SlowGuess/ACL_2020_OCR huggingface dataset:
Xinting Huang, $^{1}$  Jianzhong Qi, $^{1}$  Yu Sun, $^{2}$  Rui Zhang $^{1*}$
<sup>1</sup>The University of Melbourne, <sup>2</sup>Twitter Inc.
{xintingh@student., jianzhong.qi@, rui.zhang@}unimelb.edu.au, ysun@twitter.com
Dialogue policy optimization often obtains feedback until task completion in task-oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only provided at the end of dialogues. To address this issue, reward learning has been introduced to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. This approach requires complete state-action annotations of human-to-human dialogues (i.e., expert demonstrations), which is labor intensive. To overcome this limitation, we propose a novel reward learning approach for semi-supervised policy learning. The proposed approach learns a dynamics model as the reward function which models dialogue progress (i.e., state-action sequences) based on expert demonstrations, either with or without annotations. The dynamics model computes rewards by predicting whether the dialogue progress is consistent with expert demonstrations. We further propose to learn action embeddings for a better generalization of the reward function. The proposed approach outperforms competitive policy learning baselines on MultiWOZ, a benchmark multi-domain dataset.
Task-oriented dialogue systems complete tasks for users, such as making a restaurant reservation or finding attractions to visit, in multi-turn dialogues (Gao et al., 2018; Sun et al., 2016, 2017). Dialogue policy is a critical component in both the conventional pipeline approach (Young et al., 2013) and recent end-to-end approaches (Zhao et al., 2019). It decides the next action that a dialogue system should take at each turn. Considering its nature of sequential decision making, dialogue policy is usually learned via reinforcement learning (Su et al.,
Table 1: State Action Annotation and Utterance Example
User Side	Utterance
I would like moderate price range please.
Dialogue State annotation
Restaurant: {food=modern
european, price range=moderate}
System Side	Utterance
I found de luca cucina and riverside brasserie.
does either of them sound good for you?
System action annotation
restaurant-inform:{name=de luca
cucina, name=riiverside
brasserie}
2015; Peng et al., 2018; Zhang et al., 2019). Specifically, dialogue policy is learned by maximizing accumulated rewards over interactions with an environment (i.e., actual users or a user simulator). Handcrafted rewards are commonly used for policy learning in earlier work (Peng et al., 2018), which assigns a small negative penalty at each turn and a large positive/negative reward when the task is successful/failed. However, such reward setting does not provide sufficient supervision signals in each turn other than the last turn, which causes the sparse reward issues and may result in poorly learned policies (Takanobu et al., 2019).
To address this problem, reward function learning that relies on expert demonstrations has been introduced (Takanobu et al., 2019; Li et al., 2019b). Specifically, state-action sequences generated by an optimal policy (i.e., expert demonstrations) are collected, and a reward function is learned to give high rewards to state-action pairs that better resemble the behaviors of the optimal policy. In this way, turn-by-turn rewards estimated by the reward function can be provided to learn dialogue policy. Obtaining expert demonstrations is critical to reward function learning. Since it is impractical to assume that an optimal policy is always available,
* Rui Zhang is the corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 660-670 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
a common and reasonable approach is to treat the decision makings in human-human dialogues as optimal behaviors. To accommodate the learning of reward function, human-human dialogues need to be annotated in the form of state-action pairs from textual utterances. Table 1 illustrates an example of human-human dialogue and its state-action annotation. However, obtaining such annotations require extensive efforts and costs. Besides, a reward function based on state-action pair might cause an unstable policy learning, especially with a limited amount of annotated dialogues (Yang et al., 2018).
To address the above issues, we propose to learn dialogue policies in a semi-supervised setting where the system action of expert demonstrations only need to be partially annotated. We propose to use an implicitly trained stochastic dynamics model as the reward function to replace the conventional reward function that is restricted to state-action pairs. Dynamics models describe sequential progress using a combination of stochastic and deterministic states in a latent space, which promotes an effective tracking and forecasting (Minderer et al., 2019; Sun et al., 2019; Wang et al., 2019a). In our scenario, we train the dynamics model to describe dialogue progress of expert demonstrations. The main rationale is that the reward function should give high rewards to actions that lead to dialogue progress similar to those in expert demonstrations. This is because dialogue progress at the early stage highly influences subsequent progress, and the latter directly determines whether the task can be completed. Since the learning of dynamics model maps observations to latent states and further reason over the latent states, we are no longer restricted to fully annotated dialogues. Using dynamics model as reward function also promotes a more stable policy learning.
Learning the dynamics model in the text space is, however, prone to compounding errors due to complexities and diversities of languages. We tackle this challenge by learning the dynamics model in an action embedding space that encodes the effect of system utterances on dialogue progress. We achieve action embedding learning by incorporating an embedding function into a generative models framework for semi-supervised learning (Kingma et al., 2014). We observe that system utterances with comparable effects on dialogue progress will lead to similar state transitions (Huang et al., 2019a). Therefore, we formulate the generative
model to describe the state transition process. Using the generative model, we enrich the expert dialogues (either fully or partially annotated) with action embedding to learn the dynamics model. Moreover, we also consider the scenarios where both state and action annotations are absent in most expert dialogues, referred to as unlabeled dialogues. To expand the proposed approach to such scenarios, we further propose to model dialogue progress using action sequences and reformulate the generative model accordingly.
- To the best of our knowledge, we are the first to approach semi-supervised dialogue policy learning.
- We propose a novel reward estimation approach to dialogue policy learning which relives the requirements of extensive annotations and promotes a stable learning of dialogue policy.
- We propose an action embedding learning technique to effectively train the reward estimator from either partially labeled or unlabeled dialogues.
- We conduct extensive experiments on the benchmark multi-domain dataset. Results show that our approach consistently outperforms strong baselines coupled with semi-supervised learning techniques.
For task-oriented dialogues, a dialogue policy  $\pi(a|s)$  decides an action  $a \in \mathcal{A}$  based on the dialogue state  $s \in S$  at each turn, where  $\mathcal{A}$  and  $S$  are the predefined sets of all actions and states, respectively. Reinforcement learning is commonly applied to dialogue policy learning, where the dialogue policy model is trained to maximize cumulative rewards through interactions with environments (i.e., users):
$$
\mathcal {L} = - \mathbb {E} _ {\tau_ {i} \sim \pi} [ r (\tau) ] = - \mathbb {E} _ {\tau_ {i} \sim \pi} \left[ \sum_ {t} r \left(s _ {t}, a _ {t}\right) \right] (1)
$$
where  $\tau_{i} = \{(s_{t},a_{t})|0\leq t\leq n_{\tau}\}$  represents a sampled dialogue, and  $r(\tau_i)$  is the numerical rewards obtained in this dialogue. Instead of determining  $r(\tau_i)$  via heuristics, recent reward learning approaches train a reward function  $r_{\theta}$  to assign numerical rewards for each state-action pair. The reward function is learned from expert demonstrations  $D_{demo}$  that are dialogues sampled from an optimal policy in the form of state-action pairs. Adversarial learning is usually adopted to enforces higher rewards to state-action pairs from expert demonstrations and lower rewards to those sam
Dynamics Model
Figure 1: Overall framework of the proposed approach
pled from the learning policy (Fu et al., 2017):
$$
\mathcal {L} = - \mathbb {E} _ {\tau_ {j} \sim D _ {d e m o}} [ r _ {\theta} (\tau_ {j}) ] + \log \mathbb {E} _ {\tau_ {i} \sim \pi} \left(\frac {\exp r _ {\theta} (\tau_ {i})}{q (\tau_ {i})}\right) \tag {2}
$$
where  $\pi$  is the current dialogue policy, and  $q$  is the distribution of dialogues generated with  $\pi$ . In this way, the dialogue policy and reward function are iteratively optimized, which requires great training efforts and might lead to unstable learning results (Yang et al., 2018). Moreover, such a reward learning approach requires a complete dialogue state and system action annotation of expert demonstrations, which are expensive to obtain.
We study the problem of semi-supervised dialogue policy learning. Specifically, we consider the setting that expert demonstrations  $D_{demo}$  consist of a small number of fully labeled dialogues  $D_{\mathcal{F}}$  and partially labeled dialogues  $D_{\mathcal{P}}$ . For each fully annotated dialogue  $\tau_{i}$  in  $D_{\mathcal{F}}$ , complete annotations are available:  $\tau_{i} = \{(s_{t},a_{t},u_{t})|1\leq t\leq n_{\tau}\}$ , where  $u_{t}$  is the system utterance at turn  $t$ . Meanwhile, each partially labeled dialogue  $\tau_{j}$  in  $D_{\mathcal{P}}$  only has state annotations and system utterances:
$$
\tau_ {j} = \left\{\left(s _ {t}, u _ {t}\right) | 1 \leq t \leq n _ {\tau} \right\}.
$$
Figure 1 illustrates the overall framework of the proposed approach. Rewards are estimated by a dynamics model that consumes action embeddings  $e(a_{t})$ . Every action in the set  $\mathcal{A}$  is mapped to a fix-length embedding via a learnable embedding function  $f_{E}$ . To obtain the action embeddings for  $D_{\mathcal{P}}$  which has no action annotations, we first predict the action via a prediction model  $f_{\mathcal{A}}$  and then
transform the predicted actions to embeddings. To obtain effective action embeddings, we design a state-transition based objective to jointly optimize  $f_{E}$  and  $f_{A}$  via variational inference (Sec. 3.2). After obtaining the action embeddings, the dynamics model is learned by fitting the expert demonstrations enriched by action embeddings. Rewards are then estimated as the conditional probability of the action given the current dialogue progress encoded in latent states (Sec. 3.3). We also extend the above approach to unlabeled dialogues where both state and action annotations are absent (Sec. 3.4).
3.2 Action Learning via Generative Models
We aim to learn the prediction model  $f_{\mathcal{A}}$  and action embeddings using both  $D_{\mathcal{F}}$  and  $D_{\mathcal{P}}$ . We formulate the action prediction model as  $f_{\mathcal{A}}(a|u_t, s_t, s_{t+1})$  which takes as input the system utterance  $u_t$  and its corresponding state transition  $(s_t, s_{t+1})$ . We then introduce an mapping function:  $f_E : \mathcal{A} \to \mathcal{E}$ , where  $\mathcal{E} \subseteq \mathbb{R}^d$  is the action embedding space later used for learning the dynamics model.
We train the prediction model by proposing a variational inference approach based on a semi-supervised variational autoencoder (Semi-VAE) (Kingma et al., 2014). Semi-VAE describes the data generation process of feature-label pairs  $\{(x_{i},y_{i})|1\leq i\leq N\}$  via latent variables  $z$  as:
$$
\log p (x) = \log \sum_ {y} \int_ {z} p _ {\theta} (x, z, y) d z \tag {3}
$$
where  $p_{\theta}$  is a generative model parameterised by  $\theta$  and the class label  $y$  is treated as a latent variables for unlabeled data. Since this log-likelihood in Eqn. 3 is intractable, its variational lower bound for unlabeled data is instead optimized as:
$$
\begin{array}{l} \log p (x) \geq \mathbb {E} _ {q _ {\phi , \psi} (y, z | x)} [ \log \frac {p _ {\theta} (x , z , y)}{q _ {\phi , \psi} (y , z | x)} ] \tag {4} \\ = \mathbb {E} _ {q _ {\psi} (y | x)} [ \mathcal {L} (x, y) ] - \mathcal {H} (q _ {\psi} (y | x)) = \mathcal {U} (x) \\ \end{array}
$$
where  $q_{\phi}(z|x,y)$  and  $q_{\psi}(y|x)$  are inference models for latent variable  $z$  and  $y$  respectively, which have a factorised form  $q_{\phi,\psi}(y,z|x) = q_{\phi}(z|x,y)q_{\psi}(y|x)$ ;  $\mathcal{H}(\cdot)$  denotes causal entropy;  $\mathcal{L}(x,y)$  is the variational bound for labeled data, ans is formulated as:
$$
\begin{array}{l} \mathcal {L} (x, y) = \mathbb {E} _ {q _ {\phi} (z | x, y)} [ p _ {\theta} (x | z, y) ] + \log p (y) \tag {5} \\ - \operatorname {K L} \left(q _ {\phi} (z | x, y) | | p (z)\right) \\ \end{array}
$$
where KL is the Kullback-Leibler divergence, and  $p(y), p(z)$  are the prior distribution of  $y, z$ .
The generative model  $p_{\theta}$ , inference model  $q_{\phi}$  and  $q_{\psi}$  are optimized using both the labeled subset  $p_l$  and unlabeled subset  $p_u$  using the objective as:
$$
\mathcal {L} = \sum_ {(x, y) \sim p _ {l}} \mathcal {L} (x, y) + \sum_ {x \sim p _ {u}} \mathcal {U} (x) \tag {6}
$$
Semi-Supervised Action Prediction
We now describe the learning of action prediction model  $f_{\mathcal{A}}$  using semi-supervised expert demonstrations. We extend the semi-supervised VAE by modeling the generation process of state transitions. State transition information is indicative for action prediction and is available in both fully and partially labeled demonstrations. Thus we choose to describe the generation process of state transitions, and the optimization objective is formulated as:
$$
\begin{array}{l} \log p _ {\theta} \left(s _ {t + 1}, s _ {t}\right) = \log \sum_ {a} \int p _ {\theta} \left(s _ {t + 1}, z, s _ {t}, a\right) d z \\ = \log \sum_ {a} \int p \left(s _ {t + 1}, s _ {t} \mid , z, a\right) p (z) p (a) d z \tag {7} \\ \end{array}
$$
For partially labeled dialogues, we treat action labels as latent variables and use the action prediction model  $f_{\mathcal{A}}(a|u_t,s_t,s_{t + 1})$  to infer the value (which is denoted as  $f_{\mathcal{A}}(a|\cdot)$  later for simplicity). The variational bound of Eqn. 7 is derived as:
$$
\begin{array}{l} \mathcal {U} \left(s _ {t + 1}, s _ {t}\right) = \mathbb {E} _ {f _ {\mathcal {A}} (a | \cdot)} [ \mathcal {L} \left(s _ {t + 1}, s _ {t}, a\right) ] \tag {8} \\ - \mathcal {H} (f _ {\mathcal {A}} (a | \cdot)) \\ \end{array}
$$
where  $\mathcal{L}(s_{t + 1}, s_t, a_t)$  is the variational bound for demonstrations with action labels and is derived as:
$$
\begin{array}{l} \mathcal {L} \left(s _ {t + 1}, s _ {t}, a\right) = \mathbb {E} _ {q _ {\phi} (z | u _ {t}, a)} \left[ p _ {\theta} \left(s _ {t + 1} \mid s _ {t}, z\right) \right] \tag {9} \\ - \operatorname {K L} \left(q _ {\phi} (z | u _ {t}, a) | | p (z)\right) \\ \end{array}
$$
where  $q_{\phi}(z|u_t,a)$  is the inference model for latent variable  $z$ . Lastly, we use fully annotated samples to form a classification loss:
$$
\mathcal {L} _ {\mathrm {c l s}} = \mathbb {E} _ {\tau_ {i} \in D _ {\mathcal {F}}} [ \log f _ {\mathcal {A}} (a | u _ {t}, s _ {t}, s _ {t + 1}) ] \tag {10}
$$
The overall objectives include the loss of fully and partially labeled demonstrations:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {a c t}} = \sum_ {\tau_ {i} \in D _ {\mathcal {F}}} \mathcal {L} \left(s _ {t + 1}, s _ {t}, a\right) + (11) \\ \sum_ {\tau_ {i} \in D _ {\mathcal {P}}} \mathcal {U} (s _ {t + 1}, s _ {t}) + \mathcal {L} _ {\mathrm {c l s}} (11) \\ \end{array}
$$
Action Embeddings Learning
We then incorporate action embedding function  $f_{E}$  into the developed semi-supervised action prediction approach. The reason to introduce action embeddings is to make the learning of reward estimator more efficient and robust. Specifically, prediction error of the action prediction model might impinge the learning of reward estimator, especially for our semi-supervised scenarios where fully labeled dialogues are limited. By mapping actions to an embedding space, 'wrongly predicted' partially labeled demonstrations can still provide sufficient knowledge and thus we could achieve better generalization over actions for reward estimation.
To this aim, we consider the inference steps in the semi-supervised learning process and utilize the ones that involve action labels, i.e., the inference models for latent variables  $z$  and  $a$ . We first specify how the action prediction model is modified to include action embeddings. Inspired by (Chandak et al., 2019), we model the action selection using Boltzmann distribution for stability during training:
$$
f _ {\mathcal {A}} \left(a \mid u _ {t}, s _ {t}, s _ {t + 1}\right) = \frac {e ^ {z _ {a} / \gamma}}{\sum_ {a ^ {\prime} \in \mathcal {A}} e ^ {z _ {a ^ {\prime} / \gamma}}} \tag {12}
$$
$$
z _ {a} = e (a) ^ {\top} g \left(u _ {t}, s _ {t}, s _ {t + 1}\right), e (a) = f _ {E} (a)
$$
where  $\gamma$  is a temperature parameter, and  $g(\cdot)$  is a function that maps the input into hidden states of the same dimension as action embeddings. We also modify the inference model for latent variable by incorporating action embeddings:
$$
q _ {\phi} (z \mid u _ {t}, a) = q _ {\phi} (z \mid u _ {t}, e (a)) \tag {13}
$$
After optimizing the action prediction model  $f_{\mathcal{A}}$  and action embedding function  $f_{E}$  jointly using the objective function Eqn. 11, we use action embeddings to enrich the expert demonstrations. For fully labeled dialogues, we map the given system action labels to corresponding embeddings and obtain  $\tau_{i} = \{(s_{t}, e(a_{t})) | 1 \leq t \leq n_{\tau}\}$ . For partially labeled dialogues, we first infer the action using prediction model:  $\tilde{a}_{t} = f_{\mathcal{A}}(u_{t}, s_{t}, s_{t+1})$ , and map the inferred action to its embedding to obtain:  $\tau_{j} = \{(s_{t}, e(\tilde{a}_{t})) | 1 \leq t \leq n_{\tau}\}$ .
3.3 Reward Estimation by Dynamics Model
We aim to learn a reward estimator based on action representations obtained from the action learning module. To achieve a more stable reward estimation than adversarial reward learning, we propose
a reward estimator based on dialogues progress. Dialogue progress describes how user goals are achieved through multistep interactions and can be modeled as dialogue state transitions. We argue that an action should be given higher rewards when it leads to similar dialogue progress (i.e., state transitions) of expert demonstrations. To this aim, we learn a model to explicitly model dialogue progress without the negative sampling required by adversarial learning, and rewards can be estimated as the local-probabilities assigned to the taken actions.
To model dialogue progress, we use variational recurrent neural network (VRNN) (Chung et al., 2015). The reason to use a stochastic dynamics model is due to the 'one-to-many' nature of task-oriented dialogues. Specifically, both user and dialogue system have multiple feasible options to proceed the dialogues which requires the modeling of uncertainty. Thus, by adding latent random variables to an RNN architecture, VRNN can provide better modeling of dialogue progress than deterministic dialogue state tracking.
VRNN has three types of variables: the observations (and here we consider action embeddings), the stochastic state  $z$ , and the deterministic hidden state  $h$ , which summarizes previous stochastic states  $z_{\leq t}$ , and previous observations  $a_{\leq t}$ . We formulate the prior stochastic states to be conditioned on previous timesteps through hidden state  $h_{t-1}$ :
$$
p \left(z _ {t} \mid a _ {<   t}, z _ {<   t}\right) = \varphi^ {\text {p r i o r}} \left(h _ {t - 1}\right) \tag {14}
$$
We obtain posterior stochastic states by incorporating the observation at the current step, i.e. action embeddings  $e(a_{t})$ :
$$
q \left(z _ {t} \mid a _ {\leq t}, z _ {<   t}\right) = \varphi^ {\mathrm {e n c}} \left(h _ {t - 1}, e \left(a _ {t}\right)\right) \tag {15}
$$
Predictions are made by decoding latent states, including both the stochastic and deterministic:
$$
p \left(e \left(a _ {t}\right) \mid z _ {\leq t}, a _ {<   t}\right) = \varphi^ {\mathrm {d e c}} \left(z _ {t}, h _ {t - 1}, s _ {t}\right) \tag {16}
$$
And lastly the deterministic states are updated as:
$$
h _ {t} = \varphi^ {\mathrm {r m n}} \left(e \left(a _ {t}\right), z _ {t}, h _ {t - 1}, s _ {t}\right) \tag {17}
$$
where  $\varphi$  are all implemented as neural networks. Note that we also make the prediction and recurrence step to condition on the dialogue state  $s_t$  to provide more information.
We train the VRNN by optimizing the evidence lower bound (ELBO) as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {V R N N}} = \mathbb {E} _ {q (z _ {t} | a _ {\le t}, z _ {<   t})} \left[ \sum_ {t} \log p (e (a _ {t}) | z _ {\le t}, a _ {<   t}) \right. \\ \left. - \operatorname {K L} \left(q \left(z _ {t} \mid a _ {\leq t}, z _ {<   t}\right) \mid \mid p \left(z _ {t} \mid a _ {<   t}, z _ {<   t}\right)\right) \right] \tag {18} \\ \end{array}
$$
The rewards are estimated as the conditional probability given the hidden state of VRNN, which encodes the current dialogue progress:
$$
r \left(s _ {\leq t}, a _ {t}\right) = \log p _ {\varphi^ {\mathrm {d e c}}} \left(a _ {t} \mid a _ {<   t}, s _ {\leq t}\right) \tag {19}
$$
where  $p_{\varphi^{\mathrm{dec}}}$  is the probability given to the selected action based on the decoding step of VRNN (Eqn. 16). The larger this conditional probability is, the more similar the dialogue progress this action leads to imitates the expert demonstrations. The proposed reward estimation is agnostic to the choice of policy, and various approaches (e.g., Deep Q-learning, Actor-Critic) can be optimized by plugging into the policy learning objective (Eqn. 1).
3.4 Expanding to Unlabeled Corpus
We further describe how to expand the proposed model, including action learning and reward estimation modules, to utilize unlabeled expert demonstrations. Formally, we consider the setting that we have fully labeled dialogues  $D_{\mathcal{F}}$  and unlabeled dialogues  $D_{\mathcal{U}}$ . For each dialogue in  $D_{\mathcal{U}}$ , only textual conversations are provided and neither of state and action labels are available:  $\tau_{j} = \{(c_{t},u_{t})|1\leq t\leq n_{\tau}\}$ , where  $c_{t}$  is the context and consists of the dialogue history of both user and system utterances.
With the absence of dialogue state information, we formulate the action prediction model as  $f_{\mathcal{A}}(a|u_t,u_{t - 1},u_{t + 1})$ . This formulation can be considered as an application of Skip-Thought (Kiros et al., 2015), which originally utilizes contextual sentences as supervision signals. In our scenarios, we instead utilize the previous and next system utterances to provide more indicative information for action prediction.
We also build the joint learning of action prediction model the action embeddings on semi-supervised VAE framework. Instead of modeling state transitions, we choose the process of response generation to fully utilize unlabeled dialogues:
$$
\begin{array}{l} \log p _ {\theta} (u _ {t}) = \log \sum_ {a} \int p _ {\theta} (u _ {t}, z, a) d z \tag {20} \\ = \log \sum_ {a} \int p _ {\theta} \left(u _ {t} \mid z, a _ {t}\right) p (z) p (a) d z \\ \end{array}
$$
System action labels are treated as latent variables for unlabeled dialogues, and the variational bond is derived as:
$$
\mathcal {U} (u _ {t}) = \mathbb {E} _ {f _ {\mathcal {A}} (a | \cdot)} [ \mathcal {L} (u _ {t}, a) ] - \mathcal {H} (f _ {\mathcal {A}} (a | \cdot)) \tag {21}
$$
where  $\mathcal{L}(u_t,a)$  is variational bound for fully labeled dialogues: