Venue: NAACL 2019

# Rethinking Action Spaces for Reinforcement Learning in End-to-end

Dialog Agents with Latent Variable Models

 Tiancheng Zhao\({}^{1}\), Kaige Xie\({}^{2}\) and Maxine Eskenazi\({}^{1}\)

\({}^{1}\)Language Technologies Institute, Carnegie Mellon University

\({}^{2}\)Shanghai Jiao Tong University

{tianchezz, max+}@cs.cmu.edu, lightyear0117@sjtu.edu.cn

###### Abstract

Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have their own limitations. This paper proposes a novel latent action framework that treats the action spaces of an end-to-end dialog agent as latent variables and develops unsupervised methods in order to induce its own action space from the data. Comprehensive experiments are conducted examining both continuous and discrete action types and two different optimization methods based on stochastic variational inference. Results show that the proposed latent actions achieve superior empirical performance improvement over previous word-level policy gradient methods on both DealOrNoDeal and MultiWoz dialogs. Our detailed analysis also provides insights about various latent variable approaches for policy learning and can serve as a foundation for developing better latent actions in future research. 1

Footnote 1: Data and code are available at [https://github.com/snakeztc/NeuralDiago-LaRL](https://github.com/snakeztc/NeuralDiago-LaRL)

## 1 Introduction

Optimizing dialog strategies in multi-turn dialog models is the cornerstone of building dialog systems that more efficiently solve real-world challenges, e.g. providing information Young (2006), winning negotiations Lewis et al. (2017), improving engagement Li et al. (2016) etc. A classic solution employs reinforcement learning (RL) to learn a dialog policy that models the optimal action distribution conditioned on the dialog state Williams and Young (2007). However, since there are infinite human language possibilities, an enduring challenge has been to define what the action space is. For traditional modular systems, the action space is defined by hand-crafted semantic representations such as dialog acts and slot-values Raux et al. (2005); Chen et al. (2013) and the goal is to obtain a dialog policy that chooses the best hand-crafted action at each dialog turn. But it is limited because it can only handle simple domains whose entire action space can be captured by hand-crafted representations Walker (2000); Su et al. (2017). This cripples a system's ability to handle conversations in complex domains.

Conversely, end-to-end (E2E) dialog systems have removed this limit by directly learning a response generation model conditioned on the dialog context using neural networks Vinyals and Le (2015); Sordoni et al. (2015). To apply RL to E2E systems, the action space is typically defined as the entire vocabulary; every response output word is considered to be an action selection step Li et al. (2016), which we denote as the word-level RL. Word-level RL, however, has been shown to have several major limitations in learning dialog strategies. The foremost one is that direct application of word-level RL leads to degenerate behavior: the response decoder deviates from human language and generates utterances that are incomprehensible Lewis et al. (2017); Das et al. (2017); Kottur et al. (2017). A second issue is that since a multi-turn dialog can easily span hundreds of words, word-level RL suffers from credit assignment over a long horizon, leading to slow and sub-optimal convergence Kaelbling et al. (1996); He et al. (2018).

This paper proposes Latent Action Reinforcement Learning (LaRL), a novel framework that overcomes the limitations of word-level RL for E2E dialog models, marrying the benefits of a traditional modular approach in an unsupervised manner. The key idea is to develop E2E models that can invent their own discourse-level ac

tions. These actions must be expressive enough to capture response semantics in complex domains (i.e. have the capacity to represent a large number of actions), thus decoupling the discourse-level decision-making process from natural language generation. Then any RL technique can be applied to this induced action space in the place of word-level output. We propose a flexible latent variable dialog framework and investigate several approaches to inducing latent action space from natural conversational data. We further propose (1) a novel training objective that outperforms the typical evidence lower bound used in dialog generation and (2) an attention mechanism for integrating discrete latent variables in the decoder to better model long responses.

We test this on two datasets, DealOrNoDeal Lewis et al. (2017) and MultiWoz Budzianowski et al. (2018), to answer two key questions: (1) what are the advantages of LaRL over Word-level RL and (2) what effective methods can induce this latent action space. Results show that LaRL is significantly more effective than word-level RL for learning dialog policies and it does not lead to incomprehensible language generation. Our models achieve 18.2% absolute improvement over the previous state-of-the-art on MultiWoz and discover novel and diverse negotiation strategies on DealOrNoDeal. Besides strong empirical improvement, our model analysis reveals novel insights, e.g. it is crucial to reduce the exposure bias in the latent action space and discrete latent actions are more suitable than continuous ones to serve as action spaces for RL dialog agents.

## 2 Related Work

Prior RL research in modular dialog management has focused on policy optimization over hand-crafted action spaces in task-oriented domains Walker (2000); Young et al. (2007). A dialog manager is formulated as a Partially Observable Markov Decision Process (POMDP) Young et al. (2013), where the dialog state is estimated via dialog state tracking models from the raw dialog context Lee (2013); Henderson et al. (2014); Ren et al. (2018). RL techniques are then used to find the optimal dialog policy Gasic and Young (2014); Su et al. (2017); Williams et al. (2017). Recent deep-learning modular dialog models have also explored joint optimization over dialog policy and state tracking to achieve stronger performance Wen et al. (2016); Zhao and Eskenazi (2016); Liu and Lane (2017).

A related line of work is reinforcement learning for E2E dialog systems. Due to the flexibility of encoder-decoder dialog models, prior work has applied reinforcement learning to more complex domains and achieved higher dialog-level rewards, such as open-domain chatting Li et al. (2016); Serban et al. (2017a), negotiation Lewis et al. (2017), visual dialogs Das et al. (2017), grounded dialog Mordatch and Abbeel (2017) etc. As discussed in Section 1, these methods consider the output vocabulary at every decoding step to be the action space; they suffer from limitations such as deviation from natural language and sub-optimal convergence.

Finally, research in latent variable dialog models is closely related to our work, which strives to learn meaningful latent variables for E2E dialog systems. Prior work has shown that learning with latent variables leads to benefits like diverse response decoding Serban et al. (2017b); Zhao et al. (2017); Cao and Clark (2017), interpretable decision-making Wen et al. (2017); Zhao et al. (2018) and zero-shot domain transfer Zhao and Eskenazi (2018). Also, driven by similar motivations of this work, prior studies have explored to utilize a coarse discrete node, either handcrafted or learned, to decouple the word generation process from dialog policy in E2E systems for better dialog policy He et al. (2018); Yarats and Lewis (2017). Our work differs from prior work for two reasons: (1) latent action in previous work is only auxiliary, small-scale and mostly learned in a supervised or semi-supervised setting. This paper focuses on unsupervised learning of latent variables and learns variables that are expressive enough to capture the entire action space by itself. (2) to our best knowledge, our work is the first comprehensive study of the use of latent variables for RL policy optimization in dialog systems.

## 3 Baseline Approach

E2E response generation can be treated as a conditional language generation task, which uses neural encoder-decoders Cho et al. (2014) to model the conditional distribution \(p({\bf x}|{\bf c})\) where \({\bf c}\) is the observed dialog context and \({\bf x}\) is the system's response to the context. The format of the dialog context is domain dependent. It can vary from tex 

tual raw dialog history (Vinyals and Le, 2015) to visual and textual context Das et al. (2017). Training with RL usually has 2 steps: supervised pretraining and policy gradient reinforcement learning Williams and Zweig (2016); Dhingra et al. (2017); Li et al. (2016). Specifically, the supervised learning step maximizes the log likelihood on the training dialogs, where \(\theta\) is the model parameter:

\[{\mathcal{L}_{SL}}(\theta)=\mathbb{E}_{\mathbf{x},\mathbf{c}}[\log p_{\theta}( \mathbf{x}|\mathbf{c})] \tag{1}\]

Then the following RL step uses policy gradients, e.g. the REINFORCE algorithm Williams (1992) to update the model parameters with respect to task-dependent goals. We assume that we have an environment that the dialog agent can interact with and that there is a turn-level reward \(r_{t}\) at every turn \(t\) of the dialog. We can then write the expected discounted return under a dialog model \(\theta\) as \(J(\theta)=\mathbb{E}[\sum^{T}_{0}\gamma^{t}r_{t}]\), where \(\gamma\in[0,1]\) is the discounting factor and \(T\) is the length of the dialog. Often a baseline function \(b\) is used to reduce the variance of the policy gradient Greensmith et al. (2004), leading to \(R_{t}=\sum^{T-t}_{k=0}\gamma^{k}(r_{t+k}-b)\).

**Word-level Reinforcement Learning**: as shown in Figure 1, the baseline approach treats every output word as an action step and its policy gradient is:

\[\nabla_{\theta}J(\theta)=\mathbb{E}_{\theta}[\sum^{T}_{t=0}\sum^{U_{t}}_{j=0 }R_{tf}\nabla_{\theta}\log p_{\theta}(w_{tfj}|w_{<t>},\mathbf{c}_{t})] \tag{2}\]

where \(U_{t}\) is the number of tokens in the response at turn \(t\) and \(j\) is the word index in the response. It is evident that Eq 2 has a very large action space, i.e. \(|V|\) and a long learning horizon, i.e. \(TU\). Prior work has found that the direct application of Eq 2 leads to divergence of the decoder. The common solution is to alternate with supervised learning with Eq 2 at a certain ratio Lewis et al. (2017). We denote this ratio as RL:SL=A:B, which means for every A policy gradient updates, we run B supervised learning updates. We use RL:SL=off for the case where only policy gradients are used and no supervised learning is involved.

## 4 Latent Action Reinforcement Learning

We now describe the proposed LaRL framework. As shown in Figure 1, a latent variable \(\mathbf{z}\) is introduced in the response generation process. The conditional distribution is factorized into \(p(\mathbf{x}|\mathbf{c})=p(\mathbf{x}|\mathbf{z})p(\mathbf{z}|\mathbf{c})\) and the generative story is: (1) given a dialog context \(\mathbf{c}\) we first sample a latent action \(\mathbf{z}\) from \(p_{\theta_{e}}(\mathbf{z}|\mathbf{c})\) and (2) generate the response by sampling \(\mathbf{x}\) based on \(\mathbf{z}\) via \(p_{\theta_{d}}(\mathbf{x}|\mathbf{z})\), where \(p_{\theta_{e}}\) is the dialog encoder network and \(p_{\theta_{d}}\) is the response decoder network. Given the above setup, LaRL treats the latent variable \(\mathbf{z}\) as its action space instead of outputting words in response \(\mathbf{x}\). We can now apply REINFORCE in the latent action space:

\[\nabla_{\theta}J(\theta)=\mathbb{E}_{\theta}[\sum^{T}_{t=0}R_{t}\log p_{\theta }(\mathbf{z}|\mathbf{c}_{t})] \tag{3}\]

Compared to Eq 2, LaRL differs by:

* Shortens the horizon from \(TU\) to \(T\).
* Latent action space is designed to be low-dimensional, much smaller than \(V\).
* The policy gradient only updates the encoder \(\theta_{e}\) and the decoder \(\theta_{d}\) stays intact.

These properties reduce the difficulties for dialog policy optimization and decouple high-level decision-making from natural language generation. The \(p_{\theta_{e}}\) are responsible for choosing the best latent action given a context \(\mathbf{c}\) while \(p_{\theta_{d}}\) is only responsible for transforming \(\mathbf{z}\) into the surface-form words. Our formulation also provides a flexible framework for experimenting with various types of model learning methods. In this paper, we focus on two key aspects: the type of latent variable

Figure 1: High-level comparison between word-level and latent-action reinforcement learning in a sample multiturn dialog. The decoder network generates the response given the latent code \(\mathbf{z}\). Dashed line denotes places where policy gradients from task rewards are applied to the model.

 

\({\bf z}\) and optimization methods for learning \({\bf z}\) in the supervised pre-training step.

### Types of Latent Actions

Two types of latent variables have been used in previous research: continuous isotropic Gaussian distribution Serban et al. (2017b) and multivariate categorical distribution Zhao et al. (2018). These two types are both compatible with our LaRL framework and can be defined as follows:

**Gaussian Latent Actions** follow \(M\) dimensional multivariate Gaussian distribution with a diagonal covariance matrix, i.e. \({\bf z}\sim\mathcal{N}(\mathbf{\mu},\mathbf{\sigma}^{2}{\bf I})\). Let the encoder \(p_{\theta_{e}}\) consist of two parts: a context encoder \(\mathcal{F}\), a neural network that encodes the dialog context \({\bf c}\) into a vector representation \({\bf h}\), and a feed forward network \(\pi\) that projects \({\bf h}\) into \(\mathbf{\mu}\) and \(\mathbf{\sigma}\). The process is defined as follows:

\[{\bf h}=\mathcal{F}({\bf c}) \tag{4}\] \[\begin{bmatrix}\mathbf{\mu}\\ \log(\mathbf{\sigma}^{2})\end{bmatrix}=\pi({\bf h})\] (5) \[p({\bf x}|{\bf z})=p_{\theta_{d}}({\bf z})\quad{\bf z}\sim \mathcal{N}(\mathbf{\mu},\mathbf{\sigma}^{2}{\bf I}) \tag{6}\]

where the sampled \({\bf z}\) is used as the initial state of the decoder for response generation. Also we use \(p_{\theta}({\bf z}|{\bf c})=\mathcal{N}({\bf z};\mathbf{\mu},\mathbf{\sigma}^{2}{\bf I})\) to compute the policy gradient update in Eq 3.

**Categorical Latent Actions** are \(M\) independent K-way categorical random variables. Each \({\bf z}_{m}\) has its own token embeddings to map latent symbols into vector space \({\bf E}_{m}\in\mathbb{R}^{K\times D}\) where \(m\in[1,M]\) and \(D\) is the embedding size. Thus \(M\) latent actions can represent exponentially, \(K^{M}\), unique combinations, making it expressive enough to model dialog acts in complex domains. Similar to Gaussian Latent Actions, we have

\[{\bf h}=\mathcal{F}({\bf c}) \tag{7}\] \[p(Z_{m}|{\bf c})=\text{softmax}(\pi_{m}({\bf h}))\] (8) \[p({\bf x}|{\bf z})=p_{\theta_{d}}({\bf E}_{1:M}({\bf z}_{1:M})) \quad{\bf z}_{m}\sim p(Z_{m}|{\bf c}) \tag{9}\]

For the computing policy gradient in Eq 3, we have \(p_{\theta}({\bf z}|{\bf c})=\prod_{m=1}^{M}p(Z_{m}={\bf z}_{m}|{\bf c})\)

Unlike Gaussian latent actions, a matrix \(\mathbb{R}^{M\times D}\) comes after the embedding layers \({\bf E}_{1:M}({\bf z}_{1:M})\), whereas the decoder's initial state is a vector of size \(\mathbb{R}^{D}\). Previous work integrated this matrix with the decoder by summing over the latent embeddings, i.e. \({\bf x}=p_{\theta_{d}}(\sum_{1}^{M}{\bf E}_{m}({\bf z}_{m}))\), denoted as **Summation Fusion** for later discussion Zhao et al. (2018). A limitation of this method is that it could lose fine-grained order information in each latent dimension and have issues with long responses that involve multiple dialog acts. Therefore, we propose a novel method, **Attention Fusion**, to combine categorical latent actions with the decoder. We apply the attention mechanism Luong et al. (2015) over latent actions as the following. Let \(i\) be the step index during decoding. Then we have:

\[\alpha_{mi} =\text{softmax}({\bf h}_{i}^{T}{\bf W}_{a}{\bf E}_{m}({\bf z}_{ m})) \tag{10}\] \[{\bf c}_{i} =\sum_{m=1}^{M}\alpha_{mi}{\bf E}_{m}({\bf z}_{m})\] (11) \[\widetilde{{\bf h}}_{i} =\text{tanh}({\bf W}_{s}\begin{bmatrix}{\bf h}_{i}\\ {\bf c}_{i}\end{bmatrix})\] (12) \[p(w_{i}|{\bf n}_{i},{\bf c}_{i}) =\text{softmax}(W_{o}\widetilde{{\bf h}}_{i}) \tag{13}\]

The decoder's next state is updated by \({\bf h}_{i+1}=\text{RNN}({\bf h}_{i},w_{i+1}),\widetilde{{\bf h}}_{i})\) and \({\bf h}_{0}\) is computed via summation-fusion. Thus attention fusion lets the decoder focus on different latent dimensions at each generation step.

### Optimization Approaches

**Full ELBO**: Now given a training dataset \(\{{\bf x},{\bf c}\}\), our base optimization method is via stochastic variational inference by maximizing the evidence lowerbound (ELBO), a lowerbound on the data log likelihood:

\[\mathcal{L}_{full}(\theta)=p_{q({\bf z}|{\bf x},{\bf c})}({\bf x}|{\bf z})- \text{DKL}[q({\bf z}|{\bf x},{\bf c})\lVert p({\bf z}|{\bf c})] \tag{14}\]

where \(q_{\gamma}({\bf z}|{\bf x},{\bf c})\) is a neural network that is trained to approximate the posterior distribution \(q({\bf z}|{\bf x},{\bf c})\) and \(p({\bf z}|{\bf c})\) and \(p({\bf x}|{\bf z})\) are achieved by \(\mathcal{F}\), \(\pi\) and \(p_{\theta_{d}}\). For Gaussian latent actions, we use the reparametrization trick Kingma and Welling (2013) to backpropagate through Gaussian latent actions and the Gumbel-Softmax Jang et al. (2016) to backpropagate through categorical latent actions.

**Lite ELBO**: a major limitation is that Full ELBO can suffer from exposure bias at latent space, i.e. the decoder only sees \({\bf z}\) sampled from \(q({\bf z}|{\bf x},{\bf c})\) and never experiences \({\bf z}\) sampled from \(p_{\theta}({\bf z}|{\bf c})\), which is always used at testing time. Therefore, in this paper, we propose a simplified ELBO for encoder-decoder models with stochastic latent variables:

\[\mathcal{L}_{lite}(\theta)=p_{p({\bf z}|{\bf c})}({\bf x}|{\bf z})-\beta\text{D KL}[p({\bf z}|{\bf c}))\lVert p({\bf z})]\] (15) 

Essentially this simplified objective sets the posterior network the same as our encoder, i.e. \(q_{\gamma}(\mathbf{z|x,c})=p_{\theta_{e}}(\mathbf{z|c})\), which makes the KL term in Eq 14 zero and removes the issue of exposure bias. But this leaves the latent spaces unregularized and our experiments show that if we only maximize \(p_{p(\mathbf{z|c})}(\mathbf{x|z})\) there is overfitting. For this, we add the additional regularization term \(\beta\mathcal{D}_{\text{KL}}[p(\mathbf{z|c}))||p(\mathbf{z})]\) that encourages the posterior be similar to certain prior distributions and \(\beta\) is a hyper-parameter between 0 and 1. We set the \(p(\mathbf{z})\) for categorical latent actions to be uniform, i.e. \(p(\mathbf{z})=1/K\), and set the prior for Gaussian latent actions to be \(\mathcal{N}(\mathbf{0},\mathbf{I})\), which we will show that are effective.

## 5 Experiment Settings

### DealOrNoDeal Corpus and RL Setup

DealOrNoDeal is a negotiation dataset that contains 5805 dialogs based on 2236 unique scenarios Lewis et al. (2017). We hold out 252 scenarios for testing environment and randomly sample 400 scenarios from the training set for validation. The results are evaluated from 4 perspectives: Perplexity (PPL), Reward, Agree and Diversity. PPL helps us to identify which model produces the most human-like responses, while Reward and Agree evaluate the model's negotiation strength. Diversity indicates whether the model discovers a novel discourse-level strategy or just repeats dull responses to compromise with the opponent. We closely follow the original paper and use the same reward function and baseline calculation. At last, to have a fair comparison, all the compared models shared the identical judge model and user simulator, which are a standard hierarchical encoder-decoder model trained with Maximum Likelihood Estimation (MLE).

### Multi-Woz Corpus and Novel RL Setup

Multi-Woz is a slot-filling dataset that contains 10438 dialogs on 6 different domains. 8438 dialogs are for training and 1000 each are for validation and testing. Since no prior user simulator exists for this dataset, for a fair comparison with the previous state-of-the-art we focus on the Dialog-Context-to-Text Generation task proposed in Budzianowski et al. (2018). This task assumes that the model has access to the ground-truth dialog belief state and is asked to generate the next response at every system turn in a dialog. The results are evaluated from 3 perspectives: BLEU, Inform Rate and Success Rate. The BLEU score checks the response-level lexical similarity, while Inform and Success Rate measure whether the model gives recommendations and provides all the requested information at dialogue level. Current state-of-the-art results struggle in this task and MLE models only achieve 60% success Budzianowski et al. (2018). To transform this task into an RL task, we propose a novel extension to the original task as follows:

1. For each RL episode, randomly sample a dialog from the training set
2. Run the model on every system turn, and do not alter the original dialog context at every turn given the generated responses.
3. Compute Success Rate based on the generated responses in this dialog.
4. Compute policy gradient using Eq 3 and update the parameters.

This setup creates a variant RL problem that is similar to the Contextual Bandits Langford and Zhang (2008), where the goal is to adjust its parameters to generate responses that yield better Success Rate. Our results show that this problem is challenging and that word-level RL falls short.

### Language Constrained Reward (LCR) curve for Evaluation

It is challenging to quantify the performance of RL-based neural generation systems because it is possible for a model to achieve high task reward and yet not generate human language Das et al. (2017). Therefore, we propose a novel measure, the Language Constrained Reward (LCR) curve as an additional robust measure. The basic idea is to use an ROC-style curve to visualize the tradeoff between achieving higher reward and being faithful to human language. Specifically, at each checkpoint \(i\) over the course of RL training, we record two measures: (1) the PPL of a given model on the test data \(p_{i}=\mathrm{PPL}(\theta_{i})\) and (2) this model's average cumulative task reward in the test environment \(R^{t}_{i}\). After RL training is complete, we create a 2D plot where the x-axis is the maximum PPL allowed, and the y-axis is the best achievable reward within the PPL budget in the testing environments:

\[y=\textrm{max}_{i}R^{t}_{i}\quad\mbox{subject to}\quad p_{i}<x\] (16) 

As a result, a perfect model should lie in the upper left corner whereas a model that sacrifices language quality for higher reward will lie in the lower right corner. Our results will show that the LCR curve is an informative and robust measure for model comparison.

## 6 Results: Latent Actions or Words?

We have created 6 different variations of latent action dialog models under our LaRL framework. To demonstrate the advantages of LaRL, during model can only increase its task rewards by deviating significantly from natural language.

Closer analysis shows the word-level baseline severely overfits to the user simulator. The caveat is that the word-level models have in fact discovered a loophole in the simulator by insisting on 'hat' and 'ball' several times and the user model eventually yields to agree to the deal. This is reflected in the diversity measure, which is the number of unique responses that a model uses in all 200 testing scenarios. As shown in Figure 3, after RL training, the diversity of the baseline model drops to only 5. It is surprising that the agent can achieve high reward with a well-trained HRED user simulator using only 5 unique utterances. On the contrary, LiteCat increases its response diversity after RL training from 58 to 202, suggesting that LiteCat discovers novel discourse-level strategies in order to win the negotiation instead of exploiting local loopholes in the same user simulator. Our qualitative analysis confirms this when we observe that our LiteCat model is able to use multiple strategies in negotiation, e.g. elicit preference question, request different offers, insist on key objects etc. See supplementary material for example conversations.

\begin{table}
\begin{tabular}{l|c c c} \hline Model & Var Type & Loss & Integration \\ \hline Gauss & Gaussian & \(\mathcal{L}_{full}\) & / \\ Cat & Categorical & \(\mathcal{L}_{full}\) & sum \\ AttnCat & Categorical & \(\mathcal{L}_{full}\) & attn \\ LiteGauss & Gaussian & \(\mathcal{L}_{lite}\) & / \\ LiteCat & Categorical & \(\mathcal{L}_{lite}\) & subm \\ LiteAttnCat & Categorical & \(\mathcal{L}_{lite}\) & attn \\ \hline \end{tabular}
\end{table}
Table 1: All proposed variations of LaRL models.

Figure 2: LCR curves on DealOrNoDeal dataset.

 



### MultiWoz

For MultiWoz, we reproduce results from (Budzianowski et al., 2018) as the baseline. After RL training, the best LaRL model is LiteAttnCat and the best word-level model is word RL:SL=off. Table 3 shows that LiteAttnCat is on par with the baseline in the supervised learning step, showing that multivariate categorical latent variables alone are powerful enough to match with continuous hidden representations for modeling dialog actions. For performance after RL training, LiteAttnCat achieves near-human performance in terms of success rate and inform rate, obtaining 18.24% absolute improvement over the MLE-based state-of-the-art (Budzianowski et al., 2018). More importantly, perplexity only slightly increases from 4.05 to 5.22. On the other hand, the word-level RL's success rate also improves to 79%, but the generated responses completely deviate from natural language, increasing perplexity from 3.98 to 17.11 and dropping BLEU from 18.9 to 1.4.

Figure 4 shows the LCR curves for MultiWoz, with a trend similar to the previous section: the word-level models can only achieve task reward improvement by sacrificing their response decoder PPL. Figure 4 also shows the LCR curve for the baseline trained with RL:SL=100:1, hoping that supervised learning can force the model to conform to natural language. While PPL and BLEU are indeed improved, it also limits final reward performance. The latent-level models, on the contrary, do not suffer from this tradeoff. We also observe that LiteAttnCat consistently outperforms LiteCat on MultiWoz, confirming the effectiveness of Attention Fusion for handling long dialog responses with multiple entities and dialog acts. Lastly, Table 4 qualitatively exhibits the generation differences between the two approaches. The RL:SL=off model learns to continuously output entities to fool the evaluation script for high success rate, whereas LiteCatAttn learns to give more information while maintaining the language quality.

\begin{table}
\begin{tabular}{p{65.0pt}|p{142.3pt}} \hline
**Context** & **Sys I have [value_count] trains matching your request . Is there a specific day and time you would like to travel? **Usr l** would like to leave on [value_day] and arrive by [value_time]. \\ \hline
**Model** & **Generated Response** \\ \hline word RL:SL=off & [train.id] is leaving [value_place] on [value_day] on [train_id] [value_count] [train_id] leaving ... \\ \hline word RL:SL=100 & [train.id] leaves at [value_time] . would you like me to book you a ticket ? \\ \hline LiteAttnCat & [train.id] leaves [value_place] at [value_time] and arrives in [value_place] at [value_time]. Would you like me to book that for you ? \\ \hline \end{tabular}
\end{table}
Table 4: Example responses from baselines and LiteCatAttn on MultiWoz.

Figure 4: LCR curves on the MultiWoz dataset.

\begin{table}
\begin{tabular}{l|c c c c} \hline  & PPL & BLEU & Inform & Success \\ \hline Human & / & / & 90\% & 82.3\% \\ \hline Baseline & 3.98 & 18.9 & 71.33\% & 60.96\% \\ LiteAttnCat & 4.05 & 19.1 & 67.98\% & 57.36\% \\ \hline Baseline & 17.11 & 1.4 & 80.5\% & 79.07\% \\ +RL & & & & \\ LiteAttnCat & **5.22** & **12.8** & **82.78\%** & **79.2\%** \\ +RL & & & & \\ \hline \end{tabular}
\end{table}
Table 3: Main results on MultiWoz test set. RL models are chosen based on performance on the validation set.

Figure 3: Response diversity and task reward learning curve over the course of RL training for both word RL:SL=4:1 (left) and LiteCat (right).





## 7 Model Analysis

We compare the 6 variants of latent action models on DealOrNoDeal and MultiWoz. Table 5 shows performance of the models that are pre-trained only with supervised learning. Figure 5 shows LCR curves for the 3 models pre-trained with \({\mathcal{L}_{lite}}\) and fine-tuned with policy gradient reinforcement learning. The following are the main findings based on these results.

\({\mathcal{L}_{lite}}\)**outperforms \({\mathcal{L}_{full}}\) as a pre-train objective.** Table 5 shows that models with \({\mathcal{L}_{full}}\) fall behind their Lite counterparts on PPL and BLEU. We attribute this to the exposure bias in the latent space, i.e. the decoder is not trained to consider the discrepancy between the posterior network and actual dialog policy network. Meanwhile, the full models tend to enjoy higher diversity at pre-training, which agrees with the diversity-promoting effect observed in prior research Zhao et al. (2017). However, our previous discussion on Figure 3 shows that Lite models are able to increase their response diversity in order to win more in negotiation through RL training. This is fundamentally different from diversity in pre-training, since diversity in LaRL is optimized to improve task reward, rather than to better model the original data distribution. Table 6 shows the importance of latent space regularization. When \(\beta\) is 0, both LiteCat and LiteGauss reach suboptimal policies with final reward that are much smaller than the regularized versions (\(\beta=0.01\)). The reason behind this is that the unregularized pretrained policy has very low entropy, which prohibits sufficient exploration in the RL stage.

**Categorical latent actions outperform Gaussian latent actions.** Models with discrete actions consistently outperform models with Gaussian ones. This is surprising since continuously distributed representations are a key reason for the success of deep learning in natural language processing. Our finding suggests that (1) multivariate categorical distributions are powerful enough to model complex natural dialog responses semantics, and can achieve on par results with Gaussian or non-stochastic continuous representations. (2) categorical variables are a better choice to serve as action spaces for reinforcement learning. Figure 5 shows that Lite(Attn)Cat easily achieves strong rewards while LiteGauss struggles to improve its reward. Also, applying REINFORCE on Gaussian latent actions is unstable and often leads to model divergence. We suspect the reason for this is the unbounded nature of continuous latent space: RL exploration in the continuous space may lead to areas in the manifold that are not covered in supervised training, which causes undefined decoder behavior given \(\mathbf{z}\) in these unknown areas.

Figure 5: LCR curves on DealOrNoDeal and MultiWoz. Models with \({\mathcal{L}_{full}}\) are not included because their PPLs are too poor to compare to the Lite models.

\begin{table}
\begin{tabular}{l c c|c c c} \hline \(\beta\) & 0.0 & 0.01 & \(\beta\) & 0.0 & 0.01 \\ \hline LiteCat & 4.23 & 7.27 & LiteGauss & 4.83 & 6.67 \\ \hline \end{tabular}
\end{table}
Table 6: Best rewards in test environments on DealOrNoDeal with various \(\beta\).





## 8 Conclusion and Future Work

In conclusion, this paper proposes a latent variable action space for RL in E2E dialog agents. We present a general framework with a regularized ELBO objective and attention fusion for discrete variables. The methods are assessed on two dialog tasks and analyzed using the proposed LCR curve. Results show our models achieve superior performance and create a new state-of-the-art success rate on MultiWoz. Extensive analyses enable us to gain insight on how to properly train latent variables that can serve as the action spaces for dialog agents. This work is situated in the approach concerning practical latent variables in dialog agents, being able to create action abstraction in an unsupervised manner. We believe that our findings are a basic first step in this promising research direction.

## References

* Budzianowski et al. (2018) Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5016-5026.
* Cao and Clark (2017) Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, volume 2, pages 182-187.
* Chen et al. (2013) Yun-Nung Chen, William Yang Wang, and Alexander I Rudnicky. 2013. Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In _Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on_, pages 120-125. IEEE.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_.
* Das et al. (2017) Abhishek Das, Satwik Kottur, Jose MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In _Computer Vision (ICCV), 2017 IEEE International Conference on_, pages 2970-2979. IEEE.
* Dhingra et al. (2017) Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. Towards end-to-end reinforcement learning of dialogue agents for information access. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 484-495.
* Gasic and Young (2014) Milica Gasic and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 22(1):28-40.
* Greensmith et al. (2004) Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004. Variance reduction techniques for gradient estimates in reinforcement learning. _Journal of Machine Learning Research_, 5(Nov):1471-1530.
* He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2333-2343.
* Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_, pages 292-299.
* Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. _arXiv preprint arXiv:1611.01144_.
* Kaelbling et al. (1996) Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. _Journal of artificial intelligence research_, 4:237-285.
* Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. _arXiv preprint arXiv:1312.6114_.
* Kottur et al. (2017) Satwik Kottur, Jose Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge naturallyin multi-agent dialog. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2962-2967.
* Langford and Zhang (2008) John Langford and Tong Zhang. 2008. The epoch-greedy algorithm for multi-armed bandits with side information. In _Advances in neural information processing systems_, pages 817-824.
* Lee (2013) Sungjin Lee. 2013. Structured discriminative model for dialog state tracking. In _Proceedings of the SIGDIAL 2013 Conference_, pages 442-451.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Daquphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2443-2453.
* Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202

Bing Liu and Ian Lane. 2017. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. _Proc. Interspeech 2017_, pages 2506-2510.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Mordatch and Abbeel (2017) Igor Mordatch and Pieter Abbeel. 2017. Emergence of grounded compositional language in multi-agent populations. _arXiv preprint arXiv:1703.04908_.
* Raux et al. (2005) Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, and Maxine Eskenazi. 2005. Lets go public! taking a spoken dialog system to the real world. In _in Proc. of Interspeech 2005_. Citeseer.
* Ren et al. (2018) Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2780-2786.
* Serban et al. (2017a) Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. 2017a. A deep reinforcement learning chatbot. _arXiv preprint arXiv:1709.02349_.
* Serban et al. (2016) Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16)_.
* Serban et al. (2017b) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017b. A hierarchical latent variable encoder-decoder model for generating dialogues. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 196-205.
* Su et al. (2017) Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 147-157.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. _arXiv preprint arXiv:1506.05869_.
* Walker (2000) Marilyn A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. _Journal of Artificial Intelligence Research_, pages 387-416.
* Wen et al. (2016) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. _arXiv preprint arXiv:1604.04562_.
* Wen et al. (2017) Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve Young. 2017. Latent intention dialogue models. In _International Conference on Machine Learning_, pages 3732-3741.
* Williams et al. (2017) Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 665-677.
* Williams and Young (2007) Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. _Computer Speech & Language_, 21(2):393-422.
* Williams and Zweig (2016) Jason D Williams and Geoffrey Zweig. 2016. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. _arXiv preprint arXiv:1606.01269_.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1480-1489.
* Yarats and Lewis (2017) Denis Yarats and Mike Lewis. 2017. Hierarchical text generation and planning for strategic dialogue. _arXiv preprint arXiv:1712.05846_.
* Young et al. (2007) Stephanie Young, Jost Schatzmann, Karl Weilhammer, and Hui Ye. 2007. The hidden information state approach to dialog management. In _Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on_, volume 4, pages IV-149. IEEE.
* Young et al. (2013) Steve Young, Milica Gasic, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. _Proceedings of the IEEE_, 101(5):1160-1179.
* Young (2006) Steve J Young. 2006. Using pomdps for dialog management. In _SLT_, pages 8-13.

 

Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In _17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, page 1.
* Zhao and Eskenazi (2018) Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_, pages 1-10.
* Zhao et al. (2018) Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers_), volume 1.
* Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 654-664.

## Appendix A Supplemental Material