Venue: NAACL 2018

# A Hierarchical Latent Structure for Variational Conversation Modeling

Yookoon Park &Jaemin Cho &Gunhee Kim

Department of Computer Science and Engineering & Center for Superintelligence

Seoul National University, Korea

yookoonpark@vision.snu.ac.kr, {jaemin895,gunhee}@snu.ac.kr [http://vision.snu.ac.kr/projects/vhcr](http://vision.snu.ac.kr/projects/vhcr)

###### Abstract

Variational autoencoders (VAE) combined with hierarchical RNNs have emerged as a powerful framework for conversation modeling. However, they suffer from the notorious degeneration problem, where the decoders learn to ignore latent variables and reduce to vanilla RNNs. We empirically show that this degeneracy occurs mostly due to two reasons. First, the expressive power of hierarchical RNN decoders is often high enough to model the data using only its decoding distributions without relying on the latent variables. Second, the conditional VAE structure whose generation process is conditioned on a context, makes the range of training targets very sparse; that is, the RNN decoders can easily overfit to the training data ignoring the latent variables. To solve the degeneration problem, we propose a novel model named _Variational Hierarchical Conversation RNNs_ (VHCR), involving two key ideas of (1) using a hierarchical structure of latent variables, and (2) exploiting an _utterance drop_ regularization. With evaluations on two datasets of Cornell Movie Dialog and Ubuntu Dialog Corpus, we show that our VHCR successfully utilizes latent variables and outperforms state-of-the-art models for conversation generation. Moreover, it can perform several new utterance control tasks, thanks to its hierarchical latent structure.

## 1 Introduction

Conversation modeling has been a long interest of natural language research. Recent approaches for data-driven conversation modeling mostly build upon recurrent neural networks (RNNs) (Vinyals and Le, 2015; Sordoni et al., 2015b; Shang et al., 2015; Li et al., 2017; Serban et al., 2016). Serban et al. (2016) use a hierarchical RNN structure to model the context of conversation. Serban et al. (2017) further exploit an utterance latent variable in the hierarchical RNNs by incorporating the variational autoencoder (VAE) framework (Kingma and Welling, 2014; Rezende et al., 2014).

VAEs enable us to train a latent variable model for natural language modeling, which grants us several advantages. First, latent variables can learn an interpretable holistic representation, such as topics, tones, or high-level syntactic properties. Second, latent variables can model inherently abundant variability of natural language by encoding its global and long-term structure, which is hard to be captured by shallow generative processes (vanilla RNNs) where the only source of stochasticity comes from the sampling of output words.

In spite of such appealing properties of latent variable models for natural language modeling, VAEs suffer from the notorious _degeneration_ problem (Bowman et al., 2016; Chen et al., 2017) that occurs when a VAE is combined with a powerful decoder such as autoregressive RNNs. This issue makes VAEs ignore latent variables, and eventually behave as vanilla RNNs. Chen et al. (2017) also note this degeneration issue by showing that a VAE with a RNN decoder prefers to model the data using its decoding distribution rather than using latent variables, from bits-back coding perspective. To resolve this issue, several heuristics have been proposed to weaken the decoder, enforcing the models to use latent variables. For example, Bowman et al. (2016) propose some heuristics, including _KL annealing_ and _word drop_ regularization. However, these heuristics cannot be a complete solution; for example, we observe that they fail to prevent the degeneracy in VHRED (Serban et al., 2017), a conditional VAE model equipped with hierarchical RNNs for conversation modeling.

The objective of this work is to propose a novel VAE model that significantly alleviates the degen

eration problem. Our analysis reveals that the causes of the degeneracy are two-fold. First, the hierarchical structure of autoregressive RNNs is powerful enough to predict a sequence of utterances without the need of latent variables, even with the word drop regularization. Second, we newly discover that the conditional VAE structure where an utterance is generated conditioned on context, _i.e_. a previous sequence of utterances, induces severe data sparsity. Even with a large-scale training corpus, there only exist very few target utterances when conditioned on the context. Hence, the hierarchical RNNs can easily memorize the context-to-utterance relations without relying on latent variables.

We propose a novel model named _Variational Hierarchical Conversation RNN_ (VHCR), which involves two novel features to alleviate this problem. First, we introduce a global conversational latent variable along with local utterance latent variables to build a hierarchical latent structure. Second, we propose a new regularization technique called _utterance drop_. We show that our hierarchical latent structure is not only crucial for facilitating the use of latent variables in conversation modeling, but also delivers several additional advantages, including gaining control over the global context in which the conversation takes place.

Our major contributions are as follows:

(1) We reveal that the existing conditional VAE model with hierarchical RNNs for conversation modeling (_e.g_. Serban et al., 2017)) still suffers from the degeneration problem, and this problem is caused by data sparsity per context that arises from the conditional VAE structure, as well as the use of powerful hierarchical RNN decoders.

(2) We propose a novel variational hierarchical conversation RNN (VHCR), which has two distinctive features: a hierarchical latent structure and a new regularization of utterance drop. To the best of our knowledge, our VHCR is the first VAE conversation model that exploits the hierarchical latent structure.

(3) With evaluations on two benchmark datasets of Cornell Movie Dialog Danescu-Niculescu-Mizil and Lee, 2011) and Ubuntu Dialog Corpus Lowe et al. (2015), we show that our model improves the conversation performance in multiple metrics over state-of-the-art methods, including HRED Serban et al. (2016), and VHEED Serban et al. (2017) with existing degeneracy solutions such as the word drop Bowman et al. (2016), and the bag-of-words loss Zhao et al. (2017).

## 2 Related Work

**Conversation Modeling**. One popular approach for conversation modeling is to use RNN-based encoders and decoders, such as Vinyals and Le (2015); Sordoni et al. (2015b); Shang et al. (2015). Hierarchical recurrent encoder-decoder (HRED) models Sordoni et al. (2015a); Serban et al. (2016, 2017) consist of utterance encoder and decoder, and a context RNN which runs over utterance representations to model long-term temporal structure of conversation.

Recently, latent variable models such as VAEs have been adopted in language modeling Bowman et al. (2016); Zhang et al. (2016); Serban et al. (2017). The VHRED model Serban et al. (2017) integrates the VAE with the HRED to model Twitter and Ubuntu IRC conversations by introducing an utterance latent variable. This makes a conditional VAE where the generation process is conditioned on the context of conversation. Zhao et al. (2017) further make use of discourse act labels to capture the diversity of conversations.

**Degeneracy of Variational Autoencoders**. For sequence modeling, VAEs are often merged with the RNN encoder-decoder structure Bowman et al. (2016); Serban et al. (2017); Zhao et al. (2017) where the encoder predicts the posterior distribution of a latent variable \({\bf z}\), and the decoder models the output distributions conditioned on \({\bf z}\). However, Bowman et al. (2016) report that a VAE with a RNN decoder easily degenerates; that is, it learns to ignore the latent variable \({\bf z}\) and falls back to a vanilla RNN. They propose two techniques to alleviate this issue: _KL annealing_ and _word drop_. Chen et al. (2017) interpret this degeneracy in the context of bits-back coding and show that a VAE equipped with autoregressive models such as RNNs often ignores the latent variable to minimize the code length needed for describing data. They propose to constrain the decoder to selectively encode the information of interest in the latent variable. However, their empirical results are limited to an image domain. Zhao et al. (2017) use an auxiliary bag-of-words loss on the latent variable to force the model to use \({\bf z}\). That is, they train an auxiliary network that predicts bag-of-words representation of the target utterance based on \({\bf z}\). Yet this loss works in an opposite di 

rection to the original objective of VAEs that minimizes the minimum description length. Thus, it may be in danger of forcibly moving the information that is better modeled in the decoder to the latent variable.

## 3 Approach

We assume that the training set consists of \(N\) i.i.d samples of conversations \(\{\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{N}\}\) where each \(\mathbf{c}_{i}\) is a sequence of utterances (_i.e._ sentences) \(\{\mathbf{x}_{i1},\mathbf{x}_{i2},...,\mathbf{x}_{in_{i}}\}\). Our objective is to learn the parameters of a generative network \(\boldsymbol{\theta}\) using Maximum Likelihood Estimation (MLE):

\[\arg\max_{\boldsymbol{\theta}}\sum_{i}\log p_{\boldsymbol{\theta}}(\mathbf{c}_ {i}) \tag{1}\]

We first briefly review the VAE, and explain the degeneracy issue before presenting our model.

### Preliminary: Variational Autoencoder

We follow the notion of Kingma and Welling (2014). A datapoint \(\mathbf{x}\) is generated from a latent variable \(\mathbf{z}\), which is sampled from some prior distribution \(p(\mathbf{z})\), typically a standard Gaussian distribution \(\mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I})\). We assume parametric families for conditional distribution \(p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})\). Since it is intractable to compute the log-marginal likelihood \(\log p_{\boldsymbol{\theta}}(\mathbf{x})\), we approximate the intractable true posterior \(p_{\boldsymbol{\theta}}(\mathbf{z}|\mathbf{x})\) with a recognition model \(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})\) to maximize the _variational lower-bound_:

\[\log p_{\boldsymbol{\theta}}(\mathbf{x})\geq\mathcal{L}( \boldsymbol{\theta},\boldsymbol{\phi};\mathbf{x}) \tag{2}\] \[=\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}[-\log q _{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})+\log p_{\boldsymbol{\theta}}( \mathbf{x},\mathbf{z})]\] \[=-D_{KL}(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})\|p( \mathbf{z}))+\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}[\log p _{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})]\]

Eq. 2 is decomposed into two terms: KL divergence term and reconstruction term. Here, KL divergence measures the amount of information encoded in the latent variable \(\mathbf{z}\). In the extreme where KL divergence is zero, the model completely ignores \(\mathbf{z}\), _i.e._ it degenerates. The expectation term can be stochastically approximated by sampling \(\mathbf{z}\) from the variational posterior \(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})\). The gradients to the recognition model can be efficiently estimated using the _reparameterization_ trick Kingma and Welling (2014).

### Vhred

Serban et al. (2017) propose Variational Hierarchical Recurrent Encoder Decoder (VHRED) model for conversation modeling. It integrates an utterance latent variable \(\mathbf{z}^{\text{utt}}_{t}\) into the HRED structure Sordoni et al. (2015a) which consists of three RNN components: _encoder RNN_, _context RNN_, and _decoder RNN_. Given a previous sequence of utterances \(\mathbf{x}_{1},...\mathbf{x}_{t-1}\) in a conversation, the VHRED generates the next utterance \(\mathbf{x}_{t}\) as:

\[\mathbf{h}^{\text{enc}}_{t-1} =f^{\text{enc}}_{\boldsymbol{\theta}}(\mathbf{x}_{t-1}) \tag{3}\] \[\mathbf{h}^{\text{ext}}_{t} =f^{\text{cat}}_{\boldsymbol{\theta}}(\mathbf{h}^{\text{ext}}_{t-1} ,\mathbf{h}^{\text{enc}}_{t-1})\] (4) \[p_{\boldsymbol{\theta}}(\mathbf{z}^{\text{utt}}_{t}|\mathbf{x}_{< t}) =\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}_{t},\boldsymbol{\sigma}_{t} \mathbf{I})\] (5) \[\text{where }\boldsymbol{\mu}_{t} =\text{MLP}_{\boldsymbol{\theta}}(\mathbf{h}^{\text{ext}}_{t})\] (6) \[\boldsymbol{\sigma}_{t} =\text{Softplus}(\text{MLP}_{\boldsymbol{\theta}}(\mathbf{h}^{ \text{ext}}_{t}))\] (7) \[p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{<t}) =f^{\text{dec}}_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{h}^{ \text{ext}}_{t},\mathbf{z}^{\text{utt}}_{t}) \tag{8}\]

At time step \(t\), the encoder RNN \(f^{\text{enc}}_{\boldsymbol{\theta}}\) takes the previous utterance \(\mathbf{x}_{t-1}\) and produces an encoder vector \(\mathbf{h}^{\text{enc}}_{t-1}\) (Eq. 3). The context RNN \(f^{\text{ext}}_{\boldsymbol{\theta}}\) models the context of the conversation by updating its hidden states using the encoder vector (Eq. 4). The context \(\mathbf{h}^{\text{ext}}_{t}\) defines the conditional prior \(p_{\boldsymbol{\theta}}(\mathbf{z}^{\text{utt}}_{t}|\mathbf{x}_{<t})\), which is a factorized Gaussian distribution whose mean \(\boldsymbol{\mu}_{t}\) and diagonal variance \(\boldsymbol{\sigma}_{t}\) are given by feed-forward neural networks (Eq. 5-7). Finally the decoder RNN \(f^{\text{dec}}_{\boldsymbol{\theta}}\) generates the utterance \(\mathbf{x}_{t}\), conditioned on the context vector \(\mathbf{h}^{\text{ext}}_{t}\) and the latent variable \(\mathbf{z}^{\text{utt}}_{t}\) (Eq. 8). We make two important notes: (1) the context RNN can be viewed as a high-level decoder, and together with the decoder RNN, they comprise a hierarchical RNN decoder. (2) VHRED follows a conditional VAE structure where each utterance \(\mathbf{x}_{t}\) is generated conditioned on the context \(\mathbf{h}^{\text{ext}}_{t}\) (Eq. 5-8).

The variational posterior is a factorized Gaussian distribution where the mean and the diagonal variance are predicted from the target utterance and the context as follows:

\[q_{\boldsymbol{\phi}}(\mathbf{z}^{\text{utt}}_{t}|\mathbf{x}_{\leq t}) =\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}^{\prime}_{t},\boldsymbol{ \sigma}^{\prime}_{t}I) \tag{9}\] \[\text{where }\boldsymbol{\mu}^{\prime}_{t} =\text{MLP}_{\boldsymbol{\phi}}(\mathbf{x}_{t},\mathbf{h}^{\text{ ext}}_{t})\] (10) \[\boldsymbol{\sigma}^{\prime}_{t} =\text{Softplus}(\text{MLP}_{\boldsymbol{\phi}}(\mathbf{x}_{t}, \mathbf{h}^{\text{ext}}_{t})) \tag{11}\]

### The Degeneration Problem

A known problem of a VAE that incorporates an autoregressive RNN decoder is the degeneracy that ignores the latent variable \(\mathbf{z}\). In other words, the KL divergence term in Eq. 2 goes to zero and the decoder fails to learn any dependency between the latent variable and the data. Eventually, the model behaves as a vanilla RNN. This problem is 

first reported in the sentence VAE (Bowman et al., 2016), in which following two heuristics are proposed to alleviate the problem by weakening the decoder.

First, the _KL annealing_ scales the KL divergence term of Eq. 2 using a KL multiplier \(\lambda\), which gradually increases from 0 to 1 during training:

\[\tilde{\mathcal{L}}(\mathbf{\theta},\mathbf{\phi};\mathbf{x})=-\lambda D_{KL} (q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})\lVert p(\mathbf{z})) \tag{12}\] \[+\mathbb{E}_{q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})}[\log p_{\mathbf{ \theta}}(\mathbf{x}|\mathbf{z})]\]

This helps the optimization process to avoid local optima of zero KL divergence in early training. Second, the _word drop_ regularization randomly replaces some conditioned-on word tokens in the RNN decoder with the generic unknown word token (UNK) during training. Normally, the RNN decoder predicts each next word in an autoregressive manner, conditioned on the previous sequence of ground truth (GT) words. By randomly replacing a GT word with an UNK token, the word drop regularization weakens the autoregressive power of the decoder and forces it to rely on the latent variable to predict the next word. The word drop probability is normally set to 0.25, since using a higher probability may degrade the model performance (Bowman et al., 2016).

However, we observe that these tricks do not solve the degeneracy for the VHRED in conversation modeling. An example in Fig. 1 shows that the VHRED learns to ignore the utterance latent variable as the KL divergence term falls to zero.

### Empirical Observation on Degeneracy

The decoder RNN of the VHRED in Eq. 8 conditions on two information sources: deterministic \(\mathbf{h}^{\text{ext}}_{t}\) and stochastic \(\mathbf{z}^{\text{utt}}\). In order to check whether the presence of deterministic source \(\mathbf{h}^{\text{ext}}_{t}\) causes the degeneration, we drop the deterministic \(\mathbf{h}^{\text{ext}}_{t}\) and condition the decoder only on the stochastic utterance latent variable \(\mathbf{z}^{\text{utt}}\):

\[p_{\mathbf{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{<t})=f^{\text{dec}}_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z}^{\text{utt}}_{t}) \tag{13}\]

While this model achieves higher values of KL divergence than original VHRED, as training proceeds it again degenerates with the KL divergence term reaching zero (Fig. 2).

To gain an insight of the degeneracy, we examine how the conditional prior \(p_{\mathbf{\theta}}(\mathbf{z}^{\text{utt}}_{t}|\mathbf{x}_{<t})\) (Eq. 5) of the utterance latent variable changes during training, using the model above (Eq. 13). Fig. 2 plots the ratios of \(\mathbb{E}[\mathbf{\sigma}_{t}^{2}]/\text{Var}(\mathbf{\mu}_{t})\), where \(\mathbb{E}[\mathbf{\sigma}_{t}^{2}]\) indicates the _within variance_ of the priors, and \(\text{Var}(\mathbf{\mu}_{t})\) is the _between variance_ of the priors. Note that traditionally this ratio is closely related to _Analysis of Variance (ANOVA)_(Lemax and Hahs-Vaughn, 2013). The ratio gradually falls to zero, implying that the priors degenerate to separate point masses as training proceeds. Moreover, we find that the degeneracy of priors coincide with the degeneracy of KL divergence, as shown in (Fig. 2). This is intuitively natural: if the prior is already narrow enough to specify the target utterance, there is little pressure to encode any more information in the variational posterior for reconstruction of the target utterance.

 

This empirical observation implies that the fundamental reason behind the degeneration may originate from combination of two factors: (1) strong expressive power of the hierarchical RNN decoder and (2) training data sparsity caused by the conditional VAE structure. The VHRED is trained to predict a next target utterance \({\bf x}_{t}\) conditioned on the context \({\bf h}^{\text{ext}}_{t}\) which encodes information about previous utterances \(\{{\bf x}_{1},\dots,{\bf x}_{t-1}\}\). However, conditioning on the context makes the range of training target \({\bf x}_{t}\) very sparse; even in a large-scale conversation corpus such as Ubuntu Dialog Lowe et al. (2015), there exist one or very few target utterances per context. Therefore, hierarchical RNNs, given their autoregressive power, can easily overfit to training data without using the latent variable. Consequently, the VHRED will not encode any information in the latent variable, i.e. it degenerates. It explains why the word drop fails to prevent the degeneracy in the VHRED. The word drop only regularizes the decoder RNN; however, the context RNN is also powerful enough to predict a next utterance in a given context even with the weakened decoder RNN. Indeed we observe that using a larger word drop probability such as 0.5 or 0.75 only slows down, but fails to stop the KL divergence from vanishing.

### Variational Hierarchical Conversation RNN (VHCR)

As discussed, we argue that the two main causes of degeneration are i) the expressiveness of the hierarchical RNN decoders, and ii) the conditional VAE structure that induces data sparsity. This finding hints us that in order to train a non-degenerate latent variable model, we need to design a model that provides an appropriate way to regularize the hierarchical RNN decoders and alleviate data sparsity per context. At the same time, the model should be capable of modeling complex structure of conversation. Based on these insights, we propose a novel VAE structure named Variational Hierarchical Conversation RNN (VHCR), whose graphical model is illustrated in Fig. 3. Below we first describe the model, and discuss its unique features.

We introduce a global conversation latent variable \({\bf z}^{\text{conv}}\) which is responsible for generating a sequence of utterances of a conversation \({\bf c}=\{{\bf x}_{1},\dots,{\bf x}_{n}\}\):

\[p_{\mathbf{\theta}}({\bf c}|{\bf z}^{\text{conv}})=p_{\mathbf{\theta}}({\bf x}_{1}, \dots,{\bf x}_{n}|{\bf z}^{\text{conv}}) \tag{14}\]

Overall, the VHCR builds upon the hierarchical RNNs, following the VHRED Serban et al. (2017). One key update is to form a hierarchical latent structure, by using the global latent variable \({\bf z}^{\text{conv}}\) per conversation, along with local the latent variable \({\bf z}^{\text{utt}}_{t}\) injected at each utterance (Fig. 3):

\[{\bf h}^{\text{enc}}_{t} =f^{\text{enc}}_{\mathbf{\theta}}({\bf x}_{t}) \tag{15}\] \[{\bf h}^{\text{ct}}_{t} =\begin{cases}\text{MLP}_{\mathbf{\theta}}({\bf z}^{\text{conv}}),& \text{if $t=0$}\\ f^{\text{ext}}_{\mathbf{\theta}}({\bf h}^{\text{ext}}_{t-1},{\bf h}^{\text{enc}}_{ t-1},{\bf z}^{\text{conv}}),&\text{otherwise}\end{cases}\] \[p_{\mathbf{\theta}}({\bf x}_{t}|{\bf x}_{<t},{\bf z}^{\text{utt}}_{ t},{\bf z}^{\text{conv}})=f^{\text{dec}}_{\mathbf{\theta}}({\bf x}|{\bf h}^{\text{ ext}}_{t},{\bf z}^{\text{att}}_{t},{\bf z}^{\text{conv}})\] \[p_{\mathbf{\theta}}({\bf z}^{\text{conv}}_{t})=\mathcal{N}({\bf z}| \mathbf{0},\mathbf{I})\] (16) \[p_{\mathbf{\theta}}({\bf z}^{\text{utt}}_{t}|{\bf x}_{<t},{\bf z}^{ \text{conv}})=\mathcal{N}({\bf z}|\mathbf{\mu}_{t},\mathbf{\sigma}_{t}\mathbf{I})\] (17) \[\text{where }\mathbf{\mu}_{t}=\text{MLP}_{\mathbf{\theta}}({\bf h}^{\text{ ext}}_{t},{\bf z}^{\text{conv}})\] (18) \[\mathbf{\sigma}_{t}=\text{Softplus}(\text{MLP}_{\mathbf{\theta}}({\bf h}^ {\text{cxt}}_{t},{\bf z}^{\text{conv}})). \tag{19}\]

For inference of \({\bf z}^{\text{conv}}\), we use a bi-directional RNN denoted by \(f^{\text{conv}}\), which runs over the utterance vectors generated by the encoder RNN:

\[q_{\mathbf{\phi}}({\bf z}^{\text{conv}}|{\bf x}_{1},...,{\bf x}_{n})= \mathcal{N}({\bf z}|\mathbf{\mu}^{\text{conv}},\mathbf{\sigma}^{\text{conv}}I) \tag{20}\] \[\text{where }{\bf h}^{\text{conv}}=f^{\text{conv}}({\bf h}^{\text{ enc}}_{1},...,{\bf h}^{\text{enc}}_{n})\] (21) \[\mathbf{\mu}^{\text{conv}}=\text{MLP}_{\mathbf{\phi}}({\bf h}^{\text{ conv}})\] (22) \[\mathbf{\sigma}^{\text{conv}}=\text{Softplus}(\text{MLP}_{\mathbf{\phi}}({ \bf h}^{\text{conv}})). \tag{23}\]

The posteriors for local variables \({\bf z}^{\text{utt}}_{t}\) are then conditioned on \({\bf z}^{\text{conv}}\):

\[q_{\mathbf{\phi}}({\bf z}^{\text{ut}}_{t}|{\bf x}_{1},...,{\bf x}_{ n},{\bf z}^{\text{conv}})=\mathcal{N}({\bf z}|\mathbf{\mu}^{\prime}_{t},\mathbf{\sigma}^{ \prime}_{t}I) \tag{24}\] \[\text{where }\mathbf{\mu}^{\prime}_{t}=\text{MLP}_{\mathbf{\phi}}({\bf x}_{t},{ \bf h}^{\text{ext}}_{t},{\bf z}^{\text{conv}})\] (25) \[\mathbf{\sigma}^{\prime}_{t}=\text{Softplus}(\text{MLP}_{\mathbf{\phi}}({ \bf x}_{t},{\bf h}^{\text{ext}}_{t},{\bf z}^{\text{conv}})).\]

Our solution of VHCR to the degeneration problem is based on two ideas. The first idea is to build a hierarchical latent structure of \({\bf z}^{\text{conv}}\) for

Figure 3: Graphical representation of the Variational Hierarchical Conversation RNN (VHCR). The global latent variable \({\bf z}^{\text{conv}}\) provides a global context in which the conversation takes place.

 

a conversation and \({\bf z}_{t}^{\text{utt}}\) for each utterance. As \({\bf z}^{\text{conv}}\) is independent of the conditional structure, it does not suffer from the data sparsity problem. However, the expressive power of hierarchical RNN decoders makes the model still prone to ignore latent variables \({\bf z}^{\text{conv}}\) and \({\bf z}_{t}^{\text{utt}}\). Therefore, our second idea is to apply an _utterance drop_ regularization to effectively regularize the hierarchical RNNs, in order to facilitate the use of latent variables. That is, at each time step, the utterance encoder vector \({\bf h}_{t}^{\text{enc}}\) is randomly replaced with a generic unknown vector \({\bf h}^{\text{unk}}\) with a probability \(p\). This regularization weakens the autoregressive power of hierarchical RNNs and as well alleviates the data sparsity problem, since it induces noise into the context vector \({\bf h}_{t}^{\text{ext}}\) which conditions the decoder RNN. The difference with the word drop Bowman et al. (2016) is that our utterance drop depresses the hierarchical RNN decoders as a whole, while the word drop only weakens the lower-level decoder RNNs. Fig. 4 confirms that with the utterance drop with a probability of \(0.25\), the VHCR effectively learns to use latent variables, achieving a significant degree of KL divergence.

### Effectiveness of Hierarchical Latent Structure

Is the hierarchical latent structure of the VHCR crucial for effective utilization of latent variables? We investigate this question by applying the utterance drop on the VHRED which lacks any hierarchical latent structure. We observe that the KL divergence still vanishes (Fig. 4), even though the utterance drop injects considerable noise in the context \({\bf h}_{t}^{\text{ext}}\). We argue that the utterance drop weakens the context RNN, thus it consequently fail to predict a reasonable prior distribution for \({\bf z}^{\text{utt}}\) (Eq. 5-7). If the prior is far away from the region of \({\bf z}^{\text{utt}}\) that can generate a correct target utterance, encoding information about the target in the variational posterior will incur a large KL divergence penalty. If the penalty outweighs the gain of the reconstruction term in Eq. 2, then the model would learn to ignore \({\bf z}^{\text{utt}}\), in order to maximize the variational lower-bound in Eq. 2.

On the other hand, the global variable \({\bf z}^{\text{conv}}\) allows the VHCR to predict a reasonable prior for local variable \({\bf z}_{t}^{\text{utt}}\) even in the presence of the utterance drop regularization. That is, \({\bf z}^{\text{conv}}\) can act as a _guide_ for \({\bf z}^{\text{utt}}\) by encoding the information for local variables. This reduces the KL divergence penalty induced by encoding information in \({\bf z}^{\text{utt}}\) to an affordable degree at the cost of KL divergence caused by using \({\bf z}^{\text{conv}}\). This trade-off is indeed a fundamental strength of hierarchical models that provide _parsimonious_ representation; if there exists any shared information among the local variables, it is coded in the global latent variable reducing the code length by effectively reusing the information. The remaining local variability is handled properly by the decoding distribution and local latent variables.

The global variable \({\bf z}^{\text{conv}}\) provides other benefits by representing a latent global structure of a conversation, such as a topic, a length, and a tone of the conversation. Moreover, it allows us to control such global properties, which is impossible for models without hierarchical latent structure.

## 4 Results

We first describe our experimental setting, such as datasets and baselines (section 4.1). We then report quantitative comparisons using three different metrics (section 4.2-4.4). Finally, we present qualitative analyses, including several utterance control tasks that are enabled by the hierarchal latent structure of our VHCR (section 4.5). We defer implementation details and additional experiment results to the supplementary file.

### Experimental Setting

**Datasets**. We evaluate the performance of conversation generation using two benchmark datasets: 1) Cornell Movie Dialog Corpus Danescu

Figure 4: The comparison of KL divergences. The VHCR with the utterance drop shows high and stable KL divergence, indicating the active use of latent variables. w.d and u.d denote the word drop and the utterance drop, respectively.

 

Niculescu-Mizil and Lee, 2011), containing 220,579 conversations from 617 movies. 2) Ubuntu Dialog Corpus Lowe et al. (2015), containing about 1 million multi-turn conversations from Ubuntu IRC channels. In both datasets, we truncate utterances longer than 30 words.

**Baselines**. We compare our approach with four baselines. They are combinations of two state-of-the-art models of conversation generation with different solutions to the degeneracy. (i) Hierarchical recurrent encoder-decoder (HRED) Serban et al. (2016), (ii) Variational HRED (VHRED) Serban et al. (2017), (iii) VHRED with the word drop Bowman et al. (2016), and (iv) VHRED with the bag-of-words (bow) loss Zhao et al. (2017).

**Performance Measures**. Automatic evaluation of conversational systems is still a challenging problem Liu et al. (2016). Based on literature, we report three quantitative metrics: i) the negative log-likelihood (the variational bound for variational models), ii) embedding-based metrics Serban et al. (2017), and iii) human evaluation via Amazon Mechanical Turk (AMT).

### Results of Negative Log-likelihood

Table 1 summarizes the per-word negative log-likelihood (NLL) evaluated on the test sets of two datasets. For variational models, we instead present the variational bound of the negative log-likelihood in Eq. 2, which consists of the reconstruction error term and the KL divergence term. The KL divergence term can measure how much each model utilizes the latent variables.

We observe that the NLL is the lowest by the HRED. Variational models show higher NLLs, because they are regularized methods that are forced to rely more on latent variables. Independent of NLL values, we later show that the latent variable models often show better generalization performance in terms of embedding-based metrics and human evaluation. In the VHRED, the KL divergence term gradually vanishes even with the word drop regularization; thus, early stopping is necessary to obtain a meaningful KL divergence. The VHRED with the bag-of-words loss (bow) achieves the highest KL divergence, however, at the cost of high NLL values. That is, the variational lower-bound minimizes the minimum description length, to which the bow loss works in an opposite direction by forcing latent variables to encode bag-of-words representation of utterances. Our VHCR achieves stable KL divergence without any auxiliary objective, and the NLL is lower than the VHRED + bow model.

Table 2 summarizes how global and latent variable are used in the VHCR. We observe that VHCR encodes a significant amount of information in the global variable \({\bf z}^{\text{conv}}\) as well as in the local variable \({\bf z}^{\text{utt}}\), indicating that the VHCR successfully exploits its hierarchical latent structure.

### Results of Embedding-Based Metrics

The embedding-based metrics Serban et al. (2017); Rus and Lintean (2012) measure the textual similarity between the words in the model response and the ground truth. We represent words using Word2Vec embeddings trained on the Google News Corpus1. The _average_ metric projects each utterance to a vector by taking the mean over word embeddings in the utterance, and computes the cosine similarity between the model response vector and the ground truth vector. The _extrema_ metric is similar to the average metric, only except that it takes the extremum of each di

\begin{table}
\begin{tabular}{|c|c c c c|} \hline

- \\ VHRED & \(\leq\) 3.912 & 3.619 & 0.293 \\ VHRED + w.d & \(\leq\) 3.904 & 3.553 & 0.351 \\ VHRED + bow & \(\leq\) 4.149 & 2.982 & 1.167 \\ VHCR + u.d & \(\leq\) 4.026 & 3.523 & 0.503 \\ \hline \end{tabular} (a) Cornell Movie Dialog

\begin{tabular}{|c|c c c c|} \hline

- \\ VHRED & \(\leq\) 3.767 & 3.654 & 0.113 \\ VHRED + w.d & \(\leq\) 3.824 & 3.363 & 0.461 \\ VHRED + bow & \(\leq\) 4.237 & 2.215 & 2.022 \\ VHRC + u.d & \(\leq\) 3.951 & 3.205 & 0.756 \\ \hline \end{tabular} (b) Ubuntu Dialog

\end{table}
Table 1: Results of Negative Log-likelihood. The inequalities denote the variational bounds. w.d and u.d., and bow denote the word drop, the utterance drop, and the auxiliary bag-of-words loss respectively.

\begin{table}
\begin{tabular}{|l|r r r|r r r|} \hline  & \multicolumn{3}{c|}{Cornell} & \multicolumn{3}{c|}{Ubuntu} \\ \hline
* & 0.351 & 0.461 &
- & 0.461 \\ VHCR & 0.503 & 0.189 & 0.314 & 0.756 & 0.198 & 0.558 \\ \hline \end{tabular}
\end{table}
Table 2: KL divergence decomposition. VHRED and VHCR are trained with word drop and utterance drop respectively.
 

mension, instead of the mean. The _greedy_ metric first finds the best non-exclusive word alignment between the model response and the ground truth, and then computes the mean over the cosine similarity between the aligned words.

Table 3 compares the different methods with three embedding-based metrics. Each model generates a single response (1-turn) or consecutive three responses (3-turn) for a given context. For 3-turn cases, we report the average of metrics measured for three turns. We use the greedy decoding for all the models.

Our VHCR achieves the best results in most metrics. The HRED is the worst on the Cornell Movie dataset, but outperforms the VHRED and VHRED + bow on the Ubuntu Dialog dataset. Although the VHRED + bow shows the highest KL divergence, its performance is similar to that of VHRED, and worse than that of the VHCR model. It suggests that a higher KL divergence does not necessarily lead to better performance; it is more important for the models to balance the modeling powers of the decoder and the latent variables. The VHCR uses a more sophisticated hierarchical latent structure, which better reflects the structure of natural language conversations.

### Results of Human Evaluation

Table 4 reports human evaluation results via Amazon Mechanical Turk (AMT). The VHCR outperforms the baselines in both datasets; yet the performance improvement in Cornell Movie Dialog are less significant compared to that of Ubuntu. We empirically find that Cornell Movie dataset is small in size, but very diverse and complex in content and style, and the models often fail to generate sensible responses for the context. The performance gap with the HRED is the smallest, suggesting that the VAE models without hierarchical latent structure have overfitted to Cornell Movie dataset.

### Qualitative Analyses

**Comparison of Predicted Responses**. Table 5 compares the generated responses of algorithms. Overall, the VHCR creates more consistent responses within the context of a given conversation. This is supposedly due to the global latent variable \({\bf z}^{\text{conv}}\) that provides a more direct and effective way to handle the global context of a conversation. The context RNN of the baseline models can handle long-term context to some extent, but not as much as the VHCR.

**Interpolation on \({\bf z}^{\text{conv}}\)**. We present examples of one advantage by the hierarchical latent structure of the VHCR, which cannot be done by the other existing models. Table 6 shows how the generated responses vary according to the interpolation on \({\bf z}^{\text{conv}}\). We randomly sample two \({\bf z}^{\text{conv}}\) from a standard Gaussian prior as references (the top and the bottom row of Table 6), and interpolate points between them. We generate 3-turn conversations conditioned on given \({\bf z}^{\text{conv}}\). We see that \({\bf z}^{\text{conv}}\) controls the overall tone and content of conversations; for example, the tone of the response is friendly in the first sample, but gradually becomes hostile as \({\bf z}^{\text{conv}}\) changes.

**Generation on a Fixed \({\bf z}^{\text{conv}}\)**. We also study how fixing a global conversation latent variable \({\bf z}^{\text{conv}}\) affects the conversation generation. Table 7 shows an example, where we randomly fix a reference \({\bf z}^{\text{conv}}\) from the prior, and generate multiple examples of 3-turn conversation using randomly sampled local variables \({\bf z}^{\text{utt}}\). We observe that \({\bf z}^{\text{conv}}\) heavily affects the form of the first utterance; in the examples, the first utterances all start with a "where" phrase. At the same time, responses show

\begin{table}
\begin{tabular}{|c|c c c|} \hline
**Model** & Average & Extrema & Greedy \\ \hline  & 1-turn & & \\ \hline HRED & 0.541 & 0.370 & 0.387 \\ VHRED & 0.543 & 0.356 & 0.393 \\ VHRED + w.d & 0.554 & 0.365 & 0.404 \\ VHRED + bow & 0.555 & 0.350 & 0.411 \\ VHCR + u.d & **0.585** & **0.376** & **0.434** \\ \hline  & 3-turn & & \\ \hline HRED & 0.556 & 0.372 & 0.395 \\ VHRED & 0.554 & 0.360 & 0.398 \\ VHRED + w.d & 0.566 & 0.369 & 0.408 \\ VHRED + bow & 0.573 & 0.360 & 0.423 \\ VHCR + u.d & **0.588** & **0.378** & **0.429** \\ \hline  & \multicolumn{1}{r}{(a) Cornell Movie Dialog} \\ \hline
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] **Model** & Average & Extrema & Greedy \\ \hline  & 1-turn & & \\ \hline HRED & 0.567 & **0.337** & 0.412 \\ VHRED & 0.547 & 0.322 & 0.398 \\ VHRED + w.d & 0.545 & 0.314 & 0.398 \\ VHRED + bow & 0.545 & 0.306 & 0.398 \\ VHCR + u.d & **0.570** & 0.312 & **0.425** \\ \hline  & \multicolumn{1}{r}{(b) Ubuntu Dialog} \\ \end{tabular}
\end{table}
Table 3: Results of embedding-based metrics. 1-turn and 3-turn responses of models per context.

 

variations according to different local variables \({\bf z}^{\text{utt}}\). These examples show that the hierarchical latent structure of VHCR allows both global and fine-grained control over generated conversations.

## 5 Discussion

We introduced the variational hierarchical conversation RNN (VHCR) for conversation modeling. We noted that the degeneration problem in existing VAE models such as the VHRED is persistent, and proposed a hierarchical latent variable model with the utterance drop regularization. Our VHCR obtained higher and more stable KL divergences than various versions of VHRED models without using any auxiliary objective. The empirical results showed that the VHCR better reflected the structure of natural conversations, and outperformed previous models. Moreover, the hierarchical latent structure allowed both global and fine-grained control over the conversation generation.

## Acknowledgments

This work was supported by Kakao and Kakao Brain corporations, and Creative-Pioneering Researchers Program through Seoul National University. Gunhee Kim is the corresponding author.

\begin{table}
\begin{tabular}{|l|} \hline  **nice to meet you.** \\ \(\Rightarrow\) **where’s my wife?** \\ \(\rightarrow\) **she's sleep.** \\ go on, frank. \\ \(\downarrow\) till you what i’m doing. i’m a little tired. \\ why? \\ \hline now, you’re not going. \\ \(\rightarrow\) not for a long time. \\ but your care a little ¡unk¿ \\ \hline get out or here. \\ \(\rightarrow\) all right. \\ \(\rightarrow\) you want to go home? \\ \hline
**get out of benefit:** \\ \(\Rightarrow\)**_it’s not your fault, is at it?_ \\ \(\rightarrow\)**why? what’s wrong?** \\ \hline \end{tabular} 
\begin{tabular}{|c|} \hline where is she? \\ \(\rightarrow\) she’s they one who knows where she is, she’s going to be all right. \\ – oh, you’re tile only one who’s gon na he. she’s a “unk¿”. \\ where’s my wife? \\ \hline \(\rightarrow\) you’ve got to get out of there, yox know? yet’the one who’s gon na he. her. \\ \(\rightarrow\) oh, that’s nice. \\ where are you? \\ \(\rightarrow\) well, it was just thinking about you and i know what you’re doing. i’m going ha bae to go to the ¡unk¿ and i’m sorry. \\ — if we may get this? \\ \hline where are you going? \\ \(\rightarrow\) to get you to the airport. \\ \(\rightarrow\) ’you’re going to be late? \\ \hline where are you going? \\ \(\rightarrow\) to the ‘unk¿. j’an not going to tell you what i am. i am the only one who no. i will be the \\ \(\rightarrow\) you’ve got to stop! \\ \hline \end{tabular}
\end{table}
Table 6: An example of interpolated 3-turn responses over \({\bf z}^{\text{conv}}\) on Cornell Movie Dialog.

\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt} p{142.3pt}|p{142.3pt} p{142.3pt} p{142.3pt}|} \hline Context & HRED & VHRED & VHRED+bow & VICR \\ \hline rin going home. \(\rightarrow\) what’s wrong? \(\to\) i don’t know. & i don’t know. one of these days, i’ m gong on see lousis. my back’ killing me. & i don’t know. \\ \hline how’s sleeping beauty? \(\rightarrow\) still sleeping... \(\rightarrow\) kills him, maybe he’ll turn into a frog. & he’s a good man. & what? & we won’t? & maybe he’s dead. \\ \hline when i zoom my thd drive isn get a permission defined but i can easily solve the files ii ibot his live of any index on how it’s their aggressive view of the ctsi file system are likely preventing you thanks, any idea on options to fix? & noh, i don’t know how to fix it, but i’m not sure sure how to fix it & what kind of diesystem is it on? & you did it just first? then you’ll need to reboot and then remain it; if your problem begins, i/or give you the medium & the only thing is can think sof us to fix the permissions issue, try mountain, if your problem begins, i/or give you the medium & of is to fix the permissions issue, try mountain, if your problem begins, i/or give you the medium & its toy intuitive, try mountain and grid options in the forum \\ \hline hello i need some support, anybody there? \(\rightarrow\) go ahead with your questions. \(<\)UNK? \(>\) are not expected on fire.. \(\rightarrow\) has an hp position ¡unk¿ here and i can’t get it to install ilstumu & what’s the problem? & ¡unk¿ & what’s the problem? & i’m sorry. & they should be supported by canonical. you don’t need to use them. \\ \hline \end{tabular}
\end{table}
Table 4: Results of human evaluation via AMT. Human turkers are asked to choose which response is more appropriate in a given context, without knowing which algorithms generate which responses. For each pair of models, we carry out three evaluation batches, each of which consists of 100 random test samples evaluated by five unique humans. We report mean preferences with \(\pm 90\%\) confidence interval.

\begin{table}
\begin{tabular}{|l|} \hline  **nice to meet you.** \\ \(\rightarrow\) **where’s my wife?** \\ \(\rightarrow\) **she’s sleep.** \\ \hline go on, frank. \\ \(\uparrow\) hill you what i’m doing. i’m a little tired. \\ \(\rightarrow\) why? \\ \hline now, you’re not going. \\ \(\rightarrow\) not for a long time. \\ \(\rightarrow\) but you care a little ¡unk¿ \\ \hline get out of here. \\ \(\rightarrow\) all right. \\ \(\rightarrow\) you want to go home? \\ \hline
**got out of benefit.** \\ \(\rightarrow\) **“It’s not your fault, is at it?** \\ \(\rightarrow\) **why? what’s wrong?** \\ \hline \end{tabular} 
\begin{tabular}{|l|} \hline where is she? \\ \(\rightarrow\) she’s they one who knows where she is, she’s going to be all right. \\ \(\rightarrow\) oh, you’re tile only one who’s gon na he. she’s a “unk¿. \\ where’s my wife? \\ \(\rightarrow\) you’ve got to get out of here, yox know? you’re the one who’s gon na he. her. \\ \(\rightarrow\) oh, that’s nice. \\ where are you? \\ \(\rightarrow\) well, it was just thinking about you and i know what you’re doing. i’m going ha bae to go to the ¡unk¿ and i’m sorry. \\ where are you going? \\ \(\rightarrow\) to get you to the airport. \\ \(\rightarrow\)’yon’re going to be late? \\ \(\rightarrow\) where are you going? \\ \(\rightarrow\) to the ‘unk¿. j’an not going to tell you what i am. i am the only one who do to be. i will be the \\ \(\rightarrow\) you’ve got to stop! \\ \hline \end{tabular}
\end{table}
Table 7: An example of 3-turn responses conditioned on sampled \({\bf z}^{\text{utt}}\) for a single fixed \({\bf z}^{\text{conv}}\).

 



## References

* Bowman et al. (2016) Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In _CoNLL_. [https://doi.org/10.18653/v1/K16-1002](https://doi.org/10.18653/v1/K16-1002).
* Chen et al. (2017) Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In _ICLR_.
* Danescu-Miculescu-Mizil and Lee (2011) Cristian Danescu-miculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In _CMCL Workshop_.
* Kingma and Welling (2014) Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In _ICLR_.
* Li et al. (2017) Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. _arXiv preprint arXiv:1701.06547_ .
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In _EMNLP_.
* Lomax and Hahs-Vaughn (2013) Richard G Lomax and Debbie L Hahs-Vaughn. 2013. _Statistical concepts: A second course_. Routledge.
* Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In _SIGDIAL_.
* Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In _ICML_.
* Rus and Lintean (2012) Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In _Building Educational Applications Using NLP Workshop_. ACL. [http://www.aclweb.org/anthology/Wl2-2018](http://www.aclweb.org/anthology/Wl2-2018).
* Serban et al. (2016) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _AAAI_.
* Serban et al. (2017) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In _AAAI_.
* Shang et al. (2015) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In _ACL_. [https://doi.org/10.3115/v1/P15-1152](https://doi.org/10.3115/v1/P15-1152).
* Sordoni et al. (2015a) Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015a. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In _CIKM_.
* Sordoni et al. (2015b) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. In _NAACL-HLT_. [https://doi.org/10.3115/v1/N15-1020](https://doi.org/10.3115/v1/N15-1020).
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In _ICML Deep Learning Workshop_.
* Zhang et al. (2016) Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In _EMNLP_. [https://doi.org/10.18653/v1/D16-1050](https://doi.org/10.18653/v1/D16-1050).
* Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _ACL_. [https://doi.org/10.18653/v1/P17-1061](https://doi.org/10.18653/v1/P17-1061).