Venue: NAACL 2019

# Jointly Optimizing Diversity and Relevance in Neural Response Generation

Xiang Gao Sungjin Lee Yizhe Zhang Chris Brockett Michel Galley Jianfeng Gao Bill Dolan Microsoft Research, Redmond, WA, USA {xiag,sule,yizzhang,chrisbkt,mgalley,jfgao,billdol}@microsoft.com

###### Abstract

Although recent neural conversation models have shown great potential, they often generate bland and generic responses. While various approaches have been explored to diversify the output of the conversation model, the improvement often comes at the cost of decreased relevance (Zhang et al., 2018). In this paper, we propose a SpaceFusion model to jointly optimize diversity and relevance that essentially fuses the latent space of a sequence-to-sequence model and that of an autoencoder model by leveraging novel regularization terms. As a result, our approach induces a latent space in which the distance and direction from the predicted response vector roughly match the relevance and diversity, respectively. This property also lends itself well to an intuitive visualization of the latent space. Both automatic and human evaluation results demonstrate that the proposed approach brings significant improvement compared to strong baselines in both diversity and relevance. 1

Footnote 1: An implementation of our model is available at [https://github.com/golsun/SpaceFusion](https://github.com/golsun/SpaceFusion)

+
Footnote †: _Proceedings of NAACL-HLT 2019_, pages 1229–1238 Minneapolis, Minnesota, June 2 - June 7, 2019. ©2019 Association for Computational Linguistics

## 1 Introduction

The field of neural response generation is advancing rapidly both in terms of research and commercial applications (Gao et al., 2019; Zhou et al., 2018; Yoshino et al., 2019; Zhang et al., 2019). Nevertheless, vanilla sequence-to-sequence (S2S) models often generate bland and generic responses (Li et al., 2016a). Li et al. (2016a) encourage diversity by re-ranking the beam search results according to their mutual information with the conversation context. However, as beam search itself often produces lists of nearly identical sequences, this method can require a large beam width (e.g. 200). As a result, re-ranking can be extremely time-consuming, raising difficulties for real-time applications. This highlights the need to improve the diversity of candidates before re-ranking, and the need to optimize for diversity during training rather than just at the decoding stage.

While various approaches have been explored to diversify the output of conversation models, the improvement often comes at the cost of decreased response relevance along other dimensions. For instance, Zhao et al. (2017) present an approach to enhancing diversity by mapping diverse responses to a probability distribution using a conditional variational autoencoder (CVAE). Despite the improved response diversity, this approach reduces response relevance as measured against the baseline. One possible reason for this diversity-relevance trade-off is that such probabilistic approaches are not explicitly encouraged to induce a disentangled representation in latent space for

Figure 1: Illustration of one **context** and its multiple **responses** in the latent space induced by our model. Distance and direction from the predicted response vector given the context roughly match the relevance and diversity, respectively. Based on the example in Table 2.2

controlling diversity and relevance independently. Consider a Gaussian distribution, which is widely used for CVAE. A Gaussian distribution naturally brings frequent responses near its mean, and the resulting responses are often generic and boring. To generate diverse and interesting responses, one needs to sample a little distance from the mean. But doing so naturally leads to infrequent and thus even irrelevant responses.

In this paper, we propose a novel geometrical approach that explicitly encourages a structured latent space in which the distance and direction from a predicted response vector roughly match the relevance and diversity, respectively, as illustrated in Figure 1. To induce such a latent space, we leverage two different models: 1) a S2S model, producing the predicted response vector (the black dot at the center in Figure 1), and 2) an autoencoder (AE) model, yielding the vectors for potential responses (the colored dots). In order to make the S2S and AE share the same latent space (the cloud), we use the same decoder for both and train them jointly end-to-end with novel regularization terms. As this fuses the two latent spaces, we refer to our model as SpaceFusion.

Regularization is necessary because only sharing the decoder, as in Luan et al. (2017), does not necessarily align the latent spaces obtained by S2S and AE respectively or impose a disentangled structure onto the space. We introduce two regularization terms to tackle this issue. 1) interpolation term: we encourage a smooth semantic transition along the path between the predicted response vector and each target response vector (arrowed lines in Figure 1). This term effectively prevents semantically different responses from aligning in the same direction, essentially scattering them over different directions. 2) fusion term: we want the vectors from the two models to be distributed in a homogeneous manner, rather than forming two separate clusters (Figure 5) that can potentially make sampling non-trivial. With the resulting latent space, we can control relevance and diversity by respectively adjusting distance and direction from a predicted response vector, without sacrificing each other greatly.

Our approach also lends itself well to the intuitive visualization of latent space. Since our model allows us to geometrically find not only the predicted response vector but also the target response vector as in Figure 5, we can visually interpret the structure of latent space and identify major issues thereof. We devote Section 5.1 to show comprehensive examples for visualization-based analysis.

Automatic and human evaluations demonstrate that the proposed approach improves both the diversity and relevance of the responses, compared to strong baselines on two datasets with one-to-many context-response mapping.

## 2 Related Work

Grounded conversation modelsutilize extra context inputs besides conversation history, such as persona Li et al. (2016), textual knowledge Ghazvininejad et al. (2017); Galley et al. (2019), dialog act Zhao et al. (2017) and emotion Huber et al. (2018). Our approach does not depend on such extra input and thus is complementary to this line of studies.

Variational autoencoder (VAE) modelsexplicitly model the uncertainty of responses in latent space. Bowman et al. (2016) used VAE with Long-Short Term Memory (LSTM) cells to generate sentences. The basic idea of VAE is to encode the input \(x\) into a probability distribution (e.g. Gaussian) \(z\) instead of a point encoding. However, it suffers from the _vanishing latent variable problem_Bowman et al. (2016); Zhao et al. (2017) when applied to text generation tasks. Bowman et al. (2016); Fu et al. (2019) proposed to tackle this problem with word dropping and specific KL annealing methods. Zhao et al. (2017) proposed to add a _bag-of-word_ loss, complementary to KL annealing. Applying this to a CVAE conversation model, they showed that even greedy decoding can generate diverse responses. However, as VAE/CVAE conversation models can be limited to a simple latent representations such as standard Gaussian distribution, Gu et al. (2018) proposed to enrich the latent space by leveraging a Gaussian mixture prior. Our work takes a geometrical approach that is fundamentally different from probabilistic approaches to tackle the limitations of parameteric distributions in representation and difficulties in training.

Decoding and rankingencourage diversity during the decoding stage. As "vanilla" beam search often produces lists of nearly identical sequences, Vijayakumar et al. (2016) propose to include a dissimilarity term in the objective of beam search decoding. Li et al. (2016) re-ranked the results 

obtained by beam search based on mutual information with the context using a separately trained response-to-context S2S model.

Multi-task learningis another line of studies related to the present work (see Section 3.2). Sennrich et al. (2016) use multi-task learning to improve neural machine translation by utilizing monolingual data, which usually far exceeds the amount of parallel data. A similar idea is applied by Luan et al. (2017) to conversational modeling, involving two tasks: 1) a S2S model that learns a context-to-response mapping using conversation data, and 2) an AE model that utilizes speaker-specific non-conversational data. The decoders of S2S and AE were shared, and the two tasks were trained alternately.

## 3 The SpaceFusion Model

### Problem statement

Let \({\cal D}=[(x_{0},y_{0}),(x_{1},y_{1}),\cdots,(x_{n},y_{n})]\) denote a conversational dataset, where \(x_{i}\) and \(y_{i}\) are a context and its response, respectively. \(x_{i}\) consists of one or more utterances. Our aim is to train a model on \({\cal D}\) to generate relevant and diverse responses given a context.

### Fusing latent spaces

We design our model to induce a latent space where different responses for a given context are in different directions around the predicted response vector, as illustrated in Figure 1. Then we can obtain diverse responses by varying the direction and keep their relevance by sampling near the predicted response vector.

To fulfill this goal, we first produce the predicted response representation \(z_{\text{S2S}}\) and target response representations \(z_{\text{AE}}\) using an S2S model and an AE model, respectively, as illustrated in Figure 2. Both encoders are implemented using stacked Gated Recurrent Unit (GRU) Cho et al. (2014) cells followed by a noise layer that adds multivariate Gaussian noise \(\epsilon\sim N(0,\sigma^{2}\mathbf{I})\). We then explicitly encourage smooth semantic transition along the path from \(z_{\text{S2S}}\) to \(z_{\text{AE}}\) by imposing any interpolation between them to generate the same response via the following loss term:

\[{\cal L}_{\text{interp}}=-\frac{1}{|y|}\log p(y|z_{\text{interp}}) \tag{1}\]

where \(z_{\text{interp}}=uz_{\text{S2S}}+(1-u)z_{\text{AE}}\) and \(u\sim U(0,1)\) is a uniformly distributed random variable. \(|y|\) is the number of words in \(y\). Note that it is this regularization term that effectively prevents significantly different responses from aligning in the same direction, essentially scattering them over different directions. In order for this interpolation loss to work, we share the same decoder for both AE and S2S models as in Luan et al. (2017). The decoder consists of stacked GRU cells followed by a softmax layer. It is worth mentioning that \(z_{\text{interp}}\) is not just randomly drawn from a single line but from a richer probabilistic region as both \(z_{\text{interp}}\) and \(z_{\text{S2S}}\) are stochastic due to the random component \(\epsilon\).

Now, we want vectors from both the AE and S2S models to be distributed in a homogeneous manner scattered over the entire space while keeping the distance between \(z_{\text{S2S}}\) and \(z_{\text{AE}}\) as small as possible for any (context-response) pair in the training data. This objective is represented in the following regularization term:

\[{\cal L}_{\text{fuse}} =\sum_{i\in\text{batch}}\frac{d(z_{\text{S2S}}(x_{i}),z_{\text{ AE}}(y_{i}))}{n}\] \[-\sum_{i,j\in\text{batch},i\neq j}\frac{d(z_{\text{S2S}}(x_{i}), z_{\text{S2S}}(x_{j}))}{n^{2}-n}\] \[-\sum_{i,j\in\text{batch},i\neq j}\frac{d(z_{\text{AE}}(y_{i}), z_{\text{AE}}(y_{j}))}{n^{2}-n} \tag{2}\]

where \(n\) is the batch size and \(d(a,b)\) is the root mean square of the difference between \(a\) and \(b\). For each batch, we basically disperse vectors obtained by the same model and pull the predicted response vectors to the corresponding target response vectors. In practice, we found that the performance is better if the Euclidean distance is clipped to a prescribed maximum value.3

Footnote 3: This value is set as 0.3 for the present experiments

Finally, with weight parameters \(\alpha\) and \(\beta\), the

Figure 2: SpaceFusion model architecture.

 

loss function is defined as:

\[{\cal L}= -\frac{1}{|y|}\log p(y|z_{\text{S2S}})\] \[-\frac{1}{|y|}\log p(y|z_{\text{AE}})\] \[+\alpha{\cal L}_{\text{interp}}+\beta{\cal L}_{\text{fuse}} \tag{3}\]

As \({\cal L}_{\text{interp}}\) and \({\cal L}_{\text{fuse}}\) encourage the path between \(z_{\text{S2S}}\) and \(z_{\text{AE}}\) to be smooth and short while scattering vectors over the entire space, they effectively fuse the \(z_{\text{S2S}}\) latent space and the \(z_{\text{AE}}\) latent space. Accordingly we refer this approach as SpaceFusion with path regularization.

### Training

In contrast to previous multi-task conversation model Luan et al. (2017), where S2S and AE are trained alternately, our approach trains S2S and AE at the same time by minimizing the loss function of Equation 3.

### Inference

Like Zhao et al. (2017); Bowman et al. (2016), for a given context, we sample different latent vectors to obtain multiple hypotheses. This is done by adding a random vector \(r\) that is uniformly sampled from a hypersphere of radius \(|r|\) to the prediction \(z_{\text{S2S}}(x)\).

\[z(x,r)=z_{\text{S2S}}(x)+r \tag{4}\]

where \(|r|\) is tuned on the validation set to optimize the trade-off between relevance and diversity. \(z(x,r)\) is then fed to the decoder as the initial state of GRU cells. We then generate responses using greedy decoding.4

Footnote 4: Although we use greedy decoding in this work, other decoding techniques, such as beam search, can be applied.

## 4 Experiment Setup

### Datasets

We used the following datasets. Some of their key features are presented in Table 1.

Switchboard:We use the version offered by Zhao et al. (2017), which is an extension of the original version by Godfrey and Holliman (1997). Zhao et al. (2017) collected multiple references for the test set using information retrieval (IR) techniques followed by human filtering, and randomly split the data into 2316/60/62 conversations for train/validate/test, respectively. Each conversation has multiple turns and thus multiple \((x,y)\) pairs, as listed in Table 1. As our approach does not utilize extra information except conversation history, we removed the meta data (e.g. gender, age, prompt) from this dataset.

Reddit:As the Switchboard dataset is relatively small and multiple references are synthetically constructed, we have developed another multi-reference dataset by extracting posts and comments on Reddit.com during 2011 collected by a third party.5 As each Reddit post and comment may have multiple comments, it is a natural source of multi-reference responses. We further filtered the data based on the number of replies to obtain the final conversation dataset in which each context has at least 10 different responses, and on average the number of responses is 24.1 for a given context. The size is significantly larger than Switchboard, as listed in Table 1. The conversations are randomly shuffled before being split into train/valid/test subsets.

Footnote 5: [http://files.pushshift.io/reddit/comments/](http://files.pushshift.io/reddit/comments/)

### Model setup

Both encoders and the shared decoder consist of two GRU cells, each with 128 hidden units. The variance of the noise layer in each decoder is \(\sigma^{2}=0.1^{2}\). The word embedding dimension is 128. The weight parameters (see Equation 3) are set as \(\alpha=1\) and \(\beta=30\). For both datasets, the inference radius \(|r|\) (see Equation 4) is set to 1.5 which optimizes F1 score on the validation set. All models are trained using the Adam method Kingma and Ba (2014) with a learning rate of 0.001 on both datasets until convergence (around 4 epochs for Reddit and 10 epochs for Switchboard).

### Automatic evaluation

For a given context \(x\), we have \(N_{r}\) reference responses and generate the same number of hypothe

\begin{table}
\begin{tabular}{l|c c} \hline  & Switchboard & Reddit \\ \hline train \((x,y)\) samples & 0.2M & 7.3M \\ test \((x,y)\) samples & 5418 & 5000 \\ \hline ref. source & IR+filtering & natural \\ ref. availability & test only & train/vali/test \\ ref. per context & 7.7 & 24.1 \\ \hline \end{tabular}
\end{table}
Table 1: Key features of the datasets.

 

ses.6 We define the following metrics based on 4-gram BLEU Papineni et al. (2002), as suggested by Zhao et al. (2017).

Footnote 6: We set the number of hypotheses equal to the number of references to encourage precision and recall have comparable impact on F1

Precision \[=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}\max_{j\in[1,N_{r}]}\text{BLEU}( r_{j},h_{i})\] Recall \[=\frac{1}{N_{r}}\sum_{j=1}^{N_{r}}\max_{i\in[1,N_{r}]}\text{BLEU}( r_{j},h_{i})\] F1 \[=2\,\frac{\text{precision}\cdot\text{recall}}{\text{precision}+ \text{recall}}\]

We use Precision as an approximate surrogate metric for relevance and Recall for diversity. It should be noted that recall is not equivalent to other diversity metrics, e.g., distinct Li et al. (2016) and entropy Zhang et al. (2018), which only depend on hypotheses. One potential issue of these metrics is that even randomly generated responses may yield a high diversity score. F1 is the harmonic average of these two and is used to measure the overall response quality.

### Human evaluation

We conduct a human evaluation using crowdworkers. For each hypothesis, given its context, we ask three annotators to individually measure the quality, on a scale of 1 to 5, in terms of two aspects: relevance and interest. Interestingness is treated as an estimation of the diversity, as these two are often correlated. The hypotheses from all systems are shuffled before being provided to annotators. System names are invisible to the annotators.

### Baselines

We compare the proposed model with the following baseline models:

S2S+Sampling:We consider a vanilla version of S2S model. The dimensions are similar to our model: both encoder and decoder consist of two stacked GRU cells with 128 hidden units, and the word embedding size is 128. As in the baseline in Zhao et al. (2017), we applied softmax sampling at inference time to generate multiple hypotheses.

CVAE+BOW:For the CVAE conversation model, we use the original implementation and hyperparameters of Zhao et al. (2017) with the bag-of-words (BOW) loss. The number of trainable model parameters is 15.4M, which is much larger than our model (3.2M).

MTask:Since our approach utilizes a multi-task learning scheme, we also compare it against a vanilla multi-task learning model, MTask, similar to Luan et al. (2017), to illustrate the effect of space fusion. The model architecture and hyperparameters are identical to the proposed model except that the loss function is \(\mathcal{L}=-\log p(y|z_{\text{S2S}})-\log p(y|z_{\text{AE}})\).

## 5 Results and Analysis

### In-depth analysis of latent space

In this section, we undertake an in-depth analysis to verify whether the latent space induced by our method manifests desirable properties, namely: 1) disentangled space structure between relevance and diversity, 2) homogeneous space distribution in which semantics changes smoothly without holes. We first provide a qualitative investigation based on real examples. Then, we present a set of corpus-level quantitative analyses focused on geometric properties.

#### 5.1.1 Qualitative examples

In Table 2, we investigate three different directions from the context "Anyone want to start this game?" , which is a real example taken from Reddit. The three different directions correspond to clearly different semantics: "No I don't", "when?" and "Yes Ido." If we generate a response with the vector predicted by the S2S model (\(u=0\)), our model outputs "I would love to play this game" which is highly relevant to the context. Now as we move along each direction, we can see our model gradually transforms the response toward the corresponding responses of each direction. For instance, towards "No I don't", our model gradually transforms the response to "I amnot interested in the game" (\(u=0.18\)) and then "I amnot interested." (\(u=0.21\)). In contrary, towards "Yes Ido", the response transforms to "I would love to play it." (\(u=0.15\)). Besides the positive or negative directions, the same transition applies to other directions such as "When?". This example clearly shows that there is a rough correspondence 

between geometric properties and semantic properties in the latent space induced by our method as shown in Figure 1- the relevance of the response decreases as we move away from the predicted response vector and different directions are associated with semantically different responses.

#### 5.1.2 Direction vs. diversity

In order to quantitatively verify the correspondence between _direction_ and _diversity_, we visualize the distribution of cosine similarities among multiple references for each context for a set of 1000 random samples drawn from the test dataset. Specifically, for a context \(x_{k}\) and its associated reference responses \([y_{k,0},y_{k,1},\cdots]\), we compute the cosine similarity between \(z_{\text{AE}}(y_{k,i})-z_{\text{S2S}}(x_{k})\) and \(z_{\text{AE}}(y_{k,j})-z_{\text{S2S}}(x_{k})\). In Figure 3, we compare the distribution of our model with that of MTask, which does not employ our regularization terms. While our method yields a bell shaped curve with average cosine similarity being close to zero (0.38), the distribution of MTask is extremely skewed with average cosine similarity being close to 1 (0.95). This indicates that the directions of the reference responses are more evenly distributed in our latent space whereas everything is packed in a narrow band in the MTask's space. This essentially makes the inference process simple and robust in that one can choose arbitrary directions to generate diverse responses.

#### 5.1.3 Distance vs. relevance

In order to quantitatively verify the correspondence between _distance_ and _relevance_, we visualize the perplexity of reference responses along the path from the associated \(z_{\text{S2S}}\) (\(u=0\)) to the \(z_{\text{AE}}\) (\(u=1\)) corresponding to the predicted response. In Figure 4, we compare our model with MTask, which as already noted, does not employ our regularization terms. While our model shows a gradual increase in perplexity, there is a huge bump for MTask's line. This clearly indicates that there is a rough correspondence between distance and relevance in our latent space whereas even a slight change can lead to an irrelevant response in the MTask's space.

We further illustrate the smooth change in relevance according to distance for a specific example in Table 3. Given the context "Anyone want to start this game?", our model

Figure 3: Distribution of the directions from a given context to its multiple responses, measured by the cosine similarity between \(z_{\text{AE}}(y_{k,i})-z_{\text{S2S}}(x_{k})\) and \(z_{\text{AE}}(y_{k,j})-z_{\text{S2S}}(x_{k})\). Histogram calculated based on 1000 \(x_{k}\) from Reddit test data and visualized with bin width of 0.02.

\begin{table}
\begin{tabular}{l l|c c|l l} \hline  & \multicolumn{6}{c}{context \(x\): Anyone want to start this game?} \\ \cline{3-7} \(u\) & **towards “No I don’t.”** & \(u\) & **towards “when?”** & \(u\) & **towards “Yes I do.”** \\ \hline
0.18 & I am not interested in the game. & 0.15 & I d be interested in the game & 0.15 & I’d love to play it. \\
0.21 & I am not interested. & 0.31 & When is it? & 0.27 & Yes I do. \\
0.30 & No I don’t. & 0.40 & When will you? & & \\  & & 1.00 & When? & & \\ \hline \end{tabular}
\end{table}
Table 2: Semantic interpolation along different directions \(y\). Results decoded from \(z_{\text{interp}}\) See Fig. 1 for a visualization.

 

is able to transition from the predicted response "I would love to play this game" to a one of reference responses "Yes I do". The relevance smoothly descreases, generating intermediate responses such as "I would love to play it. " In contrary, the MTask model tends to produce irrelevant or ungrammatical responses as it moves away from the predicted response.

#### 5.1.4 Homogeneity and Convexity

Other desirable properties, with which we want to equip our latent space are _homogeneity_ and _convexity_. If the space is not homogeneous, we have to sample differently depending on the regional traits. If the space is not convex, we have to worry about running into the holes that are not properly associated with valid semantic meanings. In order to verify homogeneity and convexity, we visualize our latent space in a 2D space produced by the multidimensional scaling (MDS) algorithm [3], which approximately preserves pairwise distance. For comparison, we also provide a visualization for MTMask. As shown in Figure 5, our latent space offers great homogeneity and convexity regardless of which model is used to produce a dot (i.e. \(z_{S2S}\) or \(z_{AE}\)). In contrary, MTask's latent space forms two separate clusters for \(z_{S2S}\) and \(z_{AE}\) with a large gap in-between where no training samples were mapped to.

### Automatic evaluation

We let each system generate 100 hypotheses \(\{h_{j}\}\) for each context \(x_{i}\) in the test dataset. Assuming \(x_{i}\) has \(N_{r,i}\) references, we pick the top \(N_{r,i}\) distinct hypotheses ranked by \(\log p(h_{j}|x_{i})+\lambda|h_{j}|\). Similar to [7; 6], we takes \(|h_{j}|\) into consideration, as BLEU is sensitive to length. For fair comparison, \(\lambda\) is tuned such that the average hypothesis length becomes roughly the same for all systems and approaches the average length of the references.7

Footnote 7: Approximately 10 words/tokens for Switchboard and 12 for Reddit

The automatic evaluation results are reported in Table 4. On both datasets, the proposed system consistently outperforms the baselines by a large margin in Precision, Recall, and F1.

Examples of system outputs and human references can be found in Table 5 and Table 6 for Reddit and Switchboard, respectively. As shown in the examples, CVAE+BOW and other baseline models may generate diverse but not-so-relevant responses.

 

the same average hypothesis length. We also randomly select one reference for each context and compare them with the systems (labeled "human" in Table 7)

As illustrated in Table 7, the proposed model outperforms all systems except human, consistent with our automatic evaluation results.

## 6 Conclusion

We propose a SpaceFusion model to jointly optimize diversity and relevance that leverages novel regularization terms to essentially fuse the latent space of a S2S model with that of an autoencoder model. This fused latent space exhibits desirable properties such as smooth semantic interpolation between two points. The distance and direction from the predicted response vector roughly match relevance and diversity, respectively. These properties also enable intuitive visualization of the latent space. Both automatic and human evaluation results demonstrate that the proposed approach brings significant improvement compared to strong baselines in terms of both diversity and relevance. In future work, we will provide theoretical justification of the effectiveness of the proposed regularization terms. We expect that this technique will find application as an efficient "mixing board" for conversation that draws on multiple sources of information.

\begin{table}
\begin{tabular}{l|p{143.8pt} p{56.9pt}} \hline
* hum. \\  & \(\circ\) Oh that would be nice. \\  & \(\circ\) Is that right? \\  & \(\circ\) It s a t's a luxury \\  & \(\circ\) Well that is interesting. \\ \hline MTask & \(\circ\) Um
- hum. \\  & \(\circ\) That’s right. \\  & \(\circ\) I don’t know how much money you’re going to do it. \\  & \(\circ\) That’s right. \\  & \(\circ\) Yeah \\ \hline S2S+ & \(\circ\) That’s great. \\ sampling & \(\circ\) What had you feel about this uh
- huh. \\  & \(\circ\) It’s just kind of a glass of town and it’s even irritating to me. I don’t know \\  & \(\circ\) Somebody and it’s so much better. \\  & \(\circ\) It now and it’s hard to go. \\ \hline human & \(\circ\) Yeah it is to some degree inconvenient i’d have to say that \\ \hline \end{tabular}
\end{table}
Table 6: Example system outputs and human reference on Switchboard dataset. Controversial or offensive words are replaced by ’***.

\begin{table}
\begin{tabular}{l|c c} \hline
**context** & Everything about this movie is awesome! \\ \hline Space Fusion & \(\circ\) I love this movie. \\ • To It’s so awesome!!! I have no idea how to watch this movie. I can’t wait for the trailer. \\ • \(\circ\) I don’t think i’m a fan of the movie. \\ • \(\circ\) I would love to see this. \\ • \(\circ\) I want to watch this movie. \\ \hline CVAE & \(\circ\) Smartphones of the best games!. \\ +BOW & \(\circ\) I’m in the same boat! I feel the same way about this \\ • \(\circ\) I don’t know why but can’t tell if the New York will be the only one. \\ • \(\circ\) Caves would never say yes, but I’d love to know. \\ • \(\circ\) I know where this movie is. \\ \hline MTask & \(\circ\) This is so funny. \\ • \(\circ\) I think I saw this. \\ • \(\circ\) I don’t know how many people do you. \\ • \(\circ\) I would. \\ • \(\circ\) I remember watching was disappointed. \\ \hline S2S+ sampling & \(\circ\) Whoa man, this is amazing!! \\ • \(\circ\) Man, I love Tom Cruise and I love the show. \\ • \(\circ\) Is this a bill on the right? \\ • \(\circ\) More like samurai jack. \\ • \(\circ\) I sweep for the invemind. \\ \hline human & \(\circ\) I love this movie. \\ • \(\circ\) Watched this yesterday! so amazing. \\ • \(\circ\) My favorite Muppet movie by far. \\ • \(\circ\) Seriously. Good times. Great oldies. \\ • \(\circ\) Now you have got the song into my head. \\ \hline \end{tabular}
\end{table}
Table 7: Performance of each model on human evaluation. The highest score, except human, in each row is in **bold**.

\begin{table}
\begin{tabular}{l|c c c} \hline  & relevance & interest & average \\ \hline SpaceFusion & **2.72** & **2.53** & **2.63** \\ CVAE+BOW & 2.51 & 2.37 & 2.44 \\ Multi-Task & 2.34 & 2.14 & 2.24 \\ S2S+Sampling & 2.58 & 2.43 & 2.50 \\ \hline human & 3.59 & 3.41 & 3.50 \\ \hline \end{tabular}
\end{table}
Table 5: Example system outputs and human reference on Reddit dataset 



## References

* [1]I. Borg and P. Groenen (2003) Modern multidimensional scaling: theory and applications. Journal of Educational Measurement40 (3), pp. 277-280. Cited by: SS5.
* S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio (2016) Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21. Cited by: SS1.
* K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio (2014) On the properties of neural machine translation: encoder-decoder approaches. In Proceedings of SSSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111. Cited by: SS1.
* H. Fu, C. Li, X. Liu, J. Gao, A. Celikyilmaz, and L. Carin (2019) Cyclical annealing schedule: a simple approach to mitigating kl vanishing. arXiv preprint arXiv:1903.10145. Cited by: SS1.
* M. Galley, C. Brockett, X. Gao, J. Gao, and B. Dolan (2019) Grounded response generation task at dstc7. External Links: Link Cited by: SS1.
* J. Gao, M. Galley, and L. Li (2019) Neural approaches to conversational ai. Foundations and Trends(r) in Information Retrieval13 (2-3), pp. 127-298. Cited by: SS1.
* M. Ghazvininejad, C. Brockett, M. Chang, B. Dolan, J. Gao, W. Yih, and M. Galley (2017) A knowledge-grounded neural conversation model. arXiv preprint arXiv:1702.01932. Cited by: SS1.
* J. Godfrey and E. Holliman (1997) Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, pp. 926:927. Cited by: SS1.
* X. Gu, K. Cho, J. Ha, and S. Kim (2018) DialogWAE: multimodal response generation with conditional wasserstein auto-encoder. arXiv preprint arXiv:1805.12352. Cited by: SS1.
* B. Huber, D. McDuff, C. Brockett, M. Galley, and B. Dolan (2018) Emotional dialogue generation using image-grounded language models. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 277. Cited by: SS1.
* D. P Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS1.
* J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan (2016a) A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119. Cited by: SS1.
* J. Li, M. Galley, C. Brockett, G. Spithourakis, J. Gao, and B. Dolan (2016b) A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, pp. 994-1003. Cited by: SS1.
* Y. Luan, C. Brockett, B. Dolan, J. Gao, and M. Galley (2017) Multi-task learning for speaker-role adaptation in neural conversation models. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Vol. 1, pp. 605-614. Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Cited by: SS1.
* R. Sennrich, B. Haddow, and A. Birch (2016) Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, pp. 86-96. Cited by: SS1.
* A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, and D. Batra (2016) Diverse beam search: decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424. Cited by: SS1.
* Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, Klaus Macherey, et al. (2016) Google's neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. Cited by: SS1.
* K. Yoshino, C. Hori, J. Perez, L. Fernando D'Haro, L. Polymenakos, C. Gunasekara, W. S. Lasecki, J. K. Kummerfeld, M. Galley, C. Brockett, et al. (2019) Dialog system technology challenge 7. arXiv preprint arXiv:1901.03461. Cited by: SS1.
* Y. Zhang, M. Galley, J. Gao, Z. Gan, X. Li, C. Brockett, and B. Dolan (2018) Generating informative and diverse conversational responses via adversarial information maximization. In Advances in Neural Information Processing Systems, pp. 1813-1823. Cited by: SS1.



Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 654-664.
* Zhou et al. [2018] Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2018. The design and implementation of xia-aice, an empathetic social chatbot. _arXiv preprint arXiv:1812.08989_.