Venue: NAACL 2019

# Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memory

 Deng Cai\({}^{1}\), Yan Wang\({}^{2}\), Wei Bi\({}^{2}\),

Zhaopeng Tu\({}^{2}\), Xiaojiang Liu\({}^{2}\), Wai Lam\({}^{1}\) and Shuming Shi\({}^{2}\)

\({}^{1}\)The Chinese University of Hong Kong, \({}^{2}\)Tencent AI Lab

thisisjcykcd@gmail.com, wlam@se.cuhk.edu.hk

{brandenwang,victoriabi,zptu,kieranliu,shumingshi}@tencent.com

Work done while DC was interning at Tencent AI Lab.Corresponding author.

###### Abstract

Traditional generative dialogue models generate responses solely from input queries. Such information is insufficient for generating a specific response since a certain query could be answered in multiple ways. Recently, researchers have attempted to fill the information gap by exploiting information retrieval techniques. For a given query, similar dialogues are retrieved from the entire training data and considered as an additional knowledge source. While the use of retrieval may harvest extensive information, the generative models could be overwhelmed, leading to unsatisfactory performance. In this paper, we propose a new framework which exploits retrieval results via a skeleton-to-response paradigm. At first, a skeleton is extracted from the retrieved dialogues. Then, both the generated skeleton and the original query are used for response generation via a novel response generator. Experimental results show that our approach significantly improves the informativeness of the generated responses.

## 1 Introduction

This paper focuses on tackling the challenges to develop a chit-chat style dialogue system (also known as chatbot). Chit-chat style dialogue system aims at giving meaningful and coherent responses given a dialogue query in the open domain. Most modern chit-chat systems can be categorized into two categories, namely, information retrieval-based (IR) models and generative models.

The IR-based models Ji et al. (2014); Hu et al. (2014) directly copy an existing response from a training corpus when receiving a response request. Since the training corpus is usually collected from real-world conversations and possibly post-edited by a human, the retrieved responses are informative and grammatical. However, the performance of such systems drops when a given dialogue history is substantially different from those in the training corpus.

The generative models Shang et al. (2015); Vinyals and Le (2015); Li et al. (2016a), on the other hand, generate a new utterance from scratch. While those generative models have better generalization capacity in rare dialogue contexts, the generated responses tend to be universal and noninformative (e.g., "I don't know", "I think so" etc.) Li et al. (2016a). It is partly due to the diversity of possible responses to a single query (i.e., the _one-to-many_ problem). The dialogue query alone cannot decide a meaningful and specific response. Thus a well-trained model tends to generate the most frequent (safe but boring) responses instead.

To summarize, IR-based models may give informative but inappropriate responses while generative models often do the opposite. It is desirable to combine both merits. Song et al. (2016) used an extra encoder for the retrieved response. The resulted dense representation, together with the original query, is used to feed the decoder in a standard Seq2Seq model Bahdanau et al. (2014). Weston et al. (2018) used a single encoder that takes the concatenation of the original query and the retrieved as input. Wu et al. (2019) noted that the retrieved information should be used in awareness of the context difference, and further proposed to construct an edit vector by explicitly encoding the lexical differences between the input query and the retrieved query.

However, in our preliminary experiments, we found that the IR-guided models are inclined to degenerate into a copy mechanism, in which the generative models simply repeat the retrieved response without necessary modifications. Sharp performance drop is caused when the retrieved re

sponse is irrelevant to the input query. A possible reason is that both useful and useless information is mixed in the dense vector space, which is uninterpretable and uncontrollable.

To address the above issue, we propose a new framework, skeleton-to-response, for response generation. Our motivations are two folds: (1) The guidance from IR results should only specify a response aspect or pattern, but leave the query-specific details to be elaborated by the generative model itself; (2) The retrieval results typically contain excessive information, such as inappropriate words or entities. It is necessary to filter out irrelevant words and derive a useful skeleton before use.

Our approach consists of two components: a skeleton generator and a response generator. The skeleton generator extracts a response skeleton by detecting and removing unwanted words in a retrieved response. The response generator is responsible for adding query-specific details to the generated skeleton for query-to-response generation. A dialogue example illustrating our idea is shown in Fig. 1. Due to the discrete choice of skeleton words, the gradient in the training process is no longer differentiable from the response to the skeleton generator. Two techniques are proposed to solve this issue. The first technique is to employ the policy gradient method for rewarding the output of the skeleton generator based on the feedback from a pre-trained critic. An alternative technique is to solve both the skeleton generation and the response generation in a multi-task learning fashion.

Our contributions are summarized as below: (1) We develop a novel framework to inject the power of IR results into generative response models by introducing the idea of skeleton generation; (2) Our approach generates response skeletons by detecting and removing unnecessary words, which facilitates the generation of specific responses while not spoiling the generalization ability of the underlying generative models; (3) Experimental results show that our approach significantly outperforms other compared methods, resulting in more informative and specific responses.

## 2 Models

In this work, we propose to construct a response skeleton based on the results of IR systems for guiding the response generation. The skeleton-to-response paradigm helps reduce the search space of possible responses and provides useful elements missing in the given query.

Our model consists of two components, namely, the skeleton generator and the response generator. These components are parameterized by the above two probabilistic models, denoted by \(\theta_{ske}\) and \(\theta_{res}\) respectively. Fig. 2 depicts the overall architecture of our proposed framework.

### Skeleton Generator

The skeleton generator transforms a retrieved response into a skeleton by explicitly removing inappropriate or useless information regarding the input query \(q\). We consider this procedure as a series of word-level masking actions. Following Wu et al. (2019), we first construct an edit vector by comparing the difference between the original query \(q\) and the retrieved query \(q^{\prime}\). In (Wu et al., 2019) the edit vector is used to guide the response generation directly. In our model, the edit vector is used to estimate the probability of being reserved or being masked for every word in a sentence. We define two word sets, namely _insertion words_\(I\) and _deletion words_\(D\). The insertion words include words that are in the original query \(q\), but not in the retrieved query \(q^{\prime}\), while the deletion words do the opposite.

The two bags of words highlight the changes in the dialogue context, corresponding to the changes in the response. The edit vector \(z\) is thus defined as the concatenation of the representations of the two bags of words. We use t

Figure 1: Our idea of leveraging the retrieved query-response pair. It first constructs a response skeleton by removing some words in the retrieved response, then a response is generated via rewriting based on the skeleton.

 

word embeddings to get the dense representations of \(I\) and \(D\). The edit vector is computed as:

\[z=\sum_{w_{1}\in I}\alpha_{w_{1}}\Phi(w_{1})\oplus\sum_{w_{2}\in D}\beta_{w_{2}} \Phi(w_{2}), \tag{1}\]

where \(\oplus\) is the concatenation operation. \(\Phi\) maps a word to its corresponding embedding vector, \(\alpha_{w_{1}}\) and \(\beta_{w_{2}}\) are the weights of an insertion word \(w_{1}\) and a deletion word \(w_{2}\) respectively. The weights of different words are derived by an attention mechanism Luong et al. (2015). Formally, the retrieved response \(r^{\prime}=(r^{\prime}_{1},r^{\prime}_{2}\dots,r^{\prime}_{|r^{\prime}|})\) is processed by a bidirectional GRU network (biGRU). We denote the states of the biGRU (i.e. concatenation of forward and backward GRU states) as \((h_{1},h_{2},\ldots,h_{|r^{\prime}|})\). The weight \(\alpha_{w_{1}}\) is calculated by:

\[\alpha_{w_{1}}=\frac{\exp(s_{w_{1}})}{\sum_{w\in I}\exp(s_{w})},\]

\[s_{w_{1}}=v^{\top}_{I}\tanh(W_{I}[\Phi(w_{1})\oplus h_{|r^{\prime}|}]), \tag{2}\]

where \(v_{I}\) and \(W_{I}\) are learnable parameters. The weight \(\beta_{w_{2}}\) is obtained in a similar way with another set of parameters \(v_{D}\) and \(W_{D}\).

After acquiring the edit vector, we transform the prototype response \(r^{\prime}\) to a skeleton \(t\) by the following equations:

\[t=(\phi(r^{\prime}_{1},h_{1},z),\phi(r^{\prime}_{2},h_{2},z),\cdots,\phi(r^{ \prime}_{|r^{\prime}|},h_{|r^{\prime}|},z)),\]

\[\phi(r^{\prime}_{i},h_{i},z)=\begin{cases}<\text{blank}>&\text{if $\hat{m}_{ i}=0$}\\ r^{\prime}_{i}&\text{else}\end{cases}, \tag{3}\]

where \(\hat{m}_{i}\) is the indicator and equals 0 if \(r^{\prime}_{i}\) is replaced with a placeholder "\(<\)blank\(>\)" and 1 otherwise. The probability of \(\hat{m}_{i}=1\) is computed by

\[P(\hat{m}_{i}=1)=\text{sfigmoid}(W_{m}[h_{i}\oplus z]+b_{m}). \tag{4}\]

### Response Generator

The response generator can be implemented using most existing IR-augmented models Song et al. (2016); Weston et al. (2018); Pandey et al. (2018), just by replacing the retrieved response input with the corresponding skeleton. We discuss our choices below.

EncodersTwo separate bidirectional LSTM (biLSTM) networks are used to obtain the distributed representations of the query memories and the skeleton memories, respectively. For biLSTM,

Figure 2: The architecture of our framework. Given a query “Do you like banana”, a similar historical query “Do you like apple” is retrieved along with its response, i.e., “Yes, apple is my favorite”. Upper: The skeleton generator removes inappropriate words and extracts a response skeleton. Lower: The response generator generates a response based on both the skeleton and the query.

 

the concatenation of the forward and the backward hidden states at each token position is considered a memory slot, producing two memory pools: \({\cal M}_{q}=\{h_{1},h_{2},\ldots,h_{|q|}\}\) for the input query, and \({\cal M}_{t}=\{h^{\prime}_{1},h^{\prime}_{2},\ldots,h^{\prime}_{|t|}\}\) for the skeleton.1

Footnote 1: Note the skeleton memory pool \({\cal M}_{t}\) could contain multiple response skeletons, further discussed in the experiment section.

DecoderDuring the generation process, our decoder reads information from both the query and the skeleton using attention mechanism Bahdanau et al. (2014); Luong et al. (2015). To query the memory pools, the decoder uses the hidden state \(s_{t}\) of itself as the searching key. The matching score function is implemented by bilinear functions:

\[\alpha(h_{k},s_{t})={h_{k}}^{T}W_{q}s_{t};\beta(h^{\prime}_{k},s_{t})={h^{ \prime}_{k}}^{T}W_{t}s_{t}, \tag{5}\]

where \(W_{q}\) and \(W_{t}\) are trainable parameters. A query context vector \(c_{t}\) is then computed as a weighted sum of all memory slots in \({\cal M}_{q}\), where the weight for a memory slot \(h_{k}\) is \(\exp(\alpha(h_{k},s_{t}))/(\sum_{i=1}^{|q|}\exp(\alpha(h_{i},s_{t})))\). A skeleton context vector \(c^{\prime}_{t}\) is computed in a similar spirit by using \(\beta(h^{\prime}_{k},s_{t})\)'s.

The probability of generating the next word \(r_{t}\) is then jointly determined by the decoder's state \(s_{t}\), the query context \(c_{t}\) and the skeleton context \(c^{\prime}_{t}\). We first fuse the information of \(s_{t}\) and \(c_{t}\) by a linear transformation. For \(c^{\prime}_{t}\), a gating mechanism is additionally introduced to control the information flow from skeleton memories. Formally, the probability of the next token \(r_{t}\) is estimated by \(y_{t}\) followed by a softmax function over the vocabulary:

\[y_{t}=(W_{c}[s_{t}\oplus c_{t}])\cdot g_{t}+c^{\prime}_{t}\cdot(1-g_{t}), \tag{6}\]

where \(g_{t}=f_{g}(s_{t},c_{t},c^{\prime}_{t})\) is implemented by a single layer neural network with sigmoid output layer.

## 3 Learning

Given that our skeleton generator performs non-differentiable hard masking, the overall model cannot be trained end-to-end using the standard maximum likelihood estimate (MLE). A possible solution that circumvents this problem is to treat the skeleton generation and the response generation as two parallel tasks and solve them jointly in a multi-task learning fashion. An alternative is to bridge the skeleton generator and the final response output using reinforcement learning (RL) methods, which can exclusively inform the skeleton generator with the ultimate goal. The latter option is referred as _cascaded integration_ while the former is called _joint integration_.

Recall that we have formulated the skeleton generation as a series of binary classifications. Nevertheless, most of the dialogue datasets are end-to-end query-response pairs without explicit skeletons. Hence, we propose to construct _proxy skeletons_ to facilitate the training.

**Definition 1** Proxy Skeleton: _Given a training quadruplet \((q,q^{\prime},r,r^{\prime})\) and a stop word list \(S\), the proxy skeleton for \(r\) is generated by replacing some tokens in \(r^{\prime}\) with a placeholder "\(<\)blank\(>\)". A token \(r^{\prime}_{i}\) is kept if and only if it meets the following conditions_

1. \(r^{\prime}_{i}\notin S\)__
2. \(r^{\prime}_{i}\) _is a part of the longest common subsequence (LCS) Wagner and Fischer (1974) of_ \(r\) _and_ \(r^{\prime}\)_._

The proxy skeletons are used in different manners according to the integration method, which we will introduce below.

### Joint Integration

To avoid breaking the differentiable computation, we connect the skeleton generator and the response generator via a shared network architecture rather than by passing the discrete skeletons. Concretely, the last hidden states in our skeleton generator (i.e, the hidden states that are utilized to make the masking decisions) are used as the skeleton memories in response generation. The training objective is the sum of the proxy skeleton labels likelihood \(L(\theta_{she})\) and the response likelihood \(L(\theta_{res})\):

\[L(\theta_{res}\cup\theta_{she})=L(\theta_{res})+\eta L(\theta_{she}), \tag{7}\]

where \(\eta\) is a harmonic weight, and it is set as \(1.0\) in our experiments.

### Cascaded Integration

Policy gradient methods Williams (1992) can be applied to optimize the full model while keeping it running as cascaded process. We regard the skeleton generator as the first RL agent, and the response generator as the second one. The final output generated by the pipeline process and the 

intermediate skeleton are denoted by \(\hat{r}\) and \(\hat{t}\) respectively. Given the original query \(q\) and the generated response \(\hat{r}\), a reward \(R(q,\hat{r})\) for generating \(\hat{r}\) is calculated. All network parameters are then optimized to maximize the expected reward by the policy gradient.

The reward function \(R\) should convey both the naturalness of the generated response and its relevance to the given query \(q\). A pre-trained critic is utilized to make the judgment. Inspired by comparative adversarial learning in Li et al. (2018), we design the critic as a classifier that receives four inputs every time: the query \(q\), a human-written response \(r\), a machine-generated response \(\hat{r}\) and a random response \(\overline{r}\) (yet written by human). The critic is trained to pick the human-written response \(r\) among others correctly. Formally, the following objective is maximized:

\[\log D(r|q,\hat{r},\bar{r},r)=\log\frac{\exp(h_{r}^{\text{T}}M_{D}h_{q})}{\sum \limits_{x\in\{\tilde{r},\bar{r},r\}}\exp(h_{x}^{\text{T}}M_{D}h_{q})}, \tag{8}\]

where \(h_{x}\) is a vector representation of \(x\), produced by a bidirectional LSTM (the last hidden state), and \(M_{D}\) is a trainable matrix.2

Footnote 2: Note the classifier could be fine-tuned with the training of our generators, which falls into the adversarial learning setting Goodfellow et al. (2014).

## 4 Related Work

Multi-source Dialogue GenerationChit-chat style dialogue system dates back to ELIZA Weizenbaum (1966). Early work uses handcrafted rules, while modern systems usually use data-driven approaches, e.g., information retrieval techniques. Recently, end-to-end neural approaches Vinyals and Le (2015); Serban et al. (2016); Li et al. (2016a); Sordoni et al. (2015) have attracted increasing interest. For those generative models, a notorious problem is the "safe response" problem: the generated responses are dull and generic, which may attribute to the lack of sufficient input information. The query alone cannot specify an informative response. To mitigate the issue, many research efforts have been paid to introducing other information source, such as unsupervised latent variable Serban et al. (2017); Zhao et al. (2018); Cao and Clark (2017); Shen et al. (2017), discourse-level variations Zhao et al. (2017), topic information Xing et al. (2017), speaker personality Li et al. (2016b) and knowledge base Ghazvininejad et al. (2018); Zhou et al. (2018). Our work follows the similar motivation and uses the output of IR systems as the additional knowledge source.

Combination of IR and Generative modelsTo combine IR and generative models, early work Qiu et al. (2017) tried to re-rank the output from both models. However, the performance of such models is limited by the capacity of individual methods. Most related to our work, Song et al. (2016); Weston et al. (2018) and Wu et al. (2019) encoded the retrieved result into distributed representation and used it as the additional conditionals along with the standard query representation. While the former two only used the target side of the retrieved pairs, the latter took advantages of both sides. In a closed domain conversation setting Pandey et al. (2018) further proposed to weight different training instances by context similarity. Our model differs from them in that we take an extra intermediate step for skeleton generation to filter the retrieval information before use, which shows the effectiveness in avoiding the erroneous copy in our experiments.

Multi-step Language GenerationOur work is also inspired by the recent success of decomposing an end-to-end language generation task into several sequential sub-tasks. For document summarization, Chen and Bansal (2018) first select salient sentences and then rewrite them in parallel. For sentiment-to-sentiment translation, Xu et al. (2018) first use a neutralization module to remove emotional words and then add sentiment to the neutralized content. Not only does their decomposition improve the overall performance, but also makes the whole generation process more interpretable. Our skeleton-to-response framework also sheds some light on the use of retrieval memories.

## 5 Experiments

### Data

We use the preprocessed data in Wu et al. (2019) as our test bed. The total dataset consists of about 20 million single-turn query-response pairs collected from Douban Group3. Since similar contexts may correspond to totally different responses, the training quadruples \((q,r,q^{\prime},r^{\prime})\) for 

IR-augmented models are constructed based on response similarity. All response are indexed by Lucene.4 For each \((q,r)\) pair, top 30 similar responses with their corresponding contexts are retrieved \(\{(q^{\prime}_{i},r^{\prime}_{i})\}_{i=1}^{30}\). However, only those satisfying \(0.3\leq Jaccard(r,r^{\prime}_{i})\leq 0.7\) are leveraged for training, where \(Jaccard\) measures the Jaccard distance. The reason for the data filter is that nearly identical responses drive the model to do simple copy while distantly different responses make the model ignore the retrieval input. About 42 million quadruples are obtained afterward.

Footnote 4: [https://lucene.apache.org/core/](https://lucene.apache.org/core/)

For computational efficiency, we randomly sample 5 million quadruples as training data for all experiments. The test set consists of 1,000 randomly selected queries that are not in our training data.5 For a fair comparison, when training a generative model without the help of IR, the quadruples are split into pairs.

Footnote 5: Note the retrieval results for test data are based on query similarity, and no data filter is adopted.

### Model Details

We implement the skeleton generator based on a bidirectional recurrent neural network with 500 LSTM units. We concatenate the hidden states from both directions. The word embedding size is set to 300. For the response generator, the encoder for queries, the encoder for skeletons and the decoder are three two-layer recurrent neural networks with 500 LSTM units, where both encoders are bidirectional. We use dropout Srivastava et al. (2014) to alleviate overfitting. The dropout rate is set to 0.3 across different layers. The same architecture for the encoders and the decoder is shared across the following baseline models, if applicable.

### Compared Methods

* **Seq2Seq** the standard attention-based RNN encoder-decoder model Bahdanau et al. (2014).
* **MMI Seq2Seq** with Maximum Mutual Information (MMI) objective in decoding Li et al. (2016a). In practice, an inverse (response-to-query) Seq2Seq model is used to rerank the \(N\)-best hypothesizes from the standard Seq2Seq model (\(N\) equals 100 in our experiments).
* **EditVec** the model proposed by Wu et al. (2019), where the edit vector \(z\) is used directly at each decoding step by concatenating it to the word embeddings.
* **IR** the Lucene system is also used a benchmark.6 Footnote 6: Note its respects response candidates from the entire data collection, not restricted to the filtered one.
* **IR+rerank** rerank the results of **IR** by **MMI**.

Besides, We use **JNT** to denote our model with joint integration, and **CAS** for our model with cascaded integration. To validate the usefulness of the proposed skeletons. We design a response generator that takes an intact retrieval response as its skeleton input (i.e., to completely skip the skeleton generation step), denoted by **SKP**.7

Footnote 7: There are some other IR-augmented models using standard seq2seq models as SKP. Weston et al. (2018) used a rule to select either the generated response or the retrieved response as output, while we would like to focus on improving the quality of generated responses. Pandey et al. (2018) concentrated on closed domain conversations, their hierarchical encoder is not suitable for our open domain setting. We thus omit the empirical comparison with them.

### Evaluation Metrics

Our method is designed to improve the informativeness of the generative model and alleviate the inappropriateness problem of the retrieval model. To measure the performance effectively, we use

\begin{table}
\begin{tabular}{c|c c c} \hline model & human score & dist-1 & dist-2 \\ \hline IR & 2.093 & **0.238** & **0.723** \\ IR+rerank & 2.520 & 0.208 & 0.586 \\ Seq2Seq & 2.433 & 0.156 & 0.336 \\ MMI & 2.554 & 0.170 & 0.464 \\ EditVec & 2.588† & 0.154 & 0.394 \\ SKP & 2.581 & 0.152 & 0.406 \\ JNT & 2.612‡ & 0.147 & 0.377 \\ CAS & **2.747** & 0.156 & 0.411 \\ \hline \end{tabular}
\end{table}
Table 1: Response performance of different models. Sign tests on human score show that the CAS is significantly better than all other methods with p-value \(<0.05\), and the p-value \(<0.01\) except for those marked by †.

\begin{table}
\begin{tabular}{l|c c c c} \hline model & P & R & F\({}_{1}\) & Acc. \\ \hline JNT & 0.32 & 0.61 & 0.42 & 0.60 \\ CAS & 0.50 & 0.86 & 0.63 & 0.76 \\ \hline \end{tabular}
\end{table}
Table 2: Performance of skeleton generator.

 

human evaluation along with two automatic evaluation metrics.
* **Human evaluation** We asked three experienced annotators to score the group of responses (the best output of each model) for 300 test queries. The responses are rated on a five-point scale. A response should be scored 1 if it can hardly be considered a valid response, 3 if it is a valid but not informative response, 5 if it is an informative response, which can deepen the discussion of the current topic or lead to a new topic. 2 and 4 are for decision dilemmas.
* **dist-1 & dist-2** It is defined as the number of unique uni-grams (**dist-1**) or bi-grams (**dist-2**) dividing by the total number of tokens, measuring the diversity of the generated responses Li et al. (2016a). Note the two metrics do not necessarily reflect the response quality as the target queries are not taken into consideration.

### Response Generation Results

The results are depicted in Table 1. Overall, both of our models surpass all other methods, and our cascaded model (CAS) gives the best performance according to human evaluation. The contrast with the SKP model illustrates that the use of skeletons brings a significant performance gain.

According to the dist-1&2 metrics, the generative models achieve significantly better diversity by the use of retrieval results. The retrieval method yields the highest diversity, which is consistent with our intuition that the retrieval responses typically contain a large amount of information though they are not necessarily appropriate. The model of MMI also gives strong diversity, yet we find that it tends to merely repeat the words in queries. By removing the words in queries, the dist-2 of MMI and CAS become 0.710 and 0.751 respectively. It indicates our models are better at generating new words.

To further reveal the source of performance gain, we study the relation between response quality and query similarity (measured by the Jaccard similarity between the input query and the retrieved query). Our best model (CAS) is compared with the strong IR system (IR-rerank) and the previous state-of-the-art (EditVec) in Fig. 3. The CAS model significantly boosts the performance when query similarity is relatively low, which indicates that introducing skeletons can alleviate erroneous copy and keep a strong generalization ability of the underlying generative model.

### More Analysis of Our Framework

Here, we present further discussions and empirical analysis of our framework.

Generated SkeletonsAlthough generating skeletons is not our primary goal, it is interesting to assess the skeleton generation. The word-level precision (P), recall (R), F\({}_{1}\) score (F\({}_{1}\)) and accuracy (Acc.) of the well-trained skeleton generators are reported in Table 2, taking the proxy skeletons as golden references.

Table 3 shows some skeleton-to-response examples of the CAS model and a case study among different models. In the leftmost example in Table 3, the MMI and the EditVec simply repeat the query while the retrieved response is weakly related to the query. Our CAS model extracts a useful word 'boy' from the retrieved response and generates a more interesting response. In the middle example, the MMI response makes less sense, and some private information is included in the retrieved response. Our CAS model removes the privacy without the loss of informativeness, while the outputs by other models are less informative. The rightmost case shows that our response generator is able to recover the possible mistakes made by the skeleton generator.

Retrieved Response v.s. Generated Response To measure the extent that the generative models are copying the retrieval, we compute the edit distances between generated responses and retrieved

Figure 3: Response quality v.s. query similarity.8



responses. As shown in Fig. 4, in the comparison between the SKP and other models, the use of skeletons makes the generated response deviate more from its prototype response. Ideally, when the retrieved context is very similar to the input query, the changes between the generated response and the prototype response should be minor. Conversely, the changes should be drastic. Fig. 4 also shows that our models can learn this intuition.

Single v.s. Multiple Retrieval Pair(s)For a given query \(q\), the retrieval pair set \(R_{q}\) could contain multiple query-response pairs. We investigate two ways of using it under the CAS setting.

* **Single** For each query-response pair \((q^{\prime}_{i},r^{\prime}_{i})\in R_{q}\), a response \(\hat{r}_{i}\) is generated solely based on \(q\), and \((q^{\prime}_{i},r^{\prime}_{i})\). The resulted responses are re-ranked by generation probability.
* **Multiple** The whole retrieval set \(R_{q}\) is used in a single run. Multiple skeletons are generated and concatenated in the response generation stage.

The results are shown in Table 4. We attribute the failure of **Multiple** to the huge variety of the retrieved responses. The response generator receives many heterogeneous skeletons, yet it has no idea which to use. It remains an open question on how to effectively use multiple retrieval pairs for generating one single response, and we leave it for future work.

## 6 Conclusion

In this paper, we proposed a new methodology to enhance generative models with information retrieval technologies for dialogue response generation. Given a dialogue context, our methods generate a skeleton based on historical responses that respond to a similar context. The skeleton serves as an additional knowledge source that helps specify the response direction and complement the response content. Experiments on real world data validated the effectiveness of our method for more informative and appropriate responses.

Figure 4: Changes between retrieved and generated responses v.s. query similarity.

\begin{table}
\begin{tabular}{c|c c c} \hline setting & human score & dist-1 & dist-2 \\ \hline Single & 2.747 & 0.156 & 0.411 \\ Multiple & 1.976 & 0.178 & 0.414 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of the usages of the retrieval set.

 



## Acknowledgments

We thank the anonymous reviewers for their helpful comments. The work described in this paper is partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14203414).

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In _ICLR_.
* Cao and Clark (2017) Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In _EACL_, pages 182-187.
* Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In _ACL_.
* Ghazvininejad et al. (2018) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In _AAAI_, pages 5110-5117.
* Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In _Advances in neural information processing systems_, pages 2672-2680.
* Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In _NIPS_, pages 2042-2050.
* Ji et al. (2014) Zongcheng Ji, Zhengdong Lu, and Hang Li. 2014. An information retrieval approach to short text conversation. _arXiv preprint arXiv:1408.6988_.
* Li et al. (2018) Dianqi Li, Xiaodong He, Qiuyuan Huang, Ming-Ting Sun, and Lei Zhang. 2018. Generating diverse and accurate visual captions by comparative adversarial learning. _arXiv preprint arXiv:1804.00861_.
* Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In _NAACL_, pages 110-119.
* Li et al. (2016b) Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In _ACL_, pages 994-1003.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _EMNLP_, pages 1412-1421.
* Pandey et al. (2018) Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In _ACL_, pages 1329-1338.
* Qiu et al. (2017) Minghui Qiu, Feng-Lin Li, Siyu Wang, Xing Gao, Yan Chen, Weipeng Zhao, Haiqing Chen, Jun Huang, and Wei Chu. 2017. Alime chat: A sequence to sequence and rerank based chatbot engine. In _ACL_, pages 498-503.
* Serban et al. (2016) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _AAAI_, volume 16, pages 3776-3784.
* Serban et al. (2017) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In _AAAI_, pages 3295-3301.
* Shang et al. (2015) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In _ACL_, pages 1577-1586.
* Shen et al. (2017) Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017. A conditional variational framework for dialog generation. In _ACL_, pages 504-509.
* Song et al. (2016) Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. _arXiv preprint arXiv:1610.07149_.
* Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In _NAACL_, pages 196-205.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In _ICML (Deep Learning Workshop)_.
* Wagner and Fischer (1974) Robert A Wagner and Michael J Fischer. 1974. The string-to-string correction problem. _Journal of the ACM (JACM)_, 21(1):168-173.
* Weizenbaum (1966) Joseph Weizenbaum. 1966. Eliza--a computer program for the study of natural language communication between man and machine. _Communications of the ACM_, 9(1):36-45.
* Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. _arXiv preprint arXiv:1808.04776_

Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Wu et al. [2019] Yu Wu, Furu Wei, Shaohan Huang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In _AAAI_.
* Xing et al. [2017] Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In _AAAI_, pages 3351-3357.
* Xu et al. [2018] Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In _ACL_, page 675-686.
* Zhao et al. [2018] Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In _ACL_, pages 1098-1107.
* Zhao et al. [2017] Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _ACL_, pages 654-664.
* Zhou et al. [2018] Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In _IJCAI_, pages 4623-4629.