As how to evaluate response generation automatically is still an open problem  , we further conduct human evaluations to compare our models with baselines. We ask the same three native speakers to do a side-by-side comparison  on the 1,000 contexts. Given a context and two responses generated by different models, we ask annotators to decide which response is better (Ties are permitted).

Evaluation Metrics	We evaluate our model on four criteria: fluency, relevance, diversity and originality. We employ Embedding Average (Average), Embedding Extrema (Extrema), and Embedding Greedy (Greedy)  to evaluate response relevance, which are better correlated with human judgment than BLEU. Following  , we evaluate the response diversity based on the ratios of distinct unigrams and bigrams in generated responses, denoted as Distinct-1 and Distinct-2. In this paper, we define a new metric, originality, that is defined as the ratio of generated responses that do not appear in the training set. Here, “appear" means we can find exactly the same response in our training data set. We randomly select 1,000 contexts from the test set, and ask three native speakers to annotate response fluency. We conduct 3-scale rating: +2, +1 and 0. +2: The response is fluent and grammatically correct. +1: There are a few grammatical errors in the response but readers could understand it. 0: The response is totally grammatically broken, making it difficult to understand.

Apart from work on “safe response", there is a growing body of literature on style transfer  ,  and emotional response generation  . In general, most of previous work generates a response from scratch either left-to-right or conditioned on a latent vector, whereas our approach aims to generate a response by editing a prototype. Prior works have attempted to utilize prototype responses to guide the generation process  ,  , in which prototype responses are encoded into vectors and feed to a decoder along with a context representation. Our work differs from previous ones on two aspects. One is they do not consider prototype context in the generation process, while our model utilizes context differences to guide editing process. The other is that we regard prototype responses as a source language, while their works formulate it as a multi-source seq2seq task, in which the current context and prototype responses are all source languages in the generation process. Recently, some researches have explored natural language generation by editing  ,  . A typical approach follows a writing-then-edit paradigm, that utilizes one decoder to generate a draft from scratch and uses another decoder to revise the draft  .

Introduction	In recent years, non-task oriented chatbots focused on responding to humans intelligently on a variety of topics, have drawn much attention from both academia and industry. Existing approaches can be categorized into generation-based methods  ,  ,  ,  ,  which generate a response from scratch, and retrieval-based methods  ,  ,  ,  ,  which select a response from an existing corpus. Since retrieval-based approaches are severely constrained by a pre-defined index, generative approaches become more and more popular in recent years. Traditional generation-based approaches, however, do not easily generate long, diverse and informative responses, which is referred to as “safe response" problem  . To address this issue, we propose a new paradigm, prototype-then-edit, for response generation. Our motivations include: 1) human-written responses, termed as “prototypes response", are informative, diverse and grammatical which do not suffer from short and generic issues. Hence, generating responses by editing such prototypes is able to alleviate the “safe response" problem. 2) Some retrieved prototypes are not relevant to the current context, or suffer from a privacy issue. The post-editing process can partially solve these two problems.

Response Generation by Context-aware Prototype Editing	Open domain response generation has achieved remarkable progress in recent years, but sometimes yields short and uninformative responses. We propose a new paradigm for response generation, that is response generation by editing, which significantly increases the diversity and informativeness of the generation results. Our assumption is that a plausible response can be generated by slightly revising an existing response prototype. The prototype is retrieved from a pre-defined index and provides a good start-point for generation because it is grammatical and informative. We design a response editing model, where an edit vector is formed by considering differences between a prototype context and a current context, and then the edit vector is fed to a decoder to revise the prototype response for the current context. Experiment results on a large scale dataset demonstrate that the response editing model outperforms generative and retrieval-based models on various aspects.

Edit-N-Rerank edits all 20 responses returned by Lucene and then reranks the revised results with the dual-LSTM model. We also merge edit results of Edit-N-Rerank and candidates returned by the search engine, and then rerank them, which is denoted as Edit-Merge. In practice, the word embedding size and editor vector size are 512, and both of the encoder and decoder are a 1-layer GRU whose hidden vector size is 1024. Message and response vocabulary size are 30000, and words not covered by the vocabulary are represented by a placeholder $UNK$. Word embedding size, hidden vector size and attention vector size of baselines and our models are the same. All generative models use beam search to yield responses, where the beam size is 20 except S2SA-MMI. For all models, we remove $UNK$ from the target vocabulary, because it always leads to a fluency issue in evaluation.

That is an acceptable fluency score for a dialogue engine, indicating that most of generation responses are grammatically correct. In addition, in terms of the fluency metric, Fleiss' Kappa  on all models are around 0.8, showing a high agreement among labelers. Compared to ensemble models, our model performs much better on diversity and originality, that is because we regard prototype response instead of the current context as source sentence in the Seq2Seq, which keeps most of content in prototype but slightly revises it based on the context difference. Both of the ensemble and edit model are improved when the original retrieval candidates are considered in the rerank process. Regarding human side-by-side evaluation, we can find that Edit-Default and Edit-N-rerank are slightly better than Retrieval-default and Retrieval-rerank (The winning examples are more than the losing examples), indicating that the post-editing is able to improve the response quality. Ed-Default is worse than Ens-Default, but Ed-N-Rerank is better than Ens-Rerank. This is mainly because the editing model regards the prototype response as the source language, so it is highly depends on the quality of prototype response.

Evaluation Results	Table TABREF25 shows the evaluation results on the Chinese dataset. Our methods are better than retrieval-based methods on embedding based metrics, that means revised responses are more relevant to ground-truth in the semantic space. Our model just slightly revises prototype response, so improvements on automatic metrics are not that large but significant on statistical tests (t-test, p-value INLINEFORM0 ). Two factors are known to cause Edit-1-Rerank worse than Retrieval-Rerank. 1) Rerank algorithm is biased to long responses, that poses a challenge for the editing model. 2) Despite of better prototype responses, a context of top-1 response is always greatly different from current context, leading to a large insertion word set and a large deletion set, that also obstructs the revision process. In terms of diversity, our methods drop on distinct-1 and distinct-2 in a comparison with retrieval-based methods, because the editing model often deletes special words pursuing for better relevance. Retrieval-Rerank is better than retrieval-default, indicating that it is necessary to rerank responses by measuring context-response similarity with a matching model.

The matching model is implemented with the same setting in  , and is trained on the training data set where negative instances are randomly sampled with a ratio of INLINEFORM0 . Ensemble Model: Song et al song2016two propose an ensemble of retrieval and generation methods. It encodes current context and retrieved responses (Top-k retrieved responses are all used in the generation process.) into vectors, and feeds these representations to a decoder to generate a new response. As there is no official code, we implement it carefully by ourselves. We use the top-1 response returned by beam search as a baseline, denoted as Ensemble-default. For a fair comparison, we further rerank top 20 generated results with the same LSTM based matching model, and denote it as Ensemble-Rerank. We further create a candidate pool by merging the retrieval and generation results, and rerank them with the same ranker. The method is denoted as Ensemble-Merge. Correspondingly, we evaluate three variants of our model. Specifically, Edit-default and Edit-1-Rerank edit top-1 response yielded by Retrieval-default and Retrieval-Rerank respectively.

Then, the edit vector INLINEFORM15 is sampled from a distribution whose parameters are governed by the concatenation of INLINEFORM16 and INLINEFORM17 . Finally, the edit vector and output of the encoder are fed to the decoder to generate INLINEFORM18 . For response generation, which is a conditional setting of text editing, an interesting question raised, that is how to generate the edit by considering contexts. We will introduce our motivation and model in details in the next section.

Our methods significantly outperform generative baselines in terms of diversity since prototype responses are good start-points that are diverse and informative. It demonstrates that the prototype-then-editing paradigm is capable of addressing the safe response problem. Edit-Rerank is better than generative baselines on relevance but Edit-default is not, indicating a good prototype selector is quite important to our editing model. In terms of originality, about 86 INLINEFORM0 revised response do not appear in the training set, that surpasses S2SA, S2SA-MMI and CVAE. This is mainly because baseline methods are more likely to generate safe responses that are frequently appeared in the training data, while our model tends to modify an existing response that avoids duplication issue. In terms of fluency, S2SA achieves the best results, and retrieval based approaches come to the second place. Safe response enjoys high score on fluency, that is why S2SA and S2SA-MMI perform well on this metric. Although editing based methods are not the best on the fluency metric, they also achieve a high absolute number.

It should be noted that there are several alternative approaches to compute INLINEFORM0 and INLINEFORM1 for this task, such as applying memory networks, latent variables, and other complex network architectures. Here, we just use a simple method, but it yields interesting results on this task. We will further illustrate our experiment findings in the next section. We build our prototype editing model upon a Seq2Seq with an attention mechanism model, which integrates the edit vector into the decoder. The decoder takes INLINEFORM0 as an input and generates a response by a GRU language model with attention. The hidden state of the decoder is acquired by DISPLAYFORM0  where the input of INLINEFORM0 -th time step is the last step hidden state and the concatenation of the INLINEFORM1 -th word embedding and the edit vector obtained in Equation EQREF14 . Then we compute a context vector INLINEFORM2 , which is a linear combination of INLINEFORM3 : DISPLAYFORM0  where INLINEFORM0 is given by DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are parameters. The generative probability distribution is given by DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are two parameters.

The Jaccard similarity measures text similarity from a bag-of-word view, that is formulated as DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are two bags of words and INLINEFORM2 denotes the number of elements in a collection. Each context-response pair is processed with the above procedure, so we obtain enormous quadruples INLINEFORM3 after this step. The motivation behind filtering out instances with Jaccard similarity INLINEFORM4 is that a neural editor model performs well only if a prototype is lexically similar  to its ground-truth. Besides, we hope the editor does not copy the prototype so we discard instances where the prototype and groundtruth are nearly identical (i.e. Jaccard similarity INLINEFORM5 ). We do not use context similarity to construct parallel data for training, because similar contexts may correspond to totally different responses, so-called one-to-many phenomenon  in dialogue generation, that impedes editor training due to the large lexicon gap. According to our preliminary experiments, the editor always generates non-sense responses if training data is constructed by context similarity.

Discussions	We train variants of our model by removing the insertion word vector, the deletion word vector, and both of them respectively. The results are shown in Table TABREF29 . We can find that embedding based metrics drop dramatically when the editing vector is partially or totally removed, indicating that the edit vector is crucial for response relevance. Diversity and originality do not decrease after the edit vector is removed, implying that the retrieved prototype is the key factor for these two metrics. According to above observations, we conclude that the prototype selector and the context-aware editor play different roles in generating responses. It is interesting to explore the semantic gap between prototype and revised response. We ask annotators to conduct 4-scale rating on 500 randomly sampled prototype-response pairs given by Edit-default and Edit-N-Rerank respectively. The 4-scale is defined as: identical, paraphrase, on the same topic and unrelated. Figure FIGREF34 provides the ratio of four editing types defined above. For both methods, Only INLINEFORM0 of edits are exactly the same with the prototype, that means our model does not downgrade to a copy model. Surprisingly, there are INLINEFORM1 revised responses are unrelated to prototypes.

3) Lexical differences between contexts provide an important signal for response editing. If a word appears in the current context but not in the prototype context, the word is likely to be inserted into the prototype response in the editing process. Inspired by this idea, we formulate the response generation process as follows. Given a conversational context INLINEFORM0 , we first retrieve a similar context INLINEFORM1 and its associated response INLINEFORM2 from a pre-defined index, which are called prototype context and prototype response respectively. Then, we calculate an edit vector by concatenating the weighted average results of insertion word embeddings (words in prototype context but not in current context) and deletion word embeddings (words in current context but not in prototype context). After that, we revise the prototype response conditioning on the edit vector. We further illustrate how our idea works with an example in Table TABREF1 . It is obvious that the major difference between INLINEFORM3 and INLINEFORM4 is what the speaker eats, so the phrase “raw green vegetables" in INLINEFORM5 should be replaced by “desserts" in order to adapt to the current context INLINEFORM6 .

Background	Before introducing our approach, we first briefly describe state-of-the-art natural language editing method  . Given a sentence pair INLINEFORM0 , our goal is to obtain sentence INLINEFORM1 by editing the prototype INLINEFORM2 . The general framework is built upon a Seq2Seq model with an attention mechanism, which takes INLINEFORM3 and INLINEFORM4 as source sequence and target sequence respectively. The main difference is that the generative probability of a vanilla Seq2Seq model is INLINEFORM5 whereas the probability of the edit model is INLINEFORM6 where INLINEFORM7 is an edit vector sampled from a pre-defined distribution like variational auto-encoder. In the training phase, the parameter of the distribution is conditional on the context differences. We first define INLINEFORM8 as an insertion word set, where INLINEFORM9 is a word added to the prototype, and INLINEFORM10 is a deletion word set, where INLINEFORM11 is a word deleted from the prototype. Subsequently, we compute an insertion vector INLINEFORM12 and a deletion vector INLINEFORM13 by a summation over word embeddings in two corresponding sets, where INLINEFORM14 transfers a word to its embedding.

Related Work	Research on chatbots goes back to the 1960s when ELIZA was designed  with a huge amount of hand-crafted templates and rules. Recently, researchers have paid more and more attention on data-driven approaches  ,  due to their superior scalability. Most of these methods are classified as retrieval-based methods  ,  and generation methods  ,  ,  . The former one aims to select a relevant response using a matching model, while the latter one generates a response with natural language generative models. Prior works on retrieval-based methods mainly focus on the matching model architecture for single turn conversation  and multi-turn conversation  ,  ,  . For the studies of generative methods, a huge amount of work aims to mitigate the “safe response" issue from different perspectives. Most of work build models under a sequence to sequence framework  , and introduce other elements, such as latent variables  , topic information  , and dynamic vocabulary  to increase response diversity. Furthermore, the reranking technique  , reinforcement learning technique  , and adversarial learning technique  ,  have also been applied to response generation.

Our contributions are listed as follows: 1) this paper proposes a new paradigm, prototype-then-edit, for response generation; 2) we elaborate a simple but effective context-aware editing model for response generation; 3) we empirically verify the effectiveness of our method in terms of relevance, diversity, fluency and originality.

We hope that the decoder language model could remember the collocation of “desserts" and “bad for health", so as to replace “beneficial" with “bad" in the revised response. The new paradigm does not only inherits the fluency and informativeness advantages from retrieval results, but also enjoys the flexibility of generation results. Hence, our edit-based model is better than previous retrieval-based and generation-based models. The edit-based model can solve the “safe response" problem of generative models by leveraging existing responses, and is more flexible than retrieval-based models, because it does not highly depend on the index and is able to edit a response to fit current context. Prior work  has figured out how to edit prototype in an unconditional setting, but it cannot be applied to the response generation directly. In this paper, we propose a prototype editing method in a conditional setting. Our idea is that differences between responses strongly correlates with differences in their contexts (i.e. if a word in prototype context is changed, its related words in the response are probably modified in the editing.). We realize this idea by designing a context-aware editing model that is built upon a encoder-decoder model augmented with an editing vector.

Conclusion	We present a new paradigm, prototype-then-edit, for open domain response generation, that enables a generation-based chatbot to leverage retrieved results. We propose a simple but effective model to edit context-aware responses by taking context differences into consideration. Experiment results on a large-scale dataset show that our model outperforms traditional methods on some metrics. In the future, we will investigate how to jointly learn the prototype selector and neural editor.

Model Overview	Suppose that we have a data set INLINEFORM0 . INLINEFORM1 , INLINEFORM2 comprises a context INLINEFORM3 and its response INLINEFORM4 , where INLINEFORM5 is the INLINEFORM6 -th word of the context INLINEFORM7 and INLINEFORM8 is the INLINEFORM9 -th word of the response INLINEFORM10 . It should be noted that INLINEFORM11 can be either a single turn input or a multiple turn input. As the first step, we assume INLINEFORM12 is a single turn input in this work, and leave the verification of the same technology for multi-turn response generation to future work. Our full model is shown in Figure FIGREF3 , consisting of a prototype selector INLINEFORM13 and a context-aware neural editor INLINEFORM14 . Given a new conversational context INLINEFORM15 , we first use INLINEFORM16 to retrieve a context-response pair INLINEFORM17 . Then, the editor INLINEFORM18 calculates an edit vector INLINEFORM19 to encode the information about the differences between INLINEFORM20 and INLINEFORM21 . Finally, we generate a response according to the probability of INLINEFORM22 . In the following, we will elaborate how to design the selector INLINEFORM23 and the editor INLINEFORM24 .

Context-Aware Neural Editor	A context-aware neural editor aims to revise a prototype to adapt current context. Formally, given a quadruple INLINEFORM0 (we omit subscripts for simplification), a context-aware neural editor first forms an edit vector INLINEFORM1 using INLINEFORM2 and INLINEFORM3 , and then updates parameters of the generative model by maximizing the probability of INLINEFORM4 . For testing, we directly generate a response after getting the editor vector. In the following, we will introduce how to obtain the edit vector and learn the generative model in details. For an unconditional sentence editing setting  , an edit vector is randomly sampled from a distribution because how to edit the sentence is not constrained. In contrast, we should take both of INLINEFORM0 and INLINEFORM1 into consideration when we revise a prototype response INLINEFORM2 . Formally, INLINEFORM3 is firstly transformed to hidden vectors INLINEFORM4 through a biGRU parameterized as Equation ( EQREF10 ). DISPLAYFORM0  where INLINEFORM0 is the INLINEFORM1 -th word of INLINEFORM2 .

Experiment setting	In this paper, we only consider single turn response generation. We collected over 20 million human-human context-response pairs (context only contains 1 turn) from Douban Group . After removing duplicated pairs and utterance longer than 30 words, we split 19,623,374 pairs for training, 10,000 pairs for validation and 10,000 pairs for testing. The average length of contexts and responses are 11.64 and 12.33 respectively. The training data mentioned above is used by retrieval models and generative models. In terms of ensemble models and our editing model, the validation set and the test set are the same with datasets prepared for retrieval and generation models. Besides, for each context in the validation and test sets, we select its prototypes with the method described in Section “Prototype Selector". We follow Song et al. song2016two to construct a training data set for ensemble models, and construct a training data set with the method described in Section “Prototype Selector" for our editing models. We can obtain 42,690,275 INLINEFORM0 quadruples with the proposed data preparing method.

The edit vector is computed by the weighted average of insertion word embeddings and deletion word embeddings. Larger weights mean that the editing model should pay more attention on corresponding words in revision. For instance, in Table TABREF1 , we wish words like “dessert", “Tofu" and “vegetables" get larger weights than words like “and" and “ at". The encoder learns the prototype representation with a gated recurrent unit (GRU), and feeds the representation to a decoder together with the edit vector. The decoder is a GRU language model, that regards the concatenation of last step word embedding and the edit vector as inputs, and predicts the next word with an attention mechanism. Our experiments are conducted on a large scale Chinese conversation corpus comprised of 20 million context-response pairs. We compare our model with generative models and retrieval models in terms of fluency, relevance, diversity and originality. The experiments show that our method outperforms traditional generative models on relevance, diversity and originality. We further find that the revised response achieves better relevance compared to its prototype and other retrieval results, demonstrating that the editing process does not only promote response originality but also improve the relevance of retrieval results.

Equation EQREF18 and EQREF19 are the attention mechanism  , that mitigates the long-term dependency issue of the original Seq2Seq model. We append the edit vector to every input embedding of the decoder in Equation EQREF16 , so the edit information can be utilized in the entire generation process. We learn our response generation model by minimizing the negative log likelihood of INLINEFORM0 DISPLAYFORM0  We implement our model by PyTorch . We employ the Adam algorithm  to optimize the objective function with a batch size of 128. We set the initial learning rate as INLINEFORM0 and reduce it by half if perplexity on validation begins to increase. We will stop training if the perplexity on validation keeps increasing in two successive epochs. .

Prototype Selector	A good prototype selector INLINEFORM0 plays an important role in the prototype-then-edit paradigm. We use different strategies to select prototypes for training and testing. In testing, as we described above, we retrieve a context-response pair INLINEFORM1 from a pre-defined index for context INLINEFORM2 according to the similarity of INLINEFORM3 and INLINEFORM4 . Here, we employ Lucene to construct the index and use its inline algorithm to compute the context similarity. Now we turn to the training phase. INLINEFORM0 , INLINEFORM1 , our goal is to maximize the generative probability of INLINEFORM2 by selecting a prototype INLINEFORM3 . As we already know the ground-truth response INLINEFORM4 , we first retrieve thirty prototypes INLINEFORM5 based on the response similarity instead of context similarity, and then reserve prototypes whose Jaccard similarity to INLINEFORM6 are in the range of INLINEFORM7 . Here, we use Lucene to index all responses, and retrieve the top 20 similar responses along with their corresponding contexts for INLINEFORM8 .

The key factor for this phenomenon is that the neural editor will rewrite the prototype when it is hard to insert insertion words to the prototype. The ratio of “on the same topic" response given by Edit-N-rerank is larger than Edit-default, revealing that “on the same topic" responses might be more relevant from the view of a LSTM based reranker. We give three examples to show how our model works in Table TABREF30 . The first case illustrates the effect of word insertion. Our editing model enriches a short response by inserting words from context, that makes the conversation informative and coherent. The second case gives an example of word deletion, where a phrase “braised pork rice" is removed as it does not fit current context. Phrase “braised pork rice" only appears in the prototype context but not in current context, so it is in the deletion word set INLINEFORM0 , that makes the decoder not generate it. The third one is that our model forms a relevant query by deleting some words in the prototype while inserting other words to it. Current context is talking about “clean tatoo", but the prototype discusses “clean hair", leading to an irrelevant response.

Baseline	S2SA: We apply the Seq2Seq with attention  as a baseline model. We use a Pytorch implementation, OpenNMT  in the experiment. S2SA-MMI: We employed the bidirectional-MMI decoder as in  . The hyperparameter INLINEFORM0 is set as 0.5 according to the paper's suggestion. 200 candidates are sampled from beam search for reranking. CVAE: The conditional variational auto-encoder is a popular method of increasing the diversity of response generation  . We use the published code at https://github.com/snakeztc/NeuralDialog-CVAE, and conduct small adaptations for our single turn scenario. Retrieval: We compare our model with two retrieval-based methods to show the effect of editing. One is Retrieval-default that directly regards the top-1 result given by Lucene as the reply. The second one is Retrieval-Rerank, where we first retrieve 20 response candidates, and then employ a dual-LSTM model  to compute matching degree between current context and the candidates.

After the word substitution, the revised response becomes appropriated for current context. According to our observation, function words and nouns are more likely to be added/deleted. This is mainly because function words, such as pronoun, auxiliary, and interjection may be substituted in the paraphrasing. In addition, a large proportion of context differences is caused by nouns substitutions, thus we observe that nouns are added/deleted in the revision frequently.

Then we compute a context diff-vector INLINEFORM0 by an attention mechanism defined as follows DISPLAYFORM0  where INLINEFORM0 is a concatenation operation, INLINEFORM1 is a insertion word set, and INLINEFORM2 is a deletion word set. INLINEFORM3 explicitly encodes insertion words and deletion words from INLINEFORM4 to INLINEFORM5 . INLINEFORM6 is the weight of a insertion word INLINEFORM7 , that is computed by DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are parameters, and INLINEFORM2 is the last hidden state of the encoder. INLINEFORM3 is obtained with a similar process: DISPLAYFORM0  We assume that different words influence the editing process unequally, so we weighted average insertion words and deletion words to form an edit in Equation EQREF11 . Table TABREF1 explains our motivation as well, that is “desserts" is much more important than “the" in the editing process. Then we compute the edit vector INLINEFORM0 by following transformation DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are two parameters. Equation EQREF14 can be regarded as a mapping from context differences to response differences.

Acknowledgments	Yu is supported by AdeptMind Scholarship and Microsoft Scholarship. This work was supported in part by the Natural Science Foundation of China (Grand Nos. U1636211, 61672081, 61370126), Beijing Advanced Innovation Center for Imaging Technology (No.BAICIT-2016001) and National Key R&D Program of China (No.2016QY04W0802).

For a fair comparison, we randomly sample 19,623,374 instances for the training of our method and the ensemble method respectively. To facilitate further research, related resources of the paper can be found at https://github.com/MarkWuNLP/ResponseEdit.

The other approach follows a retrieval-then-edit paradigm, that uses a Seq2Seq model to edit a prototype retrieved from a corpus  ,  ,  . As far as we known, we are the first to leverage context lexical differences to edit prototypes.

Question: Which aspects of response generation do they evaluate on?
Answer: fluency, relevance, diversity , originality