Prototype-to-Style: Dialogue Generation with Style-Aware Editing on Retrieval Memory	The ability of a dialog system to express prespecified language style during conversations has a direct, positive impact on its usability and on user satisfaction. We introduce a new prototype-to-style (PS) framework to tackle the challenge of stylistic dialogue generation. The framework uses an Information Retrieval (IR) system and extracts a response prototype from the retrieved response. A stylistic response generator then takes the prototype and the desired language style as model input to obtain a high-quality and stylistic response. To effectively train the proposed model, we propose a new style-aware learning objective as well as a de-noising learning strategy. Results on three benchmark datasets from two languages demonstrate that the proposed approach significantly outperforms existing baselines in both in-domain and cross-domain evaluations

In summary, the contributions of this work are: (1) We propose a novel framework that tackles the challenge of stylistic dialogue generation by leveraging useful information contained in the retrieved responses; (2) We propose a new stylistic response generator by making proper adaptations to a large-scale pre-trained language model. We train our model with a new style-aware learning objective in a de-noising manner. Experiments show that the proposed model outperforms many strong baselines on three benchmark datasets on both in-domain and cross-domain evaluations.

Conclusion	In this work, we propose a novel PS framework to tackle the task of stylistic dialogue generation. Additionally, we propose a new stylistic response generator which works coherently with the proposed framework. We conduct extensive experiments on three benchmark datasets from two languages. Results of human and automatic evaluation show that the proposed approach outperforms many strong baselines by a substantial margin.

Furthermore, in order to control the style of the generated responses, we train the model with a novel style-aware maximum likelihood estimation (MLE) objective that encodes additional style knowledge into the model's parameters. Finally, to mitigate the possible effect that the retrieved response containing irrelevant and inappropriate information with respect to the input query, we adopt a de-noising learning strategy ,  to prevent the model from uncritically copying the prototype. To fully evaluate the proposed approach, we conduct extensive experiments on three benchmark datasets. Results of both human and automatic evaluation show that the proposed approach significantly outperforms several strong baselines. In addition, we also conduct an extensive cross-domain experiment to demonstrate that the proposed approach is more robust than such baselines. It should be noted that stylistic dialogue generation is different from the task of text style transfer. Text style transfer aims to rewrite the input sentences such that they possess certain language styles, while rigorously preserving their semantic meaning . On the other hand, stylistic dialogue generation does not aim at preserving the semantic meaning of the input sentences. Instead, it aims at generating sentences that are adequate and relevant responses to the input sentences, while expressing the prespecified language styles.

Experiments ::: Main Results	Both human and automatic evaluation results on the three benchmark datasets are shown in Table TABREF25, TABREF26 and TABREF27. For each dataset, we present results on individual styles as well as the overall results. We observe that the proposed model achieves the top performance results on most of the metrics. It generates responses with both intense style and high response quality. In addition, we also measure the diversity of the generated responses with two automatic metrics: Distinct-1 and Distinct-2 . The results show that the proposed model achieves the closest performance to that of the RRe approach whose responses are all written by human. On the ranking metric which jointly evaluates the content quality and the style expression, the proposed model outperforms other approaches by a substantial margin. From the results in Table TABREF26 and TABREF27, we can observe that ECM obtains the highest style expression scores on the emotion and sentiment dialogue datasets. This is because ECM directly incorporates the style information into its model architecture to force the generation of stylistic expressions. However, as shown in the quality scores, this behavior also undermines the quality of the generated responses.

Experiments ::: Further Analysis ::: Cross-Domain Evaluation	To evaluate the robustness of different approaches, we further analyze their performances when there is a notable difference between the data distribution of the training and testing set. Specifically, we use the models trained on gender-specific dataset to conduct inference on the test set of emotion-specific dataset and vise versa, which is regarded as domain variation. In Figure FIGREF50, we show the data distributions of these two datasets from which we can observe a notable distribution discrepancy. For evaluation, all results are evaluated with the same metrics as in the previous experiments. The averages response quality scores before and after domain variation are shown in Figure FIGREF55. For a direct comparison, the in-domain performance of each model can be found in Table TABREF25 and TABREF26. As shown in Figure FIGREF55, some of the strong baselines exhibit a drastic drop in response quality after domain variation such as GPT2-FT and PS w/o R. In contrast, the PS model successfully maintains high response quality in spite of domain variation. The model seems to benefit from leveraging retrieved results to bridge the gap between the two different domains.

Experiments ::: Model Comparison	We compare the proposed approach with several competitive baselines that can be categorized into two classes: generative approaches and retrieval-based approaches.

Datasets	We conduct extensive experiments on three dialogue datasets: gender-specific (Chinese) dataset, emotion-specific (Chinese) dataset, and sentiment-specific (English) dataset. For each dataset, we randomly select 200 instances as a held-out test set for evaluation.

Datasets ::: Sentiment-Specific Dialogue Dataset	To construct this dataset, we first build a classifier on the basis of BERT  and finetuned it on the the SemEval-2017 Subtask A dataset . This dataset consists of twitter instances with different sentiments including Positive, Negative and Neutral. The sentiment classifier attains 81.4% classification accuracy which is further used to annotate the OpenSubtitles dataset . The data statistic of the resulting sentiment-specific dialogue dataset is shown in Table TABREF21.

Datasets ::: Gender-Specific Dialogue Dataset	We use a publicly available gender-specific dialogue dataset . In this dataset, each response contains one specific gender preference including Female, Male and Neutral.

Datasets ::: Emotion-Specific Dialogue Dataset	We use a publicly available emotion-specific dataset  which contains responses with 6 different emotions including Like, Disgust, Happy, Anger, Sad and Other.

Therefore, the overall performance of ECM is not optimal as shown in the results of the ranking metric. From the experiment results, we observe that removing retrieved information (PS w/o R) from the proposed model causes a drastic drop on the quality score. This demonstrates that the retrieved information is indispensable for the model to generate a stylistic response and maintain a high response quality. In addition, comparing with GPT2-FT baseline, the ablated model (PS w/o R) shows similar content quality and much stronger stylistic expression, which is gained from the model architectural design and the new training strategy.

Experiments ::: Model Comparison ::: Ablation Study ::: PS:	The full model proposed in this work.

Experiments ::: Model Comparison ::: Ablation Study ::: PS w/o R:	In the ablated model, we examine how the retrieved prototype effects our model's performance. To this end, we remove the response prototype from the input representation.

Experiments ::: Model Comparison ::: Generative Approaches ::: Seq2seq:	Standard sequence-to-sequence model with attention mechanism , .

Experiments ::: Model Comparison ::: Generative Approaches ::: GPT2-FT:	To examine the effect of leveraging the pre-trained language model for the task of dialogue generation, we directly fine-tune the GPT-2 model on the dialogue data without any designed adaptations.

Experiments ::: Model Comparison ::: Generative Approaches ::: ECM:	Model proposed by  which uses memory modules to control the stylistic expressions in the generated responses.

Experiments ::: Model Comparison ::: Generative Approaches ::: Speaker:	Model proposed by  which incorporates distributed style embeddings into the structure of decoding cells to control the generation process.

Experiments ::: Model Comparison ::: Retrieval-Based Approaches ::: Skeleton-to-Response (SR):	Model proposed by  which modifies the retrieved response based on the lexical difference between the input and the retrieved query. This approach does not take the style aspect into consideration.

Experiments ::: Model Comparison ::: Retrieval-Based Approaches ::: Retrieval + Style Transfer (RST):	For this approach, we apply the state-of-the-art style transfer  model on the retrieved response. This approach does not consider the input query information during the transfer process.

In this work, we propose a novel prototype-to-style (PS) framework to tackle the challenge of stylistic dialogue generation. Our motivation is two-fold: (1) Human-written responses are informative and diverse, which could be leveraged as guidance for the generation model; (2) However, the retrieved response is not guaranteed to express the desired language style. Moreover, the quality of the retrieved response varies among different queries due to the instability of the IR system. Therefore, to transform the retrieved result into a relevant and stylistic response, an adequate editing process is necessary. An illustration of the proposed framework is shown in Figure FIGREF2, where a prototype is first extracted from the retrieved response. The stylistic response generator then takes the desired language style and the extracted prototype as additional input to obtain an adequate and stylistic response. The proposed stylistic response generator mainly inherits from the GPT-2 model  which is pre-trained with a large unlabeled text corpus. However, the GPT-2 model does not naturally fit the task of dialogue generation. To this end, we design various adaptations to the model architecture to extend the GPT-2 model to address the task of dialogue generation.

Experiments ::: Pretraining and Implementation Details	As there is no off-the-shelf pre-trained word-level language model in Chinese, we manually pre-trained one. The corpus collection and model pre-training details are presented in the supplementary material. For the English pre-trained language model, we use the PyTorch adaptation released by the HuggingFace team. To optimize the model, we use the Adam optimizer  with a batch size of 64 and learning rate of 2e-5. During inference, the retrieval system is built from the training corpus, and the retrieved responses are selected using the Jaccard similarity  between queries. During the inference stage, we retrieve the candidates from the training set. Specifically, we employ Jacquard Similarity to calculate the similarity between the input query q and queries in training set and find the most similar query q$^\prime $. Then we directly adopt the response of the retrieved query q$^\prime $ to construct the response prototype.

Methodology ::: Learning ::: Style-Aware Learning Objective	We propose to use a new style-aware learning objective to train the stylistic response generator. Consider a training instance consists of the input query ${\bf X} = (x_1, ..., x_N)$, the reference response ${\bf Y} = (y_1, ..., y_T)$, the reference language style $s$ and the response prototype ${\bf C} = (c_1, ..., c_T)$, the proposed objective is defined as where $\theta $ are the model parameters and $\mathcal {SV}$ is the stylistic vocabulary introduced in SV. By increasing $\alpha $, the proposed objective encodes more knowledge about stylistic expressions into the model parameters. We find that including the language model as an auxiliary objective in addition to the supervised style-aware learning objective helps to improve generalization as well as accelerate convergence. This observation is in line with , . In this work, the language model objective is defined as the reconstruction loss of the input query based on itself: The final learning objective is then defined as where $\beta $ regulates the importance of the auxiliary objective.

Methodology ::: Stylistic Response Generator	The proposed Stylistic Response Generator inherits from the GPT-2  model which consists of a 12-layer decoder-only Transformer . To make use of the GPT-2 model, the input tokens must be a consecutive natural sequence (e.g. sentence, document). Based on the input sequence, the input representation is constructed by adding up the token embeddings and the corresponding position embeddings. To achieve the goal of adapting the GPT-2 model under the proposed PS framework, we first make modifications to the form of the input sequence. As shown in Figure FIGREF6, we construct the input sequence as the concatenation of the input query, the response prototype and the reference response. Then we introduce a special token $[B]$ to indicate the boundary between these three parts. To further ensure the model can identify the different parts of the input sequence, we introduce a new segment level input which consists of three learnable segment embeddings $E_Q$, $E_P$ and $E_R$ to indicate the positions of the input query, the response prototype and the response history.

Experiments ::: Model Comparison ::: Retrieval-Based Approaches ::: Retrieval + Reranking (RRe):	Given the input query, a style classifier is used to rerank the top 10 retrieved responses. The response with the highest score on the desired style is selected.

Related Work ::: Stylistic Dialogue Generation:	Extensive research has tried to tackle the task of stylistic dialogue generation.  proposed to represent the user's personality with embeddings and incorporated them into the decoder structure to control the response generation process.  used reinforcement learning to train the generation model via the interaction with a pre-trained classifier to generate responses with specified attitude. , , ,  incorporated external knowledge into the model architecture either via embeddings or internal and external memory modules, such that during the generation process, emotion-based styles can be dynamically controlled.  proposed to use a shared latent space for stylistic dialogue generation.

To control the language style of the generated response, we propose to incorporate learnable style embeddings into the input representation. Specifically, we add the style embeddings to the entire part of the response history. This way, the model is constantly aware of the desired language style through the entire generation process.

Experiments ::: Case Study	We present several examples of generated responses by the proposed PS approach. Table TABREF51 shows responses with different gender and emotion styles, and Table TABREF52 shows responses with different sentiments. Examples in Table TABREF51 show that the proposed approach is able to extract informative details such as “have nightmares” and “higher salary” that are relevant to the queries from the retrieved responses. By taking the desired style as input, the proposed model generates adequate and stylistic responses while producing the informative details. Examples in Table TABREF52 also demonstrate that the proposed model is able to generate responses with desired sentiments based on the informative details (e.g. “_ want us to target _ ones _”, “_ can make _ decision.” and “_ sound _ to me _”) contained in the retrieved response.

Related Work ::: Text Style Transfer:	The task of text style transfer aims to transfer the style contained in a sentence while preserving its meaning.  proposed a DRG framework to tackle this task with the help of external knowledge. Recently, based on the pre-trained language model,  further improved the system performance under the same DRG framework.

Methodology ::: Learning ::: De-noising Training	We use a de-noising training strategy similar to DBLP:conf/nips/JainS08, DBLP:conf/cvpr/KrullBJ19 for training data construction, as shown in Figure FIGREF17. Specifically, during training, the response prototype is extracted from the reference response by the following steps. First, we mask all the stylistic words in the reference response. Second, we randomly select some words (40%) and replace it with a special token [MASK] or a random word drawn from the vocabulary. The second step is necessary otherwise the model will learn to generate a response by uncritically copying the response prototype, since the prototype after the first step is always an integral part of the golden response. This copy mechanism is undesirable since during testing the retrieved response is likely to contain information that is irrelevant to the input query. Thus, we deliberately train the response generator with noisy input to let the model learn to filter out the inappropriate information contained in the response prototype.

Methodology	The proposed framework leverages the results acquired from an IR system, A major challenge is that the retrieved response is not guaranteed to express the desired language style. At the first step, a neutral response prototype is extracted by masking all stylistic words contained in the retrieved response. A stylistic response generator then takes the desired language style and the extracted prototype as additional input to generate an adequate and stylistic response to the input query. To better emphasize the generation of stylistic expressions, we propose a style-aware learning objective. Finally, to prevent the model from learning to uncritically copy the prototype, we adopt a de-noising learning strategy ,  to train the generator.

Experiments ::: Further Analysis ::: Balance between Quality and Style	In practice, a satisfactory stylistic dialogue system should express the desired style on the premise of the response quality. Based on the criterion of human evaluation metric, 3 is the marginal score of acceptance. So we deem a response as marginally acceptable by actual users when both quality and style expression scores are greater or equal to 3. On the other hand, 4 is the score that well satisfies the users, so responses with both scores greater or equal to 4 are deemed as satisfying to actual users. The ratios of both scores $\ge 3$ and $\ge 4$ are shown in Figure FIGREF47, from which we can see that the proposed approach outperforms all other approaches on $\ge 3$-ratio and $\ge 4$-ratio. The proposed model best balances the trade-off between the response quality and style expression and therefore generating most acceptable and satisfying responses.

Introduction	Most early research on dialogue response generation focused on generating grammatical and contextually relevant responses , , . While promising results have been demonstrated , , syntactically coherent responses alone do not guarantee an engaging and attractive dialogue system. Expressing a unique and consistent speaking style has been shown to be crucial for increasing the user's engagement with dialogue systems . There are various definitions of language style , , , , . In this work, from a purely computational standpoint, we refer to language style as any characteristic style of expression. Hence, our work is in line with previous work on dialogue generation with emotion , , , ; response attitude , and speaker personality . The aforementioned approaches explicitly incorporate the language style information into the model configuration either via embeddings or memory modules to control the process of response generation. In our replication experiments, we found that these approaches tend to overemphasise the importance of the language style. As a result, the generated responses tend to be generic and non-informative , but they do express a distinct style; e.g., they generate a generic response: “I am happy to hear that." that conveys a `happy' emotion to different queries.

Experiments ::: Evaluation Metrics ::: Style Expression:	This metric measures how well the generated responses express the desired style. The annotators give a score ranging from 1 to 5 to this metric, where 5 means very strong style, 3 means no obvious style and 1 means very conflicted style. The style conflict means the generated style is conflicted to the desired one (e.g. female to male, positive to negative emotion).

An example is provided in Figure FIGREF2 where the prototype: “That's _ . I will go with my _ together !” is extracted from the retrieved response by masking the stylistic words great, bro and buddies.

Methodology ::: Prototype Extraction	The response prototype is constructed from the retrieved response by masking the stylistic words. To determine whether a word is stylistic, we use the pointwise mutual information (PMI)  metric. The relevance between the word $x$ and the style $s$ is measured as where $p(x, s)$ is the frequency that the word $x$ appears in a response with style $s$ in the training corpus. And a word $x$ is stylistic given the style $s$ if $\textup {PMI}(x,s)\ge t_s$. In our experiments, we empirically set $t_s$ as $t_s = \frac{3}{4}\times \max _{v\in \mathcal {V}}\textup {PMI}(v; s)$, where $\mathcal {V}$ is the vocabulary set of the training corpus. Given the set of all possible language styles $\mathcal {S}$, the stylistic vocabulary $\mathcal {SV}$ is defined as all words that express any style $s\in \mathcal {S}$.

Experiments ::: Evaluation Metrics ::: Ranking:	The annotators are further asked to jointly evaluate the content quality and the style expression of the generated responses from different approaches. Then the annotators give a ranking to each result where top 1 means the best.

Related Work ::: Retrieval Guided Dialogue Generation:	Many prior works , , ,  proposed to leverage information from the retrieved responses to improve the system performance on non-task oriented dialogue generation. It should be noted that all these approaches aim to improve the content quality of the generated responses but do not take the style aspect into consideration.

Experiments ::: Evaluation Metrics ::: Quality:	This metric evaluates the content quality of the generated responses. The annotators are asked to give a score within 5-point scale where 5 means perfectly human-like response (relevant, fluent and informative), 3 means marginally acceptable and 1 means unreadable and impossible to understand.

Experiments ::: Further Analysis	We present further discussions and empirical analysis of the proposed approach.

This can also be observed in the results of RST and RRe which also use the retrieved results and get a even higher performance when facing domain variation.

Experiments ::: Evaluation Metrics	The quality of dialogue responses is known to be difficult to measure automatically ; we therefore rely on human evaluation. To evaluate the responses, we hire five annotators from a commercial annotation company. To prevent introducing potential bias to the annotators, all results are randomly shuffled before being evaluated. All results are evaluated by the annotators following the metrics below.

Related Work	We summarize three categories of relevant work in the following.

Question: What are existing baseline models on these benchmark datasets?
Answer: Seq2seq, GPT2-FT, Speaker, ECM, Skeleton-to-Response (SR), Retrieval + Style Transfer (RST), Retrieval + Reranking (RRe)