Stylized Dialogue Generation with Multi-Pass Dual Learning

1 Introduction . In recent years , dialogue systems have attracted a surge of research interest and achieved great success [ 1 , 2 , 3 , 4 ] . The dialogue generation task can be formulated as a sequence-to-sequence learning problem [ 5 , 6 ] , where the source sequence is the given context , and the target sequence is the response . Stylized dialogue generation , which automatically generates response following the desired style , obtains significant progress in open-domain dialogue systems . The styles could be formal/informal , modern/Shakespearean , serious/humor , etc . A core challenge of this task is the lack of parallel training data between the contexts and responses of desired styles ( denoted as style S1 ) , which limits the performance of models . A natural solution is to establish the pseudo pairs between contexts and responses of style S1 using unsupervised or semi-supervised learning . The parallel data between contexts and responses of style S0 ( denoted as style S0 ) is leveraged . Note the conversations corpus ( context , style S0 ) are relatively easy to obtain . After that , the stylized dialogue model can be trained on the synthetic data . Specifically , Su et al . [ 7 ] propose a diversifying dialogue generation model based on iterative back translation [ 8 ] . Zheng et al . [ 9 ] propose a style routing approach with a joint training process , where an inverse model with style embeddings is used to generate pseudo pairs and train the stylized dialogue generation model . ∗corresponding authors : Rui Yan ( ruiyan @ ruc.edu.cn ) and Dongyan Zhao ( zhaody @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although the above methods achieved success in the stylized dialogue generation , there still exists lots of room to improve . First , the interaction between context , style S0 and style S1 is not fully exploited . An example is that a pseudo context can be explicitly obtained by a two-hop process , S1 → S0 →pseudo context instead of using style embedding only . Second , in the aforementioned methods , the quality of the pseudo contexts generated through back translation is not properly modeled . By manually checking the generated data , we find that there are quite a few pseudo contexts that contain irrelevant contents of the response . Several common patterns in the pseudo contexts and responses include “ I don ’ t know ” , “ I ’ m so happy ” , etc . In previous methods , each pseudo data is treated equally regardless of their quality , which introduces bias to the training set and hurts the performances of model . In this paper , we propose a new framework , multi-pass dual learning ( MPDL ) , for stylized dialogue generation . Inspired by dual learning [ 10 ] that can automatically extract information through two dual tasks in an unsupervised manner , we extend it to a multi-pass version . In MPDL , there are three dual tasks in total , including context↔style S1 , context↔style S0 and style S0 ↔style S1 . The unlabeled texts in style S1 and contexts are leveraged . We also introduce two discriminators to evaluate the quality of the generated data . The discriminators are used to measure the similarity of generated data to the corresponding domains . The similarity is used as the weight of the synthetic data for training , by which we can adaptively use the unlabeled data . We conduct experiments with benchmarks of stylized response generation in formal English and Shakespearean English . We achieved state-of-the-art results in terms of both automatic evaluation and manual judgment . Our contributions are summarized as follows : ( 1 ) We propose multi-pass dual learning ( MPDL ) framework for stylized dialogue response generation , that can effectively leverage the unlabeled data ; ( 2 ) Compared with standard dual learning , we introduce two discriminators to evaluate the quality of the pseudo parallel data . This is a new attempt for the general dual learning framework ; ( 3 ) We provide a new dataset for this task and set several benchmarks using our method ; ( 4 ) We empirically verify the effectiveness of MPDL on two datasets with formal and Shakespearean response generation . 2 Related work . Text style transfer aims to re-write a sentence with given stylistic properties . Some early works try supervised approaches , where a work is the transfer between modern English and Shakespearean style sentences [ 11 , 12 ] . Nevertheless , more and more recent research begins to pay attention to the unsupervised approach due to the lack of parallel corpora [ 13 , 14 ] . In general , there are three typical approaches . The first is to learn disentangled representations of content and style . Shen et al . [ 15 ] learned a shared representation of context across different styles and then leveraged the refined alignment of latent representations for style transfer . Fu et al . [ 16 ] utilize a multi-decoder with a style embedding model to learn separate content representations , where style representations are obtained via adversarial networks . Another is to generate stylized texts with back translation [ 17 , 18 ] , which produces pseudo parallel data to jointly train the forward and backward model . The third is based on the template , which tries to replace original stylistic words with the target stylistic words [ 19 , 14 ] . Stylized dialogue generation refers to generate a response in the target style . Zhou et al . [ 20 ] propose an Emotional Chatting Machine ( ECM ) model for the emotional response . An intuitive solution generates a response with a pre-trained style language model [ 21 ] , which builds a response generation model STYLEDGPT on top of a pre-trained language model DialoGPT [ 22 ] and devise both a word-level loss and a sentence-level loss to fine-tune the DialoGPT towards the target style . Niu and Bansal [ 23 ] propose three weakly supervised models that can generate diverse , polite ( or rude ) dialogue responses without parallel data . Some work bridge conversation modeling and non-parallel style transfer by sharing a structured latent space [ 24 ] . Recently , some work iteratively tuned to find the optimal mapping relation between conversational context and non-conversational utterances via iterative back translation [ 7 , 9 ] . These works have achieved good performance , but we have achieved better diversity and efficiency through an auxiliary domain . Back translation is to train a target-to-source seq2seq model for producing source sentences , and then construct pseudo parallel datasets . This approach is widely used in machine translation [ 8 , 25 ] , text style transfer [ 26 , 27 ] and dialogue generation [ 7 , 9 ] . Similar concurrent work is Dual learning [ 10 ] , which is first proposed for neural machine translation , involves two tasks with an immediate reward . Dual learning has proved to be very effective in neural machine translation and image translation [ 28 ] . Our work is also based on the idea of them . However , most of the recent works have focused on two domain mapping . We employ dual learning to work on a three-domain text related problem , then the contents of non-conversational text can be effectively utilized to enrich the dialogue generation . 3 Method . Let C , S0 , S1 denote the domains of context , response in style S0 and response in style S1 respectively . Our task is to learn a mapping , that can map a sequence from context domain C to a stylized response domain S1 . A core challenge is that there is no parallel data between C and S1 , but there are some related parallel corpora : ( i ) A dialogue dataset Ddia = { ( xi , yi ) } ni=1 where xi ∈ C and yi ∈ S0 ; ( ii ) A style transfer dataset Dtra = { ( yi , ỹi ) } mi=1 , where yi ∈ S0 and ỹi ∈ S1 . The corpus sizes of the above two datasets are n and m. The sequences in style S0 are not the same between Ddia and Dtra , i.e. , { y| ( x , y ) ∈ Ddia } ∩ { y| ( y , ỹ ) ∈ Dtra } = ∅ . In addition to parallel data , there is another unlabeled stylized corpus Dsty = { ỹi } li=1 with size l , where ỹi ∈ S1 . 3.1 Multi-pass dual learning framework . To fully utilize the parallel and unlabeled data , we propose multi-pass dual learning ( briefly , MPDL ) , which is illustrated in Figure 1 . There are three pairs of dual tasks involved in our framework : ( i ) Dialogue generation and the inversed task , fxy : C 7→ S0 , fyx : S0 7→ C ; ( ii ) Style transfer between S0 and S1 , fyỹ : S0 7→ S1 , fỹy : S1 7→ S0 ; ( iii ) Stylized dialogue response generation and the inversed task , fxỹ : C 7→ S1 , fỹx : S1 7→ C. Our ultimate goal is to obtain the stylized dialogue response generation model fxỹ , while the remaining tasks are all auxiliary tasks . For multi-pass dual learning , given any ỹ ∈ Dsty , we could obtain reconstruction ỹ′ = fxỹ ( fyx ( fỹy ( ỹ ) ) ) by sequentially applying fỹy , fyx and fxỹ . The reconstruction error between ỹ′ and ỹ determines the ability of model reconstruction . That is , we want to maximize the logarithmic probability : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) . ( 1 ) Directly optimization the above probability is difficult since there are three models to be updated . Consequently , we derive a relaxed version of the above probability . We have that : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) = log ∑ x∈C ∑ y∈S0 P ( ỹ′ , y , x|ỹ ; fỹy , fyx , fxỹ ) = log ∑ x∈C ∑ y∈S0 P ( ỹ′|y , x , ỹ ; fỹy , fyx , fxỹ ) P ( y , x|ỹ ; fỹy , fyx , fxỹ ) . ( 2 ) Considering that ỹ′ is only related to x and fxỹ , we have that : P ( ỹ′|y , x , ỹ ; fỹy , fyx , fxỹ ) = P ( ỹ′|x , fxỹ ) . Due to the concavity of log ( . . . ) , we can convert the Eqn . ( 2 ) to : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) ≥ ∑ x∈C ∑ y∈S0 P ( y , x|ỹ ; fỹy , fyx , fxỹ ) logP ( ỹ′|x , fxỹ ) = ∑ x∈C ∑ y∈S0 P ( x|y ; fyx ) P ( y|ỹ ; fỹy ) logP ( ỹ′|x , fxỹ ) , ( 3 ) where the last equation is obtained by the fact that x is only related to y and fyx , and y is only related to ỹ and fỹy . Instead of maximizing logP ( ỹ′ = ỹ| · · · ) directly , we maximize a lower bound as Eqn . ( 3 ) suggests . We use a sampling-based method to solve this problem : ( i ) Given a ỹ , we sample a y according to P ( ·|ỹ ; fỹy ) ; ( ii ) Based on y , we sample a x′ according to P ( ·|y ; fyx ) ; ( iii ) We maximize logP ( ỹ|x′ ; fxỹ ) . We can use either beam search or random sampling to get the intermediate sequences . On the other hand , we can pick an x from the set { x|x ∈ Ddia } , sequentially use two sampling steps y ∼ P ( ·|x ; fxy ) and ỹ′ ∼ P ( ·|y ; fyỹ ) and then maximize logP ( x|ỹ′ ; fỹx ) . In addition to the loss for unlabeled data , we propose loss functions on the parallel data . Therefore , there are four loss functions in total : Ldia = E ( x , y ) ∼Ddia − ( logP ( y|x ; fxy ) + logP ( x|y ; fyx ) ) , Ltra = E ( y , ỹ ) ∼Dtra − ( logP ( ỹ|y ; fyỹ ) + logP ( y|ỹ ; fỹy ) ) , Ldual,1 ( ỹ ) = − logP ( ỹ|x′ ; fxỹ ) , Ldual,2 ( x ) = − logP ( x|ỹ′ ; fỹx ) , ( 4 ) where x′ and ỹ′ are generated by the MPDL model as the above steps .

Summary: The authors propose a method for learning stylized dialogue response generation: given a context, generate a response conditioned on a particular style domain (e.g. modern vs. shakespearean english). The method, Multi-Pass Dual Learning (MPDL), leverages large corpora of non-parallel data in tandem. Specifically, the framework involves several forward and backward models between the context and two styles (one which has labeled data from the context to the style, and one that does not); the models produce intermediate data that is used to train the desired style generation model. To make sure that the intermediate data is being used appropriately, the authors introduce two discriminators that evaluate the quality of the data and determine whether it falls in the respective style/domain. The scores of these discriminators weight the importance of the loss computed on that data to ensure that higher-quality pseudo tuples are more impactful in the learning process. The authors evaluate the framework on two style-transfer datasets, and show that the method outperforms 5 baselines, achieving state-of-the-art results in automated metrics and outperforming nearly all models in human evaluations. A case study and ablations are presented as well that highlight and support the authors’ design decisions.