Stylized Dialogue Generation with Multi-Pass Dual Learning

1 Introduction . In recent years , dialogue systems have attracted a surge of research interest and achieved great success [ 1 , 2 , 3 , 4 ] . The dialogue generation task can be formulated as a sequence-to-sequence learning problem [ 5 , 6 ] , where the source sequence is the given context , and the target sequence is the response . Stylized dialogue generation , which automatically generates response following the desired style , obtains significant progress in open-domain dialogue systems . The styles could be formal/informal , modern/Shakespearean , serious/humor , etc . A core challenge of this task is the lack of parallel training data between the contexts and responses of desired styles ( denoted as style S1 ) , which limits the performance of models . A natural solution is to establish the pseudo pairs between contexts and responses of style S1 using unsupervised or semi-supervised learning . The parallel data between contexts and responses of style S0 ( denoted as style S0 ) is leveraged . Note the conversations corpus ( context , style S0 ) are relatively easy to obtain . After that , the stylized dialogue model can be trained on the synthetic data . Specifically , Su et al . [ 7 ] propose a diversifying dialogue generation model based on iterative back translation [ 8 ] . Zheng et al . [ 9 ] propose a style routing approach with a joint training process , where an inverse model with style embeddings is used to generate pseudo pairs and train the stylized dialogue generation model . ∗corresponding authors : Rui Yan ( ruiyan @ ruc.edu.cn ) and Dongyan Zhao ( zhaody @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although the above methods achieved success in the stylized dialogue generation , there still exists lots of room to improve . First , the interaction between context , style S0 and style S1 is not fully exploited . An example is that a pseudo context can be explicitly obtained by a two-hop process , S1 → S0 →pseudo context instead of using style embedding only . Second , in the aforementioned methods , the quality of the pseudo contexts generated through back translation is not properly modeled . By manually checking the generated data , we find that there are quite a few pseudo contexts that contain irrelevant contents of the response . Several common patterns in the pseudo contexts and responses include “ I don ’ t know ” , “ I ’ m so happy ” , etc . In previous methods , each pseudo data is treated equally regardless of their quality , which introduces bias to the training set and hurts the performances of model . In this paper , we propose a new framework , multi-pass dual learning ( MPDL ) , for stylized dialogue generation . Inspired by dual learning [ 10 ] that can automatically extract information through two dual tasks in an unsupervised manner , we extend it to a multi-pass version . In MPDL , there are three dual tasks in total , including context↔style S1 , context↔style S0 and style S0 ↔style S1 . The unlabeled texts in style S1 and contexts are leveraged . We also introduce two discriminators to evaluate the quality of the generated data . The discriminators are used to measure the similarity of generated data to the corresponding domains . The similarity is used as the weight of the synthetic data for training , by which we can adaptively use the unlabeled data . We conduct experiments with benchmarks of stylized response generation in formal English and Shakespearean English . We achieved state-of-the-art results in terms of both automatic evaluation and manual judgment . Our contributions are summarized as follows : ( 1 ) We propose multi-pass dual learning ( MPDL ) framework for stylized dialogue response generation , that can effectively leverage the unlabeled data ; ( 2 ) Compared with standard dual learning , we introduce two discriminators to evaluate the quality of the pseudo parallel data . This is a new attempt for the general dual learning framework ; ( 3 ) We provide a new dataset for this task and set several benchmarks using our method ; ( 4 ) We empirically verify the effectiveness of MPDL on two datasets with formal and Shakespearean response generation . 2 Related work . Text style transfer aims to re-write a sentence with given stylistic properties . Some early works try supervised approaches , where a work is the transfer between modern English and Shakespearean style sentences [ 11 , 12 ] . Nevertheless , more and more recent research begins to pay attention to the unsupervised approach due to the lack of parallel corpora [ 13 , 14 ] . In general , there are three typical approaches . The first is to learn disentangled representations of content and style . Shen et al . [ 15 ] learned a shared representation of context across different styles and then leveraged the refined alignment of latent representations for style transfer . Fu et al . [ 16 ] utilize a multi-decoder with a style embedding model to learn separate content representations , where style representations are obtained via adversarial networks . Another is to generate stylized texts with back translation [ 17 , 18 ] , which produces pseudo parallel data to jointly train the forward and backward model . The third is based on the template , which tries to replace original stylistic words with the target stylistic words [ 19 , 14 ] . Stylized dialogue generation refers to generate a response in the target style . Zhou et al . [ 20 ] propose an Emotional Chatting Machine ( ECM ) model for the emotional response . An intuitive solution generates a response with a pre-trained style language model [ 21 ] , which builds a response generation model STYLEDGPT on top of a pre-trained language model DialoGPT [ 22 ] and devise both a word-level loss and a sentence-level loss to fine-tune the DialoGPT towards the target style . Niu and Bansal [ 23 ] propose three weakly supervised models that can generate diverse , polite ( or rude ) dialogue responses without parallel data . Some work bridge conversation modeling and non-parallel style transfer by sharing a structured latent space [ 24 ] . Recently , some work iteratively tuned to find the optimal mapping relation between conversational context and non-conversational utterances via iterative back translation [ 7 , 9 ] . These works have achieved good performance , but we have achieved better diversity and efficiency through an auxiliary domain . Back translation is to train a target-to-source seq2seq model for producing source sentences , and then construct pseudo parallel datasets . This approach is widely used in machine translation [ 8 , 25 ] , text style transfer [ 26 , 27 ] and dialogue generation [ 7 , 9 ] . Similar concurrent work is Dual learning [ 10 ] , which is first proposed for neural machine translation , involves two tasks with an immediate reward . Dual learning has proved to be very effective in neural machine translation and image translation [ 28 ] . Our work is also based on the idea of them . However , most of the recent works have focused on two domain mapping . We employ dual learning to work on a three-domain text related problem , then the contents of non-conversational text can be effectively utilized to enrich the dialogue generation . 3 Method . Let C , S0 , S1 denote the domains of context , response in style S0 and response in style S1 respectively . Our task is to learn a mapping , that can map a sequence from context domain C to a stylized response domain S1 . A core challenge is that there is no parallel data between C and S1 , but there are some related parallel corpora : ( i ) A dialogue dataset Ddia = { ( xi , yi ) } ni=1 where xi ∈ C and yi ∈ S0 ; ( ii ) A style transfer dataset Dtra = { ( yi , ỹi ) } mi=1 , where yi ∈ S0 and ỹi ∈ S1 . The corpus sizes of the above two datasets are n and m. The sequences in style S0 are not the same between Ddia and Dtra , i.e. , { y| ( x , y ) ∈ Ddia } ∩ { y| ( y , ỹ ) ∈ Dtra } = ∅ . In addition to parallel data , there is another unlabeled stylized corpus Dsty = { ỹi } li=1 with size l , where ỹi ∈ S1 . 3.1 Multi-pass dual learning framework . To fully utilize the parallel and unlabeled data , we propose multi-pass dual learning ( briefly , MPDL ) , which is illustrated in Figure 1 . There are three pairs of dual tasks involved in our framework : ( i ) Dialogue generation and the inversed task , fxy : C 7→ S0 , fyx : S0 7→ C ; ( ii ) Style transfer between S0 and S1 , fyỹ : S0 7→ S1 , fỹy : S1 7→ S0 ; ( iii ) Stylized dialogue response generation and the inversed task , fxỹ : C 7→ S1 , fỹx : S1 7→ C. Our ultimate goal is to obtain the stylized dialogue response generation model fxỹ , while the remaining tasks are all auxiliary tasks . For multi-pass dual learning , given any ỹ ∈ Dsty , we could obtain reconstruction ỹ′ = fxỹ ( fyx ( fỹy ( ỹ ) ) ) by sequentially applying fỹy , fyx and fxỹ . The reconstruction error between ỹ′ and ỹ determines the ability of model reconstruction . That is , we want to maximize the logarithmic probability : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) . ( 1 ) Directly optimization the above probability is difficult since there are three models to be updated . Consequently , we derive a relaxed version of the above probability . We have that : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) = log ∑ x∈C ∑ y∈S0 P ( ỹ′ , y , x|ỹ ; fỹy , fyx , fxỹ ) = log ∑ x∈C ∑ y∈S0 P ( ỹ′|y , x , ỹ ; fỹy , fyx , fxỹ ) P ( y , x|ỹ ; fỹy , fyx , fxỹ ) . ( 2 ) Considering that ỹ′ is only related to x and fxỹ , we have that : P ( ỹ′|y , x , ỹ ; fỹy , fyx , fxỹ ) = P ( ỹ′|x , fxỹ ) . Due to the concavity of log ( . . . ) , we can convert the Eqn . ( 2 ) to : logP ( ỹ′ = ỹ|ỹ ; fỹy , fyx , fxỹ ) ≥ ∑ x∈C ∑ y∈S0 P ( y , x|ỹ ; fỹy , fyx , fxỹ ) logP ( ỹ′|x , fxỹ ) = ∑ x∈C ∑ y∈S0 P ( x|y ; fyx ) P ( y|ỹ ; fỹy ) logP ( ỹ′|x , fxỹ ) , ( 3 ) where the last equation is obtained by the fact that x is only related to y and fyx , and y is only related to ỹ and fỹy . Instead of maximizing logP ( ỹ′ = ỹ| · · · ) directly , we maximize a lower bound as Eqn . ( 3 ) suggests . We use a sampling-based method to solve this problem : ( i ) Given a ỹ , we sample a y according to P ( ·|ỹ ; fỹy ) ; ( ii ) Based on y , we sample a x′ according to P ( ·|y ; fyx ) ; ( iii ) We maximize logP ( ỹ|x′ ; fxỹ ) . We can use either beam search or random sampling to get the intermediate sequences . On the other hand , we can pick an x from the set { x|x ∈ Ddia } , sequentially use two sampling steps y ∼ P ( ·|x ; fxy ) and ỹ′ ∼ P ( ·|y ; fyỹ ) and then maximize logP ( x|ỹ′ ; fỹx ) . In addition to the loss for unlabeled data , we propose loss functions on the parallel data . Therefore , there are four loss functions in total : Ldia = E ( x , y ) ∼Ddia − ( logP ( y|x ; fxy ) + logP ( x|y ; fyx ) ) , Ltra = E ( y , ỹ ) ∼Dtra − ( logP ( ỹ|y ; fyỹ ) + logP ( y|ỹ ; fỹy ) ) , Ldual,1 ( ỹ ) = − logP ( ỹ|x′ ; fxỹ ) , Ldual,2 ( x ) = − logP ( x|ỹ′ ; fỹx ) , ( 4 ) where x′ and ỹ′ are generated by the MPDL model as the above steps .

Summary: This paper studies the task of stylized dialogue generation, where the goal is to generate dialogue responses in a particular target style (formal / informal / Shakespeare-like). The authors operate in a setting where they do not have access to any dialogue training data in the target style, but instead have access to dialogue data in a complementary style, along with parallel data to transfer between the two styles. For instance, the authors study dialogue generation in formal style, and assume access to dialogue data in informal style, parallel data between formal / informal sentences, and a corpus of unpaired formal sentences.  The authors model this task as a three-way dual learning problem, a technique which has proven useful for machine translation [1]. They jointly training their model to perform 1) style transfer in both directions; 2) dialogue / inverse-dialogue generation to the complementary style; 3) dialogue / inverse-dialogue generation to the target style. 1 / 2 act as auxiliary tasks to aid (3) via shared models. Since no parallel data is available for (3), the authors perform sampling (using current version of model for 1 / 2) to create pseudo parallel data for (3). Since sampling can be noisy, the authors additionally use discriminators to weight samples, which judge whether the samples are real/fake (akin to adversarial training).  The authors evaluate their approach on formal dialogue generation and Shakespearan dialogue generation. They compare their approach to 5-6 baselines, along with human upperbounds. Both automatic and human evaluation is conducted. The authors additionally conduct ablation studies, analyze their discriminator, and show the effect of the size of parallel data.  [1] - https://arxiv.org/abs/1611.00179