Multi-turn Dialogue Response Generation in an Adversarial Learning Framework

1 INTRODUCTION . Recent advances in deep neural network architectures have enabled tremendous success on a number of difficult machine learning problems . While these results are impressive , producing a deployable neural network–based conversation model that can engage in open domain discussion still remains elusive . A dialogue system needs to be able to generate meaningful and diverse responses that are simultaneously coherent with the input utterance and the overall dialogue topic . Unfortunately , earlier conversation models trained with naturalistic dialogue data suffered greatly from limited contextual information ( Sutskever et al. , 2014 ; Vinyals & Le , 2015 ) , and lack diversity ( Li et al. , 2016a ) . These problems often leads to generic and safe utterance in response to varieties of input utterance . Serban et al . ( 2016 ) and Xing et al . ( 2017 ) proposed the Hierarchical Recurrent Encoder-Decoder ( HRED ) network to capture long temporal dependencies in multi-turn conversations to address the limited contextual information but the diversity problem remained . On the other hand , some HRED variants such as variational ( Serban et al. , 2017b ) and multi-resolution ( Serban et al. , 2017a ) HREDs attempt to alleviate the diversity problem by injecting noise at the utterance level and by extracting additional context to condition the generator on . While these approaches achieve certain measures of success over the basic HRED , generated responses are still mostly generic since they do not control the generator ’ s output as the output conditional distribution is not calibrated . Li et al . ( 2016a ) , on the other hand , consider diversity promoting training objective but their model is for single turn conversations , can not not be trained end-to-end and therefore achieves little . The generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) seems to be an appropriate solution to the diversity problem . GAN matches data from two different distributions by introducing an adversarial game between a generator and a discriminator . We explore hredGAN : conditional GANs for multi-turn dialogue models with HRED generator and discriminator . hredGAN combines both generative and retrieval-based multi-turn dialogue systems to improve their individual performances . This is achieved by sharing the context and word embedding between the generator and the discriminator allowing for joint end-to-end training using back-propagation . To the best of our knowledge , no existing work has applied conditional GANs to multi-turn dialogue models and especially with HRED generators and discriminators . We demonstrate the effectiveness of hredGAN over the VHRED for dialogue modeling with evaluations on the Movie triples and Ubuntu technical support datasets . 2 ADVERSARIAL FRAMEWORK FOR MULTI-TURN DIALOGUE . Consider a dialogue consisting of a sequence of N utterances , X = ( X1 , X2 , · · · , XN ) , where each utterance Xi = ( X1i , X 2 i , · · · , X Mi i ) contains a variable-length sequence of Mi word tokens such that Xij ∈ V for vocabulary V . At any time step i , the dialogue history is given by Xi = ( X1 , X2 , · · · , Xi ) . The dialogue response generation task can be defined as follows : Given a dialogue history Xi , generate a response Yi = ( Y 1i , Y 2 i , · · · , Y Ti i ) , where Ti is the number of generated tokens . We also want the distribution of the generated response P ( Yi ) to be indistinguishable from that of the ground truth P ( Xi+1 ) and Ti =Mi+1 . Conditional GAN learns a mapping from an observed dialogue history , Xi , and a sequence of random noise vectors , Zi to a sequence of output tokens , Yi , G : { Xi , Zi } → Yi . The generator G is trained to produce output sequences that can not be distinguished from the ground truth sequence by an adversarially trained discriminator D that is trained to do well at detecting generator ’ s fakes . The distribution of the generator output sequence can be factored by the product rule : P ( Yi|Xi ) = P ( Y 1i ) Ti∏ j=2 P ( Y ji |Y 1 i , · · · , Y j−1 i , Xi ) ( 1 ) P ( Y ji |Y 1 i , · · · , Y j−1 i , Xi ) = PθG ( Y 1 : j−1i , Xi ) ( 2 ) where Y i : j−1i = ( Y 1 i , · · · , Y j−1 i ) and θG are the parameters of the generator model . PθG ( Y i : j−1i , Xi ) is an autoregressive generative model where the probability of the current token depends on the past generated sequence . Training the generator G with the log-likelihood criterion is unstable in practice , and therefore the past generated sequence is substituted with the ground truth , a method known as teacher forcing ( Williams & Zipser , 1989 ) , i.e. , P ( Y ji |Y 1 i , · · · , Y j−1 i , Xi ) ≈ PθG ( X1 : j−1i+1 , Xi ) ( 3 ) Using equation 3 in relation to GAN , we define our fake sample as the teacher forcing output with some input noise Zi Y ji ∼ PθG ( X1 : j−1i+1 , Xi , Zi ) ( 4 ) and the corresponding real sample as ground truth Xji+1 . With the GAN objective , we can match the noise distribution , P ( Zi ) to the distribution of the ground truth response , P ( Xi+1|Xi ) . Varying the noise input then allows us to generate diverse responses to the same dialogue history . Furthermore , the discriminator , since it is calibrated , is used during inference to rank the generated responses , providing a means of controlling the generator output . 2.1 OBJECTIVES . The objective of a conditional GAN can be expressed as LcGAN ( G , D ) = EXi , Xi+1 [ log D ( Xi+1 , Xi ) ] + EXi , Zi [ 1 − logD ( G ( Xi , Zi ) , Xi ) ] ( 5 ) where G tries to minimize this objective against an adversarial D that tries to maximize it : G∗ , D∗ = argmin G max D LcGAN ( G , D ) . ( 6 ) Previous approaches have shown that it is beneficial to mix the GAN objective with a more traditional loss such as cross-entropy loss ( Lamb et al. , 2016 ; Li et al. , 2017 ) . The discriminator ’ s job remains unchanged , but the generator is tasked not only to fool the discriminator but also to be near the ground truth Xi+1 in the cross-entropy sense : LMLE ( G ) = EXi , Xi+1 , Zi [ −log PθG ( Xi+1 , Xi , Zi ) ] . ( 7 ) Our final objective is , G∗ , D∗ = argmin G max D ( λGLcGAN ( G , D ) + λMLMLE ( G ) ) . ( 8 ) i+1 . Right : RNN-based discriminator that discriminates bidirectionally at the word level . It is worth mentioning that , without Zi , the net could still learn a mapping from Xi to Yi , but would produce deterministic outputs and fail to match any distribution other than a delta function ( Isola et al. , 2017 ) . This is one key area where our work is different from Lamb et al. ’ s and Li et al. ’ s . The schematic of the proposed hredGAN is depicted at the right hand side of Figure 1 . 2.2 GENERATOR . We adopted an HRED dialogue generator similar to ( Serban et al. , 2016 ; 2017a ; b ; Xing et al. , 2017 ) . The HRED contains three recurrent structures , i.e . the encoder ( eRNN ) , context ( cRNN ) , and decoder ( dRNN ) RNN . The conditional probability modeled by the HRED per output word token is given by PθG ( Y ji |X 1 : j−1 i+1 , Xi ) = dRNN ( E ( Xj−1i+1 ) , h j−1 i , hi ) ( 9 ) where E ( . ) is the embedding lookup , hi = cRNN ( eRNN ( E ( Xi ) , hi−1 ) , eRNN ( . ) maps a sequence of input symbols into fixed-length vector , and h and h are the hidden states of the decoder and context RNN , respectively . In the multi-resolution HRED , ( Serban et al. , 2017a ) , high-level tokens are extracted and processed by another RNN to improve performance . We circumvent the need for this extra processing by allowing the decoder to attend to different parts of the input utterance during response generation ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) . We introduce a local attention into equation 9 and encode the attention memory differently from the context through an attention encoder RNN ( aRNN ) , yielding : PθG ( Y ji |X 1 : j−1 i+1 , Xi ) = dRNN ( E ( Xj−1i+1 ) , h j−1 i , A j i , hi ) ( 10 ) where Aji = ∑Mi m=1 exp ( αm ) ∑Mi m=1 exp ( αm ) h ′m i , h ′m i = aRNN ( E ( X m i ) , h ′m−1 i ) , h ′ is the hidden state of the attention RNN and αk is either a logit projection of ( h j−1 i , h ′m i ) in the case of Bahdanau et al . ( 2015 ) or ( hj−1i ) T · h′mi in the case of Luong et al . ( 2015 ) . The modified HRED architecture is shown in Figure 2 . Noise Injection : We inject Gaussian noise at the input of the decoder RNN . Noise samples could be injected at the utterance or word level . With noise injection , the conditional probability of the decoder output becomes PθG ( Y ji |X 1 : j−1 i+1 , Z j i , Xi ) = dRNN ( E ( Xj−1i+1 ) , h j−1 i , A j i , Z j i , hi ) ( 11 ) where Zji ∼ Ni ( 0 , I ) , for utterance-level noise and Z j i ∼ N j i ( 0 , I ) , for word-level noise . 2.3 DISCRIMINATOR . The discriminator shares context and word embedding with the generator and can discriminate at the word level ( Lamb et al. , 2016 ) . The word-level discrimination is achieved through a bidirectional RNN and is able to capture both syntactic and conceptual differences between the generator output and the ground truth . The aggregate classification of an input sequence , χ can be factored over word-level discrimination and expressed as D ( Xi , χ ) = D ( hi , χ ) = [ J∏ j=1 DRNN ( hi , E ( χ j ) ) ] 1 J ( 12 ) where DRNN ( . ) is the word discriminator RNN , hi is an encoded vector of the dialogue history Xi obtained from the generator ’ s cRNN ( . ) output , and χj is the jth word or token of the input sequence χ. χ = Yi and J = Ti for the case of generator ’ s decoder output , χ = Xi+1 and J = Mi+1 for the case of ground truth . The discriminator architecture is depicted on the left hand side of Figure 1 .

Summary: This paper presents an adversarial learning model for generating diverse responses for dialogue systems based on HRED (hierarchical recurrent encoder-decoder) network. The contribution of the work mainly lies in: 1. adversarial learning, and 2. injecting Gaussian noise at word-level and sentence-level to encourage the diversity. Overall, the idea is interesting, and the automatic evaluation based on perplexity, BLEU, ROUGE, etc shows that the proposed methods outperform existing methods.