An Adversarial Learning Framework for a Persona-based Multi-turn Dialogue Model

1 INTRODUCTION . Recent advances in machine learning especially with deep neural networks has lead to tremendous progress in natural language processing and dialogue modeling research ( Sutskever et al. , 2014 ; Vinyals & Le , 2015 ; Serban et al. , 2016 ) . Nevertheless , developing a good conversation model capable of fluent interaction between a human and a machine is still in its infancy stage . Most existing work relies on limited dialogue history to produce response with the assumption that the model parameters will capture all the modalities within a dataset . However , this is not true as dialogue corpora tend to be strongly multi-modal and practical neural network models find it difficult to disambiguate characteristics such as speaker personality , location and sub-topic in the data . Most work in this domain has primarily focused on optimizing dialogue consistency . For example , Serban et al . ( Serban et al. , 2016 ; 2017b ; a ) and Xing et al . ( 2017 ) introduced a Hierarchical Recurrent Encoder-Decoder ( HRED ) network architecture that combines a series of recurrent neural networks to capture long-term context state within a dialogue . However , the HRED system suffers from lack of diversity and does not have any guarantee on the generator output since the output conditional probability is not calibrated . Olabiyi et al . ( 2018 ) tackles these problems by training a modified HRED generator alongside an adversarial discriminator in order to increase diversity and provide a strong and calibrated guarantee to the generator ’ s output . While the hredGAN system improves upon response quality , it does not capture speaker and other attributes modality within a dataset and fails to generate persona specific responses in datasets with multiple modalities . On the other hand , there has been some recent work on introducing persona into dialogue models . For example , Li et al . ( 2016b ) integrates attribute embeddings into a single turn ( Seq2Seq ) generative dialogue model . In this work , Li et al . consider persona models one with Speaker-only representation and the other with Speaker and Addressee representations ( Speaker-Addressee model ) , both of which capture certain speaker identity and interactions . Nguyen et al . ( 2018 ) continue along the same line of thought by considering a Seq2Seq dialogue model with Responder-only representation . In both of these cases , the attribute representation is learned during the system training . Zhang et al . ( 2018 ) proposed a slightly different approach . Here , the attributes are a set of sentences describing the profile of the speaker . In this case , the attributes representation is not learned . The system however learns how to attend to different parts of the attributes during training . Still , the above persona-based models have limited dialogue history ( single turn ) ; suffer from exposure bias worsening the trade-off between personalization and conversation quality and can not generate multiple responses given a dialogue context . This is evident in the relatively short and generic responses produced by these systems , even though they generally capture the persona of the speaker . In order to overcome these limitations , we propose two variants of an adversarially trained persona conversational generative system , phredGAN , namely phredGANa and phredGANd . Both systems aim to maintain the response quality of hredGAN and still capture speaker and other attribute modalities within the conversation . In fact , both systems use the same generator architecture ( PHRED generator ) , i.e. , an hredGAN generator ( Olabiyi et al. , 2018 ) with additional utterance attribute representation at its encoder and decoder inputs as depicted in Figure 1 . Conditioning on external attributes can be seen as another input modality as is the utterance into the underlying system . The attribute representation is an embedding that is learned together with the rest of model parameters similar to Li et al . ( 2016b ) . Injecting attributes into a multi-turn dialogue system allows the model to generate responses conditioned on particular attribute ( s ) across conversation turns . Since the attributes are discrete , it also allows for exploring different what-if scenarios of model responses . The difference between the two systems is in the discriminator architecture based on how the attribute is treated . We train and sample both variants of phredGAN similar to the procedure for hredGAN ( Olabiyi et al. , 2018 ) . To demonstrate model capability , we train on a customer service related data such as the Ubuntu Dialogue Corpus ( UDC ) that is strongly bimodal between question poser and answerer , and transcripts from a multi-modal TV series The Big Bang Theory and Friends with quantitative and qualitative analysis . We examine the trade-offs between using either system in bi-modal or multi-modal datasets , and demonstrate system superiority over state-of-the-art persona conversational models in terms of dialogue response quality and quantitatively with perplexity , BLEU , ROUGE and distinct n-gram scores . 2 MODEL ARCHITECTURE . In this section , we briefly introduce the state-of-the-art hredGAN model and subsequently show how we derive the two persona versions by combining it with the distributed representation of the dialogue speaker and utterance attributes , or with an attribute discrimination layer at the end of the model pipeline . 2.1 hredGAN : ADVERSARIAL LEARNING FRAMEWORK Problem Formulation : The hredGAN ( Olabiyi et al. , 2018 ) formulates multi-turn dialogue response generation as : given a dialogue history of sequence of utterances , Xi = ( X1 , X2 , · · · , Xi ) , where each utterance Xi = ( X1i , X 2 i , · · · , X Mi i ) contains a variable-length sequence of Mi word tokens such that Xij ∈ V for vocabulary V , the dialogue model produces an output Yi = ( Y 1i , Y 2 i , · · · , Y Ti i ) , where Ti is the number of generated tokens . The framework uses conditional GAN structure to learn a mapping from an observed dialogue history to a sequence of output tokens . The generator , G , is trained to produce sequences that can not be distinguished from the ground truth by an adversarially trained discriminator , D akin to a two-player min-max optimization problem . The generator is also trained to minimize the cross-entropy loss LMLE ( G ) between the ground truthXi+1 , and the generator output Yi . The following objective summarizes both goals : G∗ , D∗ = argmin G max D ( λGLcGAN ( G , D ) + λMLMLE ( G ) ) . ( 1 ) where λG and λM are training hyperparamters andLcGAN ( G , D ) andLMLE ( G ) are defined in Eqs . ( 5 ) and ( 7 ) of Olabiyi et al . ( 2018 ) respectively . Please note that the generator G and discriminator D share the same encoder and embedding representation of the word tokens . 2.2 phredGAN : PERSONA ADVERSARIAL LEARNING FRAMEWORK The proposed architecture of phredGAN is very similar to that of hredGAN ( Olabiyi et al. , 2018 ) . The only difference is that the dialogue history is now Xi = ( ( X1 , C1 ) , ( X2 , C2 ) , · · · , ( Xi , Ci ) ) where Ci is additional input that represents the speaker and/or utterance attributes . Please note that Ci can either be a sequence of tokens or single token such that Cij ∈ V c for vocabulary V c. Also , at the ith turn , Ci and Ci+1 are the source/input attribute and target/output attribute to the generator respectively . The embedding for attribute tokens is also learned similar to that of word tokens . Both versions of phredGAN shares the same generator architecture ( PHRED ) but different discriminators . Below is the highlight of how they are derived from the hredGAN architecture . Encoder : The context RNN , cRNN takes the source attribute Ci as an additional input by concatenating its representation with the output of eRNN as in Figure 1 . If the attribute Ci is a sequence of tokens , then an attention ( using the output of eRNN ) over the source attribute representations is concatenated with the output of eRNN . This output is used by the generator to create a context state for a turn i . Generator : The generator decoder RNN , dRNN takes the target attribute Ci+1 as an additional input as in Fig . 1 . If the attribute Ci+1 is a sequence of tokens , then an attention ( using the output of dRNN ) over the attribute representations is concatenated with the rest of the decoder inputs . This forces the generator to draw a connection between the generated responses and the utterance attributes such as speaker identity . Noise Injection : As in Olabiyi et al . ( 2018 ) , we also explore different noise injection methods . Objective : For phredGAN , the optimization objective in eq . ( 1 ) can be updated as : G∗ , D∗adv , D ∗ att = argmin G ( max Dadv λGadvLadvcGAN ( G , Dadv ) ( 2 ) +min Datt λGattLattc ( G , Datt ) + λMLMLE ( G ) ) . where LadvcGAN ( G , Dadv ) and Lattc ( G , Datt ) are the traditional adversarial and attribute prediction loss respectively and dependent on the architectural variation . It is worth to point out that while the former is adversarial , the later is collaborative in nature . The MLE loss is common and can be expressed as : LMLE ( G ) = EXi+1 [ −log PG ( Xi+1|Xi , Ci+1 , Zi ) ] . ( 3 ) where Zi the noise sample and depends on the choice of either utterance-level or word-level noise input into the generator ( Olabiyi et al. , 2018 ) . 2.3 phredGANa : ATTRIBUTES AS A DISCRIMINATOR INPUT phredGANa shares the same discriminator architecture as the hredGAN but with additional input , Ci+1 . Since it does not use attribute prediction , λGatt = 0 . The adversarial loss , LadvcGAN ( G , D ) can then be expressed as : LadvcGAN ( G , Dadv ) = EXi , Ci+1 , Xi+1 [ logDadv ( Xi , Ci+1 , Xi+1 ) ] ( 4 ) + EXi , Ci+1 , Zi [ 1− logDadv ( Xi , Ci+1 , G ( Xi , Ci+1 , Zi ) ) ] The addition of speaker or utterance attributes allows the dialogue model to exhibit personality traits given consistent responses across style , gender , location , and so on . 2.4 phredGANd : ATTRIBUTES AS A DISCRIMINATOR TARGET phredGANd does not take the attribute representation at its input but rather use the attributes as the target of an additional discriminator Datt . The adversarial and the attribute prediction losses can be respectively expressed as : LadvcGAN ( G , Dadv ) = EXi , Xi+1 [ logDadv ( Xi , Xi+1 ) ] ( 5 ) + EXi , Zi [ 1− logDadv ( Xi , G ( Xi , Ci+1 , Zi ) ) ] Lattc ( G , Datt ) = ECi+1 [ − logDatt ( Ci+1|Xi , Xi+1 ) ] ( 6 ) + ECi+1 [ − logDatt ( Ci+1|Xi , G ( Xi , Ci+1 , Zi ) ) ] Attribute Discriminator : In addition to the existing word-level adversarial discriminator Dadv from hredGAN , we add an attribute discriminator , Datt , that discriminates on an utterance level to capture attribute modalities since attributes are assigned at utterance level . The discriminator uses a unidirectional RNN ( DattRNN ) that maps the input utterance to the particular attribute ( s ) that generated it . The attributes can be seen as hidden states that inform or shape the generator outputs . The attribute discriminator can be expressed as : Datt ( Ci+1|Xi , χ ) = DattRNN ( hi , E ( χ ) ) ( 7 ) where E ( . ) is the word embedding lookup ( Olabiyi et al. , 2018 ) , χ = Xi+1 for groundtruth and χ = Yi for the generator output .

Summary: This paper proposes an extension to hredGAN, which is an adversarial framework for multi-turn dialogue model, to simultaneously learns a set of attribute embeddings that represents the persona of each speaker and generate persona-based responses. The generator of the proposed system phredGAN is conditioned on both the history utterances and the speakers’ persona by concatenating the utterance encoding with attribute embeddings. For discriminator, the authors explore two versions: 1) phredGAN_a takes attributes as inputs; 2) phredGAN_d adds a dual discriminator that predicts the attribute(s) for each utterance.