Title: AN ADVERSARIAL LEARNING FRAMEWORK FOR A PERSONA-BASED MULTI-TURN DIALOGUE MODEL Anonymous

Abstract: In this paper, we extend the persona-based sequence-to-sequence (Seq2Seq) neural network conversation model to a multi-turn dialogue scenario by modifying the state-of-the-art hredGAN architecture to simultaneously capture utterance attributes such as speaker identity, dialogue topic, speaker sentiments and so on. The proposed system, phredGAN has a persona-based HRED generator (PHRED) and a conditional discriminator. We also explore two approaches to accomplish the conditional discriminator: (1) phredGAN a , a system that passes the attribute representation as an additional input into a traditional adversarial discriminator, and (2) phredGAN d , a dual discriminator system which in addition to the adversarial discriminator, collaboratively predicts the attribute(s) that generated the input utterance. To demonstrate the superior performance of phredGAN over the persona SeqSeq model, we experiment with two conversational datasets, the Ubuntu Dialogue Corpus (UDC) and TV series transcripts from the Big Bang Theory and Friends. Performance comparison is made with respect to a variety of quantitative measures as well as crowd-sourced human evaluation. We also explore the trade-offs from using either variant of phredGAN on datasets with many but weak attribute modalities (such as with Big Bang Theory and Friends) and ones with few but strong attribute modalities (customer-agent interactions in Ubuntu dataset).

1. INTRODUCTION:

Recent advances in machine learning especially with deep neural networks has lead to tremendous progress in natural language processing and dialogue modeling research (Sutskever et al., 2014; Vinyals & Le, 2015; Serban et al., 2016) . Nevertheless, developing a good conversation model capable of fluent interaction between a human and a machine is still in its infancy stage. Most existing work relies on limited dialogue history to produce response with the assumption that the model parameters will capture all the modalities within a dataset. However, this is not true as dialogue corpora tend to be strongly multi-modal and practical neural network models find it difficult to disambiguate characteristics such as speaker personality, location and sub-topic in the data.

Most work in this domain has primarily focused on optimizing dialogue consistency. For example, Serban et al. (Serban et al., 2016; 2017b; a) and Xing et al. (2017) introduced a Hierarchical Recurrent Encoder-Decoder (HRED) network architecture that combines a series of recurrent neural networks to capture long-term context state within a dialogue. However, the HRED system suffers from lack of diversity and does not have any guarantee on the generator output since the output conditional probability is not calibrated. Olabiyi et al. (2018) tackles these problems by training a modified HRED generator alongside an adversarial discriminator in order to increase diversity and provide a strong and calibrated guarantee to the generator's output. While the hredGAN system improves upon response quality, it does not capture speaker and other attributes modality within a dataset and fails to generate persona specific responses in datasets with multiple modalities.

On the other hand, there has been some recent work on introducing persona into dialogue models. For example, Li et al. (2016b) integrates attribute embeddings into a single turn (Seq2Seq) generative dialogue model. In this work, Li et al. consider persona models one with Speaker-only representation and the other with Speaker and Addressee representations (Speaker-Addressee model), both of which capture certain speaker identity and interactions. Nguyen et al. (2018) continue along the Figure 1 : The PHRED generator with local attention -The attributes C, allows the generator to condition its response on the utterance attributes such as speaker identity, subtopics and so on. same line of thought by considering a Seq2Seq dialogue model with Responder-only representation. In both of these cases, the attribute representation is learned during the system training. Zhang et al. (2018) proposed a slightly different approach. Here, the attributes are a set of sentences describing the profile of the speaker. In this case, the attributes representation is not learned. The system however learns how to attend to different parts of the attributes during training. Still, the above persona-based models have limited dialogue history (single turn); suffer from exposure bias worsening the trade-off between personalization and conversation quality and cannot generate multiple responses given a dialogue context. This is evident in the relatively short and generic responses produced by these systems, even though they generally capture the persona of the speaker.

In order to overcome these limitations, we propose two variants of an adversarially trained persona conversational generative system, phredGAN , namely phredGAN a and phredGAN d . Both systems aim to maintain the response quality of hredGAN and still capture speaker and other attribute modalities within the conversation. In fact, both systems use the same generator architecture (PHRED generator), i.e., an hredGAN generator (Olabiyi et al., 2018) with additional utterance attribute representation at its encoder and decoder inputs as depicted in Figure 1 . Conditioning on external attributes can be seen as another input modality as is the utterance into the underlying system. The attribute representation is an embedding that is learned together with the rest of model parameters similar to Li et al. (2016b) . Injecting attributes into a multi-turn dialogue system allows the model to generate responses conditioned on particular attribute(s) across conversation turns. Since the attributes are discrete, it also allows for exploring different what-if scenarios of model responses. The difference between the two systems is in the discriminator architecture based on how the attribute is treated.

We train and sample both variants of phredGAN similar to the procedure for hredGAN (Olabiyi et al., 2018) . To demonstrate model capability, we train on a customer service related data such as the Ubuntu Dialogue Corpus (UDC) that is strongly bimodal between question poser and answerer, and transcripts from a multi-modal TV series The Big Bang Theory and Friends with quantitative and qualitative analysis. We examine the trade-offs between using either system in bi-modal or multi-modal datasets, and demonstrate system superiority over state-of-the-art persona conversational models in terms of dialogue response quality and quantitatively with perplexity, BLEU, ROUGE and distinct n-gram scores.

2. MODEL ARCHITECTURE:

In this section, we briefly introduce the state-of-the-art hredGAN model and subsequently show how we derive the two persona versions by combining it with the distributed representation of the dialogue speaker and utterance attributes, or with an attribute discrimination layer at the end of the model pipeline. 

2.1. hredGAN : ADVERSARIAL LEARNING FRAMEWORK:

Problem Formulation: The hredGAN (Olabiyi et al., 2018) formulates multi-turn dialogue response generation as: given a dialogue history of sequence of utterances,

X i = X 1 , X 2 , • • • , X i , where each utterance X i = X 1 i , X 2 i , • • • , X Mi i

contains a variable-length sequence of M i word tokens such that X i j ∈ V for vocabulary V , the dialogue model produces an output

Y i = Y 1 i , Y 2 i , • • • , Y Ti i

, where T i is the number of generated tokens. The framework uses conditional GAN structure to learn a mapping from an observed dialogue history to a sequence of output tokens. The generator, G, is trained to produce sequences that cannot be distinguished from the ground truth by an adversarially trained discriminator, D akin to a two-player min-max optimization problem. The generator is also trained to minimize the cross-entropy loss L M LE (G) between the ground truth X i+1 , and the generator output Y i . The following objective summarizes both goals:

G * , D * = arg min G max D λ G L cGAN (G, D) + λ M L M LE (G) .

(

EQUATION

where λ G and λ M are training hyperparamters and L cGAN (G, D) and L M LE (G) are defined in Eqs.

(5) and (7) of Olabiyi et al. (2018) respectively. Please note that the generator G and discriminator D share the same encoder and embedding representation of the word tokens.

2.2. phredGAN : PERSONA ADVERSARIAL LEARNING FRAMEWORK:

The proposed architecture of phredGAN is very similar to that of hredGAN (Olabiyi et al., 2018) . The only difference is that the dialogue history is now

X i = (X 1 , C 1 ), (X 2 , C 2 ), • • • , (X i , C i )

where C i is additional input that represents the speaker and/or utterance attributes. Please note that C i can either be a sequence of tokens or single token such that C i j ∈ V c for vocabulary V c. Also, at the ith turn, C i and C i+1 are the source/input attribute and target/output attribute to the generator respectively. The embedding for attribute tokens is also learned similar to that of word tokens.

Both versions of phredGAN shares the same generator architecture (PHRED) but different discriminators. Below is the highlight of how they are derived from the hredGAN architecture.

Encoder: The context RNN, cRN N takes the source attribute C i as an additional input by concatenating its representation with the output of eRN N as in Figure 1 . If the attribute C i is a sequence of tokens, then an attention (using the output of eRN N ) over the source attribute representations is concatenated with the output of eRN N . This output is used by the generator to create a context state for a turn i.

Generator: The generator decoder RNN, dRN N takes the target attribute C i+1 as an additional input as in Fig. 1 . If the attribute C i+1 is a sequence of tokens, then an attention (using the output of dRN N ) over the attribute representations is concatenated with the rest of the decoder inputs. This forces the generator to draw a connection between the generated responses and the utterance attributes such as speaker identity.

Noise Injection: As in Olabiyi et al. (2018) , we also explore different noise injection methods.

Objective: For phredGAN , the optimization objective in eq. ( 1) can be updated as:

G * , D * adv , D * att = arg min G max D adv λ G adv L adv cGAN (G, D adv ) (2) + min Datt λ Gatt L att c (G, D att ) + λ M L M LE (G) .

where L adv cGAN (G, D adv ) and L att c (G, D att ) are the traditional adversarial and attribute prediction loss respectively and dependent on the architectural variation. It is worth to point out that while the former is adversarial, the later is collaborative in nature. The MLE loss is common and can be expressed as:

L M LE (G) = E Xi+1 [-log P G X i+1 |X i , C i+1 , Z i ].

(3) where Z i the noise sample and depends on the choice of either utterance-level or word-level noise input into the generator (Olabiyi et al., 2018) .

2.3. phredGAN a : ATTRIBUTES AS A DISCRIMINATOR INPUT:

phredGAN a shares the same discriminator architecture as the hredGAN but with additional input, C i+1 . Since it does not use attribute prediction, λ Gatt = 0.

The adversarial loss, L adv cGAN (G, D) can then be expressed as:

L adv cGAN (G, D adv ) = E X i ,Ci+1,Xi+1 [log D adv (X i , C i+1 , X i+1 )] (4) + E X i ,Ci+1,Zi [1 -log D adv (X i , C i+1 , G(X i , C i+1 , Z i ))]

The addition of speaker or utterance attributes allows the dialogue model to exhibit personality traits given consistent responses across style, gender, location, and so on.

2.4. phredGAN d : ATTRIBUTES AS A DISCRIMINATOR TARGET:

phredGAN d does not take the attribute representation at its input but rather use the attributes as the target of an additional discriminator D att . The adversarial and the attribute prediction losses can be respectively expressed as:

L adv cGAN (G, D adv ) = E X i ,Xi+1 [log D adv (X i , X i+1 )] (5) + E X i ,Zi [1 -log D adv (X i , G(X i , C i+1 , Z i ))] L att c (G, D att ) = E Ci+1 [-log D att (C i+1 |X i , X i+1 )] (6) + E Ci+1 [-log D att (C i+1 |X i , G(X i , C i+1 , Z i ))]

Attribute Discriminator: In addition to the existing word-level adversarial discriminator D adv from hredGAN , we add an attribute discriminator, D att , that discriminates on an utterance level to capture attribute modalities since attributes are assigned at utterance level. The discriminator uses a unidirectional RNN (D attRN N ) that maps the input utterance to the particular attribute(s) that generated it. The attributes can be seen as hidden states that inform or shape the generator outputs. The attribute discriminator can be expressed as:

EQUATION

where E(.) is the word embedding lookup (Olabiyi et al., 2018) , χ = X i+1 for groundtruth and χ = Y i for the generator output.

3. MODEL TRAINING AND INFERENCE:



3.1. MODEL TRAINING:

We train both the generator and the discriminator (with shared encoder) of both variants of phredGAN using the training procedure in Algorithm 1 (Olabiyi et al., 2018) . For both variants, λ G adv = λ M = 1, and for phredGAN a and phredGAN d , λ Gatt = 0 and λ Gatt = 1 respectively. Since the encoder, word embedding and attribute embedding are shared, we are able to train the system end-to-end with back-propagation.

Encoder: The encoder RNN, eRN N , is bidirectional while cRRN is unidirectional. All RNN units are 3-layer GRU cell with hidden state size of 512. We use word vocabulary size, V = 50, 000 with word embedding size of 512. The number of attributes, V c is dataset dependent but we use an attribute embedding size of 512. In this study, we only use one attribute per utterance so that is no need to use attention to combine the attribute embeddings.

Generator: The generator decoder RNN, dRN N is also a 3-layer GRU cell with hidden state size of 512. The aRN N outputs are connected to the dRN N input using an additive attention mechanism (Bahdanau et al., 2015) .

Adversarial Discriminator: The word-level discriminator RNN, D RN N is a bidirectional RNN, each 3-layer GRU cell with hidden state size of 512. The output of both the forward and the backward cells for each word are concatenated and passed to a fully-connected layer with binary output.

The output is the probability that the word is from the ground truth given the past and future words of the sequence, and in the case of phredGAN a , the responding speaker's embedding.

Attribute Discriminator: The attribute discriminator RNN, D attRN N is a unidirectional RNN with a 3-layer GRU cell, each of hidden state size 512. A softmax layer is then applied to project the final hidden state to a prespecified number of attributes, V c . The output is the probability distribution over the attributes.

Others: All parameters are initialized with Xavier uniform random initialization (Glorot & Bengio, 2010) . Due to the large word vocabulary size, we use sampled softmax loss (Jean et al., 2015) for MLE loss to expedite the training process. However, we use full softmax for model evaluation. For both systems, parameters updates are conditioned on the word-level discriminator accuracy performance as in Olabiyi et al. (2018) with acc D th adv = 0.99 and acc G th = 0.75. The model is trained end-to-end using the stochastic gradient descent algorithm. Finally, the model is implemented, trained, and evaluated using the TensorFlow deep learning framework.

3.2. MODEL INFERENCE:

We use an inference strategy similar to the approach in Olabiyi et al. (2018) .

For the modified noise sample, we perform a linear search for α with sample size L = 1 based on the average word-level discriminator loss, -logD adv (G(.)) (Olabiyi et al., 2018) using trained models run in autoregressive mode to reflect performance in actual deployment. The optimum α value is then used for all inferences and evaluations. During inference, we condition the dialogue response generation on the encoder outputs, noise samples, word embedding and the attribute embedding of the intended responder. With multiple noise samples, L = 64, we rank the generator outputs by the discriminator which is also conditioned on encoder outputs, and the intended responder's attribute embedding. The final response is the response ranked highest by the discriminator. For phredGAN d , we average the confidences produced by D adv and D att .

4. EXPERIMENTS AND RESULTS:

In this section, we explore the performance of PHRED, phredGAN a and phredGAN d on two conversational datasets and compare its performance to non-adversarial persona Seq2seq models Li et al. (2016b) as well as to the adversarial hredGAN (Olabiyi et al., 2018) with no explicit persona.

4.1. DATASETS:

TV Series Transcripts dataset (Serban et al., 2016) . We train all models on transcripts from the two popular TV drama series, Big Bang Theory and Friends. Following a similar preprocessing setup in Li et al. (2016b) , we collect utterances from the top 12 speakers from both series to construct a corpus of 5,008 lines of multi-turn dialogue. We split the corpus into training, development, and test set with a 94%, 3%, and 3% proportions, respectively, and pair each set with a corresponding attribute file that maps speaker IDs to utterances in the combined dataset.

Due to the small size of the combined transcripts dataset, we first train our model on the larger Movie Triplets Corpus (MTC) by Banchs (2012) which consists of 240,000 dialogue triples. We pre-train our model on this dataset to initialize our model parameters to avoid overfitting on a relatively small persona TV series dataset. After pre-training on MTC, we reinitialize the attribute embeddings in the generator from a uniform distribution following a Xavier initialization (Glorot & Bengio, 2010) for training on the combined person TV series dataset.

Ubuntu Dialogue Corpus (UDC) dataset (Serban et al., 2017b) . We train our model on 1.85 million conversations of multi-turn dialogue from the Ubuntu community hub, with an average of 5 utterances per conversation. We assign two types of speaker IDs to utterances in this dataset: questioner and helper. We follow a similar training, development, and test split as the UDC dataset in Olabiyi et al. (2018) , with 90%, 5%, and 5% proportions, respectively, and pair each set with a corresponding attribute file that maps speaker IDs to utterances in the combined dataset While the overwhelming majority of utterances in UDC follow two speaker types, the dataset does include utterances that do not classify under either a questioner or helper speaker type. In order to remain consistent, we assume that there are only two speaker types within this dataset and that the first utterance of every dialogue is from a questioner. This simplifying assumption does introduce a degree of noise into each persona model's ability to construct attribute embeddings. However, our experiment results demonstrate that both phredGAN a and phredGAN d is still able to differentiate between the larger two speaker types in the dataset.

4.2. EVALUATION METRICS:

We use similar evaluation metrics as in Olabiyi et al. (2018) including perplexity, BLEU (Papineni et al., 2002) , ROUGE (Lin, 2014) , distinct n-gram (Li et al., 2016a) and normalized average sequence length (NASL) scores. For human evaluation, we follow a similar setup as Li et al. (2016a) , employing crowd-sourced judges to evaluate a random selection of 200 samples. We present both the multi-turn context and the generated responses from the models to 3 judges and asked them to rank the general response quality in terms of relevance, informativeness, and persona. For N models, the model with the lowest quality is assigned a score 0 and the highest is assigned a score N-1. Ties are not allowed. The scores are normalized between 0 and 1 and averaged over the total number of samples and judges. 

4.3. BASELINE:

We compare the non-adversarial persona HRED model, PHRED with the adversarially trained ones, i.e. hredGAN , phredGAN a and phredGAN d , to demonstrate the impact of adversarial training. Please note that no noise was added to the PHRED model.

We also compare the persona models to Li et al.'s work (Li et al., 2016b) which uses a Seq2Seq framework in conjunction with learnable persona embeddings. Their work explores two persona models in order to incorporate vector representations of speaker interaction and speaker attributes into the decoder of their Seq2Seq model i.e., Speaker model (SM) and Speaker-Addressee model (SAM). All reported results are based on our implementation of their models in Li et al. (2016b) .

4.4. HYPERPARAMETER SEARCH:

For both phredGAN a and phredGAN d , we determine the noise injection method and the optimum noise variance α that allows for the best performance on both datasets. We find that phredGAN d performs optimally with word-level noise injection on both Ubuntu and TV transcripts, while phredGAN a performs the best with utterance-level noise injection on TV transcripts and word-level injection on UDC. For all phredGAN models, we perform a linear search for optimal noise variance values between 1 and 30 at an increment of 1, with a sample size of L = 1. For phredGAN d , we obtain an optimal α of 4 and 6 for the UDC and TV Transcripts respectively. For phredGAN a , we obtain an optimal value of 2 and 5 for the combined TV series dataset and the much larger UDC respectively.

4.5. RESULTS:

We will now present our assessment of performance comparisons of phredGAN against the baselines, PHRED, hredGAN and Li et al.'s persona Seq2Seq models.

4.6. QUANTITATIVE ANALYSIS:

We first report the performance on TV series transcripts in table 1 . The performance of both SM and SAM models in Li et al. (2016b) compared to the hredGAN shows a strong baseline and indicates that the effect of persona is more important than that of multi-turn and adversarial training for datasets with weak multiple persona. However, once the persona information is added to the hredGAN , the resulting phredGAN shows a significant improvement over the SM and SAM baselines with phredGAN a performing best. We also observe that PHRED performs worse than the baseline S(A)M models on a number of metrics but we attribute this to the effect of persona on a limited dataset that results into less informative responses. This behavior was also reported in Li et al. (2016b) where the persona models produce less informative responses than the non-personal Seq2seq models but it seems to be even worse in multi-turn context. However, unlike the Speaker-Addressee and PHRED models that suffer from lower response quality due to persona conditioning, we note that conditioning the generator and discriminator of phredGAN on speaker embeddings does not compromise the systems ability to produce diverse responses. This problem might have been alleviated by the adversarial training that encourages the generator model to produce longer, more informative, and diverse responses that have high persona relevance even with a limited dataset.

We also compare the models performances on the UDC. The evaluation result is summarized in table 2 . While the deleterious effect of persona conditioning on response diversity is still worse with PHRED than with S(A)M models, we note that hredGAN performs much better than the S(A)M models. This is because, the external persona only provides just a little more information than is already available from the UDC utterances. We also note an improvement of phredGAN variants over the hredGAN in a variety of evaluation metrics including perplexity, ROUGE with the exception of distinct n-grams. This is expected as phredGAN should be generally less diverse than hredGAN since the number of distinct data distribution modes is more for phredGAN dataset due to the persona attributes. However, this leads to better response quality with persona, something not achievable with hredGAN . Also, the much better ROUGE(F1) score indicates that phredGAN is able to strike a better balance between diversity and precision while still capturing the characteristics of the speaker attribute modality in the UDC dataset. Within the phredGAN variants, phredGAN d seems to perform better. This is not surprising as speaker classification is much easier on UDC than on TV series. The attribute discriminator, D att is able to provide more informative feedback on UDC than on TV series where it is more difficult to accurately predict the speaker. Therefore, we recommend phredGAN a for datasets with weak attribute distinction and phredGAN d for strong attribute distinction.

4.7. QUALITATIVE ANALYSIS:

In addition to the quantitative analysis above, we report the results of the human evaluation in the last column of tables 1 and 2 for the TV Series and UDC datasets respectively. The human evaluation scores largely agrees with the automatic evaluations on the TV Series with phredGAN a clearly giving the best performance. However, on the UDC, both hredGAN and phredGAN d performs similarly which indicates that there is a trade off between diversity and persona by each model. We believe this is due to the strong persona information that already exists in the UDC utterances.

An additional qualitative assessment of these results are in Table 3 with responses from several characters in the TV series dataset and the two characters in UDC.

We see that for TV drama series, phredGAN responses are comparatively more informative than that of the Speaker-Addressee model of Li et al. (2016b) . For example, all the characters in the TV series respond the same to the dialogue context. Similar behavior is reported in Li et al. (2016b) where for the Speaker-Addressee model, nearly all the characters in the TV series respond with We also see similar results with our model's output on UDC in table 4. We demonstrate that by conditioning as either a helper or questioner from the UDC dataset, phredGAN models are able to respond differently to input utterances as well as stay close to the context of the conversation.

5. CONCLUSION AND FUTURE WORK:

In this paper, we improve upon state-of-the-art persona-based response generation models by exploring two persona conversational models: phredGAN a which passes the attribute representation as an additional input into a traditional adversarial discriminator, and phredGAN d a dual discriminator system which in addition to the adversarial discriminator from hredGAN , collaboratively predicts the attribute(s) that are intrinsic to the input utterance. Both systems demonstrate quantitative improvements upon state-of-the-art persona conversational systems such as the work from Li et al. (2016b) with respect to both quantitative automatic and qualitative human measures.

Our analysis also demonstrates how both variants of phredGAN perform differently on datasets with weak and strong modality. One of our future direction is to take advantage of phredGAN d 's ability to predict utterance attribute such as speaker identity from just the utterance. We believe its performance can be improved even with weak modality by further conditioning adversarial updates on both the attribute and adversarial discriminator accuracies. Overall, this paper demonstrates clear benefits from adversarial training of persona generative dialogue system and leaves the door open for more interesting work to be accomplished in this domain. 

:



Figures and Tables:

Figure 2: The phredGAN d dual discriminator -Left: D adv is a word-level discriminator used by both phredGAN a and phredGAN d to judge normal dialogue coherency as in hredGAN . Right: D att , an utterance-level attribute discriminator is used only in phredGAN d to predict the likelihood a given utterance was generated from a particular attribute.

phredGAN vs.Li et al. (2016b)  on BBT Friends TV Transcripts.

Table content: <table><tr><td>Model</td><td>Teacher Forcing Perplexity</td><td colspan="4">Autoregression BLEU ROUGE-2 DISTINCT-1/2 NASL Evaluation Human</td></tr><tr><td>TV Series</td><td/><td/><td/><td/><td/></tr><tr><td>SM</td><td>22.13</td><td>1.76 %</td><td>22.4 %</td><td>2.50%/18.95% 0.786</td><td>0.5566</td></tr><tr><td>SAM</td><td>23.06</td><td colspan="2">1.86 % 20.52 %</td><td>2.56%/18.91% 0.689</td><td>0.5427</td></tr><tr><td>hredGAN</td><td>28.15</td><td>2.14 %</td><td>6.81 %</td><td>1.85 %/6.93 % 1.135</td><td>0.5078</td></tr><tr><td>phred</td><td>30.94</td><td colspan="2">2.41 % 14.03 %</td><td>0.66 %/2.54 % 1.216</td><td>0.3663</td></tr><tr><td>phredGANa</td><td>25.10</td><td colspan="3">3.07 % 30.47 % 2.19 %/19.02 % 1.218</td><td>0.6127</td></tr><tr><td>phredGAN d</td><td>28.19</td><td colspan="2">2.76 % 14.68 %</td><td>0.70 %/4.76 % 1.163</td><td>0.4284</td></tr></table>

phredGAN vs. Li et al. (2016b) on UDC.

Table content: <table><tr><td>Model</td><td>Teacher Forcing Perplexity</td><td>BLEU-2/4</td><td colspan="3">Autoregression ROUGE-2 DISTINCT-1/2 NASL Evaluation Human</td></tr><tr><td>UDC</td><td/><td/><td/><td/><td/></tr><tr><td>SM</td><td>28.32</td><td>0.437%/∼ 0%</td><td>9.19 %</td><td>1.61%/5.79% 0.506</td><td>0.4170</td></tr><tr><td>SAM</td><td>26.12</td><td colspan="2">0.490%/∼ 0% 10.23 %</td><td>1.85%/6.85% 0.512</td><td>0.4629</td></tr><tr><td>hredGAN</td><td>48.18</td><td>2.16%/∼ 0%</td><td colspan="2">11.68 % 5.16%/18.21% 1.098</td><td>0.5876</td></tr><tr><td>phred</td><td>34.67</td><td>0.16%/∼ 0%</td><td>7.41%</td><td>0.56%/1.44% 0.397</td><td>0.4718</td></tr><tr><td>phredGANa</td><td>31.25</td><td>1.94%/∼ 0%</td><td>19.15%</td><td>1.05%/5.28% 1.520</td><td>0.4558</td></tr><tr><td>phredGAN d</td><td>28.74</td><td>2.02%/0.10%</td><td>16.82%</td><td>1.38%/5.77% 1.387</td><td>0.5817</td></tr></table>

hredGAN vs phredGAN vs Li et al. (2016b) on TV Transcripts

Table content: <table><tr><td>Model</td><td>Speaker</td><td>Response</td></tr><tr><td colspan="2">Big Bang Theory TV Series</td><td/></tr><tr><td>Context 0</td><td>Sheldon</td><td>Do you like me?</td></tr><tr><td>phredGANa</td><td>Leonard</td><td>Yes ?</td></tr><tr><td>phredGAN d</td><td>Leonard</td><td>You can ' t do that ? !</td></tr><tr><td>phredGANa</td><td>Penny</td><td>No , you don ' t understand . You ' re an idiot .</td></tr><tr><td>phredGAN d</td><td>Penny</td><td>You ' re right . . .</td></tr><tr><td>phredGANa</td><td>Howard</td><td>Well , I don ' t know what to say . This is the part I ' ve ever seen .</td></tr><tr><td>phredGAN d</td><td>Howard</td><td>You ' re right , you ' re just gonna go with this guy .</td></tr><tr><td>phredGANa</td><td>Raj</td><td>Well , you know , we could be a little more than my friends ' s friends .</td></tr><tr><td>phredGAN d</td><td>Raj</td><td>You know , I can ' t can ' t do I can ' go , I can ' t do that . . .</td></tr><tr><td>SM</td><td>Leonard/Penny/Howard/Raj</td><td>Yeah , I ' m sure . Oh , but I can ' t believe it . I ' m not gonna be in here</td></tr><tr><td>SAM</td><td>Leonard/Penny/Howard/Raj</td><td>I ' m not going to be a good time .</td></tr><tr><td>hredGAN</td><td>NA</td><td>Okay , I ' m gonna go see my friend .</td></tr><tr><td>Context 0</td><td>Howard</td><td>We can't go home.</td></tr><tr><td>Context 1</td><td>Leonard</td><td>Alright, I'll be right back.</td></tr><tr><td>phredGANa</td><td>Howard</td><td>Okay , well , before you and Penny get out and get married with Penny just get married.</td></tr><tr><td>phredGAN d</td><td>Howard</td><td>You ' ll see , I ' ll just go .</td></tr><tr><td>SM</td><td>Howard</td><td>Okay .</td></tr><tr><td>SAM</td><td>Howard</td><td>Okay , well , you ' re really good at this .</td></tr><tr><td>hredGAN</td><td>NA</td><td>Okay , I ' m gonna go see my old .</td></tr><tr><td>Context 0</td><td>Amy</td><td>Spent a lot of my childhood throwing coins into wishing wells hoping for friends.</td></tr><tr><td>Context 1</td><td>Penny</td><td>Whoa, whoa, give me a second to catch up here. Hold on.</td></tr><tr><td>Context 2</td><td>Amy</td><td>That'd be my boyfriend. Happier playing his dopey Star Trek game with his friends than hanging out with</td></tr><tr><td/><td/><td>me.</td></tr><tr><td>phredGANa</td><td>Penny</td><td>You know , when you ' re being so nice about Sheldon and I want to take him to like his own girlfriend</td></tr><tr><td>phredGAN d</td><td>Penny</td><td>You , I ' m sorry , I didn ' t have it . d be back in the bathroom . .</td></tr><tr><td>SM</td><td>Penny</td><td>I ' m sorry , I ' m sorry . I ' m just saying that you ' re going to have to answer to</td></tr><tr><td>SAM</td><td>Penny</td><td>It ' s not gonna happen .</td></tr><tr><td>hredGAN</td><td>NA</td><td>Okay , I ' m gonna go see my new .</td></tr><tr><td>Context 0</td><td>Sheldon</td><td>I recently read that during World War Two, Joseph Stalin had a research program to create supersoldiers</td></tr><tr><td/><td/><td>by having women impregnated by gorillas.</td></tr><tr><td>phredGANa</td><td>Howard</td><td>You ' re right . I don ' t know what you ' re talking about , Sheldon .</td></tr><tr><td>phredGAN d</td><td>Howard</td><td>You ' re kidding . .</td></tr><tr><td>SM</td><td>Howard</td><td>Well , my last was a little . I was just trying to be a good time .</td></tr><tr><td>SAM</td><td>Howard</td><td>That would be great .</td></tr><tr><td>Friends TV Series</td><td/><td/></tr><tr><td>Context 0</td><td>Rachel</td><td>Do you like me?</td></tr><tr><td>phredGANa</td><td>Joey</td><td>Yeah .</td></tr><tr><td>phredGAN d</td><td>Joey</td><td>You ' re right . . . .</td></tr><tr><td>phredGANa</td><td>Ross</td><td>Well , no , no , no . I mean , I just , I don ' t know . . .</td></tr><tr><td>phredGAN d</td><td>Ross</td><td>You ' want to know sex . .</td></tr><tr><td>phredGANa</td><td>Monica</td><td>Well , I do . And I ' m not gonna tell you another guy .</td></tr><tr><td>phredGAN d</td><td>Monica</td><td>You ' re a , you ' re really good . . . .</td></tr><tr><td>phredGANa</td><td>Chandler</td><td>No , I do not . You are a mean little man . .</td></tr><tr><td>phredGAN d</td><td>Chandler</td><td>You ' re right , I can ' t do that .</td></tr><tr><td>SM</td><td>Joey/Ross/Monica/Chandler</td><td>Yeah , I ' m sure . Oh , but I can ' t believe it . I ' m not gonna be in here</td></tr><tr><td>SAM</td><td>Joey/Ross/Monica/Chandler</td><td>I ' m not going to be a good time .</td></tr><tr><td>hredGAN</td><td>NA</td><td>Okay , I ' m gonna go see my friend .</td></tr></table>

hredGAN vs phredGAN vsLi et al. (2016b)  on UDC Of course I love you." to the dialogue context, "Do you love me?" despite the fact that some of the responders sometimes have unfriendly relationship with the addressee. Many of the novel situations explored by phredGAN are unachievable with the Speaker-Addressee model due to lack of informative responses. For example, by conditioning as Sheldon from The Big Bang Theory and

Table content: <table><tr><td>Model</td><td>Speaker</td><td>Response</td></tr><tr><td>UDC</td><td/><td/></tr><tr><td>Context 0</td><td>asker</td><td>all i need some quick amarok help. i cant hear my music</td></tr><tr><td>Context 1</td><td>helper</td><td>is amarok muted?</td></tr><tr><td>Context 2</td><td>asker</td><td>no</td></tr><tr><td>phredGANa</td><td>helper</td><td>use the UNK drivers , and then run the UNK command to get the UNK</td></tr><tr><td>phredGAN d</td><td>helper</td><td>ok , so you re not using the right driver for the network card , you re using the UNK ?</td></tr><tr><td>phred</td><td>helper</td><td>you can try to install the UNK package</td></tr><tr><td>SM</td><td>helper</td><td>ok , thanks</td></tr><tr><td>SAM</td><td>helper</td><td>ok , thanks</td></tr><tr><td>hredGAN</td><td>helper</td><td>I have no idea why it would be a bit of a bit.</td></tr><tr><td>Context 0</td><td>asker</td><td>anyone had problems with the kernel update from today? giving me a kernel panic</td></tr><tr><td>Context 1</td><td>helper</td><td>you can select previous kernels at the bootloader (grub) menu on booth.</td></tr><tr><td>phredGANa</td><td>asker</td><td>it says that it is not installed . . .</td></tr><tr><td>phredGAN d</td><td>asker</td><td>ok , so i ll have to reinstall the new kernel , i ll try that.</td></tr><tr><td>phred</td><td>asker</td><td>you can try to install the drivers from the live cd</td></tr><tr><td>SM</td><td>asker</td><td>I ' m not sure what you mean . . .</td></tr><tr><td>SAM</td><td>asker</td><td>I ' m not sure how to do that . . .</td></tr><tr><td>hredGAN</td><td>asker</td><td>I ' m not sure how to do that , but I can ' t boot from a CD . . .</td></tr><tr><td>Context 0</td><td>asker</td><td>how do I install Ubuntu?</td></tr><tr><td>phredGANa</td><td>helper</td><td>use the alternate cd , it should be in the repos , its a good place to get the source of the kernel</td></tr><tr><td>phredGAN d</td><td>helper</td><td>ok , so you have to reinstall the kernel from the CD , and you can t install the iso to the CD</td></tr><tr><td>phred</td><td>helper</td><td>yes</td></tr><tr><td>SM</td><td>helper</td><td>you can use the command line</td></tr><tr><td>SAM</td><td>helper</td><td>what is your question ?</td></tr><tr><td>hredGAN</td><td>helper</td><td>you can use the UNK to install the UNK</td></tr><tr><td>"</td><td/><td/></tr></table>

Sample of phredGAN d outputs with dual discriminator scores Source Speaker D adv (G(.)) Prediction D att (G(.)) Utterance Well, that cant be right. No ones ever done that before. Except me,because I just did it. Sheldon and his brain, yeah. you have to reinstall the kernel from the CD , and you can ' t install the iso to the CD Response 1 helper 0.3783 helper 0.8540 ok , so you have to go to the ubuntu site and see if you have the same version ? Response 2 helper 0.1618 helper 0.8540 ok , so are you using the ubuntu version ? ?

Table content: <table><tr><td colspan="2">Big Bang Theory TV Series</td><td/><td/><td/></tr><tr><td colspan="2">Context 0 Howard NA</td><td>NA</td><td>NA</td><td>Okay, you have to understand something, were in a hospital right now.</td></tr><tr><td colspan="2">Response 0 Penny 0.7389</td><td>Rachel</td><td>0.3101</td><td>You can ' believe me , you can ' t can ' t do it . .</td></tr><tr><td colspan="2">Response 1 Penny 0.5324</td><td>Rachel</td><td>0.3101</td><td>You ' re right , I'm sorry , I ' go , I can ' t do that . . .</td></tr><tr><td colspan="2">Response 2 Penny 0.4394</td><td>Rachel</td><td>0.3101</td><td>You ' re right .</td></tr><tr><td colspan="5">Context 0 Sheldon NA Did I? Response 0 Leonard 0.7028 NA NA Rachel 0.3805 You ' re really good .</td></tr><tr><td colspan="2">Response 1 Leonard 0.5998</td><td>Rachel</td><td>0.3805</td><td>You know , I can ' t believe you can ' to do that .</td></tr><tr><td colspan="2">Response 2 Leonard 0.4384</td><td>Rachel</td><td>0.3805</td><td>You ' re right .</td></tr><tr><td colspan="2">Context 0 Leonard NA</td><td>NA</td><td>NA</td><td>Shes gotten really hooked on Age of Conan, shes playing non-stop.</td></tr><tr><td>Response 0 Raj</td><td>0.6707</td><td>Rachel</td><td>0.2867</td><td>You know , I , I just want to be friends with you .</td></tr><tr><td>Response 1 Raj</td><td>0.6451</td><td>Rachel</td><td>0.2867</td><td>You ' want on to do that ?</td></tr><tr><td>Response 2 Raj</td><td>0.4138</td><td>Rachel</td><td>0.2867</td><td>You okay ?</td></tr><tr><td>UDC</td><td/><td/><td/><td/></tr><tr><td>Context 0 asker</td><td>NA</td><td>NA</td><td>NA</td><td>all i need some quick amarok help. i cant hear my music</td></tr><tr><td colspan="2">Context 1 helper NA</td><td>NA</td><td>NA</td><td>is amarok muted?</td></tr><tr><td>Context 2 asker</td><td>NA</td><td>NA</td><td>NA</td><td>no</td></tr><tr><td colspan="2">Response 0 helper 0.4489</td><td>helper</td><td>0.8582</td><td>ok , so you ' re not using the right driver for the network card , you ' re using the UNK ?</td></tr><tr><td colspan="2">Response 1 helper 0.2312</td><td>helper</td><td>0.8582</td><td>ok , so you ' re not using the right driver , right ?</td></tr><tr><td colspan="2">Response 2 helper 0.1381</td><td>helper</td><td>0.8582</td><td>ok , so you are you using the UNK ?</td></tr><tr><td>Context 0 asker</td><td>NA</td><td>NA</td><td>NA</td><td>anyone had problems with the kernel update from today? giving me a kernel panic</td></tr><tr><td colspan="2">Context 1 helper NA</td><td>NA</td><td>NA</td><td>you can select previous kernels at the bootloader (grub) menu on booth.</td></tr><tr><td>Response 0 asker</td><td>0.3431</td><td>asker</td><td>0.8126</td><td>ok , so i ' ll have to reinstall the new kernel , i ' ll try that .</td></tr><tr><td>Response 1 asker</td><td>0.2518</td><td>asker</td><td>0.8126</td><td>ok , so i ' just reinstall the ubuntu version ?</td></tr><tr><td>Response 2 asker</td><td>0.1005</td><td>asker</td><td>0.8126</td><td>ok , thanks :)</td></tr><tr><td>Context 0 asker</td><td>NA</td><td>NA</td><td>NA</td><td>how do I install Ubuntu?</td></tr><tr><td colspan="2">Response 0 helper 0.4306</td><td>helper</td><td>0.8540</td><td>ok , so</td></tr></table>

Peer review (iclr2019):
 This paper proposes an extension to hredGAN, which is an adversarial framework for multi-turn dialogue model, to simultaneously learns a set of attribute embeddings that represents the persona of each speaker and generate persona-based responses. The generator of the proposed system phredGAN is conditioned on both the history utterances and the speakers’ persona by concatenating the utterance encoding with attribute embeddings. For discriminator, the authors explore two versions: 1) phredGAN_a takes attributes as inputs; 2) phredGAN_d adds a dual discriminator that predicts the attribute(s) for each utterance. Strength: 1) to the best of my knowledge, adding persona information to an adversarial multi-turn dialogue model is novel; 2) the authors explore two different approaches to build the discriminator(s) and the idea of adding a second discriminator that predicts the attributes seems interesting. Weakness: 1) Novelty: The idea of learning speaker-specific attribute embeddings is very similar to the Speaker Model proposed by Li et al.(2016) http://www.aclweb.org/anthology/P16-1094 and the proposed system only makes minor changes to hredGAN https://arxiv.org/abs/1805.11752. 2) Presentation: The writing of this paper is a little hard to follow, for example, it presents the two discriminators after the objective function (Equation 2) and does not explain the intuition behind each model. In Equation 2, the objective function, why training the discriminator to minimize the attributes prediction probability？ Simply saying the attribute prediction loss is collaborative is not clear enough. Or is the min for the second term a typo? 3) Model: The idea of adding a discriminator that predicts the attributes seems interesting. However the loss is not adversarial for the second discriminator (Equation 6), you should not indicate L_att is GAN in your notation. I’m also not convinced that this should be collaborative. Despite that the “discriminator” is trying to predict the correct attribute id, the input of the two terms in Equation 6 is different, one comes from the true data, the other comes from the generator. Shouldn’t the discriminator try to differentiate these two cases? Otherwise, it’s not a discriminator (also raise the question for Equation 2, why argmin min(L_att)). 4) Evaluation： The evaluation is not strong enough to demonstrate the benefit of the proposed model. a. It only compares against one previous work that takes speaker identity into account on one dataset. Despite that the authors apply several different metrics to evaluate the proposed model, they only compare with previous models by Li et al. (2016) on perplexity and BLEU. b. The perplexity scores of the proposed models are worse than SAM by Li et al. (2016). The authors explain this by stating that the entropy for a multi-turn model is supposed to be higher than the single-turn model. It’s better to provide a more rigorous analysis. For a fair comparison, they could also train the proposed model using only one-turn history, which should be identical to Li et al.’s setting (How many turns history are you using?). The improvements of the BLEU score might also be the consequence of substituting the past generated sequence in the generator with ground truth (since the model uses the same training algorithm as hredGAN https://arxiv.org/abs/1805.11752). It’s unclear if this is the cause unless the authors provide the comparison among SAM, hredGAN, and phredGAN. c. Table 2 compares the non-persona hredGAN with phredGAN on UDC, but the authors do not provide a comparison between these two on the TV dataset in Table 1. d. The comparison between phredGAN_a and phredGAN_d is inconsistent for the two datasets (Table 1 and Table 2).