Alternating Recurrent Dialog Model with Large-Scale Pre-Trained Language Models
Authors: Qingyang Wu
Yichi Zhang
Yu Li
Zhou Yu
Venue:  (2020)
Keywords: NLP
Pre-training
GPT-2
Text Generation
Dialog Generation

Abstract: Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks. The recent success of large pre-trained language models such as BERT and GPT-2 have suggested the effectiveness of incorporating language priors in down-stream NLP tasks. However, how much pre-trained language models can help dialog response generation is still under exploration. In this paper, we propose a simple, general, and effective framework: Alternating Recurrent Dialog Model (ARDM). ARDM models each speaker separately and takes advantage of the large pre-trained language model. It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM outperforms or is on par with state-of-the-art methods on two popular task-oriented dialog datasets: CamRest676 and MultiWOZ. Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion. In persuasion tasks, ARDM is capable of generating human-like responses to persuade people to donate to a charity.

Under review as a conference paper at ICLR 2020
ALTERNATING ROLES DIALOG MODEL WITH LARGE- SCALE -RAANRDINED LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Existing dialog system models require extensive human annotations and are dif- ficult to generalize to different tasks. The recent success of large pre-trained lan- guage models such BERT and GPT-2 (Devlin et al. 2019: Radford et al. 2019) have suggested the effectiveness of incorporating language priors in down-stream NLP tasks. However, how much pre-trained language models response generation is still under exploration. In this paper, we propose a sim- ple, general, and effective framework: Altemnatin Roles Dialog Model (ARDM). ARDM models each speaker separately and takes advantage of the large pre- trained language model. It requires supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM outper- forms or is on par with state-of-the-an methods log datasets: CamRest676 and MultWWOZ. Moreover, we can generalize ARDM more challenging, non-collaborative tasks such as persuasion. persuasion tasks, ARDM is capable of generating human-like responses to persuade people to donate to a charity.
help dialog
popular task-oriented dia-
INTRODUCTION
has been a long-standing ambition for artificial intelligence researchers to create an intelligent conversational agent that can generate human-like responses. Recently data-driven dialog models are more and more popular. However, most current state-of-the-an art approaches still rely heavily on extensive annotations such as belief states and dialog acts (Lei et al., 2018). However, dialog content can vary considerably in different dialog tasks. Having a different intent or dialog act annotation scheme for each task is costly. For some tasks, it is even impossible, such open-domain social chat. Thus, is difficult to utilize these methods on challenging dialog tasks, such as persuasion and negotiation, where dialog states and acts are difficult to annotate.
Eric & Manning (017) proposed a simple requence-to-sequemne architecture that requires no ex- plicit annotations. The model learns extract information from dialog history with attention and copy mechanism. However, due to the limited language modeling capabilities in the previous model, Sequicity (Lei et al. 2018), which reuses belief states inputs for supervision, outperforms Eric & Manning 2017y's 's method significantly in recent dialog datasets. But with the success of large pre- trained language models such as BERT (Devlin al.) 2019) and GPT-2 (Rafford al., 2019). we re-examine Eric & Manning (2017)'s s method and investigate how large-scale pre-trained language models can help dialog tasks.
Previous large-scale pre-trained language models are used to tackle documents with only one nar- rator. However, in dialogs, two speakers have different roles; therefore, their language model dis- tributions are very different from each other. For example, customer service agents speak very differently to their customers. To address this issue, we propose ARDM, dialog model that en- codes and decodes different speaker utterances in alternating order with two pre-trained large-scale language models. To investigate whether ARDM can help dialog response generation, we evalu- ate its performance on three different task-oriented dialog datasets: CamRes676, MultiWOZ, and PersuasionForGood. The first two datasets are traditional information request dialog datasets with well-defined automatic evaluation metrics task completion. By contrast, PersuasionForGoood is a new dataset that focuses on persuading people to donate to a charity. There no explicit dialog state defined in this task such non-collaborative dialogs have various dialog actions.

---

Under review as a conference paper at ICLR 2020
We observe that ARDM is capable of improving task-oriented dialog tasks performance over the previous state-of-the-ar aar methods without incorporating any explicit supervision from belief states or dialog acts. Also, due to ARDM's simplicity and generality, one rapidly build a dialog prototype on different types of applications using only conversations without any manual annotations. We also found that ARDM works well complex dialogs, such as persuasion. The model generates dialog responses that successfully persuade people to donate to a charity, suggesting the potential of ARDM being used in scde-scale real-world settings.
2
RELATED WORK
Traditional dialog systems consist of a dialog manager to maintain dialog states and control the conversation flow. However, dialog manager requires extensive manual annotations for training the sub-modules such as dialog state tracker or policy decision-maker. An alternative is to model dialog without explicitly modeling belief states. Specificallyy, [Eric & Manning (2017) proposed recurrent neural dialogue architecture using copy history information directly from raw dialog history. This method achieved the state-of-the- art results on DSTC2 (Henderson et al.) (014), which is a simple dialog restaurant booking task with abundant data. However, such method did not perform well on more complex dialog task data sets CamRes676 (Wen al.) 2017) and KVRET (Eric et al.) 2017). Sequicity (Lei al. (018) attributed the bad performance of Eric & Manning (2017)ss method to the omission of belief tracker. They introduced the concept of belief span and added belief tracker back to the model and achieved state-of-thee ther performance.
model that utilizes ropy-mechanism to
Compared to Sequicity, Eric & Manning 2017 's method provides more general framework that reduces manual dialog state, user intent, and dialog act labeling by bypassing any symbolic anno- tations. Such model apply datasets with or partial annotations of belief states. In real-world setting, if the dialog task introduces new slot values in belief states (i.e. a new type of food), Sequicity will suffer from the belief span decoder error in response generation. Thus, EErc & Manning (217))s 's method may be potentially more robust than Sequicity in this situation. Besides, the task requires belief states for database search, we can treat belief tracking as separate task. We can train good belief tracking with only a small amount of annotated data, which reduces the annotation required and is easier to fix errors. Also, since belief states are a set of important entities condensed from dialog history (i.e., often exact words from utterances), they do not introduce extra information to the model. Therefore, a dialog model with powerful representation learning should learn form of belief states information automatically without human annotations as the scaffold.
Recent success BERT (Devlin et al. 2019) and GPT2 (Radford et al. 2019) suggests the pos- sibility of using large pre- pre-trained language models to enhance Eric & Manning (201))ss 's method. There are some studies of applying large pre-trained language model to dialog generation. Transfer- Transfo (Wolf al. 2019) fine-tuned the pre-trained language model GPT (Radford et al., 2018) on Persona-Chal dataset (Zhang et al. 2018) and obtained significant improvements on chitchat dialog generation, suggesting the potential of fine-tuning large pre-trained language model on other dialog response generation tasks. A more recent work (Budzianowski & Vulic, 2019) adopted the frame- work of TransferTransfo and made the first attempt to leverage large pre-trained language models GPT and GPT-2 on cris-triented dialog generation, but it included belief states modeling as the input and did not achieve better results than the baseline. We propose to model dialogs without any annotation but rely on pre-trained large scale language models that alternate.
Previous work shows that modeling speaker roles in conversation beneficial for language under- standing (Chi et al., 2017] Chen et al., 2017, Su et al.) 2018). Other researchers model persona information to generate language with different speaking styles (Li et al,, 2016 Joshi et al. 2017). Zhao & Kawahara (2199) propose relative speaker modeling method, where only the relative role instead of the absolute identity of the speaker is modeled. Our method is similar to Zhao Kawahara 2019) in the spirit of modeling relative speaker relationship, but focus learning role- opeecipee language models through utterances from different speakers, instead of explicitly taking role em-
beddings input.
2

---

Under review as a conference paper at ICLR 2020
Dialog
ARDM
Memory Recurrence
French resizuranti
chinese food
Pg. ireme sloll is sap restaurant serving
Time
A
chinese food.
User LM
System LM
My: address
number
Ry The address (oddress slod and the phono
number (chure, slod).
Kg
Nu You sre welcome Have good dayy
(a)
(b)
Figure I: Alternating Roles Dialog Model (ARDM) Overview. (a) shows how feed the entire dialog to ARDM. (b) shows the recurrence mechanism we used to preserve memory.
APPROACH
Our goal is to leverage large pre-trained language models to improve dialog response generation. Favoring Eric & Manning (2017)'s approach without using additional annotations such dialog states or dialog acts, we propose Alternating Roles Dialog Model (ARDM) by compositing two separate pre-trained language model in alternate order learn the user and system utterance distribution. Figure I shows an overview of ARDM.
3.1 ALTERNATING ROLES DIALOG MODEL
We aim model both user and system utterances distribution simultaneously. Given multi- turn dialog (d) between a user (u) and a system (s), we can represent d as a series of utterances {u, 81, 02, .2..... WT, 8T). where T denotes the total number of turns. We decompose the proba- bility distributions over the utterances in into two language models for the user and system respec- tively, denoted Pu and p.- Then we define dialog model p(d) with the equation:
p(d) II Pa (ullures
Pa and are standard language models where the task is to predict the next token given the pre- ceding context. For utterance u, or 8t with m tokens {wi,.... W. the joint probability of an utterance is as follows:
Muc
Pa Pu(uuluets8es))
II 2=1
P(ws|wei, thee, 8ct)
(2)
may
( 82t)
II Pww:/ee Uge. 821)
(3)
Finally, we train the dialog model by maximizing the likelihood over Equation 1.
We apply simple memory mechanism to grant the model the capability of memorizing conversa- tion history. For an utterance at turn f, we reuse the hidden states h stored in the memory M, to obtain hú, and store the ht back to the memory as M,. As for the pre-trained Transformer lan- guage model, we implement the memory mechanism using self-attention given the query/key/value features denoted as Q, K, V, where the equation defined as:
Attention (Q, K, V) softmax (QKTT)
(4)
For simplicity, assume there is only one layer in Transformer, and h, is the hidden states which consist of vectors for the current input tokens in the utterance at time t. Then a recurrence relation for hy is defined by computing Qt. Kgg, Vse from h and the current utterance. In

---

Under review as a conference paper at ICLR 2020
practice, reuse Kg-- and V=i-1 (i.e. history keys and values) M,1 instead of 4 avoid recomputing history information. Therefore, the final h2 is computed as:
M-- [Kgr- Vse-1]
(5)
Kgo Vgi [Kg-1:: Ke). VVs-- Vi]
(6)
h4 Attention Q₁,
(7)
One can use he (consisting of vectors for each token) get each token's probability to calculate the language model cross entropy loss to maximize p(w; / i, Uctt, 8et)), shown in Figure
However, one major drawback of this memory mechanism is that the memory consumption grows as the number of turns increases, until a point that the dialog cannot continue because of the memory limit. straightforward way solve this to discard the distant history. But because most dialogs lengths in our datasets can fit in the GPU memory limit (i.e., approx. 1,000 tokens for 11GB GPU), we leave the memory issue for future work.
3.2 TRAINING DETAILS
We initialize the user and the system language model with large pre-trained language model GPT-2 small with 117M parameters (Radford et al. 2019). It is a Transformer (Vaswani et al. 2017) model with 12 heads, 768 hidden size, and 12 layers. The model is trained large scale corpus called WebText extracted from Reddit with at least three upvotes. The tokenizer is 50,257 size byte pair encoding (BPE) (Semnrich et al.. 2016) that can encode and decode any text in a lossless manner to avoid out-of-vocabulary tokens. We follow a special format in GPT-2 as the "trigger" so that the model can zero-shot dialog response generation, by prefixing the user role token "A:" or "B:", and suffixing the end of utterance token ",n)n',n". This "trigger" approach is similar in other zero-shot scenarios mentioned in GPT-2 paper (e.g., that a "TL:DR" token can trigger GPT-2 to summarize the input text.) We further fine-tune ARDM on the specific task dataset. We apply A optimizer (Loshchilov & Hutter 2019), and the number of warm-up steps set to the number of batches in one epoch. The learning rate is set to 3 10 and the dropout rate is set to 0.1 for all tasks.
3.3 DECODING DETAILS
We decode utterances by nucleus sampling (Holtzman et al. 2019) with different (top-p, top-k) for down-stream dialog tasks. We also vary the temperature of T find the best setting for the specific down-stream dialog task. To handle both situations in the evaluation and the real-world use case, have two decoding modes. For evaluation mode, feed all past ground truth history before turn / to generate the corresponding utterance, so that we can evaluate the quality of generated dialog responses without concerning about the conversion flow. While in real-world use case, we do not have ground truth history. and therefore we use the memory from previously generated responses and let the model dynamically interact with human or another bot in turns. Because dialogs have different lengths, it is hard for ARDM to efficiently decode responses using traditional batch padding method. As solution, we develop a dynamic dialog filtering algorithm to support fast decoding in batch. Such method speeds up the generation eight times faster. Please refer to Appendix B for the method's details.
4 EXPERIMENTS AND RESULTS
Data scarcity is one of the biggest challenges in dialog research. is costly collect humman conversations under specific setting. It is even more tine-consuming to annotate belief states and dialog acts. With the success of transfer learning in NLP, we aim to mitigate the low-resource problem with the large pre-trained language model. We validate our proposed ARDM on three task-oriented dialog datasets, CamRest676, MulitWOZ, and PersuasionForGood..
4.1 CAMREST676
CamRest676 is relatively small dataset with 408/136/136 dialogs for train/validation/tss.. We follow Sequicity (Lei et al. 2018) to delexicalize tokens such as restaurant names, phone numbers, postcodes by replacing them with their slot names in utterances. We prepend database search results
4

---

Under review as a conference paper at ICLR 2020
to the system utterance. An example database search results "restaurant:3", where the first slot indicates its dialog domain, which is always "restaurant" in CamRest767, and the second slot represents the number of matched items in the database. We use nucleus sampling for all methods in decoding for a fair comparison. Here, we set top-p 0.2 and temperature 0.7 for our model. We use BLEU4 and Success F1 to evaluate language generation quality and Success F1 evaluate task success. Success FI computes the Fl score of the generated responses on requested slots such as an address, phone number, or food type. Other than Sequicity, we also compare results by using GPT-2 alone language model for the entire dialog.
4.1.1 RESULTS
We first test our method on a restaurant search dataset, CamRest676 (Wen et al. 2017).
Ground Truth Belief State
Generated Belief State BL.U-4
Model
Entity Match rate
BI.U--4 Success. F1
Success. F1
Regular Expression
0.960
Sequicity
0.923
21.4
0.852
21.4
0.853
Sequicity (w/o RI)
0.940
22.9
0.821
23.4
0.834
GPI-2-finetune
21.8
0.851
19.2
0.862
ARDM
26.0
0.875
25.2
0.871
ARDM (50% data)
25.9
0.859
23.4
0.851
Table I: Results on CamRest676 dataset.
Table T shows all models' results with ground truth belief state or generated belief state. We first use ground truth belief state in all methods to evaluate their response generation quality. ARDM achieves the best BLEU and Success F1 score. We observe that after fine-tuning GPT-2 on the CamRest676, it achieves similar results compared the previous state-of-the-ard method, Sequicity with reinforcement fine-tuning. This suggests pre-trained large-scale language model, such as GPT- 2, transfers the meaningful representations to help fine-tuning. However, without the alternating mechanism, GPT-2 alone does not perform as well as ARDM in terms of both BLEU-4 and Success F1, especially in BLEU-4 (improved 19%). Without modeling the speaker role, the model blends two speakers language distribution and ignores the inherent speaker role difference. Moreover, to test if our model preserves its performance with even less training data, we reduce the training data to 50%, and the performance only drops slightly. With half of the training data, our method still performs significantly better than Sequicity. This result suggests ARDM is robust on low-resource settings due to the advantage of the large-scale pre- pre-training language model.
We also evaluate all models with generated belief states instead ground truth belief states. Se- quicity generates belief tracker results, and its Entity Match rate is 0.927. Our model does not have a state tracker, so we write a separate simple regular expression to extract the occurrence of entities that appear in the database to support our model. Such state tracker achieves 0.960 in Entity Match rate. It suggests that state tracking may be accomplished in more straightforward ways other than training neural network model on large set of annotated data. With simple state tracker, our proposed method still performs better than Sequicity, which trains the belief state and the response generation task jointly.
4.2 MULTIWOZ.
Here, we only use the ground truth database search result to be consistent with other methods. We perform delexicalization which is mentioned in the original MultiWOZ. (Budzianowski al., 2018). We prepend the database search results to the system response for conditional input. Also, the database results now contain information about whether the booking is successful or not (i.e., suc- ceed or fail). Note that we do not use belief state or dialog act annotation provided by the dataset to train ARDM. We set the top-p to 0.2 and the temperature to 0.7. The results are evaluated on BLEU- 4, Inform Rate, and Success Rate. Inform and Success Rate measure whether the system response provides the recommendation. and requested information given in the goal. We compare our model to the cecntion-based seq2seq model which is proposed as the MultiWOZ Baseline (Budzianowski
5

---

Under review as a conference paper at ICLR 2020
et al. 2018), the HDSA (Chen et al. 2019) model that incorporates dialog act supervision in- ductive prior for model architecture, and the LaRL (Zhao et al. 2019) model which leverages latent action modeling and reinforcement learning improve performance. We do not compare with GPT- 2-finetune with our model in MultiWOZZ because GPT-2-finetune's performance on CamRest676 is significantly worse than model. We normalize the time's slot value in all dialogs into the 24- hour format and perform tokenization via spaCy]] We found that different papers report results with different versions of the evaluator, which makes it difficult to compare different methods fairly. We explain the differences among all versions of the evaluator in Appendix A In this paper, we follow LaRL's evaluator implementation, as is more reasonable than others. We re-evaluate results for all methods with the same evaluator to ensure fairness.
4.2.1
RESULTS
Supervision
Model
Inform (%)
Success (%)
BLEU-4
Dialog State
Dialog Act
Human
98.9
96.5
Baseline
82.5
72.9
18.9
HDSA
V
87.7
73.4
23.6
LaRL
82.8
79.2
12.8
ARDM
87.4
72.8
20.6
Table 2: Results on MultiWOZ. Supervision denotes whether model leverages dialog state or/and
dialog act annotations. All models use the ground truth dialog state for database search. ARDM without supervision from annotation can still achieve comparable results.
The evaluation results are shown in Table 2] Without any supervision from dialog states or dialog acts, ARDM significantly outperforms the MultWOZ Baseline and LaRL on BLEU-4 and Inform rate, and is on par with HDSA. However, HDSA uses dialog act supervision and large pretrained language model, BERT. Our model requires no annotation and can achieve similar results. This suggests our speaker role modeling and large-scale pre-training methods work similarly as the use- ful dialog act annotations. All the results show that our method's 's excellent performance remains consistent in multi-domain dialogs.
We analyze the generated responses and find that if multiple domains have appeared in the conver- sation history, our model tends to make mistakes in answering the right domain for user requests. This finding suggests that the Maximum Likelihood Estimation (MLE) has limitations in directly optimizing the metric, while reinforcement Learning (RL) can hugely improve the task completion a dialog system. This is why LaRL has a higher Success rate. However, we also observe that LaRL has low BLEU-4 score, which indicates low readability in responses. Therefore, there is trade-off between the generation quality and the task success rate in the RL setting.
4.3 PERSUASTONFORGooM
To showcase ARDM's performance on dialog dataset where is much more difficult to obtain belief states and dialog annotations, we train and evaluate model on PersuasionForGood [Wang et al. 2019) dataset. In this dataset, the persuader must persuade an assigned persuadee (i.c., person who asked donate) to donate money (from their task payment) "Save the Children". This dataset has much larger vocabulary size (8,141) than the previous task- oriented dialog datasets due its col-collaborative dialog property. The conversation content richer because two speakers are negotiating back and forth. The dataset consists of 1,017 dialogs where only 300 dialogs are annotated with dialog acts. Therefore, models that require dialog state or dialog act annotation are not applicable in this dataset. ARDM does not require dialog acts for training. As ARDM has already learned good language priors through pre-training, which makes applicable on PersuasionForGood Due to the similarity to chit-chat,
charity called
TransferTransfo (Wolf et al. 2019) model as a strong baseline model. Transfer TranferTrransf is also based on large pre-trained
'htpss//spacy.i////

---

Under review as a conference paper at ICLR 2020
language model, but it uses token type embedding to encode role information of the speaker. We concatenate all history text together as input to generate the next utterance.
To generate diverse responses, decode the response using the nucleus sampling (Holtzman al, 2019) with a top-p of 0.9 and a temperature of 0.7. It is impossible to conduct an automatic eval- uation on task success on this task due to the lack of annotation. We use perplexity, BLEU-I, and BLEU-2 to evaluate each model's 's language generation quality. We also conduct a human evaluation validate each model's task success rate. We show some generated examples in the Appendix to provide more information on both models' generation quality.
4.3.1
RESULTS
Perplexity
BLEU-1 f
BLEU-21
Human Preference f
Average Donation Amount
TransferTransfo
19.9
17.0
8.57
34.7%
0.538
ARDM
10.1
16.5
6.44
65.3%
0.807
Table 3: Automatic Evaluation and Human Evaluation Results
Role
Utterance
SYS
Hi, how are you doing today?
USR
am good!
SYS
Have you ever heard of the charity Save the Children?
USR
No. have not heard about it. Would you like to talk about it?
Save the Children is international non governmental organization that promotes children's 's rights, provides relief and helps support children in developing countries.
SYS
USR
That's cool. is an organization Yes, is. In the first two months of 2018 alone, 1,000 children were reportedly killed or injured in intensifying violence in Syria.
good of this world.
SYS
USR
That's sad.
We do need to do something about that. One of the things that we can do is make sure that we have system in place that will ensure that every penny spend goes directly to the children in need. agree.
SYS
USR
How much do you like donate to the charity now? Your donation will directly deducted from your task payment. You can choose any amount from S0 to all your payment (S2).
SYS
USR
I am willing to donate full my payment $2.
SYS
Wow, that is awesome! That is so kind of you!
USR
You are welcome.
Table 4: A conversation between a machine persuader trained by ARDM and a human persuadee.
Table 3 shows the results for PersuasionForCiood. Because ARDM applies better speaker modeling and recurrence mechanism, our model achieves lower perplexity compared to TansferTranssf In terms of BLEU scores, TransferTransfo is better than ARDM. However, BLEU-1 cannot reflect the actual generation quality because a random sentence with common tokens the, of, is, are already has 10.0+ BLEU-1 score. Also because the validation set only contains 100 samples, the result can have a high variance. To comprehensively evaluate each model's performance, we recruit 14 human evaluators chat with the two persuasive systems ten times to avoid the randomness produced by each model. In total, we collected 140 ratings. We ask them to select a preferred chat-bot and indicate how much they are willing to donate after talking to the chat-bot. As result, human judges prefer ARDM over TransferTransfo and tends to donate more when talking to ARDM produced chat-bot. Our model achieved 27% more donations compared to TransferTransh that our systems are more persuasive. In some examples, such as the one in Table 4 our model generates coherent, natural, and persuasive responses.
This indicates
5 ERROR ANALYSIS
Since CamRest676 is similar to MultiWOZ in terms of task content and dialog structure, we only describe the errors in MultiWOZ for simplicity. We randomly selected 30 generated error responses
7

---

Under review as a conference paper at ICLR 2020
from model with zero inform and success score. To our surprise, we observed that nearly 63.3% of errors are not really mistakes. is mainly due to the limitation of the automatic evaluator. For example, at turn one, the user asks about restaurant, and the ground truth system response is "the [restaurant name] is located at... at...", ", but the generated system response is "what food preference do you have?". Our generated response is correct with respect to the dialog context. It is narrowing down the restaurant choices before providing a restaurant recommendation. However, the evaluator sticks to the only possible response it has. Unless the user can dynamically interact with the system, there is no good way to change such mistakes in the automatic evaluator. We find that another 20% errors our model makes are when the system asks information the user already provided. This type of errors calls for better history representation. Another 10% errors are due to ignoring the user's request for information, such as phone number. However, when we look at the ground truth responses, some crowd workers also made such errors. So resolving these errors requires cleaner training dataset. Finally, the rest of 6.7% errors are about incorrect dialog domain understanding. For example, the user is asking for a hotel, but present a restaurant recommendation. This is because of the data noise during the delexicalization process in which some domain labels are wrong.
The donation persuasion system trained with TransferTransfo and our model has some common problems, such as inconsistency, lack of logic, and hallucination. For example, if the persuader provides the information about "Save the Children", then the persuadee asks "Can you tell me more about it?". The system ends up providing the same information as before. It also sometimes makes up facts that have never happened, such as "Save the Children has an operation about hurricane in Hawaii". All those errors would prevent users from trusting the bot, and therefore resulting in less donation. However, we also observe that users have higher tolerance for errors in the persuasion setting than the customer service setting. Overall, our model performs better on PersuasionForGood by having longer and diversified utterances. This suggests our model which utilizes alternating parameters for different speaker roles is more effective than TransferTransfo which only injects role information into the input embedding.
DISCUSSIONS AND ETHICAL CONSIDERATION
ARDM models speakers separately on top of a large pre-trained language model. Such simple adaptation demonstrates substantial performance gain. We suspect it is because the interleaved structure of two language models provides a collaborative learning frame of both the user and the system language distribution modeling. The memory is the only way for the user and system to communicate, as they do not share any weights in their networks. Thus, the user encoder needs to leam useful representations to make the system model for understanding its intent. Similarly, the system needs to do the same for the user model to improve its understanding. This alternative repeating process forces both the user and system models to preserve the dialog history effectively in the memory. One can interpret the memory as the implicit representation of belief states or dialog acts.
Another benefit of ARDM is that we will obtain both user and system utterance generators. We can let the two models talk each other to generate new self-play dialogs (Silvrr et al.) (017). We show some self-play dialog examples in the Appendix E.] With self- play. one can rapidly build large scale dialog dataset using adversarial filtering (Zellers al. 2018). Such models can be used in reinforcement learning as user simulator to study complex dialog strategies as well.
Persuasion is double-edged sword. Given the fast development of dialog systems, an ethical design principle must be in place throughout all stages of the development and evaluation. We choose the donation task is because it is a relatively simple task that benefits children. Second, when deploying the persuasive agents in real conversations, we need to keep the users informed of the nature of the system. By revealing the identity of the persuasive agent, the user should also have options to communicate directly with the human team behind the system. Lastly, by investigating persuasive dialog systems, we also envision to use them as an educational tool for the general public to learn to defend themselves against machine persuasion.
8

---

Under review as a conference paper at ICLR 2020
7 CONCLUSIONS
We propose to build Alternating Roles Dialog Model (ARDM), a simple, general, and effective di- alog method that models user and system separately with large-scale pre-trained language models. Since ARDM does not require any annotations, it generalizes to different dialog applications. Ex- perimental results CamRest676 and MultiWOZ suggest that ARDM outperforms on-par with the current state-of-the-art methods that use manual annotation information, such as belief states and dialog acts. Furthermore, find our model's excellent performance generalizes more complex non-collaborative dialog settings. It can generate high-quality responses to persuade people to do- nate to charity. However, the easiness of training ARDM raises concerns about the misuse of the model in scenarios such as sales, harassment, or scam on a mass scale. We caution the public in deploying such systems in the real world.
REFERENCES
Pawel Budzianowski and Ivan Vulic. Hello, it's GPT-2 how can help you? towards the of pretrained language models for task-oriented dialogue systems. CoRR, abs/1907.05774 2019.
http:://axxvvv.ovo abs/ 05774
Pawel Budzianowski, Buing-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Riloff et al. (2018), pp. 5016-5026. ISBN 978-1-948087-
84-1. URL [att::: wwwweb. org/ wwt.oo//g/anogg///oooooooooo
1547/
Po-Chun Chen, Ta-Chung Chi, Shang-Yu Yu Su, and Yun-Nung Chen. Dynamic time-aware attention to speaker roles and contexts for spoken language understanding. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 560. IEEE, 2017.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. Semantically condi- tioned dialog response generation via hierarchical disentangled self-attention. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3696-3709, Association for Computational Linguistics, 2019. ISBN 978-1- 950737-48-2. URL BtRLhttts:/// //ww: ac wwwb.webb... org/ http:/nnhoooogyyo 13667/
Ta-Chume Chung Chi, Po-Chun Chen, Shang-Ya Yu Su, and Yun-Nung Chen. Speaker role contextual model- ing for language understanding and dialogue policy learning, arXiv preprint arXiv:1710.0016: 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171 4171-4186, Minneapolis, Minnesota, org/
June 2019. Association for Computational Linguistics. URL anthology/NI9- 1423
Mihail Eric and Christopher D. Manning. A copy-augmented sequence-to-sequeeec architecture gives good performance on task-oriented dialogue. CoRR, abs/1701.04024 2017. URL http:
7/arxivv. owt.vvorg/aae hbt///1s/111111 04024
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. Key-value retrieval networks for taik-triented dialogue. In Kristiina Jokinen, Manfred Stede, David Vault,, and An- nie Louis (eds.), Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, August 15-17, 2017, pp. 37-49. Association for Computational Linguis- tics, 2017. ISBN 978-1-945626-82-11 URI. https::/p.............ttt...s..t /wwww ac web. org/ httgg//nthhoooooooooo hoyo W17 R17-55067
Matthew Henderson, Blaise Thomson, and Jason D. Williams. The second dialog state tracking challenge. In Proceedings of the SIGDIAL 2014 Conference, The 15th Annual Meeting of the Special Interest Group Discourse and Dialogue, 18-20 June 2014, Philadelphia. PA, USA, pp. 263 272. The Association for Computer Linguistics, 2014. ISBN 778-1-941643-21-1 URL https: www. ac web. www.awwwwwww...//.eeeeeeeeeeeeee www.trttg/nnooooooooooooooooooo. ogy 443377

---

Under review as a conference paper at ICLR 2020
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. CoRR, abs/1904.0077/1 2019.
09751
Chaitanya Joshi, Fei Mi, and Boi Faltings. Personalization in goal-oriented dialog. arXiv preprint wrXi::1706.07503, 2017.
Wenqiang Lei, Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequeece architectures. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1437-1447, Melbourne, Australia, July 2018, Association for Computational antttooohoooyyo
Linguistics. URL ht tattpps:/ /wwww acc
1133
Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan, A persona-based neural conversation model. arXiv preprint arXiv: /55, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, OpenAI, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, 2018. Association for Computational Linguisties. ISBN 1-948877 84-1. URL https:/ / www. ac web. www.oorg//aaaaaa 1 ooyy /vol umes /D1me17/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Compu-
tational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
The Association for Computer Linguistics, 2016. ISBN 978-1-945626-00-5 URL https:
7/ /www. ac web. ont.holoogy/I16111622
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. CoRR, abs/1712.01815, 2017. URL http://////// http///axxvv../ org/ abs/ 1712. 01815|
Shang- Yu Su, Pei-Chieh Yuan, and Yun-Nung Chen. How time matters: Learning time-decay at- tention for contextual spoken language understanding in dialogues. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 2133-2142, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need. CORR, abs/1706.03762. 2017.
URL httthttt//axixii org/ hbtp///s.//1/./.// 03762
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. CoRR, htss/190.06772, 2019. UtLhht::://arat org/ abs/ / 1 9066606. 06725
Tsung- Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve J. Young. Latent intention dialogue mod- cls. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 3732-3741. PMLR, 2017. URL http: http::///cceeeinn..mmmm .mlr. www.:press/vvee httl///0//een// 7a. html
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, 0tbs//1911.00 149,999,, 2019.
URL URL httttt/tp///axt org a 1901 08149
10

---

Under review as a conference paper at ICLR 2020
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Riloff et al. (2018), pp. 93-104, ISBN 978-1- 948087-84-1. URLktts:: wwwwwb. orz www.tantoooooaoaaoaaaaogoogooe
1009/
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Lingnistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, PP- 2204-221 2213. Association for Computational Linguistics, 2018. ISBN 978-1-949088-32-22 doi: N0.18653/v1/P188 1205. htR.http::// /ww. ac web.orgg/amcchooccr//////// 1205/
Tiancheng Zhao, Kaige Xie, and Maxine Eskénazi. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 1208-1218. Association for Computational Linguistics, 2019. ISBN 978-1-950737-13-0, URL https:
7/www. aww.bb.wwbbb..... org/ www.o//ggtoooooo/aooooooo /N19 1123/
Tianyu Zhao and Tatsuya Kawahara. Effective incorporation of speaker information in utterance encoding in dialog, arXiv preprint arXiv: /rXiv:/90.005559 2019.

---

Under review as a conference paper at ICLR 2020
A MULTIWOZ EVALUATOR INCONSISTENCY
We rerun baseline models to compare our methods and find discrepancy among different papers' reported results. In order understand the reason, we compared between LaRL's evaluator and MultiWOZ Baseline's evaluator 3 We found that they make different assumptions to handle the "train" domain (line 637-639 at LaRL evaluator.py). After carefully analyzing the code and dis- cussing with authors of these two papers, we believe that LaRL's evaluator is more reasonable. However, in LaRL, the authors reported MultiWOZ. Baseline's s scores with a different evaluator: Therefore, re-evaluated all methods, including LaRI, HDSA, and MultiWOZ Baseline using the same evaluator for fairness.
Baseline Evaluator Inform
LaRL Evaluator Inform
Success
Success
Human
75.7%
97.9%
90.0%
82.3%
Human (the cleaned version)
82.4%
78.9%
98.9%
96.5%
MultiWOZ Baseline
71.3%
61.0%
82.5%
72.9%
Table 5: Re-evaluation Results MultiWOZ.
B DYNAMIC DIALOG FILTERING ALGORITHM
Since ARDM takes the entire dialog as input, so different input has different number of turns and different length in each turn. Also, there are two sub-language models in ARDM. If we pad the utterance in each turn and concatenate results of all turns, it is inefficient. Instead, we introduce a filtering algorithm to dynamically release memory once a dialog finished. This enables us to speed up our model four times in training and eight times in inference. Please refer to our code for details.
Algorithm Dialog Filtering algorithm
Batch Size n
Initialize Memory M for workers. for Batch 1, 2, 3... do
T maximum number of turns in Batch
for Turn = 1, 2, ....., T Batch do
Filter M and Batch for the finished worker
Feed remaining Batch to the model Compute loss and gradients
0aa f- 8
end for
end for
hanerngghhhh..oaaa.cmcm.maallaaa dialogevvannttorssspp
12

---

Under review as a conference paper at ICLR 2020
C TRANSFERTRANSFO EXAMPLES
Role
Utterance
ER
hello!
EE
Hello
ER
have you heard of the charity save the children?
EE
no, haven't it heard
it's an international non governmental organization that promotes children's rights, provides relief and helps support children in developing countries.
ER
EE
ok
ER
so would you like to donate some of your payment to this charity? can you me more information about it?
ER
save the children is an international non governmental organization that promotes children's rights, provides relief and helps support children developing countries.
EE
which developing countries?
ER
the united states and latin america. have you heard of this charity?
EE
no, I haven't
ER
it's an organization that helps ensure children's rights to health, education and safety. sounds cool
EE
ER
have you ever donated to this organization? no
EE
Table 6: A conversation between machine persuader (ER) trained by TransferTrran and human persuadee (EE)
Role
Utterance
ER
hello!
EE
hi
ER
how are you today?
EE
am good
ER
have you ever heard of the children?
EE
no, I haven't
ER
is organization that helps children around the world. it's great organization that does good work and is highly rated with many positive rewards. good
ER
have you ever heard of it?
EE
no, haven't 't heard of it
ER
it's great organization that does good work and is huge part of the world's popula- tion.
EE
okay
ER
so, are you interested in donating?
EE
yes, I am
ER
how much would you like to donate to the charity?
EE
$1
ER
wow, that is really great. that is really great to hear. so, are you going to donate anything at this time??
Table A conversation between machine persuader (ER) trained by TransferTransfo and human persuadee (EE)
D PERSUASIONFORGOOD HUMAN-MACHING EXAMPLES
We show examples of dialogs generated using our proposed ARDM.
13

---

Under review as a conference paper at ICLR 2020
Role
Utterance
ER
Hello! How you doing today? am doing well!
EE
ER
Awesome. Have you heard about the Save the Children charity?
EE
Yes have heard about it.
Save the Children an international non-go oongovvrnmeetal organization that promotes children's rights, provides relief and helps support children in developing countries. So which developing countries do they help?
ER
EE
ER
Save the Children is working supporting children in the US and all over the world. that's cool. want to help children as well.
ER
That's s awesome! I know there are lot of kids in the world that are in dire situations. Right. What should do to help them?
Save the Children is great organization, you should donate some of your payment to them.
ER
EE
how much is my task payment?
am not sure exactly how much you will donate, but you can donate any amount from $0 to all of your payment ($2).
ER
EE
would like to donate $1.00.
ER
That's awesome. Your donation will make huge impact for those kids. Thank you.
EE
Table 8: A conversation between a machine persuader (ER) trained by ARDM and a human per- suadee (EE)
Role
Utterance
Hi, how are you doing today?
EE
am good!
ER
Have you ever heard of the charity Save the Children?
EE
No. I have not heard about it. Would you like to talk about it?
Save the Children an international non- non-governmental organization that promotes children's rights, provides relief and helps support children in developing countries. That's 's cool. It a organization for the good this world. Yes, it is. In the first two months of 2018 alone, 1,000 children reportedly killed or injured in intensifying violence in Syria. That's ' sad.
ER
EE
ER
EE
We do need to do something about that. One of the things that we can do is make sure that we have a system place that will ensure that every penny we spend goes directly to the children in need. agree.
EE
How much do you like to donate to the charity now? Your donation will be directly deducted from your task payment. You can choose any amount from $0 to all your payment ($2).
EE
am willing to donate full my payment $2. Wow, that is awesome! That is kind of you!
ER
EE
You are welcome.
Table 9; A conversation between a machine persuader (ER) trained by ARDM and a human per- suadee (EE)
14

---

Under review as a conference paper at ICLR 2020
Role
Utterance
ER
Hello
EE
Hi
ER
How you doing today? am good. thank you
EE
was wondering if you would be interested in donating portion of your payment from this HIT to a children's 's charity called Save the Children?
ER
Of course, would like to. Can I know their website address? Yes, URL
ER
Oh, cool. They are good organization.
ER
How much would you like to donate?
would like to donate $1.00
That is great! The research team will collect all donations and send it to Save the Children.
ER
EE
You welcome.
Table 10: A conversation between machine persuader (ER) trained by ARDM and human per- suadee (EE)
E PERSUASIONFFRRGOO SELF-PLAY EXAMPLES
We also present several self-play examples generated using our proposed ARDM.
Role
Ufterance
ER
Hello! How you doing today?
EE
Good, thank you. How are you?
ER
am good, thank you. Have you heard of the charity Save the Children? Yes, have heard about would like to donate a dollar to it.
EE
That is wonderful, it is important to provide education and support the children. am sure you will agree that this is a great cause. Do you donate to charities? Yes, do donate several charities.
ER
EE
Are you sure you want to donate dollar to Save the Children? Children all over the world are suffering due to poverty, war, and other issues. Would you consider donating little bit of your incoming task payment?
EE
may donate a dollar. don't 't have lot of money right now but would like to help. That is great to hear. I know your donation is small amount but the impact this will have on world can significant. Your donation would be directly deducted from your task payment.
ER
That is true. will donate one dollar to the charity. Thank you much for your donation, it will help many children. Thank you. hope you will continue to donate to the charity.
ER
EE
Table 11: A conversation between a machine persuader (ER) and machine persuadee (EE) trained by ARDM in self-play setting
15

---

Under review as a conference paper at ICLR 2020
Role
U'terrnce
ER
Hi there!
EE
Hi, I'm Carolyn. How are you today?
good, thank you. am also here talk about charity called Save the Children. Do you have any idea about that?
ER
EE
think it's something that should know about, but don't 't have the funds.
ER
understand that, know that for sure.
am glad to hear that. How does help children in the USA?
It is really great. I am here to talk to you about something that is really important to me. I am here tell you about something that am really passionate about. think it's very important issue. don't t want to donate to charities that I know not helping children. Yes, it is.
ER
EE
ER
EE
Do you donate regularly to others?
ER
have yet, not
EE
feel a little guilty for not donating when think about the children have lost. Yeah, totally understand.
ER
would like to donate to a charity that I feel would be able to help all the children in the world.
ER
Awesome! That is great. How much would you like to donate to Save the Children? would like to donate $1.00 Great! That is awesome. am happy to hear that. have heard about Save the Children, and it something that I really, truly need.
EE
ER
EE
ER
That is so great you.
EE
Thank you for the opportunity.
Table 12: A conversation between machine persuader (ER) and machine persuadee (EE) trained by ARDM in self-play setting
Role
Utterance
hello how are you? I'm doing well, how are you?
EE
I'm doing well, thanks for asking. I was wondering you would be willing to donate portion of your task payment to save the children.
ER
I'm interested, how much will you donate?
would like to donate the entire amount, but I'm not how much I'd be able to donate. Do you have any other questions?
ER
EE
think know the amount would donate.
completely understand, am hoping you would be willing to donate even a small amount. Save the children an amazing charity that helps children in need around the world.
ER
EE
How much do you think can donate? would like to donate minimum of $2. I was wondering if you would be willing to donate any more, would you be willing to donate even little bit of your task payment?
ER
EE
think can donate $1.
That's s very kind you. would be happy to do that, but I can't think of any other charities I would be able to donate to. am glad to hear that.
ER
EE
think you are very generous, and I am sure this will help in the future.
ER
It is wonderful hear. Thank you for your donation.
EE
You are very welcome, and thank you for the opportunity to help.
Table 13: A conversation between a machine persuader (ER) and a machine persuadee (EE) trained by ARDM in self-play setting
16

---

Under review as a conference paper at ICLR 2020
Role
Utterance
ER
Hi, how you today?
EE
I'm doing well! How are you?
ER
doing well as well, hope you have nice day. am as well.
EE
ER
Do you have children?
EE
Yes, do. have three boys and one girl.
Have you ever donated to a charity?
EE
No. don't.
Have you heard of Save the Children? have not.
EE
The charity is an international non-govermmenall organization that promotes chil- dren's 's rights, provides relief and helps support children in developing countries. think they are good organization.
ER
EE
ER
sure they would be happy to help.
EE
think they would.
ER
Do you donate charities? I do.
EE
How much do you like donate the charity now? Your donation will be directly deducted from your task payment. You can choose any amount from $0 to all your payment ($2).
ER
EE
would like to donate $0.50.
That very generous you.
EE
hope they able to help lot of children.
Table 14: A conversation between a machine persuader (ER) and machine persuadee (EE) trained by ARDM in self-play setting
17


Reviews:
[{"review_id": "HJgL0TmFKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is concerned with reducing the amount of manual dialog state or dialog act labeling in task-oriented dialog, following the lead of Eric and Manning (2017) who did not require any such explicit annotation. The authors note that Eric and Manning’s approach didn’t do well on more recent dialog datasets in comparison to e.g. Sequicity (Lei et al., 2018), which trains an explicit belief state. The first main contribution of the submission is to exploit GPT-2 to outperform Sequicitiy and other techniques, doing so without explicit dialog state/act supervision. The second contribution is an alternating parameterization of the model that distinguishes between agent and user utterances. Experimental results show that the method of the paper (ARDM) is either superior or on par with methods that exploit hand-labeled dialog states/acts.\n\nOriginality/impact:\n\nI think the approach of the paper is well motivated, though quite simple. The idea of improving on a generation task with GPT-2 is a tried one, and its effectiveness isn’t really surprising. The alternating parameterization is interesting but has been done before in (e.g.) Zhao and Kawahara (2019) and (Zhang et al., 2019) [1]. Overall, I think the paper doesn’t bring much novelty, but results are promising, and this work represents a step towards making task-oriented dialog system less reliant and hand coded linguistic information. It could have an impact for the dialog community.\n\nEmpirical contribution:\n\nMy main concern with the paper is a lack of ablation, which makes it hard to understand where the improvement comes from. More specifically:\n\n* It is not known how much the alternating nature of ARDM contributes the overall gains compared to (1) other details of the model; (2) pre-training with GPT-2. One way to ablate ARDM’s alternating mechanism from other characteristics of the model would be for example to tie the parameters of the user and the agent. \n\n* Table 1 compares GPT-2 with ARMD, so we don’t know if the improvement comes from either fine tuning on CamRest676, from ARDM’s alternation between speakers, or from other characteristics of ARDM. Note: The text characterizes GPT-2 as “a pre-trained large-scale language model GPT-2”, so I presume “GPT-2” here means without fine-tuning.\n\n* Results of Table 3 are perplexing and more analyzes would be needed to understand what is really happening. Considering that both TransferTransfo and ARDM are based on transformer and GPT-2, and are apparently fine-tuned on the same data, I find it strange the gap between the two is so big on perplexity (10.1 vs TransferTransfo’s 19.9) while TransferTransfo is actually superior on BLEU. I understand BLEU is problematic for dialog evaluation as it treats references (gold responses) as the only reasonable responses (Liu et al., 2016), but that makes perplexity as an evaluation metric problematic for exactly the same reason. In fact, (Schwenk et al., 2006)[2], (Luong et al., 2016)[3], and others have shown that BLEU and perplexity are highly correlated, which isn’t surprising as both penalize generated responses to the extent that they lexically differ from the gold standard. So, these results look a bit strange/suspicious to me, and I think deserve further investigation. Sample outputs of both TransferTransfo and ARDM might help better understand the problem (Why showing only ARDM’s? When showing generation examples, I think it is standard practice to show outputs of multiple systems, and not just for the system of the paper). \n\nMinor comments:\n\n- [Manning and Eric, 2017] should be [Eric and Manning, 2017]\n- ARDM with “recurrent” maybe isn’t a good name as it may imply it is RNN instead of Transformer based.\n- Section 3.1: I suggest you explain the “memory mechanism” more formally and in more details, e.g., M_{t-1} doesn’t seem to be used anywhere.\n- “assume there is only one layer in Transformer”: Is it for the sake of the presentation only, I assume?\n- Tables 1-3: perhaps add confidence intervals as some of the differences are quite small, and so are some of the test sets.\n- Table 3 doesn’t seem to be referenced anywhere.\n- Table 4: I would recommend showing ARDM and TransferTransfo side by side.\n\n[1]: https://arxiv.org/abs/1903.05759\n[2]: https://www.aclweb.org/anthology/P06-2093/ (e.g., Fig. 5 shows the high correlation between BLEU and perplexity)\n[3]: https://arxiv.org/abs/1410.8206\n\nQuestions:\n\n(1) In 4.3.1, are capacities of the two models the same? If ARDM has (roughly) double the number of parameters due to its modeling of two speakers, what about comparing TransferTransfo at same model capacity to make the comparison more meaningful?\n\n(2) I am confused how you (the authors) compute belief state success F1 for GPT-2, as the text says there is only pre-training with this model, which I think implies there is no fine-tuning on the belief state identification task. Could you provide more details?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bygq23JFFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores methods to incorporate a pretrained language model into a dialog system. The authors propose Alternating Recurrent Dialog Model (ARDM) where the pretrained language model is used to initialize both the user LM and the system LM. The authors also present a memory module to augment this dialog system where each memory slot contains a <key, value> pair derived from hidden states for each dialog turn.\n\nI think the writing of the model section could be improved.\nA few clarification questions for the authors:\n- How is the hidden state for the dialog turn t (i.e., h_t) derived from hidden states of tokens in turn t? Supposed there are N tokens in turn t, which Transformer hidden state is used as h_t? \n- How is h_t used to compute p(w_i | w_{<i}, u_{<t}, s_{<t})?\nIf I understand correctly, the authors use t to index dialog turns, but then they seem to use the same symbol to denote token indices in the same section. So the exact model, although appears to be simple, is very confusing to me.\n\nIn terms of experiment results, the authors show that their approach improves over the basic GPT-2 and is competitive with baseline methods that rely on more supervision.A few clarification questions regarding the experiments:\n- Did the authors tune the GPT-2 model on each dataset, similar to ARDM as well? Or are the GPT-2 results shown in Table 1 after fine-tuning?\n\nIn summary, this paper is a plug-and-play extension of the GPT-2 pretrained model that could be explained more clearly.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1g7ZMmRYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pre-trained language model architecture specifically used for task-oriented dialogue systems. The basic idea is to alternate between the likelihood of two parties in a dialogue. This simple yet effective approach yield significant improvements in baseline dialogue system datasets in terms of both BLEU score and accuracies. Experiments are done throughly by comparing to BERT and GPT-2, which reflected the cutting edge research in pre-trained language model. This paper opened up potential new domains that can inspire a wide range of work and it fits very well into the ICLR community. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an alternating dialog model based on transformers and GPT-2, that model each conversation side separately and aim to eliminate human supervision. Results on two dialog corpora are either better than or comparable to state-of-the-art. Two of the reviewers raise concerns about the novel contributions of the paper, and did not change their scores after authors' rebuttal. Furthermore, one reviewer raises concerns about the lack of detailed experiments aiming to explain where the improvements come from. Hence,  I suggest rejecting the paper.
Decision: Reject