SlowGuess/ACL_2020_OCR huggingface dataset:
The results are presented in Table 3. We see that for models with similar parameter counts the GCC-DEC has the advantage, and that under similar decoder sizes having direct access to the reference material (i.e. processing the reference and conversation together in a single decoder) results in superior performance. This indicates that the added complexity from additional encoding is not needed and that concatenating all relevant context is both the simplest, and most effective means of incorporating previous information. Since the parameters are shared and no latent variable bottleneck is used, the model has full access to the information from the references. With this, the self attention operation is able to automatically modify the model's output distribution in complex, non-linear ways without the need for manual architecture design.
Pre-training and References We will use GCC-DEC going forward. It is important to see if we can gain additional predictive power using pre-trained models trained on large, diverse, language modeling corpora, or at the very least utilize less computing resources to achieve similar performance. The GCC-DEC trained from scratch in the previous section will be compared against another model of the same size that was pre-trained using Megatron-LM (Shoeybi et al., 2019). The pre-trained GCC-DEC will be fine-tuned for 70,000 iterations at a global batch size of 128. We will also compare against GCC-NRC fine-tuned from the same checkpoint with the same batch size and amount of iterations.
The results can be seen in Table 4. We observe that with less data, the pre-trained model quickly eclipses the model trained from scratch and achieves better perplexity, highlighting the need for models with robust linguistic features learned from non-Reddit corpora. Additionally, including refer
Model	P.T.	Iter.	Batch	PPL
GCC-DEC	X	200K	256	19.10
GCC-DEC	✓	70K	128	18.92
GCC-NRC	✓	70K	128	19.39
Table 4: Comparison of models with and without pretrained initializations. All models in this study have  $355\mathrm{M}$  parameters with  $h = 1024$ ,  $l = 24$ , and  $A = 16$ .
Model	h	l	A	Params	PPL
GCC-DEC	768	12	12	117M	23.14
GCC-DEC	1024	24	16	355M	18.92
GCC-DEC	1280	36	16	774M	17.18
GCC-DEC	1536	40	16	1.2B	16.08
GCC-DEC	3072	72	24	8.3B	13.24
Table 5: Comparison of model performance as size varies. All models are trained from a pre-trained checkpoint for 70K iterations with a batch size of 128.
ence history improves performance as well. This difference of 0.47, while smaller than differences between results from different model sizes, is notable due to the large amount of out of sample data that the models were tested on.
Model Size Finally, we performed an ablation study on the size of GCC-DEC used. The different size configurations and results can be seen in Table 5. All models fine-tuned from a pre-trained checkpoint for 70,000 iterations at a global batch size of 128. As shown in Shoeybi et al. (2019), perplexity decreases as the size of the model increases. This increase in performance is significant as it has been shown that for conversational models there is a correlation between held-out perplexity measures and human-likeness of sampled turns, especially for models within the same family (Adiwardana et al., 2020).
The goal of the human evaluations is to verify the results of the quantitative ablations studies concerning both model size and presence of reference history. This is done by presenting participants on Mechanical Turk with 375 different ground truth conversations of variable lengths (2, 4, and 8 turns) in even proportions. We utilize 3 raters per example in our setting. To filter out spurious raters we explicitly detail in the instructions that payment is contingent on spending at least a certain amount of time on the samples and completing a survey about their Reddit use. If a rater fails to satisfy both these conditions we discard their label. Adopting this simple heuristic for rater quality led to the disqual-
ification of  $33.2\%$  of our labels. As is common in other work a single conversation is presented with two different realizations for the last turn (Serban et al., 2017). These last turns can be either machine-generated or ground truth depending on the experiment; however, every model generates exactly one reply for each of the ground truth to be used across all experiments. Samples where three new turns are generated can be seen in Table 1 or in Tables 7 - 13 in the Appendix.
When presented with these different realizations, the participant is asked to rate the pair on several qualities such as which is likely to be human generated, which follows the references well, which has good quality, and which exhibits good turn-to-turn coherency. For each of these the rater is asked to decide which in the pair showcases these qualities better. Note that the rater has the option of selecting both of them exhibit the quality of interest, or neither of them do. These were conducted in pairs to provide a frame of reference for the rater. We present the findings as paired results to account for grounding effects. Exact phrasings of these questions, several sample conversations, and details on our Turk setup can be found in Appendix A. We found inter-rater agreement in our studies about  $75 - 80\%$  of the time between 2 of the 3 users who judged samples, and about  $10\%$  of the time all 3 agreed unanimously. This is in light of 4 possible choices and 3 raters. It should be noted that our goal is not to make the distribution between model and human statistically different, but rather to make them as close as possible. We have taken several steps to assure the quality of our human evaluations as mentioned in the previous paragraph. Beyond that, any experiment with sufficient statistical power would need a prohibitively expensive number of samples per comparison.
The results of this study can be seen in Table 6. We find that in pairwise comparisons bigger models nearly always outperform their smaller counterpart across all tests we ran. For our pairwise tests we only considered pairings between a model and the next largest model size due to the prohibitive cost of computing all pairwise comparisons. For tests against our ground truth we found the results to be rather noisy. Generally, we observed that the models were close to  $30 - 40\%$  in all categories meaning that they were sufficiently similar to the ground truth distribution of data (the neutral options were chosen more frequently). However, we found that
Source A	Realistic	Reference	Quality	Coherency	Source B
GCC-NRC (355M)	31% - 35%	37% - 41%	29% - 36%	32% - 39%	Human
GCC-DEC (355M)	32% - 34%	38% - 40%	31% - 33%	32% - 36%	Human
GCC-DEC (774M)	31% - 35%	40% - 39%	33% - 33%	34% - 36%	Human
GCC-DEC (1.2B)	32% - 37%	40% - 40%	34% - 38%	29% - 36%	Human
GCC-DEC (8.3B)	37% - 40%	42% - 38%	42% - 42%	40% - 42%	Human
GCC-DEC (355M)	31% - 34%	41% - 39%	37% - 36%	33% - 35%	GCC-NRC (355M)
GCC-DEC (774M)	33% - 33%	39% - 40%	34% - 29%	34% - 36%	GCC-DEC (355M)
GCC-DEC (1.2B)	31% - 31%	40% - 38%	33% - 32%	38% - 38%	GCC-DEC (774M)
GCC-DEC (8.3B)	41% - 37%	39% - 43%	38% - 38%	42% - 39%	GCC-DEC (1.2B)
Table 6: Experiment results for pairwise comparisons grading if conversation samples seemed human-like (Realistic), were inline with the reference history (Reference), were interesting and had good grammar (Quality), and if they fit the conversation as a whole (Coherency). Percentages reported in the format “A% - B%” indicate how often users reported that samples from source A were better than samples from source B for a given category, and vice versa. Percentage pairs do not sum to  $100\%$  due to users being able to report both samples as being of equal standing, thus resulting in a third (omitted) percentage representing neutral opinions.
our 8.3B parameter model was significantly more polarizing than the rest. The model was capable of generating unique and engaging conversations that, when compared to the ground truth, led to it being explicitly preferred more than other models in all tests. It proved to adhere to the persona more than even the ground truth conversations. In addition to effectively utilizing its references to modulate style as we'd hoped, we also found that its realism, linguistic quality, and coherency was superb. Furthermore, we also tested pairwise comparisons between samples from successive model sizes. On average, the larger model tended to achieve similar or superior performance in all of the categories. All in all, these findings reinforce the results from the quantitative experiments in that larger models better match the target distribution.
Reference use From our qualitative study we can clearly see the benefit of using reference history as was alluded to in prior sections. In all four experiments the presence of references leads to better ground truth performance compared to GCC-NRC. In Figure 2 we delve deeper into the results of the ground truth experiments and display labeler preference as a function of conversation length. As can be seen, when the conversation has built up a lot of context, GCC-NRC (355M) moves away from the user style, instead focusing presumably on the style within the conversation. Alternatively, GCC-DEC (355M) adheres more closely to the references instead of the prior conversation context, thus resulting in higher style match for longer conversations. However, this over-adherence to the conversation style does seem to impact conversation quality for longer conversations. It is possible that our inclusion of random reference conversations leads to
Figure 2: Test scores compared against the number of dialog turns given as context prior to generating samples for GCC-DEC (355M) and GCC-NRC (355M).
this quality degradation. To investigate this future work could consider incorporating information retrieval components to select contextually relevant reference conversations for more accurate personality transfer that does not degrade conversation quality.
Transformer Language Models Radford et al. released the first widely used transformer based generative language model, GPT. Follow up work, GPT-2, showed that language modeling quality improved as model size grew, up to 1.5B parameters (Radford et al., 2019), and that large transformer language models were able to successfully incorporate long term dependencies to model and generate diverse content. Further work with generative transformer language models would go on to push model scale by testing up to 8.3B param
eters and 11B parameters in two separate studies (Shoeybi et al., 2019; Raffel et al., 2019). These results have demonstrated performance scaling not only for the original language modeling task, but also on plenty of downstream NLP tasks as well (Radford et al., 2019; Dai et al., 2019; Howard and Ruder, 2018; Liu et al., 2019; Zellers et al., 2019; Yang et al., 2019; Devlin et al., 2018). We demonstrate that this scaling trend applies to the conditional conversation modeling task as well and validate the efficacy of transformer based language models for dialog modeling.
Dialog Modeling Generative, non-goal oriented dialog modeling (i.e. chit-chat) has a history of difficulty with modeling long contexts (Serban et al., 2016b), exhibiting a consistent personality (Li et al., 2016), and producing interesting and engaging responses (Li et al., 2015). In general approaches to mitigating these issues have included: tweaking the base recurrent network architecture to introduce persona-based latent variables (that are either learned, amortized, or adversarially generated) (Serban et al., 2017; Bak and Oh, 2019; Chan et al., 2019; Olabiyi et al., 2019), learning speaker embeddings to modulate style (Li et al., 2016), and conditioning on outside information or heuristics to control generation (Young et al., 2018; Joshi et al., 2017; Ghazvininejad et al., 2018). One particular way that inconsistent personalities have been addressed is by conditioning the model on a set of sentences describing the target personality (Zhang et al., 2018; Mazaré et al., 2018). As described in the prior section large transformer models have demonstrated success in generating diverse and engaging content. Recent work in conversational modeling has built upon the success of these transformer-based architectures to allow for longer contexts and incorporating multiple turns (Wolf et al., 2019; Olabiyi and Mueller, 2019).
Several datasets have been proposed for multi-turn conversation modeling (Serban et al., 2016a; Lowe et al., 2015); however, these are limited to relatively short median conversation lengths of 3 and 6-turn respectively. Contexts of these lengths are not able to take full advantage of GPT-2 and other large transformer's modeling capabilities. Addressing this shortcoming and curating a dataset of diverse conversations that cover a wider distribution of conversation lengths from 0 to 15 turn contexts is a central goal of this work. Concurrent work has shown the value of leveraging large
amounts of Reddit data to harvest naturally occurring conversations for the purposes of downstream conversational tasks (Zhang et al., 2019). However, this work does not address the issue of stylistic control or the effects of scaling models to large sizes, which are central themes of our work. Other concurrent work has also shown the benefit of learning from large amounts of social media conversations, but it also did not attempt to influence the model output style nor did it scale up the model to 8.3 billion parameters (Adiwardana et al., 2020).
When a large conversational model is trained on a diverse collection of multi-turn conversations, it is able to generate quality conversations that are engaging, coherent, and plausibly human. Furthermore, when conditioned on prior conversations, the model is able to utilize a speaker's personality when choosing how to reply in a conversation to allow for greater control and more diverse responses. In the future, we aim to leverage these pre-trained models to advance SOTA on downstream conversational tasks, such as knowledge-grounded conversations or question answering. Recent advancements in learnable information retrieval systems could select contextually relevant references to further strengthen the quality of generated dialogue.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
JinYeong Bak and Alice Oh. 2019. Variational hierarchical user-based conversation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1941-1950.
Zhangming Chan, Juntao Li, Xiaopeng Yang, Xiuying Chen, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019. Modeling personalization in continuous space for response generation via augmented Wasserstein autoencoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1931-1940.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan
Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Chaitanya K Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. arXiv preprint arXiv:1809.01984.
Oluwatobi Olabiyi, Anish Khazane, Alan Salimov, and Erik T Mueller. 2019. An adversarial learning framework for a persona-based multi-turn dialogue model. arXiv preprint arXiv:1905.01992.
Oluwatobi Olabiyi and Erik T Mueller. 2019. Multi-turn dialogue response generation with autoregressive transformer models. arXiv preprint arXiv:1908.01841.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016b. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models usinggpu model parallelism. arXiv preprint arXiv:1909.08053.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug-mentation end-to-end dialogue systems with commonsense knowledge. In Thirty-Second AAAI Conference on Artificial Intelligence.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. arXiv preprint arXiv:1905.12616.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.
A Mechanical Turk Setup
Below are sandbox examples of our Mechanical Turk layout for each task as seen in Figures 3-6. Note that in reality the synthetic example of the pair may come first, or both examples in the pair may be randomly ordered synthetic examples. Also, only the reference task displays the speaker's past reference replies. Hiding the reference in other tasks has the goal of trying to decorrelate the experiments. Example layouts for the realistic, reference, quality, and coherency tests can be found in Figures 3, 4, 5, & 6 respectively.
B Samples
Listed in Tables 7 - 13 are three turn generation samples conditioned on varying lengths of real conversations taken from the validation collection. They were all sampled from our Generative Conversation Control model, GCC-DEC (8.3B) using nucleus sampling with  $p = 0.95$ .
All typos present within these samples are intentional and reflective of their sources, be it human generated or sampled.
C Human Evaluation Results
Listed in Tables 14 - 17 are the complete results from our human evaluations, including the percentages that were omitted in Table 6. The rates that were omitted are two neutral form of ratings: for a given evaluation characteristic (e.g. quality, coherency, etc.) either both sources exhibited it, or neither did.
Figure 3: Realism Task Turk Layout.
Figure 4: Reference Task Turk Layout.
Table 7: A set of three generated responses continuing from the first turn of a real conversation about technical support for a video game between the developer's account and a customer. Upon investigation, it appears that the developer's account signs off their comments with "ÉT" in order to signal what person in their team wrote the comment. Likewise, it appears the model is emulating this practice as evidenced by signing of "WH".
Figure 5: Quality Task Turk Layout.
Which conversation is higher quality?
View full instructions
View tool guide
Note: you must spend at least 15 seconds and indicate your prior social media experience for approval and continued inclusion in future studies. Determine which conversation has the best quality language. Is the grammar, use of language, and overall linguistic quality better for Conversation 1 or Conversation 2? Or do both conversations have equally good or equally bad language quality?
How many hours per week do you spend on social media forums like Reddit?
Conversation 1:	Conversation 2:
turn 0	turn 0
turn 1	turn 1
turn 2	turn 2
turn 3	synthetic turn
Select an option
1 is good
#3 is good
Both are good 3
Neither are good
Figure 6: Coherency Task Turk Layout.
Note: you must spend at least 15 seconds and indicate your prior social media experience for approval and continued inclusion in future studies. Both conversations start off with human content, but may later include machine-generated content. Determine which conversation has the best coherency between responses (ie. do later turns match the topics/discussion from earlier turns). Options include Conversation 1 is coherent, Conversation 2 is coherent, both are coherent, and neither are coherent?
Which conversation is the most coherent between responses?
1 is coherent
#2 is coherent.
both are coherent.
neither are coherent 4