SlowGuess/NAACL_2022_OCR huggingface dataset:
Is DF a Legal Evaluation Metric? Although DF is used for both weighted learning and evaluation, we see DF as a suitable evaluation metric for our task and not biased in favor of weighted learning due to: 1) A word receives multiple DF values in the training process given the corpus that a context belongs to; 2) in the generation process, DF is never used. 3) In the evaluation process, DF can be calculated purely on the test sets. Note that since a word receives multiple DF values in the training step, it is equivalently likely for the model trained with weighted learning to be influenced by DF weights of incorrect corpus. Above all, in the evaluation step, if the trained model is influenced more by DF weights from the correct corpus, it already means that the model is good
at distinguishing which corpus a given context is from, thus is suitable for our task.
Data Collection We collected 4 commonly used English corpora of different domains from the ParAI platform (Miller et al., 2017): OpenSubtitles corpus  $(\mathrm{OSDB})^2$  (Lison et al., 2018), Twitter corpus<sup>3</sup> (Miller et al., 2017), Ubuntu chatlogs corpus (Lowe et al., 2015)<sup>4</sup>, and PersonaChat corpus (Zhang et al., 2018) from the NeurIPS 2018 ConvAI2 Challenge (Dinan et al., 2019). Each corpus contains 250K context-response pairs, as much as the size of the original PersonaChat used in ConvAI2 competition. This gives us 1M context-response pairs in total. The corpus for training is a combination of these 4 corpora. For comparison, we have a single corpus-PersonaChat-trained on both base models. For testing, each of the 4 corpora has a test set of 30K context-response pairs, which is the same size of the test set of PersonaChat.
The OpenSubtitles corpus (OSDB) is a noisy dataset of film subtitles. We removed films that belonged to genres that usually had few conversations, such as musical and documentary films. We regarded two neighboring sentences as a context-response pair following Vinyals and Le (2015). The Twitter corpus contains one-turn dialogues extracted from Twitter. The original author has already cleaned it, so we only removed special symbols such as hashtags, Emojis, and @. The Ubuntu corpus contains dialogues about solving technical problems of Ubuntu. The PersonaChat corpus contains dialogues between two workers acting as specific personas; we focused on the dialogue part and ignored the persona part. This corpus allows us to compare our base models with state-of-the-art performance. These 4 corpora have very different characteristics, confirmed by the imbalanced performance of GPT-2 fine-tuned on a single corpus (see Table 2).
5.2 Training and Decoding
We used Pytorch (Paszke et al., 2017) to implement the LSTM Seq2Seq model with attention and the pre-trained GPT-2 models. For GPT-2, we adapted
$^{2}$ http://www.opensubtitles.org/  
 $^{3}$ https://github.com/Marsan-Ma/chat corpus/  
 $^{4}$ https://github.com/rkadlec/ubuntu-ranking-dataset-creator
Model	Corpus / Method	Test set
OSDB
Precision	Recall	F1	Twitter
Interleaved	23.2	9.6	11.1	24.1	10.1	11.7
Table 4: Precision, recall and F1 of ROUGE-1  $(\%)$  for baselines and proposed methods fine-tuned on 4 corpora (stop words eliminated)
Model	Corpus / Method	Test set
OSDB	PersonaChat	Twitter	PersonaChat	Ubuntu	PersonaChat	PersonaChat
Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train
Test Set (Standard Score)	7.0	9.7	3.6	3.7	9.1	11.0	3.6	3.8	19.4	23.2	2.7	2.8	9.5
LSTM	PersonaChat (single)	2.9	3.4	9.2	9.9	2.8	3.4	8.6	9.2	2.7	3.1	8.6	9.1	11.9
Concatenated	2.9	3.3	7.6	8.6	3.6	4.3	8.0	8.7	7.6	7.7	5.6	6.0	12.5
Interleaved	3.9	4.1	5.0	5.3	4.7	4.9	4.1	4.5	11.8	11.3	3.7	4.0	11.5
Labeled	3.9	4.2	5.0	5.3	5.0	5.3	3.9	4.3	12.5	11.8	3.4	3.8	12.1
Multi-task Labeled	3.8	4.0	5.0	5.4	4.5	4.7	4.1	4.5	11.2	10.7	3.8	4.1	11.4
Weighted	5.6	6.3	4.1	4.5	9.9	10.1	3.8	4.3	27.7	25.4	2.7	3.0	17.7
GPT-2	PersonaChat (single)	2.8	3.2	10.5	11.1	2.9	3.3	9.5	9.8	4.1	4.6	8.3	8.4	12.9
Concatenated	3.1	3.6	8.8	9.4	3.3	3.9	8.2	8.7	6.5	7.1	7.0	7.4	12.1
Interleaved	4.9	5.8	4.8	5.0	4.6	5.1	4.4	4.7	15.7	16.0	3.1	3.4	12.1
Labeled	4.9	5.8	4.8	5.0	4.7	5.2	4.1	4.3	16.7	17.0	2.9	3.2	12.4
Multi-task Labeled	4.8	5.7	4.8	5.1	4.6	5.1	4.4	4.6	15.5	15.8	3.1	3.4	12.1
Weighted	6.0	7.5	4.1	4.4	8.1	8.8	3.7	4.1	25.7	24.4	2.4	2.6	16.0
Table 5:  $\alpha\mathrm{DF}_d$  scores for generated responses from multiple corpora. The columns "train" indicate train-set- $\alpha\mathrm{DF}_d$ . The columns "test" indicate test-set- $\alpha\mathrm{DF}_d$ .
our model from the implementation of the HuggingFace team<sup>5</sup>. The LSTM model has 4 layers and the dimension is 512. The training procedure was with a batch size of 256, learning rate of 1.0, dropout rate of 0.2, and gradient clip threshold of 5. The vocabulary size is 50000. GPT-2 has 12 layers, 12 heads, and the dimension is 768, the same as the pre-trained model. The training procedure was with Adam and we adopted a similar setup as Wolf et al. (2019): the batch size was 32, learning rate was  $6 \times 10^{-5}$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , L2 weight decay set to 0.01, learning rate linearly decreased to zero at the end. We followed these hyper-parameters to ensure state-of-the-art performance for the base models. We use the same hyper-parameters for both base models and models with our proposed methods, so the proposed methods work slightly
(but not much) worse than it should be. This is to avoid the extra improvement caused by hyperparameters. We pre-trained the LSTM model on 3 large-scale corpora (OSDB, Twitter and Ubuntu) with interleaved learning until converging. GPT-2 is already pre-trained, so we directly used it for finetuning (details about pre-training convergence can be found in Section B). For decoding, we adopted greedy decoding for all the models to ensure an equal condition.
For automatic metrics, to measure the relevance of the generated responses, we eliminated punctuation and stop words, and adopted Rouge-1 $^{6}$  (precision, recall, F1) as multi-grams become meaningless
$^{6}$ We used implementation from https://github.com/google-research/google-research/tree/master/rouge.
$^{5}$ https://huggingface.co/.
without stop words. However, Rouge-1 compares the generated responses with the golden ones, while there is never a standard response for any context, so in addition to Rouge, we use  $\alpha$ DF score that shows to what extent the generated responses use important words of the corresponding corpus, as stated in Section 4.4. Due to the limitation of automatic evaluation methods (Liu et al., 2016), we also conduct an extensive human evaluation on the relevance of generated responses to contexts (see Section 6.1 for details).
Our base models achieve perplexity scores of 28.9 (LSTM model) and 19.6 (GPT-2) on the test set of the PersonaChat dataset from the ConvAI2 competition when fine-tuned with the single PersonaChat corpus (more details can be found in Section C). These results would likely advance the models to the second round in the competition.
Table 4 shows that models trained with our proposed methods gain better performance on Rouge than baselines. Baselines concentrate on the last trained corpus (PersonaChat), while with the proposed methods, performance is more balanced on multiple corpora. Weighted learning has the best overall performance on all metrics, and it performs especially well on the Ubuntu corpus, indicating that it might be good at distinguishing the unique technical words from the Ubuntu corpus. Labeled learning is the second best with stable improvement from interleaved learning, indicating that the corpus embeddings function as expected. Multi-task labeled learning has slightly worse performance than interleaved learning, indicating that predicting the corpus of a contexts is not easy, and wrong predictions result in worse performance.
Table 5 shows  $\alpha\mathrm{DF}_d$  scores for generated responses of each corpus. Full results can be found in Section E. We use both  $\alpha\mathrm{DF}_d$  calculated purely on the train set (train-set- $\alpha\mathrm{DF}$ ) and  $\alpha\mathrm{DF}_d$  calculated purely on the test set (test-set- $\alpha\mathrm{DF}$ ). The black scores are scores for the corresponding corpus (we expect high scores for these parts), while the grey scores are scores for non-related corpus—PersonaChat (we expect low scores for these parts). Note that scores for different corpora are in different scales. From the table, we can see that train-set-DF scores and test-set-DF scores are similar, and weighted learning always has the highest score, indicating that weighted learning distinguishes well
which corpus a context comes from. Labeled learning is the second best, indicating that the learned corpus embeddings help the model to use more important words of the corresponding corpus. Compared to the concatenated corpus, the improvement is at least  $20\%$ , while the decrease in PersonaChat is just  $9\%$  at most.
We conducted a human evaluation on all GPT-2 models: base models and models adapted with our proposed methods. We randomly picked 2400 responses: 400 different contexts evenly from 4 corpora with 6 responses generated by each of our models. 3 judges<sup>7</sup> are asked to pick the most and the least relevant response(s) for the given context. The most relevant response(s) are given score 3, the least relevant response(s) are given score 1, and the other(s) are given score 2. Table 6 shows the overall scores of all GPT-2 based models. Table 7 shows the p-value for the t-test conducted between every two models. The overall scores of our proposed methods are all highly significantly  $(p < 0.001)$  higher than the concatenated models, especially the weighted learning method.
6.2 Response Examples
The generated responses from better methods are more relevant to the corresponding corpus, while worse methods cannot distinguish contexts from different corpora (e.g., they may answer any questions in a "PersonaChat" way). To show an intuition of the difference among our proposed methods, we present some response examples generated by GPT-2 in Section G.
6.3 Possible Limitations
Our proposed methods are meant to be able to work in most models, which is why we choose the most common conversational models as our base models. However, there are many variants of conversational models focusing on different aspects, such as integrating knowledge, avoiding dull responses, keeping the speech style, etc. We cannot ensure that our methods work for all of these variant models. Also, dialogues are always multi-turn, while we focus on a simpler task: single-turn response generation.
7Similar to previous work like Zhang et al. (2020), we have 3 judges. We have one random worker from https: //www.mturk.com/worker, one bachelor student, and one graduate student. An example of the mTurk interface can be found in Section F.
Model \ Corpus	OSDB	Twitter	Ubuntu	PersonaChat	Overall
PersonaChat (single)	1.53	1.43	1.21	2.09	1.56
Concatenated	1.67	1.71	1.60	2.16	1.78
Interleaved	2.04	1.89	2.18	2.24	2.09
Labeled	2.10	2.10	2.32	2.24	2.19
Multi-task Labeled	2.05	1.98	2.11	2.24	2.10
Weighted	2.40	2.45	2.61	2.47	2.48
Table 6: Average scores of human evaluation for GPT-2 based models on each corpus
Model \ Model	PersonaChat	Concatenated	Interleaved	Labeled	Multi-Task Labeled	Weighted
PersonaChat	1.00	\	\	\	\	\
Concatenated	2.54 × 10-7**	1.00	\	\	\	\
Interleaved	4.71 × 10-34**	2.09 × 10-12**	1.00	\	\	\
Labeled	1.08 × 10-46**	9.41 × 10-21**	1.18 × 10-2*	1.00	\	\
Multi-task Labeled	6.65 × 10-35**	6.96 × 10-13**	8.86 × 10-1	1.17 × 10	1.00	\
Weighted	1.65 × 10-103**	2.86 × 10-63**	6.54 × 10-26**	1.59 × 10-15**	2.01 × 10-25**	1.00
Table 7: P-value for t-test on overall human evaluation scores of GPT-2 based models,  $^{**}p <   0.001$
Furthermore, the methods are trained and evaluated on English corpora. There can be a limitation on applying the methods to other languages.
We have experimented with 4 methods—interleaved learning (baseline), labeled learning, multi-task labeled learning, and weighted learning—to help common open-domain conversational systems generate relevant responses for multiple corpora of different domains. We adopted Rouge (precision, recall, F1) for auto evaluation. In addition, we used DF to evaluate how well a model uses relevant words for a corresponding corpus. We also did an extensive human evaluation. Our results show significant improvement in performance for our proposed methods, especially weighted learning. Future work of multi-turn response generation is potential. We have focused on one-turn response generation, while dialogue is naturally multi-turn so further research is needed.
This paper is funded by the collaborative project of DNB ASA and Norwegian University of Science and Technology (NTNU). We also received assist on computing resources from the IDUN cluster of NTNU (Själander et al., 2019). We would like to thank Aria Rahmati, Zhirong Yang (Norwegian Research Council, 287284) and Özlem Özgöbek for their helpful comments.
Reina Akama, Kazuaki Inada, Naoya Inoue, Sosuke Kobayashi, and Kentaro Inui. 2017. Generating Stylistically Consistent Dialog Responses with Transfer Learning. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 408-412, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of Representations for Domain Adaptation. In B. Scholkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 137-144. MIT Press.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum Learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 41-48. ACM.
Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385-391. Association for Computational Linguistics.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa
Fe, New Mexico, USA. Association for Computational Linguistics.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2019. The Second Conversational Intelligence Challenge (ConvAI2). arXiv:1902.00098 [cs].
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput., 9(8):1735-1780.
Chenyang Huang, Osmar Zaiane, Amine Trabelsi, and Nouha Dziri. 2018. Automatic Dialogue Generation with Expressed Emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 49-54. Association for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Mahesh Joshi, Mark Dredze, William W. Cohen, and Carolyn Rose. 2012. Multi-Domain Learning: When Do Domains Matter? In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1302-1312. Association for Computational Linguistics.
Nate Kornell and Robert Bjork. 2008. Learning concepts and categories is spacing the "Enemy of induction"? Psychological science, 19:585-92.
Edda Leopold and Jörg Kindermann. 2002. Text categorization with support vector machines. how to represent texts in input space? Machine Learning, 46(1):423-444.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A Persona-Based Neural Conversation Model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). European Language Resource Association.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132. Association for Computational Linguistics.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294. Association for Computational Linguistics.
Yi Luan, Chris Brockett, Bill Dolan, Jianfeng Gao, and Michel Galley. 2017. Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 605–614. Asian Federation of Natural Language Processing.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. *ParlAI: A Dialog Research Software Platform*. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 79-84. Association for Computational Linguistics.
Tong Niu and Mohit Bansal. 2018. Polite Dialogue Generation Without Parallel Data. Transactions of the Association for Computational Linguistics, 6:373-389.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In EACL.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Better Language Models and Their Implications. Technical report, Technical report, OpenAI.
Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-Driven Response Generation in Social Media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 583-593. Association for Computational Linguistics.
Avishek Saha, Piyush Rai, Hal Daum $\mathring{\mathbb{A}}$  III, and Suresh Venkatasubramanian. 2011. Online Learning of Multiple Tasks and Their Relationships. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 643-651. PMLR.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? How controllable attributes affect human judgments. arXiv:1902.08654 [cs].
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural Responding Machine for Short-Text Conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577–1586. Association for Computational Linguistics.
Yangyang Shi, Martha Larson, and Catholijn M. Jonker. 2015. Recurrent Neural Network Language Model Adaptation with Curriculum Learning. Comput. Speech Lang., 33(1):136-154.
Magnus Själander, Magnus Jahre, Gunnar Tufte, and Nico Reissmann. 2019. EPIC: An energy-efficient, high-performance GPGPU computing research infrastructure.
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021-2030, Online. Association for Computational Linguistics.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 196-205. Association for Computational Linguistics.