SlowGuess/ACL_2020_OCR huggingface dataset:
Dev	Test
Baseline category	Baseline method	R@1	R@2	MRR	R@1	R@2	MRR
Baseline	Human	-	-	-	0.938	0.971	0.964
Random	0.250	0.500	0.604	0.250	0.500	0.604
Individual scoring method (discrimination)	TF-IDF	0.276	0.541	0.541	0.279	0.536	0.542
Dual LSTM (Lowe et al., 2015)	0.266	0.528	0.538	0.260	0.491	0.743
SMN (Wu et al., 2017)	0.274	0.524	0.575	0.299	0.585	0.595
DAM (Zhou et al., 2018)	0.239	0.463	0.575	0.241	0.465	0.518
BERT (Devlin et al., 2019)	0.657	0.867	0.803	0.648	0.847	0.795
RoBERTa (Liu et al., 2019)	0.695	0.878	0.824	0.713	0.892	0.836
Individual scoring method (generation)	GPT-2 (Radford et al., 2019)	0.335	0.595	0.586	0.332	0.602	0.584
GPT-2-FT (Radford et al., 2019)	0.398	0.646	0.628	0.392	0.670	0.629
Multi-choice method	BERT-MC (Devlin et al., 2019)	0.661	0.871	0.806	0.667	0.878	0.810
RoBERTa-MC (Liu et al., 2019)	0.693	0.887	0.825	0.686	0.887	0.822
Table 3: Comparison of varying approaches on MuTual.
Dev	Test
Baseline category	Baseline method	R@1	R@2	MRR	R@1	R@2	MRR
Baseline	Human	-	-	-	0.930	0.972	0.961
Random	0.250	0.500	0.604	0.250	0.500	0.604
Individual scoring method (discrimination)	TF-IDF	0.283	0.530	0.763	0.278	0.529	0.764
SMN (Wu et al., 2017)	0.264	0.524	0.578	0.265	0.516	0.627
DAM (Zhou et al., 2018)	0.261	0.520	0.645	0.272	0.523	0.695
BERT (Devlin et al., 2019)	0.514	0.787	0.715	0.514	0.787	0.715
RoBERTa (Liu et al., 2019)	0.622	0.853	0.782	0.626	0.866	0.787
Individual scoring method (generation)	GPT-2 (Radford et al., 2019)	0.305	0.565	0.562	0.316	0.574	0.568
GPT-2-FT (Radford et al., 2019)	0.226	0.577	0.528	0.226	0.611	0.535
Multi-choice method	BERT-MC (Devlin et al., 2019)	0.586	0.791	0.751	0.580	0.792	0.749
RoBERTa-MC (Liu et al., 2019)	0.621	0.830	0.778	0.643	0.845	0.792
Transfer method	RoBERTa (Liu et al., 2019)	0.559	0.827	0.746	0.558	0.827	0.746
RoBERTa-MC (Liu et al., 2019)	0.384	0.815	0.656	0.402	0.845	0.673
Table 4: Results on MuTualplus. Transfer method denotes that we train it on MuTual and test on MuTualplus.
can see that well-designed matching models do not give better performance compared with simple dual LSTM, moreover, they drop by more than 50 absolute R@1 points compared to their performance on the Ubuntu Corpus, indicating that text matching models cannot handle reasoning problem well.
Both BERT and RoBERTa outperform other models in MuTual, which is consistent with results in other literatures (Talmor et al., 2019). This is mainly because models learn reasoning capability during the pre-training on a large corpus. Although RoBERTa only gets  $71.3\%$  on  $\mathbb{R}@\mathbb{1}$ , it achieves a surprising number,  $89.2\%$ , on  $\mathbb{R}@\mathbb{2}$ , indicating that the model is able to rank the correct response to the top-2 position. BERT-MC and RoBERTa-MC obtain similar results with BERT and RoBERTa, respectively. However, even RoBERTa is far behind human performance 23 points on  $\mathbb{R}@\mathbb{1}$ , indicating that MuTual is indeed a challenging dataset, which opens the door for tackling new and complex reasoning problems in multi-turn conversations.
GPT-2 and GPT-2-FT also perform undesirably on MuTual, even if the averaged perplexity on MuTual testset is 10.40. This phenomenon illustrates that 1) sentences in MuTual are fluent; and
2) current generative models still have plenty of room to improve their reasoning ability.
4.2.2 Results on MuTualplus
As shown in Table 4, all models perform worse on MuTualplus, indicating the dataset is more difficult than MuTual, which is consistent with our assumption. We find that the performance of multi-choice method is significantly better than individual scoring method. One possible explanation is that multi-choice methods consider candidates together, so they can distinguish whether or not the safe response is the best one. In contrast, individual scoring methods are not robust, and safe responses are easy to confuse methods in the training stage. Moreover, RoBERTa-MC outperforms others by a large margin, showing its outstanding performance on reasoning problems.
Furthermore, we conduct a transfer experiment, in which models are trained on MuTual but tested on MuTualplus without fine-tuning. The experiment investigates whether the model handles safe responses well if they have never seen them in training corpus. As shown in Table 4, RoBERTa-MC and RoBERTa drops  $24.1\%$  and  $6.8\%$ , respectively,
Figure 4: BERT-MC and RoBERTa-MC performance on different reasoning types.
in the transfer setting, demonstrating the benefits of seeing safe responses during the training process. Moreover, the individual scoring RoBERTa outperforms RoBERTa-MC, showing that the individual scoring method is more robust, when the safe response is not fed during training.
To analyze model performance across different reasoning types, we calculate BERT-MC and RoBERTa-MC performance on various question types as we introduce in Section 3.2. As shown in Figure 4, we find that the trends of BERT-MC and RoBERTa-MC are similar across different categories. RoBERTa-MC significantly outperforms BERT-MC in attitude reasoning and multi-fact reasoning. One potential reason is that there are some normal patterns between action and attitude captured by RoBERTa-MC, such as "play football" and "excited". However, instances that involve algebraic and situation show poor performance. These two reasoning types heavily depend on commonsense reasoning. Taking Figure 5 as examples, it takes a simple subtraction step to derive the time difference (5:00 pm - 6h = 11:00 am), but this turns out a significant challenge for RoBERTa-MC. In the second case, RoBERTa-MC fails to infer the dialogue situation, where the goal is to find a flat to rent.
Performance across different context lengths: It is interesting that the performance of RoBERTa does not decrease significantly with the number of turns increasing, which is different from the phenomenon observed on other datasets. As shown in Table 5, the performance drops by only 1.9 points R@1 from 2 turns to long turns  $(>6)$ , and the performance of 5 turns is higher than those with 4
F: Do you know what time it is right now in New York?  
M: Let me see. It's 5:00 pm now, in New York is 6 hours behind.
F: Let me see, 7 hours behind. It is 11:00 am now in New York
F: 5 hours ahead. It is 11:00 pm now in New York.
X F: Is it 5:00 pm as well?
√ F: It is 11:00 am now in New York.
F: Good morning. What can I do for you?
M: I am looking for a flat for 2 people near the university.
F: Well. There are several places available and the rent ranges from 80 to $150 a month. What are your requirements?
M: I think of flat for no more than $100 a month is good. I prefer to live in a quiet street and I need at least 2 bedrooms.
X F: If you have any questions about enrollment, do not hesitate to ask me.  
√ F: How about this flat? If you are satisfied, we can sign the contract tomorrow.
F: We have 2 floors in our supermarket.
F: You want only 1 bedroom, so we have three flats that meet your requirement.
Figure 5: Error analysis.  $\times$  indicates RoBERTa-MC's prediction.
Figure 6: Ablation of context information. w/o context means all contexts are removed, so models just predict correct choice based on four candidates. context-n denotes the earliest n utterances are removed.
#T=2	#T=3	#T=4	#T=5	#T>6
#Instances	290	143	115	51	287
RoBERTa	0.731	0.657	0.635	0.804	0.712
RoBERTa-MC	0.681	0.622	0.609	0.725	0.750
Table 5: Performance comparison (R@1) of different number of turns on the test set. #T denotes number of turns. #Instances is the number of instances
turns, indicating the reasoning problems do not become much harder when the context becomes longer. The results also show that the difficulty of MuTual is attributed to reasoning instead of complex conversation history.
Context ablation study: We further verify whether our dataset requires multi-turn understanding rather than degenerating to a single turn reasoning problem. We evaluate Roberta and Roberta-MC performance when some utterances are manually removed. Figure 6 shows the performance when the earliest  $n$  utterances are removed in testing. As the ablation utterance increases, the performance of RoBERTa and RoBERTa-MC significantly decreases, which conforms to intuition. RoBERTa and RoBERTa-MC achieve only  $43.7\%$  and  $47.7\%$
after ablating all utterances in the context, respectively, indicating the importance of each utterance and the quality of the dataset. Moreover, if we shuffle the sequence of utterance, the performance of RoBERTa-MC drops by  $3.8\%$  only, showing that it is insensitive to the utterance sequence information.
We introduced MuTual, a high-quality manually annotated multi-turn dialogue reasoning dataset, which contains 8,860 dialogues and aims to test reasoning ability of dialogue models. We describe the process for generating MuTual, and perform a detailed analysis. We find that various state-of-the-art models show poor performance in MuTual. The best model RoBERTa only obtains  $71.3\%$  R@1. There is a large gap between the model performance and human performance. We hope that this dataset facilitates future research on multi-turn conversation reasoning problem.
We thank Yulong Chen, Duyu Tang, Zhiyang Teng and Sen Yang for their insightful discussions. We also thank all anonymous reviewers for their constructive comments. The corresponding author is Yue Zhang. We thank the support by a BrightDreams Robotics - Westlake University research grant.
Joakim Nivre et al. 2019. Universal dependencies 2.4. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UfAL), Faculty of Mathematics and Physics, Charles University.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart reply: Automated response suggestion for email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 955-964.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Wojciech Kryciski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012, pages 552-561.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294, Prague, Czech Republic. Association for Computational Linguistics.
Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn incomplete utterance restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1824-1833, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249266.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586, Beijing, China. Association for Computational Linguistics.
Heung-yeung Shum, Xiao-dong He, and Di Li. 2018. From eliza to xiaoice: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19(1):10-26.
Robert Speer, Joshua Chin, and Catherine Havasi. 2016. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019. One time of interaction may not be enough: Go deep with an interaction-over-interaction network for response selection in dialogues. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1-11, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Ellen Voorhees. 2000. The trec-8 question answering track report.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [link].
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Taesun Whang, Dongyub Lee, Chanhee Lee, Kisu Yang, Dongsuk Oh, and HeuiSeok Lim. 2019. Domain adaptive training bert for response selection.
Yu Wu, Wei Wu, Chen Xing, Can Xu, Zhoujun Li, and Ming Zhou. 2019. A sequential matching framework for multi-turn response selection in retrieval-based chatbots. Computational Linguistics, 45(1):163197.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
496-505, Vancouver, Canada. Association for Computational Linguistics.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2324-2335, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug-menting end-to-end dialogue systems with commonsense knowledge. In Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4970-4977.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018b. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3740-3752, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1118-1127, Melbourne, Australia. Association for Computational Linguistics.