SlowGuess/ACL_2020_OCR huggingface dataset:
Leyang Cui†‡*, Yu Wu, Shujie Liu, Yue Zhang‡, Ming Zhou
†Zhejiang University
$^{\ddagger}$ School of Engineering, Westlake University
$\ddagger$  {cuileyang,zhangyue}  $@$  westlake.edu.cn  $\diamond$  {Wu.Yu,shujliu,mingzhou}  $@$  microsoft.com
Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empirical results show that state-of-the-art methods only reach  $71\%$ , which is far behind the human performance of  $94\%$ , indicating that there is ample room for improving reasoning ability. MuTual is available at https://github.com/Nealcly/MuTual.
Building an intelligent conversational agent is one of the longest running goals in AI. Existing conversational agents can be categorized into task-oriented dialogue systems (Kannan et al., 2016) and non-task-oriented chatbot systems (Shum et al., 2018; Wu et al., 2019). Owing to the rise of deep learning techniques and the large amount of conversation data for training (Lowe et al., 2015; Wu et al., 2017; Zhang et al., 2018b), we are now witnessing promising results of chatbots both in academia and industry (Pan et al., 2019; Tao et al., 2019).
Neural dialogue systems are trained over a large dialogue corpus and used to predict responses given a context. There are two lines of methods. Retrieve-based methods and generation based methods rely
M: Ma'am, you forgot your phone.
F: Oh, thanks, I couldn't live without this little thing.
M: I know what you mean. It is of great significance to you. So did you enjoy your dinner?
F: Oh yes, everything was just perfect. It's so hard to take the whole family out to eat, but
your restaurant was perfect. Johnny had his own place to play in and I had time to talk
with my sisters and their husbands.
$\checkmark$  (A) M: Thanks for your compliment for the restaurant.
X (B) M: I'm sorry that you don't have a good time.
X (C) M: Goodbye brother! Love you.
X (D) M: Hurry up honey, or we will be late for the dinner.
Figure 1: B is incorrect because there is no reason to apologize. C and D can be excluded because the relationship between two speakers are waiter and customer based on the context.
on matching scores and perplexity scores, respectively. Due to the development of text matching and pre-training models (Devlin et al., 2019; Liu et al., 2019), a machine is able to achieve highly competitive results on these datasets, even close to human performance. For instance, ESIM (Chen et al., 2017) achieves  $88\%$  on the Dialogue NLI (Welleck et al., 2019), and BERT achieves  $85.8\%$ ,  $93.1\%$  and  $98.5\%$  in terms of  $\mathbf{R}_{10}@\mathbf{1}$ ,  $\mathbf{R}_{10}@\mathbf{2}$  and  $\mathbf{R}_{10}@\mathbf{5}$  on the Ubuntu Corpus (Whang et al., 2019).
However, there is still a huge gap between high performance on the leader-board and poor practical user experience. Chatbot engines often generate responses that are logically incorrect or violate commonsense knowledge (Shum et al., 2018). A likely reason is that current dialogue systems do not have strong reasoning skills, and most of the cases in previous benchmarks can be tackled by linguistic information matching. Previous work has demonstrated that neural encoders capture a rich hierarchy of syntactic and semantic information (Jawahar et al., 2019; Clark et al., 2019). However, reasoning capability and commonsense knowledge are not captured sufficiently (Young et al., 2018).
One important research question is how we can evaluate reasoning ability in chatbots, which can potentially allow us to bridge the gap between high performance on leader-board and unsatisfactory practical performance. To this end, we develop
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1406-1416 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dataset	Task	Reasoning	Domain	Manually
Ubuntu (Lowe et al., 2015)	Next Utterances Prediction	✕	Technique	✕
PERSONA-CHAT (Zhang et al., 2018a)	Next Utterances Prediction	✕	Persona	✓
Dialogue NLI (Welleck et al., 2019)	Next Utterances Prediction	✕	Persona	✕
CoQA (Reddy et al., 2019)	Conversational QA	✓	Diverse	✓
Douban (Wu et al., 2017)	Next Utterances Prediction	✕	Open	✕
DREAM (Sun et al., 2019)	Reading Comprehension	✓	Open	✓
WSC (Levesque et al., 2012)	Coreference Resolution	✓	Open	✕
SWAG (Zellers et al., 2018)	Plausible Inference	✓	Movie	✕
CommonsenseQA (Talmor et al., 2019)	Reading Comprehension	✓	Open	✓
RACE (Lai et al., 2017)	Reading Comprehension	✓	Open	✕
ARC (Clark et al., 2018)	Reading Comprehension	✓	Science	✕
DROP (Dua et al., 2019)	Reading Comprehension	✓	Open	✕
Cosmos (Huang et al., 2019)	Reading Comprehension	✓	Narrative	✓
MuTual	Next Utterances Prediction	✓	Open	✓
Table 1: Comparison between our dataset and other datasets. "Manually" indicates that human writing of the question or answers is involved in the data annotation process, rather than mere manual selection of data.
an open domain Multi-Turn dialogue reasoning dataset (MuTual) to facilitate conversation model reasoning capabilities. In particular, given a context, we prepare four response candidates, each of which is relevant to the context, but only one of them is logically correct. As shown in Figure 1, all responses follow the same topic, but only the first one is appropriated. It requires reasoning ability on social etiquette and relationship to make the correct choice, which is not considered by existing dialogue benchmarks.
We build our dataset based on Chinese high school English listening comprehension test data, where students are excepted to select the best answer from three candidate options, given a multi-turn dialogue and a question. The original data is formatted as <dialogue, question, answer>, which is not directly suitable for our goal since chatbots only concern about how to respond contexts instead of answering an additional question. Therefore, we ask human annotators to rewrite the question and answer candidates as response candidates. Then our dataset follows the traditional response selection setting (Lowe et al., 2015), where a model should recognize a correct response from others for a multi-turn dialogue.
The resulting dataset, MuTual, consists of 8,860 challenge questions, in terms of almost all questions involving reasoning, which are designed by linguist experts and high-quality annotators. We evaluate state-of-the-art retrieval-based models and pre-training models on MuTual. The best method gives a R@1 of  $71\%$ , which significantly underperforms human performance  $(94\%)$ . To the best of our knowledge, MuTual is the first human-labeled
reasoning-based dataset for multi-turn dialogue. We provide detailed analysis to provide insights into developing potentially reasoning-based chitchat dialogue systems.
Table 1 compares our dataset with prior dialogue and reasoning related benchmarks.
Dialogue: The Ubuntu Dialogue Corpus is a large retrieval-based dataset (Lowe et al., 2015), extracted from Ubuntu chat logs. PERSONA-CHAT (Zhang et al., 2018a) considers consistent personality in dialogue. Crowd workers are required to act the part of a given provided persona, and chat naturally. Dialogue NLI (Welleck et al., 2019) is a natural language inference dataset modified from PERSONA-CHAT. It demonstrates that NLI can be used to improve the consistency of dialogue models. CoQA (Reddy et al., 2019) is collected by pairing two annotators to chat about a passage in the form of questions and answers. Each question is dependent on the conversation history. There are also several large-scale datasets in Chinese, such as Sina Weibo (Shang et al., 2015), Douban Conversation Corpus (Wu et al., 2017) and E-commerce Dialogue Corpus (Zhang et al., 2018b).
As shown in Table 1, most of the existing conversation benchmarks do not focus on testing reasoning ability. One exception is CoQA, which considers pragmatic reasoning. The difference is that CoQA is a machine comprehension dataset, in which conversations are based on a given passage. Another related reading comprehension dataset is DREAM (Sun et al., 2019), which is designed specifically for challenging dialogue-based reading
Figure 2: The process of modifying the listening comprehension test data.
comprehension. It relies on an external question to test the model's understanding capability. In contrast to the above dataset, our dataset is a next utterance prediction task, which is the fundamental problem in retrieval-based chatbots. In addition, our dataset requires various specific reasoning abilities, such as algebraic reasoning, intention prediction and so on, which is the main characteristic of our dataset.
Reasoning: Recently, efforts have been made to develop benchmarks and tasks to address reasoning for language understanding. Winograd Schema Challenge (Levesque et al., 2012) is a reasoning-based coreference resolution task. Each pair of sentences differs by only one phrase. SWAG (Zellers et al., 2018) is derived from pairs of consecutive video captions, including 113k short context each with four candidates endings. CommonsenseQA (Talmor et al., 2019) is a question answering dataset extracted from CONCEPTNET (Speer et al., 2016). Utilizing CONCEPTNET to construct the dataset ensures that questions directly target commonsense reasoning. RACE is a machine reading comprehension dataset collected from English exams for Chinese students. AI2 Reasoning Challenge (Clark et al., 2018) contains 7,787 genuine grade-school level science questions with a corpus of 14M science reference sentences. DROP (Dua et al., 2019) and COSMOS (Huang et al., 2019) focus on factual understanding and commonsense comprehension, respectively.
Despite their success, these datasets can hardly help chatbots directly. Following the traditional dia
logue response selection setting, we deeply modify English listening comprehension conversation to form an utterance prediction task.
3.1 Collection
The original listening comprehension materials and question-answer pairs are designed by linguist experts. Students are required to choose the best answer from three options for a question based on a piece of audio. To ensure students fully understand the audio, most of the questions need to be answered with reasoning capability.
We crawled the listening exams from public websites<sup>1</sup>. Since the audio is either a conversation between two people or a simple passage, we only crawled data in the conversation format. The raw data is formatted as triples  $\langle$  Conversation (audio), Question and Choices (text), Answer (image)  $\rangle$ . The following data pre-processing methods are applied to convert raw data to data in Figure 2.
Step 1 Pre-processing: If question and candidate choices in two problems are the same, we consider them as duplicates and delete one of them. If there are more than three candidate options in one problem, we randomly drop incorrect options until three candidates are left.
The answers are stored as images. We apply a commercial OCR system to convert images to text. It is easy to recognize the printed alphabet answer for the OCR system. We manually correct all OCR
MuTual
<sup>1</sup>All the problems in our dataset are freely accessible online without copyright by consulting the legal adviser.
outputs to ensure quality. In the original listening comprehension test, the conversation is stored as audio. We adopt a commercial ASR system to convert speech to text, and further recruit experienced annotators to correct the transcription errors. To further ensure the quality of the transcripts, they are double-checked by annotators in the next step.
Step 2 Candidate Response Creation: Figure 2 illustrates the process of modifying the listening comprehension problem. At first, an annotator is required to segment the original conversation, after clues to answer the question have appeared. Then, they construct positive response (Response A in Figure 2) and negative responses (Response C and Response D) by consulting correct choice (Choice A) and incorrect choices (Choice B and Choice C), respectively. To make MuTual more challenging, we further ask the annotator to construct one more negative response (Response B) based on the correct choice. Through these steps, MuTual not only keeps the reasoning test designed by experts, but also introduces one more another type of reasoning for each instance. As shown in Figure 2, Response C and D can be excluded based on the relationship between two speakers. But B is incorrect due to the attitude reasoning.
It is worth noting that all negative responses are logically correct if the context is not considered, but they are not appropriated responses if the context is taken into account. Therefore, our dataset focuses on multi-turn conversation reasoning rather than the logic of a sentence. When framing a negative response, we encourage annotators to copy some phrases in the context to discourage a model that can solve the problem by text matching. We further calculate the lexical overlap between response and context. There are  $9.98\%$  (10.63%) words in the positive (negative) response that occur in the corresponding context, suggesting that MuTual is hard to solve by plain text matching.
Annotators in Step 2 are all English-major graduate students in Chinese, who are familiar with English language exams in China and fluent in English (pass the TEM-8²). Annotators are required to draft annotate 170 instances repeatedly, until their labeling is sufficiently accurate to provide useful annotation. Because not all conversations are adapted to construct a reasoning-based response problem, the annotator has the right to skip the con
MuTual
# Context-Response Pairs	8,860
# Avg. Turns per Dialogue	4.73
# Avg. Words per Utterance	19.57
Vocabulary Size (Context)	8,809
Vocabulary Size (Response)	8,943
Vocabulary Size	11,343
# Original Dialogues	6,371
# Original Questions	11,323
# Response Candidates	4
Table 2: Data statistics of MuTual.
versation. We employ five annotators to construct the response, and two quality inspectors to check it. We discard the instance when inspectors doubt the uniqueness or correctness of the answer.
The detailed statistics of MuTual are summarized in Table 2. MuTual has an average of 4.73 turns. The vocabulary size is 11,343, which is smaller than other dialogue datasets (Lowe et al., 2015; Wu et al., 2017). Because MuTual is modified from listening tests of English as a foreign language, the complexity of morphology and grammar is much simpler than other datasets.
For human-annotated datasets, there is always a trade-off between the number of instances being annotated and the quality of annotations (Kryciski et al., 2019). Our dataset is smaller than the previous crawling-based dialogue dataset (Lowe et al., 2015; Wu et al., 2017) due to the collection method. But it is comparable with high-quality reasoning based dataset (Clark et al., 2018; Khashabi et al., 2018; Talmor et al., 2019) and human-designed dialogue dataset (Zhang et al., 2018a). Moreover, around  $10\mathrm{k}$  is sufficient to train a discriminative model (Nivre et al., 2019) or fine-tuning the pretraining model (Wang et al., 2019).
To assess the distribution of different reasoning types, we annotate the specific types of reasoning that are involved for instance, sampled from the test set and categorize them into six groups. The definition and ratio of each group are shown as follows.
Attitude Reasoning: This type of instance tests if a model knows the speaker's attitude towards an object.
Algebraic Reasoning: This type of instances tests whether a model is equipped with algebraic abilities when it chooses a response.
Intention Prediction: This type tests whether a model can predict what the speaker is going to do next.
2The highest level test for English majors as a foreign language in China.
Figure 3: Examples from the MuTual dataset. All choices are relevant to context, but only one of them is logic correct. Some negative choices might be reasonable in extreme cases, but the positive one is the most appropriate. Clue words are purple and underline.
Situational Reasoning: Situation information (e.g., Location, Relationship between two speakers) is considered in this type of instance. A model should mine the implicit information from the previous context.
Multi-fact Reasoning: In this type of instance, the correct response is related to multiple facts in context, which requires the model to deeply understand the context rather than simply text matching.
Others:. There are  $9\%$  of instances that require other commonsense knowledge. For example, at the bottom of Figure 3, the model should know that a fully reserved restaurant is usually very popular.
The six types of reasoning are considered the most relevant to real chatbots. For example, it enables chatbots to make personal recommendations if a machine knows the user's attitude. The ability of intention prediction allows chatbots to respond more intelligently in a long conversation session.
3.3 MuTualplus
To further increase the difficulty, we use safe response to replace one of the candidate responses for each instance in MuTual. To guarantee diversity, the safe response is sampled from a list including "I'm afraid I didn't quite catch what you were saying.", "Could you repeat that?", "I'm really sorry, I didn't catch that.", etc. In particular, once the
instance is chosen, we randomly select a response to replace. If the positive response is replaced, the correct one is the safe response. If the negative response is replaced, the original positive response is still the best one.
The motivation to build MuTualplus is to evaluate whether a model is able to select a safe response when the other candidates are inappropriate. When we replace the positive response with a safe response, it simulates a scenario in which all the other candidates are incorrect. The phenomenon is common in retrieval-based chatbots, because limited candidate responses cannot handle all cases in practice. Similarly, we can evaluate if the model can choose the correct response instead of a safe response when a correct response exists.
We split the data into training, development and test sets, with an  $80\%$ ,  $10\%$  and  $10\%$  ratio. We pack instances constructed from the same conversation during splitting to avoid data leakage. Following the standard dialogue setting (Lowe et al., 2015; Wu et al., 2017), we consider our task as a response selection task and employ traditional information retrieval evaluation methods, including recall at position 1 in 4 candidates  $(R@1)$ , recall at position 2 in 4 candidates  $(R@2)$  and Mean Reciprocal Rank
(MRR) (Voorhees, 2000). We compare the performance of several response selection models as well as pre-training models. We simply introduce these works as follows:
We evaluate individual scoring methods, multi-choice methods and human performance in our experiment. Given a context  $c$  and four candidates  $(r_1, r_2, r_3, r_4)$ , the individual scoring method computes a score for each choice independently with a score  $g(c, r_i)$ , and selects the individual with the highest score among four candidates. On the contrary, the multi-choice method selects the best one by classification over all choices, formulated as  $h(c, r_1, r_2, r_3, r_4)$ .
TF-IDF: The correct response tends to share more words with the context than the incorrect ones. Following Lowe et al. (2015), we calculate the TF-IDF vectors for the context and each of the candidate responses, respectively, and then select the highest cosine similarity between the context and the candidate response as the model output. The "IDF" is calculated only on the training set.
Dual LSTM (Lowe et al., 2015): Two LSTMs are used to encode context and response, respectively. The relevance between context and response is calculated by the similarity of the final hidden state from both LSTMs.
Sequential Matching Network (Wu et al., 2017): To avoid losing information in the context, SMN constructs a word-word and a sequence-sequence similarity matrix, instead of utilizing the last hidden state only, and then aggregates similarity matrix as a matching score.
Deep Attention Matching Network: Zhou et al. (2018) adopt self attention module (Vaswani et al., 2017) to encode response and each utterance, respectively. To match utterance and response, DAM further applies cross-attention module and 3D matching to obtain final score.
BERT (Devlin et al., 2019): Pre-training models have shown promising results on various multi-choice and reasoning tasks (Whang et al., 2019; Xu et al., 2019). Following Devlin et al. (2019), we concatenate the context (sentence A), and a candidate response (sentence B) as BERT input. On the top of BERT, a fully-connected layer is used for transforming the [CLS] token representation to the matching score.
RoBERTa: Liu et al. (2019) re-establish
BERT's masked language model training objective by using more data and different hyper-parameters. We fine-tune RoBERTa in the same way as BERT.
GPT-2 (Radford et al., 2019): Given a context, the positive response has a higher probability compared with negative responses. Motivated by this, we concatenate context and response as a sequence, and calculate the joint probability of an entire sequence. The response in the lowest perplexity sequence is considered as the positive response. Moreover, we fine-tune the GPT-2 on [Context, Positive Response] pairs in MuTual training set, denoted as GPT-2-FT.
Multi-choice Method: Inspired by BERT for multiple choice (Devlin et al., 2019), the task is considered as picking the most suitable response by comparing four candidates responses. In particular, we concatenate each candidate response with the corresponding context. Each input sequence is subsequently encoded to produce a [CLS] representation. The positive response is predicted based on the concatenation of all [CLS] representations, on which a fully connected layer with softmax is used. The method is denoted as BERT-MC. Similarly, we implement RoBERTa-MC as another multi-choice method.
Human Performance: To obtain the human performance, we employ 3 NLP experts to measure the ceiling performance on the test set.
We report the performance of approaches introduced in 4.1, and human performance. Implementation details are shown in Appendix B.
4.2.1 Results on MuTual
All models perform significantly worse than on other popular conversation datasets, such as the Ubuntu Corpus (Lowe et al., 2015) and the Dialogue NLI dataset (Welleck et al., 2019), while human can address the reasoning problems easily. For example, BERT gives  $85.8\% \mathrm{R}_{10} @ 1$  on the Ubuntu Corpus, but RoBERTa only gives  $71.3\% \mathrm{R}_4 @ 1$  on MuTual.
TF-IDF only slightly better than randomly guessing, which indicates that there is no obvious statistic clue between context and positive response. In contrast, TF-IDF achieves  $54.98\%$  R@1 score on the Ubuntu Corpus, showing our dataset is more difficult to get the correct answer by text overlap. We evaluate typical retrieved-based dialogue models' performance on MuTual. From Table 3, we