ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

1 INTRODUCTION . Machine reading comprehension ( MRC ) is a fundamental task in Natural Language Processing , which requires models to understand a body of text and answer a particular question related to the context . With success of unsupervised representation learning in NLP , language pre-training based models such as GPT-2 ( Radford et al. , 2019 ) , BERT ( Devlin et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) and RoBERTa ( Liu et al. , 2019 ) have achieved nearly saturated performance on most of the popular MRC datasets ( Rajpurkar et al. , 2016 ; Lai et al. , 2017 ; Rajpurkar et al. , 2018 ; Wang et al. , 2018 ) . It is time to challenge state-of-the-art models with more difficult reading comprehension tasks and move a step forward to more comprehensive analysis and reasoning over text ( Dua et al. , 2019 ) . In natural language understanding , logical reasoning is an important ability to examine , analyze and critically evaluate arguments as they occur in ordinary language according to the definition from Law School Admission Council ( 2019a ) . It is a significant component of human intelligence and is essential in negotiation , debate and writing etc . However , existing reading comprehension datasets have none or merely a small amount of data requiring logical reasoning , e.g. , 0 % in MCTest dataset ( Richardson et al. , 2013 ) and 1.2 % in SQuAD ( Rajpurkar et al. , 2016 ) according to Sugawara & Aizawa ( 2016 ) . One related task is natural language inference , which requires models to label the logical relationships of sentence pairs . However , this task only considers three types of simple logical relationships and only needs reasoning at sentence-level . To push the development of models in logical reasoning from simple logical relationship classification to multiple complicated logical reasoning and from sentence-level to passage-level , it is necessary to introduce a reading comprehension dataset targeting logical reasoning . A typical example of logical reasoning questions is shown in Table 1 . Similar to the format of multiple-choice reading comprehension datasets ( Richardson et al. , 2013 ; Lai et al. , 2017 ) , it contains a context , a question and four options with only one right answer . To answer the question ∗Equal contribution . 1Project page : http : //whyu.me/reclor/ in this example , readers need to identify the logical connections between the lines to pinpoint the conflict , then understand each of the options and select an option that solves the conflict . Human minds need extensive training and practice to get used to complex reasoning , and it will take immense efforts for crowdsourcing workers to design such logical reasoning questions . Inspired by the datasets extracted from standardized examinations ( Lai et al. , 2017 ; Clark et al. , 2018 ) , we build a dataset by selecting such logical reasoning questions from standardized exams such as GMAT 2 and LSAT 3 . We finally collect 6,138 pieces of logical reasoning questions , which constitute a Reading Comprehension dataset requiring logical reasoning ( ReClor ) . Human-annotated datasets usually contain biases ( Schwartz et al. , 2017 ; Cai et al. , 2017 ; Bugert et al. , 2017 ; Poliak et al. , 2018 ; Gururangan et al. , 2018 ; Zellers et al. , 2019 ) , which are often exploited by neural network models as shortcut solutions to achieve high testing accuracy . For data points whose options can be selected correctly without knowing the contexts and questions , we classify them as biased ones . In order to fully assess the logical reasoning ability of the models , we propose to identify the biased data points and group them as EASY set , and put the rest into HARD set . Based on our experiments on these separate sets , we find that even the state-of-the-art models can only perform well on EASY set and struggle on HARD set as shown in Figure 1 . This phenomenon shows that current models can well capture the biases in the dataset but lack the ability to understand the text and reason based on connections between the lines . On the other hand , human beings perform similarly on both the EASY and HARD set . It is thus observed that there is still a long way to go to equip models with true logical reasoning ability . The contributions of our paper are two-fold . First , we introduce ReClor , a new reading comprehension dataset requiring logical reasoning . We use option-only-input baselines trained with different random seeds to identify the data points with biases in the testing set , and group them as EASY set , with the rest as HARD set to facilitate comprehensive evaluation . Second , we evaluate several stateof-the-art models on ReClor and find these pre-trained language models can perform well on EASY set but struggle on the HARD set . This indicates although current models are good at exploiting biases in the dataset , they are far from capable of performing real logical reasoning yet . 2 RELATED WORK . Reading Comprehension Datasets . A variety of reading comprehension datasets have been introduced to promote the development of this field . MCTest ( Richardson et al. , 2013 ) is a dataset with 2,000 multiple-choice reading comprehension questions about fictional stories in the format similar to ReClor . Rajpurkar et al . ( 2016 ) proposed SQuAD dataset , which contains 107,785 questionanswer pairs on 536 Wikipedia articles . The authors manually labeled 192 examples of the dataset and found that the examples mainly require reasoning of lexical or syntactic variation . In an analysis of the above-mentioned datasets , Sugawara & Aizawa ( 2016 ) found that none of questions requiring logical reasoning in MCTest dataset ( Richardson et al. , 2013 ) and only 1.2 % in SQuAD dataset ( Rajpurkar et al. , 2016 ) . Lai et al . ( 2017 ) introduced RACE dataset by collecting the English exams for middle and high school Chinese students in the age range between 12 to 18 . They hired crowd workers on Amazon Mechanical Turk to label the reasoning type of 500 samples in the dataset and show that around 70 % of the samples are in the category of word matching , paraphrasing or single-sentence reasoning . To encourage progress on deeper comprehension of language , 2https : //en.wikipedia.org/wiki/Graduate Management Admission Test 3https : //en.wikipedia.org/wiki/Law School Admission Test more reading comprehension datasets requiring more complicated reasoning types are introduced , such as iterative reasoning about the narrative of a story ( Kočiskỳ et al. , 2018 ) , multi-hop reasoning across multiple sentences ( Khashabi et al. , 2018 ) and multiple documents ( Welbl et al. , 2018 ) , commonsense knowledge reasoning ( Mihaylov et al. , 2018 ; Zhang et al. , 2018 ; Huang et al. , 2019 ) and numerical discrete reasoning over paragraphs ( Dua et al. , 2019 ) . However , to the best of our knowledge , although there are some datasets targeting logical reasoning in other NLP tasks mentioned in the next section , there is no dataset targeting evaluating logical reasoning in reading comprehension task . This work introduces a new dataset to fill this gap . Logical Reasoning in NLP . There are several tasks and datasets introduced to investigate logical reasoning in NLP . The task of natural language inference , also known as recognizing textual entailment ( Fyodorov et al. , 2000 ; Condoravdi et al. , 2003 ; Bos & Markert , 2005 ; Dagan et al. , 2005 ; MacCartney & Manning , 2009 ) requires models to take a pair of sentence as input and classify their relationship types , i.e. , ENTAILMENT , NEUTRAL , or CONTRADICTION . SNLI ( Bowman et al. , 2015 ) and MultiNLI ( Williams et al. , 2018 ) datasets are proposed for this task . However , this task only focuses on sentence-level logical relationship reasoning and the relationships are limited to only a few types . Another task related to logical reasoning in NLP is argument reasoning comprehension task introduced by Habernal et al . ( 2018 ) with a dataset of this task . Given an argument with a claim and a premise , this task aims to select the correct implicit warrant from two options . Although the task is on passage-level logical reasoning , it is limited to only one logical reasoning type , i.e. , identifying warrants . ReClor and the proposed task integrate various logical reasoning types into reading comprehension , with the aim to promote the development of models in logical reasoning not only from sentence-level to passage-level , but also from simple logical reasoning types to the complicated diverse ones . Datasets from Examinations . There have been several datasets extracted from human standardized examinations in NLP , such as RACE dataset ( Lai et al. , 2017 ) mentioned above . Besides , NTCIR QA Lab ( Shibuki et al. , 2014 ) offers comparative evaluation for solving real-world university entrance exam questions ; The dataset of CLEF QA Entrance Exams Task ( Rodrigo et al. , 2015 ) is extracted from standardized English examinations for university admission in Japan ; ARC dataset ( Clark et al. , 2018 ) consists of 7,787 science questions targeting student grade level , ranging from 3rd grade to 9th ; The dialogue-based multiple-choice reading comprehension dataset DREAM ( Sun et al. , 2019 ) contains 10,197 questions for 6,444 multi-turn multi-party dialogues from English language exams that are designed by human experts to assess the comprehension level of Chinese learners of English . Compared with these datasets , ReClor distinguishes itself by targeting logical reasoning . 3 RECLOR DATA COLLECTION AND ANALYSIS . 3.1 DATA COLLECTION . The format of data in ReClor is similar to other multiple-choice reading comprehension datasets ( Richardson et al. , 2013 ; Lai et al. , 2017 ) , where a data point contains a context , a question and four answer options , among which only one option is right/most suitable . We collect reading comprehension problems that require complicated logical reasoning . However , producing such data requires the ability to perform complex logical reasoning , which makes it hard for crowdsourcing workers to generate such logical questions . Fortunately , we find the reading comprehension problems in some standardized tests , such as GMAT and LSAT , are highly in line with our expectation . We construct a dataset containing 6,138 logical reasoning questions sourced from open websites and books . In the original problems , there are five answer options in which only one is right . To comply with fair use of law4 , we shuffle the order of answer options and randomly delete one of the wrong options for each data point , which results in four options with one right option and three wrong options . Furthermore , similar to ImageNet dataset5 , ReClor is available for non-commercial research purpose only . We are also hosting a public evaluation server on EvalAI ( Yadav et al. , 2019 ) to benchmark progress on Reclor . 3.2 DATA ANALYSIS . As mentioned above , we collect 6,138 data points , in which 91.22 % are from actual exams of GMAT and LSAT while others are from high-quality practice exams . They are divided into training set , validation set and testing set with 4,638 , 500 and 1,000 data points respectively . The overall statistics of ReClor and comparison with other similar multiple-choice MRC datasets are summarized in Table 2 . As shown , ReClor is of comparable size and relatively large vocabulary size . Compared with RACE , the length of the context of ReCor is much shorter . In RACE , there are many redundant sentences in context to answer a question . However , in ReClor , every sentence in the context passages is important , which makes this dataset focus on evaluating the logical reasoning ability of models rather than the ability to extract relevant information from a long context . The length of answer options of ReClor is largest among these datasets . We analyze and manually annotate the types of questions on the testing set and group them into 17 categories , whose percentages and descriptions are shown in Table 3 . The percentages of different types of questions reflect those in the logical reasoning module of GMAT and LSAT . Some examples of different types of logical reasoning are listed in Figure 2 , and more examples are listed in the Appendix C. Taking two examples , we further express how humans would solve such questions in Table 4 , showing the challenge of ReClor .

Summary: This paper presents a new reading comprehension dataset for logical reasoning. It is a multi-choice problem where questions are mainly from GMAT and LSAT, containing 4139 data points. The analyses of the data demonstrate that questions require diverse types of reasoning such as finding necessary/sufficient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation. The paper includes comprehensive experiments with baselines to identify bias in the dataset, where the answer-options-only model achieves near half (random is 25%). Based on this result, the test set is split into the easy and hard set, which will help better evaluation of the future models. The paper also reports the numbers on the split data using competitive baselines where the models achieve low performance on the hard set.