ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Authors: Weihao Yu
Zihang Jiang
Yanfei Dong
Jiashi Feng
Venue:  (2020)
Keywords: reading comprehension
logical reasoning
natural language processing

Abstract: Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models. 

Published as a conference paper at ICLR 2020
RECLOR: A READING COMPREHENSION DATASET REQUIRING LOGICAL REASONING
Weihao Yu", Zihang Jiang', Yanfei Dong & Jiashi Feng
National University of Singapore weihaoyu6e gmail. 11......... com, {izihang, dyanfei eu. nus. edu, e elefjia@euu.eeuu.s
ABSTRACT
Recent powerful pre-trained language models have achieved remarkable perfor- mance on most of the popular datasets for reading comprehension. is time to introduce more challenging datasets to push the development of this field to- wards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, annotatdd datasets usually contain biases, which often exploited by models to achieve high accuracy without truly understanding the text. In order comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest HARD set. Empirical results show that ofate-of-the-art models have outstanding ability to capture biases contained in the dataset with high accuracy EASY set. However, they struggle HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models]
INTRODUCTION
Machine reading comprehension (MRC) is a fundamental task in Natural Language Processing, which requires models to understand a body of text and answer a particular question related the context. With success of unsupervised representation learning in NLP, language pre-training based models such as GPT-2 (Radford al., 2019). BERT (Devlin al.) 2019), XLNet (Yang et al. 2019) and RoBERTa (Liu et 2019) have achieved nearly saturated performance on most of the popular MRC datasets (Rajpurkar et al. 2016 Lai et al., 2017 Rajpurkar et al. 2018 Wang et al. 2018 It is time to challenge Mate-offthhe-ar models with more difficult reading comprehension tasks and move a step forward to more comprehensive analysis and reasoning over text (Dua et al. 2019).
In natural language understanding, logical reasoning is important ability to examine, analyze and critically evaluate arguments as they occur in ordinary language according to the definition from Law School Admission Council (2019a). It is a significant component of human intelligence and is essential in negotiation, debate and writing etc. However, existing reading comprehension datasets have none merely small amount of data requiring logical reasoning, e.g., 0% in MCTest dataset (Richardson et al., (2013) and 1.2% in SQuAD (Rajpurkar et al.. 2016) according to Sug- awara & Aizawa (2016). One related task is natural language inference, which requires models to label the logical relationships of sentence pairs. However, this task only considers three types of simple logical relationships and only needs reasoning at sentence-level. To push the development of models in logical reasoning from simple logical relationship classification to multiple complicated necessary to introduce a reading
logical reasoning and from sentence-level to passage-leved comprehension dataset targeting logical reasoning.
typical example of logical reasoning questions is shown in Table T Similar to the format of multiple-choice reading comprehension datasets (Richardson et al., 2013 Lai et al. 2017), it con- tains context, a question and four options with only one right answer. To answer the question
"Equal contribution.
Project page: http:/ http://hw.meeeecll.../

---

Published as a conference paper at ICLR 2020
in this example, readers need to identify the logical connections between the lines to pinpoint the conflict, then understand each of the options and select an option that solves the conffict. Human minds need extensive training and practice to get used to complex reasoning, and it will take im- mense efforts for crowdsourcing workers to design such logical reasoning questions. Inspired by the datasets extracted from standardized examinations (Lai et al., 2017, Clark al.) 2018)) we build dataset by selecting such logical reasoning questions from standardized exams such as GMAT and LSAT We finally collect 6,138 pieces of logical reasoning questions, which constitute a Reading Comprehension dataset requiring logical reasoning (ReClor).
datasets usually contain biases (Schwartz et al. 2017 Cai et al. 2017: Bugert et al. 2017] Poliak et al., 22018: Gururangan et al. [0118] Zellers et al. 2019). which are often exploited by neural network models as shortcut solutions to achieve high testing accuracy. For data points whose options can be selected correctly without knowing the contexts and questions, we classify them as biased ones. In order to fully assess the logical reasoning ability of the models, propose to identify the biased data points and group them as EASY set, and put the rest into HARD set. Based on our experiments on these separate sets, we find that even the ofate-of-the-art models can only perform well on EASY set and struggle HARD set as shown in Figure IT This phenomenon shows that current models can well capture the biases in the dataset but lack the ability to understand the text and reason based on connections between the lines. On the other hand, human beings perform similarly on both the EASY and HARD set. It is thus observed that there is still a long way to go to equip models with true logical reasoning ability.
The contributions of our paper are two-fold. First, we introduce ReClor, a new reading comprehen- sion dataset requiring logical reasoning. We use option-only-input baselines trained with different random seeds to identify the data points with biases in the testing set, and group them as EASY set, with the rest as HARD set to facilitate comprehensive evaluation. Second, we evaluate several state- of-the-an art models on ReClor and find these pre-trained language models can perform well on EASY set but struggle on the HARD set. This indicates although current models are good at exploiting biases in the dataset, they are far from capable of performing real logical reasoning yet.
guess
L XLNclaaes
RoBERTayaG:
Humaass (Gradante Students)
CPT-2
Accurayy (S4
80
60
40-
EASY (Maaed) set
HARD Inon blasod) set
Figure 1: Performance comparison of state-of-the-art models and humans (graduate students) on EASY and HARD set of ReClor testing set.
2 RELATED WORK
Reading Comprehension Datasets. A variety of reading comprehension datasets have been intro- duced promote the development of this field. MCTest (Richardson et al., 2013) is dataset with 2,000 multiple-choice reading comprehension questions about fictional stories in the format similar to ReClor. RRajpurkar al. (2016) proposed SQuAD dataset, which contains 107,785 question- answer pairs on 536 Wikipedia articles. The authors manually labeled 192 examples of the dataset and found that the examples mainly require reasoning of lexical or syntactic variation. In an anal- ysis of the above-mentioned datasets, (Sugawara & Aizawal (2016) found that none of questions requiring logical reasoning in MCTest dataset (Richardson et al. 2013) and only 1.2% in SQuAD dataset (Rajpurkar et al. 2016). Lai et al. (2017) introduced RACE dataset by collecting the En- glish exams for middle and high school Chinese students in the age range between 12 to 18. They hired crowd workers Amazon Mechanical Turk to label the reasoning type of 500 samples in the dataset and show that around 70 % of the samples are in the category of word matching, paraphras- ing or single-sentence reasoning. To encourage progress deeper comprehension of language,
httaas:/ooniiiieedaaao///////// https://en..iiipela..ogg.//
Management Admi ssi on Test
School Admi 88 on Test
2

---

Published as a conference paper at ICLR 2020
Context:
jurisdictions where use of headlights is optional when visibility is good, drivers who use headlights at all times are less likely to be involved in a collision than are drivers who use headlights only when visibility is poor. Yet Highway Safety Department records show that making use of headlights mandatory all times does nothing reduce the overall number of collisions.
Question: Which one of the following. if true, most helps to resolve the apparent discrepancy in the information above?
Options:
A. In jurisdictions where use of headlights is optional when visibility is good, one driver in four uses headlights for daytime driving in good weather.
Only very careful drivers use headlights when their use is not legally required.
C. The jurisdictions where use of headlights is mandatory at all times are those where daytime visibility is frequently poor.
D. A law making use of headlights mandatory all times is not especially difficult to enforce.
Answer:
Table 1: An example in the ReClor dataset which is modified from the Law School Admission
Council
2019b).
more reading comprehension datasets requiring more complicated reasoning types are introduced, such as iterative reasoning about the narrative of story (Kočiský et al. 2018). multi-hop reasoning across multiple sentences (Khashabi et al. 2018) and multiple documents (Welbl et al.) 2018) com- monsense knowledge reasoning (Mihaylov et al.) 2018 Zhang et al. 2018 Huang al. (019)) and numerical discrete reasoning over paragraphs (Dua et al. (2019). However, to the best of our knowl- edge, although there some datasets targeting logical reasoning in other NLP tasks mentioned in the next section, there is no dataset targeting evaluating logical reasoning in reading comprehension task. This work introduces
dataset to fill this gap.
Logical Reasoning in NLP. There are several tasks and datasets introduced to investigate logical reasoning in NLP. The task of natural language inference, also known as recognizing textual en- tailment (Fyodorov et al.) 2000 Condoravvii et al. 2003) Bos & Markert, 2005; Dagan et al.) (0055] MacCartney Manning 2009) requires models take a pair of sentence as input and classify their relationship types, i.e., ENTAILMENT, NEUTRAL, or CONTRADICTION. SNLI (Bowman et al. 2015) and MultiNLI (Williams et 2018) datasets are proposed for this task. However, this task only focuses on sentence-level logical relationship reasoning and the relationships are limited to only a few types. Another task related to logical reasoning NLP is argument reasoning comprehension task introduced by Habernal et al. (2018) with dataset of this task. Given an argument with claim and premise, this task aims to select the correct implicit warrant from two options. Although the task is passage-level logical reasoning, it is limited to only one logical reasoning type, i.e., identifying warrants. ReClor and the proposed task integrate various logical reasoning types into reading comprehension, with the aim to promote the development of models in logical reasoning not only from sentence-level to passage-level, but also from simple logical reasoning types to the complicated diverse ones.
Datasets from Examinations. There have been several datasets extracted from human standardized examinations in NLP, such RACE dataset (Lai et al. 2017) mentioned above. Besides, NTCIR QA Lab (Shibuki et al. 2014) offers comparative evaluation for solving real-world university entrance exam questions: The dataset of CLEF QA Entrance Exams Task (Rodrigo al. 2015) extracted from standardized English examinations for university admission in Japan; ARC dataset (Clark et al., 2018] consists of 7,787 science questions targeting student grade level, ranging from 3rd grade to 9th; The dialogue-based multiple-choice reading comprehension dataset DREAM (Sun et al.) 2019) contains 10,197 questions for 6,444 multi-turn multi-party dialogues from English language exams that are designed by human experts to assess the comprehension level of Chinese learners of English. Compared with these datasets, ReClor distinguishes itself by targeting logical reasoning.
3
RECLOR DATA COLLECTION AND ANALYSIS
3.1 DATA COLLECTION
The format of data in ReClor is similar to other multiple-choice reading comprehension datasets (Richardson et al. 2013 Lai et al. 2017), where data point contains context, question and four
3

---

Published as a conference paper at ICLR 2020
ReClor
DREAM
MCTest
ARC exams
RACE
construction method
exams written text
exams dialogues
crowd-sourcing child's stories
exams
context type
written text
# of options
4
3
4
4
# of context
6,138
6,444
660
27,933
# of questions
6.138
10.197
2.640
7.787
97,687
Vocab size
26,576
13,037
8,000
6,329
136,629
Context Len
73.6
85,9
210.1
321.9
Question Len
17.0
8.6
7.8
20.5
10.0
Option Len
20.6
5.3
3.4
4.2
5.3
Table 2: Statistics of several multiple-choice MRC datasets.
answer options, among which only one option is right/most suitable. We collect reading comprehen- sion problems that require complicated logical reasoning. However, producing such data requires the ability perform complex logical reasoning. which makes it hard for crowdsoureing workers to generate such logical questions. Fortunately, we find the reading comprehension problems in some standardized tests, such as GMAT and LSAT, are highly in line with our expectation.
We construct dataset containing 6,138 logical reasoning questions sourced from open websites and books. In the original problems, there are five answer options in which only one right. To comply with fair use of law we shuffle the order of answer options and randomly delete one of the wrong options for each data point, which results in four options with one right option and three wrong options. Furthermore, similar to ImageNet datase ReClor is available for non-commercial research purpose only. We are also hosting public evaluation server on EvalAl (Yadav et al. 2019) to benchmark progress on Reclor.
3.2 DATA ANALYSIS
As mentioned above, we collect 6,138 data points, in which 91.22% are from actual exams of GMAT and LSAT while others from high-quality practice exams. They are divided into training set, val- idation set and testing set with 4,638, 500 and 1,000 data points respectively. The overall statistics of ReClor and comparison with other similar multiple-choice MRC datasets are summarized in Ta- ble 2 As shown, ReClor is of comparable size and relatively large vocabulary size. Compared with RACE, the length of the context of ReCor is much shorter. RACE, there are many redundant sen- tences in context to answer a question. However, in ReClor, every sentence in the context passages important, which makes this dataset focus on evaluating the logical reasoning ability of models rather than the ability to extract relevant information from long context. The length of answer options of ReClor is largest among these datasets. We analyze and manually annotate the types of questions on the testing set and group them into 17 categories. whose percentages and descriptions are shown in Table 3 The percentages of different types of questions reflect those in the logical reasoning module of GMAT and LSAT. Some examples of different types of logical reasoning listed in Figure and more examples are listed in the Appendix(C] Taking two examples, we further express how humans would solve such questions in Table 4 showing the challenge of ReClor.
3.3 DATA BIASES IN THE DATASET
The dataset is collected from exams devised by experts in logical reasoning, which means it is annotated by humans and may introduce biases the dataset. Recent studies have shown that models can utilize the biases in dataset of natural language understanding to perform well on the task without truly understanding the text (Schwartz et al., 2017 Cai Cai et al., 2017: Bugert et al. 2017 Poliak et al. 2018 Gururangan et al. 2018 Zellers et al. 2019). It is necessary to analyze such data biases to help evaluate models. In the ReClor dataset. the common context and question are shared across the four options for each data point, so focus on the analysis of the difference in lexical choice and sentence length of the right and wrong options without contexts and questions. We first investigate the biases of lexical choice. We lowercase the options and then WordPiece tokenization (Wu et al. 2016) of BERTBASE (Devlin et al. 2019) to get the tokens. Similar to
http:///mmge-nnet.orgg/o// -ad-
4

---

Published as a conference paper at ICLR 2020
Type
Description
Necessary Assumptions (1 (11,4%)
identify the claim that must he true or is required in order for the argument to work.
Sufficient Assumptions (3.0%)
identify sufficient assumption, that is, assumption that, if added to argument, would make it logically valid.
Strengthen (9.4%)
identify information that would strengthen an argument identify information that would weaken an argument identify information that would be useful to know evaluate argument
Weaken (1 11.3%)
Evaluation (1.3%)
Implication (4.6%)
identify something that follows logically from a set premises identify the conclusion/main point of line of reasoning find the choice that is most strongly supported by stimulus identify information that would explain resolve situation identify principle, or find situation conforms princi- ple, or match the principles identify or infer an issue in dispute identify the technique used in the reasoning of an argument deseribe the individual role that a statement playing in larger argument
Comclusion/Main Point (3.6%)
Most Strongly Supported (5.6%)
Explain or Resolve (8.4%)
Principle (6.5%
Dispute (3.0%)
Technique (3.6%)
Role (3.2%)
Identify Flaw (11.7%)
identify flaw in an argument's 's reasoning find choice containing argument that exhibits the same flaws the passage's argument
Match Flaws (3.1%)
Match the Structure (3.0%
match the structure of an argument a choice to the structure of the argument in the passage
Others (7.3%)
other types of questions which are not included by the above
Table 3:
The percentage and description of each logical reasoning type. The descriptions are adapted from those specified by Khan Academy (2019).
Poliak et al. (2018), for the tokens in options, we analyze their conditional probability label Te (right, wrong) given by the token by p(l[t) count(t, 1)/ /cound((). The larger the correlation score is for a particular token, the more likely it contributes to the prediction of related option. Table reports tokens in training which occur least twenty times with the highest scores since many of the tokens with the highest scores are of low frequency. We further analyze the lengths of right and wrong options (Gururangan et 2018) in training set. We notice slight difference in the distribution of sentence length for right and wrong options. The average length for wrong options is around 21.82 whereas that for right options is generally longer with an average length of 23.06.
Token
Score (%)
Freq
motive
65.2
23
0.05
right
#ilce
62.5
24
wrong
0.04
thereby
56.0
25
Probablio
consequence
52.4
21
0.03
warm
52.4
0.02
interfere
52.2
23
contributes
52.2
23
0.01
manufacture
52.0
25
0.00
included
52.0
25
20
30
40
50
preferences 52.0
25
Length
Table 5: Top 10 tokens that correlate to right op- tions with more than 20 occurrences.
Figure 3: The distribution of the option length in ReClor with respect to right and wrong labels.
EXPERIMENTS
4.1 BASELINE MODELS
Many neural network based models such as FastText (Joulin et al. 2017). Bi-LSTM, GPT (Rafford al. 2018), GPT-2 (Radford al. 2019), BERT (Devlin al. 2019), XL.Net (Yang al. 2019)

---

Published as a conference paper at ICLR 2020
Context:
the purpose of laws is to contribute to people's happiness, have basis for criticizing existing laws well as proposing new laws. Hence, if that is not the purpose, then we have no basis for the evaluation of existing laws, from which we must conclude that existing laws acquire legitimacy simply because they are the laws The reasoning in the argument is flawed in that the argument
Question:
Options:
A. takes a sufficient condition for state of affairs to a necessary condition for it
B. draws a conclusion about how the world actually is on the basis of claims about how it should be
C. infers causal relationship from the mere presence of correlation
D. trades on the of term in sense in premise and in different sense in the conclusion
Answer:
Reasoning Process of Humans:
We may first look at the question to understand the specific task of the question identify a flaw. We then amalyze the argument in the context. The conclusion 'existing laws acquire legitimacy simply because they are the laws.' based on the argument (purpose is NOT happiness) (NOT basis for criticizing laws), which is obtained from the first statement: (purpose is happiness) - (baxis for criticizing laws). However, we know -A + -B cannot be obtained from A + B. Therefore, we should choose option A that describes this flaw. The distractors here are different types of reasoning flaws. Prior knowledge of basic logical rules is needed to correctly answer this question.
Context:
Psychologist: Phonemic awareness, or the knowledge that spoken language can be broken into component sounds, is essential for learning to read an alphabetic language. But one also needs to learn how sounds are symbolically represented by means of letters; otherwise, phonemic awareness will not translate into the ability read an alphabetic language. Yet many children who are taught by the whole-language method, which emphasizes the ways words sound, learn to read alphabetic languages.
Question: Which one of the following can be properly inferred from the psychologist's statements? Options:
A. The whole- whole-language method invariably succeeds in teaching awareness of how speken language be broken into component sounds.
B. Some children who are taught by the whole-language method are not prevented from learning how sounds are represented by means of letters.
C. The language method succeeds in teaching many children how represent sounds symbolically by means of letters.
D. When the whole-language method succeeds in teaching someone how to represent sounds by means of letters, that person acquires the ability to read an alphabetic language.
Answer: B
Reasoning Process of Humans:
Looking at the question and we know that is asking about implication. From the first two sentences in con- text, we know that there are two necessary conditions to read an alphabetic language: phonemic awareness and symbolic letters. We also learn [(NOT symbolic letters) AND (phonemic awareness) A read an alpha- betic language (denoted Formula 1). The last sentence in the context says that many children are taught by the whole-language method to learn language. As for option A, from the context, we only know the whole language method works for 'many' children, which cannot be inferred to 'invariably' works. As for option B, combing three sentences in the context, we know that the whole-language method meets the two necessary con- ditions to learn a language, especially the last sentence mentions 'learn to read alphabetic languages", Children learn to read alphabetic languages means that they must recognize symbolic letters that represent sound because symbolic letters a necessary condition read an alphabetic language; otherwise, they cannot read because Formula mentioned above. Therefore, option B is correct. As for option C, from context we only know the whole-language method teaches phonemic awareness and read an alphabetic language. Symbolic letters may be taught by other methods, so C wrong, As for D, similar to C, symbolic letters may be taught by other methods and also cannot obtain: symbolic letters read alphabetic language.
Table 4: Two examples to show how humans would solve the questions.

---

Published as a conference paper at ICLR 2020
Context
Q Question- newers
Rossoingg Type
Q. Which ser of the fallowing is an aesumption requized by the aquunet?
Context The cumrent paterr of human consumation of resources which we rely on nonrenewable resources, for ecample
V A.
We cannul indelinitdl replace inhuusted nuarenwwable resources with other nonresovable resources.
must evestually change. Siuce su much metal ure available, altimately, we must tithrr do without or timn to renowable resouroes tike its place.
only
B. Consungtion of torreswoable mesurces will not oondime to Ingrase la the néir hutue.
Necessary A Assumptions
C Thae resowubbr ressuroe replecements for ell of the curmatly beag consumed.
acurmevable
Uitiminlly conosto with vonnerwable
Q The arguments conclesion follows legicilly which one of the following is oesumeed?
Context: Some thecists ugue literary critics should value-neutzil their liteeay cttiiisss These theoriss maintaia that by exposing tae meening of Literayy works without mrai uoting them, crbes will enablc moders make their own judgments about the works' mertss But literary critcismm camnot be completall value seutral. Taus, some thcorists are mistakun ahout what approgriato gool Interayy enticiam.
A.
Any uitii who is alde to help meskes make (hxi von juigments about ltanry wocks notts stoule strre to produce value nestril orliciam.
Sufficiect A Assumptions
v B. If it impoosible to produce completely aduu-neutral literary crilicuu. then crities should not even try to be value neutral.
C. The loss rederrs understand the maaning of itteray work, the less cspable they will evelusting that work's meriss.
D. Crities more likely to precide crticians the works they like works they dislie.
The tolevision show Herry not widely watched until It was scheduled for Tuesday ev eveslings inmeddabelly after That's a Lift, the most, pogular show on inlevision. D During the your after the most watched shows on telentaion. Since Henry' recent move to Wednesday evenings, howaver. it hos been watched by far fewer people. We must ocaclude that Heury was widely
Context:
than provid critiisms
Which the following, true, strongthers the theargumen?
The show now
Life on Tuedayss double sumber viewers had before being
Henny condi stendly
of the ton
B. Henry
the three years, but that's Life has been oz the air faranly too years.
Strengthen
C.
nomi more Wodnesday, Herry was alrod some lime as the secund mast pepular shuw un isteriiuuz
watched Infiome the move Wednesday evenings becansee folllwed 'That s life and not berause people esporialyy liloo it
D. That' Life was not widely watrbed burinn the first it was aired.
Q: Which the following thes most artondly wakkrss the doctors recommmnolation?
Contect: hen certain gland becomes concerous hamons, it produces high levels a particular protein. A blood lest can determine the level of this pevitiia well before a canuer of the gland could be deterted other
A.
for some tinse momter conditor patients who been diagnosod harring ornore shaglind.
The Nlod for protet
Some doctors recommend that aggressive anticancer er treatizen should be begun sarly us possibbl for anyune who is testasd and is found have high levels of the peotsia.
D. Defore the luood bet becume artilable about one third cases of cascet of gland were detected excty siages.
Weskus
C. So faz, no potents whose protiin levels were fornd to be normal have
sabeeguerlly developed concer of the gland
v D. E Enlarqement of the gland, common condition indrequently
associaled with cancer, resultss in high levels of the protein.
2ich the followtng would be most useful to know onder to ereliate the ergumez??
Context: In study; pairs trained dogs were placed, side by side and given tommand such as "sit" Aut both ubeeyad the commandd dog was given trat while its portnor given no reward at all. Over time, the dogs who went unzewarded began to disobey the commundl This shows that. dogs have avcersion being trustad unfairty.
A.
Wore dags who were inpottomed to mostivng wegitar reandds price
to the study more inclized to obey the command!
B. How many repettions were requirel before the unzewardedd dags
Evaluation
bogan to alsobey the command?
v. C. therw dec line obedience rewer ds are withheld from both dogs in the pair??
D. Were dugs who reodived trests one tria used dogs that did
not tmots other ttak??
Figure 2: Examples of some question types. The correct options marked by V. More examples are shown in the Appendix
RoBERTa (Liu et al. 2019) have achieved impressive results in various NLP tasks. We challenge these neural models with ReClor to investigate how well they can perform. Details of the baseline models and implementation are shown in the Appendix and B
4.2 EXPERIMENTS TO FIND BIASED DATA
datasets (Poliak et al. 2018 Gu- rurangan et al. 2018 Zellers al. 2019 Niven & Kao 2019), which are often exploited by models to perform well without truly understanding the text. Therefore, it is necessary to find out the biased data points in ReClor in order evaluate models in a more comprehensive manner (Sugawara et al., 2018). To this end, we feed the five strong baseline models (GPT, GPT-2, BERTBASE: XLNeIBASE and RoBERTaBASE) with ONLY THE ANSWER OPTIONS for each problem. In other words, we purposely remove the context and question in the inputs. In this way, we are able to identify those problems that can be answered correctly by merely exploiting the biases in answer options without knowing the relevant context and question. However, the setting of this task is multiple-choice question with 4 probable options, and even a chance baseline could have 25% probability to get right. To eliminate the effect of random guess, we set four different random seeds for each model and pick the data points that are predicted correctly in all four cases to form the EASY set. Then, the data points which are predicted correctly by the models at random could be nearly eliminated, since any data point only has a probability of (25%)+ 4 = 0.39% to be guessed right consecutively for four times. Then unite the sets of data points that consistently predicted right by each model,
As mentioned earlier, biases prevalently exist in
7

---

Published as a conference paper at ICLR 2020
because intuitively different models may learn different biases of the dataset. The above process is formulated as the following expression,
CEASY
(Capl)
n
GPT n
GGPT n
CGPT
U (Ceeed
0
GPT-2 0 Caeeda n Ceeed
Ceeeda
GGP-22
U
BERT
n
Cseed
BERT
0
BERT
n BERT)
(Ceedd
XLNet
Geedd
XLNNet
(Seedd
XLNet O
Cseeda
(1)
U (Ceeed,
RoBERTa
Cseeda
RoBERT.
Oseeda
RoBERTa
0 RoBERTa
Osscdd
CHARD = CrEST - CEASY;
where BERT denotes the set of data points which are predicted correctly by BERT BASE with seed 1, and similarly for the rest. shows the average performance for each model trained with four different random seeds and the number of data points predicted correctly by all of them. Finally, we get 440 data points from the testing set CTEST and we denote this subset as EASY set CEASY and the other as HARD set CHARD.
Model
Val
Test
Number
Chance
25.0
25.0
3.9
GPT
45.8
42.2
238
GPT-2
46.8
42.6
245
BERT BASE
47.2
43.2
234
XLNetBASE
47.5
43.2
225
ROBERTaBASE
48.8
41.7
200
Union
440
Table 6: Average accuracy of each model using four different random seeds with only answer options as input, and the number of their common correct predictions.
4.3 TRANSFER LEARNING THROUGH FINE-TUNING
Among multiple-choice reading comprehension or QA datasets from exams, although the size of ReClor is comparable to those of ARC (Clark et al. 2018) and DREAM (Sun al. 2019), it is much smaller than RACE Lai et al. (201)). Recent studies (Min et al.) 2017 Howard & Ruder 2018 Huang et al. 2019 Jin et al. 2019) have shown the effectiveness of pre-training on similar tasks datasets then fine-tumin -uming
the target dataset for transfer learning. Pin et al. (019)) find that by first training on RACE (Lai et al. 2017) and then further fine-tuning on the target dataset, the performances of BERT BASE multiple-choice dataset MC500 (Richardson et al. 2013) and DREAM (Sun et al. 2019) can significantly boost from 69.5% to 81. 81.2%, and from 63.2 to 70.2%, respectively. However, they also find that the model cannot obtain significant improvement even performs worse if it is first fine-tuned on span-based dataset like SQuAD (Rajpurkar et al. 2016). ReClor a multiple-choic dataset, so we choose RACE for fine-tuning study.
4.4 RESULTS AND ANALYSIS
The performance of all tested models on the ReClor is presented in Table 7 This dataset is built on questions designed for students who apply for admission to graduate schools, thus we randomly choose 100 samples from the testing set and divide them into ten tests, which are distributed to ten different graduate students in a university. We take the average of their scores and present as carefully chosen and modified from only
the baseline of graduate students. The data of ReClor high-quality questions from standardized graduate entrance exams. We set the ceiling performance to 100% since ambiguous questions not included in the dataset.
The performance of fastText is better than random guess, showing that word correlation could be used help improve performance some extent. It is difficult for Bi-LSTM converge this
8

---

Published as a conference paper at ICLR 2020
Model
Input
RACE
Val
Test
Test-E
Test-H
Chance
(C. Q. A)
25.0
25.0
25.0
FastText
32.0
30.8
40.2
23.4
Bi-LSTM
27.8
27.0
26.4
27.5
(C. A)
GPT
47.6
45.4
73.0
23.8
GPT-2
52.6
47.2
73.0
27.0
(C. Q. A)
54.6
47.3
71.6
28.2
BERTRASE
(C, Q. A)
V
55.2
49.5
68.9
34.3
(A)
46.4
42.4
(Q. A)
48.8
43.4
72.7
20.4
BERTLARGE
(C, Q, A)
53.8
49.8
72.0
32.3
(C, A)
55.6
54.5
73.9
39.3
(C. A)
55.8
50.4
75.2
30.9
XLNetBASE
(C, A) A)
62.0
55.5
76.1
39.3
(A)
45.0
42.9
66.1
24.6
(Q, A)
47.8
43.4
68.6
23.6
XLNctLARGE
(C, A)
62.0
56.0
75.7
(C, Q. A)
70.8
62.4
77.7
50.4
(C, A)
55.0
48.5
71.1
30.7
RoBERTaBASE
(C, A)
56.8
53.0
72.5
37.7
(A)
48.8
43.2
69.5
22.5
(Q, A)
49.8
45.8
72.0
25.2
RoBERTALARCE
(C, Q, A)
62.6
55.6
75.5
40.0
(C, Q,
68.0
65.1
78.9
54.3
Graduate Students
(C, Q, A)
63.0
57.1
Ceiling Performance
(C, A)
100
100
100
Table 7: Accuracy (%) of models and human performance. The column Input means whether to input context (C), question (Q) and answer options (A). The RACE column represents whether to first use RACE to fine-tune before training ReClor.
dataset. Transformer-basee bassed pre- pre-training models have relatively good performance, close the per- formance of graduate students. However, we find that these models only perform well on EASY set with around 75% accuracy, showing these models have an outstanding ability to capture the biases of the dataset, but they perform poorly on HARD set with only around 30% accuracy. In contrast, humans can still keep good performance on HARD set. notice the difference testing accuracy performed by graduate students EASY and HARD but this could be due to the small number of students participated the experiments. Therefore, we say humans perform relatively consistent on both biased and non-biased dataset.
is noticed that the models are first trained on RACE and then fine-tuned on ReClor, they could obtain significant improvement, especially RoBERTaIAARG is even better than that of graduate students. This similar phenomenon can also be observed DREAM dataset (Sun et al., 2019) by Jin al. (2019), which shows the potential of transfer learning for reasoning tasks. However, even after fine-tuning on RACE, the best per- formance of these strong basclines HARD set is around 50%. still lower than that of graduate students and far away from ceiling performance.
HARD set. The overall performance of
Experiments in different input settings are also done. Compared with the input setting of answer options only (A), the setting of questions and answer options (Q, A) can not bring significant im- provement. This may because some questions e.g., Which one of the following an assumption required by the argument?, Which one of the following, if true, most strengthens the argument? can be used in the same reasoning types of question, which could not offer much information. Further adding context causes significant boost, showing the high informativeness of the context.
We further analyze the model performance with respect to different question types of logical rea- soning. Some results are shown in Figure and and the full results are shown in Figure 5 (nnnd] in the Appendix[E] Three models BERTLARGE XLNetLARGE and RoBERTaLARGE perform well on most of types. On HARD the three models perform poorly certain types such STRENGTHEN, WEAKEN and ROLE which require extensive logical reasoning. However, they perform relatively better other certain types, such CON POINT and MATCH STRUCTURES that

---

Published as a conference paper at ICLR 2020
more frrright-forwrd.. For the result of transfer learning, we analyze XLNetLARGE in detail. Though the overall performance is significantly boosted after fine-tuning on RACE first, the his- tograms in the bottom of Figure 4 show that EASY set, accuracy of the model with fine-tuning on RACE is similar to that without it among most question types, while on HARD set, significant improvement on some question types is observed, such as CONCLUSION/MAIN POINT and MOST STRONGLY SUPPORTED. This may be because these types require less logical reasoning to some extent compared with other types, and similar question types may also be found in RACE dataset. Thus, the pre-training on RACE helps enhance the ability of logical reasoning especially of relatively simple reasoning types, but more methods are still needed to further enhance the ability especially that of relatively complex reasoning types.
XLNetuance
RoBERTauacE
BERT
XLNetuaac
ROBERTbuace
100
Accuracy (%6
60
Accuracy
62
60-
20
20
Necenorry Aasumptoos
cuftfe Aocurptoons
Shorpphen
Wealen
E Evaraation
imip lcation
Condiclonno mo
Wast savinl Supported
Exp En Reeate
Peinc
Mendrs Row
Mates ste adurns
others
Necwoorry Assumption
Sufte Aocumptoo
Streggthe
kmp lcation
Conduation Putn
sorungls Supportzd
Expaiio Reeak
Princ pit
Doouute du
Techin que
Rale
Menlfy Fiwe
March Puws
Nutes Sp adurss
oeher
Fine-tune
XLNel Janu
100
Accuracy (36)
02
80-
20
20
Neceusrry Avauruuia
Suptireett Sutfrteet Avoumattonn
Strengthen
Wasken
Exall cander
Insticatiin
Conduzlonoian Paire
Nox Stungh Susparizd
ox Senclee
Hath stuuuurs
other:
Neceuarry Assumattons
Sutteienn Acsumptions
Bireggtben
Insiication
Condustoonltun Pore
Most Stungly Susporrod
ex Revsavee
Prtric pee
Diroute de
Technique
no
idnnlly Sban
Huch ruw
Hitco soudures
others
dondty
Figure 4: Performance of models on EASY (left) and HARD (right) testing sets and that of models.
XLNetLARGG +Fine-Tum means the model is first fine-tuned on RACE before training on ReClor.
CONCLUSION
In this paper, we introduce ReClor, a reading comprehension dataset requiring logical reasoning, with the aim push research progress on logical reasoning in NLP forward from sentence-level to passage-level and from simple logical reasoning to multiple complicated one. We propose to identify biased data points and split the testing set into EASY and HARD group for biased and non-biased data separately. We further empirically study the different behaviors of state-of-the-an models on these two testing sets, and find recent powerful transformer based pre-trained language models have an excellent ability to exploit the biases in the dataset but have difficulty in understand- ing and reasoning given the non-biased data with low performance close to or slightly better than random guess. These results show there is long way to equip deep learning models with real log- ical reasoning abilities. We hope this work would inspire more research in future to adopt similar split technique and evaluation scheme when reporting their model performance. We also show by first fine-tuning on a large-scale dataset RACE then fine-tuning on ReClor, the models could obtain significant improvement, showing the potential of transfer learning solve reasoning tasks.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their insightful comments and suggestions; thank Rishabh Jain from Georgia Tech for helping build up the leaderboard of ReClor on EvalAI. Jiashi Feng was partially supported by NUS IDS MOE Tier-II II R-263-000-D17-112 and ALSG would like to thank TFRC program for the support of computational resources.
ECRA R-263-000-C87-133
Weihao Yu and Zihang Jiang
REFERENCES
Common crawl.
Khan http:///ttte///mmmmmnaaa....rr 2019.
http:///wwwwee..ooooeeee/o/e///a catalog 2019, Accessed Sept. 16, 2019.
Johan Bos and Katja Markert. Recognising textual entailment with logical inference. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 028-635. Association for Computational Linguisties, 2005.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Michael Bugert, Yevgeniy Puzikov, Andreas Rücklé, Judith Eckle-Kohler, Teresa Martin, Eugenio Cartinez-Cámarc Danil Sorokin, Maxime Peyrard, and Iryna Gurevych. Lsdsem 2017: Explor- ing data generation methods for the story cloze test. In Proceedings of the 2nd Workshop Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 56-61, 2017.
Zheng Cai, Lifu Tu, and Kevin Gimpel. Pay attention to the ending: Strong neural baselines for the story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 616-622, 2017.
Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Clueweb09 data set, 2009.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 180:::80044450 2018.
Cleo Condoravdi, Dick Crouch, Valeria De Paiva, Reinhard Stolle, and Daniel G Bobrow. Entail- ment, intensionality and text understanding. In Proceedings of the HLT-NAACL 2003 workshop on Text meaning, pp. 38 38-45, 2003.
Law School Admission Council.
Cormat//ooiical-reaseeee http:///www..aaccc///aaa//aaaatp
2019a. Accessed Sept. 16, 2019.
Law School Admission Council. http:s///wwweeaaccr/////////ttta http
2019b. Accessed Sept. 16, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Miig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning paragraphs. In Proceedings of NAACL-HLT, pp. 2368-2378, 2019.
Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. A natural logic inference system. In Pro- ceedings of the 2nd Workshop Inference in Computational Semantics (ICoS-2). Citeseer, 2000.
11

---

Published as a conference paper at ICLR 2020
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts natural language inference data. arXiv preprint arXiv:1803.02324, 2018.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human Language Technologies, Volume / (Long Papers), pp. 1930-1940, 2018.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-339, 2018.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-JJCNLP), pp. 2391-2401, 2019.
Di Jin, Shuyang Gao, Jiun- Yu Kao, Tagyoung Chung, and Dilek Hakkani-tur. Mmm: Multi-stage multi-task learning for multi-choice reading comprehension. arXiv preprint 2019.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics; Volume 2, Short Papers, pp. 427-431. Association for Computational Linguistics, April 2017.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies. Volume 1 (Long Papers), pp. 252-262, 2018.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Trunsactions of the Association for Computational Linguistics, 6:317-328, 2018.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: vrXivv:704.000 170400468, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Bill MacCartney and Christopher D Manning. An extended model of natural logic. In Proceedings of the eighth international conference on computational semantics, pp. 140-156. Association for Computational Linguistics, 2009.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, arXiv preprint arXiv:1809.02739 027899 2018.
Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. Question answering through transfer learn- ing from large fine-grained supervision data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 510-517, 2017.
Timothy Niven and Hung-Yt Yu Kao. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Asxociation for Computational Linguistics, pp. 4658-4664, 2019.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword fifth edition, 2011.
12

---

Published as a conference paper at ICLR 2020
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines natural language inference. In Proceedings of the Seventh Joint Conference Lexical and Computational Semantics, pp. 180-191, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ily Sutskever. Improving language un-
derstanding by generative pre-training. URL hrtps://s3-ss-westt amazonaws. com/openai- understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188), 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 100,05500,,, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions 2018.
for squad. arXiv preprint arXiv: arXiv:/806.nm
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 193-203, 2013.
Alvaro Rodrigo, Anselmo Penas, Yusuke Miyao, Eduard H Hovy, and Noriko Kando. Overview of clef qa entrance exams task 2015. In CLEF (Working Notes), 2015.
Roy Schwartz, Maarten Sap, loannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. Story cloze task: Uw nlp system. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics. pp.
Hideyuki Shibuki, Kotaro Sakamoto, Yoshinobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y Itakura, Wang, Tatsunori Mori, and Noriko Kando. Overview of the ntcir- qa-lab task. In Nicir, 2014.
Saku Sugawara and Akiko Aizawa. An analysis of prerequisite skills for reading comprehension. In Proceedings of the Workshop on Uphill Battles Language Processing: Scaling Early Achieve- ments to Robust Methods, pp. 1-5, 2016.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. What makes reading comprehen- sion questions easier? In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4208-4219 4219, 2018
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. Dream: A challenge data set and models for dialogue-based reading comprehension. Trunsactions of the Association for Computational Linguistics, 7:217-231, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 253-355, 2018
Johannes Welbl, Pontus Stenetorp. and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A croad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122, 2018.
13

---

Published as a conference paper at ICLR 2020
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: /44, 2016.
Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranject Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra. Evalai: Towards better evaluation systems for ai agents. arXiv preprint arXiv:1902.03570, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08.. 082377 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rXiv:1905.07880 2019.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 1810. 118881...888.. /2885, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 29-27, 2015.
14

---

Published as a conference paper at ICLR 2020
A BASELINE MODELS
fastText. FastText (Joulin et al. 2017) models sentences bag of n-grams, and tries to predict the probability of each answer being correct independently. We choose the answer with the highest score the prediction for the muultiplechhiie setting.
LSTM sentence encoder. A two-layer bi- bi-LSTM is randomly initialized as a sentence encoder with Gl word embedding (Pennington et al. 2014). With span of text imput, the last hidden state of the second layer is max-pooled and then fed into fully-connected layer to compute the output score.
GPT and GPT-2. GPT (Rafford et al., 2018) and GPT-2 Radford et al., 2019) are both trans- former (Vaswani et al. 2017) based models which pre-trained using unsupervised method with a standard language modeling objective. GPT is pre-trained on BooksCorpus; GPT-2 is pre-trained using a larger dataset called WebText, Here we use the smallest model proposed in (Radford al. 2019) as our GPT-2 baseline. To fine-tune on ReClor, the final hidden vector corresponding to the last input token (C c-alass fy 11) used as the aggregate representation followed by an extra fully connected layer to compute the score.
BERT. BERT (Devlin et al., 2019) is also a transformer (Vaswani al. 2017) based model which is trained by using BooksCorpus (Zhu et al.) 2015) and English Wikipedia in two unsupervised tasks, i.e. Masked LM (MLM) and Next Sentence Prediction (NSP). During fine-tuning, the final hidden vector corresponding to the first input token ([CLS)) ) is used as the aggregate representation followed by two extra fully connected layers to compute the score.
XLNet. XLNet (Yang al. 2019) is trained with Permutation Language Modeling and without NSP. In addition, beside BooksCorpus and English Wikipedia used BERT, it uses Giga5 (Parker et al. 2011), ClueWeb 2012-B B (extended from (Callan et al., 2009)), and Common Crawl (com 2019 for pre-training. We use the final hidden vector corresponding to the last input token <els> the aggregate representation and introduce two fully connected layers predict the score.
RoBERTa. RoBERTa (Liu et al. 2019) is an improved pre-training procedure of BERT with training the model longer, with bigger batches over more data and removing NSP objective etc.. Extra two fully connected layers are added to transform the final hidden vector of the first input token (
 the score.
The input format of different models shown in Table
Model
Input Format
CPPTRaddod al 2018
.start. Context dellmiter. Question Option .classily. start Context delimiter Question option classify [CLS] Context [SEP] Question II Option [SEF] (PAD]... <pad>... Coatext <sep> Question Option <aep> <ele> <a>> conzext </>> 
 Question II Option </>> <pad>...
GP Kadlerd
2019
BERT Devina el a. 2019
XLNNt Yong 2019
RoBERTa Lau st.
Table 8: Input formats of different models. Context, Quest and Opt on represent the token sequences of the context, question and option respectively, and denotes concatenation.
B IMPLEMENTATION DETAIL
Adam is used by all models. For fastText, we use its python library by converting ReClor to the required form, and keep the default setting of the hyper parameters. For Bi-LSTM, we use a two- layer Bidirectional LSTM with the GloVe 300d word embedding (Pennington et al. (014)) followed by max-poot max-pooling and fully-connected layer. We train the model for 100 epochs using a batch size of 64 and learning rate of 0.1. A learning rate decay of 0.5 is also applied every 10 epochs. For pre-training models, we modify the code of Transformers of Hugging Face to implement them on ReClor. We use a batch size of 24 and fine-tune for 10 epochs. The maximum input sequence length for all models is 256. The detailed hyperparameters are shown in Table
87
15

---

Published as a conference paper at ICLR 2020
HYPERPARAM
GPT 6.25p-3
GPT-2 6.252-
BERT 1r.3
BERTLAGGE 20.5
XLNeffASE 20-5
ROBERTAASS XLNNttARGE
RoBERTALARGE
Rate
2e.5
Je.5
Te.5
Size
Soq Longth
256
Learning Rate Decay
Lincur
Number of Epochts
Proportion
Weight Decay
0.01
0.01
0.01 Ic-6 (0.9 0.999)
0.01 1c-6 (0990.9))
0.01
Adam Epsilon
1c-8
lc-8 (0.9.0999)
Ic-6
Ic-6 (0.9, 0.999)
Ic-6
Ic-6 (0.9009))
Adam Betas
(0.9, 0,999)
(0.9, 0,999
(0.9, 0.999)
Clip
Table 9: Hyperparameters for finetuning pre-training language models on ReClor
C EXAMPLES
Type:
Necessary Assumptions
Definition: identify the claim that must be true or is required in order for the argument to work
Context:
Slash-and-burn agriculture involves burning several acres of forest, leaving vegetable ash that provides ample fertilizer for three or four years of bountiful crops. On the cleared land nutrients leach out of the soil, however, and the land becomes too poor to support agriculture. New land is then cleared by burning and the process starts again. Since most farming in the tropies uses this method, forests in this region will eventually be permanently eradicated.
Question:
The argument depends on assumption that
Options:
A. forests in the tropics do not regenerate well enough to restore themselves once they have been cleared by the slash-and-burt method some other methods agriculture are not as destructive the environment in tropical regions as the slash-and and-burn method
C. forests in the tropics are naturally deficient in nutrients that are needed to support the growth of plants that are not native to those regions
D. slash-and-burn agriculture is particularly suitable for farming tropical areas
Answer:
Table 10: The definition and example of the logical reasoning type Necessary Assumptions
Type:
Sufficient Assumptions
Definition: identify a sufficient assumption, that is, an assumption that, added to the argument, would make it logically valid
Context:
Geologist: new method for forecasting carthquakes has reliably predicted several carthquakes. Unfor- tunately, this method can predict only that an earthquake will fall somewhere within range of two and a half points on the Richter scale. Thus, since difference of two and a half points can be the difference between marginally perceptible shaking and quake that causes considerable damage, the method unlikely useful.
Question: Which one of the following, assumed, enables the geologist's conclusion to be properly inferred?
Options:
A. An nurthquake-forecastio method unlikely he useful unless its predictions always differentiate earthquakes that are barely noticeable from ones that result substantial destruction.
B. Several well-established methods for forecasting earthquakes can predict within much narrower ranges than two and half points the Richter scale.
C. Even if an wethhhuaak--ffrcssttii method makes predictions within very narrow range the Richter scale, this method is likely useful unless its predictions are reliable. D. An seahhuuak--frrcassttie method has not been shown to be useful until it has been used to reliably predict large number of carthquakes. Answer: A
Table 11:
The definition and an example of the logical reasoning type Sufficient Assumptions
16

---

Published as a conference paper at ICLR 2020
Type:
Strengthen
Definition:
identity information that would strengthen argument
Context:
Financial success does not guarantee happiness. This claim is not mere proverbial wisdom hut a fact verified by statistics. In a recently concluded survey, only one-third of the respondents who claimed to have achieved financial success reported that they were happy. Question: Which one of the following, true, most strongly supports the conclusion drawn from the survey results?
Options:
A. Most of the respondents who reported they were unhappy were in fact happy.
The respondents who reported financial success were, for the most part, financially successful.
Many of the respondents who claimed not to have achieved financial success reported that they were
happy five years ago.
D. Many of the respondents who failed to report financial success were in fact financially successful.
Answer:
Table 12: The definition and an example of the logical reasoning type- Strengthen
Type: Weaken
Definition: identify information that would weaken an argument
Context:
"DNA fingerprinting" is recently-introduced biochemical procedure that uses pattern derived from person' genetic material match suspect' s genetic material against that of specimen from crime scene. Proponents have claimed astronomically high odds against obtaining match by chance alone. These odds are based on an assumption that there is independence between the different characteristics represented by single pattern.
Question: Which one of the following, if true, casts the most doubt the claim of the proponents of DNA fingerprinting?
Options:
A. The skill required of laboratory technicians performing the DNA fingerprinting procedure is not ex- traordinary.
B. There is generally accepted theoretical basis for interpreting the patterns produced by the procedure. In the whole population there are various different subgroups, within each of which certain sets of genetic characteristics are shared.
D. In the investigation of certain genetic discases, the techniques used in DNA fingerprinting have traced the transmission of the diseases among the living members of very large families.
Answer: C
Table 13: The definition and an example of the logical reasoning type- - Weaken
Type:
Evaluation
Definition:
identify information that would be useful to know to evaluate an argument
Context:
George: Some scientists say that global warming will occur because people releasing large amounts of carbon dioxide into the atmosphere by burning trees and fossil fuels. We can see, though, that the predicted warming is occurring alrcady. the middle of last winter, we had month springlike weather in our area, and this fall, because of unusually mild temperatures, the leaves on our town' trees were three weeks late in turning color.
Question:
Which one of the following would be most relevant investigate in evaluating the conclusion George's argument?
Options:
A. whether air pollution is causing some trees in the area to lose their leaves
B. what proportion of global emissions of carbon dioxide is due to the burning of trees by humans
whether unusually warm weather is occurring elsewhere on the globe more frequently than before when leaves on the trees in the town usually change color
Answer: C
Table 14: The definition and an example of the logical reasoning type Evaluation
17

---

Published as a conference paper at ICLR 2020
Type: Implication
Definition:
identify something that follows logically from a set of premises
Context:
horrific, monster must be threatening. Whether or not presents psychological, moral or social dangers, or triggers enduring infantile fears, if a monster is physically dangerous then is threatening. fact, even a physically henign monster is horrific if it inspires revulsion.
Question:
Which one of the following logically follows from the statements ahove?
Options:
A. Any horror-story monster that threatening also horrific.
B. If a monster triggers infantile fears but is not physically dangerous, then it is not horrific.
C. All monsters that are not physically dangerous, but that are psychologically dangerous and inspire revulsion, are threatening.
If a monster both horrific and psychologically threatening, then does not inspire revulsion.
Answer:
Table 15: The definition and an example of the logical reasoning type Implication
Type: Conclusion/Main Point
Definition: identify the conclusion/main point of line of reasoning
Context:
Whether or not one can rightfully call a person's s faithfulness a virtue depends in part on the object of that
person' faithfulness. Virtues are by definition praiseworthy. which is why no one considers resentment virtuous, even though it is in fact kind of faithfulness faithfulness to hatreds or animosities.
Which one of the following most accurately expresses the overall conclusion drawn in the argument?
Question:
Options:
A. The object of person's 's faithfulness partially determines whether or not that faithfulness is virtuous.
B. Virtuous behavior is praiseworthy by definition.
Resentment should be considered virtuous emotion.
D. Behavior that emerges from hatred or animosity cannot be called virtuous.
Answer: A
Table 16: The definition and an example of the logical reasoning type Conclusion/Main Point
Type:
Most Strongly Supported
Definition:
find the choice that most strongly supported by a stimulus
Context:
After a nuclear power plant accident, researchers found radioactive isotopes of iodine, tellurium, and cesium-but heavy isotopes-in the atmosphere downwind. This material came either from spent fuel rods from the plant' core. Spent fuel rads Radioactive material ejected into the atmosphere directly from the core would include heavy isotopes. After the accident, steam, which may have been in contact with the core, was released from the plant. The core contains iodine, tellurium, and cesium isotopes, which easily dissolved by steam.
contain significant quantities of tellurium isotopes.
Question:
Of the following statements, which one most strongly supported by information above?
Options:
A. The nuclear power plant's ' spent fuel rods were not damaged.
B. Spent fuel rods do contain hcavy isotopes in significant quantitics.
The rescarchers found some radioactive material from spent fuel rods as well some material that was ejected into the atmosphere directly from plant's 's core.
D. The radioactive material detected by the researchers was carried into the atmosphere by the steam that
was released from the plant.
Answer: D
Table 17: The definition and an example of the logical reasoning type Most Strongly Supported
18

---

Published as a conference paper at ICLR 2020
Type: Explain or Resolve
Definition: identify information that would explain or resolve situation
Context:
To reduce the mosquito population in resort area, hundreds of trees were planted that bear fruit attractive birds. Over the years, as trees matured, they attracted a variety of bird species and greatly increased the summer bird population in the area. As expected, the birds ate many mosquitoes. However, the planting of the fruit trees had the very opposite of its intended effect.
Question:
Which one of following, true, most helps explain the apparently paradoxical result?
Options:
A. Most of the species of birds that were attracted by the trees that were planted did not cat mosquitoes.
B. Increases and decreases in mosquito populations tend follow eyclical pattern.
C. The species of birds that were attracted in the greatest number by the fruit of the trees that were planted did not eat mosquitoes.
D. The birds attracted to the area by the trees ate many more insects that prey on mosquitoes than they did mosquitoes. Answer: D
Table 18: The definition and an example of the logical reasoning type Explain or Resolve
Type: Principle
Definition: identify the principle, or find a situation that conforms to principle, or match principles
Context:
Buying elaborate screensavers programs that put moving images computer monitor to prevent damage- can oost company far in employee time than it saves in electricity and monitor protection. Employces cannot resist spending time playing with screensavers that flash interesting graphics across their screens.
Question:
Which one of the following most closely conforms to the principle illustrated above?
Options:
A. An electronic keyboard may be cheaper to buy than a piano but more expensive to repair.
B. An meegyy-fffcient insulation system may cost more front but will ultimately save money over the
life of the house.
The time that it takes to have pizza delivered may be longer than it takes cook complete dinner.
D. A complicated hotel security system may cost more customer goodwill than it saves in losses by theft.
Answer: D
Table 19: The definition and an example of the logical reasoning type Principle
Type: Dispute
Definition: identify or infer an issue dispute
Context:
Raphacla: Forcing people to help others is morally wrong. Therefore, no government has the right to redistribute resources via taxation. Anyone who wants help others voluntarily. Edward: Govermments do have that right, insofar they give people the freedom to leave and hence live under their authority.
Question:
Raphaela and Edward disagree about the truth of which one of the following?
Options:
A. Any government that forces people help others should permit emigration.
Any government that permits emigration has the right to redistribute resources via taxation.
C. Any government that redistributes resources via taxation forces people to help others.
D. Every government should allow people to help others voluntarily.
Answer:
Table 20: The definition and an example of the logical reasoning type Dispute
19

---

Published as a conference paper at ICLR 2020
Type:
Technique
Definition:
identify the technique used in the reasoning of argument
Context:
Joanna: The only way for a company to be successful, after emerging from bankruptcy, is to produce the same goods or services that it did before going bankrupt. It is futile for such a company to try to Iearn a whole new business. Ruth: Wrong. The Kelton Company bankruptcy. emerging from bankruptcy, Kelton turned its mines into landfills and presently a highly successful mantemmmn concem.
major mining operation that went into
Question:
Ruth uses which one of the following argumentative techniques in countering Joanna's 's argument?
Options:
A. She undermines a claim by showing that rests on an ambiguity.
B. She offers an alternative explanation for phenomenon.
C. She presents a counterexample to claim.
D. She establishes conclusion by excluding the only plausible alternative that conclusion.
Answer:
Table 21: The definition and an example of the logical reasoning type Technique
Type:
Role
Definition: describe individual role that statement playing a larger argument
Context:
The position that punishment should be proportional to how serious the offense is but that repeat offenders should receive harsher punishments than first-time offenders is unsustainable. implies that considera- tions as remote as what an offender did years ago are relevant remote considerations were relevant, almost every other consideration would be too. But this would make determining the seriousness of an offense so difficult that it would be impossible to apply the proportion- ality principle.
seriousness of an offense. such
Question:
The statement that considerations as remote as what an offender did years ago are relevant to the serious- ness of an offense plays which one of the following roles in the argument?
Options:
A. It is allegedly untenable consequence of view rejected in the argument's overall conclusion. statement the argument provides grounds accept and from which the overall conclusion is inferred.
C. It is the overall conclusion in favor of which the argument offers evidence.
D. It is premise offered in support of an intermediate conclusion of the argument.
Answer:
Table 22: The definition and an example of the logical reasoning type Role
Type: Identily Flaw
Definition: identify a flaw in an argument's reasoning
Context:
The tidal range at particular location is the difference in height between high tide and low tide, Tidal studies have shown that of the greatest tidal ranges in the world is found in the Bay of Fundy and reaches more than seventeen meters. Since the only forces involved inducing tides are the sun' moon't s gravity, the magnitudes of tidal ranges also must be explained entirely by gravitational forces.
Question:
Which one of the following most accurately describes flaw in the reasoning above?
Options:
does not differentiate between the tidal effect the sun the tidal effect of the moon.
B. It fails to consider that the size of a tidal range could be affected hy the conditions in which gravitational
forces act.
C. It presumes, without providing warrant, that most activity within the world's oceans result of interplay gravitational forces.
D. It gives only one example of a tidal range.
Answer: B
Table 23: The definition and an example of the logical reasoning type Identify Flaw
20

---

Published as a conference paper at ICLR 2020
Type: Match Flaws
Definition:
find a choice containing an argument that exhibits the same flaws as the passage's 's argument
Context:
The museum' night security guard maintains that the thieves who stole the portrait did not enter the museum at any point above ground level. Therefore, the thieves must have gained access to the museum from below ground level.
Question:
The flawed pattern of reasoning in the argument above is most similar to that in which one of the follow- ing?
Options:
A. As had generally been expected, not all questionnaires were sent in by the official deadline. follows that plans must have been made for the processing of questionnaires received late.
B. The store's competitors claim that the store, in selling off the shirts those prices, neither made any profit broke even. Consequently, the store's customers must have been able to buy shirts there at less than the store's 's cost.
C. The product label establishes that this insecticide is safe for both humans and pets. Therefore, the insecticide must also be safe for such wild mammals as deer and rabbits.
D. If the census is to be believed, the percentage of men who are married is higher than the percentage of women who are married. Thus, the census must show a higher number of men than of women overall, Answer:
Table 24: The definition and an example of the logical reasoning type- Match Flaws
Type: Match the Structure
Definition: match the structure an argument a choice to the structure of the argument in the passage
Context:
It is an absurd idea that whatever artistic endeavor the government refuses to support it does not allow, one see by rephrasing the statement read: No one allowed to create art without government subsidy.
Question:
The pattern of reasoning in which one of the following is most similar to that in the argument above?
Options:
A. The notion that every scientist who has been supported by a government grant will be successful is absurd, as one can see by rewording it:No scientist is allowed to do research without a government grant.
B. The notion that every scientist who is supported by a government grant will be successful is absurd, as one can see by rewording it:No scientist lacking governmental support will be successful.
C. The claim that any driver who is not arrested does not break the law is absurd, one can see by
rewording it: Every driver who gets arrested has broken the law.
D. The claim that any driver who is not arrested does not break the law is absurd, as one can see by rewording it: Every driver who breaks the law gets arrested.
Answer: D
Table 25: The definition and an example of the logical reasoning type- Match the Structure
Type: Others
Definition: other types of questions which are not included by the above
Context:
PhishCo runs a number of farms the arid province of Nufa, depending largely on irrigation. Now, as part of a plan to efficiently increase the farms' total production, it plans to drill down to an aquifer containing warm, slightly salty water that will be used to raise fish in ponds. The water from the ponds will later be used to supplement piped-in irrigation water for PhishCo's vegetable fields, and the ponds and accompanying vegetation should help reduce the heat in the area of the farms.
Question:
Which of the following would, if true, most strongly suggest that the plan, if implemented, would increase the overall efficiency of PhishCo's 's farms?
Options:
A. Organic waste from fish the pond water will help to fertilize fields where is used for irrigation.
B. Fish raised on PhishCo's 's farms are likely to be salcable in the nearest urban areas.
C. Ponds will be located on low-lying land now partially occupied by grain crops.
D. The government of Nufa will help arrange loan financing to partially cover the costs of drilling.
Answer: A
Table 26: The definition and an example of the logical reasoning type Others
21

---

Published as a conference paper at ICLR 2020
D CONSISTENCY OF DIFFERENT MODELS
GPT
GPT-2
BARTBASE
XLNetBASE
RoBERTaRASE
GPT
245
164
152
142
116
GPT-2 BERTBASE
238
151
144
123
234
138
124
XLNeIBASE RoBERTaBASE
225
125
200
Table 27: Overlap of each pair of models after intersection among 4 random seeds.
22

---

Published as a conference paper at ICLR 2020
E RESULTS WITH RESPECT TO DIFFERENT QUESTION TYPES
fastText BI-LSTM
BERT BERT
Ne XLNetuance
RoBERTa RoBERTaunnc
GPT-2
Necessary Assumptions
Sufficient Assumptions
Strengthen
Weaken
Evaluation
Implication
ConclusioryMain Point
Most Strongly Supported
Explain or Resolve
Principle
Dispute
Technique
Role
Identify
Match Flaws
Structures
Others
20
40
60
80
100
Accuracy (96)
Figure 5: Accuracy of all baseline models overall testing
23

---

Published as a conference paper at ICLR 2020
fastText BI-LSTM
GPT
BERT
XLNetaas:
RaBERTaga RaBERTa.ARC
GPT-2
BERT
XLNetyaer
Necessary Assumptions
Sufficient Assumptions
Strengthen
Weaken
Evaluation
Implication
ConclusioryMain Point
Most Strongly Supported
Explain Resolve
Principle
Dispute
Technique
Identify Flaw
Match Structures
Others
20
60
100
Accuracy (96)
Figure 6: Accuracy of all baseline models on EASY set of testing set
24

---

Published as a conference paper at ICLR 2020
fastText BI-LSTM
GPT GPT-2
BERToAs
XLNetaas:
RaBERTagaa
BERT
XLNetuaer
RaBERTa.ARC
Necessary Assumptions
Sufficient Assumptions
Strengthen
Weaken
Evaluation
Implication
ConclusioryMain Point
Most Strongly Supported
Explain Resolve
Principle
Dispute
Technique
Identify Flaw
Match Structures
Others
20
60
100
Accuracy (96)
Figure 7: Accuracy of all baseline models on HARD set of testing set
25

---

Published as a conference paper at ICLR 2020
100
Accura 196)
20
Sufttri www.mppoopeooooo
Strenghhem
Wessken
imsl canoo
Concus Monnaa Painz
Med Stongh Supportc
Panddee
awo
others
Neroenarry Assumptinns
Prevdiee
Dioouee
Technique
loune Ron
maaco Snucurss
others
Stoungly Supoo shaio
Idart
Math
Sufffrien::
RoBERTauace
ROBERTBuKGa
RoBERTaunnz
RoBERTauNGs
100
100
Accuracy (%)
20
Mecenaary Assurations
cumtee and Aesumptions
Grreghhee
Weaten
Exatuattion
Impitaation
Wust seongl Supporso
Besto
Pranc
Flo
Onher
Mecencary Necumbtions
suffee Ascumitions
Grrogghee
Mussseundd Supporoo
Expan Beeo
Princ phople
Dippttc
Technique
Wonbly Ron
watch
Nozs Scoucura
Figure 8: Performance of BERT (top) and ROBERTaLARGE (bottom) on EASY (left) and
HARD (right) testing sets.
26


Reviews:
[{"review_id": "r1lhnRVTOB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a small multiple choice reading comprehension dataset drawn from LSAT and GMAT exams.  I like the idea of using these standardized tests as benchmarks for machine reading, and I think this will be a valuable resource for the community.\n\nThe two major concerns I have with this paper are with its presentation and with the quality of the baselines.  These concerns leave me at a weak accept, instead of a strong accept, which I would be if these issues were fixed.\n\nPresentation:\n\nThe main contribution here is in collecting a new dataset drawn from the LSAT and GMAT.  (I assume that is all that is used, though the text actually says \"such as\".  Or is this from practice exams and not the actual exams?)  The job of this paper is to convince me, a researcher focusing on reading comprehension, that I should use this dataset as a target of my research.  Out of 8 pages, however, at most 2 are devoted to actually describing this contribution and why it's a good target for reading comprehension.  Much of the interesting description of the phenomena in the dataset is relegated to the appendix, which I did not really look at because it far exceeds the page limit of an ICLR submission.  I have a lot of questions about this dataset that could have been answered in the main text had more of the paper been given to actually describing the data.  Such as:\n\n- What are some actual examples of the different kinds of reasoning in table 2?\n- How did you decide the proportions of questions listed in table 2?  Was that manual?\n- Where did the questions actually come from?  Real exams?  Practice exams?  Which ones?\n- What is the reasoning process that a person would actually use to answer some of these questions?  Preferably answered for several of the question types that you listed.  You have 8 pages; there's a lot of space that could be given to this.\n- What kinds of things do you see in the distractors that make this hard?\n\nI'm recommending a weak accept for this paper, but only because of my background knowledge about the LSAT and GMAT, and my prior understanding of reading comprehension and why this would be a really interesting dataset.  I think the paper itself could do a *much* better job convincing a reader about the dataset than what is done here.\n\nWhat should be cut to make room for this?\n\nI think far too much space is given to the discussion of easy vs. hard (~2.5 pages).  Given that the best performance is at ~54% on the full test set, there isn't a lot of need for this.  The argument about four flips of a 25% coin giving a 0.39% chance of guessing right every time is only true if the coin flips are independently drawn; training the same pretrained model on the same data with a different random seed is clearly not independent, so this argument rings hollow.  It's not really clear what to conclude from the easy vs. hard split, other than the models that you used to create it expectedly do well on the easy split and hard on the test split.  You'd want to have separate models that create the split than what you're evaluating them with, but even then, you're training them on the same data, so it's still not really clear what to conclude.  It's sufficient in a case like this to just evaluate as a baseline a model that shouldn't have access to enough information to solve the task, and measure its performance on the test set.  That would have taken a few sentences to describe, instead of 2.5 pages.\n\nYou could also recover a lot of space by evaluating fewer baselines.  Describing six baseline models and including all of their performance numbers takes up a lot of space, and we really don't need to know what all of those numbers are.  One or two, along with question-only and option-only baselines, would be plenty.  If you really want to include all of them, that's information that should go into the appendix, instead of something that's core to your paper's contribution, like describing the dataset that you're releasing.\n\nQuality of baselines:\n\nWith a dataset this small, you definitely want to pretrain BERT / RoBERTa on a larger QA dataset before training on ReClor.  I would guess that performance would be significantly higher if you used a RACE-trained model as a starting point, instead of raw BERT / RoBERTa.  And if it doesn't end up helping, then you've made a stronger argument about how your dataset is different from what has come before, and a good target for future research.  What I would recommend for table 6:\n\n- chance\n- option-only\n- question-only\n- raw XLNet or RoBERTa\n- XLNet or RoBERTa pretrained on RACE, or SQuAD, or similar\n\nAnd I would pick either XLNet or RoBERTa based on which one did the best after pre-training.  That way, related to my first point, you can remove the description of all other models, and table 4, and free up a bunch of space for more interesting things.\n\nAs an aside, I think it's good that the dataset is relatively small, as it gives these over-parameterized models less opportunity to overfit, and it makes us come up with other ways of obtaining supervision.  But you've got to be sure to use the best baselines available, and with small datasets, that means more pretraining on other things.\n\nEDIT Nov. 19, 2019: The authors' revision has satisfied my concerns.  I think that this is a good paper and it should be accepted.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1gcuJ9qtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPaper Summary:\n\nThis paper presents a machine reading comprehension dataset called ReClor. It is different from existing datasets in that ReClor targets logical reasoning. The authors identified biased data points and separated the testing dataset into biased and non-biased sets. Experimental results show that state-of-the-art models such as XLNet and RoBERTa struggle on the non-biased HARD set with poor performance near that of random guess.\n\nStrengths:\n\n—The dataset, which is extracted from standardized tests such as GMAT and LSAT, requires the ability to perform complex logical reasoning. It is difficult for crowdsourcing workers to generate such logical questions.\n\n—The authors carefully analyzed the biases, which are often exploited by models to achieve high accuracy without truly understanding the text.\n\n—The authors thoroughly investigated how well strong models such as RoBERTa can perform.\n\n—The paper is well organized and well written.\n\nWeaknesses:\n\n—The dataset seems small to acquire the ability to perform complex logical reasoning. The training, validation, and testing datasets consist of 4,651, 500, and 1,000 questions, respectively.\n\n—The paper does not show statistics of the dataset such as question/passage length and question vocabulary size.\n\n—The paper does not show results of models trained with other reading comprehension datasets such as RACE and fine-tuned with ReClor.\n\n—Unlike other datasets, the questions themselves show their required logical reasoning types in ReClor. For example, the question “Which one of the following is an assumption required by the argument?” shows the “Necessary Assumptions” type and has no information with respect to passages. This characteristic makes it difficult to use the ReClor dataset as an evaluation benchmark for models trained with large-scaled reading comprehension datasets such as RACE.\n\n—The human performance with respect to different question types of logical reasoning is not analyzed in Figure 3.\n\nReview Summary:\n\nThe paper is well motivated. ReClor can be a useful dataset to evaluate the ability of logical reasoning, while the dataset seems small to acquire the ability to perform complex logical reasoning.   I think it can benefit a lot with a more comprehensive analysis of transfer learning with other reading comprehension datasets.\n\nEDIT Nov. 20, 2019: \nI appreciate the authors' revision.  \nAlthough my concern about the size of the dataset is not satisfied, I decided to increase the score of the paper (weak reject -> weak accept) upon looking at the author response about transfer learning.\n\nThere is a typographical error, in Table 2.  ACR -> ARC.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJe7pOKoKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new reading comprehension dataset for logical reasoning. It is a multi-choice problem where questions are mainly from GMAT and LSAT, containing 4139 data points. The analyses of the data demonstrate that questions require diverse types of reasoning such as finding necessary/sufficient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation. The paper includes comprehensive experiments with baselines to identify bias in the dataset, where the answer-options-only model achieves near half (random is 25%). Based on this result, the test set is split into the easy and hard set, which will help better evaluation of the future models. The paper also reports the numbers on the split data using competitive baselines where the models achieve low performance on the hard set.\n\nStrengths\n1) The introduced dataset is timely and is more challenging than existing reading comprehension datasets, based on the examples presented in the paper.\n2) Analyses of reasoning types (Table 2) are done in a very comprehensive way, especially because they are comparable with descriptions from previous literature. This analysis clearly demonstrates that this data requires diverse types of challenging reasoning.\n3) The experiments are comprehensive with many competitive baseline models, and their efforts to identify bias and use the result to split the test set are impressive.\n\nWeaknesses\n1) Although the main claim of this paper is about logical reasoning, I believe the term “logical reasoning” is somewhat subjective. What exactly is the definition of logical reasoning? The paper never defines it in any way; it only shows Table 1 as an example.\n2) (Continuing the point above) The paper mentions other datasets’ statistics on logical reasoning (in Section 1, they mention “0% in MCTest dataset and 1.2% SQuAD” requires logical reasoning). How was this analysis done?\n3) (Continuing the point above) Are all types of reasoning in Table 2 logical reasoning? I agree that those reasoning are very challenging, but I am not convinced why some of them belong to logical reasoning (E.g. summary). I expect this problem can be resolved if authors bring a clearer definition of logical reasoning.\n4) Although I appreciate the experiments to identify bias, I believe that the bias seems to be somewhat significant if 440 out of 1000 examples are identified as biased examples. I believe that the authors should do more comprehensive analysis on biased examples and explain what are possible biases here, in order to justify the significant bias.\n\nMarginal comments\n1) The authors claim they collected the questions from open websites and books, but they didn’t provide the URLs or names. More information about which source was used, how data points were filtered and so on would be necessary to assess the quality of the dataset. Specifically, is there any particular reason for removing one of wrong options to make it have four choices, instead of five choices?\n2) Why is the number of Chance model 0.39? Shouldn’t it be 1000 * 0.39% = 3.9?\n3) In Table 5, it’s interesting to see that different models identify different numbers of biased examples. Have you looked at how much do they overlap each other if you take a pair of models? (A number of the union kinda shows it, but I wonder if there is a more intuitive way to see how consistent the models’ decisions are.)\n4) It would be also helpful to see the baseline results of “question & answer options only model” and “context & answer options only model”.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper presents a new reading comprehension dataset for logical reasoning. It is a multi-choice problem where questions are mainly from GMAT and LSAT, containing 4139 data points. The analyses of the data demonstrate that questions require diverse types of reasoning such as finding necessary/sufficient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation. The paper includes comprehensive experiments with baselines to identify bias in the dataset, where the answer-options-only model achieves near half (random is 25%). Based on this result, the test set is split into the easy and hard set, which will help better evaluation of the future models. The paper also reports the numbers on the split data using competitive baselines where the models achieve low performance on the hard set.

--

Discussion:

While the authors agree this is an important direction, there are reservations concerning the small size of the dataset, that have not been fully addressed.

--

Recommendation and justifcation:

I still believe this paper should be accepted as the existing datasets for reading comprehension are inadequate and it is important for the field not to be climbing the wrong hill.
Decision: Accept (Poster)