Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a new reading comprehension dataset for logical reasoning. It is a multi-choice problem where questions are mainly from GMAT and LSAT, containing 4139 data points. The analyses of the data demonstrate that questions require diverse types of reasoning such as finding necessary/sufficient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation. The paper includes comprehensive experiments with baselines to identify bias in the dataset, where the answer-options-only model achieves near half (random is 25%). Based on this result, the test set is split into the easy and hard set, which will help better evaluation of the future models. The paper also reports the numbers on the split data using competitive baselines where the models achieve low performance on the hard set.

Strengths
1) The introduced dataset is timely and is more challenging than existing reading comprehension datasets, based on the examples presented in the paper.
2) Analyses of reasoning types (Table 2) are done in a very comprehensive way, especially because they are comparable with descriptions from previous literature. This analysis clearly demonstrates that this data requires diverse types of challenging reasoning.
3) The experiments are comprehensive with many competitive baseline models, and their efforts to identify bias and use the result to split the test set are impressive.

Weaknesses
1) Although the main claim of this paper is about logical reasoning, I believe the term “logical reasoning” is somewhat subjective. What exactly is the definition of logical reasoning? The paper never defines it in any way; it only shows Table 1 as an example.
2) (Continuing the point above) The paper mentions other datasets’ statistics on logical reasoning (in Section 1, they mention “0% in MCTest dataset and 1.2% SQuAD” requires logical reasoning). How was this analysis done?
3) (Continuing the point above) Are all types of reasoning in Table 2 logical reasoning? I agree that those reasoning are very challenging, but I am not convinced why some of them belong to logical reasoning (E.g. summary). I expect this problem can be resolved if authors bring a clearer definition of logical reasoning.
4) Although I appreciate the experiments to identify bias, I believe that the bias seems to be somewhat significant if 440 out of 1000 examples are identified as biased examples. I believe that the authors should do more comprehensive analysis on biased examples and explain what are possible biases here, in order to justify the significant bias.

Marginal comments
1) The authors claim they collected the questions from open websites and books, but they didn’t provide the URLs or names. More information about which source was used, how data points were filtered and so on would be necessary to assess the quality of the dataset. Specifically, is there any particular reason for removing one of wrong options to make it have four choices, instead of five choices?
2) Why is the number of Chance model 0.39? Shouldn’t it be 1000 * 0.39% = 3.9?
3) In Table 5, it’s interesting to see that different models identify different numbers of biased examples. Have you looked at how much do they overlap each other if you take a pair of models? (A number of the union kinda shows it, but I wonder if there is a more intuitive way to see how consistent the models’ decisions are.)
4) It would be also helpful to see the baseline results of “question & answer options only model” and “context & answer options only model”.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
Paper Summary:

This paper presents a machine reading comprehension dataset called ReClor. It is different from existing datasets in that ReClor targets logical reasoning. The authors identified biased data points and separated the testing dataset into biased and non-biased sets. Experimental results show that state-of-the-art models such as XLNet and RoBERTa struggle on the non-biased HARD set with poor performance near that of random guess.

Strengths:

—The dataset, which is extracted from standardized tests such as GMAT and LSAT, requires the ability to perform complex logical reasoning. It is difficult for crowdsourcing workers to generate such logical questions.

—The authors carefully analyzed the biases, which are often exploited by models to achieve high accuracy without truly understanding the text.

—The authors thoroughly investigated how well strong models such as RoBERTa can perform.

—The paper is well organized and well written.

Weaknesses:

—The dataset seems small to acquire the ability to perform complex logical reasoning. The training, validation, and testing datasets consist of 4,651, 500, and 1,000 questions, respectively.

—The paper does not show statistics of the dataset such as question/passage length and question vocabulary size.

—The paper does not show results of models trained with other reading comprehension datasets such as RACE and fine-tuned with ReClor.

—Unlike other datasets, the questions themselves show their required logical reasoning types in ReClor. For example, the question “Which one of the following is an assumption required by the argument?” shows the “Necessary Assumptions” type and has no information with respect to passages. This characteristic makes it difficult to use the ReClor dataset as an evaluation benchmark for models trained with large-scaled reading comprehension datasets such as RACE.

—The human performance with respect to different question types of logical reasoning is not analyzed in Figure 3.

Review Summary:

The paper is well motivated. ReClor can be a useful dataset to evaluate the ability of logical reasoning, while the dataset seems small to acquire the ability to perform complex logical reasoning.   I think it can benefit a lot with a more comprehensive analysis of transfer learning with other reading comprehension datasets.

EDIT Nov. 20, 2019: 
I appreciate the authors' revision.  
Although my concern about the size of the dataset is not satisfied, I decided to increase the score of the paper (weak reject -> weak accept) upon looking at the author response about transfer learning.

There is a typographical error, in Table 2.  ACR -> ARC.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper presents a small multiple choice reading comprehension dataset drawn from LSAT and GMAT exams.  I like the idea of using these standardized tests as benchmarks for machine reading, and I think this will be a valuable resource for the community.

The two major concerns I have with this paper are with its presentation and with the quality of the baselines.  These concerns leave me at a weak accept, instead of a strong accept, which I would be if these issues were fixed.

Presentation:

The main contribution here is in collecting a new dataset drawn from the LSAT and GMAT.  (I assume that is all that is used, though the text actually says "such as".  Or is this from practice exams and not the actual exams?)  The job of this paper is to convince me, a researcher focusing on reading comprehension, that I should use this dataset as a target of my research.  Out of 8 pages, however, at most 2 are devoted to actually describing this contribution and why it's a good target for reading comprehension.  Much of the interesting description of the phenomena in the dataset is relegated to the appendix, which I did not really look at because it far exceeds the page limit of an ICLR submission.  I have a lot of questions about this dataset that could have been answered in the main text had more of the paper been given to actually describing the data.  Such as:

- What are some actual examples of the different kinds of reasoning in table 2?
- How did you decide the proportions of questions listed in table 2?  Was that manual?
- Where did the questions actually come from?  Real exams?  Practice exams?  Which ones?
- What is the reasoning process that a person would actually use to answer some of these questions?  Preferably answered for several of the question types that you listed.  You have 8 pages; there's a lot of space that could be given to this.
- What kinds of things do you see in the distractors that make this hard?

I'm recommending a weak accept for this paper, but only because of my background knowledge about the LSAT and GMAT, and my prior understanding of reading comprehension and why this would be a really interesting dataset.  I think the paper itself could do a *much* better job convincing a reader about the dataset than what is done here.

What should be cut to make room for this?

I think far too much space is given to the discussion of easy vs. hard (~2.5 pages).  Given that the best performance is at ~54% on the full test set, there isn't a lot of need for this.  The argument about four flips of a 25% coin giving a 0.39% chance of guessing right every time is only true if the coin flips are independently drawn; training the same pretrained model on the same data with a different random seed is clearly not independent, so this argument rings hollow.  It's not really clear what to conclude from the easy vs. hard split, other than the models that you used to create it expectedly do well on the easy split and hard on the test split.  You'd want to have separate models that create the split than what you're evaluating them with, but even then, you're training them on the same data, so it's still not really clear what to conclude.  It's sufficient in a case like this to just evaluate as a baseline a model that shouldn't have access to enough information to solve the task, and measure its performance on the test set.  That would have taken a few sentences to describe, instead of 2.5 pages.

You could also recover a lot of space by evaluating fewer baselines.  Describing six baseline models and including all of their performance numbers takes up a lot of space, and we really don't need to know what all of those numbers are.  One or two, along with question-only and option-only baselines, would be plenty.  If you really want to include all of them, that's information that should go into the appendix, instead of something that's core to your paper's contribution, like describing the dataset that you're releasing.

Quality of baselines:

With a dataset this small, you definitely want to pretrain BERT / RoBERTa on a larger QA dataset before training on ReClor.  I would guess that performance would be significantly higher if you used a RACE-trained model as a starting point, instead of raw BERT / RoBERTa.  And if it doesn't end up helping, then you've made a stronger argument about how your dataset is different from what has come before, and a good target for future research.  What I would recommend for table 6:

- chance
- option-only
- question-only
- raw XLNet or RoBERTa
- XLNet or RoBERTa pretrained on RACE, or SQuAD, or similar

And I would pick either XLNet or RoBERTa based on which one did the best after pre-training.  That way, related to my first point, you can remove the description of all other models, and table 4, and free up a bunch of space for more interesting things.

As an aside, I think it's good that the dataset is relatively small, as it gives these over-parameterized models less opportunity to overfit, and it makes us come up with other ways of obtaining supervision.  But you've got to be sure to use the best baselines available, and with small datasets, that means more pretraining on other things.

EDIT Nov. 19, 2019: The authors' revision has satisfied my concerns.  I think that this is a good paper and it should be accepted.
review_title: Official Blind Review #1

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper presents a new reading comprehension dataset for logical reasoning. It is a multi-choice problem where questions are mainly from GMAT and LSAT, containing 4139 data points. The analyses of the data demonstrate that questions require diverse types of reasoning such as finding necessary/sufficient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation. The paper includes comprehensive experiments with baselines to identify bias in the dataset, where the answer-options-only model achieves near half (random is 25%). Based on this result, the test set is split into the easy and hard set, which will help better evaluation of the future models. The paper also reports the numbers on the split data using competitive baselines where the models achieve low performance on the hard set.

--

Discussion:

While the authors agree this is an important direction, there are reservations concerning the small size of the dataset, that have not been fully addressed.

--

Recommendation and justifcation:

I still believe this paper should be accepted as the existing datasets for reading comprehension are inadequate and it is important for the field not to be climbing the wrong hill.
Decision: Accept (Poster)