Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new task/dataset for language-based abductive reasoning in narrative texts.

Pros: 

-	The proposed task is interesting and well motivated. The paper contributes a dataset (20,000 commonsense narratives and 200,000 explanatory hypotheses). The construction of the dataset was performed carefully (e.g., avoiding annotation artifacts).  

-	The paper established many reasonable baselines.

-	The paper conducted detailed analysis, which invites more research on this task: despite the strong performance of many existing systems on NLI/RTE, there are larger gaps between the performance of these models and human performance on the proposed task. The experiments well support the conclusions made in the paper.

-	The paper is well structured and easy to follow. It is well written.

Cons/comments: 

-	While this is a new and interesting task, the contribution (as discussed above in “pros” above) is somewhat limited. I also suggest the paper discusses e-SNLI a bit more. 

-	The paper has a specific form of formulation for abductive reasoning, where there are exactly two observations and one proceeds the other; the explanation happens in between. I can see this helps collect and annotate data, but also limit the form of abductive reasoning and how models should be developed. 

-	Should the title of the paper specify the paper is about “language-based” abductive reasoning. 

-	A minor one: “Table 7 reports results on the αNLI task.” Should it be “Table 2”?

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper introduces two new natural language tasks in the area of commonsense reasoning: natural language abductive  inference and natural language abductive  generation. The paper also introduces a new dataset, ART, to support training and evaluating models for the introduced tasks. The paper describes the new language abductive tasks, contrasting it to the related, and recently established, natural language inference (entailment) task.  They go on to describe the construction of baseline models for these tasks. These models were primarily constructed to diagnose potential unwanted biases in the dataset (e.g., are the tasks partially solved by looking at parts of the input, do existing NLI models far well on the dataset, etc.), demonstrating a significant gap with respect to human performance.

The paper, and the dataset specifically, represent an important contribution to the area of natural language commonsense reasoning. It convincingly demonstrates that the proposed tasks, while highly related to natural language entailment, are not trivially addressed by existing state-of-the-art models. I expect that teaching models to perform well on this task can lead to improvements in other tasks, although empirical evidence of this hypothesis is currently absent from the paper.

Below are a set of more specific observations about the paper. Some of these comments aim to improve the presentation or content of the paper.

1. In Section “5.1 - Pre-trained Language Models” and attendant Table 1describe results of different baselines on the ART inference task. The results in the table confused me for quite some time, I’d appreciate some clarifications. With respect to the differences with columns 1 (GPT AF) and 2 (ART, also BERT AF) I would like the comparison to be made more clear. As far as I understand it, there are 2 parts of the dataset that can be varied: (1) the train+dev sets and (2) the test set. Furthermore, it seems that it makes sense to vary each of these at a time, if we are to compare results with variants. For example: we can fix the test set, and vary how we generate training and dev examples. If a model does better with the same test set, we can assume the train+dev examples were better for the model (for whatever reasons, closer distribution to test, harder or more 
informative training examples, etc). We can also keep the train+set constant, and vary the test set. This allows us to evaluate which test set is harder with respect to the training examples. The caption of Table 1 implies that both columns are evaluations based on the “ART” test set. If that is correct, then the train+dev set generated from the GPT adversarial examples is of better quality, generating a BERT-ft (fully connected) model that is 3% better. But the overall analysis seems to indicate that this is not what was done in the experiment. Rather, it seems that *both* the train+dev _and_ the test sets were modified concurrently. If that is the case, I would emphasize that the text needs to make this distinction clear. Furthermore, I would say that varying both train and test sets concurrently is sub-optimal, and makes it a bit harder to draw the conclusion that BERT adversarial filtering leads to a stronger overall dataset.

2. Along the lines of the argument in (1), above, I would urge the authors to publish the *entire* set of generated hypotheses (plausible and implausible) instead of only releasing the adversarially filtered pairs. Our group’s experience with training inference models is that it is often beneficial to train using “easy” examples, not only hard examples. I suspect the adversarially filtered set will focus on hard examples only. While this is fine to do in the test set, I think if the full set of annotated/generated hypotheses are released, model designers can experiment with combining pairs of hypothesis in different ways.

3. Furthering the argument of (2): in certain few-shot classification tasks, one is typically asked to identify similarity between one test example and different class representatives. Experience shows that it is often beneficial to train the model on a larger number of distractor classes than what the model is eventually evaluated on (e.g., https://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning). In the alpha-NLI setting, have you experimented with training using multiple distractors, instead of only 1, during training (even if you end up evaluating over 2 hypotheses)?

4. One potential argument for introducing a new natural language task is of transfer learning: learning to perform a complex natural language task should lead to better natural language models more generally, or, for some other related tasks. This paper does not really touch on this aspect of the work. But, potentially, one investigation that could be conducted is through a reversal of the paper’s existing NLI entailment experiment. The paper shows that NLI entailment models do not perform well on alpha-NLI. But it would be interesting to see if a model trained on alpha-NLI, and fine-tuned or multi-tasked on NLI entailment, does better on NLI entailment (i.e., is there transfer from alpha-NLI to entailment NLI?).

5. Another option is to evaluate whether alpha-NLI helps with other commonsense tasks. One other example is Winograd Schema Challenge, which current systems also perform well below human performance. It also seems that the Winograd Schema Challenge questions are not too far from abductive inference.

6. In the abstract of the paper, before the paper defines the abductive inference/generation task specifically, the claim that abductive reasoning has had much attention in research seemed awkward. Informally, most commonsense reasoning (including NLI entailment) could be cast as abductive reasoning.

7. In at least one occasion, I found an acronym which was hard to find the definition for (“AF” used in Section “5.1 - Pre-trained Language Models”; I assumed it was “adversarial filtering”.)

8. In Section “5.1 - Pre-trained Language Models” it seems that the text quotes an accuracy for BERT-ft (fully connected) of 68.9%, but Table 1 indicates 69.6%.

9. In Section “5.1 - Learning Curve and Dataset Size”, there is a claim that the performance of the model plateaus at ~10,000 instances. This does not seem supported by Figure 5. There appears to be over 5-7% accuracy (absolute) improvements from 10k to 100k examples. Maybe the graph needs to be enhanced for legibility?

10. It is great that the paper includes human numbers for both tasks, including all the metrics for generation.

11. Period missing in footnote 8.

12. The analysis section is interesting, it is useful to have in the paper. However, Table 3 is a bit disappointing in that ~26% of the sampled examples fit into one of the categories. It would be great if the authors could comment on the remaining ~74% of the sampled dataset.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: the paper purposes a dataset of abductive language inference and generation. The dataset is generated by human, while the testing set is adversarially selected using BERT. The paper experiments the popular deep learning models on the dataset and observe shortcoming of deep learning on this task.

Comments: overall, the problem on abductive inference and abductive generation in language in very interesting and important. This dataset seems valuable. And the paper is simple and well-written.

Concerns: I find the claim on deep networks kind of irresponsible. 
1. The dataset is adversarially filtered using BERT and GPT, which gives deep learning model a huge disadvantage. After all, the paper says BERT scores 88% before the dataset is attacked. 
2. The human score of 91.4% is based on majority vote, which should be compared with an ensemble of deep learning prediction. To compare the author should use the average score of human.
3. The ground truth is selected by human.

On a high level, the main difficulty of abduction is to search in the exponentially large space of hypothesis. Formulating the abduction task as a (binary) classification problem is less interesting. The generative task is a better option.

Decision: despite the seeming unfair comparison, this task is novel. I vote for weak accept.
review_title: Official Blind Review #3

Meta-review: This paper presents a dataset, created using a combination of existing resources, crowdsourcing, and model-based filtering, that aims to tests models' understanding of typical progressions of events in everyday situations. The dataset represents a challenge for a range of state of the art models for NLP and commonsense reasoning, and also can be used productively as a training task in transfer learning.

After some discussion, reviewers came to a consensus that this represents an interesting contribution and a potentially valuable resource. There were some concerns—not fully resolved—about the implications of using model-based filtering during data creation, but these were not so serious as to invalidate the primary contributions of the paper.

While the thematic fit with ICLR is a bit weak—the primary contribution of the paper appears to be a dataset and task definition, rather than anything specific to representation learning—there are relevant secondary contributions, and I think that this work will be practically of interest to a reasonable fraction of the ICLR audience. 
Decision: Accept (Poster)