Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ## Summary

This paper introduces a new dataset, WikiWhy, along with an explanation task, that aims to measure the reasoning and common sense abilities of generative language models. The work describes the motivation for this new dataset and task, mainly that most existing question-answering tasks require limited reasoning and common-sense knowledge. The paper describes how the new dataset was collected, based on Wikipedia with a couple of MTurk tasks: (1) generating question-answer pairs given a specially selected passage, and (2) generating a series of sentence-oriented rationales. The paper describes experiments using baseline models (based on GPT-2 with fine-tuning, and GPT-3 with prompting), demonstrating low performance on the proposed task. Finally, the paper proposes an automatic metric for evaluating system outputs on the new task, demonstrating correlation of the metric with human judgements.
strength_and_weaknesses: ## Strengths

1. The paper addresses an important weakness in many existing question-answering datasets that focus on answering questions with limited feedback on system reasoning or explanations. The proposed formulation of explicitly requiring systems to produce causal explanations to “why” questions is an interesting step,

2. The proposed task formulation is intuitive and elegant. While the two types of explanations proposed (step sequence and sets) are likely not comprehensive, they probably cover a large proportion of explanations to simple ‘why’ questions,

3. Collecting datasets is challenging, but can be helpful directing improvements to future systems. The release of this dataset to the community will likely help drive new work in QA explanations and reasoning.

4. The baselines described in the paper are quite reasonable starting points for modeling,

5. The human judgements of the baselines are surprisingly low which means that this task has large room for improvement in future work. (Note to authors: it would be nice if the baseline predictions were released along with your code+dataset so that they can be inspected by the community).

## Weaknesses

1. The main weakness in this work relates to the protocol used to elicit ‘why’ questions conditioned on a passage that contains a causal connective. Unfortunately, I believe this contains some of the same issues the community has observed with SQuAD task, namely: (1) generated questions are artificial since conditioned a specific passage, and rarely reflect naturally occurring ‘why’ questions, and (2) unusually high overlap in phrasing between the question and the answer passage, and (3) reliance of passage context for question to be fully interpretable. Fortunately, the paper addresses issue (3), but still has issues (1) and (2).

   Looking over most questions in the dataset, they mostly contain the issues listed. Examples:

   a) Question: "Why did the member of The Cure, Lol Tolhurst, make minimal contributions to the album Disintegration?"

     Passage excerpt: “... revealed that while Tolhurst had contributed to the song "Homesick", his contributions to the rest of the album were minimal due to his alcoholism”


    b) Question: “"Why is it unlikely that any sequels would be made for the Power Rangers film?"

      Passage excerpt: “However, in May 2017, Forbes noted that due to the underwhelming performance of the film in most markets, it was unlikely any sequels would be made.”

    In these examples (and others I browsed) it seems that (1) questions are formulated slightly unnaturally, (2) very high word overlap between answer phrase and question, (3) questions are all answerable within a single sentence.

    It is probably too late and costly to ask authors to revise their protocol for question generation. For future reference, one protocol that avoids the pitfalls of SQuAD is the TyDiQA dataset (https://arxiv.org/pdf/2003.05002.pdf).

 2. Probably due to the issue above, the retrieval performance using BM25 is unusually high  (Section 5.1). For comparison, the “Deep Passage Retrieval” paper (https://arxiv.org/pdf/2004.04906.pdf) shows BM25 performance in the 40s for Natural Questions. Similarly, answer identification should be quite easy as well for a model fine-tuned on this data given the high word overlap between answer phrase and question.

   So while this dataset will be useful to understand whether models are able to produce good reasoning/explanations for certain types of ‘why’ questions, it is, unfortunately, not a good dataset for QA retrieval and answer identification.

3. There is a part of the human evaluation (Section 5.3), specifically the Win/Tie/Lose which seems problematic. In our previous experience, disclosing to raters which outputs are “reference” and which are “system” biases ratings measurably. It is typically needed to make this type of analysis blind by (1) randomly shuffling which output is reference vs. system when presenting them to users, and (2) never disclosing the source of the explanation (reference or system).

## Other notes / suggested citations or references:

1. A dataset that seems quite related is ELI5 (https://arxiv.org/abs/1907.09190 , https://facebookresearch.github.io/ELI5/) which also aims to measure model’s ability to generate QA explanations.

2. Another dataset is the TellMeWhy (https://arxiv.org/pdf/2106.06132.pdf) which asks models to produce simple, reasoning-based explanations for answers in the context of stories.

3. One other dataset that I thought worth mentioning in the related work is e-SNLI (https://arxiv.org/pdf/1812.01193.pdf , https://github.com/OanaMariaCamburu/e-SNLI ). While not strictly a QA task, the explanations collected in e-SNLI are quite related to the ones presented in this work.

4. Another system that should be referenced is WT5 (https://arxiv.org/pdf/2004.14546.pdf) which is a T5-based system for generating explanations/rationales. Some of the checkpoints are available at https://github.com/google-research/google-research/tree/master/wt5#released-model-checkpoints and could serve as a stronger baseline than GPT-2, if fine-tuned for your dataset/task.

clarity,_quality,_novelty_and_reproducibility: ### Clarity:
The writing and presentation of the work are quite good. I had no issues understanding the basic ideas presented and experimental results. One issue is that there is some important content that has been pushed to the Appendix .

### Quality:
Overall, except for the protocol issue I outlined in question elicitation, I think the work is generally high quality,

### Novelty:
This work is moderately novel: explanations and reasoning is not a new topic in QA (see the related work along with my suggested references). Nonetheless, the dataset resource, along with proposed task formulation should be a worthwhile addition to this area.

### Reproducibility:
As far as I can tell, most of the resources for annotation and baselines experiments are made available. The one lacking resource is in Section 5.3 (Human Evaluation) which used student raters. The specific interface was not made available (like the MTurk one from Appendix A4), but perhaps it was just a spreadsheet?

summary_of_the_review: Overall I’m leaning to recommend acceptance of this paper. It has become clear to the QA community that we should move beyond factoid seeking queries and “exact match” as an evaluation metric. This paper proposes an interesting task variant. Despite some of the flaws in the data collection protocol (see above), I think this resource will be of interest to the community. The task formulation may also be extended in the future for other types of rationales. Finally, the initial baseline experiments indicate a large gap for improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Goal: The paper introduces a resource for explaining the cause-and-effect relationships on a wide variety of topics. 

Method: The resource is crowdsourced in two steps. First, crowdworkers are asked to turn potential causal statements (identified via discourse cues) into Cause-effect questions. Then, the crowdworkers are asked to explain the cause effect relation to obtain “why” answers. 

Using this process, a resource of about 9K why questions was curated from Wikipedia articles covering a wide variety of topics. 

Benchmarking: The paper presents benchmarking results that show the 

Key Contributions: The main contribution is in the resource itself. The resource presents a new test bed to assess abilities of NLP models to reason about cause and effect relations. The task requires a wide variety of knowledge and the benchmarking shows that the task is difficult for the current models and definitely warrants further research. 

strength_and_weaknesses: Strengths:

The paper introduces a high quality resource that fills a gap regarding evaluation of LLM’s ability to not only provide answers about causes and effects but also explain why using a wide variety of knowledge.

The paper is well-written and provides clear descriptions of how the data was collected, providing enough information to verify the quality of the resource.

The benchmarking experiments convincingly demonstrate the difficulty of the task. This resource could be a key driver for research into this problem space. 

clarity,_quality,_novelty_and_reproducibility: As stated above, the paper presents the main ideas, the data collection procedure and the experimentation details with clarity. 

The paper fills an important gap. Novelty or originality is not pertinent here with respect to the methodology.

There are adequate amount of details here and the main tasks are structured simply, so the data could be reproduced to some approximation with reasonable effort.
summary_of_the_review: The paper makes an interesting contribution to explainable reasoning for cause-effect relations, which can drive interesting research into this space.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new QA task on answering and explaining cause-and-effect questions. Different from previous approaches, the authors create the WIKIWHY dataset which contains the explanation within cause-effect relations. Taking the effect as the question source, the goal is to not only get the answer (cause) but also predict the explanations in sequential or set settings. Experiments show that the state of the art generative models are good at QA task but still a large space for improvement on the explanation task.


strength_and_weaknesses: ## Strength:
- It is the first work that investigates Q&A over cause-effect reasoning with explanations.
- The authors create a consolidated pipeline to define the problem and build the dataset, resulting a WIKIWHY dataset covering 11 topics.
-The authors propose new evaluation metrics for this cause-and-effect question answering task. Experiments verifies the reasoning ability of LLMs. 

## Weakness:
- The problem formulation in which a sequence or a set of explanations are required make this task intrinsically difficult to evaluate.

clarity,_quality,_novelty_and_reproducibility: Clarity: This work is generally well-written and easy to follow.

Quality: This work is technically sound.

Novelty: The novelty is not on the approach side but it is novel in terms of the task proposal, the dataset and the evaluation protocol.

Reproducibility: This work can be reproduced using the shared source code.


summary_of_the_review: In general, this is an interesting work. Though there are some improvement space from the problem formulation and evaluation, this version of work is still good for the task proposal, dataset preparation and the experiment design.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents WikiWhy, a Question Answering dataset of "Why" questions constructed from Wikipedia text. WikiWhy contains questions requiring two steps of reasoning or based on a rationale set.
strength_and_weaknesses: ## Strong

* The Cause and Effect QA with text generation problem is interesting (WikiWhy tasks 2 and 3)
* The automatic evaluation metric proposed is interesting.

## Weak

* (Minor) Table 1 has messy formatting. The Size and Topic columns should be right-aligned to support easy comparisons.
* Some hyper-parameters are missing (see the Clarity, Quality, Novelty, and Reproducibility section below).
* (Minor) The Conclusion states, "With this paper, we release WIKIWHY, a Question Answering dataset interested in understanding and facilitating the improvement of LLMs’ reasoning capability." The word "interested" is a strange choice for this sentence. I recommend rephrasing.
clarity,_quality,_novelty_and_reproducibility: * The paper is well-written and easy to follow. I particularly appreciate Figures 1 and 2, which give a good overview of how WikiWhy was constructed.
* There are many figures and tables in the appendices to help the reader understand the details of the paper and support reproduction. That being said, I cannot find all the hyper-parameters used for the GPT-2 Task 2 experiments.
summary_of_the_review: An interesting QA dataset. The paper has some minor issues.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new dataset of question answering that also focuses on selecting the reasons for a particular answer. In particular, the authors observed a lack of "why" questions in existing datasets, and created a dataset that focuses on these questions, while also adding a concept of explaining why a particular answer was chosen for the question. The reasoning is required to be in the form of a few sentences that when put together explain the answer. The authors also showed that existing LLMs do not work on this task out of the box hence demonstrating the difficulty of the task.

Strength
1) Very clearly written paper with thorough explanation of data collection methodology
2) Release of a dataset that is useful for the community and will advance research in reasoning
3) Establishment of baseline experiments for QA and QA+reasoning on this dataset

Weakness
1) There is some issue with the nature of the "why" questions because these are not totally open-ended but rather very focused on the text present in the answer passages. However, this cannot be easily rectified as soliciting open-ended questions from annotators come with its own set of challenges.
justification_for_why_not_higher_score: n/a
justification_for_why_not_lower_score: There is no particular reason to reject the paper or provide a lower score as shown by the strong agreement among the reviewers. Further, the authors have addressed most of the concerns in the updated version of the paper.
Decision: Accept: notable-top-5%