WikiWhy: Answering and Explaining Cause-and-Effect Questions
Authors: Matthew Ho
Aditya Sharma
Justin Chang
Michael Saxon
Sharon Levy
Yujie Lu
William Yang Wang
Venue: ICLR 2023 notable top 5% (2023)
Keywords: NLP
Question Answering
LLM
Dataset
Explanation

Abstract: As large language models (LLMs) grow larger and more sophisticated, assessing their "reasoning" capabilities in natural language grows more challenging. Recent question answering (QA) benchmarks that attempt to assess reasoning are often limited by a narrow scope of covered situations and subject matters. We introduce WikiWhy, a QA dataset built around a novel auxiliary task: explaining why an answer is true in natural language. WikiWhy contains over 9,000 "why" question-answer-rationale triples, grounded on Wikipedia facts across a diverse set of topics. Each rationale is a set of supporting statements connecting the question to the answer. WikiWhy serves as a benchmark for the reasoning capabilities of LLMs because it demands rigorous explicit rationales for each answer to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to be easily memorized. GPT-3 baselines achieve only 38.7% human-evaluated correctness in the end-to-end answer & explain condition, leaving significant room for future improvements.

Published as a conference paper at ICLR 2023
WIKIWHY: ANSWERING AND EXPLAINING CAUSE-AND-EFFECT QUESTIONS
Matthew Ho, Aditya Sharma', Justin Chang,,
Michael Saxon, Sharon Levy, Yujie Lu, William Yang Wang Department of Computer Science, University of California, Santa Barbara, USA fmsho, aditya sharma, justin chang] .edu, (saxon, sharonlee rovlevy,, yuji i eelllllleee @ucsb. edu, wi willi amêcs ucsb edu
ABSTRACT
As large language models (LLMs) grow larger and more sophisticated, assessing their "reasoning" capabilities in natural language grows more challenging. Recent question answering (QA) benchmarks that attempt assess reasoning are often limited by a narrow scope of covered situations and subject matters. We intro- duce WIKIWHY' QA dataset built around a novel auxiliary task: explaining why an answer is true in natural language. WIKIWHY HY contains over 9,000 "why" question-ansser--ationnnl triples, grounded on Wikipedia facts across a diverse set of topics. Each rationale is a set of supporting statements connecting the question to the answer. WIKI HIKIHHY HY serves as a benchmark for the reasoning capabilities LLMs because it demands rigorous explicit rationales for each answer to demon- strate the acquisition of implicit commonsense knowledge, which unlikely to be easily memorized. GPT-3 baselines achieve only 38.7% eeallueed correct- ness in the end-to-end answer explain condition, leaving significant room for future improvements.
INTRODUCTION
Error analyses of practical NLP systems in recent history demonstrate that some of the mistakes made by state-of-the-ar models would be avoided by basic human intuition (Shuster et al., 2022), and some of the most challenging tasks for models are the same ones that might be trivial to hu- man children. With modern systems' impressive performance on tasks such as grammar correction showing that manipulating language is not the issue, I.LMs seem to face a fundamental lack of com- mon sense- an understanding of everyday phenomena and how they interact with each other and the world at large. As striking gains in subjective performance tion, and apparent language understanding continue to be called into question, the development of strong benchmarks to assess reasoning capabilities for these LLMs grows more important.
summarization, creative text genera-
One popular approach to measuring reasoning capability is through performance on question an- swering (QA) benchmark tasks where direct queries for information act straightforward exam- ination of a system's "understanding." Classic QA datasets, however, are primarily concerned with retrieving factoids to answer questions of "Who", "What", "When", and "Where". These questions have been shown to be answerable (with high accuracy) by simple (Wadhwa al., 2018), thereby limiting their ability to measure the aforementioned reasoning capa- bility. Looking to maintain the breadth of topics covered while increasing the difficulty of the QA task, researchers introduced multi-hop QA datasets like HotpotQA (Yang et al., 2018). While chal- lenging, the task's extra complexity mostly leads to unnatural questions that can be addressed with iterated factoid retrieval and entity resolution, rather than necessary understanding of how different entities interact. Noticeably absent in these prior datasets are "why" questions, which prompt for not factoids, but explanations- reasoning made explicit.
approaches
"Co-first authors. Author contribations listed at end of paper. 'https://gi thub. com coom/mate /tt// seb-ho/ ho/wi wi wiy

---

Published as a conference paper at ICLR 2023
PASSAGE
CAUSE
REASONING
Numerous plans for the Second Avenue Subway appeared throughout the 20th century, these were usually due to lack funds..."
Lack of Funds.
Contractors complete construction
Contractors need to be compensated.
QA
QUESTION:
Why were numerous plans for the Second Avenue Subway of New York City deferred throughout the 20th century?
EFFECT
Numerous plans for the Second Avenue Subway of New York City were deferred throughout the 20th century.
ANSWER:
Lack of Funds.
Figure A simple example of an entry from WIKIWHY; a cause and effect sourced from Wikipedia passage. "wwhy" question and its answer about this relation, and most importantly rationale that explains why cause leads to effect.
The task of explanation uses reasoning and produces explicit, interpretable "thought" processes. Capitalizing on these properties, this paper introduces WIKIWHY, novel dataset containing "why" question-answer pairs. Each WIKIWHY entry contains rationale explaining the QA pair's causal relation (Figure 1), summing to total of 14,238 explanation elements. the context of recent multimodal, self-supervised approaches aiming to capture intuitions unlearnable from text alone (Chadha Jain, 2021), WIKIWHY presents an opportunity investigate specific kind of infor- mation absent in text: implicit commonsense assumptions. Compared to other QA datasets with rationales, WIKIWILY covers a significantly broader range of 11 topics which may prove valuable for developing the skill of applied reasoning various specific situations.
Our experiments in explanation generation and human evaluation demonstrate that state-of-the-an art generative models struggle with producing satisfying explanations for WIkIWHH cause-effect rela- tions. Our experiments also demonstrate how our proposed task might be used to diagnose a lack of "understanding" in certain relations. Our key contributions are thus:
We propose explanation within relaec novel problem formulation for exploring LLM reasoning ability.
We create WIKIWHY, the first relations, spanning 11 topics.
dataset focusing reasoning within causal
We perform experiments state-of-the-are generative models to investigate various settings and establish baseline results with sizable room for improvement.
We introduce idea-level evaluation metrics for free-form text (explanation) generation and hu- man judgment correlation analysis, demonstrating that (1) reference similarity is strongly corre- lated with explanation correctness, and (2) the metrics introduced correlate with this proxy.
RELATED WORK
Cause and Effect. Causality has been a subject of rigorous work in various fields. In science phi- losophy, Pearl (2009) has contributed seminal work relating to causal models, Bayesian networks, and causal strength via interventions and counterfactuals. These ideas have even been incorporated into QA tasks through Knowledge Graph approaches, such as filtering spurious latent correlations (Sui et al., 2022). While work emphasizes cause-and-effect, we unconcerned with causal strength. Starting with Eokipedia-groundee relations ensures valid relations. Instead, we are inter- ested in the information encoded into LLMs rather than augmented structures such as knowledge graphs.
Multi-hop Question Answering. While datasets such HotpotQA (Yang al., 2018) and Hy- bridQA (Chen et al., 2020) are instrumental in gauging models' ability to handle multiple sources and modalities, they focused on iterated factoid retrieval. Although chaining multiple facts into
2

---

Published as a conference paper at ICLR 2023
Table 1: comparison of WIKIWHY with previous QA datasets relating to explanation
Dataset
Size
Answer Type
Explanation Type
Topics
Source
CoS-E
9,500
MCQ
1-step
ConceptNet
eQASC²
9,980
MCQ
2-step
WorldTree
CausalQAs
24,000
Short
None
Yahoo Finance
EntailmentBank
1,840
Short
Tree
1
WorldTree
WIKIWHY
9,406
Short
Set/Chain
11
Wikipedia
'(Rajni et al., 2019), "Jhamtani & Clark, 2020), *(Yang et al., 2022), A(Dalvi et al., 2021)
multi-hop answer is useful for products, WIKIWHY focuses on in-filling rationales to demonstrate reasoning.
Visual Question Answering. Vision and language tasks have also intersected with both QA and reasoning. The Visual Question Answering (VQA) dataset (Agrawal et al., 2015) prompts textual answers questions about images. However, the caption-based generation leads to surface- level questions that require little reasoning ability, and the multiple-choice output format precludes explicit reasoning. The vision-based Sherlock dataset (Hessel al., 2022) is much closer our work, focusing on abductive reasoning (working backward from consequence). Setting aside modality differences, WIKIWHY requires deeper reasoning with its multi-hop explanations.
Explainable QA. One previous approach to building explanation resources collects direct answers to "why" questions. Tel (Lal al., 2021) features question-anee annwerr pairs tied to short story narrative contexts. The dataset skips step-wise explanations, prioritizing reading comprehension instead. On the other hand, ELI5 (Fan et al., 2019) dives deep into reasoning with long-form, detailed explanations. However, the open-endedness (compared to explaining a specific cause-effect relation) complicates evaluating candidate responses.
Another line of QA work emphasizes rationale component support for answer predictions. Datasets like CoS-E (Rajani et al., 2019), chhSC(aaamtani & Clark, 2020), and EntailmentBank (Dalvi et al., 2021) focus explanation and reasoning much like WIKIWHY, albeit with significant differences (Table 1). CoS-E's explanations for CommonsenseQA (Talmor et al., 2019) mark an important first step, but the commonsense explanations have limited depth, often requiring sin- gle hop of reasoning. eQASC and EntailmentBank feature richer explanations with more complex structure, tightly focusing on grade school level science facts. Regarding structure, fixed-length tionale in CoS-E. eQASC, FEVER (Thore et al., 2018). and e-SNLI (Camburu et al., 2018 80) capture less granularity, while entailment trees accept limitations in scale and naturalness in exchange for complete ordering information. Previous datasets tend towards retrieval tasks with eQASC's 's corpus of all rationale sentences and EntailmentBank's 's collection of root causes. Retrieval enables simple evaluation, at the cost of decreased difficulty, the possibility for exploiting spurious artifacts, and reduced debugging opportunity.
BACKGROUND
3.1
WHY FOCUS ON "WHY" QUESTIONS?
"Why" questions are underrepresented in other QA datasets. Users tend to ask straightforward questions that use words like "who", "what", "when" or "where." Questions of this more common form have simple answers that state standalone facts which may be elaborated but do not require explanation. Consider the pair, "Q: What the fifth most abundant metal the Earth's 's crust? A: Calcium." The answer is straightforward.
In contrast, a "why" QA-pair encodes a cause-effect relation. Take, for example, "Q: Why can't 't I eat calcium metal? A: Calcium reacts exothermically with water and acids. This pair encodes the causal relation "calcium has an exothermic reaction with water, therefore eating calcium is not advised." (Figure 2). The answer to a "why""question -question is explanation itself (the reaction being exothermic explains the toxicity), but we can take it a step further and ask "why" again to request the understanding, intuition, of this process. While there some processes at the edge of human
3

---

Published as a conference paper at ICLR 2023
Step Sequence
Rationale Set
CAUSE:
CAUSE:
Calcium metal reacts exothermically with water and acids
The USSR mostly traded with Eastern Bloc neighbors
EFFECT: The merchant marine was not used much under Joseph Stalin
ELEMENT
ELEMENT 1:
Eastern Bloc countries connected by land
Exothermic reactions release heat
S₂
Sq
ELEMENT
ELEMENT
Intense heat damages tissues
Merchant Marine trades by sea
ELEMENT 3:
EFFECT:
More direct routes are preferable trading
E
Calcium metal toxic
Figure 2: Explanation topologies in WIKIWIIY mainly vary between sequence of intermediate conclusions (chain-likee like)) and a set rationale that combine with the original cause to entail the final effect.
understanding or taken as axioms, we assert that there are valid explanations for most processes due the layered nature of human understanding. This extra step is especially worth taking since allows WIKIWHY to not only test if a model "knows" that "an exothermic waler reaction makes calcium toxic" but also if it "unddesttadd"" the underlying mechanics of why that is the case.
3.2 TASK FORMULATION
Formally defined in $5, we propose generative explanation task. Previous works have made strides in assessing reasoning through multiple choice (Lu et al., 2022), retrieval (Asai et al., 2019), and partial generation (Dalvi et al., 2021). While these works are undoubtedly crucial towards end goal of understanding and reasoning. their task formulations have some drawbacks. Referring back to education, studies on human students have shown that multiple choice questions "obscure nuance in student thinking" (Hubbard et al., 2017). Likewise, selection decision be correct for retriever systems but for the wrong reasons. Augmenting multi-hop factoid questions with an additional task of selecting the relevant supporting facts from the context passage, Inoue et al. (2020) emphasizes that interpretability is lost in the absence of explanation. Furthermore, text generation combine existing ideas arguably different task than generating from scratch. The field of psychology defines recall (mental retrieval of information) as a distinct process from recognition (mental familiarity with the cue) (Mohr al., 1989). Neural nets" biological inspiration suggests that there might be similar difference between cue-aided retrieval and freeform generation. In the context of NLP, we are interested in the implicit understandings and assumptions embedded in LLMMs and hypothesize that an entirely generative approach is most conducive to this study.
3.3 EXPLANATION STRUCTURE
Explanations come in various structures, as seen in the typology defined by Ribeiro et al. (2022). Shown Figure 2, our work focuses on subset said typology. WIKIWHY includes two struc- tures that explain cause-and-effect relations: (1) multi-hop step sequences and (2) rationale sets. While the chain structure adds intermediate conclusions between cause and effect, rationale sets contain elements that support the relation from without. The rationale set topology acts as our gen- cral, catch-all case that other structures can be condensed to. Since our data collection procedure promotes stepwise, ordered approach, also consider the sequential topology respect the structure exhibited in applicable explanations. We forego the unstructured approach as even lim- ited structure helps bring freeform generated text evaluation within reach. Finally, opt against pursuing the most complex entailment tree organization to maintain naturalness and facilitate crowd- sourcing scalability.
4

---

Published as a conference paper at ICLR 2023
4 DATASET
4.1 DATA COLLECTION
The objective of WIKIWHY is present high-quality, challenging dataset of QA pairs with corre- sponding causes, effects, and explanations. We developed an extensive data collection and validation pipeline around Amazon Mechanical Turk, depicted in Figure 5 (appendix). For each stage involving crowdsourced annotations, we perform rigorous worker-level quality control to ensure the dataset's quality. The exact procedures are detailed in subsection A.2 in the appendix.
Preprocessing. We begin with English Wikipedia's 's corpus of "Good Articles," whose strict crite- ria of verifiability and neutrality (among others) ensure that WIKIWHY does not evaluate models on misinformation opinionated views. From these articles, we extract passages containing causal re- lations using causal connectives. We selected a list of causal keywords (Appendix, $subsection A.1) from a more extensive set of causal connectives as their presence in a passage guarantees the exis- tence of cause and effect relation- seationn-omeeeee excluded connectives such as "since" or "as" are highly prevalent but are not necessarily causal. The presence of causal word pattern on its own is a very simple heuristic-in the subsequent collection steps, we hired crowdworkers to ensure the quality of each sample.
QA Synthesis (Stage 1). Randomly sampled preprocessed Wikipedia passages containing potential causal statements were shown to qualified Amazon Mechanical Turk (MTurk) workers (see ethics statement for details), who were tasked with extracting the highlighted causal relation from the passage and re- re-framing it as "why" question when possible. While automatic cause-effect relation extraction has seen recent progress (Yang et al., 2022), this human intelligence task (HIT) remains vital for two reasons. First, we find that quality in cause-effect is crucial for meaningful and valid explanations in the following stage. More importantly, we depend on human annotators to add sufficient context to the text of the cause, effect, and question to disambiguate them. This enables the question and cause-effect relation to be presented to models without the context we prepared (e.g., "Why was the river diverted?" is unanswerable without additional context). This feature is key to enabling WIKIWHY to assess the information and ideas within L.MSs as opposed to whatever may be present in the context.
Explanation Synthesis (Stage 2). After verifying the quality of the examples, we prompt crowd workers to explain cause-effect pairs from stage To encourage structured explanation, we supply an interface that allows sentences or ideas to be entered one at a time in separate fields. Though the input pairs should be context-indeeendent, we provide the original passage as an aid for under- standing the topic. Furthermore, we provide the link to the source article to encourage explanations leveraging topic-specific information in addition to commonsense knowledge.
4.2 DATASET DESCRIPTION
Entry Contents. In addition to the main fields of the question, answer, and explanation, each dataset entry contains the underlying relation's 's cause and effect, the passage the question was ex- tracted from, the article the passage is from, and Wikipedia's 's topic categorization for that article.
Topic Diversity. WIKIWILY improves upon other datasets due to its ability to examine reasoning proficiency across broader range of concepts (Table 9 in Appendix contains examples from the six most frequent topics).
The statistics for the reasoning component are shown in Appendix Table 11. On aver- age, each rationale contains 1.5137 elements. Figure 4 (Appendix) shows a histogram of rationale length by sentence count. WIKIWHY includes range of rationale lengths, with more than third of examples (36%) containing two or more reasoning steps.
Rationale.
5

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
5.1
EXPERIMENTAL. SETTINGS AND MODELS
Task Notation Let C' be cause clause; E be an effect clause corresponding to C; Q be ques- tion equivalent to "Why is it the case that E?"; A be the answer to Q 3; X be the explanation (S1, S2, , Sk) where S, is a sentence such that:
CAS A S2 ........
E
Task I: Question Answering (QA). Input = Q, Output = A. For thoroughness, we confirm high performance on Task (Standard QA) in the open-book setting. For this of experiments, we use the classic approach of breaking the task into separate retrieval and reading comprehension phases. We experiment with BM25 (Robertson et al., 2009) and Dense Passage Retriever (DPR) (Karpukhin et al., 2020) as our document retriever, using their Pyserini implementations (Lin et al., 2021). Using the Natural Questions (Kwiatkowski et al., 2019) encoder, as the original DPR paper, we build custom indices around segments from the subset of Wikipedia Articles shown to workers at collection time. For reading comprehension. we experimented with RoBERTa (Liu et al., 2019) and Big Bird (Zaheer et al., 2020) QA models. We also fine-tune Fusion-in-Decoder (FiD) (Izacard & Grave, 2020) model 180-10-10 10 split; default configurations), hypothesizing the decode- time combination of ideas could better model cause- relations.
The performance was unsurprisingly high, with BM25 achieving a high Top-1 Accuracy score of 0.810 in retrieval and FiD reaching mean BERT-f1 of 0.78 (Table 7 in Appendix). While retrieving the appropriate Wikipedia passage relating to some topic is straightforward, we found that producing an explanation of comparable quality our gold rationales was difficult for the models we tested.
Task 2: Explanation Only (EO). Input = (C, E), Output= = X. First, we examine task 2: gener- ating an explanation given an initial cause-effect pair. Given their stronger zero-shot generalization (Wang et al., 2022). we choose decoder-onk only models for our baselines. In this vein, we investigate the few-shot abilities of GPT-3 (Brown al., 2020) with OpenAl's most capable model, DaVinci- 002, at otherwise default settings. To better coax out the intermediates between cause and effect, we conduct prompt engineering over Wei al. (2022)'s Chain of Thought method. Our exemplars are shown in Figure 6.
We also make use of WIKIWILY's scale for fine-tuning GPT-2 (Radford et al., 2019). this set of experiments, we attempt to balance improving GPT-2's understanding of the task's structure while preserving the model's "intuitions" for examination. We train GPT-2 for ten epochs using the train- ing split (=800 of the data) and Adam (Kingma& & Ba, 2014) optimizer with standard hyperparame- ters (learning rate: 7 = 0,001, 31 = 0.9, 82 = 0.999, e = 1e-8, decay: À = 0). For this tuned model introduce special delimiter tokens <cause>, <e <ef fect>, and <exp 1 anat i on in addition to the beginning and end tokens <bos> and <eos>. To support the delimiters and help the model distinguish the segments, we add token type embeddings (marking cause, effect, and explanation) as part of the preprocessing phase. At decoding time, we experiment with multiple temperatures.
Task 3: Answer and Explanation (A&E). Input = Q, Output (A, X). To investigate the performance of jointly predicting an answer and explanation given only a "why"" question, we carry forward with our best performing baseline from the EO task. The first setting in this experiment set tasks single model with the full end-to-end procedure. Once again, we utilize Chain-of prompting, albeit with a modified prompt that also requests an answer to handle the different input format. Considering the impressive performance of existing IR techniques on the QA task described above, we also study an additional setting incorporating the QA task. In the "pipeline" setting, the explainer model still lacks access to the ideal answer (the explanation's 's starting point) but benefits from a reader model's 's access to the original context. Here combine our strongest performing approaches to the QA and EO tasks to make a 3-step pipeline of retrieval (BM25), reading (FiD), and explanation (GPT-3).
prompted GPT-3.
5.2 AUTOMATIC EVALUATION METRICS
While the still developing area of text generation has measures and proxies for similarity that help with simple sequences, comparing reasoning sequences or rationale sets requires more involved
Note that Q is query that provides F and is correctly answered by C, C A.

---

Published as a conference paper at ICLR 2023
Reference
Prediction
Alignment
Alignment
Unordered
The US supported Israel during the Yom Kippur War
Yom Kippur War, the US sided with Israel.
R1
R1
R1
An OPEC embargo halted US exports 1973
OPEC organization with 13 par participating countries
The UK was unable to import oll make vinyl records
UK vinyl record production slowed without raw materials
1973, decided halt US ail exports
R2
R2
Figure 3: Alignment example for sentence-level metrics. Ordered evaluation uses the longest com- subsequence shown by alignment and 2. The final alignment's length is used to compute F-score metrics.
measures. With the two topologies introduced $3.3 in mind, we propose two related metries, unordered and ordered, to handle sets and sequences, respectively.
Unordered Evaluation. This first approach compares the ideas contained in the predictions and ref- erences. First, we split predicted and reference explanations into "ideas" or "steps" by sentence. We then compute matrix of pairwise similarity scores before using a threshold to classify "matches". Since single prediction sentence may contain multiple reference ideas, keep separate counts of precise prediction steps and covered reference steps. These counts are then micro-averaged for the test set's overall precision, recall, and F1 scores.
Ordered Evaluation. To respect the structure of multi-hop explanations, we penalize incorrectly the previously generated pairwise score matrix and its align-
ordered explanations. Here, ments to generate all possible assignments of prediction sequence elements to reference elements. As demonstrated in Figure 3, compute the length of the longest common subsequence (L.SS between a prediction alignment against the reference labels for each candidate assignment. This length becomes the count of correctly incorporated structural elements- positives. Note that the LCS alignment discounts repeated ideas in the prediction.
Metric Validity. To understand the usefulness of our constructed metrics, we compare them against human judgements. A panel of 3 undergraduate students compared pairs of predictions and references on two binary scales: (1. Similarity) "Is the prediction similar to the reference?" and (2. Correctness) "Is the prediction valid or correct explanation of the cause-effect pair?" Summing the panelist scores for each pair, we found a strong correlation (r = 0.82) between the similarity and correctness judgement. This validates comparison with WIKIWHY gold explanations useful proxy for explanation quality. Our proposed sentence-level processing incorporates the intuitions of checking for completeness with recall and penalizing over-explanation with precision.
Further, we use a single-xxplanation version of F-score to compare this proposed automatic metric with human judgement (the proposed F-score measures aggregate through the whole dataset). With this variation, we find a modest correlation (r = 0.35) between ordered F1 and similarity, among other weaker correlations.
Besides supporting our proposed methods, this correlation analysis also enabled a data-driven ap- proach to calibrating our similarity metric and match criteria. For each similarity metric, selected triples (which thresh-
a starting point through manual inspection of rrediction-rfferene--imiiiaa old value divides "genuine" from mistaken similarity) and used correlation for refinement. After trials with BLEURT (Sellam et al., 2020) and BERTScore (Zhang et al., 2020), different underlying models and different match thresholds, we selected BERTScore using large DeBERTa (He al., 2021) model (mi crosoft. deberteeere.. rtae rta-etlleeee x wwwrgereeeee aarreemmmn 1 at threshold of 0.64.
5.3
HUMAN EVALUATION
Recent studies by Goyal et al. (2022) show that automatic metrics may not reliably evaluate results produced by models with few-shot capabilities like GPT-3. In light of this, supplement our auto-

---

Published as a conference paper at ICLR 2023
matic evaluation with additional human evaluation. We first evaluate each setting in each experi- ment using the binary correctness scale (see criteria definition below). Following this evaluation, we select the highest scoring explanations for each set of experiments for additional fine-gt uineggrined evalu- ation. For each human evaluation task, we present a panel of three undergraduate students a random sample of 50 entries from each setting and the following binary True/False criteria guidelines:
Correctness: Mark true if and only if the explanation is both complete and satisfying.
Concision: Mark true if the explanation says everything it needs to say and nothing more. Mark false if extra information is included.
Fluency: Is the explanation writing fluent? Mark false if there are any mechanical mistakes.
Does the explanation make logical sense? Ignore whether not the explanation suc- cessfully explains the cause/effect relation. Mark false if the explanation contains any illogical or untrue conclusions.
Validity:
Compare the generated explanation against the provided reference (WIKIWHY gold explanation). Mark Win if you prefer the generated explanation, Tie if you have no prefer- ence, and Lose if you prefer the reference explanation.
Win/Tie/Lose:
5.4 RESULTS
Fine-Grained Human Evaluation. With our human evaluation experiments, we find significant room for improvement across the board. Our strongest baseline, GPT-3 with greedy decoding, produced explanations judged to be satisfactory only 66% of the time in the most favourable setting of Task 2: EO (Table 3). Moreover, these explanations were judged to be worse than the gold reference 58% of the time. These results from our strongest baseline leave plenty of room to improve upon and motivate future work this reasoning task.
Decoding. Our experiments show increased performance with lower temperature sampling and best results with greedy decoding (Table 2). This aligns with existing notions of higher temperatures better suiting "creative," open-ended tasks as opposed to more grounded ones. Explaining, as we hypothesize, relies more on the embedded assumptions in a generative model rather than the tenuous associations made more likely at higher temperatures.
Model Differences. We find that GPT-3 significantly outperforms GPT-2. Comparing GPT-3's output against its predecessor's s strongest setting shows increases in both ordered and unordered F1 scores by over 50%. Despite benefiting from fine-tuning and additional structural support from token type embeddings, GPT-2's explanations are lacking compared to GPT-3's few-shot explanations using only exemplars. We find that GPT-2's statements are often not only incomplete//unsatittying for explaining the cause-effect relation at hand but also simply invalid. 94% of GPT-2's statements were deemed worse than VIKIWHY' gold references. The only area GPT-2 outperformed GPT- 3 was in concision, however this is more a demerit of GPT-3 rather than merit of GPT-2. We found that GPT-3 tended to occasionally add unnecessary detail to its explanations, often needlessly defining one of the entities in the prompt.
Answer & Explanation. On the A&E task, we find results that align cleanly with preconceived intuitions. Our baseline model able better handle explanations from points A B when A is fixed and provided. Requiring the same procedures to generate more output creates more vari- ance as incorrect or alternative starting points mislead the remaining generation. The "pipeline" setting strengthens this trend, as the better-informer answer generation allows for higher quality explanation. This setting, simulating three-step process with different models handling each step, demonstrates an intermediate performance between having the oracle-provided answer and requir- ing the explainer to manage the entire process. The "validity" criteria of our human evaluation especially interesting under this setting, where the model's input excludes the correct answer (the cause). While the majority of the end-to-end setting's explanations were marked incorrect or unsat- isfying, notable proportion was still marked having valid chain of reasoning. This supports the chain-of-tee thothht premise that short, logical strides are more manageable for LLMs, but the approach is still insufficient for generating satisfactory explanations of target phenomena.
Explanation Failure. A typical error observed in GPT-3's predictions is repeating the cause-effect relation. To explain why [A] leads to [B], GPT-3 might only write " [[B] because [A] " another
8

---

Published as a conference paper at ICLR 2023
Table 2: Baseline Performance on Explanation Tasks (EO Explanation-Only, A&E: Answer and Explanation). For Task 3, the Single Model setting has the generative model complete the end-to- end task single pass. The Pipeline setting allows each stage be handled separately (QA is handled by BM25+FiD and explanation is done by GPT-3). Human evaluation was done with binary scale (correct/incoreect) and we report the proportion of correct evaluations.
Experiments
Unordered
Ordered
Human
Precision
Recall
BERT-f1
Precision
Recall
BERT-FI
Correct
Task 2: EO GPT-2 Greedy
0.249
0.196
0.220
0.239
0.179
0.204
0.100
T = 0,50
0.218
0.164
0.188
0,194
0,146
0.166
0.065
1.00
0.072
0.056
0.063
0.071
0.054
0.062
0.064
GPT-3
Greedy
0.347
0.388
0.366
0.307
0.355
0.329
0.660
= 1.00
0.326
0.356
0.340
0,291
0.328
0.308
0,481
Task 3: A&E
GPT-3
Single-Model
0.092
0.095
0.094
0.082
0.092
0.087
0.140
Pipeline
0.229
0.233
0.231
0.211
0.220
0.215
0.387
Human evaluation. Overall correctness is marked on a binary scale- an explanation is complete and satisfying not. Concision penalizes for repeated or unnecessary information, fluency evaluates grammar, and validity measures if the generated sequence makes logical sense regardless if it correctly explains the relation. For Win/Losee/Tie, annotators compared the generations against. WIKIWHY's gold references.
Table 3:
Setting
Fine Grained Human Evaluation
Correctness
Concision
Fluency
Validity
Win (?)
Tie
Lose (L)
GPT-2: EO
0.100
0,880
0.860
0,520
0.040
0.040
0,920
GPT-3: EO
0.660
0.680
1.00
0.960
0.080
0.360
0,580
GPT-3: A&E
0.140
0.680
0.900
0.720
0.080
0.100
0.820
semantically equivalent formulation. This pattern may explainable with a fine-tuned baseline where annotation errors of the same kind might have slipped into the training set, but GPT-3 was prompted with hand-picked exemplars with no such mistakes. Furthermore, we observe success- ful explanations some inputs expect to be more difficult alongside errors relatively less challenging inputs. These observations, together with the consistently high fluency scores showing syntactic competence,
to indicate reasoning failure opposed systematic "misunder- standing" of the task at hand. Per the original goal of better understanding what and how LLMs "understand" the world, this might indicate gap commonsense: that GPT simply memorized the fact that [A] leads to [B].
CONCLUSION
With this paper, we release WIKIWHY, Question-Answerte dataset enabling the analysis and im- provement of LLMs' reasoming capability. We propose explanation between grounded cause-effect pairs to distinguish memorization of the relation from a genuine understanding of the underlying mechanics. Compared to related works on explainable QA, our explanation format finds natural middle ground that balances complexity and depth, allowing our crowdsoureing methods to pro- duce
examples while being highly scalable. We exploit this scalability to cover topics previously overlooked by other explanation datasets and demonstrate our proposed task to be difficult with strong baselines (our experiments feature models failing to produce satisfying explana- tions even under ideal conditions). Finally, we motivate the development of new automatic metrics that are better able handle the complexities of generated reasoning.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
For data collection, our listing required workers to have high HIT approval rating (2 96%) and be located in English speaking regions (Australia, Canada, New Zealand, the United Kingdom, and the United States). The average hourly pay is 12.00 dollars, which exceeds the income requirements proposed the human subjects research protocols. The project is classified as exempt status for IRB. Our interfaces include notices that forms, and a link for inquiries and concerns. Our MTurk interfaces are displayed in the $A. Due the experimental nature, limited production applicability, and relatively small dataset scale, believe the potential for misuse or harm negligible.
collecting information for dataset creation, consent
REPRODUCIBILITY STATEMENT
We publically release our dataset and codebase at https::/ggthheie com/matt seb-ho/ WikiWhy containing the model tuning procedures, settings, few-shot prompts, and evaluation script.
AUTHOR CONTRIBUTIONS
Matthew Ho developed the crowdsourcing interfaces, performed data cleaning, ran baseline re- theval/reading/explaaatio experiments, designed and implemented evaluation procedures, and drafted the paper. Aditya Sharma contributed to crowdsourcing interfaces, performed data clean- ing, organized dataset statistics, and created tables and designed figures. Justin Chang developed the data collection infrastructure, the validation web app, and ran OpenAl baselines. Michael Saxon, Sharon Levy, Yujie Lu, and William Wang advised, guided, and ideated. All authors edited the manuscript.
ACKNOWLEDGEMENTS
Thank you to Early Research Scholars Program (ERSP) advisors Diba Mirza and Chinmay Sonar, and Wenhu Chen for further advice. Thank you Alex Mei for comments on the manuscript, and Ed and Lauren for input on data collection. We thank the reviewers for helpful feedback. This work is supported by the Amazon AWS AI/ML Research Award and AWS Cloud Credit for Research. MH, AS, and JC were supported by ERSP under National Science Foundation (NSF) Grant 1821415 as well generous donations from anonymous source towards undergraduate research at UCSB. AS was supported by College of Creative Studies Traveling Undergraduate Research Fellowship (TURF). MS was supported in part by the NSF Graduate Research Fellowship, NSF Grant 16501 14. This work was also supported by the NSF Award 2048122. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies. either expressed or implied, of the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2015. URL httpt:t////aae /ttp:::///xxxvv.r///// abs/1505. 00468.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learn ing to retrieve reasoning paths over wikipedia graph for question answering, 2019. URL
https:// /ttp:i//aaivvvv//a org/ abs//991 104700047.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
10

---

Published as a conference paper at ICLR 2023
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-smli: Natural language inference with natural language explanations, 2018. URL https::/arxiv..rr abs/1812. 01193.
Aman Chadha and Vinija Jain. ireason: Multimodal commonsense reasoning using videos and nat- ural language with interpretability, 2021. URL putttps///rxiyy org. org.orgg//oo.o.. /a////b/////// 20os/210 2217 10300.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang- HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026-1036, Online, Novem- ber 2020. Association for Computational Linguisties. doi: 10. (1653/v1/2022 wttp:///........nnneeeeemmmm findings-emnlp.91. URL https: / /etes//aaaaaattt aclanthology......ooo.oo.ooooahh org/ 2020 findings wwwiinnggsseppppp. 91.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Anmual Meeting the Association for Computational Linguistics, pp. 7748-2760 Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/P19 1264. URL https: ptttpp::/accaathhte http///aaaaanootyyy..o.o org/ 1264.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xic, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7358-7387 7370, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 12118/111122211 10116551122011.mmmmmnnn...53.. URL https: /t///acaaaa /aclaaaaatoooo.....coc.oooo 202/2021. emm p- ma n. 585.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli Eli5:
Long form question answering, 2019. URL https: : wttpab//rrrgg//////////a/ 09190.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett, News summarization and evaluation the era of gpt-3, 2022. URL https: //tp/vaivvv///////////a///aa.at/ wwt./or///////oo.///.//ooo abs//b/9...2./ 12356.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: secoding-enhanced bert with disentangled attention. ArXiv, abs/2006.00654, 2021.
Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of sherlock holmes: A dataset for visual abductive reasoning. 2022. doi: 1044550/ARXXV.2202.0022 2202. 04800.
URL https: //tp://aavvv.//..///a/eeeeeaaevr org/ /bs/
Joanna Hubbard, Macy A. Potts, and Brian Couch. How question types reveal student thinking: An experimental comparison of multiple-true trapple--tr--flsee and free-response formats. CBE Life Sciences Education, 16, 2017.
Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. R4c: A benchmark for evaluating systems to get the right answer for the right reason. In ACL, 2020.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2020. URL https: //tp///avvvvv///a//////eea/.awvl org/ abs 00..2..........
Harsh Jhamtani and Peter Clark. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-ansserrng. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 137-150, Online, November 2020. Association for Computational Linguisties. doi: 10.18653/v 1/1033//1122200.mmmmmmmmmaa ht https: http://aaaanttttoooeeee 20200000.0.0.000000002 www.:22000m0mmmmaammmmmmmmmmmmma emnlp-main. 10.
URL
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering, 2020. URL
https: /taps/////vv...//
orgg a 2004. 04906.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2, 2014.
11

---

Published as a conference paper at ICLR 2023
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl 00276. URL pettps:/accaantiii
(trg/Q19-1022
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian, Tellme- why: A dataset for answering why-questions in narratives. Findings of the Association for Com- putational Linguistics: ACI-IJCNIPP 202/. Association for Computational Linguistics, 2021. doi:
http:///2222...nnn-n--eeeeee findings--all.53, findings-aci. acw.angssacl.....
10.18653/v1/202
URL https: / /////// . (tte:/..116332211222222222
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: / hww.antooonoooooooooooooooooo.oe org /O 1013.
Jimmy Lin, Xueguang Ma, Sheng-Chich Lin, theng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Confer- ence Research and Development in Information Retrieval, SIGIR '21, pp. 2356-2362, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 9781450380379, doi: 00.1145/3404835344632.. URL https: / http:/dti..ogoo.///////o 111.1...11141448811111 344853444435354 0044445533333223 3463238.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: arxiv. org/ abs/a907....///.// 11692..
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Kei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. URL https::pp............ //////aaaa /arxivii.. /atp//aii.....a/eeee//../eea/aee /2tg//99///229....// 2209. 09513.
Gilbert Mohr, Johannes Engelkamp. and Hubert D. Zimmer. Recall and recognition of self- performed acts. Psychological Research, 51:181-187, 1989.
Judea Pearl. Causality. Cambridge university press, 2009.
Matt Post. A call for clarity reporting BLEU scores. Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https:////// www.waclll..........
0118-619..
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932-4942. Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/119-1455 URL https: / //aclanthology org. -t/.//9991111111
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henry Zhu, Xinchi Chen, Zhiheng Huang, Peng Xu, Andrew O. Arnold, and Dan Roth. Entailment tree explanations via iterative retrieval-generation reasoner. In NAACL-HILT. 2022.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4))333-33) 2009.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Re-evaluating word mover's distance, 2021.
URL attps://arxiv. org/ abs/2105. 14403.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. In ACI., 2020.
12

---

Published as a conference paper at ICLR 2023
Kurt Shuster, Jack Urbanek, Arthur D. Szlam, and Jason Weston. Am i me you? state-of of ff-the-art dialogue models cannot maintain an identity. In NAACL-HLT, 2022.
Yuan Sui, Shanshan Feng, Huaxiang Zhang, Jian Cao, Liang Hu, and Nengjun Zhu. Causality-
aware enhanced model for multi-hop question answering over knowledge graphs. Knowledge- Based Systems, 250: 108943, 2022. ISSN 0950-7051. doi:
101.////..r//111101............ 2022. 10894.. URL https:/ /www. sww.sceeiicciiiccccooo rect. com 50950705122004567.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguisticy; Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 1111665//11N11114222 URL ht
14211
James Thore, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1//118-10772 URL
https:: https///alaaathhhooooooo hww.:/looyyorg/g11111 org/N18-1074..
Soumya Wadhwa, Varsha Embar, Matthias Grabmair, and Eric Nyberg. Towards inference-oriented reading comprehension: Parallelqa. ArXiv, abs/1 8001033550 2018
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? In ICML, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 2022.
Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. Towards fine-grained causal reasoning and qa. arXiv preprint arXiv:2204.074400 2022.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, October-November 2018, Association for Com- putational Linguistics. doi: 10.18653/v1/D188 1259. URL ht /atp//ttaaaanooooooocaoao. .org/ D18-12 -129..
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. ArXiv, abs/2007. 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. ArXiv, 00t5/1994400675, 2020.
A APPENDIX
A.1 DATA COLLECTION
Our corpus consists the entirety of the English Wikipedia, snapshotted on 23 May 2022. Wikipedia presents list of curated "Good Articles", which have been nominated and reviewed to fit the "Good Article Criteria". Articles from this category are guaranteed to have correct spelling and grammar, as well as clear and concise diction. Our final keyword list includes: "because", "due to", "therefore", "consequently", "resulted in", "resulting in", and "as a result".
13

---

Published as a conference paper at ICLR 2023
A.2 DATA COLLECTION VALIDATON
Each stage in our data collection process is followed by two additional validation layers. For Stage 1. workers prohibited from submitting more than 20 entries until their annotations have been manually validated. The annotation result passes through another phase of manual validation to ensure that the quality is kept up after workers' initial submissions are accepted by quality control. For Stage 2, we track separate list of qualified workers for explanation quality.
Similar to Stage Stage 2's initial submit limit (the "speed bump") 10. Undergraduate students manually reviewed the examples from stage-2-qualifee workers. These panelists were instructed and shown demonstrations of marking explanations as satisfying or not and correcting minor errors for slight quality improvements. While manually approved workers write each WIKIWHY explana- tion, these hand-reviewed samples ultimately comprise the test and development sets. The continu- flow between stages is enabled by a backend system implemented to maintain database of submissions. This system serves inputs to both MTurk interfaces, as well as the front-end validation interfaces provided to the undergraduate panelists.
A.3 ADDITIONAL RESULTS
We include additional evaluations of our generated explanations using simple metrics, Table 4 shows performance on the EO task, and Table 5 show performance the A&E task. We also include results from the QA task in Table 7 and Table 8. Automatic evaluation on individual topics categories are included in Table 10.
A.4 CROWD WORKER INTERFACE
Figure and Figure 8 display the interfaces for the first and second stages respectively. In addition to the list of requirements, we provide examples and tips to further clarify our expectations. The passage is displayed with link to the full article so workers can view the complete context needed. Every passage contains highlighted causal connective, allowing workers to quickly scan and skip irrelevant portions. Each passage is retrieved from our custom database through our API. the passage is too difficult for the worker to understand or lacks a cause-effect relation, the worker can click the button below for another random passage.
14

---

Published as a conference paper at ICLR 2023
Table 4: Explanation Evaluation Results of WIKIWHY dataset according the following metries: SacreBLEU (S- (S-BLEU) Post (2018), Word-Mover's distance (WMD) Sato et al. (2021), Sentence Mover's Similarity Metrics (SMS) Clark al. (2019), BERT-F1 Score Zhang al. (2020), ROUGE-
1, ROUGE-2, and ROUGE-L (all ROUGE-fI Scores Lin (2004) averaged). SMS is scaled by 1000 readability.
Model
Fine-tuned GPT-2 vs. Few-shot GPT-3
S-BLEU
WMD
SMS
BERT-F1
ROUGE-1
ROUGE-2
ROUGE-L
GPT-2
Greedy
0.042
0,541
15.81
0.773
0.212
0.057
0,184
Temp 0.5
0.037
0.540
15.30
0.770
0.198
0.047
0,169
Temp 1.0
0.022
0.536
13.25
0.760
0.161
0,022
0,134
GPT-3
Temp 1.0
0.055
0.555
14.93
0.792
0.240
0.057
0.199
Table 5: GPT-3 explanation results with various input settings: Ideal- gold cause/answer, Well- Selected- provided cause/answer predicted by rest-performing reader model (FiD), End-to-end- provided only question/effect (GPT-3 completes end and-to-end task)
Model
GPT-3 Prompt Input Experiments
S-BLEU
WMD
SMS
BERT-fI
ROUGE-1
ROUGE-2 ROUGE-L
Input Setting
Ideal
0.055
0.555
14.93
0.792
0.240
0.057
0.199
Well-Selected
0.030
0.546
13.27
0.776
0.203
0.049
0.149
End-to-end 0.023
0.542
13.22
0.768
0.200
0.038
0.144
Table 6: WIKIWHY dataset contains diverse set 11 genres. The raw counts topic themes in articles is presented in the second column. The relative frequency is the percentage of ar- ticles in CausalQA sub-sampled from the Good Wikipedia articles list.
Table 7: Document Retrieval for WIKIWHY. BM25 consistently outperforms DPR.
MODEL
WIKIWHY
Top-1 Acc MRR
BM25
0.810
0.858
DPR
0.340
0.448
GENRES
RAW #
FREQ.
AGRICULTURE
131
0.436
ARTS
577
0.396
Table 8: Answer Evaluation Results for WIKI-
WHY dataset. Stage 1: RoBERTa, BigBird, and
ENGINEERING
952
0.336
GEOGRAPHYY
754
0.624
FiD. FiD Gold is fine-tuned on 80% train split & evaluated on 10% dev split.
HISTORY
1023
0.433
LITERATURE
455
0.340
MATHEMATICS
27
0.227
MODEL
WIKIWHY
MEDIA
1773
0.399
S-BLEU
BERT-11
WMD
MUSIC
1070
0.229
RoBERTa
NATURAL SCIENCES
2952
0.768
Gold
0.246
0.860
0.637
PIILOSOPHTY
302
0,465
BM25
0.214
0.832
0.620
BigBird
Gold
0.258
0.825
0.615
BM25
0.223
0.802
0.602
FiD
Gold
0.373
0.863
0.658
BM25
0.259
0.827
0.617
15

---

Published as a conference paper at ICLR 2023
Table 9: Examples from most frequent topics covered in WIKIWHY. denotes cause, effect, and 8; the ith rationale sentence.
Genres
Example
Geography
The geographic isolation of the Hupa homeland
Hupu's homeland sepurated by boties
Not many people could to the Hupn's homeland
The Hupa had few interactions with early European explorens up to the 19th cenituy
Literature
Increased language Increased contact between people requires increased communication
globalizing world
22 Speaker of uncommon languages switch to more common languages
Switching away from unzommon languages leads them heing forgotten Many small languagos are becoming codangered as their speakers shift other languages
Media
Seeing Castle Cagliostro eatrenched Yamazaki that Japan can make high-quality films Viewing The Castle of Caglinstro inspired Takashi Yamazaki
Out of national pride. Takashi Yamazaki followed model that he believed would produce quality films Castle of Cagliostro
Director Takashi Yamazaki modeled 2019 Lupin
Music
The duration Hotel California longer than songs generally played radio stations Most songs are only 3-4 minutes long
Hotel California
Peuple would not want to listen to same song on radio for that long
Don
Eagles song Hotel California
Natural Sciences
The thermal lemperatures change so drastically the rocks expand and cuntract integrity
22
process weakens the The boulders Cores are brittle and degrade rapidly
Technology
The use of coal power in Turkey pollution
81 Burning leads
Air pollution causes sickness and early death
83 Sick dead people cannot work million working
lost the population of Turkey in 2019
Table 10: Explanation performance (unordered fl) over the most frequent topics. We GPT-2 under the greedy setting and GPT-3 under the same defaults Table 2
Most Frequent Genres
ARTS
GEOG
HISTORY
MEDIA
MUSIC
SCIENCE
TECH
Models
GPT-2
0.256
0.221 0.372
0.202
0.161 0.335
0.239 0.301
0.252 0.371
0.236
GPT-3
0,412
0.341
0.333
Table 11: WikiWhy Summary Statistics
WikiWhy Statisties
# of Train
7,397
10
# of Dev
1,004
Prop. Chalnn
# of Test
1,005
30-1
# of Rationale
9,406
# of Rationale Elements
14,238
Avg. # Rationall Length
1.5137
Avg. # Tokens per Element
16.697
explanattoons
Figure 4: Rationale Length Distribution
16

---

Published as a conference paper at ICLR 2023
1. "Good Articles" from English Wikipedia
amazon
4. MTurk stages for cause, effect and explanation chain
Validation stage for quality control
?
H
mechanical turk
Parse articles for passages with causality
RAILS Database with REST API
WikiWhy
Figure 5: Dataset Collection and Validation Pipeline
Cause: There were time constraints to writing "Boruto: Naruto the Movie" Effect: Hiroyuki Yamashita felt pressured writing "Boruto: Naruto the Movie" Explanation: Creativity is difficult when put on a strict timetable. There was a need to both produce a good movie and do so on strict time budget. These two demands put stress on Hiroyuki Yamashita while he worked.
Cause: Homer P. Rainey had liberal views.
Effect: Homer P. Rainey was fired by the University of Texas in 1944. Explanation: If the University of Texas is conservative, they wouldn't want people working there who have liberal views.
Cause: the large size and reddish tint of red maple buds
Effect: Red maple buds which form in fall and winter are often visible from distance.
Explanation: The color red stands out from distance, so if the buds are red in the fall and winter, you'd be able to see them from distance.
Cause: There were advances in technology, lower energy prices, a favorable exchange rate of the United States dollar, and lower alumina prices.
Effect: Productions costs of aluminum changed in the late 20th century. Explanation: With advances in technology, prices of manufacturing change usually because they are now easier and cheaper to make. In this case it is aluminum that the price changed on because the technology improved the process.
Figure 6: GPT-3 Few-shot Exemplars
17

---

Published as a conference paper at ICLR 2023
Find a Cause-Effect Relation & Turn it into a Why Question.
Your question must be specific enough to make sense ON ITS OWN (without the passage)
Example 1: Use Full Names of People!
Example 2: Specify When and Where the situation is totally clear!
Your Passage (click link to find details)
Inttos wikppedia orgietkiChitssie Watts (link opens tab)
The antermath dominated EastEnders 2005 and helped to revive the fortures the show. Accorcing the former head BBC Drama Seridls, Mei Young, this was dependent on the character of Chrissie, who was responsible for "anchoring the success of the anniversary storyline. A similar sentiment was expressed by lan Hyland the Sunday Mirror, who although critical the convuluted piot felt EastEnders was improving "mainly because
daing bost resoue fallout performing "rescue auf wannabe", and felt her doparture was ennabiing EastEnders to move forward. contrast, the TV editor of The Daily Telegraph Telograph ha laled Chrissle as "helping rovive the show's fortunes that had been lagging somewhat in recent years".
storyline dirty show. However, Shellay of Daily Mirror highly criiial Chrissio, calling her "the ludiorous Lady ManBeth
click me only. the current passage lacks cause effect relation
Step 1: Find Cause & Effect from the Passage
To make the next step easler, write effect full sentence with details it's clear what the exact situation is repeat cause effect
Cause: write your specific cause here
Effect:
write your specific effect here
Step 2: Turn the Cause-Effect into Why Question & its Answer
The Question should ask about Effect, and the Answer should be Cause
Example: Cause=drug overdose, Effect=Heath Ledger died.
Question: did Heath Ledger die?
Your Question must make sense on Its own (without passage)
Add details (who, what, when, where) so it's clear what the exact situation is
Use full names for people, groups, places
Someone seeing only your QUESTION should NOT need ask any clarifying questions
They SHOULD NOT need to ask "which are you talking about?'
since your question should already be explicit about which
It's talking about
Why. Question:
queetion
Answer (Cause): (wrlle cause & effect first]]
Falling Please Note: Falling to follow instructions will result your Workerid being blocked from future tasks published by our By submitting, you agres to the torms this consent form.
Submit
Figure 7: Amazon Mechanical Turk Interface for Stage
18

---

Published as a conference paper at ICLR 2023
1. Choose Question to Answer (each involve cause & effect)
Why did "The similartt between The Fault in Our Stars and Perks Being Walllower: lead to "Staphen Chbosky turned down the Stars."?
opportunity to
Why did "Stophen Chbosky turned down the opportunity direct The Fault in Our Stars." result from "The similarity betwoon The Fault Our Stars and Perks of Boing a Wallfoowrr""?
does "The similarity between The Fault Our Stars and Perks of Being Walllower." "Stephen Chbossy turned down the opportunity to diract The Fault Our Stars.?
Why's "Stephen Chbosky
opportunity
consequence of similarity
Our Parks
Wallfower"?
Adcitional Context for the Questions
January 31, 2012, was announced that Fox 2000, a division 20th Century Fox, had optioned the rights to adapl John Green's produce
Wyek Godfrey Marty company, Temple Entertainment. Stephen Chbossky, who directed The Parks of Wallfower (alsa flimed Pittsbrgh), talks to direct the fllm but tumned down because similartty Perks, On February 18, 2013, Joeh Boone was hired as director; Scott Neusladter and Micheel H. Weber were hired to adspt the novel into ecreenplaytheir second adaptation for Fox, following Rosaline.
production
2. Answer the Question
Example 1:
Most explanations ons are fairyy short (1 entreas
Example 2: Most explanat ons only require basic logic
Examplc 8: You need to search online if you unsure of the answer
Requirements
Add new entry for each step/sentence
complete sentences with good spelling grammar
Carefully read the question you chose and actually respond to
DO NOT only write rephrase "cause leads effect*
We already know this! want you explain WHY that is the case
Your Question
The similerty direct Fault in Our Stars.
Fault
Perks Being Walffower." lead to "Stephen Chbosky tumed opportunity
he had directed both, could have endengered his nomination for the Academy Award "Perks Walffower*
Filing movies make look one pony.
Add Explonation
Submitting work with egregious grammar errors, inappropriately copied text, or nonsense anewers (or otherwise failing to follow instructions) your being submittingg you agree
Please Note:
tasks published by our group.
Submit
Figure 8: Amazon Mechanical Turk Interface for Stage
19


Reviews:
[{"review_id": "pxVcdxJpSuC", "reviewer": "Reviewer_tSWL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "## Summary\n\nThis paper introduces a new dataset, WikiWhy, along with an explanation task, that aims to measure the reasoning and common sense abilities of generative language models. The work describes the motivation for this new dataset and task, mainly that most existing question-answering tasks require limited reasoning and common-sense knowledge. The paper describes how the new dataset was collected, based on Wikipedia with a couple of MTurk tasks: (1) generating question-answer pairs given a specially selected passage, and (2) generating a series of sentence-oriented rationales. The paper describes experiments using baseline models (based on GPT-2 with fine-tuning, and GPT-3 with prompting), demonstrating low performance on the proposed task. Finally, the paper proposes an automatic metric for evaluating system outputs on the new task, demonstrating correlation of the metric with human judgements.", "summary_of_the_review": "Overall I’m leaning to recommend acceptance of this paper. It has become clear to the QA community that we should move beyond factoid seeking queries and “exact match” as an evaluation metric. This paper proposes an interesting task variant. Despite some of the flaws in the data collection protocol (see above), I think this resource will be of interest to the community. The task formulation may also be extended in the future for other types of rationales. Finally, the initial baseline experiments indicate a large gap for improvement.", "strength_and_weaknesses": "## Strengths\n\n1. The paper addresses an important weakness in many existing question-answering datasets that focus on answering questions with limited feedback on system reasoning or explanations. The proposed formulation of explicitly requiring systems to produce causal explanations to “why” questions is an interesting step,\n\n2. The proposed task formulation is intuitive and elegant. While the two types of explanations proposed (step sequence and sets) are likely not comprehensive, they probably cover a large proportion of explanations to simple ‘why’ questions,\n\n3. Collecting datasets is challenging, but can be helpful directing improvements to future systems. The release of this dataset to the community will likely help drive new work in QA explanations and reasoning.\n\n4. The baselines described in the paper are quite reasonable starting points for modeling,\n\n5. The human judgements of the baselines are surprisingly low which means that this task has large room for improvement in future work. (Note to authors: it would be nice if the baseline predictions were released along with your code+dataset so that they can be inspected by the community).\n\n## Weaknesses\n\n1. The main weakness in this work relates to the protocol used to elicit ‘why’ questions conditioned on a passage that contains a causal connective. Unfortunately, I believe this contains some of the same issues the community has observed with SQuAD task, namely: (1) generated questions are artificial since conditioned a specific passage, and rarely reflect naturally occurring ‘why’ questions, and (2) unusually high overlap in phrasing between the question and the answer passage, and (3) reliance of passage context for question to be fully interpretable. Fortunately, the paper addresses issue (3), but still has issues (1) and (2).\n\n   Looking over most questions in the dataset, they mostly contain the issues listed. Examples:\n\n   a) Question: \"Why did the member of The Cure, Lol Tolhurst, make minimal contributions to the album Disintegration?\"\n\n     Passage excerpt: “... revealed that while Tolhurst had contributed to the song \"Homesick\", his contributions to the rest of the album were minimal due to his alcoholism”\n\n\n    b) Question: “\"Why is it unlikely that any sequels would be made for the Power Rangers film?\"\n\n      Passage excerpt: “However, in May 2017, Forbes noted that due to the underwhelming performance of the film in most markets, it was unlikely any sequels would be made.”\n\n    In these examples (and others I browsed) it seems that (1) questions are formulated slightly unnaturally, (2) very high word overlap between answer phrase and question, (3) questions are all answerable within a single sentence.\n\n    It is probably too late and costly to ask authors to revise their protocol for question generation. For future reference, one protocol that avoids the pitfalls of SQuAD is the TyDiQA dataset (https://arxiv.org/pdf/2003.05002.pdf).\n\n 2. Probably due to the issue above, the retrieval performance using BM25 is unusually high  (Section 5.1). For comparison, the “Deep Passage Retrieval” paper (https://arxiv.org/pdf/2004.04906.pdf) shows BM25 performance in the 40s for Natural Questions. Similarly, answer identification should be quite easy as well for a model fine-tuned on this data given the high word overlap between answer phrase and question.\n\n   So while this dataset will be useful to understand whether models are able to produce good reasoning/explanations for certain types of ‘why’ questions, it is, unfortunately, not a good dataset for QA retrieval and answer identification.\n\n3. There is a part of the human evaluation (Section 5.3), specifically the Win/Tie/Lose which seems problematic. In our previous experience, disclosing to raters which outputs are “reference” and which are “system” biases ratings measurably. It is typically needed to make this type of analysis blind by (1) randomly shuffling which output is reference vs. system when presenting them to users, and (2) never disclosing the source of the explanation (reference or system).\n\n## Other notes / suggested citations or references:\n\n1. A dataset that seems quite related is ELI5 (https://arxiv.org/abs/1907.09190 , https://facebookresearch.github.io/ELI5/) which also aims to measure model’s ability to generate QA explanations.\n\n2. Another dataset is the TellMeWhy (https://arxiv.org/pdf/2106.06132.pdf) which asks models to produce simple, reasoning-based explanations for answers in the context of stories.\n\n3. One other dataset that I thought worth mentioning in the related work is e-SNLI (https://arxiv.org/pdf/1812.01193.pdf , https://github.com/OanaMariaCamburu/e-SNLI ). While not strictly a QA task, the explanations collected in e-SNLI are quite related to the ones presented in this work.\n\n4. Another system that should be referenced is WT5 (https://arxiv.org/pdf/2004.14546.pdf) which is a T5-based system for generating explanations/rationales. Some of the checkpoints are available at https://github.com/google-research/google-research/tree/master/wt5#released-model-checkpoints and could serve as a stronger baseline than GPT-2, if fine-tuned for your dataset/task.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nyz_rzG1IK", "reviewer": "Reviewer_V2He", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents WikiWhy, a Question Answering dataset of \"Why\" questions constructed from Wikipedia text. WikiWhy contains questions requiring two steps of reasoning or based on a rationale set.", "summary_of_the_review": "An interesting QA dataset. The paper has some minor issues.\n", "strength_and_weaknesses": "## Strong\n\n* The Cause and Effect QA with text generation problem is interesting (WikiWhy tasks 2 and 3)\n* The automatic evaluation metric proposed is interesting.\n\n## Weak\n\n* (Minor) Table 1 has messy formatting. The Size and Topic columns should be right-aligned to support easy comparisons.\n* Some hyper-parameters are missing (see the Clarity, Quality, Novelty, and Reproducibility section below).\n* (Minor) The Conclusion states, \"With this paper, we release WIKIWHY, a Question Answering dataset interested in understanding and facilitating the improvement of LLMs’ reasoning capability.\" The word \"interested\" is a strange choice for this sentence. I recommend rephrasing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ms0WC77Whx", "reviewer": "Reviewer_reEb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new QA task on answering and explaining cause-and-effect questions. Different from previous approaches, the authors create the WIKIWHY dataset which contains the explanation within cause-effect relations. Taking the effect as the question source, the goal is to not only get the answer (cause) but also predict the explanations in sequential or set settings. Experiments show that the state of the art generative models are good at QA task but still a large space for improvement on the explanation task.\n\n", "summary_of_the_review": "In general, this is an interesting work. Though there are some improvement space from the problem formulation and evaluation, this version of work is still good for the task proposal, dataset preparation and the experiment design.", "strength_and_weaknesses": "## Strength:\n- It is the first work that investigates Q&A over cause-effect reasoning with explanations.\n- The authors create a consolidated pipeline to define the problem and build the dataset, resulting a WIKIWHY dataset covering 11 topics.\n-The authors propose new evaluation metrics for this cause-and-effect question answering task. Experiments verifies the reasoning ability of LLMs. \n\n## Weakness:\n- The problem formulation in which a sequence or a set of explanations are required make this task intrinsically difficult to evaluate.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ydF5sg-LWb", "reviewer": "Reviewer_RHVP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Goal: The paper introduces a resource for explaining the cause-and-effect relationships on a wide variety of topics. \n\nMethod: The resource is crowdsourced in two steps. First, crowdworkers are asked to turn potential causal statements (identified via discourse cues) into Cause-effect questions. Then, the crowdworkers are asked to explain the cause effect relation to obtain “why” answers. \n\nUsing this process, a resource of about 9K why questions was curated from Wikipedia articles covering a wide variety of topics. \n\nBenchmarking: The paper presents benchmarking results that show the \n\nKey Contributions: The main contribution is in the resource itself. The resource presents a new test bed to assess abilities of NLP models to reason about cause and effect relations. The task requires a wide variety of knowledge and the benchmarking shows that the task is difficult for the current models and definitely warrants further research. \n", "summary_of_the_review": "The paper makes an interesting contribution to explainable reasoning for cause-effect relations, which can drive interesting research into this space.", "strength_and_weaknesses": "Strengths:\n\nThe paper introduces a high quality resource that fills a gap regarding evaluation of LLM’s ability to not only provide answers about causes and effects but also explain why using a wide variety of knowledge.\n\nThe paper is well-written and provides clear descriptions of how the data was collected, providing enough information to verify the quality of the resource.\n\nThe benchmarking experiments convincingly demonstrate the difficulty of the task. This resource could be a key driver for research into this problem space. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%