IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
Authors: Wenhao Yu
Meng Jiang
Peter Clark
Ashish Sabharwal
Venue: EMNLP 2023 Main (2023)
Keywords: open-domain question answering
counterfactual reasoning

Abstract: Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an ``if'' clause. Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset is highly challenging for existing open-domain QA methods, including supervised retrieve-then-read pipeline methods (F1 score 44.5), as well as recent few-shot approaches such as chain-of-thought prompting with ChatGPT (F1 score 57.2). We hope the unique challenges posed by IfQA will push open-domain QA research on both retrieval and reasoning fronts, while also helping endow counterfactual reasoning abilities to today's language understanding models.

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
Wenhao Yu Meng Jiang*, Peter Clark Ashish Sabharwal
Tecent AI Seattle Lab; * University of Notre Dame; Allen Institute for AI wenhaowyu@global. tencent... com; aashisss@alleaai. org
2000; Atherton, 2005). Counterfactual reasoning has long been considered necessary part of com- plete system for AL. However, few NLP resources aim at gauging the effectiveness of such reasoning capabilities in Al models, especially for the open- domain QA task. Instead, existing formulations of open-domain QA tasks mainly focus on questions whose answer can be deduced directly from global, factual knowledge (e.g., What was the occupation of Lovely Rita according the song by the Beat- les?) available on the Internet (Joshi et al., 2017; Kwiatkowski et al., 2019; Yang et al., 2018).
Abstract
Although counterfactual reasoning is funda- mental aspect of intelligence, the lack of large- scale counterfactual open-domain question- answering (QA) benchmarks makes it difficult evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an "if" clause. Such questions require models to go beyond retrieving direct factual knowl- edge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains 3,800 questions that were anno- tated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset highly challenging for exist- ing open-domain QA methods, including su- pervised sarrivve-then-eant pipeline methods (FI score 44.5), as well as recent few-shot ap- proaches such -hthn-tf-thought prompting with ChatGPT (FI score 57.2). We hope the unique challenges posed by IfQA will push open- open-domain QA research both retrieval and reasoning fronts, while also helping en- dow counterfactual reasoning abilities to to- day's language understanding models. The IfQA dataset can be found and downloaded https: https://allenatt wwtp//ra///igaaaa...
Counterfactual presupposition in open-domain domain QA can viewed as a causal intervention. Such intervention entails altering the outcome of events based on the given presuppositions, while obeying the human readers' shared background knowledge of how the world works. To answer such questions, models must go beyond retrieving direct factual knowledge from the Web. They must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters.
Although some recent work has attempted to an- swer questions based on counterfactual evidence in the reading comprehension setting (Neeman et al., 2022), or identified and corrected false presuppo- sition in given question (Min al., 2022), none of existing works have been developed for evaluating and improving counterfactual reasoning capabili- ties in open-domain QA scenarios. To this gap. we present a novel benchmark dataset, named IfQA, where each of over 3,800 questions based on a counterfactual presupposition defined via an "if" clause. Two examples are given in Figure 1. TfQA combines causal inference questions with factual text sources that are comprehensible without an understanding of formal causation. It also allows us to evaluate the capabilities and lim- itations of recent advances in QA methods the context counterfactual reasoning.
1 Introduction
Counterfactual reasoning captures human tendency to create possible alternatives to past events and imagine the consequences of something that is contrary to what actually happened or is factually true (Hoch, 1985). Take, for example, the business arena where corporate leadership team might rig- orously analyze the potential ripple effects had they opted for an alternative investment strategy (Baron,
layman
The majority of the work was completed during Won- hao's internship at the Allen Institute for Al.

---

Question: If the American Kennel Club had not granted the English Cocker Span- iel separate breed name, what breed of dog would they have been shown with?
English Cocker Spaniel
-Wiik: The English Cocker Spaniel and American Cocker Spaniel shown together in America until English Cocker Spantel received status as separate breed. The American Kennel Club granted separate breed designation to the English Cocker Spaniel in 1946. The American
English American
"
Question: the movement of the earth's 's crust caused the height of Mount Eve- rest to drop by 300 meters, what would be the highest mountain the world?
- "
WWiki: Mount Everest is Earth's highest mountaiin above sea level, its elevation of 8,848m was most recently established. Wiiki K2 (also known as Godwin Austen at 8,61 meters above sea level, is the second-highan mountäin on Earth, after Mount Everest. It lles in the Karakoram range. partially in
No. Mouriaah name.
Height
Mount Everest
8,548m
Godwin Austen
8,611m
Kangchenjunga 6.586m
"
Figure I:
In the IfQA dataset, each question is based on counterfactual presupposition via an "if" clause. To answer the question, one needs to retrieve relevant facts from Wikipedia and perform counterfactual reasoning.
IfQA introduces new challenges in both retrieval and reading. For example, to answer the 2nd exam- ple question in Figure 1, "If the movement of the earth's crust caused the height of Mount Everest drop by 300 meters, which mountain would be the highest mountain in the world?", the search and reasoning process can be divided into four steps: (i) retrieve documents relevant to the current height of Mount Everest (8,848 metres); (ii) cal- culate the height based the counterfactual presup- metres); (iii) retrieve documents relevant the current second-highens mountain in the world (K2: 8,6/1 metres); and (iv) compare the heights lowered Mount Everest and K2, then generate the answer (K2).
and reasoning. Specifically, we make the following observations. First, in retrieval, traditional dense re- trieval methods based on semantic matching cannot well capture the discrepancy between counterfac- tual presuppositions and factual evidence, resulting failing to retrieve the gold passages in nearly 35% of the examples. Second, state-of-the-ar reader models, such as FiD, achieve an score of only 50% even when the gold passage is contained in the set of retrieved passages. Third, close-book CoT reasoning can effectively improve the end-QA per- formance, but still heavily lags behind open-book models. Lastly, combining passage retrieval and large model reasoner achieves the best results.
position (8,
We hope the new challenges posed by IfQA will help push open-domain QA research in an interest- ing new direction, as well as towards more effective general-purpose purpose retrieval and reasoning methods.
To establish an initial performance level on IfQA, we evaluate both state-of-the-art close-book and open- book- models. Close-book models, such chain-of-thouge (CoT) reasoning with Chat- GPT (Wei et al., 2022), generate answers and op- tionally intermediate reasoning steps, without ac- cess to external evidence. On the other hand, open- book models, such as RAG (Lewis et al., 2020) and FiD (Izacard and Grave, 2021), first lever- age retriever over large evidence corpus (e.g. Wikipedia) to fetch set of relevant documents, then use reader to peruse the retrieved documents and predict answer.
2
Related Work
2.1 Open-domain Question Answering
The task of answering questions using large col- lection of documents (e.g., Wikipedia) of diver- sified topics, has been longstanding problem in NLP, information retrieval (IR), and related fields (Chen et al., 2017: Brill et al., 2002: Yu al., 2022). large number of QA benchmarks have been released in this space, spanning the different types of challenges represented behind them, in-
Our experiments demonstrate that IfQA a chal- lenging dataset for both retrieval as well as reading

---

cluding single-hop questions (Joshi et al., 2017:
terfactual samples into model training improves the generalization ability (Kaushik et al., 2019).
Kwiatkowski et al., 2019; Berant et al., 2013), multi-hop questions (Yang et al., 2018: Trivedi et al., 2022), ambiguous questions (Min et al., 2020), multi-answer questions (Rubin et al., 2022; Li et al., 2022), multi-modal questions (Chen et al., 2020; Zhu et al., 2021a), real time questions (Chen et al., 2021; Kasai et al., 2022), and etc.
inspiring a line of research to explore incorporat- ing counterfactual samples into different learning paradigms such as adversarial training (Zhu et al., 2020) and contrastive learning (Liang et al., 2020), These work lie in the orthogonal direction of in- corporating counterfactual presuppositions into model's 's decision-making process.
To the best of our knowledge, all existing for- mulations assume that each question is based on factual presuppositions of global knowledge. In contrast, the questions in our IfQA dataset are given counterfactual presuppositions for each question, so the model needs to reason and produce answers based on the given presuppositions combined with the retrieved factual knowledge. This makes IfQA a novel and qualitatively different dataset.
In the field of NLP, existing counterfactual in- ferences are ubiquitous in many common infer- ence scenarios, such as counterfactual story gen- eration (Qin et al., 2019), procedural text genera- tion (Tandon et al., 2019). For example, in TIME- TRAVEL, given an original story and an intervening counterfactual event, the task is to minimally revise the story to make it compatible with the given coun- terfactual event (Qin et al., 2019). In WIQA, given procedural text and some perturbations to steps mentioned in the procedural, the task is predict whether the effects of perturbations to the process can be predicted (Tandon et al., 2019). However, to the best of our knowledge, none of existing bench- mark datasets was built for the open-domain QA.
Mainstream open-domain QA methods employ a retriever-reader architecture, and recent follow- up work has mainly focused on improving the re- triever or the reader (Chen and Yih, 2020; Zhu et al., 2021b; Ju et al., 2022). For the retriever traditional methods such as TF-IDF and BM25 explore sparse retrieval strategies by matching the overlapping contents between questions and passages (Chen al., 2017; Yang et al., 2019). DPR (Karpukhin al., 2020) revolutionized the field by utilizing dense contextualized vectors for passage indexing. Furthermore, other research improved the perfor- mance by better training strategies (Qu et al., 2021; Asai et al., 2022), passage re-ranking (Mao et al.. 2021) and etc. Recent work has found that large lan- guage models have strong factual memory capabili- ties, and can directly generate supporting evidence in some scenarios, thereby replacing retrievers (Yu al.. 2023). Whereas for the reader, extractive readers aimed to locate a span of words in the re- trieved passages as answer (Karpukhin et al., 2020; Iyer et al., 2021; Guu et al., 2020). On the other hand, FiD and RAG, current state-of-the-e read- ers, leveraged encoder-decoder models such as T5 to generate answers (Lewis et al., 2020; Izacard and Grave, 2021; Izacard al., 2022).
3 IfQA: Task and Dataset
3.1 Dataset Collection
All questions and answers in our IfQA dataset were collected on the Amazon Mechanical Turk (AMT)!, a crowdsourcing marketplace for individuals to out- source their jobs to a distributed workforce who can perform these tasks. We offered all AMT work- ers $0.8 per annotation task, which leads to $15 to $20 per hour in total. To maintain the diversity of labeled questions, we set limit of 30 questions per worker. the end, the dataset was annotated by total of 188 different crowdworkers.
Our annotation protocol consists of three phases. First, we automatically extract passages from Wikipedia which are expected to be amenable to counterfactual questions. Second, we crowdsource dnestion-answer pairs on these passages, eliciting questions which require counterfactual reasoning. Finally, validate the correctness and quality of annotated questions by one or two additional work- ers. These phases are described below in detail, and the annotation task form shown in Figure 3.
2.2 Counterfactual Thinking and Causality
Causal inference involves question about coun- terfactual world created by taking an intervention, which have recently attracted interest in various fields of machine learning (Niu et al., 2021), in- cluding natural language processing (Feder et al., 2022). Recent work shows that incorporating coun-
pettps:/www..nturrt

---

Table 1: Example questions from the IfQA dataset, with the proportions with different types of answers.
Answer Type
Passage (some parts shortened)
Question
Answer
Entity (49.7%)
LeBron James: ... On June 29, 2018, James opted
If LeBron James had not been (Cleveland)
out of his contract with the Cavaliers and became unrestricted free agent. On July I, his manage- ment company, Klutch Sports, announced that he would sign with the Los Angeles Lakers.
traded to the Los Angeles Lak- Cavaliers
ers, which team would he have played for in 2018-2019 season?
Number (15.9%)
7-Hleven: ... Japan Co., Ltd. in 2005, and is now held by Chiyoda, Tokko-based Seven & Hold- ings. 7-Eleven operates, franchises, and licenses 71, 100 stores in 17 countries as of July 2020.
If 7-Hleven expanded its reach 22 (countries)
to five more countries in 2020, how many countries would have
7-Eleven by the end the year?
Date (14.5%)
2020 Summer Olympics: ... originally scheduled to take place from 24 July to 9 August 2020, the event was postponed to 2021 in March 2020 result of the COVID-19 pandemic....
If Covid- hadn't it spread rapidly July 24, 2020
across the globe, when would the Tokyo Olympics in Japan start?
Others (19.9%)
1991 Belgian Grand Prix: Patrese's 's misfortune promoted Prost to second, with Nigel Mansell third, Gerhard Berger fourth, Alesi fifth, and NeI- son Piquet sixth while the sensation of qualifying, Schumacher, was an amazing seventh... ...
If Gerhard Berger and Nelson Pi- fourth quet had switched starting posi- tion at the 1991 Belgian Grand Prix. what would have been Nel- son Piquet's starting position?
Massospondylus: ... "Pradhania" was originally If the new clade analysis per- Basal sauropo- regarded as more basal sauropodomorph hut new cladistic analysis performed by Novas al.. 2011 suggests that "Pradhania" is a mas- sospondylid. "Pradhania" presents
formed by Novas in 2011 did domorph not indicate that "Pradhania" a large vertebrate, what animal would have been identified as?
3.1.1 Question and Answer Annotation
when examples annotated questions provided. Therefore, we provided workers with five questions at the beginning of each HIT to better prompt them to annotate qucstions and answers. However, we noticed that fixed examples might bring some bias to annotation workers. For example, when we pro- vided the following example: If German football club RB Leipzig doubled their donation to the city of Leipzig in August 2015 to help asylum seek- ers, how many euros would they donate in total? The workers would be more inclined mimic the sentence pattern to annotate questions, such as: If Wells Fargo doubled its number of ATMs world- wide by 2022, how many ATMs would it have? To enhance the diversity of annotated questions, we devised new strategy. Instead of providing the same fixed examples, we presented five exam- ples randomly sampled from previously annotated examples for each new annotation task. This ap- proach ensures that each annotator sees distinct examples, thereby sparking creativity and minimiz- ing the likelihood of bias in the annotation process.
(1) Passage Selection. Creating counterfactual presupposition based on a given Wikipedia page is a non-trivial task, requiring both the rational- ity of the counterfactual presupposition and the predictability of alternative outcomes. Since the entire Wikipedia has more than 6 million entries, we first perform preliminary screening to fil- ter out passages that are not related to describing causal events. Specifically, we exploit keywords to search Wikipedia for passages on causality (e.g.. lead to, cause, because, due to, originally, initially) on events, particularly with a high proportion of past tense, as our initial pilots indicated that these passages were the easiest to provide a counterfac- tual presupposition about past events. Compared with randomly passage selection, this substantially reduces the difficulty of question annotation.
(2) Question Annotation. To allow some flexi- bility in this question annotation process, in each human intelligence task (HIT), the worker received a random sample of 20 Wikipedia passages and was asked to select at least 10 passages from them to annotate relevant questions.
Additionally, we allow workers to write their own questions if they want to do so or if they find it difficult to ask questions based on a given Wikipedia passage (see annotation task form
During the etrly-stage annotation, we found that the quality of annotation was significantly low

---

Table Data statistics IfQA, for both supervised and few-shot settings.
IfQA-S: Supervised Setting
IfQA-F: Few-shot Setting
Train
Dev.
Test
Train
Dev.
Test
Number of examples
2400
700
700
600
1600
1600
Question length (words)
23.35
23.26
23.09
23.78
23.29
23.10
Answer length (words)
1.84
1.83
1.71
1.88
1.83
1.78
Vocabulary size
12,902
5,460
4,596
4,578
9,967
9,339
Figure 4). Such annotation process can prevent the workers from reluctantly asking a question for a given passage. At the same time, workers can be encouraged to ask interesting questions and increase the diversity of data. We require that this self-proposed question must also be based on Wikipedia, and the worker is required to provide the URL of Wikipedia page and copy the correspond- ing paragraph. Ultimately, 20.6% of the questions were annotated in this free-form annotation.
3.1.3 Answer Post-processing
Since the answers are in free forms, different sur- face forms of the same word or phrase can make syntactic matching based end end-QA evaluation unreli- able. Therefore, we further normalize the different types of answers as follows and include them in addition to the original article span.
Entity. Entities often have other aliases. For exam- plc, the aliases of "United States" include "United States of America", "USA", "U.S.A", "America", "US" and etc. The same entity often exists with dif- ferent aliases in different Wikipedia pages. There- fore, in addition shown in the given passage, we add the canoni- cal form the entity - the title of the Wikipedia page to which the entity corresponds.
(3) Answer Annotation. Workers then are required to give answers to the annotated questions. We provided additional answer boxes where they could add other possible valid answers, when appropriate.
the entity aliases currently
3.1.2 Question and Answer Verification
Number. number could be written numeric and textual forms, such as "5" and "five", "30" and "thirty". When the number has a unit, such as "5 billion", it is difficult for us to traverse all possi- ble forms, such as "5,000 million" and "5,000,000 thousand".
The verification step mainly evaluates three dimen- sions of the labelled questions in the first step.
Q1: this a readable, passage-related question? The first question is used to filter mislabeled ques- tions, such as unreadable questions and questions irrelevant to the passage. For example, we noticed that very few workers randomly write down ques- tions, in order to get paid for the task.
annotate the answer based the unit that appears in the given Wikipedia passage, for example, if the word "billion" appears in the given passage, we take "5" as the numeric part, so only "5 billion" provided as an additional answer.
Q2: Is the question not well-defined without the Wikipedia passage? L.e., can the question not be properly understood without the passage as the con- text? not, could you modify the question make it context-free? This ensures that the questions are still answerable without the given passage, to avoid ambiguity (Min et al., 2020).
Date. In addition of keeping the original format mentioned in the given passage, we use the ISO 8601 standard to add an additional answer, namely "Month Day, Year", such as "May 18, 2022".
3.2 Dataset Analysis
Answer Type and Length. The types of answers can be mainly divided into the following four cat- egories: entity (49.7%), date (14.5%), number (15.9%), and others (19.9%), shown in Table The "others" category includes ordinal numbers, combinations of entities and numbers, names of people or location that do not have a Wikipedia entry, and etc. The average length of the answers in IfQA is 1.82 words, mainly noun words, noun
Q3: Is the given answer correct? If not, could provide the correct answer to the question? The third question is to ensure the correctness of the answer. If the answer annotated in the first step is incorrect, it can be revised in time from the second step. If the workers submit different answer, further add one more worker, so that total of three workers answered the question, thereby selecting the final answer by voting.
Phttps://ent https:///ewwwwiieeeeeeeeeee wwkipedia wwkipadia.org/wiki/tto... wwggw/kii/soo.ssa.

---

80
BBM25 DDPR
73.5
69.9
50
DRAG-I FiD-s mFiD
44.5
70
45
50.5
39.5
60
S5.7
54.2
40
36.1
36.6
50
47.7
35
31.4
40
36.9
33.5
30
28.5
30
25
20
20
10
15
10
R@05
R@20
R@100
EM score
score
(a) Retrieval performance, measured by R
(b) Reader performance, measured by EM FI.
Figure 2: Retrieval and end-QA performance using the retrieve-then-read models on the IfQA-S split. It should be noted that under the supervised setting, all models, except BM25, are fine-tuned on the training split. For retrieval, BM25 demonstrates superior performance than DPR. For end-QA, FiD-I demonstrates the best performance.
phrases, or prepositional phrases. This answer length is similar to many existing open-domain QA benchmarks, such as NQ (2.35 words), TriviaQA (2.46 words), and HotpotQA (2.46 words).
important. Our dataset requires the model to rea- son over counterfactual presuppositions, which is a natural test bed for evaluating their counterfac- tual reasoning abilities. Therefore, we also set up another split for few-shot learning (IfQA-F) that has only 600 examples for training, and half of the rest for validation and half for test. The dataset statistics of two splits are shown Table 2.
Question Type and Length. The types of ques- tions can be mainly divided into the following seven categories according to the interrogative words: what (51.7%), who (14.6%), when (5.1%), which (10. (1), where (3.5% and how many/much (12.0%). Among the seven categories, "what" has the highest proportion, but it also includes some questions about time/date or location, such as "what year" and "what city". The average length of question in IfQA is 23.2 words, which are signifi- cantly longer than many existing open-domain QA benchmarks, such as NQ (9.1 words), TriviaQA (13.9 words), HotpotQA (15.7 words), mainly due to the counterfactual presupposition clause.
4 Experiments
4.1 Retrieval Corpus
We use Wikipedia as the retrieval corpus. The Wikipedia dump we used is dated 2022-05-01 and has 6,394,490 pages in total. We followed prior work (Karpukhin et al., 2020; Lewis et al., 2020) to preprocess Wikipedia pages, splitting each page into disjoint 100-word passages, resulting 27.572,699 million passages in total.
Span vs. Non-span Answer. As the question an- notation is based on the given Wikipedia passage. most answers (75.15 1 %) in the dataset are text spans extracted from the provided passage. Non-span answers usually require some mathematical reason- ing (e.g., the 2nd example in Table 1) combining multiple text spans in the passage (e.g., the 3rd example Table 1) as the final answer.
4.2 Comparison Systems
Closed-book models are pre-trained models that store knowledge in their own parameters. When answering a question, close-book models (Codex and ChatGPT (Brown et al., 2020)) only encode the given question and predict answer without access to any external non-parametric knowledge. Instead of directly generating the answer, chain-of- thought (CoT) leverages ChatGPT to generate a series of intermediate reasoning steps before pre- senting the final answer (Wei et al., 2022).
3.3 Dataset Splits
We provide two official splits of our dataset. The first one is regular split for supervised learning (IfQA-S). This split has 2,400 (63.2%) examples for training, 700 (18.4%) examples for validation and 700 (18.4%) examples for test. With the pop- ularity of large language models, the reasoning ability of the model the few-shot setting is also
Open-Book models first leverage retriever over large evidence corpus (e.g. Wikipedia) to fetch set of relevant documents that may contain the
Phttps://dumps. wikimedia.oo

---

Table 3: End-QA performance on both IfQA-S and IfQA-F splits. We can observe that combining passage retricval and large model reasoner achieve the best performance, the entire pipeline enjoy both the factual evidence provided by the retriever and the powerful deductive reasoning ability of the large language model. It should be noted that all models are deployed under the few-shot setting, even when being evaluated on the supervised split.
IfQA-S: Supervised Setting
IIQA-F: Few-shot Setting
Methods
Codex
ChatGPT EM
Codex
ChatGPT EM F1
EM 1
F1
Fl
EM
F1
"without retriever, and not using external documents
ChatGPT (QA prompt)
25.25 32.91
24.42 34.33
25.73 32.88
25.25 | 34.49
Chain-of-thouggh (CoT)
27.39 34.22
25.55 35.82
27.08 I 34.28
25.75 I 35.96
"with retriever, and read passages using ChatGPT (few-shot)
DPR + ChatGPT
40.80 I 48.82
40.65 48.96
(DPR is only for supervised setting)
BM25 + ChatGPT
46.08 55.27
46.28 57.81
46.81 I 55.46
45.56 57.21
answer, then reader to peruse the retrieved doeu- ments and predict an answer. The retriever could be sparse retrievers, such as BM25, and also dense retrievers, such as DPR (Karpukhin et al., 2020), which a dual-encodec based model. Whereas for the reader, FiD and RAG, current state-of-the-- art readers, leveraged encoder-decoder models, such T5 (Raffel et al., 2020), to generate answers (Lewis et al., 2020; Izacard and Grave, 2021).
learning rate of le-5, We used Adam (Kingma and Ba, 2015) as the optimizer, and set its hyperparame- ter e to 1e-8 and (31, .2) to (0.9, 0.999). The batch size is set as 32 on 8x32GB Tesla V100 GPUs.
Reader. We employed the FiD (Izacard and Grave, 2021) model that built up on T5-large (Raf- fel et al., 2020). For model training, we used AdamW (Loshchilov and Hutter, 2019) with batch size 32 on 8x32GB Tesla V100 or A A100 GPUs. We experimented with learning rates of le-5/3e- 5/6e-5/le-4 and we found that in general the model performed best when to 3e-5. All reader models were trained with 20,000 steps in total where the learning rate was warmed up over the first 2,000 steps, and linear decay of learning rate.
4.3 Evaluation Metrics
Retrieval Performance. We employ @ecall@K (short as R@K K) as an intermediate evaluation met- ric, measured as the percentage of top-K K retrieved passage that contain the ground truth passage.
End-QA Performance. We two commonly used metries to evaluate the end- end-QA performance: exact match (EM) and Fl score (Karpukhin et al., 2020; Izacard and Grave, 2020; Sachan et al., 2022). EM measures the percentage of predictions having an exact match in the acceptable answer list. Fl score measures the token overlap between the pre- diction and ground truth answer.
4.5 Results and Discussion
(1) Retrieval in IfQA is challenging. As shown in Figure 2, when retrieving 20 Wikipedia pas- sages, both sparse and dense searchers could only achieve Recall @0 scores of about 60%, so the reader model cannot answer the remaining 40% of questions based on accurate supportive evidence. Although recall goes higher when more number of passages retrieved, would significantly increase the memory cost of the reader model, making it hard to further add complex reasoning modules. This phenomenon of rapid increase in memory cost is also observed in FiD (Izacard and Grave, 2021), i.e., when reading 100 passages, 64 V100 GPUs are required to train the model. Besides, when using large language models for in-context learn- ing, more input passages lead to an increase in the number of input tokens, limiting the number of in-context demonstrations. For example, the lat- est variants of ChatGPT, such as code-davinci and text-davinci, have an input limit of 4096 tokens.
4.4 Implementation Details
Under the supervised learning setting, the DPR retriever (Karpukhin et al., 2020) and FiD reader (Izacard and Grave, 2021) are fine-tuned on the IfQA-S training split. The implementation details of training are as follows.
Retriever. We employed two independent pre- trained BERT-base models with 110M parame- ters (Devlin al., 2019) as query and document encoders. BERT-base consists of 12 Transformer layers. For each layer, the hidden size is set to 768 and the number of attention head is set to 12. All dense retrievers were trained for 40 epochs with a

---

Table 4: Case study; a comparative analysis of two IfQA questions and answers generated by different methods.
Question: If the Game Boy Color video game Elmo's ABCs was released in North America at the same time as it was released in Europe, in what year was released in Europe?
Wikipedia passage: Elmo's s ABCs is 1999 educational Game Boy Color video game. It was relcased in March 1999 in North America. Like "Elmo's 's 123s". it was also released on April 6, 2001 in Europe....
ChatGPT (QA prompt, No Retrieval): 2000 DPR + FiD: 1999 BM25 + ChatGPT: 2001 V
Ohain-of-though (CoT), No Retrieval: The Game Boy Color video game Elmo's ABCs was released in North America on October 31, 2000, and in Europe on March 9, 2001. So, if the game was released in Europe at the same time as North America, would be available in Burope in 2000. The answer is 2000 X
Gold answer: 2001. Due to the lack of retrieved evidence, ChatGPT (QA prompt) and Chain-of-thought can hallucinate factual events, resulting in wrong answers. DPR-FiD, on the other hand, relies on retrieved passages from existing factual evidence to generate responses. However, it still produces inaccurate answers due to its limited reasoning ability. Only BM25+ChaGPT generates the accurate answer.
adopt local time zones instead of single standard time, what would he the maximum time difference between the easternmost and westermmost parts of the country?
Question: If China
Wikipedia Document: The time in China follows a single standard time offset of (eight hours ahead of Coordinated
Universal Time), even though China spans almost five geographical time zones. The official national standard time is called "Beijing Time", domestically and "China Standard Time" internationally.
ChatGPT (QA prompt, No Retrieval): Five X DPR + FiD: five X BM22 + ChatGPT: Five X
Chain-of-though (CoT), No Retrieval: China spans five geographical time zones, if each region were have its own local time zone, the time difference between the eastermmost and westernmost parts would be five hours X Gold answer: Four. Although China spans five geographical time zones, the maximum time difference between the easternmost and westernmost parts would be four hours, not five. All provided methods erroneously suggest five-hour difference, underscoring a limitation in their reasoning capabilities.
Furthermore, the IfQA benchmark has some unique features in terms of retrieval compared to existing open-domain QA benchmarks. On one hand, questions in IfQA datasets are usually longer than many existing QA datasets (e.g. NQ and Triv- iaQA), because each question in IfQA contains a clause mentioning counterfactual presuppositions. The average question length of questions in IfQA (as shown in Table 2) is 23.2 words, which is much higher than the question length in NQ (9.1 words), TriviaQA (13.9 words), HotpotQA (15.7 words) and etc. Longer questions make current retrieval methods based on keyword matching (e.g., BM25) easier because more keywords are included in the question, but make latent semantic matching (e.g.. DPR) methods harder because single embedding vector cannot well represent enough Information. On the other hand, in many cases, the retriever suf- fers from fetching relevant documents by simple semantic matching because of the discrepancies between counterfactual presuppositions and factual evidence. For example, the question "If the sea level continues to rise at an accelerated rate, which country is likely to be submerged first?", the tar- geted passage for retrieval might not directly men- tion "sea level", "rise", and "submergerd", where the question is essentially to ask "which country is the lowest-lying one in the world".
(2) Reading and reasoning in IfQA are challeng- ing. Deriving answers from retrieved passages quires reader models to reason over counterfactual presuppositions in questions and retrieved factual passages. As shown in Figure 2, even the state-of- the-art reader model FiD struggles. In the subset of examples where the retrieved passages contained the golden passages, only around 40% of the an- swers are correct. Thus, while FiD can achieve state-of-the-an performance many open-domain QA benchmarks, without any reasoning module performs poorly on IfQA. We also find that the FiD model performs worse (around 32%) on ques- tions that require some complex reasoning, such as numerical reasoning examples.
(3) Chain-of-thought improves LLMs' counter- factual reasoning. LLMs perform particularly well on reasoning tasks when equipped with chain- of-thought (Wei et al., 2022) to generate a series of intermediate reasoning steps before presenting the final answer. Since IfQA requires models to reason over counterfactual presuppositions, pothesize that such a reasoning process would also be effective on IfQA. Table shows that chain-of- thought generation, which was mainly evaluated in complex multi-step reasoning questions earlier, can effectively improve the performance of LLMs
hy-

---

on IfQA. However, since LLMs are closed-book models, they still lack non-parametric knowledge and, on IfQA, lag behind state-of-the- retrieve- then-read methods, such as FiD.
present them human annotators, which might lead to pattem-oriented bias in the annotated data.
Acknowledgements
(4) Passage retriever + Large model reasoner performs the best on IfQA. We saw that passage retrieval is necessary step for IfQA. In the ab- sence of grounding evidence, it is difficult for even LLMs accurately find relevant knowledge from parameterized memory, and accurately predict an- swer. From the results, the performance of close- book models on IfQA data is also far behind the retrieve-then tee--teheerrdddddd models. However, an inherent disadvantage of relying on small readers is that they do not enjoy the world knowledge or deduc- tive power of LLMs, making reasoning based on retrieved passages perform poorly. Therefore, provided in-context demonstrations to ChatGPT, and prompt read the retrieved passages, so that the entire pipeline can enjoy both the factual evi- dence provided by the retriever and the powerful reasoning ability of the large language reader. As shown in Table 3, we found that the combination of BM25 (as retriever) and ChatGPT (as reader) can achieve the best model performance.
This work was supported by NSF ISS-21 IIS-2137396, USS-2142827, IIS-2234058, CCF- 1901059, and ONR N00014-22 1-2507. Wenhao Yu is also supported by Bloomberg Data Science Ph.D Fellowship and Tencent AT Lab.
References
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard. Sebastian Riedel, Hannaneh Ha- jishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXh::221
Andrew Atherton. 2005. A future for small business? prospective scenarios for the development of the economy based on current policy thinking and coun- terfactual reasoning. Futures, (787777777
Robert A Baron. 2000. Counterfactual thinking and venture formation: The potential effects of thinking about "what might have been". Journal of business venturing. 15()):99-91
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freehase from question-answe pairs. EMNLP, pages 1533- 1544.
5 Conclusion
We introduce IfQA, a novel dataset with 3,800 ques- tions, each of which is based on counterfactual presupposition and has an "if" clause. Our empir- ical analysis reveals that IfQA is challenging for existing open- open-domain QA methods in both retrieval and reasoning process. It thus forms valuable re- source to push open-domain QA research on both retrieval and counterfactual reasoning fronts.
Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the askmsr question-answering system. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 257-264
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
6 Limitations
The main limitation of IfQA dataset is that it only covers event-based questions, due to the nature of creating counterfactual presuppositions. Therefore, our dataset is not intended for training general open- domain QA models or evaluate their capabilities.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1870-1879,
For data collection, we relied heavily on human annotators, both for question annotation and veri- fication. Despite efforts to mitigate annotator bias by providing explicit instructions and exam- ples and by sampling annotators from diverse popu- lations, is not possible to completely remove this bias. Besides, we use heuristic rules to select only a small portion of Wikipedia passages and then
Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. In Proceedings of the 58th an- nual meeting of the association for computational linguistics: nutorial abstracts, pages 34-37.
Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W Cohen. 2020. Open question answering over tables and text. In ternational Conference on Learning Representations.

---

Wenhu Chen, Xinyi Wang. and William Yang Wang. 2021. A dataset for answering sensstvnitive ques- tions. In Thirty-fitth Conference Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)) pages 6769-6781
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
Jungo Kasai, Keisuke Sakaguchi. Yoichi Takahashi. Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's s the answer right now? arXiv preprint 23333:2207.13332.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a differ- ence with aaanttrrfacuall--agemennt data. In Inter- national Conference on Learning Representations.
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference in natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Transactions of the Association for Computational Linguistics, 10:1 138-1158.
Pryzant, Dhanya Sridhar, Zach
Diederik P Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In International Conference for Learning Representation (ICLR).
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: A benchmark for question answering research. TACL, pages 452- 466.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXi::2002.0890
Stephen Hoch. 1985. Counterfactual reasoning and accuracy predicting personal events. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11(4):719.
Patrick Lewis, Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel. et al. 2020. Retrieval- augmented auggeeeeede generation for knowledge-intesivee nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9473
Srinivasan Iyer. Sewon Min, Yashar Mehdad, and Wen- tau Yih. 2021. Reconsider: Improved re-ranking using span-focused cross-attentoon for open domain question answering. In Procs. of NAACL-HLI
Haonan Li, Martin Tomko, Maria Vasardani, and Tim- othy Baldwin. 2022. Multispanqa: dataset for multi- question answering. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 1250-1260.
Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question an- swering. In International Conference on Learning Representations.
Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu. 2020. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 conference on empirical methods in naturul language processing (EMNLP). pages 3285-3292.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL. pages 874- 880.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. International Confer- ence for Learning Representation (ICLR).
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao. Jiawei Han, and Weizhu Chen. 2021. Reader-guided passage reranking for open- domain question answering. In Findings of ACL- IJCNLP
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zeullemoye: 2017. Triviaqa: large scale distantly supervised challenge dataset for reading comprehen- sion. In ACL, pages 1601-1611.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797.
Mingxuan Ju. Wenhao Yu. Tong Zhao, Chuxu Zhang, and Yanfang Ye. 2022. Grape: Knowledge graph enhanced passage reader for open-domain question answering. In Findings of Empirical Methods in Natural Language Processing.

---

Sewon Min, Luke Zettlemoyer, Hannaneh Hajishirzi. et al. 2022. Crepe: Open- question answering with false presuppositions. arXiv e-prints, pages arXiv-2211.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.
Ella Neeman, Roce Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint arXiv:22 2.006655..
Wei Yang. Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open- open-domain question answering with bertserini. In NAACI, 2019 (demo).
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Puttern Recognition, pages 12700- 12710.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNLP-JJNNLL pages 5043-5053.
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. International Conference for Learning Representation (ICIR)..
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanee text generation. ACM Computing Surveys (CSUR).
Yingqi Qu, Yuehen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835 (335-5877.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang. Jiancheng Lv. Fuli Feng, and Tat-Seng Chua. 2021a. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics. pages 3277-3287.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21 21(140):1-67.
Fengbin Zhu, Wenqiang Lei, Chao Wang. Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021b. Retrieving and reading: A comprehensive survey open-domane question answering. arXiv preprint arXiv:210..00774
Samuel Joseph Amouyal Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. Qampari:: An open-domain question an- swering benchmark for questions with many an- swers from multiple paragraphs. arXiv preprint arXiv:2205.1266
Qingfu Zhu, Weinan Zhang, Ting Liu, and William Yang Wang. 2020. Counterfactual off-policy training for neural dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3438-3448.
Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2022. Questions are all you need to train a dense passage retriever. arXiv preprint arXiv:2206.106588
A Appendix
Niket Tandon. Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. Wiqa: A dataset for "what if... reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing DOMNLP-IJCNLAS pages 6086-6085.
A.1 Annotation Task Form on AMT
We have provided the annotation form used on Amazon Mechanical Turk (AMT) in Figure 3 and Figure 4, which demonstrate the two methods of an- restricted annotation, where Wikipedia page is provided, and free-form annotation, where no Wikipedia page is provided. This annotation process helps prevent workers from asking ques- tions without sufficient context or reluctance. Fur- ther details on the data collection is in $3.1.
notation:
Harsh Trivedi, Niramjan Balasubramaniam, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.

---

Restricted Annotation of IfQA Instruction
Goal: Our goal to collect a question answering dataset with counterfactual presuppositions to train better artificial intelligence system with counterfactual reasoning ability. qualified question should be based counterfactuals assumptions on Wikipedia facts and have definite answer that can be evaluated.
Example 1: (Wikipedia URL) http:://wwwwkkkppdaaaaa///////// Summer Olympics
endik argoo000000 Summer Obmoai
2020 Summer Olympics
Tokyo was selected the host city during the 125th 100 Seesion Buenos Aires, Argentina, on September 2013 The Games were originaly scheduled lake place from 24 July to Auguet 2020, but due the global global COVID-11 penderni on March 2020, the event was postponed to 2021 first such instance the history the Olympic Games (previous games boon cancelled but not reschoduled).
WIKIPDIAA panipan
(Question) Covid-13
effectively contained, in which year
the Tokyo Olympics held?
(Answer) 2020
(Copied sentence) The Games were originally scheduled to take place from July to 9 August 2020, but due to global COVID-19 pandemic, the event was postponed 2021.
Example 2: (Wikipedia URL) https://en, http://wwwww...aaaaao///////////
(2018 film)
envencipedi orgini
Overboard (2018 film)
Overboard' released
Gond-Goddyy-Mayss Lionggtee
film credited under Pantellon civision).
originally scheduled for April 20, 2018 thllghh January 2018, the date reecheculed for Apeil 13, 2010. in March 2010, the fimmls relesse was reacheduded for May 4, 2018,122 avoid competing against the new April relesse Avengera: Infenly Wa/!11)
WIKIEDLLA Be tie ceneeeneei
(Question) If Overboard came out on April 27, 2018, which movie would it mainly compete with at the box office? (Answer) Avengers: Infinity War (Copled sentence) was originally scheduled for April 20, 2018, though In January 2018. In March 2018, the film's release was rescheduled for May 4, to avoid competing against the new April 27 release of Avengers: Infinity War.
Example 5: (Wikipedia URL) https::/een.wkkieddda..rr
K2
K2, 8,611 metres (28,251 It) above sea level, the second-higheet mountain on Earth, after Mount Everest (at 6,849 metres
(29 032 fi), lis the Karakoram
pertially the Gilgit-Baliisan region of takistan-ddmiisstree Kashmir and partially admiinstsdii temrtory the Kashmir region included the Taxkorgan Tajik Autonomous County Xinjiang. (419)/
WIRIPEDIA
(Question) Everest were 1000 meters lower, what would be the highest mountain in the world? (Answer) K2
(Copied sentence) K2, at 8,611 metres (28,251 ft) above sea level, is the second-highest mountain on Earth, after Mount Everest (at 8,849 metres (29,032 ft)).
Based the example above, please write down question and answer based the Wikipedia passage provided below. Make sure to copy the supporting sentence into the designated box.
en.nkipedts ww2ooo Lunted wwwee wemaddntttLrrciiin
2012 United States presidential election
The 2012 United States presidential election 57th quadrennial preeidential election, held Tuesday, November 2012 Incumbent Demooratic President Barack Obama and his running mate, incumbent Vice President Joe Bicen, were re elected to a second torm. They defeated the Republican tokot businessman and former Governor Mit Romney of Massachusetts and Representative Paul Ryan of Wisconsin.
WIKIPEDIA
1. Question:
Please type your question here. The question should have a definite answer, not open-ended!
2. Answer:
Please type your here. If the question has multiple answers, please separate then by
Supporting sentences:
Please copy the supporting sentence here. This should be sentences copied from above Wikipedia.
Figure 3: Restricted annotation of IfQA instruction used on Amazon Mechanical Turk (AMT). This figure only shows the process of annotating one question. In practice, workers are presented with 20 Wikipedia passages and required complete total of 10 tasks ensure the avoidance of nonsensical questions.

---

Free-form Annotation of IfQA Instruction
Goal: Our goal is to collect question answering dataset with counterfactual presuppositions to train better artificial intelligence system with counterfactual reasoning ability. qualified question should be based on counterfactuals assumptions Wikipedia facts have definite that be evaluated.
Example 1: (Wiklpedia URL) https://en. https://ee.wkkpeedaaarg///////// Summer Olympics
2020 Summer Olympics
was selected the host during the 125th IOC Session Buenos Aires, Argenting, September 2013 (i) The Games
originally echeduled take place from 24 July to Auguet 2020, but the globel penderii March 2020, the event postponed to 2021 the first such instance the history of the Olympic Games (previous games had been cancelled but not rescheduled).
WIKIFEDIA
(Question) Covid-19 19 was effectively contained, in which year would the Tokyo Olympics be held? (Answer) 2020
(Copied sentence) The Games were originally scheduled to take place from July August 2020, but due to the global C0 pandemic, the event was postponed to 2021.
Example 2: (Wikipedia
(2018 film)
shaekipodia.oo o wwwiii/oroooooo.o....o...oo 120.ooorr__200000000000 1km)
Overboard (2018 film)
Overboard was released by Metro-Goldw Metro-Goldyy--Mayer and Lioregate Films (though the film credited under Pantelion Films division), was originally
2018 thought Jen.ary 2018,
2016,
WIKIPEDIA
rélesse reachedueed for May 4, 2018,12 1 avoid competing againal the new April mélenoe of Avengera: Infisty WM]
(Question) Overboard came on April 27, 2018, which movie would mainly compete with at box office? (Answer) Avengers: Infinity War (Copled sentence) was originally scheduled April 20, 2018, though In January 2018. In March 2018, the film's release rescheduled May avoid competing against the new April 27 release of Avengers: Infinity War.
Example 5: (Wikipedia URL) https://en.wkkipellarrwwkwkk
2
(28,251 above level, second-higheet
032 Hi). les the Karakoram range, partially the Gilgi--Ballsian region Pakisan-admiiisbrre Kashmir and partlally
WIKIPEDIA The Poee Eeceeogggell
China aiinnnnddiiiteeeedd territoy of the Kashmir region included in the Taxkorgan Tajjk Autonamous County Xinjiang Xinjiang.M41910 [4j80eaa]
(Question) Everest (Answer) K2
1000 meters lower, what would be the highest mountain the world?
(Copied sentence) K2, at 8,611 metres (28,251 above sea level, the second-highest mountain on Earth, after Mount Everest (at 8,849 metres (29,032 ft)).
Based the example above, please write your question and answer, include relevant evidence from Wikipedia. Make provide the URL of the Wikipedia page and copy the supporting sentence into the designated box.
1. Qonstion:
Please type your question here. The question should have definite answer, not open-ended!
2.
Answer:
Please type your answer here. If question multiple answers, please separate by
3. Wikipedia URL:
Please type the URL here. The auto-URL check has been conducted to ensure its validity.
4.
Supporting sentences:
Please copy the supporting sentence here. This should be sentences copied from above Wikipedia.
Figure Free-form annotation of IfQA instruction used on Amazon Mechanical Turk (AMT), Free-form annotation allows workers write their own questions if they want to do or if they find difficult to ask questions based on given Wikipedia passage. We require that this self-proposed question must also based on Wikipedia, and the worker is required to provide the URL of Wikipedia page and copy the corresponding sentence/paragraph.


Reviews:
[{"review_id": "nUe6pjzYVS", "reviewer": "Reviewer_8qae", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. We concur that providing a direction of thought for future techniques is valuable.  Here are some promising directions, which we would be happy to include in the paper:\n\n\n> 1.The author(s) could have provided a direction of thought or discussion as to their perspective, how current techniques (or newer ones) might be modeled to tackle such counterfactual reasoning scenarios.\n\n\n**-- Overcoming Retrieval challenge:** (1) Questions in the IfQA dataset are typically longer, owing to the inclusion of counterfactual presuppositions. This causes challenges for current IR models, such as DPR, which tend to utilize a single embedding vector to represent comprehensive information. One promising way to get around this limitation is to consider multi-vector representations that capture different nuances in the question, and explicitly perform IR for both the original fact as well as the presupposition.\n\n(2) The inherent discrepancies between counterfactual presuppositions and factual evidence frequently cause the retriever to falter in obtaining relevant documents when relying solely on semantic matching. Incorporating clear instructions for retrieval might help bridge the gap between the counterfactual and factual worlds.\n\n**-- Overcoming Reasoning Challenge:** For counterfactual questions, models must perform complex reasoning on potential situations that diverge from the factual data built into their parameters. This may be achieved via a multi-stage reasoning mechanism, wherein the initial stages deal with understanding and extracting the counterfactual premise, and the subsequent stages handle the implications of that premise on the original reasoning.\n\n**-- Adaptive Chain-of-Thought Reasoning:** We showed the chain-of-thought reasoning approach offers a potential avenue for counterfactual reasoning. However, it still struggles with many IfQA questions.One way to improve it could be to employ adaptive reasoning chains that dynamically adjust based on the nature of the counterfactual premise. This could involve iterative questioning or even self-dialogue approaches where the model questions its assumptions and iteratively refines its reasoning process.\n\nWe will enhance the paper by incorporating a more detailed discussion on these fronts, elaborating on both the current limitations of models on IfQA and potential pathways to improve performance. Once again, thank you for the suggestion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a very important aspect of reasoning that human tends to do in certain scenarios. So, this type of \"thinking\" is a crucial step for AGI development. I believe this dataset would spark interest in this direction and lead to fruitful research contribution.", "reasons_to_reject": "1. The author(s) could have provided a direction of thought or discussion as to their perspective, how current techniques (or newer ones) might be modeled to tackle such counterfactual reasoning scenarios.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper provides a new dataset in the domain of QA, but based on counterfactual reasoning. The author(s) show that current existing methods are not geared to tackle such reasoning, and thus perform quite low.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jaD1an0j9B", "reviewer": "Reviewer_hauT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We are very encouraged that you found our dataset novel and valuable to the community. We address your concerns as follows.\n\n> 1.There should be some analysis on similarity of questions because the 5 randomly sampled examples may still have an impact on annotation quality. For example, replace named entities with their respective category and use some similarity metric (e.g., word embs, WMD) to cluster to see how diverse annotations actually are. Generally, they should be evenly spaced for diverse questions.\n\n-- Following your recommendation, we organized the training data into two sets of clusters: one set with 5 clusters and another with 10. We observed that the distribution of instances across these clusters was relatively even. Below are the details:\n\n| #clusters | mean | std |  #1 |  #2 |  #3 |  #4 |  #5 |  #6 |  #7 |  #8 |  #9 | #10 |\n|:---------:|:----:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|     5     |  480 |  88 | 541 | 468 | 405 | 542 | 447 |     |     |     |     |     |\n|     10    |  240 |  47 | 257 | 209 | 206 | 240 | 250 | 210 | 251 | 254 | 255 | 271 |\n\n\n> 2.There is a lot of overlap in counterfactual questions and in other tasks such as common-sense QA and mathematics word question answering. As mentioned in the paper, the keyword filter on Wikipedia seems to have made the dataset heavily skewed towards event-related questions. It would be interesting to see these aforementioned domains being included in the dataset. For example, many math questions present counterfactuals which can be expanded to things like corporate decision making. If this is not the goal, it may be more apt to make the dataset more specific - \"Event-based counterfactual open-domain QA\".\n\n\n-- While a significant portion of the IfQA dataset comprises event-based questions, owing to their relevance to real-world situations and potential alternative outcomes, the scope of IfQA extends beyond events. As highlighted by Figure 1 and Table 1, our dataset also incorporates questions that necessitate both commonsense reasoning and mathematical proficiency.\n\nWe note that an important distinguishing aspect of IfQA is that it’s based on verifiable (counterfactual) facts in an open-domain setting. E.g., for the question “who would have become the president of the USA if Obama lost in 2008?”, there is a unique factual answer (John McCain), deriving which requires external IR and reasoning. This is generally not the case for commonsense QA or math word problems, even if they are phrase as “if” questions (e.g., “if John had 3 apples and gave 1 to Mary, …”). Answering some counterfactual questions might call upon commonsense or math skills, but the challenge generally goes further – models must identify the right information to retrieve from an external source and reason in a way that might even go against facts built into their parameters. \n\nThus, while there are some intersections between counterfactual QA and commonsense or mathematical QA, the main challenge of IfQA is orthogonal to them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Counterfactual reasoning is an important strength of human intelligence and to achieve more robust question answering models, it is important to have a publically available dataset/set of baseline metrics. This paper presents clear and concise insights into how the dataset was built as well as an in-depth analysis into the dataset and how a broad set of state-of-the-art models perform on the given task. The discussion is thorough and insightful and most design choices are motived/justified well. ", "reasons_to_reject": "Aside from common concerns in most crowdsourced dataset papers (e.g., bias in views, human error due to limited validation resources, etc.), the weaknesses of this paper (or other suggestions) can be summarized as:\n1. There should be some analysis on similarity of questions (maybe in Appendix) because the 5 randomly sampled examples may still have an impact on annotation quality. For example, replace named entities with their respective category and use some similarity metric (e.g., word embs, WMD) to cluster to see how diverse annotations actually are. Generally, they should be evenly spaced for diverse questions.\n2. There is a lot of overlap in counterfactual questions and in other tasks such as common-sense QA and mathematics word question answering. As mentioned in the paper, the keyword filter on Wikipedia seems to have made the dataset heavily skewed towards event-related questions. It would be interesting to see these aforementioned domains being included in the dataset. For example, many math questions present counterfactuals which can be expanded to things like corporate decision making. If this is not the goal, it may be more apt to make the dataset more specific - \"Event-based counterfactual open-domain QA\".\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a dataset of counterfactual questions and answers (challenging for existing retrieval/open-book and closed-book models). Counterfactuals are particularly challenging for LLMs because they rely on a combination of world-sense/reasoning and retrieval. The authors also establish an initial set of performance baselines with state of-the-art close-book and open-book models (e.g., chain-of-thought GPT-3, RAG, FiD, etc.). This paper demonstrates that the set of counterfactuals presented by IfQA remains a significant challenge for existing models, finding that there is a large gap between human performance and the best approach (passage retrieval + large model reasoner). ", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos on line 42-43 and line 501."}, "extra_text": {}}, {"review_id": "eQl2c1oyb3", "reviewer": "Reviewer_sGbp", "summary": "", "questions": "How does this work compare with other counterfactual reasoning approaches in the literature?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are very encouraged that you found our dataset novel and valuable to the community. We address your concerns as follows.\n\n> 1.The paper seems to rely heavily on Wikipedia as a source, which might limit its applicability in more diverse contexts.\n\n-- While we agree that using other sources in addition to Wikipedia would make the benchmark even stronger, we believe Wikipedia already provides diverse enough topics and extensive coverage to build a meaningful dataset to develop and evaluate novel models. This is akin to numerous existing open-domain QA benchmarks which also use Wikipedia as the underlying knowledge source. Secondly, the foundational challenge introduced by our IfQA dataset can be extrapolated to other domains as well, namely evaluating a model's ability to identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into its parameters.\n\n> 2.The evaluation metrics and results are not clearly mentioned in the extracted content, making it hard to gauge the effectiveness of the proposed approach.\n\n-- We note that we didn’t fully understand what the reviewer meant by “extracted content”. We will try to answer the question as best as we can here, but are happy to expand further.\n\nThe evaluation metrics are listed in section 4.3 and experiential results in section 4.4. In particular, the retrieval performance is evaluated as recall@k, the percentage of top-K retrieved passages that contain the ground truth passage. The end-QA Performance is evaluated by standard exact match and F1 score metrics, which measure the token overlap between the prediction and the ground truth answer.\n\n> 3.How does this work compare with other counterfactual reasoning approaches in the literature?\n\n-- Our work primarily introduces a novel counterfactual QA dataset, which is the first dataset of open-domain QA focused on counterfactual reasoning. We compare our paper with prior work in the area of counterfactual reasoning in section 2.2. We will clarify that prior work is either about models (whereas ours is about a new benchmark), or about counterfactual datasets in the reading comprehension setting (while ours is in the open-domain setting).\nRegarding counterfactual reasoning methods, traditional approaches in existing literature are not directly applicable due to the unique challenges presented by IfQA. To navigate this open-domain characteristic effectively, we mainly experimented with retrieve-then-read pipelines, since they are SoTA for existing benchmarks. However, they are not geared to tackle such challenging reasoning tasks. IfQA requires models to not only pinpoint the relevant information to retrieve but also to cogitate over counterfactual scenarios, some of which might challenge or even contradict the inherent facts within their training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a novel and challenging area in NLP, which is counterfactual reasoning in question answering.\n2. The approach of creating a dataset based on counterfactual presuppositions can be a valuable resource for the NLP community.\n3. The inclusion of a detailed annotation process ensures the quality and reliability of the dataset.", "reasons_to_reject": "1. The paper seems to rely heavily on Wikipedia as a source, which might limit its applicability in more diverse contexts.\n2. The evaluation metrics and results are not clearly mentioned in the extracted content, making it hard to gauge the effectiveness of the proposed approach.", "questions_to_authors": "How does this work compare with other counterfactual reasoning approaches in the literature?", "paper_topic_and_main_contributions": "The paper seems to delve into the realm of question answering with a specific focus on counterfactual reasoning. The goal, as outlined, is to create a dataset for question answering that is based on counterfactual presuppositions. This is aimed at training AI systems to have a better counterfactual reasoning ability. The paper provides examples of questions that are based on counterfactual assumptions on Wikipedia facts. The paper also touches upon the annotation process, using platforms like Amazon Mechanical Turk, to ensure the quality of the questions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new dataset for open-domain question answering where the questions contain counterfactual/hypothetical situations. The model must retrieve passages based on these counterfactual situations to correctly answer the question. The reviewers found that the proposed dataset targets an important ability for ODQA systems and demonstrates that existing state-of-the-art systems perform poorly on the task. They also found that the paper presents sufficient detail and provides interesting insights about the dataset and models.
Decision: Accept-Main