CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
Authors: Jian Wu
Linyi Yang
Zhen Wang
Manabu Okumura
Yue Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM evaluation
Multi-hop QA evaluation
Primary area: datasets and benchmarks
TL;DR: A novel evaluation method that comprehensively and objectively reveal LLMs' real multi-step reasoning performance without data contamination.

Abstract: While Large Language Models (LLMs) excel in question-answering (QA) tasks, their real reasoning abilities on multiple evidence retrieval and integration on Multi-hop QA tasks remain less explored. Firstly, LLMs sometimes generate answers that rely on internal memory rather than retrieving evidence and reasoning in the given context, which brings concerns about the evaluation quality of real reasoning abilities. Although previous counterfactual QA benchmarks can separate the internal memory of LLMs, they focus solely on final QA performance, which is insufficient for reporting LLMs' real reasoning abilities. Because LLMs are expected to engage in intricate reasoning processes that involve evidence retrieval and answering a series of sub-questions from given passages. Moreover, current factual Multi-hop QA (MHQA) benchmarks are annotated on open-source corpora such as Wikipedia, although useful for multi-step reasoning evaluation, they show limitations due to the potential data contamination in LLMs' pre-training stage. To address these issues, we introduce the Step-wise and Counterfactual benchmark (CofCA), a novel evaluation benchmark consisting of factual data and counterfactual data that reveals LLMs' real reasoning abilities on multi-step reasoning and reasoning chain evaluation. Our experimental results reveal a significant performance gap of several LLMs between Wikipedia-based factual data and counterfactual data, deeming data contamination issues in existing benchmarks. Moreover, we observe that LLMs usually bypass the correct reasoning chain, showing an inflated multi-step reasoning performance. We believe that our CofCA benchmark will enhance and facilitate the evaluations of trustworthy LLMs.

Published as a conference paper at ICLR 2025
COFCA: A STEP-WISE COUNTERFACTUAL MULTI-HOP QA BENCHMARK
Jian Wul Linyi Yang?* Zhen Wang'
Manabu Okumura'
Yue Zhang 21
Institute of Science Tokyo Uniiversity College London School of Engineering, Westlake Univeristy
ABSTRACT
While Large Language Models (LLMs) excel question-answeriigg (QA) tasks, their real reasoning abilities multiple evidence retrieval and integration on Multi-hop QA tasks remain less explored. Firstly, LLMs sometimes generate answers that rely internal memory rather than retrieving evidence and reasoning in the given context, which brings concerns about the evaluation quality real reasoning abilities. Although previous counterfactual QA benchmarks separate the internal memory of LLMs, they focus solely on final QA performance, which is insufficient for reporting LLMs' real reasoning abilities. Because LLMs are expected to engage in intricate reasoning processes that involve evidence retrieval and answering a series of sub-questions from given passages. Moreover, current factual Multi-hop QA (MHQA) benchmarks are annotated on open-source corpora such as Wikipedia, although useful for multi-step reasoning evaluation, they show limitations due to the potential data contamination in LLMs' pre-training stage. To address these issues, we introduce a Step-wise Counterfactual benchmark (CofCA), novel evaluation benchmark consisting of factual data and counterfactual data that reveals LLMs' real reasoning abilities on multi-step reasoning and reasoning chain evaluation. Our experimental results reveal significant performance gap of several LLMs between Wikipedia-basee based factual data and counterfactual data, deeming data contamination issues in existing benchmarks. Moreover, we observe that LLMs usually bypass the correct reasoning chain, showing an inflated multi-step reasoning performance. We believe that our CofCA benchmark will enhance and facilitate the evaluations of trustworthy LLMs.
INTRODUCTION
Retrieval- -oumentll-uugmeneed generation (RAG) (Chen et al. 2023 3 Ma et al. 2023 Gao et al. 2023b). has garnered increasing research attention TTngg& & Yang 2024 Poliakov & Shvai 2024). As shown in Figure p given a query, a RAG system first retrieves pieces of evidence from a set of relevant passages, and then generates a response by taking the passages additional context LLMs. While most existing work concentrates on the setting where the answer can be derived from a single retrieved passage, there are situations where LLMs are required to integrate information from multiple resources evidence to infer the answer. As shown in Figure Db. given the query, the single retrieved passages cannot offer enough information to generate correct answers. Instead, the evidence "Earth is the third planet" and "Mars is after Earth" are required to be integrated for deciding "The fourth planet of the sun".
Recently, line of work (Wang et al. 2023b; Staniszewski et al.) 2023 Trivedi et al. 2022 has investigated the evidence integration capabilities of LLMs combined with RAG, using existing Multi-hop hop QA (MHQA) datasets such as HotpoiQA (Yang al.) 2018) and 2WikiMultihopAA (Ho et al. 2020). However, such work focuses only on the correctness of the final answer, without evaluating the reasoning process itself. It is thus difficult to understand whether shorteut reasoning exists (Tang et al.] 2021 Ho et al. 2022 Yang et al.) 2022) and whether such reasoning is robust
"Equal contribution. Jian Wu did this work during his internship at Westlake University Corresponding author.

---

Published as a conference paper at ICLR 2025
Multi Passage RAG
Factubal MUJQA Evakodden
Cructerfactuaa Multi step Remsoning Evallzation
Waahypeets forth deset lan!
Makin poot benoeac pllp pot Ieam unto born 29977
couch parvic Chioae INNDUA
IES SoL com in 15675
and the conattro octible
Mare
sidnel tractinn bomily Lisit and cariinal jair oecocii years
span reigle bo all in mooe
Brown
meeees and ixpert onstis anch
Anower Mars
doaen love Age 1947.
Single Panaaee RAG
100/1 stso knisen de Leu senui, Asgeminn mudor andooadd ltans
Audils
fen podfosiinn coatctull plgen
peitle (lay?
puant lo
perves as tor bat the losusee jears
hid cocactro autetle fi
Lnoe treae, son ti80,, sas jeen of
en quaed liro senvrs as sooun
Chinose mecornal tours.
Find Ausact
dotu
LEM
LLM
Rearoning Chain
Evahuatoon
Previous VHQA Evaluntion
CoFCA
Figure I: Differences between (a) ab). The words in red are the pieces of evidence for the given questions. The differences between factual MHQA (c) and counterfactual MHQA (d). c: we input a factual MHQA with and without context into ChatGPT. ChatGPT could output the correct answer based on its internal memory regardless of the context. d: When inputting a counterfactual QA, where the passage is rewritten from the existing factual passage in с (words in the red), ChatGPT cannot rely on its memory and must reason on the given context, deeming that counterfactual QA can decouple LLMs' internal memory and reasoning abilities.
across dusto--distribution (OOD) tasks. Existing research (Tang et al., 2021 shows that strong QA models such as DFGN (Xiao et al. 2019), DecompRC (Min et al. 2019). and CogQA (Ding et al. 2019) can correctly answer multi-hop questions but bypass the correct reasoning chain. However, no existing research has evaluated the performance of LLMs to the same problem.
We aim to address the above issues by making systematic evaluation of the LLMs' reasoning process. To this end, an existing sub-question style benchmark (Tang et al. 2021) derived from HotpotQA Yang et al. (2018) can be useful, in which L.MMs are required to answer a series of sub-questions to reach the target rather than giving final answer solely. However, existing datasets cannot be directly used for reporting LLM real performance, due to the risk of data leakage in the pretraining stage (L (L'opez et al. 2024 Zhou et al. 2023a; Fu et al. 2023). As shown in Figure Db. for instance, given the question Who was the captain of Argentine team that was born in 1987?", ChatGPT might give the correct answer even without context, since relevant knowledge is learned by the model during the pre-training stage. Therefore, a sub-question benchmark that avoids the negative influence of data leakage is needed. To this purpose, consider the counterfactual evaluation method (Li et al.) 2023 Neeman et af.) 2023 Zhou et al.] 20236) Wu et al. 2024 Yu et al.] 2023). We make modifications existing knowledge
Wikipedia, deriving new passages and ensuring that knowledge in these passages is non-existent in the real world, thus those answers have little chance of being inherent in LLMs. For example, in Figure Dd. given the question "Which position did the coach play that serves Chinese national basketball team and born in 1997?", LLMs cannot rely on their memories and must answer the first sub-question "Which coach serves as Chinese national basketball team and born in 1997?", before finding the final answer by asking second sub-question "Which position did Alice Brown play?". The passage Figure Tt. is modified from the passage in Figure [T.. Specifically, we generate counterfactual data to Wikipedia knowledge by replacing key information (named entities, noun phrases, and synonyms) and then paraphrasing (back translation).
We call our dataset CofCA, step-wise and Counterfactual MHQA benchmark, which consists of counterfactual passages with corresponding multi-hop QA pairs and sub-QA pairs, and an equal amount of factual MHQA data from HotpotQA, 2WikiMultihopQA and MuSiQue as the control group. Using CofCA, we evaluate and report LLMs' real reasoning ability. Experimental results show two types of inflated performance in LLMss 1) an obvious performance gap between Wikipedia-hasee
2

---

Published as a conference paper at ICLR 2025
factual MHQA benchmarks and counterfactual MHQA data; 2) inflated performance due to low proportion of correct reasoning chains as well as a high proportion of incorrect reasoning chains, with GPT-4 achieving only 36.3% correct reasoning chains across the entire dataset. Additionally, we observe that incorporating sub-questions into the prompt as part of the reasoning chain is a more efficient approach for improving LLMS' performance.
To the best of our knowledge, we are the first to introduce counterfactual data into the evaluation of the multi-step reasoning ability of LLMs, finding that there a significant performance gap between LLMs' performance on factual data and counterfactual data. Furthermore, the reasoning abilities of LLMs are exaggerated because of the poor reasoning chain performance. The Whole CofCA data are 4open.
available at
2 RELATED WORK
Retrieval Augmented Generation (RAG) improves LLM's response (Borgeaud et al. 2021 and also mitigates the occurrence of hallucinations, thereby enhancing the models' credibility (Gao et al. 2023a). As demonstrated by Khattab et al. (202T), designs a RAG system for MHQA and claim verification tasks. These tasks require the extraction of evidence from two more documents to produce a correct answer. Tang & Yang (024) propose a Multi-hop RAG benchmark, which consists of a large collection of multi-hop queries, ground-truth answers, and the corresponding supporting evidence. Multi-hop RAG requires LLM to reason and answer multi-hop queries given the evidence. However, LLMs' memorized knowledge sometimes conflicts with the given context, emphasizing the importance of correcting LLMs' generations with new facts. [i et al. (2023): Neeman et al. (2023); Zhou et al. (2023b); Wu et al. (2024); Yu et al. (2023) propose counterfactual QA benchmarks to separate LLMs' parametrical knowledge (internal) and contextual knowledge (outer) that fix LLMs to reasoning on the given context strictly by editing the contextual information or prompts. Previous work motivates to explore LLMs' real reasoning ability by reasoning in counterfactual contexts. However, counterfactual QA datasets still only assess final QA performance and lack reasoning process evaluation.
Multi-hop QA Datasets Multi-hop hopp QA requires more than one reasoning step in multiple paragraphs answer a question (Dua et al. 2019 Yang al.. 2018: Ho al., 2020 Trivedi al. 2021 Notably, Tang et al. (202T) introduce a human-validated sub-question dataset derived from the HotpotQA dataset ( YYang et al. 2018). undertaking detailed investigation of the models' capabilities to reason through sub-questions. Their findings revealed that notable models like DFGN (Xiao et al. 2019), DecompRC (Min et al. 2019). and CogQA (Ding et al., 2019) exhibit deficiencies in resolving sub- questions, even though they may successfully address the overarching multi-hop question. Moreover, Wikipedia-based based MHQA datasets face the challenge of data contamination that hard to objectively and truthfully evaluate the reasoning ability of LLMs. Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a major issue in measuring LLMs' real performance on other tasks. For example, HotpotQA (Yang al. 2018). 2iikiMultihopQA (Ho et al., 2020), and MuSiQue (Trivedi et al.) (02T) can be applied to evaluate the multi-step reasoning performance of LLMs. Typically, evaluating L.LMs in MHQA datasets involves using RAG to retrieve and reason over context with a single step of retrieval. However, single-step retrieval can result in insufficient context retrieval for complex questions,
provides a limited scope of information (Gao et al. 2023b). Prasad et al. (2023) proposes a framework that defines good reasoning chains in Correctness and Informativeness illustrate whether the previous reasoning step could help the current reasoning step and the final answer.
Benchmarking Data Leakage A handful of recent studies have provided several strategies. methods, and benchmarks for detecting contamination without the need to access pre-training data (Shi et al.) 2023, Roberts et al., 2023 Golchin Surdeanu 2023 Zhu et al. 2023a). |Ravaut et al. (2024) surveys recent work on detecting data contamination and releases a python library named LLMSanitize that implements major contamination detection methods. However, these data contamination detection benchmarks are required to dynamically update because of the development of LLMs and the expansion of pertaining data. Dynamical maintenance is time-consuming and effortless, while our proposed benchmark CofCA, based on the knowledge edition, is fixed and maintains the cleanness of the test data. To statistically and quantitatively detect the LLMs' data contamination extent, Xu et al. (2024b) proposes detection pipeline by computing perplexity and N-gram accuracy to evaluate
3

---

Published as a conference paper at ICLR 2025
Data Annotation with Human-in-th--Loop
B. Counterfactual MB3Q Evaluation
Nulti- hhp Question
for
Archieccue Award? Alice
Step 3: High- quality CofCA data
Step 4:LLM Multi-step Reasoning
Sub- Ques Lions
Awarff
Who tes gaod for Whltee Husse Aareet Alice
Question Review
Anuotated Passage
Alter Americas interir designer and intentor devigner for fadvas brunth
as
Arditeeiare
Step Reassoning Chain Evaluation
paár Annocator
Alead nationaiooal Des
patticiuated
mazy faskian saaws loth denestially Interadooraly responsi bie
design
Passage Iluman Review
Context
Mahneen Klon as Palceteni brshton and contome designer, abn en mond sitner facrice de deeigner for fedoin labels like "The Embecidery House Maleen' and "Gulabe" She has there many catircal and anousiianal fislion events und shows. Size uode emboodyy y for the dilm Snow White and the [Rattonon and television series
d:Performmnce Comparion
Annotator
Figure 2: The framework of our LLM automatic data annotation pipeline. From left right, A: we first ask LLM to act as passage annotator to replace the keywords and paraphrasing. Then we manually ensure that the correctness grammar and the key information have been changed. We send the reviewed high-quality data to GPT-4 to generate QA pairs and manually check the quality. B: After receiving the reviewed high-quality counterfactual QA data, we evaluate LLMs on generated data to test. their inherent ability on MHQA.
potential data leakages. Zhang et al. (024) designs new Grade School Math 1000 (GSMIK) to mirror the GSM8k benchmark (Cobbe et al., [021) and evaluate LLMs' mathematical reasoning ability. Clean-Eval (Zhu et al., 2023b), benchmark to assess the inflated performance of LLMs. The experimental results show a drop in performance for GPT-3.5 and Llama-2 in the Clean-Eval data, deeming the data contamination problem. Xu al. (2024a) proposes detection pipeline with the help of perplexity and N-gram accuracy gap between the previous MathQA dataset and synthesized data, pinpointing the potential data leakage problem. The measurement illustrates that underscores the urgency for an evaluation paradigm shift in how we approach the development and evaluation LLMs. To properly and objectively investigate multi-step reasoning abilities, is crucial to disentangle LLMs' inherent memory from their reasoning capabilities, Hua et al. (2024) introduces a novel benchmark for counterfactual QA tasks by editing six common reasoning schemes in the real world, while still lacking muti-step reasoning evaluation for LLMs.. Our work is different from such works in two main aspects: I) a comprehensive reasoning chain evaluation; and 2) Counterfactual Question Answering that reduces the risk of data contamination.
MULTI-HOP REASONING EVALUATION
In this section, we design a human-in-the-loop annotation framework and our novel evaluation method. As shown in Figure 2 the framework consists of three components: 1) we first ask LLM such as GPT-4 to act as a passage annotator that rewrites passages from Wikipedia with human evaluation and feedback; 2) LLM's automatic counterfactual QA pairs annotation with human evaluation and feedback; 3) After obtaining the synthesized counterfactual MHQA data, counterfactual multi-hop reasoning to evaluate several strong LLMs to report the real multi-step reasoning performance.
4

---

Published as a conference paper at ICLR 2025
3.1 DATA CONSTRUCTION
Data Collection and Passage Rewriting We randomly select 300 Wikipedia passages from Huggingface as the raw context. Inspired by recent studies on LLM's s ability to aid human annotation (Bartolo et al., [02T Törnberg 2023), we design a pipeline for automatically annotating Wikipedia passages into counterfactual passages. Given a raw Wikipedia passage, L.LMs are required to act as passage annotator to do the named entity, noun phrase, and synonym replacement. Since LLMs are pre-trained on the open-source corpus collected from the Internet, we manually search the key information of the annotated passage on the Internet, e.g. the characters, times, events, causes, processes, and results.
After the replacement stage, translate the replaced text into Chinese and finally back translation into English, e.g., the words in red of the annotated passage in figure are the replaced named entities, noun phrases, and synonyms, and the new counterfactual passage is rewritten from the original Wikipedia passage. Upon acquiring the new and counterfactual passages, human experts conduct manual evaluation of data quality by focusing on two key aspects: 1) assessing the grammatical integrity of the annotated passages to ensure there are no grammatical errors; and 2) verifying that the annotated passages are new and counterfactual to Large Language Models (LLMs).
QA-pair Annotation and Checking We use LLMs to generate new multi-hop questions to fit the rewritten passages. To make sure the generated QA pairs are correct and related to the given passages, we check the QA pairs from two perspectives: 1) Grammar issue; 2) Answerabilities of the generated question, whether the question is related to the passage and make sure the answer can be reasoned from the passages based in the generated questions. To evaluate LLMs' performance on different complexity of multi-hop questions, for each passage, annotate 3 complex questions: one for 2-hop, one for 3-hop, and one for 4-hop questions. L.LSs are required to generate multi-hop questions along with the corresponding sub-questions for reasoning chain evaluation. For example, at the middle part of Figure |illluttatess the re-annotated context, newly generated multi-hop questions, sub-questions, and intermediate answers. The annotated answers are all with short answer span that follows the settings of HotpotQA, 2WkimulihopQA, and MuSiQue (Yang et al., 2018 Ho et al. [2020] Trivedi al. 202T). To ensure the evidence of answers are integrated from multiple paragraphs. we follow the settings of HotpotQA, 2WkimulthopOO.
and MuSiQue, the supporting facts are from at least two paragraphs. Here we use EM and Fl scores to measure the L.LMs' output. After obtaining the rewritten passages with corresponding QA pairs and sub-qa pairs, we also check the logic of the whole passage and QA pairs. We expect that the passage is coherent and the answers can be reasoned from the passage based on the given questions. The prompts of the data annotation and more annotated examples are shown in Appendix B
Dataset Analysis and Statistics Table shows the statistics of our dataset. We re-annotated 300 unique Wikipedia passages, with three multi-hop questions (one 2-hop, one 3-hop, and one 4-hop for each passage), a total of 3600 unique QA pairs including 900 multi-hop questions, and 2700 sub-questions with corresponding answers. Besides, we also randomly selected 900 factual MHQA data as the control group (300 from HotpotQA, 300 from 2WikiMultihopQA. and 300 from MuSiQue) and 1,800 data of CofCA in total. Following the settings of previous LLM evaluation benchmarks (Wang et al. 2023a| 2024). we treat the total of 1800 data as the test set. Following benchmarks such as HotpotQA (Yang et af., 2018), 2WikiMulthopPAA (Ho et al., 2020), we propose a taxonomy on fine-grained question types and examples commonly used in multi-hop QA illustrated in Table [1] In HotpotQA, 2WikimultihopQA, and MuSiQue datasets, the multi-hop questions consist of two types of questions, or the combination of the two types of questions, Bridge and Comparison: Bridge question is required to find the bridge entity that connects the sub-questions, while Comparison question is type of question that compares two or more entities for the parallel sub-questions. We focus on these two types of questions for annotating multi-hop questions.
Inter Human Agreement Following a thorough manual inspection of the annotated data, we randomly select 300 samples, dividing them equally among 2-hop, 3-hop, and 4-hop instances, with each sample being evaluated by two expert reviewers. These reviewers are tasked with identifying and eliminating any data that significantly deviates from the established guidelines. Initially, the
https: httts////teeiaaaeeeaaaee/rllllll
5

---

Published as a conference paper at ICLR 2025
Table Statistics our CofCA dataset.
hop dataset
hop dataset
hop datasct
Whole data property
Valuc
hop QA pairs
300
hop QA pairs
300
hop QA pairs
300
Unique Passages
1800
Sub-QA pairs
600
Sub-QA peirs
900
Sub-QA pairs
1200
Total QA pairs
4500
Correctness
94
Correctness
Correctness
92
Sentences per data (Avg)
42
Informativeness
85
Informativeness
Informativeness
82
Inter annotator Agreement
94%
Table 2: Differences between our CofCA with previous factual and counterfactual QA benchmarks.
Benchmarks
Data Type
Data Souree
Task
Reasoning Chain
HotpotQA
Factual
Wikipedia Wikipedia
Muli-bop/lina--AA
2WikiMulthhpQA
Factual
Multi-hop/inal-AA
MusiQue
Factual
Wikipedia
Multi-hop//inn QA
DiseniQA
Counterfactual & Factunl
Natural Questions
Single hop/tinal QA
IfQA
Counterfactual & Factual
Crowdsourcing
Open Aonain/inal-QAA
CofCA(Ourr)
Counterfactual & Factual
Rewriting Wikipedia
experts focus on assessing the grammatical correctness of the questions, sub-questions, sub-answers, and answers within the annotations. Subsequently, they verify the originality of the content in the annotated passages by checking for their presence on the Internet.
According to our statistics, the agreement rate between the annotators in the randomly selected CofCA samples 94% and the human agreement rates are 96%, 92%, and 95% in the 2-hop, 3-hop, and 4-hop datasets, respectively. This suggests that our synthesized instances reflect good data quality on annotation guideline following and achieve high human agreement among expert annotators. To quantitatively illustrate data quality, we also utilize GPT-4 to assign scores to each selected data from two perspectives: correctness and informativeness. Each data is assigned with I or 0, which means correct or incorrect, informative or not informative. Correctness indicates whether the answer can be reasoned from the given question and context, while informativeness means whether the QA pairs and context are related or not.
3.2 EVALUATION METRICS
Given a set of input documents, we employ three representative QA evaluation methods to assess the correctness of A..M-generated MHQA responses: sub-question answering evaluation, reasoning chain evaluation, and the joint performance of sub-qa and MHQA.
Sub-QA QA Evaluation This part is the basis of all evaluation results. Following reading comprehen- sion (Rajpurkar al. 2016), evaluation is conducted through lexical matching using two widely used metrics to assess the performance of models. We employ F1 and EM scores to evaluate the answers sub-questions, similar to the single-hop QA task.
Reasoning Chain Evaluation of Multi-hop QA Table illustratss the differences between evaluation method and previous evaluation methods counterfactual QA and factual MHQA datasets. To interpret the behavior of existing LLMs on each hop of the reasoning process required for multi- hop questions, and to determine their reasoning ability to answer simple questions, we followed the experiment setting proposed by (Tang al. 2021). For example, in the 2-hop dataset, each data contains a 2-hop question, 2 sub-questions. intermediate answers, and final answer. In order to understand whether LLMs can correct answers by following the right reasoning chain, we calculate the proportion of right and incorrect reasoning chains to compare LLMs' reasoning performance. Each question or sub-question has two results, correct incorrect, thus N-hop question with its N sub-questions has N different reasoning chains. Due the space limitation, we measure and collect correctness statistics for the 2-hop question dataset, qeabl, qeud2, and g. and show the percentage reasoning chains given by L.MSs.
Joint Performance Previous MHQA benchmarks traditionally evaluated on the EM or F1 score on the final answer (Rajpurkar et al.) 2016) Yang et al.] 2018 Ho et al.) 2020), which is partially figure
correct. The previous MHQA systems and L.Mss treated black box and

---

Published as a conference paper at ICLR 2025
how they find the final answer. To understand the impact of sub-qa MHQA, introduce a joint performance that combines the evaluation of Sub-QA performance and MHQA performance. The details of computing the joint scores are shown in Appendix C
Table 3: Performance of Proprictary LLMs and Open Source LLMs Wikipedia- based factual multi-hop QA datasets. The performance is measured by EM and FI scores with a zero-shot setting. PMf indicates the partial match of LLMs' outputs evaluated with GPT--4-turbo with the prompt.
Datasets
Wikipedia
HotpotQA FI
2Wiki FI
MuSiQue F1
Metrics
EM
PM i
FM
PM
FM
PM i
Proprictary LLMs
GPT-4
69.9
82.3
74.8 112
59.7-14
67.4
64.8-n9
57.3+19
65.4-28
63.9s14
GPT-3.5
6289
69.1st11
8.0.
56,3-09
67.6aas
59.4.n9
49.3ans
63.2a13
53.1
GEMINI-pro
58.2
68.4
63.5w0.
48.5-16
58.5
54.7a12
41.3+15
54.5a0
46.9a13
text
50.3sav
61.4
$4.9
42.3a14
53.9
46.7=a7
40.2am
51.0s15
44.6
Bing Chat
78.3+12
42.1
58.9
69.9
63.4-ux
49.6-11
64.
52.3-03
Ol-
72.2
82.7 20.7
76.9 all
68.7 20.9
79.8 2n
72.3
63.9 209
72.4 so.s
67.9G08
Open Source LLMs
2-7b
34.5a12
41.3g11
38.5
30.6-1.1
34.7a1.1
33.8an9
31.7aus
15.66112
34,2a09
Mistral-7b
30.6115
37,2
34.9
27.4.06
29.8.09
31.4:08
25.2.a7
28.9.0s
29.2.0d
Qwen 2-7b
36.2a15
43.5a13
39.3a04
31.7a10
35.8a0.
36.8.n
28.2a1.1
31.2a11
33.5a04
Table Results on CofCA, the results reveal that LLMs show an obvious performance gap between previous Wikipedia- based factual data and our CofCA.
Datasets
CofCA
2-hop
3-hop
4-hop F1
Metrics
RM
PM f
FM
PM f
PM
Proprietary LLMs
GPT-4
L3.5
62.8
57.6x1.1
44.5a13
56.4.17
49.5
42.3.06
53.5.09
48.8
GPT-3.5
40.6
56.7a0.5
43.7a09
37.7.05
50.9211
42.1a1.
32.5
44.620.5
36.2a1.1
GEMINI-pro
35. 040.7
45.3
38.2ao.
29.6.us
42.7
31.9406
26.1
35.341.1
29.8a1.
text
32.6x0.
48.520.
37.4a09
27.8109
46.3:n8
33.5x12
24.8.0.3
44.1
27.6an7
Bing Chat
41. 9.as
53.420.
45.4
39.6211
49 4,12
43.7
30,7a09
42.2a0.
35.6207
Ol-preview
59.4 444
68.5.0.
63.9-os
52.3 40.5
66. 1.04
58.4
50.7 10.4
62.3-06
53.2-02
Open Source L.MSs
Llama 2-7b Mistral-7b
26.1
34.3a1.2
30.5a04
22.6x1.1
26.7a13
25.8aa8
24.9a112
28.7a1.1
30.5a09
24.7209
29.5
28.5
20.8
25.3a11
24.8
18.6
22.2a1.3
23.5ams
Qwen 2-7b
30.8
38.2
34.1
27.2
31.7-13
31.8104
25.2-0a
28.6
29.2.ns
EXPERIMENTS
We conduct comprehensive experiments and evaluate different LLMs, using CofCA answer the following questions: 1) Do LLMs show a performance gap between the Wikipedia-basee factual MHQA datasets and our synthesized counterfactual MHQA data? 2) When inputting counterfactual questions, how do LLMs perform in terms of their reasoning ability? 3) How do sub-questions affect the performance of LLMs? 4) How do LLMs perform on reasoning chain evaluation? These investigations aim to shed light on the capabilities and limitations of LLMs when dealing with counterfactual MHQA and multi-step reasoning tasks.
4.1 EXPERIMENTAL SETTINGS
We evaluate LLMs the CofCA benchmark, including 900 randomly selected data from Wikipedia- based MHQA datasets (300 QA pairs from HotpotQA (Yang et al.] 2018), 300 QA pairs from @WikiMultihopOAAHHo al. 2020), 300 QA pairs from MuSiQue (Trivedi al. 2021)), and 900
7

---

Published as a conference paper at ICLR 2025
annotated counterfactual MHQA data (divided into 2-hop, 3-hop, and 4-hop subsets). We employ proprictary LLMs and open-source LLMs in experiments. To enhance reproducibility, we set the temperature to 0 for proprictary models. and all the experiment results are the average scores of three experiment results. We adopt the proprietary L.Mss: GPT-4 (Achiam et al. 2023), GPT-3.5 (Ouyang et al., 2022). text-davinci-003, Bing Chat, GEMINI-pro TTeam et al. 2023) and Open Source L.LMs such as Llama 2-7b, Mistral-7b and Qwen-7b as the baselines. To decouple LLMs' internal memory and reasoning ability, and let LLMs retrieve answers from the given passage as much possible, we design prompt that requires LLMs to only retrieve answers based on the given context. The prompt of QA shown in the Appendix B
4.2 RESULTS
Reasoning VS Memorization The results of the comparison between the selected Wikipedia- based MHQA data and our CofCA can be found in Table 3 and Table LLMS show a perfor- mance gap between the selected data and ours. Taking GPT-4 as an example, GPT-4 achieves high EM and F1 scores (69.9 and 82.3, respec- tively), which are even close to well-finetuned small QA models. For our 2-hop dataset, EM and Fj scores are sharply declined (53.1 in ta- ble and 62.8 in ta)le(4). For 3-hop and 4-hop datasets, GPT-4 even performs worse. Since our synthesized data is new, unprecedented knowl- edge, our results objectively reflect the real rea- soning performance of LLMs. There also 2-hop dataset.
100
60
20
Figure 3: The performance change of F1 score and EM scores when answering 2 sub-questions on the
concern that EM may not be dependable method due to its constraints in accurately representing real performance, particularly in scenarios where answers involve aliases or abbreviations. For example, the gold answer is "Lionel Messi" and the generated answer is "Messi". The two results should be treated the same but the EM score is not applicable in this scenario. As a result, we introduce GPT-4-turbo as the answer evaluator to measure the partial match (PM) between LLMs' generated answers and gold answers The prompts are shown in appendix B The PM scores in table 3 and table reveal that LLMs are better answering factual MHQA questions than counterfactual questions. In light of the results, we can find that LLMs achieve an inflated high performance on the Wikipedia-based MHQA dataset possibly because of the data contamination that leads utilizing LLMs' memory ability rather than reasoning ability.
Impact of Annotation LLMs Our an- notation framework involves two stages: key- words replacement followed by paraphrasing. To demonstrate the impact of annotated pas- sages in different stages, three experiments were conducted using the original passages, passages generated from the first stage, and passages gen- crated from the second stage. We additionally select and annotate another 300 passages from the HotpotQA dataset. Additionally, to ensure consistency in question complexity, we only an- notated 2-hop questions on both sets of passages, aligning with the HotpotQA dataset.
Table 5: Impact on LLMs' performance at different stages in the annotation framework.
Model
HotpotQA
1st Stage
2nd Stage
EM
F1
EM
F1
EM
F1
O1-preview
78.35
86.91
71.55
78.36
68.18
74.69
GPT4
71.35
78.62
65.62
70.47
63.49
67.32
GPT-3.5
64.58
72.36
60.74
66.82
57.94
60.16
text
56.67
61.45
52.43
55.87
49.65
$2.88
GEMINI-Pro
63.28
68.36
57.94
60.33
55.46
58.39
Bing-Chat
61.72
65.16
53.78
57.69
51.97
54.37
Results are illustrated in Table 5 we find that: 1) there is also an obvious performance gap of LLMs between the original HotpotQA data and our two-stage data. Taking GPT-4 as an example, GPT-4 performs better on HotpotQA than our two-stage data, EM score of 71.35 and Fl score of 78.62 respectively. In the 1st stage data, GPT-4 shows a significant decrease, 65.62 EM and 70.47 Fl respectively. 2) The two stages of bias bring the different extents of performance drop. Comparing the performance of GPT-4 in stages I and 2, there is also an obvious drop. from 65.62 to 63.49 and from 70.47 to 67.32. It is mainly because of the data produced from the first stage although some
8

---

Published as a conference paper at ICLR 2025
keywords such named entities and noun phrases, other words, sentence structure, and internal logic are the same as the original passages. The second stage passages are totally different from the original passages. Table also illustrates an example from HotpotQA that is annotated in different stages. We find that the passage in stage still maintains the same sentence structure. While the passage in stage 2 is totally different from the original passage.
AAb-QA Evaluation Figure 3 illustrates the performance degradation because LLMs also suffer from error propagation with the reasoning depth increasing. shows the performance of LLMs on the different hops of questions. According to the observation of the three figures, we find that with the hop increases, the complexity of multi-hop questions also increases, leading to the LLMs' performance decrease. When incorrectly answering the previous sub-question, the latter one will also be influenced. Consequently, the performance of
2 is worse than that of Sub Question 1. Tables 13 and 14 also illustrate the sub-qa performance of LLMs on the 3-hop and 4-hop datasets
in appendix D
Table 6: Ablation study of the MHQA task, where we remove the sub-question information from the prompt and only ask LLMs to give the final answer. Here all the prompts are in Chain-of- Chain-of-Thought setting.
Setting
2 hup
3 hup
hop
EM
F2
EM
F1
EM
F1
GPT-4 w/o Sub-Q
43.8-n3
65.2-03
41.4
61.6
38.1.05
48.9403
w Sub-Q
53.2.05
67.7
44,5e02
64.5
42.1a0
53.1a94
GPT-3.5 w/o Sub Q
34.3am:
51.3ami
32.7a03
48.6wss
31.2w0
41.7a04
sub Q
40.4
56.9.o4
37.5ia
50.2ma
33.5m0
45.9ma
GEMINI-pro w/o Sub-q
25.2-05
55, 2502
20. 850n
38.7
14.1 20
31.0
w sub-Q
34.62n5
64.2zn3
27.3202
42.1a02
25.9a0
33,8 203
text-davinci-003 wro Sub-q sub-Q
1
48 8
22.1
45 6a02
20.0 s0.1
42. 1202
32.3am4
52.7204
27.3
46.420
24.2a05
42.8
Bing Chat wio Sub w sub-Q
37.2.0a
52.4.os
33.340s
44.2
29.3
38.7
41.8-03
56.8-05
40.
48.4
32.4.0
41.3.03
01- Ol-previow w/o Sub-q
52.3ama
63.4204
49.3a05
60.2aas
42.5
55.9
w sub-Q
56.7-01
72.3-03
65,4
46.4.0
60..70.
Joint Performance The joint F2 RC and joint EM RC scores in Table are the whole reasoning chain evaluation results. We find that with the increases in the reasoning chain, the performances of LLMs dropped swiftly. For example, Bing Chat gets comparable performance with GPT-4 (0.7 joint Fi) on answering hop questions and gets a 0.9 joint F, score. However, in the 3-hop question, the joint F2 RC and joint EM RC scores of Bing Chat are 4.2 and 8.4. In the 4-hop dataset, Bing Chat gives 4.7 joint F1 RC, and 8.9 joint EM RC scores, respectively. Since the joint performance is a negative log, the larger scores mean the worse performance on the reasoning chain. We can conclude that LLMs' reasoning ability decreases with the increases in reasoning chain length.
Reasoning Chain Evaluation We calculate the proportion of the reasoning chain on the 2-hop dataset and follow the settings of (Tang et al. 2021 in calculating the percentage of correct or incorrect answers and record the results.
Table showss the reasoning chain evaluation results. The green row shows the percentage of examples whose multi-hop questions can be correctly answered from the right reasoning chain. The red rows show the percentage of examples whose multi-hop questions can be correctly answered but through an incorrect reasoning chain. Among these examples, we notice that there is low percentage of the correct final answer based on the right reasoning chain. There is also large proportion of incorrect final answers shown in rows 2,4,6 and 8.
Taking the results of GPT-3.5 as an example, the right reasoning chain only accounts for 13.3% although it shows relatively high QA performance in previous tables. The percentage of incorrect reasoning chain of GPT-3.5 is 17.7% (sum of the three red rows). However, total failure cases account for 69% (sum of rows 2, 4, 6, and 8) which is substantial for the whole dataset. We conclude that LLMs only get a small proportion of the right reasoning chain and their high performance is relatively inflated due to the considerable proportion of incorrect reasoning chain.

---

Published as a conference paper at ICLR 2025
Table 8: Categorical EM statistics (%) of sub-question evaluation for the five LLMs on 2-hop dataset.c stands for correct and w stands for wrong. For example, the third row shows the percentage of questions where models correctly answer both 2-hop questions and the first sub- sub-question but incorrectly answer the second sub-question. We abbreviate text-davinci-003 as text.
goubl guuta
q
O1-preview
GPT-4
GPT-3.5
GEMINI-pro
text
Bing Chat
GPT-4o
45.2
36.3
13.3
15.0
17.3
28.3
44.5
c
w
9.8
12.3
9.3
9.0
10.7
7.7
10.1
w
2.0
6.7
5.3
7.7
6.0
2.7
w
15.4
24.3
14.7
25.0
16.3
12.7
w
5.7
3.7
5.3
5.8
3.7
3.7
5.3
3.7
3.0
4.7
0.3
7.3
13.3
8.7
1.4
16.6
14.3
31.7
32.3
30.3
31.3
18.1
4.3 IMPACT OF SUB-QUESTION
To evaluate the impact of sub-questions for LLMs, we conduct an ablation study testing the performance of answering the final an- swer and removing the sub-questions from prompts. All LLMS are combined with Chain- of-Thought (Wei et al. 2022) settings, in which LLMs incrementally derive series interme- diate steps. The results, shown in Table 6 in- dicate that when directly asking LLM a multi- hop question and corresponding passage, the performance is much lower than that of adding sub-questions require LLMs reasoning step- by-step. For example, computed from table 6 the performance of GPT-4 on the 2-hop dataset decreased the F1 score and EM by 2.5 and 9.4 respectively. The results show the sub-questions could help LLMs improve the performance of final-QA.
Table 7: LLMs' joint performance on the whole reasoning chain. The scores the average scores of three experiment results. The larger score means worse performance on the whole reasoning chain.
hop
EM
RC
EM
RC
EXT
prrniew
277
GEMINI you
2.1
4.6
9.3
test-davinc--03
2.4
2.9
39
5.2
5.5
74
Bing Chur
0.9
1.9
4.2
8,4
4.7
89
ERROR ANALYSIS
We select total of 20 incorrect final answers generated by GPT-4 from the 4-hop dataset comprehensively illustrate how LLMs make decision on multi-step reasoning tasks. We first verify the proportion of each incorrect sub-answer and final answer. Among the 20 incorrect final answers, 9 of them are wrongly answered in the first sub-questions which leads to the incorrect final answers. While for the remaining 11, of them are incorrect second sub-questions that lead to the wrong final answers. The rest of 4 are influenced by the wrong third answers and fourth answers. From this analysis, we estimate that roughly half of the incorrect final answers are incorrectly reasoning from scratch. We select 20 correct final answers generated by GPT-4 and find that about 4 of them are reasoned from incorrect reasoning chains (wrong sub-answers). revealing that LLMs also sometimes bypass the incorrect reasoning chain and get correct final answers. The insights of the whole experiment results are illustrated in Appendix
CONCLUSION
We presented a novel evaluation framework CoFCA for evaluating L.LMs' evidence integration and multi-step reasoning capabilities by combining sub-question evaluation and counterfactual data annotation. To disentangle LLMSs' memory and reasoning ability, we design a human-in-ther -ooo--lop framework to synthesize counterfactual data. Our results show that, although LLMs performed relatively well on QA tasks, the performance dropped on multi-hop questions that were based on new, counterfactual knowledge. In addition, their high performances inflated and benefit from high proportion incorrect reasoning chains. Our work can facilitate future research on developing faithful knowledge editing methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprini urXiv:2210.02441) 2022.
Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp. Robin Jia, and Douwe Kiela. Models in the loop: Aiding crowdworkers with generative annotation assistants. In North American Chapter of the Association for Computational Linguistics, 2021. URL https
Seopi.semantiin
orgg/CorpusID::..o 245218995
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, 2021. URL http::tttsssatt semanticscho.... D /B//CCorPuSID:244454722 244954723
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmmented generation. In AAAI Conference on Artificial Intelligence, 2023. URL
https://api www.nticschol.e..e.... wwtpssschhhaaoooo///ooooooeeoe httg//orppsID::215350
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168 2021. URL
https://api. commanticscholar wwtpoooaaarrgg/oopuuueee wowp//orrsSDD 239998651
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with Mfain-of-thoughe for large language models. arXiv preprint arXiv:2302.12244: 2023.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi- hop reading comprehension at scale. ArXiv, abs/1905.05460, 2019. URL httpps:/appi
semanticscholars www.orr/////oooiee 153312668
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Chapter of the Association for Computational Linguistics, 2019.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. 13394, 2023, URL
https://api. www.sseimtcooollaaae....e httprioorrr.rrrooooooooor...:::/ 2592433928
Tianyu Gao, Ho-Ching Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. ArXiv, 0bs/23055146277, 2023a. URL https://pp/ppp semanticscholar. orrg//
CorpusIDD:55866571
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang. and Haofen Wang. Retrieval-augmented generation for large language models: A survey. ArXiv, 12./22312.19999 2023b. WtLLhttps://apg semanticscholar. wttpriccchohooooooooooeoeeeeeeee org//orpssi/
2663591511
Shahriar Golchin and Mihai Surdeamu. Data contamination quiz: A tool to detect and estimate contamination in large language models. ArXiv, abs/2311.06233, 2023. URL pettss///ap!
commatticscchlaar.og//oooooooo 2t0p//c69

---

Published as a conference paper at ICLR 2025
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprchensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguistics, pp. 6609-66255 Barcelona, Spain (Online), December 2020. International Committee on Computational I Linguistics. doi: 10.1 10.18/33/1/2020 coling-main.500 URL http:tttps://allanttoooggg www.ggy.rrg//2gggg co coling main.580
Xanh Ho, Johannes Mario Mcissner, Saku Sugawara, and Akiko Aizawa. A survey measuring and mitigating reasoning shortcuts machine reading comprehension. ArXiv, abs/2209.01824, 2022.
URL https://api http:/pcp..smmmntccschhho.aaaa wwt.org/////o.oo htt.://r....r/rrrruuu//:///21990
Wenyue Hua, Jiang Guo, Mingwen Dong, He Zhu, Patrick Ng, and Zhiguo Wang. Propagation and
pitfalls: Reasoning-bas -baed assessment knowledge editing through counterfactual tasks. ArXiv, abs/2401.17585. 2024. URL BRL/https::// //api. semanticscholar.orrt wtt.cc/........roo.ooouus//////.
O. Khattab, Christopher Potts, and Matei A. Zaharia. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. ArXiv, abs/2101.0044 2021. htR.hhtttps//ppp.ssmmnttt
org/CorpusID: 239437663
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023. pp. 1774-1793, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 111138/11/2000 20t.://w1222...nnngg--aeeee findinggs-all. URL https: https::/aclaanthooogy wgg../g//ooor/32333
Sindings-acl... 112
Jos'e Antonio Hern andez I. ope,, Boqi Chen, Tushar Sharma, and D'aniel Varr'o. On inter-dataset code duplication and data leakage in large language models. ArXiv, abs/2401.079330, 2024. URL
https://api. www.steisscoooolaee..o wtt.:/ccc...o.oooooooooooooooa99
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for
retrieval-augmeenedd large language models. ArXiv, abs/2305.14283. 2023. URL https
//api. sempanticmaniicccho..a. httpp//eeaaaaaaci.oooooooooooooo wrgp/orppr/.: 258841283
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading compre- hension through question decomposition and rescoring, In ACL, 2019.
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10056-10070, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.11665//v12002..allooo acl-oog.599., UtL/https://accatthooloo
2023 acl-long. 1ong.555.55 559
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, Abs/2203.02155. 2022. URL htRpst //ppi./pp.ppp// oot.eaaneeeccoolooooooooorrroooo :40.rpu 246426909
Mykhailo Poliakov and Nadiya Shvai. Multi-meta-rag: Improving rag for multi-hop queries using database filtering with llm-extracted metadata. ArXiv, 1bb/2400.13211,, 2024. URL https
seapi.semanticseeee hwwpaaarrrrr//ooosi::ia: 27702202
Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. Receval: Evaluating reasoning chains via correctness and informativeness. In Conference on Empirical Methods in Natural Language Processing, 2023.
semanticscholar http://ccco....ooooooooooo/////1
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. Squad: 100,000+ questions for machine comprehension text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL httpps:/appi scoatiiccooa ort.aaarrrrrg//rruss:111111001
12

---

Published as a conference paper at ICLR 2025
Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Cheng- wei Qin, Caiming Xiong, and Shafiq R. Joty. How much are 1lms contaminated? a com- prehensive survey and the llmsanitize library. ArXiv, abs/2404.00699, 2024. URL https
/emant ccchollaar org/ 268819579.
Manley Roberts, Himanshu Thakur, Christine Herliby, Colin White, and Samuel Dooley, Data contamination through the lens of time. ArXiv, abs/2310.10628, 2023. URL https://api semanticscholar.c (CooppsSD:22411120000
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. ArXiv, abs/2310.16789, 2023. URL/htts:://// //api. www.seeaaiisiee www.ntticccoolarnncoooo http:/ola...rgg/ooouusi:::::/:/0
Konrad Staniszewski, Szymon Tworkowski, Yu Zhao, Sebastian Jaszezur, Henryk Michalewski, Lukasz Kuci nski, and Piotr Milo's. Structured packing in Ilm training improves long context utilization. ArXiv, abs/2312.17296, 2023, URLhhtps:://ap wew.aaisssmmanisccoolloee | 266690935
Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking regaraanniinrerrriieaaaaammeeee generation for multi-he -hop queries. ArXiv, abs/2401.15391. 2024. URL httpps:/app. /emp:::////i...aanttta www.sttiimcnoooo..oo..m... org/CorpusID; 267312593
Yixuan Tang, Hwee Tou Ng, and Anthony Tung. Do multi-hop question answering systems know how to answer the single-hop sub-questions? In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2244-3249 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 wea/222eeeaclmmaaii maa.-aaam.mm..... eaac-mmain. 283
URL https://acllatthologg/ www.ey.orr//0000 2021
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint 231 1805, 2023.
Petter Tömneerg Chatgpt-4 outperforms experts and crowd workers annotating political twit- ter messages with zero-shot learning. ArXiv, abs/2304.06588, 2023. URL https://api
semanticscholar. Baaiiaa.org//oorro wwwp/rrppssD::2588888888 228100255
H. Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop ques- tions via single-hop question composition. Transactions of the Association for Computational Lin- guistics, 10:539-554, 2021. https:/tp:://app www.seeetttttooleeeeee httpssccoooaaoooooooooeeeereeeee orgg/CorpuSDD:226777977
H. Trivedi, Niramjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval retriveval with chain-of-thougge reasoning for knowledge-intenssive multi-step questions. ArXiv, abs/2212.105099 2022. URL
http:/ppp..mmmtttttcccoooorrrrrs http://CCoopsSI02557777999999 254877499
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Zhikun Xu, Bowen Ding, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-qa evaluation. In Neural Information Processing Systems, 2023a. URL / semanticscholar http:iicccccoooooorr//rrrrpppppo 259951112
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. Novelqa: A benchmark for long-range novel question an- swering. ArXiv, abs/2403.12766. 2024. URL httpps://app www.sseesmcaotooeeee.eme. 268531221
Utt.ittpps///p....eattttcccooooo wwtps/schhh..r.ooooooorououeueue orgg/Crppsll
Kuan Wang, Yadong Lu, Michael Santacroce, Yeyun Gong, Chao Zhang, and Yelong Shen. Adapting Ilm agents with universal feedback in communication. 2023b. URL https://app semanticscholar .or...rg//eee wwwpoooopps..:2333e663
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 19003, 2022.
//api. semanticscholar seenanttccccolllarrror/r/rouuupp 246411662
13

---

Published as a conference paper at ICLR 2025
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug- war between Ilm's inter- nal prior and external evidence. 2024. URL https://api 269157310
Yunxuan Xiao, Yanru Qu, Lin Qiu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. ArXiv, abs/1905.06933, 2019. URL httpss//a/p
semanticscholar orr.poopui:/ 15510011
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. 2024a. URL https: www.nnnicccoooooorrrgggg///oo//e : 269448637
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404. 2024b..
Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. Towards fine-grained causal reasoning and qa. ArXiv, abs/2204.07408, 2022. URL https://api. www.pp.smmattccccollrr...rr org/
CorpusID:248218748
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. URL
https://apl. semanticscholar hww.sccholarrrr/////oooooooo 22024
52822214
Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. Ifqa: A dataset for open-domain question answering under counterfactual presuppositions. arXiv preprint arXiv:2305. 2023.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. 2024. URL https://api. semanticscholar...... hrt.aaarrrrr//oooosi::22:25933
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't 't make your Ilm an evaluation benchmark cheater. ArXiv, 1..02331.01194, 2023a. WRRL/htps::/ //api. www.ntccscaoolaattttomoeo http://aaatttt.....oooooooooooo. orgp/Corpuss::22551122222
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14544-14556, Singapore, December 2023b. Association for Computational Linguistics. doi: URL httpt:::/aalaattoollg.ooo//// org/ 2023. mondinggs-mnlpp 968
http://222..iinnsoooome madings--mmlp.966
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Graph-informed dynamic evaluation large language models. arXiv preprint arXiv:2309. 17167, 2023a.
Wenhong Zhu, Hong ping Hao, Zhiwei Hc, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. Clean-eval: Clean evaluation contaminated large language models. ArXiv, 0bs/2311..0955. 2023b. httppss/:://ap wemanticccooooaaaa........a.. http::///.....nttoc.oooooooooooo 265213054
A REPRODUCIBILITY STATEMENT
To make the results and models reproducible and verifiable, we provide our full data annotation guideline, data link, implementation details, and prompts: We detail the process of data annotation in section 3.] and the implementations are in Appendix All the prompts required to reproduce the results illustrated in Appendix B
14

---

Published as a conference paper at ICLR 2025
Table 9: The prompt template of passage rewriting and question generation. We here take 2-hop data annotation as the example. [WORDS] denotes the information we should give.
Prompts of NER, Noun Phrase and Adjective replacement
Prompt Now you are passage annotator. you need to recognize all the named entities, noun phrases, and adjectives from the given [CONTEXT), then translate the passage into Chinese and translate to English. Please output the response in JSON format (Passage: String] [CONTEXT) The given context.
Prompts Question Generation
Example One-shot example with multi-hop QA pairs, Sub-QA pairs, and passage.
Prompt Now you multi-hop question generation machine, given an example of hop question and its sub-questions, sub-answers, and final answer is [2 hop [rrestion][[Subb [Suestions][[SubAAA [Final Answer]. you need to generate new 2 hop multi-hop question same with the given example and its sub-questions sub-answers and final answer from the given [Context]. Please follow the sentence structure of give examples and output the response in JSON format (2 hop question: String, sub-questions: List, sub-answers:List. final answer:String]:
and
12 hop question| The given example of hop question. ISub-Questions] The given example of sub-questions.
[Sub-Answers] The given example of sub-answers.
[Final Answer] The given example of final answer. [CONTEXT] The given passage
Prompts of QA
Prompt You QA machine, you need to answer the [Question] from given the [Context], and you only need to come out with the correct answer without other words. Let's think step by step, and please output the answer to the [Question] in the format of: (Final Answer: String). [QUESTION] The given question. [CONTEXT] The given passage.
Prompts Partial Match Evalutaion
Prompt You Answer evaluator, you need measure the semantic similarity between [Generated Answer] and [Gold Answer), and give the score, mcans equal, 0 mcans not. Some answers may have abbreviations or alias, example, Lionel Messi is equal Messi, Donald Trump equal to Trump. Please only output the score or 0 without any other words. [Generated Answer] The LLM generated answer. [Gold Answer/ The ground truth.
B PROMPTS AND EXAMPLES
When evaluating large language models, prompting is brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort should be dedicated to designing a painstakingly crafted perfect prompt for the given task (Arora et al., 2022; Diao et al. 2.23). In this study. We investigate the performance of zero-shot our benchmark. To climinate the randomness, we manually select one demonstration for each task, ensuring that all tasks covered.
We give our designed input examples for three different tasks to help readers understand our imple- mentation, shown in Table 9 table 12
respectively. The original and rewritten passages are shown in the
The annotated multi-hop questions are shown in the table 11
15

---

Published as a conference paper at ICLR 2025
Table 10: Examples of passage annotated in different stages. The words in red indicate the revision part. Passage in stage still has the same sentence structure with the original passage while passage in stage 2 is totally different.
Stage
Passages
HotporQA
Elliott Lester is an English film and television director, best known for directing the film "Blizz. He made his releused on April 4, 2017. Marvin Kellaway is a Britshh movic and radio belmsman, foremost noted for belming the movic Flashstrike.. commenccd helning genesis in 2007 with Affeection the Potion', , and his most recent movie, Repercussion was unveiled on May 3, 2018.
directing debut in 2006 with "Love the Drug", and lis latest film." "Afenmath"
Stage
Stage
Marvin Kellaway, a prominent British director in both tilm and radio, best known for directing the popular movie "Flassttrik"" He began his directing career in 2007 with "Affection Is the Potion', and his latest film, 'Repereussion', , was released on May 3, 2018.
Table 11: Examples of annotated different question types and question hops. We emphasize keywords for their respective categories.
Question Type
Hop
Multi-hop Question
2 hop
When was the actor who played Helen in FBC series The Murder born?
Bridge
3 hop
Who were the learners of the people that was the principal violist in the Fioba Symphony Band and instructed
music to Michard Rokney?
4 hop
Which is Inter, the birthday of Zephyr Vole-Anderson or the time that 2060 Kingdom of Azkahan ATP
Conqueror occurred in Gleeful Peak, Atlantis?
2 hop
Where the Bluc Falls Empire located and what products are it responsible for importing?
Comparison
3 hop
Which is later, the opening time of Gold or the opening time of the Mad Book in 2006?
4 hup
War the release of the movic Ocean Secrets before after Echoes of Tomorrow & Victoria Wrigh??
Table 12: The original passages with the rewritten passages. The first and third rows of the table are the original passages, and the second and fourth rows show the corresponding rewritten passages.
Title
Passages
Radio City (Indian radio station)
Radio City India's first private FM racho station and was started on 3 July 2001. It broadcasts on 91.1 (earber 91.0 in most cities) megahertz from Mumbai (where il was sturted in 2004), Bengaluru (started first in 2001). Lucknow, and launched in Hydershad in March 2006, Chennai on 7 July 2006, and in Visakhapatnam in October 2007. Radio City recently forayed into New Media in May 2008 with the launch music portal PlanetRadiocity. com that offers music related news, vidcos, songs, and other music related features. The Radio station currenily plays a mix of Hindi and Regional music. Abraham Thonas is the
New Delhi (since 2003). plays Hindi, Fnglish, and regional songs.
CEO of the company.
Permission (African
Permission Affica's second public FM radio station, launched on August 2002. broadcasts 95.1 (previously 95.0 in most cities) megahertz from Baili (where was launched in 2006), Hanwi (first launched in 2008). Shuyu, and Sadem (since 2004-09), It plays Japanese, Chinese,
radio station)
folk songs. started in Hindu in April 2007, Beuge 8 August 2007, and in Adler in November 2008. The Permission recently forayed into Old Business in June 2009 with the launch music portal - Mennsesnssionncc com, which offers music-related news, videos, songs, and other music-related fentures. The Permission currently plays a mix of Japanese and folk music. Amma is the founder of the company.
Lights Out Paris
Lights Out Paris is the first studio album by American hip-hop artist Sims, memher of Minncapolis indie hip-hop collective Doomtree. It was released July 28, 2005, on Doomtree Records and includes guest appearances from P.O.S. Crescent Moon, and Toki Wright, among others. The alhum was re-relensed with four remixes and five songs from Sims' False Hopes Fourun vinyl in June 2015.
Brilliant
Brilliant is studio album hy Australian Shout artist Allen, member London indie Shout collective Die. was published on 29 October 2006 on Die Records and features guest appearances from Lucia, Lisa, and Bill, among others. The album was relisted on vinyl in July 2016, along with seven remixes and nine tracks from Allen's 's Right.
C JOINT COMPUTATION
We here list the details of computing the joint scores on the whole reasoning chain: For exam- ple, a N-hop question and its N sub-questions, given their precisions and recalls on the MHQA (PIMIION) ((MITA)) and the Sub-QA (Pisub_qa') R(lsub "),.. (p(sub_ Ritaab ga"" 1), respectively, (
we calculate joint performance as:
p(joint) p(MHHA) pas qa't) pis.ppusuubuuu aa
R(jinn) = R(MHQA) gu'' qlsub_qaa
2 P(jiin ((jiiin)
Joint F, RC = - -log Pijointi + Rojiin
16

---

Published as a conference paper at ICLR 2025
Table 13:
The LLM evaluation on CofCA hop dataset. We here measure the sub-qa task and compare the performance between each hop. Qi means the ith sub-questions.
3 hop
QT HM
Q1
Q2 EM
Q2
Q3 EM
Q3
GPT-4
70.9
80.8a06
59.7.03
74.9
18.1
68.8an.5
GPT-3.5
43.0 33.0
56,4207
38.6.01
49.3a0.2
29.0a0.3
40.6a0.2
GEMINI-pro
5.820.4
33.8a0s
4.4g06
30.8
4.1
31.5a09
text-davinci-003
23.3s0.8
42.4g03
20.5.04
33.7a03
19.5x0,5
29.6x0.6
Bing Chat
7.2a09
34.0s06
5.8
31.5u0.
3,1206
32.3a0.4
Table 14: The LLM performance on CofCA hop dataset. We here measure the sub-qa task and compare the performance between each hop.
4 hop
QI EM
QI
Q2 EM
Q2
Q3 EM
Q
Q4 EM
Q4 Fi
GPT-4
60.9s04
66.7+03
56.4
62.6.n.4
28.4xu2
58.7 =04
23.1+02
56.3+03
GPT-3.5
40.7=04
46.9+0.
30.1
36.3+0.
20.2+0.1
47.2+0.5
14.7+04
44.8
GEMINI-pro
14.9+as
39.2+a.
10.4+0.
38.3+n.4
9.1+0.
34.9+0.4
7.2-03
29.5+0.
text-davinci-003
19.8
39.220.4
19.2
30.720.6
18. 820.
28.610.6
18.5207
27.820.2
Bing Chat
20.8
39.420.
16.920.
37.1a0.3
6.220.5
35.8
5.5207
15.
where the Joint F1 RC means the joint F1 performance of the reasoning chain.
Given their EM scores on the MHQA (EMO)OAA) and the Sub-QA E.M(subgaa E. EMiisbbgaa
2E (MHQA) ...EM qa
Joint EM RC = - log
EM + ..... ........M... issbb _aaaaa__aaaaa )
where the Joint EM RC means the joint EM performance of the reasoning chain.
D PERFORMANCE ANALYSIS
hop 3 hop
hop hop
70
4
80
80
80
30
20
20
Bing
GP7-4
GPT-35
GcMiNists Bol-deiese001 Dag Chat
Figure 4: The performance change of EM and F1 scores when answering from 2 hop questions to hop questions.
As the quantitative complementary of Sub-QA evaluation, we here list the results of LLMs' perfor- mance on 3-hop and 4-hop datasets. The LLMs' reasoning performance dropped dramatically, e.g. in table 13 GPT-4 achieves 70.9 EM and 80.8 F, scores on sub-questionl but only gets 59.7 EM, 74.9 F1, and 58.1 EM, 68.8 F, scores on sub-question2 and uub-question3 respectively.
table 14] we further find that when answering 4 hop questions, the results show cliff-like descent from sub-question2 to sub-question qube-uustion3, especially GPT-3.5 gets 46.9 F, in sub-question2 but drop to 36.3 F, score in sub-question3.
17

---

Published as a conference paper at ICLR 2025
comparioon beirreaa
compariion between differcnt
90
Genti XX
80
Geatb X
Dig Chi
Dag Oli
O2 Ouetor
70
gueer
8
30
Moepo1QA
MaSiQuz
Moop
3-hap
4-hop
Tepp-OA
207ki
MediQuz
Mhpp
34hag
Figure 5: Performance gap between Wikipedia- based factual multi-hop QA datasets and our 2-hop, 3-hop, and 4-hop counterfactual MHQA data of table 3 and table 4 The line charts reveal that LLMs show an obvious performance gap between previous datasets and CofCA.
Table 15: Error comparison factual and counterfactual data on different question types.
Question Type
Data Type
Wrong
Correct
factual
6
14
Comparison
counterfactual
11
factual
4
16
Bridge
counterfactual
8
We randomly sampled 40 factual data (20 bridge questions and 20 comparison questions) from the HotpotQA dataset and 40 counterfactual data (20 bridge questions and 20 comparison questions) from our 2-hop data. We count the number of wrong and correct answers given by GPT-4 on different question and data types. Table [13 illusttras the results on our selected 80 data. We find that Although GPT-4 is worse on counterfactual data than factual data, GPT-4 is better at answering bridge-type questions than comparison questions. We further analyze the error categories on the errors. We
Table 16: Statistics of different error categories
Data types
Factual
Counterfactual
Question Types
Bridge
Comparison
Bridge 5
Comparison
Reasoning failures
3
5
Retrieval Errors
Contradictions
0
2
3
categorize errors into three types: reasoning failures, evidence retrieval errors, and contradictions. We compare the number of different error categories and analyze their relative prevalence factual versus counterfactual datasets. In factual data, since all knowledge of passages is from the real world, there are no contradictions. However, in counterfactual data, there are contradiction errors which means the knowledge is contradicted to LLMs' knowledge. E.g. the question: "When did Africa's 's second public FM radio station launch?" In Wikipedia, the second radio station is "Rádio Nacional de Angola (RNA) which was launched in 1951, while in our counterfactual passage, the answer is "Permission", launched 2002.
E INSIGHTS OF LLM MULTI-STEP REASONING EVALUATION
Drawing from the above experimental results, we draw the conclusions:
18

---

Published as a conference paper at ICLR 2025
Exact Matching While exact matching is simple and effective method for MHQA evaluation, struggles with issues when the answers have abbreviations or other expressions. For example, in synthesized counterfactual MHQA data, if the golden answer the question "When did Africa's second public FM radio station launch?" "2002" and the generated answer of GPT-4 "4 August 2002", the exact match not be computed accurately. All the answers generated by LLMs have this issue. Thus, It is urgent to propose a more universal QA evaluation score in LLMs' reasoning performance evaluation.
Multi-step Reasoning Although the experiment results demonstrate that LLMss can perform multi- step reasoning ability to a certain extent, they remain sensitive to prompts and the impact of additional contexts, especially the sub-questions. Providing sub-questions additional information into prompts can help guide LLMs to reason the correct direction and show a relatively strong performance.
Reasoning chain Evaluation (Joint F1 RC and Joint EM RC in this work) The advantage of our reasoning evaluation method is we jointly consider the sub-QA performance and final-QA performance when LLMs bypass the incorrect reasoning chain and achieve correct final answer, the scores remain very low. However, this evaluation method is easily influenced by the LLMs' performance on the first sub-questions, since the answer order is sequential. if the first sub-question is incorrectly answered, the following sub-questions and the final question will also be influenced, leading to a very low score. How to answer sub-questions more correctly remains exploration.
F IMPLEMENTATION DETAILS
For proprietary models, we employ official APIs to interact with exclusive LLMs and prompts are well-defined. For open-source models, all experiments are conducted on 8 A A100 GPUs.
G LIMIATATIONS
In this paper, we focus the evaluation of L.MMs' real multi-step reasoning ability our annotated counterfactual MHQA data. Although L.LMs show an obvious performance gap between previous factual MHQA datasets and our dataset, the data size of our dataset still remains improved. The Exact Match (EM) for reporting QA performance still faces challenges, because EM does not report LLMs real performance due to the variation in the expression of the answers.
19


Reviews:
[{"review_id": "lX1Pi0YX5r", "reviewer": "Reviewer_6xmP", "summary": "This paper proposed a new dataset, CofCA for evaluating LLM’s counterfactual multi-hop reasoning ability, which focuses on both final answer evaluation and reasoning step evaluation. The dataset construction involves 2 steps, 1) the sampled Wikipedia paragraphs are sent to GPT-4 for entity replacement and paraphrase, and then back-translation is done to further modify the paragraphsm and then these paragraphs are manually checked 2) GPT-4 is then used to generate QA pairs based on the newly generate paragraphs. Here GPT-4 is asked to generate both overall multi-hop QA pair and subquestion answer pairs, and the resulting data is also manually verified. \nThe authors then conducted experiments on both newly generated data and HotpotQA, 2WikiHop and Musique, using both closed-sourced models and open-source models. The results show that all LLMs suffer significant performance drop on counterfactual questions and the joint performance which considers both final answer and reasoning process correctness can be even worse, exposing the weakness of current LLMs.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CofCA could be a valueable resource for studying LLM’s true reasoning ability. Due to its counterfactual nature, the LLMs can no longer use their internal knowledge for finding reasoning shortcut, hence the model have to reason over provided context. Also, the annotation of intermediate sub question answer pairs can also enable examination of the reasoning process. \n\n2. The experiments and analysis are very thorough. The results show a clear trend of performance vs question complexity and the analysis on the reasoning process further exposes the weaknesses of the SOTA LLMs.", "weaknesses": "My main concern is the scale of the passages set when constructing the data. For previous datasets such as. HotpotQA and 2WikiHop, they usually construct questions using multiple passages to ensure that the answer have to be deducted from multiple pieces of evidence. However, it seems that for CofCA, the questions are generate using only a single passage? If this is the case, how can you make sure the generated multi-hop QA pairs are fully grounded to the passage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6JQpTX1j0g", "reviewer": "Reviewer_royr", "summary": "The paper introduces CofCA, an evaluation benchmark for multi-hop reasoning abilities of LLMs over counterfactual data. CofCA is created by (1) LLMs alter Wikipedia passages by replacing keywords with counterfactual information, and (2) LLMs generate multi-hop questions over the modified passages. Both steps are manually verified by human annotators, and the benchmark includes 900 multi-hop questions from 300 passages. Experimental results show a gap between performance on the counterfactual CofCA and existing multi-hop QA datasets. Additionally, the paper shows that the low performance is due to reasoning failures.", "questions": "See weaknesses, mainly:\n- Does the counterfactual information contradicts real-world facts? If so, isn't some degradation in performance expected? \n- How often are shortcuts in the collected examples? \n- What qualitative analysis did you consider that might strengthen the main claims of the paper?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- A counterfactual multi-hop benchmark can be helpful to understand the ability of LLMs to perform multi-hop reasoning. The benchmark will be publicly available in addition to the annotation guidelines and implementation details.\n\n- The paper features extensive quantitative analysis examining the effect of the annotation pipeline and the performance of sub-questions amongst others.", "weaknesses": "- Given the examples presented in the paper, I was not fully convinced about several aspects of the collected data. Mainly, how do you ensure that the counterfactual information does not contradict real-world facts? Also, both examples in the main paper seem to have shortcuts, e.g., there is only one position in the example in Fig.1, and only one designer in the example in Fig. 2.\n\n- Qualitative analysis - the analysis presented in Section 5 did not fully address the concerns above, and is limited to 20 correct/incorrect questions with GPT-4. I think the paper could benefit from a thorough analysis regarding the quality of the generated data examining contradiction and shortcuts and verifying that the main claims of the paper (multi-hop QA is more challenging on counterfactual information) is supported. For example, it could have been interesting to have an error analysis that compares different error categories between factual and counterfactual questions.\n\n- Presentation - the presentation in several parts of the paper was a bit hard to follow. For example, in Fig.1, the text is very small and there is an error (the bottom-right box the question is about the coach and the answer is Point Guard). In Tab.2, the ‘Whole data property’ is not the sum of the other columns (I assume this is due to the factual questions sampled from other datasets?). The main results in Section 4.2 require comparing between information presented in different tables (Tables 3 and 4). Table 8 is referenced before Table 7 but presented afterwards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iJzQ00y7xg", "reviewer": "Reviewer_cSw3", "summary": "This paper introduces a new dataset called CofCA, designed to effectively assess LLMs' capability in counterfactual multi-hop question answering. The authors generate the dataset by starting with Wikipedia passages and applying techniques such as entity replacement and back-translation to create question-answer pairs. The models are then evaluated at both the final-answer level and the sub-question level. To ensure the quality of the dataset, the authors perform thorough quality checks and include human-in-the-loop validation throughout the dataset construction process.", "questions": "1. Could you provide a more intuitive explanation for the **JOINT COMPUTATION** metric mentioned in Appendix B? An example or simplified breakdown might help readers understand its significance and how it is calculated in practical terms. This would not only make technical details more accessible but also allow readers to better appreciate its role in evaluating model performance.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, presenting the content in a clear and accessible manner.\n\n2. The authors demonstrate an awareness of the limitations of using Exact Match (EM) and F1 scores when evaluating the output of large language models (LLMs). To address this, they introduce an additional Partial Match metric, which helps mitigate the issue and provides a more nuanced evaluation.", "weaknesses": "1. One major drawback of the paper is that, while it claims to assess LLMs' true reasoning abilities, it primarily reports the accuracy of different sub-questions without fully addressing whether the models are capable of reasoning holistically. For a more comprehensive evaluation of reasoning abilities, other factors should also be considered, such as whether the model is able to effectively decompose the problem into meaningful or correct sub-questions, rather than treating them as isolated tasks.\n\n2. The quality of the dataset is also in question. Currently, the correctness and informativeness of the dataset are evaluated using GPT-4. However, as the paper itself shows, GPT-4 struggles to correctly answer these questions, which suggests that it may not be fully reliable for assessing the quality of the dataset. This raises concerns about the robustness of the dataset validation process.\n\n3. There is further concern about ensuring that the generated questions are non-trivial. As shown in Table 9, the prompt generates the multi-hop questions and the sub-questions, along with their answers, simultaneously. This raises a natural concern that the generated multi-hop questions may either be independent of these sub-questions or follow a single pattern (e.g., q1->q2->q3), leading to a lack of diversity in question structure. The paper does not address or discuss this potential limitation.\n\n4. When benchmarking models, the paper lacks evaluation on more recent and advanced models, such as GPT-4 Turbo, GPT-4O, and Claude series, as well as newer open-source models like LLaMA 3 or Deepseek. Including these models would provide a more up-to-date and comprehensive analysis of the current state-of-the-art in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gLptpDOgUA", "reviewer": "Reviewer_gcH3", "summary": "The paper is designed to evaluate large language models (LLMs) on multi-hop counterfactual reasoning abilities. CofCA generates counterfactual contexts that LLMs likely haven’t encountered during pre-training, and ask questions based on the counterfactual contexts.  It also emphasizes sub-question evaluation for each reasoning step. \n\nThe evaluation in this paper reveals that models like GPT-4 show inflated scores in standard QA due to these contamination issues, while performance drops on CofCA due to its reasoning on counterfactual contexts, highlighting the gap between LLM memory and genuine reasoning.", "questions": "As given in the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CofCA minimizes data contamination by creating a cleaner benchmark for evaluating LLM reasoning.\n\n2. The diverse counterfactual and factual datasets allow researchers to observe performance discrepancies, adding depth to LLM evaluation.\n\n3. The constructed dataset is clearly different from existing counterfactual QA benchmarks, including distentQA and IfQA. The multi-hop challenge lacks of enough study in counterfactual setting.", "weaknesses": "1. More open-source LLMs should be evaluated on the benchmark, like llama-3 or 13B-level models.\n\n2. OpenAI o1 was evaluated in the paper, it shows using long reasoning-chains could help a lot on this reasoning tasks, and more insights on comparing o1 reasoning with 4o reasoning could be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)