CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark

## Abstract

Abstract While Large Language Models (LLMs) excel in question-answering (QA) tasks, their real reasoning abilities on multiple evidence retrieval and integration on Multi-hop QA tasks remain less explored. Firstly, LLMs sometimes generate answers that rely on internal memory rather than retrieving evidence and reasoning in the given context, which brings concerns about the evaluation quality of real reasoning abilities. Although previous counterfactual QA benchmarks can separate the internal memory of LLMs, they focus solely on final QA performance, which is insufficient for reporting LLMs’ real reasoning abilities. Because LLMs are expected to engage in intricate reasoning processes that involve evidence retrieval and answering a series of sub-questions from given passages. Moreover, current factual Multi-hop QA (MHQA) benchmarks are annotated on open-source corpora such as Wikipedia, although useful for multi-step reasoning evaluation, they show limitations due to the potential data contamination in LLMs’ pre-training stage. To address these issues, we introduce a Step-wise Co unter f a c tu a l benchmark (CofCA), a novel evaluation benchmark consisting of factual data and counterfactual data that reveals LLMs’ real reasoning abilities on multi-step reasoning and reasoning chain evaluation. Our experimental results reveal a significant performance gap of several LLMs between Wikipedia-based factual data and counterfactual data, deeming data contamination issues in existing benchmarks. Moreover, we observe that LLMs usually bypass the correct reasoning chain, showing an inflated multi-step reasoning performance. We believe that our CofCA benchmark will enhance and facilitate the evaluations of trustworthy LLMs.

## 1 Introduction

Retrieval-augmented generation (RAG) (Chen et al., ; Ma et al., ; Gao et al., ), has garnered increasing research attention (Tang & Yang, ; Poliakov & Shvai, ). As shown in Figure 1a, given a query, a RAG system first retrieves pieces of evidence from a set of relevant passages, and then generates a response by taking the passages as additional context to LLMs. While most existing work concentrates on the setting where the answer can be derived from a single retrieved passage, there are situations where LLMs are required to integrate information from multiple resources as evidence to infer the answer. As shown in Figure 1b, given the query, the single retrieved passages cannot offer enough information to generate correct answers. Instead, the evidence “Earth is the third planet” and “Mars is after Earth” are required to be integrated for deciding “The fourth planet of the sun”.

Recently, a line of work (Wang et al., ; Staniszewski et al., ; Trivedi et al., ) has investigated the evidence integration capabilities of LLMs combined with RAG, using existing Multi-hop QA (MHQA) datasets such as HotpotQA (Yang et al., ) and 2WikiMultihopQA (Ho et al., ). However, such work focuses only on the correctness of the final answer, without evaluating the reasoning process itself. It is thus difficult to understand whether shortcut reasoning exists (Tang et al., ; Ho et al., ; Yang et al., ) and whether such reasoning is robust across out-of-distribution (OOD) tasks.
Existing research (Tang et al., ) shows that strong QA models such as DFGN (Xiao et al., ), DecompRC (Min et al., ), and CogQA (Ding et al., ) can correctly answer multi-hop questions but bypass the correct reasoning chain.
However, no existing research has evaluated the performance of LLMs to the same problem.

Figure: Figure 1: Differences between (a) and(b). The words in red are the pieces of evidence for the given questions. The differences between factual MHQA (c) and counterfactual MHQA (d). c: we input a factual MHQA with and without context into ChatGPT. ChatGPT could output the correct answer based on its internal memory regardless of the context. d: When inputting a counterfactual QA, where the passage is rewritten from the existing factual passage in c (words in the red), ChatGPT cannot rely on its memory and must reason on the given context, deeming that counterfactual QA can decouple LLMs’ internal memory and reasoning abilities.
Refer to caption: x1.png

We aim to address the above issues by making a systematic evaluation of the LLMs’ reasoning process.
To this end, an existing sub-question style benchmark (Tang et al., ) derived from HotpotQA Yang et al. () can be useful, in which LLMs are required to answer a series of sub-questions to reach the target rather than giving a final answer solely.
However, existing datasets cannot be directly used for reporting LLM real performance, due to the risk of data leakage in the pretraining stage (L’opez et al., ; Zhou et al., ; Fu et al., ).
As shown in Figure 1c, for instance, given the question “Who was the captain of Argentine team that was born in 1987?”, ChatGPT might give the correct answer even without a context, since relevant knowledge is learned by the model during the pre-training stage.
Therefore, a sub-question benchmark
that avoids the negative influence of data leakage is needed. To this purpose, we consider the counterfactual evaluation method (Li et al., ; Neeman et al., ; Zhou et al., ; Wu et al., ; Yu et al., ).
We make modifications to existing knowledge in Wikipedia, deriving new passages and ensuring that knowledge in these passages is non-existent in the real world, thus those answers have little chance of being inherent in LLMs.
For example, in Figure 1d, given the question “Which position did the coach play that serves Chinese national basketball team and born in 1997?”, LLMs cannot rely on their memories and must answer the first sub-question “Which coach serves as Chinese national basketball team and born in 1997?”, before finding the final answer by asking a second sub-question “Which position did Alice Brown play?”. The passage in Figure 1d, is modified from the passage in Figure 1c.
Specifically, we generate counterfactual data to Wikipedia knowledge by replacing key information (named entities, noun phrases, and synonyms) and then paraphrasing (back translation).

We call our dataset CofCA, a step-wise and Counterfactual MHQA benchmark, which consists of counterfactual passages with corresponding multi-hop QA pairs and sub-QA pairs, and an equal amount of factual MHQA data from HotpotQA, 2WikiMultihopQA and MuSiQue as the control group.
Using CofCA, we evaluate and report LLMs’ real reasoning ability. Experimental results show two types of inflated performance in LLMs: 1) an obvious performance gap between Wikipedia-based factual MHQA benchmarks and counterfactual MHQA data; 2) inflated performance due to a low proportion of correct reasoning chains as well as a high proportion of incorrect reasoning chains, with GPT-4 achieving only 36.3% correct reasoning chains across the entire dataset. Additionally, we observe that incorporating sub-questions into the prompt as part of the reasoning chain is a more efficient approach for improving LLMs’ performance.

To the best of our knowledge, we are the first to introduce counterfactual data into the evaluation of the multi-step reasoning ability of LLMs, finding that there is a significant performance gap between LLMs’ performance on factual data and counterfactual data. Furthermore, the reasoning abilities of LLMs are exaggerated because of the poor reasoning chain performance. The Whole CofCA data are available at [https://anonymous.4open.science/r/LLM_inherent_multi_step_eval-3818/](https://anonymous.4open.science/r/LLM_inherent_multi_step_eval-3818/).

## 2 Related Work

Retrieval Augmented Generation
(RAG) improves LLM’s response (Borgeaud et al., ) and also mitigates the occurrence of hallucinations, thereby enhancing the models’ credibility (Gao et al., ). As demonstrated by Khattab et al. (), designs a RAG system for MHQA and claim verification tasks. These tasks require the extraction of evidence from two or more documents to produce a correct answer. Tang & Yang () propose a Multi-hop RAG benchmark, which consists of a large collection of multi-hop queries, ground-truth answers, and the corresponding supporting evidence. Multi-hop RAG requires LLM to reason and answer multi-hop queries given the evidence.
However, LLMs’ memorized knowledge sometimes conflicts with the given context, emphasizing the importance of correcting LLMs’ generations with new facts. Li et al. (); Neeman et al. (); Zhou et al. (); Wu et al. (); Yu et al. () propose counterfactual QA benchmarks to separate LLMs’ parametrical knowledge (internal) and contextual knowledge (outer) that fix LLMs to reasoning on the given context strictly by editing the contextual information or prompts. Previous work motivates us to explore LLMs’ real reasoning ability by reasoning in counterfactual contexts. However, counterfactual QA datasets still only assess final QA performance and lack reasoning process evaluation.

Multi-hop QA Datasets
Multi-hop QA requires more than one reasoning step in multiple paragraphs to answer a question (Dua et al., ; Yang et al., ; Ho et al., ; Trivedi et al., ). Notably, Tang et al. () introduce a human-validated sub-question dataset derived from the HotpotQA dataset (Yang et al., ), undertaking a detailed investigation of the models’ capabilities to reason through sub-questions. Their findings revealed that notable models like DFGN (Xiao et al., ), DecompRC (Min et al., ), and CogQA (Ding et al., ) exhibit deficiencies in resolving sub-questions, even though they may successfully address the overarching multi-hop question.
Moreover, Wikipedia-based MHQA datasets face the challenge of data contamination that hard to objectively and truthfully evaluate the reasoning ability of LLMs. Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a major issue in measuring LLMs’ real performance on other tasks.
For example, HotpotQA (Yang et al., ), 2WikiMultihopQA (Ho et al., ), and MuSiQue (Trivedi et al., ) can be applied to evaluate the multi-step reasoning performance of LLMs. Typically, evaluating LLMs in MHQA datasets involves using RAG to retrieve and reason over context with a single step of retrieval. However, single-step retrieval can result in insufficient context retrieval for complex questions, as it provides a limited scope of information (Gao et al., ).
Prasad et al. () proposes a framework that defines good reasoning chains in Correctness and Informativeness to illustrate whether the previous reasoning step could help the current reasoning step and the final answer.

Benchmarking Data Leakage A handful of recent studies have provided several strategies, methods, and benchmarks for detecting contamination without the need to access pre-training data (Shi et al., ; Roberts et al., ; Golchin & Surdeanu, ; Zhu et al., ). Ravaut et al. () surveys recent work on detecting data contamination and releases a python library named *LLMSanitize* that implements major contamination detection methods. However, these data contamination detection benchmarks are required to dynamically update because of the development of LLMs and the expansion of pertaining data.
Dynamical maintenance is time-consuming and effortless, while our proposed benchmark CofCA, based on the knowledge edition, is fixed and maintains the cleanness of the test data. To statistically and quantitatively detect the LLMs’ data contamination extent, Xu et al. () proposes a detection pipeline by computing perplexity and N-gram accuracy to evaluate potential data leakages. Zhang et al. () designs a new Grade School Math 1000 (GSM1k) to mirror the GSM8k benchmark (Cobbe et al., ) and evaluate LLMs’ mathematical reasoning ability. Clean-Eval (Zhu et al., ), a benchmark to assess the inflated performance of LLMs.
The experimental results show a drop in performance for GPT-3.5 and Llama-2 in the Clean-Eval data, deeming the data contamination problem.
Xu et al. () proposes a detection pipeline with the help of perplexity and N-gram accuracy gap between the previous MathQA dataset and synthesized data, pinpointing the potential data leakage problem. The measurement illustrates that underscores the urgency for an evaluation paradigm shift in how we approach the development and evaluation of LLMs. To properly and objectively investigate multi-step reasoning abilities, it is crucial to disentangle LLMs’ inherent memory from their reasoning capabilities.
Hua et al. () introduces a novel benchmark for counterfactual QA task by editing six common reasoning schemes in the real world, while still lacking muti-step reasoning evaluation for LLMs. Our work is different from such works in two main aspects: 1) a comprehensive reasoning chain evaluation; and 2) Counterfactual Question Answering that reduces the risk of data contamination.

## 3 Multi-hop Reasoning Evaluation

In this section, we design a human-in-the-loop annotation framework and our novel evaluation method.
As shown in Figure LABEL:Annotation_Framework, the framework consists of three components: 1) we first ask LLM such as GPT-4 to act as a passage annotator that rewrites passages from Wikipedia with human evaluation and feedback; 2) LLM’s automatic counterfactual QA pairs annotation with human evaluation and feedback; 3) After obtaining the synthesized counterfactual MHQA data, we use counterfactual multi-hop reasoning to evaluate several strong LLMs to report the real multi-step reasoning performance.

### 3.1 Data Construction

#### Data Collection and Passage Rewriting

We randomly select 300 Wikipedia passages from Huggingface(^1^11[https://huggingface.co/datasets/lucadiliello/english_wikipedia](https://huggingface.co/datasets/lucadiliello/english_wikipedia)) as the raw context.
Inspired by recent studies on LLM’s ability to aid human annotation (Bartolo et al., ; Törnberg, ), we design a pipeline for automatically annotating Wikipedia passages into counterfactual passages. Given a raw Wikipedia passage, LLMs are required to act as a passage annotator to do the named entity, noun phrase, and synonym replacement. Since LLMs are pre-trained on the open-source corpus collected from the Internet, we manually search the key information of the annotated passage on the Internet, e.g. the characters, times, events, causes, processes, and results.

After the replacement stage, we translate the replaced text into Chinese and finally back translation into English, e.g., the words in red of the annotated passage in figure LABEL:Annotation_Framework are the replaced named entities, noun phrases, and synonyms, and the new counterfactual passage is rewritten from the original Wikipedia passage. Upon acquiring the new and counterfactual passages, human experts conduct a manual evaluation of data quality by focusing on two key aspects: 1) assessing the grammatical integrity of the annotated passages to ensure there are no grammatical errors; and 2) verifying that the annotated passages are new and counterfactual to Large Language Models (LLMs).

#### QA-pair Annotation and Checking

We use LLMs to generate new multi-hop questions to fit the rewritten passages. To make sure the generated QA pairs are correct and related to the given passages, we check the QA pairs from two perspectives: 1) Grammar issue; 2) Answerabilities of the generated question, whether the question is related to the passage and make sure the answer can be reasoned from the passages based in the generated questions.
To evaluate LLMs’ performance on different complexity of multi-hop questions, for each passage, we annotate 3 complex questions: one for 2-hop, one for 3-hop, and one for 4-hop questions. LLMs are required to generate multi-hop questions along with the corresponding sub-questions for reasoning chain evaluation. For example, at the middle part of Figure LABEL:Annotation_Framework illustrates the re-annotated context, newly generated multi-hop question, sub-questions, and intermediate answers.
The annotated answers are all with a short answer span that follows the settings of HotpotQA, 2WkimultihopQA, and MuSiQue (Yang et al., ; Ho et al., ; Trivedi et al., ). Here we use EM and F1 scores to measure the LLMs’ output.
After obtaining the rewritten passages with corresponding QA pairs and sub-qa pairs, we also check the logic of the whole passage and QA pairs. We expect that the passage is coherent and the answers can be reasoned from the passage based on the given questions. The prompts of the data annotation and more annotated examples are shown in Appendix A.

#### Dataset Analysis and Statistics

Table 1 shows the statistics of our dataset. We re-annotated 300 unique Wikipedia passages, with three multi-hop questions (one 2-hop, one 3-hop, and one 4-hop for each passage), a total of 3600 unique QA pairs including 900 multi-hop questions, and 2700 sub-questions with corresponding answers. Besides, we also randomly select 900 factual MHQA data as the control group (300 from HotpotQA, 300 from 2WikiMultihopQA, and 300 from MuSiQue). Following the settings of previous LLM evaluation benchmarks (Wang et al., ; ), we treat the total of 1800 data as the test set.
Following benchmarks such as HotpotQA (Yang et al., ), 2WikiMulthopQA (Ho et al., ), we propose a taxonomy on fine-grained question types and examples commonly used in multi-hop QA illustrated in Table 11.
In HotpotQA, 2WikimultihopQA, and MuSiQue datasets, the multi-hop questions consists of two types of questions, or the combination of the two types of questions, *Bridge* and *Comparison*: *Bridge* question is required to find the bridge entity that connects the sub-questions, while *Comparison* question is a type of question that compares two or more entities for the parallel sub-questions.
We focus on these two types of questions for annotating multi-hop questions.

#### Inter Human Agreement

Following a thorough manual inspection of the annotated data, we randomly select 300 samples, dividing them equally among 2-hop, 3-hop, and 4-hop instances, with each sample being evaluated by two expert reviewers. These reviewers are tasked with identifying and eliminating any data that significantly deviates from the established guidelines. Initially, the experts focus on assessing the grammatical correctness of the questions, sub-questions, sub-answers, and answers within the annotations. Subsequently, they verify the originality of the content in the annotated passages by checking for their presence on the Internet.

According to our statistics, the agreement rate between the annotators in the randomly selected CofCA samples is 94% and the human agreement rates are 96%, 92%, and 95% in the 2-hop, 3-hop, and 4-hop datasets, respectively.
This suggests that our synthesized instances reflect good data quality on annotation guideline following and achieve high human agreement among expert annotators.
To quantitatively illustrate data quality, we also utilize GPT-4 to assign scores to each selected data from two perspectives: correctness and informativeness. Each data is assigned with 1 or 0, which means correct or incorrect, informative or not informative. Correctness indicates whether the answer can be reasoned from the given question and context, while informativeness means whether the QA pairs and context are related or not.

Table: Table 1: Statistics of our CofCA dataset.

Table: Table 2: Differences between our CofCA with previous factual and counterfactual QA benchmarks.

### 3.2 Evaluation Metrics

Given a set of input documents, we employ three representative QA evaluation methods to assess the correctness of LLM-generated MHQA responses: sub-question answering evaluation, reasoning chain evaluation, and the joint performance of sub-qa and MHQA.

#### Sub-QA Evaluation

This part is the basis of all evaluation results. Following reading comprehension (Rajpurkar et al., ), evaluation is conducted through lexical matching using two widely used metrics to assess the performance of models. We employ F1 and EM scores to evaluate the answers to sub-questions, similar to the single-hop QA task.

#### Reasoning Chain Evaluation of Multi-hop QA

Table 2 illustrates the differences between our evaluation method and previous evaluation methods of counterfactual QA and factual MHQA datasets.
To interpret the behavior of existing LLMs on each hop of the reasoning process required for multi-hop questions, and to determine their reasoning ability to answer simple questions, we followed the experiment setting proposed by (Tang et al., ). For example, in the 2-hop dataset, each data contains a 2-hop question, 2 sub-questions, 2 intermediate answers, and a final answer. In order to understand whether LLMs can correct answers by following the right reasoning chain, we calculate the proportion of right and incorrect reasoning chains to compare LLMs’ reasoning performance.
Each question or sub-question has two results, correct or incorrect, thus an N-hop question with its N sub-questions has $2^{(N+1)}$ different reasoning chains. Due to the space limitation, we measure and collect correctness statistics for the 2-hop question dataset, $q_{sub1}$, $q_{sub2}$, and $q$, and show the percentage of 8 reasoning chains given by LLMs.

#### Joint Performance

Previous MHQA benchmarks are traditionally evaluated on the EM or F1 score on the final answer (Rajpurkar et al., ; Yang et al., ; Ho et al., ), which is partially correct. The previous MHQA systems and LLMs are treated as a black box and we can not figure out how they find the final answer. To understand the impact of sub-qa on MHQA, we introduce a joint performance that combines the evaluation of Sub-QA performance and MHQA performance. The details of computing the joint scores are shown in the Appendix B.

Table: Table 3: Performance of Proprietary LLMs and Open Source LLMs on Wikipedia-based factual multi-hop QA datasets. The performance is measured by EM and F1 scores with a zero-shot setting. PM$\dagger$ indicates the partial match of LLMs’ outputs evaluated with GPT-4-turbo with the same prompt.

Table: Table 4: Results on CofCA, the results reveal that LLMs show an obvious performance gap between previous Wikipedia-based factual data and our CofCA.

## 4 Experiments

We conduct comprehensive experiments and evaluate different LLMs, using CofCA to answer the following questions: 1) Do LLMs show a performance gap between the Wikipedia-based factual MHQA datasets and our synthesized counterfactual MHQA data? 2) When inputting counterfactual questions, how do LLMs perform in terms of their reasoning ability? 3) How do sub-questions affect the performance of LLMs? 4) How do LLMs perform on reasoning chain evaluation? These investigations aim to shed light on the capabilities and limitations of LLMs when dealing with counterfactual MHQA and multi-step reasoning tasks.

### 4.1 Experimental Settings

We evaluate LLMs on the CofCA benchmark, including 900 randomly selected data from Wikipedia-based MHQA datasets (300 QA pairs from HotpotQA (Yang et al., ), 300 QA pairs from 2WikiMultihopQA(Ho et al., ), 300 QA pairs from MuSiQue (Trivedi et al., )), and 900 annotated counterfactual MHQA data (divided into 2-hop, 3-hop, and 4-hop subsets).
We employ proprietary LLMs and open-source LLMs in experiments. To enhance reproducibility, we set the temperature to 0 for proprietary models, and all the experiment results are the average scores of three experiment results.
We adopt the proprietary LLMs: GPT-4 (Achiam et al., ), GPT-3.5 (Ouyang et al., ), text-davinci-003, Bing Chat, GEMINI-pro (Team et al., ), and Open Source LLMs such as Llama 2-7b, Mistral-7b and Qwen-7b as the baselines. To decouple LLMs’ internal memory and reasoning ability, and let LLMs retrieve answers from the given passage as much as possible, we design a prompt that requires LLMs to only retrieve answers based on the given context.
The prompt of QA is shown in the Appendix A.

### 4.2 Results

Figure: Figure 3: The performance change of F1 score and EM scores when answering 2 sub-questions on the 2-hop dataset.
Refer to caption: extracted/5927436/Figures/G3.png

#### Reasoning VS Memorization

The results of the comparison between the selected Wikipedia-based MHQA data and our CofCA can be found in Table 3 and Table 4.
LLMs show a performance gap between the selected data and ours. Taking GPT-4 as an example, GPT-4 achieves high EM and F1 scores (69.9 and 82.3, respectively), which are even close to well-finetuned small QA models. While for our 2-hop dataset, EM and F1 scores are sharply declined (53.1 in table 3 and 62.8 in table 4). For 3-hop and 4-hop datasets, GPT-4 even performs worse. Since our synthesized data is new, unprecedented knowledge, our results objectively reflect the real reasoning performance of LLMs.
There is also a concern that EM may not be a dependable method due to its constraints in accurately representing real performance, particularly in scenarios where answers involve aliases or abbreviations. For example, the gold answer is "Lionel Messi" and the generated answer is "Messi". The two results should be treated the same but the EM score is not applicable in this scenario. As a result, we introduce GPT-4-turbo as the answer evaluator to measure the partial match (PM) between LLMs’ generated answers and gold answers The prompts are shown in appendix A. The PM scores in table 3 and table 4 also reveal that LLMs are better at answering factual MHQA questions than counterfactual questions.
In light of the results, we can find that LLMs achieve an inflated high performance on the Wikipedia-based MHQA dataset possibly because of the data contamination that leads to utilizing LLMs’ memory ability rather than reasoning ability.

#### Impact of Annotation on LLMs

Table: Table 5: Impact on LLMs’ performance at different stages in the annotation framework.

Our annotation framework involves two stages: keywords replacement followed by paraphrasing. To demonstrate the impact of annotated passages in different stages, three experiments were conducted using the original passages, passages generated from the first stage, and passages generated from the second stage. We additionally select and annotate another 300 passages from the HotpotQA dataset. Additionally, to ensure consistency in question complexity, we only annotated 2-hop questions on both sets of passages, aligning with the HotpotQA dataset.

Results are illustrated in Table 5, we find that: 1) there is also an obvious performance gap of LLMs between the original HotpotQA data and our two-stage data. Taking GPT-4 as an example, GPT-4 performs better on HotpotQA than our two-stage data, EM score of 71.35 and F1 score of 78.62 respectively. In the 1st stage data, GPT-4 shows a significant decrease, 65.62 EM and 70.47 F1 respectively. 2) The two stages of bias bring the different extents of performance drop. Comparing the performance of GPT-4 in stages 1 and 2, there is also an obvious drop, from 65.62 to 63.49 and from 70.47 to 67.32. It is mainly because of the data produced from the first stage although some keywords such as named entities and noun phrases, other words, sentence structure, and internal logic are the same as the original passages. The second stage passages are totally different from the original passages.
Table 10 also illustrates an example from HotpotQA that is annotated in different stages. We find that the passage in stage 1 still maintains the same sentence structure. While the passage in stage 2 is totally different from the original passage.

#### Sub-QA Evaluation

Figure 3 illustrates the performance degradation because LLMs also suffer from error propagation with the reasoning depth increasing. Figure 4 shows the performance of LLMs on the different hops of questions. According to the observation of the three figures, we find that with the hop increases, the complexity of multi-hop questions also increases, leading to the LLMs’ performance decrease.
When incorrectly answering the previous sub-question, the latter one will also be influenced. Consequently, the performance of Sub_Question 2 is worse than that of Sub_Question 1. Tables 13 and 14 also illustrate the sub-qa performance of LLMs on the 3-hop and 4-hop datasets in appendix C.

Table: Table 6: Ablation study of the MHQA task, where we remove the sub-question information from the prompt and only ask LLMs to give the final answer. Here all the prompts are in a Chain-of-Thought setting.

#### Reasoning Chain Evaluation

We calculate the proportion of the reasoning chain on the 2-hop dataset and follow the settings of (Tang et al., ) in calculating the percentage of correct or incorrect answers and record the results.

Table 8 shows the reasoning chain evaluation results. The green row shows the percentage of examples whose multi-hop questions can be correctly answered from the right reasoning chain. The red rows show the percentage of examples whose multi-hop questions can be correctly answered but through an incorrect reasoning chain. Among these examples, we notice that there is a low percentage of the correct final answer based on the right reasoning chain. There is also a large proportion of incorrect final answers as shown in rows 2,4,6 and 8.

Taking the results of GPT-3.5 as an example, the right reasoning chain only accounts for 13.3% although it shows a relatively high QA performance in previous tables. The percentage of incorrect reasoning chain of GPT-3.5 is 17.7% (sum of the three red rows). However, total failure cases account for 69% (sum of rows 2, 4, 6, and 8) which is substantial for the whole dataset.
We conclude that LLMs only get a small proportion of the right reasoning chain and their high performance is relatively inflated due to the considerable proportion of incorrect reasoning chain.

Table: Table 7: LLMs’ joint performance on the whole reasoning chain. The scores are the average scores of three experiment results. The larger score means a worse performance on the whole reasoning chain.

#### Joint Performance

The joint F1 $RC$ and joint EM $RC$ scores in Table 7 are the whole reasoning chain evaluation results. We find that with the increases in the reasoning chain, the performances of LLMs dropped swiftly. For example, Bing Chat gets comparable performance with GPT-4 (0.7 joint F1) on answering 2 hop questions and gets a 0.9 joint F1 score. However, in the 3-hop question, the joint F1 $RC$ and joint EM $RC$ scores of Bing Chat are 4.2 and 8.4. In the 4-hop dataset, Bing Chat gives 4.7 joint F1 $RC$, and 8.9 joint EM $RC$ scores, respectively. Since the joint performance is a negative log, the larger scores mean the worse performance on the reasoning chain. We can conclude that LLMs’ reasoning ability decreases with the increases in reasoning chain length.

Table: Table 8: Categorical EM statistics (%) of sub-question evaluation for the five LLMs on our 2-hop dataset.c stands for correct and w stands for wrong. For example, the third row shows the percentage of questions where models correctly answer both 2-hop questions and the first sub-question but incorrectly answer the second sub-question. We abbreviate text-davinci-003 as text.

### 4.3 Impact of Sub-Question

To evaluate the impact of sub-questions for LLMs, we conduct an ablation study testing the performance of answering the final answer and removing the sub-questions from prompts. All LLMs are combined with Chain-of-Thought (Wei et al., ) settings, in which LLMs incrementally derive a series of intermediate steps. The results, shown in Table 6, indicate that when directly asking LLM a multi-hop question and corresponding passage, the performance is much lower than that of adding sub-questions to require LLMs reasoning step-by-step. For example, computed from table 6 the performance of GPT-4 on the 2-hop dataset decreased the F1 score and EM by 2.5 and 9.4 respectively. The results show the sub-questions could help LLMs improve the performance of final-QA.

## 5 Error Analysis

We select a total of 20 incorrect final answers generated by GPT-4 from the 4-hop dataset to comprehensively illustrate how LLMs make decision on multi-step reasoning tasks. We first verify the proportion of each incorrect sub-answer and final answer. Among the 20 incorrect final answers, 9 of them are wrongly answered in the first sub-questions which leads to the incorrect final answers.
While for the remaining 11, 5 of them are incorrect second sub-questions that lead to the wrong final answers. The rest of 4 are influenced by the wrong third answers and fourth answers.
From this analysis, we estimate that roughly half of the incorrect final answers are incorrectly reasoning from scratch.
We select 20 correct final answers generated by GPT-4 and find that about 4 of them are reasoned from incorrect reasoning chains (wrong sub-answers), revealing that LLMs also sometimes bypass the incorrect reasoning chain and get correct final answers. The insights of the whole experiment results are illustrated in Appendix D.

## 6 Conclusion

We presented a novel evaluation framework CoFCA for evaluating LLMs’ evidence integration and multi-step reasoning capabilities by combining sub-question evaluation and counterfactual data annotation. To disentangle LLMs’ memory and reasoning ability, we design a human-in-the-loop framework to synthesize counterfactual data. Our results show that, although LLMs performed relatively well on QA tasks, the performance dropped on multi-hop questions that were based on new, counterfactual knowledge. In addition, their high performances are inflated and benefit from a high proportion of incorrect reasoning chains. Our work can facilitate future research on developing faithful knowledge editing methods.

## 7 Reproducibility Statement

To make the results and models reproducible and verifiable, we provide our full data annotation guideline, data link, implementation details, and prompts: We detail the process of data annotation in section 3.1 and the implementations are in Appendix E. All the prompts required to reproduce the results are illustrated in Appendix A.

## Appendix A Prompts and Examples

When evaluating large language models, prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort should be dedicated to designing a painstakingly crafted perfect prompt for the given task (Arora et al., ; Diao et al., ). In this study, We investigate the performance of zero-shot on our benchmark. To eliminate the randomness, we manually select one demonstration for each task, ensuring that all tasks are covered.

We give our designed input examples for three different tasks to help readers understand our implementation, as shown in Table 9, respectively.
The original and rewritten passages are shown in the table 12.

The annotated multi-hop questions are shown in the table 11.

**Table 9: The prompt template of passage rewriting and question generation. We here take 2-hop data annotation as the example. [WORDS] denotes the information we should give.**
| Prompts of NER, Noun Phrase and Adjective replacement |
| --- |
| Prompt Now you are a passage annotator, you need to recognize all the named entities, noun phrases, and adjectives from the given [CONTEXT], then translate the passage into Chinese and translate to English. Please output the response in JSON format {Passage: String}<br>[CONTEXT] The given context. |
| Prompts of Question Generation |
| Example One-shot example with multi-hop QA pairs, Sub-QA pairs, and passage.<br>Prompt Now you are a multi-hop question generation machine, given an example of 2 hop question and its sub-questions, sub-answers, and final answer is [2 hop question],[Sub-Questions],[Sub-Answers] and [Final Answer], you need to generate a new 2 hop multi-hop question same with the given example and its sub-questions, sub-answers and final answer from the given [Context]. Please follow the sentence structure of give examples and output the response in JSON format {2 hop question: String, sub-questions: List, sub-answers:List, final answer:String}:<br>[2 hop question] The given example of 2 hop question. |
| [Sub-Questions] The given example of sub-questions. |
| [Sub-Answers] The given example of sub-answers. |
| [Final Answer] The given example of final answer. |
| [CONTEXT] The given passage |
| Prompts of QA |
| Prompt You are a QA test machine, you need to answer the [Question] from given the [Context], and you only need to come out with the correct answer without other words. Let’s think step by step, and please output the answer to the [Question] in the format of: {Final Answer: String}.<br>[QUESTION] The given question.<br>[CONTEXT] The given passage. |
| Prompts of Partial Match Evalutaion |
| Prompt You are an Answer evaluator, you need to measure the semantic similarity between [Generated Answer] and [Gold Answer], and give the score, 1 means equal, 0 means not. Some answers may have abbreviations or alias, for example, Lionel Messi is equal to Messi, Donald Trump is equal to Trump. Please only output the score 1 or 0 without any other words.<br>[Generated Answer] The LLM generated answer.<br>[Gold Answer] The ground truth. |
| Table 9: The prompt template of passage rewriting and question generation. We here take 2-hop data annotation as the example. [WORDS] denotes the information we should give. |

Table: Table 10: Examples of a passage annotated in different stages. The words in red indicate the revision part. Passage in stage 1 still has the same sentence structure with the original passage while passage in stage 2 is totally different.

Table: Table 11: Examples of annotated different question types and question hops. We emphasize keywords for their respective categories.

Table: Table 12: The original passages with the rewritten passages. The first and third rows of the table are the original passages, and the second and fourth rows show the corresponding rewritten passages.

## Appendix B Joint Computation

We here list the details of computing the joint scores on the whole reasoning chain: For example, a N-hop question and its N sub-questions, given their precisions and recalls on the MHQA ($P^{\text{(MHQA)}},R^{\text{(MHQA)}}$) and the Sub-QA ($P^{\text{(sub_$qa^{1}$)}},R^{\text{(sub_$qa^{1}$)}}$), … ($P^{\text{(sub_$qa^{N}$)}},R^{\text{(sub_$qa^{N}$)}}$), respectively, we calculate joint performance as:

$$ $\displaystyle P^{\text{(joint)}}=P^{\text{(MHQA)}}P^{\text{(sub_$qa^{1}$)}}.. .P^{\text{(sub_$qa^{N}$)}},$ $\displaystyle~{}~{}R^{\text{(joint)}}=R^{\text{(MHQA)}}R^{\text{(sub_$qa^{1}$ )}}...R^{\text{(sub_$qa^{N}$)}},$ $$

$$ $\displaystyle\text{Joint F${}_{1}$ $RC$}=-\log\frac{2P^{\text{(joint)}}R^{ \text{(joint)}}}{P^{\text{(joint)}}+R^{\text{(joint)}}}.$ $$

where the Joint F1 $RC$ means the joint F1 performance of the reasoning chain.

Given their EM scores on the MHQA ($EM^{\text{(MHQA)}}$) and the Sub-QA $EM^{\text{(sub_$qa^{1}$)}}$), … $EM^{\text{(sub_$qa^{N}$)}}$.

$$ $\displaystyle\text{Joint EM $RC$}=-\log\frac{2EM^{\text{(MHQA)}},...EM^{\text{ (sub_$qa^{N}$)}}}{EM^{\text{(MHQA)}}+,...EM^{\text{(sub_$qa^{N}$)}}}.$ $$

where the Joint EM $RC$ means the joint EM performance of the reasoning chain.

## Appendix C Performance Analysis

Table: Table 13: The LLM evaluation on CofCA 3 hop dataset. We here measure the sub-qa task and compare the performance between each hop. $Q_{i}$ means the $ith$ sub-questions.

Table: Table 14: The LLM performance on CofCA 4 hop dataset. We here measure the sub-qa task and compare the performance between each hop.

Figure: Figure 4: The performance change of EM and F1 scores when answering from 2 hop questions to 4 hop questions.
Refer to caption: extracted/5927436/Figures/G1.png

Figure: Figure 5: Performance gap between Wikipedia-based factual multi-hop QA datasets and our 2-hop, 3-hop, and 4-hop counterfactual MHQA data of table 3 and table 4. The line charts reveal that LLMs show an obvious performance gap between previous datasets and CofCA.
Refer to caption: extracted/5927436/Figures/EM.png

As the quantitative complementary of Sub-QA evaluation, we here list the results of LLMs’ performance on 3-hop and 4-hop datasets. The LLMs’ reasoning performance dropped dramatically, e.g. in table 13, GPT-4 achieves 70.9 EM and 80.8 F1 scores on sub-question1 but only gets 59.7 EM, 74.9 F1, and 58.1 EM, 68.8 F1 scores on sub-question2 and sub-question3 respectively.
In table 14, we further find that when answering 4 hop questions, the results show a cliff-like descent from sub-question2 to sub-question3, especially GPT-3.5 gets 46.9 F1 in sub-question2 but drop to 36.3 F1 score in sub-question3.

## Appendix D Insights of LLM Multi-step Reasoning Evaluation

Drawing from the above experimental results, we draw the conclusions:

#### Exact Matching

While exact matching is a simple and effective method for MHQA evaluation, it struggles with issues when the answers have abbreviations or other expressions. For example, in our synthesized counterfactual MHQA data, if the golden answer to the question "When did Africa’s second public FM radio station launch?" is "2002" and the generated answer of GPT-4 is "4 August 2002", the exact match can not be computed accurately. All the answers generated by LLMs have this issue. Thus, It is urgent to propose a more universal QA evaluation score in LLMs’ reasoning performance evaluation.

#### Multi-step Reasoning

Although the experiment results demonstrate that LLMs can perform multi-step reasoning ability to a certain extent, they remain sensitive to prompts and the impact of additional contexts, especially the sub-questions. Providing sub-questions as additional information into prompts can help guide LLMs to reason in the correct direction and show a relatively strong performance.

#### Reasoning chain Evaluation (Joint F1 RC and Joint EM RC in this work)

The advantage of our reasoning evaluation method is we jointly consider the sub-QA performance and final-QA performance when LLMs bypass the incorrect reasoning chain and achieve a correct final answer, the scores remain very low. However, this evaluation method is easily influenced by the LLMs’ performance on the first sub-questions, since the answer order is sequential. if the first sub-question is incorrectly answered, the following sub-questions and the final question will also be influenced, leading to a very low score. How to answer sub-questions more correctly remains exploration.

## Appendix E Implementation Details

For proprietary models, we employ official APIs to interact with exclusive LLMs and prompts are well-defined. For open-source models, all experiments are conducted on 8 A100 GPUs.

## Appendix F Limiatations

In this paper, we focus on the evaluation of LLMs’ real multi-step reasoning ability on our annotated counterfactual MHQA data. Although LLMs show an obvious performance gap between previous factual MHQA datasets and our dataset, the data size of our dataset still remains improved. The Exact Match (EM) for reporting QA performance still faces challenges, because EM does not report LLMs’ real performance due to the variation in the expression of the answers.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. CofCA could be a valueable resource for studying LLM’s true reasoning ability. Due to its counterfactual nature, the LLMs can no longer use their internal knowledge for finding reasoning shortcut, hence the model have to reason over provided context. Also, the annotation of intermediate sub question answer pairs can also enable examination of the reasoning process. 

2. The experiments and analysis are very thorough. The results show a clear trend of performance vs question complexity and the analysis on the reasoning process further exposes the weaknesses of the SOTA LLMs.
summary: This paper proposed a new dataset, CofCA for evaluating LLM’s counterfactual multi-hop reasoning ability, which focuses on both final answer evaluation and reasoning step evaluation. The dataset construction involves 2 steps, 1) the sampled Wikipedia paragraphs are sent to GPT-4 for entity replacement and paraphrase, and then back-translation is done to further modify the paragraphsm and then these paragraphs are manually checked 2) GPT-4 is then used to generate QA pairs based on the newly generate paragraphs. Here GPT-4 is asked to generate both overall multi-hop QA pair and subquestion answer pairs, and the resulting data is also manually verified. 
The authors then conducted experiments on both newly generated data and HotpotQA, 2WikiHop and Musique, using both closed-sourced models and open-source models. The results show that all LLMs suffer significant performance drop on counterfactual questions and the joint performance which considers both final answer and reasoning process correctness can be even worse, exposing the weakness of current LLMs.
weaknesses: My main concern is the scale of the passages set when constructing the data. For previous datasets such as. HotpotQA and 2WikiHop, they usually construct questions using multiple passages to ensure that the answer have to be deducted from multiple pieces of evidence. However, it seems that for CofCA, the questions are generate using only a single passage? If this is the case, how can you make sure the generated multi-hop QA pairs are fully grounded to the passage?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses, mainly:
- Does the counterfactual information contradicts real-world facts? If so, isn't some degradation in performance expected? 
- How often are shortcuts in the collected examples? 
- What qualitative analysis did you consider that might strengthen the main claims of the paper?
soundness: 3
strengths: - A counterfactual multi-hop benchmark can be helpful to understand the ability of LLMs to perform multi-hop reasoning. The benchmark will be publicly available in addition to the annotation guidelines and implementation details.

- The paper features extensive quantitative analysis examining the effect of the annotation pipeline and the performance of sub-questions amongst others.
summary: The paper introduces CofCA, an evaluation benchmark for multi-hop reasoning abilities of LLMs over counterfactual data. CofCA is created by (1) LLMs alter Wikipedia passages by replacing keywords with counterfactual information, and (2) LLMs generate multi-hop questions over the modified passages. Both steps are manually verified by human annotators, and the benchmark includes 900 multi-hop questions from 300 passages. Experimental results show a gap between performance on the counterfactual CofCA and existing multi-hop QA datasets. Additionally, the paper shows that the low performance is due to reasoning failures.
weaknesses: - Given the examples presented in the paper, I was not fully convinced about several aspects of the collected data. Mainly, how do you ensure that the counterfactual information does not contradict real-world facts? Also, both examples in the main paper seem to have shortcuts, e.g., there is only one position in the example in Fig.1, and only one designer in the example in Fig. 2.

- Qualitative analysis - the analysis presented in Section 5 did not fully address the concerns above, and is limited to 20 correct/incorrect questions with GPT-4. I think the paper could benefit from a thorough analysis regarding the quality of the generated data examining contradiction and shortcuts and verifying that the main claims of the paper (multi-hop QA is more challenging on counterfactual information) is supported. For example, it could have been interesting to have an error analysis that compares different error categories between factual and counterfactual questions.

- Presentation - the presentation in several parts of the paper was a bit hard to follow. For example, in Fig.1, the text is very small and there is an error (the bottom-right box the question is about the coach and the answer is Point Guard). In Tab.2, the ‘Whole data property’ is not the sum of the other columns (I assume this is due to the factual questions sampled from other datasets?). The main results in Section 4.2 require comparing between information presented in different tables (Tables 3 and 4). Table 8 is referenced before Table 7 but presented afterwards.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you provide a more intuitive explanation for the **JOINT COMPUTATION** metric mentioned in Appendix B? An example or simplified breakdown might help readers understand its significance and how it is calculated in practical terms. This would not only make technical details more accessible but also allow readers to better appreciate its role in evaluating model performance.
soundness: 3
strengths: 1. The paper is well-written and easy to follow, presenting the content in a clear and accessible manner.

2. The authors demonstrate an awareness of the limitations of using Exact Match (EM) and F1 scores when evaluating the output of large language models (LLMs). To address this, they introduce an additional Partial Match metric, which helps mitigate the issue and provides a more nuanced evaluation.
summary: This paper introduces a new dataset called CofCA, designed to effectively assess LLMs' capability in counterfactual multi-hop question answering. The authors generate the dataset by starting with Wikipedia passages and applying techniques such as entity replacement and back-translation to create question-answer pairs. The models are then evaluated at both the final-answer level and the sub-question level. To ensure the quality of the dataset, the authors perform thorough quality checks and include human-in-the-loop validation throughout the dataset construction process.
weaknesses: 1. One major drawback of the paper is that, while it claims to assess LLMs' true reasoning abilities, it primarily reports the accuracy of different sub-questions without fully addressing whether the models are capable of reasoning holistically. For a more comprehensive evaluation of reasoning abilities, other factors should also be considered, such as whether the model is able to effectively decompose the problem into meaningful or correct sub-questions, rather than treating them as isolated tasks.

2. The quality of the dataset is also in question. Currently, the correctness and informativeness of the dataset are evaluated using GPT-4. However, as the paper itself shows, GPT-4 struggles to correctly answer these questions, which suggests that it may not be fully reliable for assessing the quality of the dataset. This raises concerns about the robustness of the dataset validation process.

3. There is further concern about ensuring that the generated questions are non-trivial. As shown in Table 9, the prompt generates the multi-hop questions and the sub-questions, along with their answers, simultaneously. This raises a natural concern that the generated multi-hop questions may either be independent of these sub-questions or follow a single pattern (e.g., q1->q2->q3), leading to a lack of diversity in question structure. The paper does not address or discuss this potential limitation.

4. When benchmarking models, the paper lacks evaluation on more recent and advanced models, such as GPT-4 Turbo, GPT-4O, and Claude series, as well as newer open-source models like LLaMA 3 or Deepseek. Including these models would provide a more up-to-date and comprehensive analysis of the current state-of-the-art in LLMs.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: As given in the weakness.
soundness: 3
strengths: 1. CofCA minimizes data contamination by creating a cleaner benchmark for evaluating LLM reasoning.

2. The diverse counterfactual and factual datasets allow researchers to observe performance discrepancies, adding depth to LLM evaluation.

3. The constructed dataset is clearly different from existing counterfactual QA benchmarks, including distentQA and IfQA. The multi-hop challenge lacks of enough study in counterfactual setting.
summary: The paper is designed to evaluate large language models (LLMs) on multi-hop counterfactual reasoning abilities. CofCA generates counterfactual contexts that LLMs likely haven’t encountered during pre-training, and ask questions based on the counterfactual contexts.  It also emphasizes sub-question evaluation for each reasoning step. 

The evaluation in this paper reveals that models like GPT-4 show inflated scores in standard QA due to these contamination issues, while performance drops on CofCA due to its reasoning on counterfactual contexts, highlighting the gap between LLM memory and genuine reasoning.
weaknesses: 1. More open-source LLMs should be evaluated on the benchmark, like llama-3 or 13B-level models.

2. OpenAI o1 was evaluated in the paper, it shows using long reasoning-chains could help a lot on this reasoning tasks, and more insights on comparing o1 reasoning with 4o reasoning could be insightful.

Meta-review: There is a discrepancy among the review scores. Yet it is worthy to note that the two reviewers who gave 5 did not participant in the after-rebuttal discussion.

I have personally read this paper and find it to be a valuable contribution. It tackles an important challenge in our field: how to disentangle the evaluation of LLMs’ reasoning abilities from their reliance on internal memory. The dataset construction process proposed in the paper is well-conceived, and the experimental results support its validity. Despite the evaluation being limited to specific scenarios and some unresolved issues raised by the reviewers, I believe the paper makes meaningful progress toward the long-standing goal of evaluating pre-trained models effectively.

That said, I concur with the reviewers on several notable weaknesses that remain unresolved:

- The paper lacks additional insights, particularly regarding the decomposition of questions.
- There is insufficient clarity on how the counterfactual information is ensured to not contradict real-world facts.

Furthermore, the absence of human performance benchmarks in the results table is a notable omission. Including a comparison involving human participants unfamiliar with the dataset construction would provide a more comprehensive evaluation.

Decision: Accept (Poster)