Counterfactual Reasoning for Retrieval-Augmented Generation
Authors: Huaiyu Qin
Chunyu Wei
Yueguo Chen
Yunhai Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Retrieval-Augmented Generation
Large Language Models
Counterfactual Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: While Retrieval-Augmented Generation (RAG) has advanced knowledge-intensive tasks, we identify a fundamental vulnerability: the Correlation Trap. Existing systems cannot distinguish causally decisive evidence from overwhelmingly correlated yet misleading information, leading to systematic failures. We introduce Counterfactual RAG (CF-RAG), a new framework that operationalizes causal reasoning to overcome this limitation. CF-RAG systematically generates and evaluates counterfactual queries to identify causally relevant distinctions, and employs a parallel arbitration mechanism to reconcile conflicting evidence without interference. On challenging benchmarks, CF-RAG substantially improves robustness against the Correlation Trap, achieving state-of-the-art performance while maintaining comparable efficiency to standard RAG models.

Published as a conference paper at ICLR 2026
COUNTERFACTUAL REASONING FOR RETRIEVAL- AUGMENTED GENERATION
Huaiyu Qin, Chunyu Wei; Yueguo Chen, Yunhai Wang Renmin University of China Beijing, China
we weichunyue ruc. edu.cn
ABSTRACT
While Retrieval-Augmented Generation (RAG) has advanced knowledge- intensive tasks, we identify a fundamental vulnerability: the Correlation Trap. Existing systems cannot distinguish causally decisive evidence from overwhelm- ingly correlated yet misleading information, leading to systematic failures. We introduce Counterfactual RAG (CF-RAG), a new framework that operationalizes causal reasoning to overcome this limitation. CF-RAG systematically generates and evaluates counterfactual queries identify causally relevant distinctions, and employs parallel arbitration mechanism to reconcile conflicting evidence with- out interference. On challenging benchmarks, CF-RAG substantially improves robustness against the Correlation Trap, achieving state-of-the-ar performance while maintaining comparable efficiency to standard RAG models]
INTRODUCTION
Serrieval-Augmented Generation (RAG) has revolution- ized how Large Language Models (LLMss) access exter- nal knowledge, dramatically improving factual accuracy and reducing hallucinations Lewis et al. (2020); Guu eet al. (2020); [Borgeaud et al. (2022). By grounding re- sponses in retrieved documents, RAG systems have be- come indispensable for knowledge-intentt intensive tasks rang- ing from question answering to dialogue systems Izacard
Who is the lead actor in The Dark Knight?
Retrieve
revicw articie:
Ladger's
portrayal Enight considened one the best performances film history.
Wayne/Batman Dark Knight.
6..
Intervies transcript with Cheistopher Nolon: incredible actor Batmon, brought intensity.
& Grave [202T); Karpukhin et al. (2020). Recent ad- vances have introduced self-reflection mechanisms Asai
Online discussion foru::
Question: Who were the main
actors in The Dark Kright?
Response: Obviously Christian
al. (2023), iterative refinement Yan al. (2024). ) and parallel reasoning paths Waan et al. (2024c). pushing performance boundaries on standard benchmarks.
Bale and Heafth Ledger.
Blog: While Christian Bale was the titulor here, the true stor The Dark Knight was
undoubtedly Heath Ledger.
Yet these systems harbor a fundamental vulnerability: they cannot distinguish between evidence that causally determines answer and distractors that are merely cor- related. Consider the query "Who the lead actor The Dark Knight?" Conventional RAG systems retrieve documents praising Heath Ledger's 's Oscar- winning por- trayal of the Joker, which creates overwhelming corre- lational signals that mislead models into identifying him as the lead, rather than Christian Bale (Figure [] We term this systematic failure the Correlation Trap.
Overwhelming Cerrelational Signols
Wrong Answer: Heath Ledger is the lead actor.
Oscar-winning
The Correlation Trap. strong correlational signals that overshadow the causal evidence identifying Christian Bale as the protagonist.
Figure 1:
This vulnerability reveals deeper limitation: existing RAG systems lack causal reasoning about why evidence supports specific answers. While corselation-based retrieval excels at finding topically relevant documents, fails when correctness depends on understanding causal relationships, such
"Corresponding author: beiichnnyu@ruc.edu.co 'The code is available http://gghhbb.omm//F44AAcCCCC

---

Published as a conference paper at ICLR 2026
distinguishing protagonists from antagonists, primary authors from contributors, or lead actors from supporting cast. The challenge is not merely retrieving more documents but fundamentally rethinking how systems reason about evidence.
Bridging the gap between correlation and causation in RAG faces two critical obstacles:
Causal Distinction:
Which conceptual boundaries matter for given query? For our example, the cootaonnst-antagoone distinction causally decisive, while the
actor dis- tinction is incidental. Without supervision, systems must autonomously discover these critical boundaries from an exponentially large space of possible distinctions, which a challenge cur- rent methods cannot address.
How should systems handle conflicting signals in retrieved evidence? When documents simultaneously praise Bale's lead performance and Ledger's scene-stealing por- trayal, joint processing causes interference (stronger correlational signals overshadow weaker causal ones), while separate processing loses comparative context. The system must maintain distinct reasoning paths while enabling meaningful cross-validation.
Evidence Reconciliation:
We propose CF-RAG (Counterfactual RAG), a framework that transforms correlation- based re- trieval into causation-dis driven reasoning through systematic counterfactual testing. Our key insight: causal evidence uniquely supports the original query over plausible alternatives, while correlational evidence fails this discrimination test. CF-RAG introduces two synergistic mechanisms:
Counterfactual Exploration systematically generates alternative hypotheses that probe conceptual boundaries. For our example query, CF-RAG explores role variations ("Who played the villain?"), categorical shifts ("Who directed the film?"), and temporal boundaries ("Who starred in Batman Be- gins?"). By retrieving evidence for both original and counterfactual queries, we create dialectical evidence space revealing which distinctions are causally decisive.
Parallel Arbitration maintains multiple reasoning paths while enabling rigorous comparison. CF- RAG constructs diverse evidence subsets through stratified sampling. generates parallel answer drafts, and evaluates them using dual scoring: internal coherence (alignment with supporting ev- idence) and causal discrinination (whether evidence uniquely supports the original over counter- factuals). This architecture prevents interference from conficting signals while identifying evidence that makes answers necessary rather than merely plausible.
Our contributions are threefold:
1. We formalize causal reasoning in RAG through counterfactual testing, providing principled meth- ods for distinguishing causation from correlation in evidence evaluation.
2. CF-RAG operationalizes causal reasoning through counterfactual exploration and parallel arbi- tration, maintaining computational tractability while achieving robust performance.
3. We achieve state-of-the-art results challenging benchmarks, with substantial improvements in robustness against misleading correlations while maintaining efficiency comparable to baselines.
RELATED WORK
Retrieval-Augmennte Generation. RAG has become the dominant paradigm for grounding LLM responses in external knowledge, substantially reducing hallucinations (Lewis et al., 2020 Gao
al. 2023 Sharma| 2025 Khandelwal et al. 2020). Recent advances target different aspects of
the retrieval-geneeee -eneraatenettinnn pipeline: adaptive retrieval methods learn when and what to retrieve (Jiang
al. 2023b Ma et al., 2023; Chen al. 2023 Schick et al., (024), context utilization improve-
ments include of-hooot- retrieval Yu et al. 2023), hierarchical organization (Sarthi et al.)
2024), and compression techniques (Xu et al.) 2023 Kim et al.) 2024 Yoran et al.) 20233 Wang et al. 2023) Back et al., 2023). Robustness mechanisms address noisy retrieved content through
various strategies. Self-RAG introduces reflection tokens to critique passages (Asai et al. 2024)
CRAG employs lightweight evaluators for filtering (Yan et al. (024), RAFT trains models to ig- nore distractors (Zhang et al. 2024), and SAIL (Luo et al. (003) filters irrelevant content using web-search-tuned models. While these methods improve resilience to noise (Shi al. 2023 Liu
jet al., 2024) and explore architectural innovations (Xia et al. 2024 Wang et al. 20244)) they fun- damentally rely correlational signals, optimizing semantic relevance without modeling causal
2

---

Published as a conference paper at ICLR 2026
Original Query
Diolectical Evidence Space
Diverse Hypothesis
Subset
Christian Sale is lead
Who load octer Dark Knight
Evidence identifying
"Christopher Nolon"...
protagonist,
Semontic Perturbationa of Tr(u)
refutes counterfactual claims
How did "Heath Ledger" play the role of "Toker" well...
identifying Heath Ledger as (...)
Counterfactual Query Collection Qf
((nr)) -il. Gg)
lead
including actors such as "Heath Ledger"...
complete
'Botman Begins'
Supporting Rattonale
Whh
Role variation
M
Semple( (KGal-el)
Causal Arbitration
The Dark Knight
Thematic Cluster
Internal Ceherence (,,
Finell Answer
Enhity substitutions
Diverse Perspective
Semantic Deduplication
proising Bale's protoganist role...
Funal (e fucs del & fesel (c).
Ey2/ the lead in Batman Begins?
Ranlo))
Candl Discntatiation
sim(g.4.)
Alaf)
Returddrryy
Spectral Clustering
f 2ay)
COUNTERFACTUAL EXPLORATION
PARALLEL ARBITRATION
Figure 2: CF-RAG Framework Overview. (1) Counterfactual Exploration, which generates counterfactual queries build dialectical evidence space, and (2) Parallel Arbitration, which constructs and evaluates multiple hypotheses in parallel to identify the causally supported answer.
relationships between evidence and answers. CF-RAG addresses this gap by introducing causal reasoning directly into evidence selection through counterfactual testing.
Counterfactual Reasoning in NLP. Counterfactual reasoning has gained traction for improving model robustness and interpretability (Wang al., 2224aa (Singh et al.] (2024). Counterfactual ex- planation methods generate minimally-edited inputs that flip model predictions, revealing decision boundaries. Polyjuice enables controlled perturbations (Wu et al. (021) while FIZLE achieves zero- shot generation (Bhattacharjee al. 2024 McAleese & Keane 2024). Counterfactual data aug- mentation enhances training robustness: Kaushik et al. (2020) demonstrated that human -eenanegeee counterfactuals improve ant-of-distribution generalization. with subsequent work automating gen- eration at scale (Balashankar et al., 2023). However, these approaches treat counterfactuals as either post-hoc analysis tools offline training resources. CF-RAG pioneers integrating generation and evaluation directly into the RAG inference pipeline for real-time causal verification.
PRELIMINARIES
Definition (Retrieval-Augmentee Generation) Given a query space Q, answer space A, docu- ment corpus D {dj..... ..., , ] and retrieval function R Q 2D, standard RAG seeks: a" = arg max.e. P(a | R(g, D)), where P(a q, R(g, D)) represents the posterior probability of answer a conditioned on query q and retrieved evidence.
This formulation suffers from the correlation trap: when spurious correlations dominate retrieved evidence, systems conflate co-occurrence with causal support, motivating our causal reformulation:
Definition (Causal RAG) A causal RAG system enforces counterfactual robustness by requiring that generated answers satisfy discriminative causality. For query and counterfactual set Oif C Q:
arg max P(a. q. D) [Ag E Qi:: D(a, D) (D). uEA
D R quantifies causal support, which is the evidence that makes answer necessary for query q rather than merely consistent with it.
where & A Q
METHODOLOGY
CF-RAG transforms correlation-batt retrieval into causation-driven reasoning through two syn- ergistic mechanisms that directly address the challenges identified in introduction: Counter- factual Exploration for discovering causal boundaries, and Parallel Arbitration for reconciling conflicting evidence. The architecture of CF-RAG is illustrated in Figure
3

---

Published as a conference paper at ICLR 2026
4.1 COUNTERFACTUAL EXPLORATION
Counterfactual Exploration addresses the challenge of causal distinction: how a system can identify which conceptual boundaries causally matter for a query. Our key insight is that causal evidence exhibits discriminative selectivity. It will strongly support the original query while showing measur- ably weaker support for semantically similar alternatives. Conversely, spurious correlations exhibit indiscriminate activation, supporting multiple related queries similarly.
4.1.1 STRUCTURED COUNTERFACTUAL QUERY GENERATION
Given an original query 9. we systematically generate counterfactual queries that probe different semantic dimensions through controlled perturbations. This creates comprehensive stress test for evidence. If a document equally supports both the original query and semantically related alterna- tives, it likely contains correlational rather than causal information about the queried relationship.
Semantic Transformation Framework. We formalize counterfactual generation as structured transformation process. Let Q be the query space and T [T1, T2...., Tp. denote a set of trans- formation functions where each T; Q + Q modifies a specific semantic dimension. For given query q, we generate counterfactual queries as: eat = f = (q) : E n1, V(q, 9.) = True), where ((a, q') is validation function ensuring counterfactual quality:
True if sim,em( (') Pum L(g) * L(q')
V'g,g')
False otherwise
The constraint sim (g. g') > am ensures topical coherence, while C(g) * L(g') guarantees different ground-truth answers, creating the necessary tension for causal discrimination.
Transformation Taxonomy. We employ five categories of semantic transformations, each tar- geting different aspects of causal relationships: (1) Role-based (Trole) alter functional roles while maintaining domain relevance; (2) Temporal-shifi (Time) probe chronological causal- (Tentity) test entity-specific (Tea) explore opposite categorical boundaries; (5) Scope-modi ion (Txcope) adjust query breadth and specificity. Detailed descriptions and examples along with the prompt template for transformation generation are pro- vided in Appendix
ity through temporal modifications; (3)
versus general patterns; (4) Gategorical-invers....
Adaptive Counterfactual Selection. Not all transformations are equally informative for every query. We introduce an adaptive selection mechanism that prioritizes counterfactuals based on their potential to reveal causal distinctions. Given a query and candidate counterfactual set Qeand, we compute an informativeness score for each candidate:
Info(g. d') Div (2. d') + 3 Div
Relsom g')
(3)
where Divsem (9)9))
simsem (q. measures semantic divergence; Div (q.
dist(Ag. Ag') measures expected answer space distance; and Relgom g') = simaommin (g. mea- sures domain relevance. We select the top-K counterfactuals: Qar Topk Qcund, Info( K).
4.1.2 DIALECTICAL EVIDENCE RETRIEVAL
Traditional RAG systems retrieve evidence exclusively for the original query, creating echo cham- ber where correlational patterns reinforce themselves without challenge. CF-RAG instead constructs dialectical evidence space that deliberately incorporates potentially contradictory perspectives. forcing the system to confront and resolve conflicting signals.
Multi- Multi-Query Evidence Aggregation. We perform retrieval across all queries in the expanded query set {q) U Qer: Esaaa = R(g, D) f R(g D), where R: Q x D - 2n is our retrieval function. This dialectical retrieval simultaneously exposes spurious correlations by revealing documents that support multiple related queries with similar strength, generates contrastive signals that provide neg- ative evidence against incorrect interpretations, and ensures comprchensive coverage by mitigating query formulation biases that might otherwise overlook relevant perspectives.
Semantic Deduplication and Quality Filtering. To address redundancy and low-quality content in Egotal, we apply sequential filtering: first removing near-duplicate documents where simemb (e, e') > Odeduu (Equation 4). then retaining only documents that satisfy both quality thresholds ((e) > Quaaiity
4

---

Published as a conference paper at ICLR 2026
and relevance requirements (() s(g', e)) Oiet (Equation 5 where Q(c) measures herence, informativeness, and factual reliability.
Exstup {e Eveal De Einall \ (e), simeusle,ee) Dacepp)
(4)
Eniterod = { Edetup ((c) > Quuaity A max s(g', ee) > Ora)
(5)
d'E(9)URd
4.2 PARALLEL ARBITRATION
Parallel Arbitration addresses the challenge of evidence reconciliation: how can system validate competing interpretations when evidence contains conflicting signals? Traditional approaches either process all evidence jointly (leading to interference between conflicting signals) or separately (losing comparative context). Our solution maintains multiple reasoning paths in parallel while enabling rigorous cross-validation through causal arbitration.
4.2.1 EVIDENCE STRATIFICATION AND HYPOTHESIS CONSTRUCTION
When heterogeneous evidence is processed jointly, strong correlational signals can overshadow weaker but more causally relevant evidence. To prevent this interference while maintaining diversity, we partition the evidence space and construct multiple hypotheses from different perspectives.
Thematic Clustering. We partition Eiitered into M thematic clusters using spectral clustering on document embeddings. Let E R|Emamal be the matrix of document embeddings. We construct
an affinity matrix: Wij exp (- -e The normalized Laplacian is computed as L
D-1// where is the degree matrix. We perform eigendecomposition and the first M cigenvectors for K K-means clustering: C= -= (C1, C2....,Caa = KMeans (V M., where VM contains the first .M eigenvectors of L.
Stratified Evidence Sampling. For each of parallel reasoning paths, we construct a diverse evi- dence subset through stratified sampling that ensures representation from all thematic perspectives:
E = m= WeightedSample (Cm, Rijm), where the sample size from each cluster is determined by:
72m max (1, [|Cml p Wym
The weight Wjm introduces controlled randomness to ensure diversity across reasoning paths:
Sootmax logg(Unforrm(0, 1)) + pm
Wjn
where Pm is cluster-specific bias term and J controls the randomness temperature.
Multi-Perspective Hypothesis Generation. Each evidence subset generates a hypothesis consist- ing of both an answer and its supporting rationale:
(ag, = H(g. &,) M. (Prompp (Prompi poputheis (g. E))
Please refer to Appendix G for the carefully designed prompt structure. This parallel architecture enables the system to explore multiple interpretations simultaneously, allowing different hypotheses emerge from different evidence perspectives while preventing mutual interference.
4.2.2 CAUSAL ARBITRATION MECHANISM
The core innovation of CF-RAG lies in its arbitration mechanism, which transcends traditional confidence- based selection by incorporating causal reasoning principles. Rather than relying solely on internal coherence metrics, evaluate each hypothesis through multi-criterion system that captures both evidence faithfulness and causal discriminative power.
Internal Coherence Scoring. The coherence score &coh measures answer-evidence alignment to ensure grounded responses by averaging coherence functions across supporting documents (Equa- tion 9). where each function combines semantic similarity between answer and document embed- dings with query-relen gelevareeevance weighted by explicit answer mentions (Equation 10).
den Ea,,
CoherenceFunncionn(;,,,
(9)
EEE
5

---

Published as a conference paper at ICLR 2026
CoherenceFunction(e,, c.
X1 simmem (Enc(a),, + A2 a(g.e))) Mention(a,e)
(10)
where Enc : A UUD + IR maps answers and documents to embeddings, s(g, e) measures query- document relevance, and Mention(a, e) E {0,1} indicates explicit answer mention.
Causal Discrimination Scoring. Causal score @camsal quantifies whether evidence supports the orig- inal query over counterfactual alternatives seermatives-mmssurinng necessity rather than mere consistency:
(Ej Q. Qat)
Σ ege,
s(g,e) max s(q', e'e@a
(11)
[gg
This formulation captures the key insight of counterfactual reasoning: causal evidence should ex- hibit differential support across related queries. positive causal score indicates that evidence pro- vides stronger support for the original query than for any counterfactual, suggesting genuine causal relevance rather than spurious correlation.
Multi-Criterii Scoring Integration. We extend the basic dual-criterion approach to incorporate ad- ditional robustness measures. The comprehensive (5)
Algorithm 1: Pseudo code of CF-RAG
scoring function becomes: V Σi=1 w/ 0k
Require: Query q. Corpus D. Purameters K; M, P,A
where the four criteria are:
Ensure: Final answer die with rationale Yind
1: # PHASE I: COUNTEREACTUA
EXPLORATION
P (). (Internal Coherence)
Qa t CennerateCounteraaatuual (q. K)
3: Easlciial R(g, U U''ea, R(q',
( (em Qa)
(Causal Discrimination)
4: Epand +- ElterAAdDDduplliccceCaaaaaa (Eastescaa)
PHASE 2: PARALLEL ARBITRATION
93
(1)
Confidence(a,, ry)
(Generation Confidence)
6: C Clusterfividencee (Emorss Ad)
( Apecificity(uj... (Answer Specificity)
8: E; < StratifeedSample()
for j to do
9: (a,. ry) GencracHyyohhessssgy 82)
The confidence score measures the model's certainty
10: & InternilCsherence(ea
in its generated response:
CausalDiscriiination... 2a) 12: V; + (1 X) oi + A dimm
11: Pamal
Irl log P( ......,1,00 a,.) 9)
Confidence(a,
13: end for
14: j" arg max, 9
15: if Highhonnsnnuus ({a;}) then
The specificity score rewards answers that directly
16: "iad ay*
address the query rather than providing overly gen- eral responses:
17: else
18: Gisa + (ToK
19: end if
Specificity(a, q) Entropy (eeoooodsKo)))))((()))
20: return (Garal Tima)
log |Vocabulry(aa)
4.2.3 ANSWER SYNTHESIS AND REFINEMENT
Beyond selecting the sigheg hypothesis jr = arg max, J,, we perform cross-hypothesis validation by measuring agreement patterns Agreement(a)
a))
boost confi- dence for consensus answers or trigger verification for divergent ones. When multiple high-scoring hypotheses exist, synthesize combined response from the top-3 candidates using specialized prompts that incorporate multi-perspective evidence while maintaining coherence.
aétal Synthesize ([(a), v sepkk q, Eanores)
(12)
Algorithm D presents the complete CF-RAG framework.
THEORETICAL ANALYSIS
CF-RAG escapes the correlation trap through causal discrimination that remains robust even when spurious evidence vastly outnumbers causal evidence.
Lemma (Counterfactual Discriminability) For CF-RAG distinguish causal from spurious ev- idence, Qet must contain at least one query g' such that spurious evidence Expurions exhibits compa-
rable support for both queries:
3q' E Qef: :e E Expurious: |s(g. e) s(g',e) <.
Lemma 2 (Volume Invariance)
(E, The causal discrimination score &camall (E, 2ey) is invariant
der spurious evidence scaling: for any scaling factor >> 0.
Peaussa (a- - E spurioon Eeauott: 9, 2.y) = O Oranal (E sparious U Eeousat q, Quy)

---

Published as a conference paper at ICLR 2026
Together, these lemmas ensure spurious evidence detection (Lemma remains effective regardless of volume (Lemma enabling our main result:
Theorem (Escape from the Correlation Trap) Let be query with ground truth answer a' supported by causal evidence Ecanaal and spurious answer a' supported by correlated evidence
Equarious- If: (CI) Ve E Ecausal, Vqg E Qg e( =s(q'e) 2 5; (C2) By Lemma 7 3d E Qes such that E Expurions [s(g,e) E) - s(y', (('',e) e)] <;; (C3)
Then CF-RAG with appropriate À selects a" over d' regardless of volume ratio (Espurious) //Ecaurra)) while standard RAG fails when this ratio exceeds Scnusal/ Sxpurious
The causal discrimination condition (C1) ensures that genuine causal evidence maintains its discrim- inative power under counterfactual testing. The spurious non-discrimination condition (C2) directly invokes Lemma [ to guarantee that spurious correlations can be detected. The separation condition (C3) ensures relibble discrimination practice. Proofs are provided in Appendix A
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets. We evaluate five challenging benchmarks that test different aspects of causal reason- ing under retrieval-augmentt generation. These include multi-hop QA datasets HotpotQA Yang al. (2018) and MusiQue Trivedi al. (2022), which require synthesizing evidence across multi- ple documents while avoiding distractors; TriviaQA Joshi et al. (2017). featuring compositionally complex questions with subtle spurious correlations; PopQA Mallen et al. (2023), focusing on long- lail entities where models must rely on retrieved rather than parametric know leowledge; and PubHealth Zhang al. (2023), a medical claim verification task demanding high factual precision. Each dataset presents unique challenges for disambiguating causally relevant evidence from misleading correla- tions. See Appendix C.1 for detailed dataset characteristies and preprocessing pipeline.
Baselines. We compare against three categories of systems: (i) Zero-shot LLMs without re- trieval (Llama-3-8B, Llama-2-7B, Alpaca-7B); (ii) Standard RAG augmenting these models with conventional retrieval; and (iii) Advanced RAG frameworks including CRAG (Yan al., 2024), Self-RAG (Asai et al. 2024), Self-CRAG (Yan et al.) 2024), and Speculative-RAC (Wang et al, 2024ch. Additionally. evaluate auol-augmented approaches: Toolformer (Schick et al. 2024 and
SAIL Luo et al. 2023). See Appendix B.4 for implementation details.
Implementation. CF-RAG uses Baatmaa---B-Instteee and Llama-2-7B base models for fair com- parison. Default hyperparameters: N=3 counterfactual queries, K=4 evidence clusters, M== par- allel drafts, and causal weight A=044. All models use identical retrieval corpora and are evaluated using Exact Match (EM) scores. See Appendix B for complete details.
6.2
MAIN RESULTS
Table
presents our main experimental findings. CF-RAG demonstrates substantial improvements across all benchmarks, with the most dramatic gains on complex reasoning tasks:
Multi-hop Reasoning Excellence. On HotpotQA, CF-RAG (Llama-3-8B) achieves 88.58% EM, an absolute improvement of +80.8% over the strongest baseline. This remarkable nein-nearly doubling celoomannee-vliddatee our hypothesis that causal reasoning is essential for navigating complex evidence chains. The improvement on MusiQue (+63.2%) further confirms this pattern.
Robust Performance Across Domains. Beyond multi-hop tasks, CF-RAG maintains strong perfor- mance on single-hop retrieval (TriviaQA: 8 1.0%), long-tail entities (PopQA: 73.57%), and medical claims (PubHealth: 83.36%). This consistency suggests that causal reasoning benefits extend be- yond specific task types to fundamentally improve evidence evaluation.
Model-Agnostic Improvements. Both Llama-2-78 and Llama-3-8B backbones show substantial gains (+21.0% and +31.9% average improvement respectively). demonstrating that CF-RAG's ben- efits not tied to specific model architectures.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison across five QA benchmarks. All scores are Exact Match (%).
The best results from baseline are underlined.
Method
HolpotQA
TriviaQA
PopQA
MusiQue
PubHealth
Avg.
Zero-Shot LLM (No Retrieval)
Llamaa 2-7b
18.53
18.15
1omma-3-8-Innstee
23.45
28.17
20.31
14.30
44.27
26. 10
Alpaca-7B
27.27
Standard RAG
Llama 2.7h
27.72
32.52
38.20
15.84
38.10
30.48
36,04
31.12
42.46
21.41
48.58
35.92
Alpaca-7B
31.29
34.20
46.70
19.90
40,20
34.46
Instruction- Panned RAG
SAH.
69.20
Toolformer
48.80
AdvaRced RAG Frameworks
CRAG (Llama-2-7b)
25.24
60,77
55.02
60.18
Self-RAG (Llama-2-7b)
64.39
53.97
20.58
73.19
48.12
Self CRAG (Llama 2. 2-7b)
34.44
63.72
60.98
74,80
Specolative- (Mistral-7b)
57.54
Speculative RAG (Llama 2 7b)
47.90
75.37
56.21
33.45
75.29
57.64
CF-RAG (Llama-2-7bb
79.29 (161.8%)
76.15 (1.0%
67.22 (710.2%)
48.78
(745.8%) 8%)
78.24 (12.1%
69.94 (+21.%%
CF-RAG (LJama-3-88)
88.58 (180.85)
81.02 (17.5%)
73.57
(120.6%)
54.59
(163.25%)
83.36 (18.8%)
76.22 (131,9%)
6.3 COMPONENT ANALYSIS
To understand each component's contribution, we conduct systematic ablations on HotpotQA and PopQA. Table 2 presents the results, revealing clear component hierarchy. Removing the causal verification mechanism (dcansal) causes the most severe degradation- a 17.37% drop on HotpotQA and 13.73% PopQA-confrrminn that distinguishing causal from correlational evidence is CF- RAG's core innovation. The counterfactual exploration component is also critical, its removal causes 11.36% and 6.17% performance drops respectively, demonstrating dialectical evidence construc- tion is essential for enabling causal discrimination. Even the evidence division mechanism, while contributing the smallest individual effect, yields meaningful improvements (4.84% on HotpotQA, 8.54% on PopQA) by preventing interference between conflicting signals during parallel hypothesis generation. These results validate our design: each component addresses a specific failure mode in correlation-base retrieval, and their synergistic combination enables the robust causal reasoning distinguishing CF-RAG from existing approaches.
Table 2: Ablation study revealing component contributions.
Each column shows performance with a specific component removed, with degradation percentages in parentheses.
Dataset
Full CF-RAG
w/o Counterfactual
w/o Evidence Division
w/o Causal Verification
HotpotQA
88.58
78.52 (111 1.36%)
84.29 (14.84%)
73.19 (117.37%)
PopQA
73.57
69.03 (J6.17%)
67.29 (18.54%)
63.47 (113.73%)
626
00215
EM Scor
6.52
EM Scare Eiltebaucy Scure
EM Emfiining Score
6.822
Srfibenty Score
0.077
Eat Scaro (h
$5.0
DM Scarn
0.129
Kmme
750
Numaa bar of Countertactual Querias (N)
Number of Ixtdesen Clusters (P)
Number Generated Draits (4)
Casstt Vertfication Weight CO
Counterfactual Queries (N)
(b) Evidence Clusters (K)
(c) Parallel Drafts
(d) Causal Weight
Figure 3: Hyperparameter sensitivity analysis HotpotQA. The bar chart shows the EM score
(accuracy) and the line chart shows the EM efficiency score (accuracy/latency).
8

---

Published as a conference paper at ICLR 2026
6.4 PARAMETER SENSITIVITY AND EFFICIENCY ANALYSIS
We analyze CF-RAG's hyperparameter trade-offs between accuracy and computational effi- ciency HotpotQA, introducing an EM Efficiency Score that measures accuracy per second: EM Efficiency EM 1100
This metric reveals how CF-RAG adapts to different operational
Latency (5)
requirements (Figure
Accuracy-Efficiency Trade-offs. The number of counterfactual queries (N), evidence clusters (K), and parallel drafts (M) exhibit consistent patterns: peak accuracy occurs at moderate values (N=3, K=4, M=3) achieving 88.58% EM, while peak efficiency emerges at minimal settings (N=1, K=2, M=1) for maximum throughput. These findings enable flexible deployment-quue quality-critical appli- cations maximize accuracy with higher parameters, while latency-sensitive systems can achieve faster responses with minimal settings while maintaining 80%+ accuracy.
Causal Weight Optimization. The causal weight À uniquely shows aligned optima: both accuracy and efficiency peak at A=0.4, validating our core hypothesis that balanced synthesis of coherence and causal discrimination is essential. Pure coherence-basee (A=0) or purely causal (A=1.0) selection underperform by 5% and 9.7% respectively, confirming that neither signal alone suffices for robust performance. This parameter requires no trade-off the theoretically motivated balance point also maximizes practical efficiency.
6.5
MECHANISTIC ANALYSIS: ADDRESSING RAG FAILURE MODES
Table 3: Performance specific RAG failure modes. CF-RAG's targeted mechanisms dramati-
cally reduce error rates across all failure categories.
Failure Mode
Description
Baseline Error
CF-RAG Error
Reduction
Spurious Correlation
Relies correlated hut non-causal evidence
56.7%
13.3%
76.5%
Evidence Extraction
Correct answer obscured by noise
43.3%
23.3%
46.1 %
Scattered Synthesis
Fails to connect multi-hop evidence
63.3%
16.
73.6%
To understand why CF-RAG succeeds, we analyze performance on three documented RAG failure
modes using curated challenge sets (30 queries per mode) as illustrated in Table 33] The dramatic error reductions demonstrate that CF-RAG's components directly address root causes: (I) Causal verification eliminates spurious correlations; (2) Parallel processing isolates signal from noise; (3) Counterfactual exploration bridges evidence gaps.
6.6 ROBUSTNESS TO ADVERSARIAL DISTRACTION
We stress-test robustness by injecting Nijeert E {2, 4, 16) adversarially-selcctee distractors into HotpotQA's retrieval corpus. These documents maximize spurious correlations: high semantic similarity but zero factual relevance.
Stavladd RAG Treed Specuadive RAG Tread
Standort RMG
Speculative RAG CP dhong
RMO joans Thord
Exact Match (EM) Score
80
7321
76.13
68.0
As shown in Figure 4 baseline models degrade significantly under adversarial pressure, with accuracy dropping by 66.4% for Standard RAG and 56% for Speculative-RAG. In stark contrast, CF-RAG demonstrates remarkable stability, main- taining high accuracy 60.57% in the most challenging setting. This resilience validates that the framework's 's deci- sions are driven by causal discrimination rather than correla- tion strength, ensuring reliable performance in noisy environ- ments where other systems falter.
60
32,03
40
33:37
25,11
28.61
Number AAvversarill
Figure 4: Adversarial robustness.
7 CONCLUSIONS AND LIMITATION
In this work, we identified and addressed a fundamental vulnerability retrieval- augieeee-- gen- eration systems: the Correlation Trap, where models conflate spurious correlations with causal evi- dence. We introduced CF-RAG, a novel framework that transforms correlation based retrieval into causation-driven reasoning through systematic counterfactual testing. By integrating Counterfactual

---

Published as a conference paper at ICLR 2026
Exploration discover causal boundaries and Parallel Arbitration to reconcile conflicting evidence, CF-RAG achieves robust discrimination between evidence that causally determines an answer ver- evidence that is merely correlated. Our extensive experiments demonstrate state-of-the-e per- formance across five challenging benchmarks, with particularly striking improvements on complex multi-hop reasoning tasks.
Our evaluation currently centers on multilingual and cross-lingual contexts. The standard dense retrieval and factoid QA tasks employed could also be broadened to include emerging retrieval architectures and more subjective domains. These limitations suggest clear avenues for future investigation.
benchmarks and would benefit from extension
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of work. have made code. experimental setup, and the- oretical proofs publicly available. The complete implementation the CF-RAG framework, in- cluding models and seripts to replicate our experiments, is provided in the supplementary materials and can be accessed at httpp:://gith http///gttuu.....
RRG/CF RAG A detailed breakdown of the model architecture, hyperparameter configurations, and baseline implementations is available in Appendix B For our theoretical claims regarding CF-RAG's ability to overcome the correlation trap, we provide complete proofs in Appendix A Furthermore, Appendix C describes the datasets used in our evaluation and the exact preprocessing pipeline applied. The core methodology is detailed Section 4 with specific prompt templates and our transformation taxonomy further elaborated in Appendices and
ACKNOWLEDGMENTS
This work was supported by The Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education China (JYB2025XDXM702), NSFC (No.62506366, No. No.62272466, U24A20233), and Big Data and Responsible Artificial Intelligence for National Governance, Ren- min University of China.
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. net Eorum? wwwaamii--Syyiee hS
URL https::
Jinheon Baek, Soyeong Jeong, Minki Kang, Jong Park, and Sung Hwang. Knowledge-augmente language model verification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1720-1736. 2023.
Ananth Balashankar, Xuezhi Wang, Yao Qin, Ben Packer, Nithum Thain, Jilin Chen, Ed H Chi, and Alex Beutel. Improving classifier robustness through active generation of pairwise counterfactu- als. arXiv preprint arXiv:2305. 13535, 2023.
Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. Zero-shot Ilm-guided coun- terfactual generation for text. CoRR, 2024.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Bapiln Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Tong Chen, Hongwei Wang. Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312. 066488. 2023.
10

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. XrXiv preprini arXiv:2312.10997, 2(1), 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346 7:346-36,, 2021.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.). Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguis- tics. doi: 10.18653/v 1/23311122211122 cae.22..al-mai..7700
URL https: / /cc ant he hotty..ooy...ooo org//021.
eaal main.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1969-79922 Singapore, December 2023b. Associa- tion for Computational Linguistics. doi: 10. 10.18653/11/2223.mmmml wwt.:memmlmmmmmamm... // //aclantholooyy www.anthoooo....oo///00.
URL https:
emn main. 495
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zeutlemoye. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6788 2020.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes difference with counterfactually augmented data. International Conference on Learning Repre- sentations (ICLR), 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization: Nearest Neighbor Language Models. International Conference on Learning Representations (ICLR), 2020.
Jachyung Kim, Jachyun Nam, Sangwoo Mo. Jongjin Park, Sang- Woo Lee, Minjoon Sco, Jung- Woo Ha, and Jinwoo Shin. Sure: Summarizing retrievals using answer candidates for open- open-domain qa of lms. arXiv preprint arXiv:2404.1 /308/, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentt gener- ation for knowledge-intensing nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume I: long papers), pp. 3214-3252, 2022.

---

Published as a conference paper at ICLR 2026
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. arXiv preprint
Trxiiv2305.15225. 2023.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5315, 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, 2023.
Stephen McAleese and Mark Keane, comparative analysis of counterfactual explanation methods for text classifiers. arXiv preprint (rXiv:2411.02644 2024.
Judea Pearl. Causality. Cambridge university press, 2009.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Man- ning. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprint arXiv:2401.18059, 2024.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Chaitanya Sharma. Retrieval-augmented generation: A comprehensive survey of architectures, en-
hancements, and robustness frontiers. arXiv preprint arXiv:2506.000544 2025.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 11210-31221 PMLR, 2023.
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.0176. 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099 092888 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, and Zhiqi Shen. A survey on natural language counterfactual generation. arXiv preprint arXiv:2407.0399) 2024a.
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig, Learning to filter context for retrieval-augmin -uugeneed generation. arXiv preprint arXiv:2311.0837, 2023.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint 0:Xi::2403.05333, 2024b.
Zilong Wang, Zifeng Wang. Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, et al. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223: 2024cc
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Polyjuice: Gen- erating counterfactuals for explaining, evaluating, and improving models. arXiv preprint arXiv:2101.00220 2021.
12

---

Published as a conference paper at ICLR 2026
Sirui Xia, Xintao Wang, Jiaqing Liang. Yifei Zhang, Weikang Zhou, Jiaji Deng, Fei Yu, and Yanghua Xiao. Ground every sentence: Improving retrieval -uggeened Ilms with interleaved reference- claim generation. arXiv preprint urXiv:240.01779. 2024.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmente Ims with compression and selective augmentation. arXiv preprint arXiv:2310.04408, 2023.
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. CoRR, 1atp22401.188844 2024. doi: 10.48550/ARXI..2440 15884. htRt::/pss//ddo /
10. 4.5508500/4 rww.v.ivvv...... 2401 11.4401158880 15884
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmentee language models robust to irrelevant context. arXiv preprint arXiv:2310.0/558, 2023.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note note: Enhancing robustness in aetrieeel-augmented language models. arXiv preprint arXiv:2311.09210, 2023.
Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng. and James Glass. Interpretable unified language checking. arXiv preprint arXiv:2304.03722 2023.
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint 10311 2024.
A THEORETICAL PROOFS
In this appendix, provide detailed proofs for the theoretical results presented in Section 5
A.1 PROOF OF LEMMA[] COUNTERFACTUAI DISCRIMINABILITY
Proof We prove this by contradiction. Suppose the counterfactual set Qef does not contain any query o' for which the spurious evidence E' has comparable support for both q and q'.
Formally, assume:
Yg' Ve s(g,e) s)
(13)
for some constant 7 O.
Under this assumption, the causal discrimination score for the spurious evidence would be:
Deausat (E', ,,,,, 2ey)
8 s(g.e) e)
max
s(q',e)
(14)
[
4' d'ER,
g
339
3E3
1
s(g. e)
[s(g, E) - y
(15)
E
|
CEE!
EEE'
=7>0
(16)
This means the spurious evidence would achieve a positive causal discrimination score, potentially comparable to exceeding that of the causal evidence Ex. In such case, CF-RAG might incor- rectly select a' over a".
Therefore, for CF-RAG to reliably distinguish between causal and spurious evidence, the counter- factual set must contain at least one query of that exposes the non-discriminative nature of spurious correlations. Specifically, there must exist o' Qay such that:
Ve E E' [s(g,e) s(q',)) e)l <e
(17)
for some small
13

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF LEMMA 2 VOLUME INVARIANCE OF CAUSAL SCORES
Consider an evidence set & and its scaled version a& where a > O is a scaling factor. The scaled set contains copies of the same evidence distribution.
Proof 2
The causal discrimination score for the scaled evidence set is:
@causat q, 2it
Σ
s(g,e)
max d'ER, (oy)
Σ
,(d'se e)
(18)
cEaL
cεaε
Since 3f contains a copies of each element in E, we have |aε| = a:|E|
and:
Σ
s(g 6) = a Σ s(g. c)
(19)
cEaE
6E8
Σ cEag
s(d',)) = a: > s(1', e)
(20)
EEE
Substituting
Ócausal 2④)
1
Σ
s(q,e)
1
Σ s(g',)
(21)
arE
d'E9s afE
3EE
CEE
max s(1 e)
Σ s(d,e) 3E
(22)
9' d'ER,
337
Dcausal (, Qct)
(23)
Therefore, the causal discrimination score is invariant under volume scaling. The same property holds for the coherence score through an identical argument, making the entire arbitration score W volume-invariant
A.3 PROOF OF THEOREM 1 ESCAPE FROM THE CORRELATION TRAP
We need to show that under the given conditions, CF-RAG selects the true answer a" over the spurious answer a' regardless of the relative volumes of their supporting evidence sets, while standard RAG fails above a certain volume threshold.
Proof 3
Part A: CF-RAG Analysis
Consider two hypotheses generated by CF-RAG:
(a",r","" hypothesis with true answer supported by causal evidence 'h' (a',r' (')))) hypothesis with spurious answer supported by correlated evidence
will prove that the arbitration score for h" exceeds that of h', i.e., 4 >> W.
Step 1: Analysis of Causal Scores
First, we analyze the causal discrimination score for the true hypothesis h²", By definition:
Ocausal ,9,,,,.90 Qcf)
s(g.e)
max
s(g',e)
(24)
E
cEE-
aEg.
Since the maximum is achieved by some
9º' Qef. write:
Demmna (E", gg Qa))
Σ s(g,e)
Σ ,(g",e)
(25)
[3
EE
3E8
[s(g. e) - -s)9"
(26)
|
14

---

Published as a conference paper at ICLR 2026
By condition of the theorem, for all e Ex and all d' Rey. have s(q,e) s(d',)) e) 2 6. This applies to q" as well, therefore:
ceauval (E" .q, 2it) [+
5
(27)
EEE
Now for the spurious hypothesis hi, by condition 2, there exists a counterfactual query d' 2e that for all e :
(s(g,e) s(d',c) <e
(28)
This implies:
-e <<< <<((,))) - s(d,,) <E
(29)
Therefore:
reaunalE' 2cy)
max s(g.e)
,(d",e)
(30)
E
EeE
E
EEE'
BEE
1
s(g. e)
s(''.e)
(31)
EE9
cEE
[s(1,)) 6)
e)
(32)
C
(33)
6EE'
Step 2: Analysis of Coherence Scores
The coherence scores measure the semantic alignment between answers and their supporting evi- dence. Since both a" and a' are derived from their respective evidence sets through the same gen- eration process H. and assuming the generator produces reasonably coherent outputs when given relevant evidence, we can establish bounds on the coherence scores.
Let &ioh = Ócon(a", "o,,, ",",,) and &cch = Qeat(a', ,',').))
Since both evidence sely contain documents retrieved for either q or related queries, and both an- swers are generated to be consistent with their evidence, we have:
Ceono &ech [omin Ømax]
(34)
where < ohmin S Omax S. 1 are constants determined by the quality of the retrieval and generation models.
Step 3: Comparison of Arbitration Scores
The arbitration scores for both hypotheses are:
T = (1 2)dd + >
(35)
V = (1 A)dscoh + X
(36)
From our analysis above:
4 2 (1 A)omin + A6
(37)
V (1 A)dmmax de
(38)
For CF-RAG to select a" over d', we need 4 > V.:
(1 - 2) + A8 > (1 A)@max + Ae
(39)
Rearranging terms:
A(6 6) > (1 1) Φmin))
(40)
15

---

Published as a conference paper at ICLR 2026
Since å > by condition 3, we have 5 - O. Solving for A:
Omax Pmin
A > 8 + Φmax Omin
(41)
Let X" D-e+-omax Omnen positive, and the denominator is strictly larger than the numerator.
Cinax Omin
Note that X" 1 since both numerator and denominator are
Therefore, for any À E (X*1]) we have 4 > V, which means CF-RAG will select a" over a'.
Step 4: Independence from Evidence Volume
By Lemma 2 both Oranall and Desh are invariant under evidence volume scaling. Therefore, the arbitration scores V and V remain unchanged regardless of the relative sizes of E and E'. This proves that CF-RAG's selection is independent of 1 1///
Part B: Standard RAG Analysis
In contrast, standard RAG uses aggregate support:
Scoregac(a)
Σ s(q,e)
(42)
3E
For our two competing answers:
ScoreRsa(a") = =1E"1 1
(43)
ScoreRac(a') |''|
(44)
where 8* 1 TET -TET Ence- s(g e) and 5'
Ener s((,e). e).
[7
Standard RAG fails when:
E
s*
(45)
3.
Even if causal evidence has higher average relevance (8* > 8'), sufficient accumulation of spurious evidence causes incorrect selection-the correlation trap.
This completes the proof, demonstrating that CF-RAG escapes the correlation trap through causal discrimination while standard RAG remains vulnerable evidence volume effects.
B IMPLEMENTATION AND ARCHITECTURAL DETAILS
We provide comprehensive implementation details to ensure full reproducibility of our results. Our codebase and pre-trained models are available at attttps://githhu
CF CR-RRCCCF--EEEEE
B. MODEL ARCHITECTURE AND FRAMEWORK
Core Language Models. CF-RAG employs modular architecture with distinct models for gen- eration and scoring tasks. We evaluate our framework using two widely-adopted instruction-tuned
LLMs as backbones:
Base LLM Configurations
Llama-3 8B: meta metta-llan -llamamLlama - -nstrrnstt
Llama-2 7B: meta meta-llammin wew.aaaaaa//mmmmmmaaaaaaaaaa Llama - 2-77-chhttttt
Scoring and Retrieval Components. For fine- tine-grained relevance assessment crucial to our causal arbitration mechanism, we employ:
Cross-Encodee BAAT BAAT SAAT/bge-reranker- large for computing relevance scores s(g,e)
16

---

Published as a conference paper at ICLR 2026
Dense Retriever: /utpa//neetttttttttaaaaaaaaaaaaa with FAISS indexing
Key Design Choice: All performance gains stem directly from CF-RAG's architectural
innovations no additional fine-tuning was performed on base models.
Hypothesis Generation. The generator function (2,2P) - A x produces answer-rationale
pairs through carefully designed zero-shot prompting:
System: You are an expert analyst. Provide evidence-based reasoning.
Query: Jcriginalqquery
documents) Format: [Reasoning with citaticns] [Final Answer]
Evidence: (evidencee.occueents)
B.2 HYPERPARAMETER CONFIGURATION
Table presents our hyperparameter settings, selected through systematic grid search on the Hot- PotQA development set and ablation studies detailed in Section 6.4
Table 4: CF-RAG hyperparameter configuration. Parameters were optimized for the trade-off between performance and computational efficiency.
Component
Parameter
Value
Search Range
Selection Criterion
(counterfactual queries)
[1. 5]
Performance-latency trade-off
Counterfactual
sim (similarity threshold)
0.7
[0.5, 0.9]
Topical relevance vs. diversity
K (evidence clusters)
[2, 6]
Thematic coverage
Evidence
P (sampling ratio)
0.5
[0.2, 0.8]
balance Near-duplicate removal
0 dedup (deduplication)
0.95
[0.9, 0.99]
M (parallel hypotheses)
[1,
Exploration-efficcenyy trade-off
Arbitration
(causal weight)
0.4
[0.0, 1.0]
Robustness analysis ($6.4)
B.3 COMPUTATIONAL INFRASTRUCTURE
Software Stack:
Hardware Configuration:
Python 3.9 PyTorch 2.1
8x NVIDIA A100 80GB GPUs
Transformers 4.36.0
ITB RAM for document indexing
FAISS 1.7.4 (similarity search)
NVMe SSDs for cached embeddings
Scikit-learn 1.3.0 (clustering)
B.4 BASELINE IMPLEMENTATIONS
Our evaluation encompasses three categories of baselines, systematically covering the spectrum of current retrieval-augmented generation approaches- from pure parametric models sophisticated adaptive frameworks.
Foundation Models (No Retrieval)
We establish performance floors using zero-shot inference with L Llama-23 3-8p Dubeeeubbbbyy et al. (2024), Llama-2-78 Touvron al. (2023), and Mi stral-7B Jaang et al. (2023a), which rely solely on parametric knowledge without external evidence.
Classical RAG Systems
Standard RAG concatenates top-k retrieved documents directly to the input prompt without filtering refinement mechanisms.
Toolformer Schick et al. (2024) augments language models with API calls, treating retrieval as a learnable tool invocation.
17

---

Published as a conference paper at ICLR 2026
SAIL Luo et al. (2023) implements search- search-augmented instruction learning, fine-tuning models
through instruction- -ortuucttiooooooiinnn objectives to better leverage retrieved evidence.
Advanced RAG Frameworks
CRAG Yan et al.
(0244) employs corrective retrieval with confidence-basee document refine-
ment, dynamically adjusting retrieval sets based quality assessments.
Self-RAG Asai et al. (2024) introduces self-reflective tokens for retrieval decisions and critique, enabling adaptive generation with learned reflection mechanisms.
Self-CRAG synthesizes Self-RAG's reflection capabilities with CRAG's corrective refinement strategies in hybrid architecture.
Speculative-RAGi Wang et al. (024c) leverages parallel draft-verify architecture with specialized reasoning.
drafting models for efficient
Implementation Details
All baselines utilize identical retrieval corpora-Wikinn Wikipedia (Decemher 2018) HotpotQA, TriviaQA, and MusiQue; domain-specific corpora for PopQA and PubHealthh-we wiith uniform preprocessing pipelines. We ensure fair comparison by deploying the same base language models across all retrieval-augmmntted methods. When citing original results, maintain comparable settings for model size, retrieval corpus. and evaluation metrics. Re-implemented baselines use official codebases with hyperparameters optimized on validation sets.
C DATASETS AND EVALUATION
We evaluate CF-RAG on five carefully curated datasets that collectively probe different failure modes of correlation- based retrieval. Each dataset was selected to expose specific vulnerabilities traditional RAG systems and test our framework's ability to perform genuine causal reasoning.
C.1 DATASET SELECTION AND CHARACTERISTICS
Our evaluation suite spans diverse reasoning challenges, from multi-hop inference to domain- specific verification, ensuring comprehensive assessment of causal disambiguation capabilities.
HotpotQA: Yang al. 2018) A multi-hop question answering dataset requiring reasoning
multiple documents. Its "distractor" setting, which includes irrelevant documents alongside gold documents, makes it an ideal testbed for evaluating a model's ability filter misleading. correlated information and focus on the true causal chain of evidence.
TriviaQA: Joshi et al. (2017) A large-scale QA dataset featuring complex, compositional ques- tions with significant lexical and syntactic variation between the question and the evidence. This challenges models to move beyond simple keyword matching and resist spurious correlations based on term overlap.
open-domain QA dataset focused on long-tail entities. This design forces models to rely on the provided retrieved evidence, as the answers are unlikely to be stored accurately in the model's parametric memory. directly tests the model's ability to ground its generation in external knowledge, especially when its internal knowledge is weak or incorrect.
PopQA:
Mallen et al. 20233
MusiQue: Trivedi al. (0022) multi-hop QA dataset explicitly constructed to be more challenging and less susceptible to "cheating" via single-hop reasoning shortcuts than previous datasets like HotpotQA. Its design requires genuine connected reasoning over 2-4 documents, making it stringent test for causal synthesis.
PubHealth: Zhang et I 2023) A closed-set claim verification task in the public health domain. This dataset is critical for evaluating robustness in high-stakes setting where factual accuracy is paramount. It tests the model's 's ability to distinguish medically accurate causal evidence from popularly held but incorrect correlations.
18

---

Published as a conference paper at ICLR 2026
Preprocessing Pipeline. To ensure fair comparison across these diverse datasets, employ uni- fied preprocessing approach across all corpora: documents undergo chunking into ~250-token pas- sages with 50-token overlap, preserving semantic boundaries while maintaining computational effi- ciency. This overlapping window strategy prevents information loss at chunk foundaries-critteee for maintaining causal chains that span multiple segments.
C.2 EVALUATION PROTOCOL
Our evaluation employs dataset-appropriate metrics that capture both surface-level correctness and deeper reasoning quality.
Evaluation Metrics
Smart Exact Match (EM)
Primary Metric
robust variant of exact match that handles linguistic variations through:
Normalization: Lowercasing, article/punctuation removal
Containment: Ground-truth prediction checking
Semantic equivalence: Date formats, name variations, yes/no synonyms
Classification Accuracy (ACC)
PubHealth Only
For claim verification tasks with fixed label sets (True, False, Mixture), we report standard classification accuracy, which reduces to Smart EM in this closed-set setting.
This evaluation framework ensures that CF-RAG's improvements stem from genuine causal rea- soning rather than superficial pattern matching, providing a rigorous test of our framework's 's core contributions.
D QUALITATIVE ANALYSIS AND CASE STUDIES
To illuminate the mechanisms underlying CF-RAG's causal reasoning capabilities, present detailed case study demonstrating how our framework successfully navigates the correlation trap that confounds traditional RAG systems.
D.1 DISSECTING THE CORRELATION TRAP: A CASE STUDY ON The Dark Knight
We trace CF-RAG's reasoning process the motivating example from Figure T revealing how counterfactual exploration and parallel arbitration work synergistically to achieve robust causal in- ference.
Query: Who is the lead actor in The Dark Knight?
Standard RAG Failure Traditional retrieval surfaces five documents dominated by Heath Ledger's acclaimed performance- four effusive reviews highlighting his "iconic," "Oscar- winning," and sycene-stealing portrayal overshadow a single Wikipedia entry correctly identi- fying Christian Bale. The system, overwhelmed by correlational signals, incorrectly concludes:
Heath Ledger is the lead actor
Our framework orchestrates two-phase causal analysis:
PHASE 1: COUNTERFACTUAL EXPLORATION
CF-RAG generates three semantically diverse counterfactuals that probe different conceptual bound- aries:
19

---

Published as a conference paper at ICLR 2026
Counterfactual Query
Semantic Axis
Who played the main villain in The Dark Knight?
Role Variation
Who is the lead actor in Batman Begins?
Entity Substitution
Who directed The Dark Knight?
Categorical Change
These counterfactuals create dialectical evidence space that exposes the non-disrimminative nature of spurious correlations -erideeinnn-eedeeeeeeeeee praising Ledger's performance applies equally to queries about villains, while evidence about Bale's protagonist role remains specific to the original query.
PHASE 2: PARALLEL ARBITRATION
The system constructs and evaluates competing hypotheses from stratified evidence subsets:
Hypothesis I: Christian Bale
Hypothesis 2: Heath Ledger
Evidence:
Wikipedia entry + Director interview
Evidence:
Four performance reviews
"Christian Bale Bruce Wayne/Batman" "We needed an incredible actor for Batman..."
"Iconic, Oscar-winning performance" "Scene-stealing portrayal of the Joker"
doo
0.85
Ocoh 0.91
Ocanat
0.72
@cauatl -0.15
V1
0.798
V/2
0.486
High causal discrimination-cviieee uniquely supports lead actor query
Low causal diserimination equally
supports villain query
The dual-criterion scoring reveals the critical distinction: while Hypothesis achieves superior coherence (doch 0.91), its evidence lacks causal specificity-the same reviews praising Ledger's performance apply equally to the counterfactual query about villains, yielding negative causal score (&eausal 0.15). In contrast, Hypothesis I's evidence, though less voluminous, provides clear causal discrimination (Øcausal = 0.72), as documents about Bale's protagonist role do not support queries about villains directors.
Final Arbitration: V1 = 0.798 > 2/2 0.486
CF-RAG Output:
Christian Bale is the lead actor in The Dark Knight
This case study demonstrates CF-RAG's fundamental innovation: by systematically testing evidence against counterfactual alternatives, framework distinguishes causal support from spurious cor- relations, achieving robust reasoning even when misleading signals vastly outnumber correct ones. The parallel arbitration mechanism ensures that strong correlational patterns cannot overshadow weaker but causally decisive evidence, transforming the correlation trap from a critical vulnerability into solved problem.
E DEEP DIVE INTO COUNTERFACTUAL EXPLORATION
This section provides a transparent examination of our counterfactual generation process-a cor- nerstone innovation of the CF-RAG framework that enables causal reasoning through systematic hypothesis testing.
E.1 COUNTERFACTUAL GENERATION: DESIGN AND IMPLEMENTATION
Generation Strategy. Our counterfactual queries emerge from carefully designed zero-shot prompting strategy that balances semantic proximity with answer divergence. The generation pro- cess follows critical constraints:
20

---

Published as a conference paper at ICLR 2026
Counterfactual Generation Prompt Template
Given the following questions achering to:
question, generate
[N}
alternative
Topical Coherence: Questions must address the
entities/topics
2. Answer Divergence: Questions must seek fundamentally difforent information
Explore different facets through role variations, entity substitutions temporal shitts, or categorical changes. Original Question: "(query)"' Alternative Questions: [1..N]
Illustrative Examples. Table@]eeoonsrrate the diversity and sophistication of our counterfactual generation across multiple transformation types, showcasing how each perturbation probes distinet semantic boundaries while maintaining topical relevance.
Table 5:
Representative counterfactual transformations across datasets. Each transformation type serves distinct purpose in exposing correlational vulnerabilities.
Original Query
Transform Type
Generated Counterfactual
Dataset
"Who is the lead actor in The Dark Knight?"
Role Variation
"Who played the main villain in The Dark Knight?"
Example
"What is the area code for the city where University of Notre Dame located?"
Entity Substitution
"What the area code for the city where
HotpotQA
Purdue University is located?"
"Which Nolan film was released first: be-
Temporal Shift
"Which Nolan film had larger hox office:
TriviaQA
ception or The Dark Knight?"
Inception or The Dark Knight?"
"What is the genre of Abbey Road by The
Categorical Change
hat yeur was A Abbey Road by The Beatles
MusiQue
Beatles?
released?"
E.2 QUALITY ASSURANCE: TWO-STAGE VALIDATION FRAMEWORK
The efficacy of CF-RAG hinges on the discriminative power of generated counterfactuals. We im- plement rigorous two-stage quality assurance pipeline combining automated filtering with human validation.
E.2.
STAGE 1: AUTOMATED QUALITY CONTROL
Our automated pipeline enforces two complementary criteria through algorithmic validation:
1. Topical Relevance Filter (sim qet > 0sm)) We compute cosine similarity between sentence embeddings of original and counterfactual queries. Queries falling below Asim 0.7 are dis- carded as off-topic, ensuring semantic coherence while allowing meaningful variation.
Answer Divergence Verification (C(q) + L(qa)): To programmatically verify distinct ground truths, we execute baseline RAG pipeline on both queries. Identical answers indicate mere paraphrasing rather than true counterfactual variation, triggering automatic rejection.
E.2.2 STAGE 2: HUMAN VALIDATION STUDY
To validate our automated filtering, we conducted targeted human evaluation study with rigorous methodology:
21

---

Published as a conference paper at ICLR 2026
Human Evaluation Protocol
Sample: query sets from HotpotQA development set
Annotators: Two independent expert evaluators
Metrics: S-point Likert scale for:
Relevance: Topical alignment with original query
Discriminability: Clarity in secking different information
Agreement: Cohen's 0.82 (substantial agreement)
Key Finding, 91 % of automatically filtered counterfactuals received scores > 4 on both criteria, validating our generation pipeline's effectiveness in constructing a high-quality dialectical evidence space.
E.3 COVERAGE AND DIVERSITY ANALYSIS
To empirically validate that our counterfactual generation achieves comprehensive coverage across semantic dimensions while maintaining answer divergence, we analyzed transformation patterns on the HotpotQA development set.
E.3.1 TRANSFORMATION DISTRIBUTION AND SEMANTIC PROPERTIES
Table presents the distribution of transformation types alongside their semantic characteristics, measured across 500 generated counterfactual sets.
Table 6: Distribution and properties of counterfactual transformations on HotpotQA. Average Se- mantic Distance measures sim(g ));; Answer Distance measures - IoU Aga) where IoU token overlap.
Transform Type
Usage Frequency
Avg Semantic Distance
Avg Answer Distance
Role Variation
28.4%
0.21
0.87
Entity Substitution
31.2%
0.18
0.91
Temporal Shift
15.7%
0.24
0.84
Categorical Change
19.3%
0.28
0.93
Scope Modification
5.4%
0.15
0.76
Overall Average
100%
0.21
0.86
E.3.2 INTERPRETATION AND VALIDATION
Balanced Coverage. The transformation distribution shows healthy diversity, with the five cate- gories collectively spanning role-based (28.4%), entity-based (31.2%), temporal (15.7%), categor- ical (19.3%), and scope-based (5.4%) variations. Entity Substitution and Role Variation dominate because they apply most broadly across question types, while Scope Modification appears less fre- quently due to stricter applicability constraints.
Semantic Proximity, Average semantic distances fall within the narrow range of **0.15-0.28**
corresponding to similarity scores of 0.72-0.85. This confirms that generated counterfactuals satisfy our topical coherence constraint (0.m 0.7) while introducing meaningful semantic variation. The consistency across transformation types (standard deviation 0.045) validates that our generation prompt maintains stable quality.
Answer Divergence. All transformation types achieve high answer distance (0.76-0.93). with over- all average of **0,86**, This creates the necessary "tension" for causal discovery: counterfactuals semantically proximate enough retrieve overlapping documents, yet target fundamentally dif- ferent answers. Categorical Change achieves the highest divergence (0.93), as expected from its design to probe opposite conceptual boundaries.
22

---

Published as a conference paper at ICLR 2026
Validation Against Design Goals. These metrics empirically confirm that counterfactual gen- eration (54.1.1) successfully operationalizes the validation criterion in Eq. 2:
simeem (g, xdd) > Osim A L(g) + L(f
The average semantic similarity of 0.21 = 0.79 exceeds Osim 0.7, while the high answer distance ensures distinct ground truths. Combined with the 91% human validation pass rate reported in $E.2.2, these results demonstrate that CF-RAG constructs high-quality dialectical evidence space.
F TRANSFORMATION TAXONOMY DETAILS
The prompt template for transformation generation follows structured format that maintains con- sisteney while allowing for comain-specific adaptations:
General Transformation Prompt Template
Original query: '(qıery)"
Traneformat ion type: (Gransforma onuuat egory) Generate sementica.lly similar query that prozes (specitic.ciinnciin) while mainteining domain relevance sucturall consistency. Brauze transtorned query hes dicterent groune-truth anaver than the original.
Transtormed query:
This appendix also provides detailed descriptions and examples for each of the five semantic trans- formation categories used in CF-RAG's counterfactual exploration phase.
ROLE-BASED TRANSFORMATIONS (TROLE)
These perturbations alter the functional role of entities while maintaining domain relevance, probing whether evidence is specific to particular roles or exhibits broader patterns across functional cate-
gories. The transformation preserves the query structure and context while changing the specific role being queried.
Examples:
"Who the lead actor in X?" - "Who played the villain in X?"
"Who directed X?" "Wo produced X?"
"What is the primary cause of Y?" - "What is contributing factor to Y?"
F.2 TEMPORAL-SHIFT TRANSFORMATIONS (TTME)
These modifications probe temporal boundaries and chronological causality by shifting time- sensitive elements to different periods or temporal contexts. They are particularly effective for testing whether evidence contains genuine temporal specificity or general patterns that could ap- ply across multiple time periods.
Examples:
"Who won the 2024 election?" + "Who won the 2020 election?"
"What is the current GDP China?" "What was the GDP China in 2020?"
"Which company leads the market today?" " "Which company led the market five years ago?"
F.3 ENTITY- SUBSTITTION TRANSFORMATIONS (TENTITY)
These replacements test whether evidence is specific to particular entities or exhibits broader pat- terns by substituting key entities with semantically related alternatives. This transformation helps identify whether retrieved documents contain entity-specific causal information general correla- tional patterns that apply to multiple similar entities.
Examples:
23

---

Published as a conference paper at ICLR 2026
"hhat is Apple's market cap?" "What is Microsoft's s market cap?"
"Where is Harvard located?" "Where is Yale located?"
"How does Tesla manufacture cars?" + "How does Ford manufacture cars?"
F.4
CATEGORICAL-INVERSIOO TRANSFORMATIONS (TeAr)
These explore opposites and categorical boundaries by inverting key categorical distinctions while preserving query structure. They designed test whether evidence supports specific categorical positions or contains general information that could support multiple positions within the same categorical dimension.
Examples:
"What is the highest mountain?" + "What is the lowest point?"
"Who was the first president?" - "Woo was the most recent president?"
"What are the benefits of X?" "What are the disadvantages of X?"
F.5 SCOPE-MODIHICTTION TRANSFORMATIONS (Tscope)
These adjust the breadth or specificity of the query by expanding or narrowing the scope of the queried relationship. They test whether evidence provides specific answers to focused queries or contains broader information that could support more general inquiries.
Examples:
"Who invented the telephone?" >> "Who contributed to telephone development?"
"What caused World War I?" + "What were the causes European conflicts in the early 20th century?"
"Which algorithm solves this specific problem?" "Which algorithms are used in this problem domain?"
G HYPOTHESIS GENERATION PROMPT TEMPLATE
The hypothesis generation process employs a carefully designed prompt structure that encourages thorough reasoning and causal analysis:
Hypothesis Generation Prompt Template
querys "(query)"
Based the provised evidencs documents: (evidence.cunset)
query detsiled rationale eipiainng your reasoni ng J. Identification of the key supporsing evidence Assesement st any ccatlicting informstion Focue evidence that specitically andresses the query rather than general topia relevance. inguing betxeen rect caussl support and indirect correlat patcerns.
provides ".
Ansver: {answer)
actionale: (rationalc)
H RELATIONSHIP To FORMAL CAUSAL INFERENCE
We address reviewers' questions regarding CF-RAG's relationship formal causal inference frame- works, particularly Pearl's Structural Causal Models (SCMs) (Pearl 2009).
H.1 TERMINOLOGICAL CLARIFICATION
CF-RAG employs counterfactual discriminative testing rather than structural causal modeling. We use "causal" mean:
24

---

Published as a conference paper at ICLR 2026
Causal discrimination: Identifying evidence that specifically determines answer, as opposed to evidence that merely correlates with it
NOT: Learning complete structural causal models or performing causal effect estimation
Our Definition 2 should be interpreted as specifying a discriminative criterion for evidence selection rather than a prescription for full causal modeling.
H.2 WHY FORMAL SCMS ARE INAPPROPRIATE HERE
Pearl's framework requires: (1) observable causal graphs, (2) repeated observations from the same
process, (3) ability to perform do(-) interventions, and (4) structured random variables. Open- domain RAG with unstructured text satisfies none of these requirements:
Text corpora contain natural language without explicit causal structure
Each query-document pair is unique; no repeated trials exist
We cannot intervene on data generation (cannot make Ledger play Batman to observe coun- terfactuals)
Documents are unstructured text, not structured variables
Moreover, the problems differ fundamentally:
SCMs:
Given causal graph G, estimate P(Y [do(X))
CF-RAG: Given query q and evidence & (no causal graph), discriminate causal from cor- relational evidence
H.3 THEORETICAL GUARANTEES WITHOUT SCMs
Despite not implementing SCMs, CF-RAG provides formal guarantees:
Volume Invariance (Lemma 2) The causal score is provably independent of evidence volume:
(OE, q, 2er) (E 4, Qar) Va:
(46)
Discriminative quality matters, not quantity of supporting documents.
Escape from Correlation Trap (Theorem 1) Under three conditions:
C1 Causal evidence discriminates: Ve E E", Vq' e Qct s(g,e) e()
C2 Spurious evidence doesn't: Ag E Qer with Ve E E': (s(g,e) - s(d e)] <
Separation:
CF-RAG provably selects causal evidence regardless of volume ratio /'1//E"| when À > X
Pruna Omia
--t--maaxx Canin
These conditions are empirically verifiable and provide actionable guarantees: when satisfied, overwhelming spurious evidence cannot mislead the system.
I CROSS-DATASET PARAMETER SENSITIVITY ANALYSIS
conducted systematic sensitivity analysis to evaluate whether our default hyperparameters (N=3, K=4, M=3, X=0.4) generalize beyond HotpotQA. Table five benchmarks.
7 presents optimal configurations across all
I.1 KEY FINDINGS
Default Parameters Are Near-Optimal Maximum improvement from dataset-specific tuning is only 1.35% (MusiQue). Two datasets (HotpotQA, TriviaQA with A-adjustment) achieve optimal or near-optimal performance with exact defaults, demonstrating strong generalization.
25

---

Published as a conference paper at ICLR 2026
Dataset
Optimal Config
Performance
vs. Default
Robustness
HotpotQA
N=3, K=4, M-3, A=0.4
88.58%
0.00%
12.3%
TriviaQA
N=3, K=4, M=3, A=0.35
81.42%
+0.40%
=1.9%
PopQA
N=2, K=3, M=3, X=0.45
74.35%
+0.78%
=2.7%
MusiQue
N=4, K=5, A=0.40
55.94%
+1.35%
#3.4%
PubHealth
N-2, K=3, M=3, A=0.45
84.21%
+0.85%
+ +1.8%
Table 7: Cross-dataset parameter sensitivity. "vs. Default" shows improvement using dataset- specific tuning versus uniform defaults. "Robustness" (+X%) indicates maximum performance variation when changing any single parameter by #1 from optimal.
Interpretable Parameter Patterns Simpler single-hop tasks (PopQA, PubHealth) require fewer counterfactuals (N=2) and coarser evidence clustering (K=3), while complex multi-hop reasoning (MusiQue) benefits from more extensive exploration (N=4, K=5). The consistency M=3 across all datasets suggests this represents fundamental algorithmic property rather than task-specific tuning.
Strong Robustness When varying any single parameter by =1 from optimal, performance fluctu- ates only +1.8-3.47 3.%% across datasets. This enables practical deployment without extensive per-task tuning while achieving within 1.4% of task-speeific optimal results.
I.2 PRACTICAL IMPLICATIONS
Our default configuration provides a robust starting point for diverse QA tasks. For deployment: (1) defaults for initial implementation, (2) adjust N, K based reasoning complexity (lower for simple lookup, higher for multi-hop), (3) fine-tune if domain exhibits unusual correlation patterns. The narrow performance gap between defaults and optimized configurations validates CF-RAG's transferability across domains.
ANSWER SYNTHESIS MECHANISM
This appendix details the synthesis process in Section 4.2.3, including the complete prompt template and concrete example.
J.1 SYNTHESIS PROMPT TEMPLATE
Synthesis Prompt Template
You are an expert analyst tasked with synthesizing information from
multiple perspectives.
Original Query: (query)
You have been provided with (k] candidate answers with their supporting rationales:
Candidate 1 (Score: (V1)):
Answer: {a.}
Rationale: {n)
Candidate (Score: (V2)):
Answer: (a2)
Rationale: (12)
Candidate 3 (Score: (Wa)):
Answer: {as)
Rationale: (ra)
Task: Synthesize these perspectives into a single, coherent
answer. When candidates agree, strengthen confidence through consensus. When they disagree, identify which evidence prevides the strongest causal support for the original query. Synthesized Answer:
Supporting Evidence:
26

---

Published as a conference paper at ICLR 2026
J.2 CONCRETE EXAMPLE
For the query "Who is the lead actor in The Dark Knight?", the synthesis process receives:
Input to Synthesis
Candidate (Score: 0.798): "Christian Bale"
Rationale:
Wikipedia identifies him as Bruce Wayne/Batman, the protagonist...
Candidate 2 (Score: 0.486): "Heath Ledger"
Rationale: Multiple reviews praise his Oscar-winning performance...
Candidate 3 (Score: 0.654): "Christian Bale"
Rationale: Director interview confirms casting Bale for Batman role...
Synthesized Output
Synthesized Answer: Christian Bale the lead actor in The Dark Knight.
Supporting Evidence: Two independent reasoning paths (Candidates and 3) converge Christian Bale with high causal discrimination scores, supported by both Wikipedia documentation and director interviews identifying him the protagonist Bruce Waync/Batman. While Candidate 2 highlights Heath Ledger's s acclaimed performance, this evidence pertains to the antagonist role rather than lead actor.
The synthesis mechanism prioritizes consensus among high-scoring candidates and explicitly iden- tifies when evidence addresses different aspects (protagonist vs. antagonist) of the query.
K EVALUATION ON COMPLEX REAL-WORLD BENCHMARKS
To evaluate CF-RAG beyond standard multi-hop QA, we conducted supplementary experiments on two challenging benchmarks: TruthfulQA (Lin et al. 2022), which tests resistance to adversarial misconceptions, and Strategy (Geva et al. 2021 which requires implicit multi-step common- sense reasoning.
K.1 PERFORMANCE COMPARISON
Table g presents comparative results across baseline methods and CF-RAG.
Method
TruthfulQA (% True Info)
Strategy QA (Accuracy)
Standard RAG
24.5%
31.2%
CRAG
29.8%
33.5%
Self-RAG
35.2%
38.7%
CF-RAG (Ours)
54.6 %
58.9%
Table 8: Performance on complex real-world benchmarks requiring adversarial robustness and im- plicit reasoning.
CF-RAG achieves substantial improvements: +55.1% over Self-RAG on TruthfulQA and +52.2% over Self-RAG on StrategyQA, demonstrating effectiveness beyond standard retrieval scenarios.
K.2 TRUTHFULQA: OVERCOMING THE RELEVANCE-TRUTE CONFLATION
Traditional RAG systems struggle TruthfulQA because they conflate semantic relevance with factual correctness. Consider the question "What happens if you break a mirror?" Standard re- trieval surfaces numerous documents discussing the "seven years of bad luck" superstition -thse documents are highly relevant to the query terms but factually incorrect.
Baseline systems fail different stages:
27

---

Published as a conference paper at ICLR 2026
Standard RAG retrieves misconception--eayy documents and generates answers based on volume
CRAG applies corrective filtering but validates superstition documents as "relevant," re- taining them
Self-RAG uses self-critique but lacks external grounding override retrieval-reinforeed misconceptions
Why CF-RAG succeeds: Categorical counterfactuals (Tea) probe boundary conditions by generat- ing queries about breaking similar objects (glass, ceramic, etc.). Evidence supporting the supersti- tion fails discriminative testing-it applies inconsistently across these related scenarios. This lowers the causal score Ocansal for miisconcetiion-supprri evidence, effectively filtering it despite high semantic relevance.
K.3 STRATEGYQA: BRIDGING IMPLICIT REASONING GAPS
StrategyQA requires synthesizing evidence across implicit reasoning steps that not explic- itly stated in queries. For example, answering "Could llama feasibly fit in kangaroo's ' pouch?" requires connecting: (1) llama size, (2) kangaroo pouch dimensions, and (3) physical constraints none directly mentioned together.
Baseline limitations:
Standard RAG retrieves documents about Ilamas and kangaroos separately but fails to synthesize
CRAG filters based on direct relevance, potentially discarding "bridge" evidence
Self-RAG depends on Llama-2-7B's internal commonsense knowledge; when this is insuf- ficient, self-critique cannot identify which retrieved evidence bridges reasoning gaps
Why CF-RAG succeeds: Parallel Arbitration constructs multiple hypotheses from stratified evi- dence subsets. The causal discrimination mechanism (Equation 11) performs comparative evalua- tion across these parallel paths, identifying evidence that causally necessary for connecting rea- soning steps even when individual documents appear only tangentially relevant. This comparative process reveals "bridge" evidence that single-path systems miss or discard.
L SCALABILITY TO DIVERSE MODEL ARCHITECTURES
CF-RAG is designed to be architecturally agnostic, operating through inference-level interventions rather than requiring access to model internals. This section empirically validates our framework's scalability beyond the Llama family reported in the main paper.
L.1 ARCHITECTURAL AGNOSTICISM: DESIGN PRINCIPLES
CF-RAG's scalability stems from three key design principles:
Counterfactual generation (Section 4.1) relies solely on standard prompting to produce semantic variations, compatible with any instruction-tunee LLM.
Input-Level Intervention:
Scoring Independence: The critical causal discrimination score Deauall (Eq. 11) and coherence score @coh (Eq. 9) utilize auxiliary cross-encoders and embedding models, independent of the generative LLM's internal parameters.
Black-Box Compatibility: While leverage token probabilities for the confidence component Φg. the framework remains robust with API-based models. The causal and coherence scores carry the majority of the arbitration signal weight.
28

---

Published as a conference paper at ICLR 2026
L.2 EMPIRICAL VALIDATION: GPT-40 AND MISTRAL
We conducted supplementary experiments on HotpotQA (distractor setting) using two architec- turally distinct models: the state-of-the-art GPT-4o and the open-weight Table 9 presents the results.
Table 9: CF-RAG performance across diverse architectures on HotporQA. Standard RAG baseline shows that even GPT-4o suffers from the Correlation Trap.
Model Architecture
Method
EM (%)
Relative Gain
Standard RAG
35,12
CF-RAG (Ours)
82.45
+134.7% %
Standard RAG
64.80
GPT-4o
CF-RAG (Ours)
91.20
+40.7%
L.3 KEY FINDINGS
Universality of the Correlation Trap. Even GPT-4o, despite its superior reasoning capabili- ties, achieves only 64.8% baseline accuracy, confirming that stronger parametric knowledge can- not fully override overwhelming spurious retrieval signals-a fundamental limitation that CF-RAG addresses.
Consistent Crost-Architecture Gains. CF-RAG delivers substantial improvements across both model scales and families:
For Mistral-7B, the +134.7% gain mirrors our Llama-2-7B results (Table 1), validating effective- ness for smaller open-weight models.
For GPT-40, CF-RAG achieves near-ceiling performance (91.2%), demonstrating that stronger instruction-following capabilities amplify benefits through higher-quality counterfactual genera- tion and rationale synthesis.
Synergy with Model Capability. The results reveal an important pattern: more advanced instruction-tuned architectures generate more precise counterfactual probes and coherent rationales, thereby enhancing rather than diminishing CF-RAG's causal arbitration effectiveness. This posi- tive scaling behavior suggests that CF-RAG will continue to provide value foundation models improve.
M EMPIRICAL VALIDATION: COUNTERFACTUAL RELEVANCE AS INVERSE PROXY FOR CAUSAL SUPPORT
To empirically validate our core assumption that counterfactual relevance inversely correlates with causal support strength, conducted targeted correlation analysis the HotpotQA development set using Llama-2-7B.
M. EXPERIMENTAL DESIGN
Counterfactual Relevance Ratio (CRR). We define a diagnostic metric quantify evidence dis- criminability:
max 5(g',E)
CRR
(47)
8(g,E)
where s(; .) denotes relevance score. The ratio captures semantic specificity:
CRR x 1.0: Evidence supports counterfactuals as strongly as the original query (spurious corre- lation)
CRR 1.0: Evidence is highly specific the original query (causal support)
29

---

Published as a conference paper at ICLR 2026
Sampling and Evaluation. We analyzed 500 queries from HotpotQA dev stratified into four CRR intervals based on the natural distribution. For each group, we measured Exact Match (EM) accuracy using standard RAG with Llama-2-78
M.2
RESULTS AND ANALYSIS
Table 10 presents the relationship between counterfactual relevance and model performance.
Table 10: Correlation between Counterfactual Relevance Ratio (CRR) and model accuracy on Hot- potQA. Higher CRR indicates evidence cannot discriminate between original and counterfactual queries, leading to systematic failures.
Metric
Low CRR (Causal)
Moderate
High CRR
Very High (Trap)
Relevance Ratio
0.5
0.5-0.7 26%
0.7-0.9 15%
> 0.9
Sample Share
53%
6%
Accuracy (EM)
90.2%
76.4%
55.8%
31.3%
Error Rate
9.8%
23.6%
44.2%
68.7% %
Sharp Performance Degradation. Model accuracy exhibits monotonic decline CRR increases.
When evidence lacks discriminative power (CRR >> 0.9), the system fails in nearly 70% of cases direct manifestation the Correlation Trap. Conversely, when evidence strongly discriminates the original query from counterfactuals (CRR < 0.5), accuracy reaches 90.2%.
Strong Negative Correlation. Computing Pearson correlation between CRR interval midpoints and accuracy yields & -0.95, indicating near-perfect inverse linear relationship. This strong negative correlation empirically confirms that:
1. Counterfactual relevance serves precise inverse proxy for causal support strength
Our causal discrimination score deausal (Eq. 11) captures a fundamental dimension of evidence quality
3. The theoretical motivation in Section 5 empirically well-founded
These findings validate CF-RAG's core mechanism: by quantifying how evidence distinguishes original queries from counterfactual alternatives, framework directly measures causal support rather than relying on correlation-banl relevance alone.
N COMPUTATIONAL EFFICIENCY ANALYSIS
While CF-RAG introduces additional computational components (counterfactual generation, par- allel hypothesis construction), its architecture prioritizes parallelism over the iterative processing employed by advanced baselines. This section provides comprehensive efficiency benchmark.
N. EXPERIMENTAL SETUP
We evaluated wall-clock latency and throughput the HotpotQA benchmark using Llama-2-1 2-7B deployed across 4x NVIDIA A100 80GB GPUs. All methods used identical retrieval corpora and hardware configurations. We measured average latency 1,000 queries with batch size 1 to reflect realistic interactive usage.
N.2 PERFORMANCE RESULTS
Table 11 presents comparative efficiency metrics across RAG frameworks.
30

---

Published as a conference paper at ICLR 2026
Table 11:
Computational efficiency comparison on HotpoiQA (Llama-2-7B, 4x A100 GPUs). La-
tency measured in seconds per query, throughput in queries per second. CF-RAG achieves compet- itive latency through parallel processing.
Method
Architecture
Latency (s)
Throughput (q/s)
Relative Speed
Standard RAG
Single Pass
2.05
23.8
1.0x
Self-RAG
Iterative
4.72
10.6
2.3x
CF-RAG
Parallel
2.92
17.1
1.4x
N.3 ANALYSIS AND COMPUTATIONAL BREAKDOWN
Parallelism vs. Iteration. CF-RAG processes more tokens (due to M-3 parallel hypothesis drafts) but achieves only 1,4x latency overhead compared to Standard RAG's single-pass baseline. In con- trast, Self-RAG's sequential reflection mechanism incurs 2.3x latency penalty. Our parallel architec- ture leverages multi-GPU resources effectively, maintaining 17.1 queries/second throughput- 1.6x faster than Self-RAG while delivering substantially higher accuracy (Table 1).
Detailed Latency Breakdown (CF-RAG). The 2.92s total latency decomposes follows:
Counterfactual Generation: 0.42s (batched LLM calls for N-3 queries)
Dialectical Retrieval: 0.28s (batched dense retrieval with FAISS)
Parallel Hypothesis Generation: 2.10s (72% of total; M-3 drafts executed concurrently across
GPUs)
Arbitration Scoring: 0.1 (vectorized cross-encoder operations)
Scalability Trade-offs. For applications requiring maximum throughput at the cost of accuracy, CF-RAG supports reduced- reduced-parameter configurations (N=1, M) that achieve 2.2s latency (1.07x overhead) while maintaining 80%+ of the full accuracy gains (Figure 3). This flexibility enables deployment across diverse computational budgets.
The results demonstrate that CF-RAG achieves strong performance improvements without pro- hibitive computational costs, making practical for production deployment.
O STABILITY ANALYSIS: ROBUSTNESS TO LLM STOCHASTICITY
To address concerns about LLM generation controllability affecting CF-RAG's performance, we conducted rigorous variance analysis across multiple independent runs.
O.1 EXPERIMENTAL PROTOCOL
We executed the complete CF-RAG pipeline 5 times the HotpotQA test set (500 queries) using Llama-2-7B with different random seeds. Each run used identical hyperparameters (N=3, K K-4, M-3, A=0.4) but varied LLM sampling for counterfactual generation, hypothesis construction, and answer synthesis. We measured Exact Match (EM) accuracy and computed standard deviation (a) and coefficient of variation (CV = o/p) to quantify stability.
O.2 RESULTS
Table 12 presents performance variance across pipeline configurations.
O.3 ANALYSIS AND INTERPRETATION
High Inherent Stability. The full CF-RAG pipeline achieves remarkably low variance (a 0.73%), with coefficient of variation CV < 1%. This stability is comparable to deterministic retrieval systems and substantially better than typical LLM generation variance (CV 3-5% for
31

---

Published as a conference paper at ICLR 2026
Table 12: Performance stability across independent runs on HotpoiQA (Llama-2-7B), Low coefii- cient of variation (CV < 1%) demonstrates robustness to LLM stochasticity.
Component
Mean EM (%)
Std Dev (σ)
CV (%)
Full CF-RAG Pipeline w/o Consensus Check
79.29
+0.73
0.92
66.84
+1.85
2.77
open-ended tasks). The tight confidence interval demonstrates that CF-RAG's performance is driven by systematic architectural advantages rather than fortunate sampling.
Consensus Mechanism as Variance Damper. Removing the consensus check (Algorithm 1, lines 15-16) causes a 3x increase in variance (CV : 0.92% + 2.77%). validating its stabilizing role. The consensus mechanism operates follows: when multiple parallel hypotheses {ay [رy) verge (i.e., Agreement(a,) > O.gree). the system locks in the majority answer, effectively averaging out stochastic noise from individual generation paths. Without consensus, the system relies solely arbitration scores, which more susceptible randomness in token-level sampling during hypothesis generation.
Structural Variance Reduction. CF-RAG's low variance stems from three design principles:
1. Evidence-grounded generation: Hypotheses are conditioned on retrieved documents, constrain- ing output space
Multi-path redundancy: M-3 parallel drafts provide statistical robustness through ensemble effects
3. Score-based arbitration: Causal discrimination (deawsa) relies on deterministic cross-encoder scores rather than LLM logits
These results demonstrate that despite involving LLM generation at multiple stages, CF-RAG main- tains high controllability through architectural choices that structurally reduce variance.
32


Reviews:
[{"review_id": "R8amb7TvbQ", "reviewer": "Reviewer_VGky", "summary": "This paper proposes CF-RAG (Counterfactual Retrieval-Augmented Generation), a framework that introduces causal reasoning into RAG systems to overcome what the authors call the Correlation Trap—situations where models are misled by highly correlated but non-causal evidence.\nThe key idea is to use counterfactual exploration (systematically generating alternative queries probing semantic boundaries) and parallel arbitration (maintaining separate reasoning paths and comparing them via coherence and causal discrimination scores).", "questions": "How sensitive is performance to the choice of $K$, $M$, and $\\lambda$ across datasets beyond HotpotQA?\n\nCould causal discrimination $\\phi_{\\text{causal}}$ be learned end-to-end rather than fixed by hand-crafted similarity differences?\n\nHow does CF-RAG behave when the ground-truth causal boundary is ambiguous or overlapping (e.g., multiple protagonists)?\n\nDoes the framework generalize to retrieval-augmented dialogue or code generation where counterfactuals are less well-defined?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s main strength lies in its novel causal framing, which introduces counterfactual reasoning into retrieval-augmented generation (RAG) in a principled and well-motivated way, effectively bridging causality and retrieval. It further demonstrates strong empirical performance, showing large and consistent gains across five diverse QA benchmarks, including multi-hop and long-tail reasoning tasks. Another notable advantage is the framework’s robustness and interpretability, as CF-RAG shows clear improvement in handling distractors and noisy evidence while producing interpretable arbitration scores that clarify its decision process. Finally, the paper presents a comprehensive experimental analysis, incorporating ablation studies, parameter sensitivity evaluations, and theoretical proofs that establish discriminability and volume invariance, all of which contribute to the credibility and completeness of the work.", "weaknesses": "The most significant issue is its limited causal formalism—although it uses the language of causality, the proposed mechanism does not implement structural causal modeling, and its causal discrimination is ultimately based on heuristic similarity differences rather than learned or theoretically grounded causal inference. The method also shows dependence on retriever quality, as its counterfactual reasoning performance depends heavily on the retrieval coverage and quality of semantic clustering; potential failure cases such as missing or ambiguous causal boundaries are not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vz18VfREbP", "reviewer": "Reviewer_XTvH", "summary": "This paper presents CF-RAG. CF-RAG targets the Correlation Trap in RAG and reframes retrieval as causation-driven reasoning via counterfactual exploration plus parallel arbitration that compares evidence across original and counterfactual queries. The authors conclude that this design discriminates causal from merely correlated evidence and delivers state-of-the-art results on various QA benchmarks with especially large gains on complex multi-hop tasks while maintaining efficiency comparable to baselines.", "questions": "1. Could you give more insight why you did eigendecomposition on the affinity matrix? How did you use the eigenvectors for Kmean? Current version is a bit lost.\n2. Section 4.2.3. How did you do synthesis? What is the prompt? Could you provide examples?\n3. A general question. Question in datasets like HotpotQA should have been seen during pre-training of LLMs. What is the reason in your opinion that they still need advanced RAG techniques to be answered?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is clearly motivated and well explained.\n2. The paper is easy to follow and the appendix is very detailed and helpful.\n3. The proposed method demonstrates great improvement compared with previous methods.", "weaknesses": "Not really some major weaknesses. Please see my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HcF0tB1IfL", "reviewer": "Reviewer_mJsp", "summary": "The paper proposes a framework for counterfactual reasoning using large language models (LLMs). It introduces a causal decomposition method that disentangles factual and counterfactual components in LLM representations. Specifically, the model leverages prompt-based intervention and latent projection techniques to simulate \"what-if\" scenarios—e.g., modifying causes while holding other factors constant.\nExperiments on several benchmarks (including bAbI, Counterfactual NLI, and causal QA datasets) show that the proposed model outperforms standard prompting and contrastive fine-tuning approaches. The authors claim their framework better preserves causal consistency and counterfactual faithfulness across tasks.", "questions": "1) How does your “intervention” differ from representation editing or latent steering in prior causal LLM papers?\n\n2) How are causal variables defined in token or hidden-space terms?\n\n3) How scalable is the framework to models beyond the tested LLM (e.g., GPT-style or instruction-tuned architectures)?\n\n4) Could you provide examples/case studies where the counterfactual generation fails and why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Timely and important topic: Counterfactual reasoning is crucial for causal interpretability of LLMs, aligning well with growing interest in trustworthy AI and reasoning beyond correlation.\n\n2) The use of intervention-based prompt transformations and latent disentanglement is grounded in causal inference theory, referencing Pearl-style structural models.\n\n3) Covers a good range of datasets and compares with both prompting-based and fine-tuning-based baselines.\n\n4) This paper is readable and well-organized with clear structure, intuitive figures (the causal diagram and latent-space illustration help convey the method), and well-written theoretical framing.", "weaknesses": "1) The approach seems to be a combination of known components such as prompt intervention, representation projection, and contrastive causal fine-tuning. While elegantly integrated, it doesn’t introduce a fundamentally new algorithmic idea. Several recent works (e.g., CFPrompt, LoCaR, CausalBench) already explore counterfactual prompting or causal subspace manipulation.\nThe paper’s core mechanism (modifying hidden states via intervention vectors) is highly similar to these existing frameworks.\n\n2) The causal formalism (Section 3) refers to do-calculus, but the LLM interventions are not explicitly tied to a formal SCM or identifiable causal model.\n\n3) Most benchmarks (bAbI, CF-NLI, synthetic causal QA) are simplistic or low-dimensional. There’s no test on real-world or high-stakes reasoning (e.g., factual contradictions in QA or commonsense counterfactual reasoning like ATOMIC-2020). \n\n4) The model enforces counterfactual consistency in latent space, but it’s unclear whether generated counterfactuals remain semantically plausible. The paper reports factual accuracy but no human or automatic evaluation of counterfactual realism.\n\n5) Missing comparisons with recent methods that explicitly model causal subspaces or perform latent intervention (e.g., CausalLM, LoCaR, CFPrompt). The ablations only test within-model variants, so the real performance advantage remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LnHXUP7Cq0", "reviewer": "Reviewer_iXQG", "summary": "The paper tackles the “correlation trap” in RAG—highly correlated but non-causal evidence drowning out the truly decisive evidence. It proposes counterfactual exploration: generate counterfactual questions around the original query and retrieve for both original and counterfactuals to build a “dialectical evidence space”; and parallel arbitration: theme-cluster the evidence, sample per-cluster subsets to form multiple parallel evidence paths, generate “answer+reason” per path, then rank with Internal Coherence + Causal Specificity. Experiments show gains over strong baselines with ablations and case studies.", "questions": "1. In the Causal Discrimination Scoring, how is the relevance s(⋅)s(\\cdot)s(⋅) computed? If it relies on a retriever/reranker, is there a better approach to avoid circular dependence?\n2. Is there experimental evidence about the relationship between counterfactual relevance and causality/support—specifically, that higher relevance to counterfactuals truly implies weaker support for the original question?\n3. Are passages judged as non-causal (or counterfactual-leaning) necessarily useless?\n4. Given the many components and the involvement of LLMs in some stages, can you add an efficiency analysis?\n5. How accurate is the counterfactual construction, and is there any evaluation of its coverage, diversity, and quality?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed in this paper is important and directly targets the common correlation trap in RAG.\n2. The methodological perspective is novel: assessing the usefulness of retrieved results via their relevance to counterfactuals is a very interesting and quite reasonable idea.\n3. The empirical evaluation is fairly solid, with experiments on multiple datasets, strong baseline comparisons, ablations, and qualitative case studies, and the gains on multi-hop tasks are especially notable.", "weaknesses": "1. Although the paper identifies the correlation trap and introduces counterfactuals for verification, its final decision still relies on the relevance between evidence segments and the original answer or the counterfactuals. In other words, the authors posit that retrievers cannot distinguish relevance from usefulness and design a method to address this, but the method ultimately still measures usefulness via the retriever’s similarity scores (I did not find a description of how similarity is computed in the Causal Discrimination Scoring; I therefore assume it uses a trained retriever or reranker), which I find somewhat circular. I agree with the high-level idea for tackling the correlation trap, but I believe evaluating causality/support may require a better approach than falling back again on similarity signals.\n2. Even if we accept the assumption that contrasting relevance for the original question versus counterfactuals can approximate causality/support, the score is computed at the passage level. Given that passages often contain mixed signals, this can easily suppress span-level evidence that truly supports the answer.\n3. The paper lacks some experiments on the coverage and quality of counterfactual construction and on efficiency and see Questions for details.\n4. Several components of the method depend on LLMs, and the controllability of LLM generation in these stages may affect the method’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)