On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
Authors: Shuai Yang
Qi Yang
Luoxi Tang
Yuqiao Meng
Nancy Guo
Jeremy Blackburn
Zhaohan Xi
Venue: ICLR 2026 Poster (2026)
Keywords: Counterfactual Reasoning
Large Language Models (LLMs)
Experimental Study
Primary area: datasets and benchmarks
TL;DR: In-depth understanding of LLMs' counterfactual reasoning eligibility

Abstract: Counterfactual reasoning has emerged as a crucial technique for generalizing the reasoning capabilities of large language models (LLMs). By generating and analyzing counterfactual scenarios, researchers can assess the adaptability and reliability of model decision-making. Although prior work has shown that LLMs often struggle with counterfactual reasoning, it remains unclear which factors most significantly impede their performance across different tasks and modalities. In this paper, we propose a decompositional strategy that breaks down the counterfactual generation from causality construction to the reasoning over counterfactual interventions. To support decompositional analysis, we investigate 11 datasets spanning diverse tasks, including natural language understanding, mathematics, programming, and vision-language tasks. Through extensive evaluations, we characterize LLM behavior across each decompositional stage and identify how modality type and intermediate reasoning influence performance. By establishing a structured framework for analyzing counterfactual reasoning, this work contributes to the development of more reliable LLM-based reasoning systems and informs future elicitation strategies.

Published as a conference paper at ICLR 2026
ON THE ELIGIBILITY OF LLMs FOR COUNTERFACTUAL REASONING: A DECOMPOSITIONAL STUDY
Shuai Yang'
Qi Yang²
Luoxi Tang'
Yuqiao Meng'
Nancy Guo
Jeremy Blackburn' Zhaohan Xi
Binghamton University zxi wewbinghmttonnede
SShanhai University
ABSTRACT
Counterfactual reasoning has emerged as crucial technique for generalizing the reasoning capabilities of large language models (LLMs). By generating and analyzing counterfactual scenarios, researchers can assess the adaptability and reliability of model decision-making. Although prior work has shown that LLMs often struggle with counterfactual reasoning, it remains unclear which factors most significantly impede their performance across different tasks and modalities. In this paper, we propose a decompositional strategy that breaks down the counterfactual generation from causality construction to the reasoning over counterfactual inter- ventions. To support decompositional analysis, we investigate 11 datasets spanning diverse tasks, including natural language understanding, mathematics, program- ming, and vision-language tasks. Through extensive evaluations, we characterize LLM behavior across each decompositional stage and identify how modality type and intermediate reasoning influence performance. By establishing a structured framework for analyzing counterfactual reasoning, this work contributes the development of more reliable LLM-based reasoning systems and informs future elicitation strategies.
INTRODUCTION
Large language models (LLMs) have exhibited remarkable proficiency across diverse range of tasks, including natural language understanding (Devlin et al. 2019 Kuang et al. 2025), multimodal reasoning (Hu et al. 2017 Lu et al. 2022] Yang et al. 2[23]]
[0ho]Zhuu Zhou et al. 2024]] Tang et al. 2025b), and
domain- -snaifc generation (Desai et al.) 2026 Kalvakuntla et al.) 2026 Tang et al. 2025a). Despite these advancements, concerns persist regarding their reasoning and generalization capabilities. A particularly challenging aspect of model evaluation is Counterfactual Reasoning, i.e., the ability adjust responses when presented with modified premises (Pearl & Mackenzie 2018) (e.g.. What is the outcome in hypothetical condition?). Investigating the counterfactual reasoning of LLMs provides an interpretable step to understand their adaptability under hypothetical alterations to input conditions (Gat et al. 2024 Huang et al. 2024).
Prior studies have demonstrated that LLMs often struggle with counterfactual reasoning and fre- quently fail to maintain logical consistency or adjust to context shifts (Li et al. 2023 Nguyen al. 2024. Wang et al. 2024). While these works highlight notable performance gaps, they lack standardized framework for systematically analyzing and understanding counterfactual behaviors in I.LMs. Consequently, it remains unclear what factors most significantly impact L.LM performance in counterfactual scenarios. Furthermore, counterfactual reasoning has often been evaluated in direct and monolithic manner, primarily by introducing interventions and assessing model responses (Li et al. 2023). without grounding the analysis in the underlying causal structure that gives rise such interventions. This overlooks the foundational role causal modeling. Specifically, the identification of causal variables and their dependencies essential for understanding counterfactuals.
To address these gaps, we are motivated by the structural formulation of counterfactual reasoning under the Structural Causal Model (SCM) formalism (Pearl 2009), in which counterfactual reasoning

---

Published as a conference paper at ICLR 2026
Example
Causal Graph (Factual)
Causal Graph (Counterfactual)
Browsing Nistrry recommendation
Eocasing listtloliooootoi
purchasee evaluste how persoraliee product con influence buying behavior.
Browsing Nistory Punchwee Itelhood
Browsing History Purchase itelhood
Recommencaatoos - Time spent on the wabpage
No Recommendiriooe -n (7) Time spent on welpaage (?) Time spent wulpaage (?) Purchase ikalhood
Time spant the wibpage - Purchase Ikathood
1. Variable
2. Causality
3. Intervention Identification
4. Outcome
Counterfactual Output
Input Context
Identification
Construction
Reasoning
Variables
Counterfactual
Output Spent recommenniaion vecpaage
Expasure:
Covariate: brawing history
pereoralioed recommendationn
Mediator: Time spent on the webpage
Original Exposure: recevved recomme soat
Purchase (Outcomal: Less like due the recuced time and lock recommnnaation
Outcome: if the user made a purchase
Counterfactual: did not receive recommendiiion
Figure 1: A workflow and illustrative example that decomposes LLM-based counterfactual reasoning into four stages: (1) identifying causal variables (e.g., whether web recommendation is shown), (2) constructing the causal graph (e.g., browsing history a recommendation is shown). (3) specifying the counterfactual intervention (e.g., no recommendation shown), and (4) reasoning about the coun- terfactual outcome (e.g., less likely to purchase product online).
must proceed through sequence of regularized steps including inferring latent variables from observations, modifying the SCM via intervention, and computing the updated outcome. Accordingly. we outline Decompositional Strategy that breaks down the analysis of counterfactual reasoning into distinct stages. Our approach departs from prior work that focuses solely on counterfactual generation. Instead, we begin by examining the causal structure of factual conditions, which serves as the necessary foundation for valid counterfactual reasoning.
As illustrated in Figure [ our methodology is outlined into four stages. First, we assess (i) whether LLMs can accurately identify the four variable groups critical to causal reasoning: Exposure, Co- variate, Mediator, and Outcome. Next, we evaluate (ii) whether LLMs can correctly construct a corresponding causal graph in the form of directed acyclic graph (DAG). Building this causality modeling, we then study LLMs' counterfactual reasoning abilities by evaluating (iii) whether they identify the correct intervened variable (i.e., the Exposure), and (iv) whether they accurately infer the counterfactual mediators and final outcomes by reasoning over the updated causal graph.
To support our decompositional study, we construct benchmark by collecting and curating 11 counterfactual datasets across diverse tasks, including natural language understanding. mathematics, programming, and vision- languaee reasoning. We curate each dataset by extracting factual and counterfactual variables, identifying causal elements, and constructing corresponding causal graphs as reference structures for evaluation purpose. In experiments, we test the performance of leading LLMs across each decompositional stage to analyze their sufficiency in handling individual reasoning components. Based on the observed performance among these decomposed evaluations, we propose targeted improvements, such as integrating modality-speciee augmented learning paradigm, to address critical reasoning bottlenecks. Additionally, we evaluate impact of different elicitation (prompting) strategies, including Chain-of- Thought (CoT) (Wei et al., 2022), Chain-of- Chain-of-Thought with Celf-Connsstency (CoT-SC) (Wang et al., 2022), and Tree-of- Thought (ToT) (Yao et al. 2023) reasoning. Collectively, evaluations provide a solid step for understanding and enhancing [.MMs in complex reasoning tasks and imaginative scenarios.
function-calling interfaces within tool-
In summary, make the following contributions:
Decompositional Framework -We propose a decompositional strategy that spans from causal modeling to counterfactual reasoning, enabling systematic evaluation of LLMS' capabilities in understanding and performing counterfactual tasks.
Benchmark Construction-Ve We construct a comprehensive evaluation benchmark by curating causal structures and counterfactual instances across multiple domains. This benchmark standard- izes decompositional evaluations and supports consistent analysis across tasks and modalities.
Evaluation and Improvement Strategy-Wo We evaluate leading LLMs under diverse tasks. By identifying LLMs' capabilities in specific decompositional stage, we propose actionable strategies to improve LLMs" counterfactual adaptability.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Counterfactual Reasoning. A fundamental component of causal inference is to examine hypotheti- cal scenarios, which addresses the question: What would have occurred had a particular factor or decision differed? (Pearl & Mackenzie 2018) This method facilitates causal analysis by compar- ing observed outcomes with those projected under alternative conditions. Empirical research has demonstrated the broad applicability of counterfactual reasoning across multiple domains, including healthcare, business, and fairness (Gvozdenović et al. [022]] Kyrimi et al. 2025 Gow et al. 2016 Kasirzadeh & Smart 202T Koonce et al. (011). This stands in contrast to many contemporary Al systems, which predominantly rely on statistical correlations while lacking robust capacities for abstract reasoning and causal inference (Jiao et al. 2024).
Counterfactual Reasoning in AI and NLP. Counterfactual reasoning has emerged as a powerful framework for enhancing model interpretability and causal understanding in AI and NLP. In medical AI, SyncTTwin (Qian et al. 2021) proposed counterfactual estimation framework that constructs syn- thetic patient data to predict potential outcomes under alternative treatments. NL.,, a Counterfactual Reasoning Model (CRM) (Feng et al. 2021 is developed using L.MSs to generate contrastive samples, improving sentiment analysis and inference tasks. There are also order- fatthulaitsfullness metries (Gat et al.) 2024) to evaluate causal explanations in black-box models. These contributions demonstrate the versatility of counterfactual methods in improving model transparency and reliability across domains.
LLMs and Elicitation. Recent advances have significantly enhanced LLMs' reasoning capabilities through several elicitation (prompting) approaches. The introduction of Chain-ofi of Th-uhht (CoT) prompting (Wei et al., 2022) and its extensions, the Self-Consistency CoT (Wang al. (022)) and Tree-of- Thought (ToT) (Yao et al. 2023), have enabled more structured and reliable multi-step reasoning. These innovations collectively represent paradigm shift from simple pattern recognition to deliberate, verifiable reasoning in LLMs (Meng et al. 2025 Liu et al. 2025).
Evaluation of Counterfactual Reasoning. Evaluating counterfactual reasoning in LLMs has garnered growing attention. However, most prior works assess counterfactual through end-to-end evaluations such as contrastive counterfactuals (e.g.. "What would happen if X didn't 't occur?") (Huang et al. 2024 Frohberg & Binder 2021 Zhang et al. 2024b; | Le et al., (023). Other studies such as MalAlgoQA (Sonkar et al.] (024) introduce the concept of algorithms assess LLMs' ability to reason about flawed hypothetical paths. Their setup focuses on identifying distractor rationales in
multiple-choice formats, revealing L.M struggles in understanding student misconceptions Other
efforts like DICE (Shrivastava & Aoyagui 2025) and CausalProbe (Chi al. 2024) create diagnostic benchmarks to evaluate causal sensitivity or counterfactual faithfulness in static question-answering formats, without decomposing the reasoning process into interpretable modules. Similarly, synthetic datasets have been used to analyze RNN inductive biases in agreement prediction (Ravfogel et al.) 2019) but do not extend to structured counterfactual inference tasks. Compared to these works, our study introduces a modular evaluation aligned with Pearl's structural causal model (Pearl & Mackenzie] 2018). We decompose counterfactual reasoning into four interconnected sub-tasks to open up fine-grained attribution of model failures and provide a more diagnostic and interpretable assessment of LLM reasoning capabilities.
3 METHODOLOGY: DECOMPOSING COUNTERFACTUAL REASONING
This section presents our methodology for decomposing counterfactual reasoning. We begin by intro- ducing foundational concepts in causality and counterfactual reasoning (Section 3.I. Subsequently, detail the evaluation tasks used to assess models (Section 3.2 and our corresponding construction of benchmarks over multimodal datasets (Section 3.3
3.1 PRELIMINARY: FROM CAUSALITY TO COUNTERFACTUAL REASONING
Causality. Causality depicts the dependencies about how one variable influences another, i.e., the underlying causal effects. There are four types of variables commonly used in causal analysis: exposure, covariate, mediator, and outcome. Specifically: (1) Exposure (or treatment, intervention, denoted X) refers to the action or condition imposed on a system; (2) Outcome (Y) denotes the resulting response effect influenced by the exposure; (3) Covariate (Z) is the pre-treatment variable
3

---

Published as a conference paper at ICLR 2026
Z Low Income
Low income
M
Y
M
Study time
score
hours/week
Soore 70
Intervention
houra/week
Score
X Not Received Tutoring
X Received Tutoring
(a) Causal Graph Structure
(b) Factual Graph
(c) Counterfactual Graph
Figure 2: (a) Causal graph structure and (b)() A factual/counteeractuual example.
that may influence both X and Y: (4) Mediator (M) lies the causal pathway from X to Y, representing intermediate mechanisms through which the exposure exerts its influence.
Example Consider dataset that records students' academic performance in the presence of tutoring tool. Here, the exposure X indicates whether a student used the tool. The outcome Y corresponds to the student's is final exam score. The covariate Z may include socioeconomic factors (e.g.. parental income), which could influence both the use tutoring and academic outcomes. The mediator M is the number of hours the student spends studying per week.
The relationships among exposure, covariate(s), mediator(s), and outcome(s) can be formally represented using directed acyclic graph (DAG), commonly named causal graph (Pearl& & Mackenzie, 2018) that captures causal relationships, where the exposure X influences the outcome Y both directly and indirectly through mediator M. Covariate Z may affect X, M, and Y. as illustrated in Figure 2
Causal Graph.
Example 2 The corresponding causal graph, illustrated in Figure (a). would include: an arrow from X + M (e.g.. tutoring influences study time), M + (study time influences exam performance). X + Y (direct effect of tutoring on scores), Z + X and Z + Y (c.g.. socioeconomic status affects both tutoring usage and academic performance).
Counterfactual Reasoning. Counterfactual reasoning aims to answer:
Given an observed instance (X = x, Z = 2. M = m, Y = y), what would the outcome Y be if the exposure X were set a diffferent value z', while keeping the covariate Z fixed?
The observed instance (x, 2, m, y) is also known the factual case. In contrast, counterfactual reasoning seeks to determine the outcomes under an alternate intervention, that is, when X = x'.
In causal graph, we assume a two-stage causal mechanism: (1) Mediator Function: M (X, Z) and (2) Outcome Function: Y = (X, M, Z). Then, the counterfactual outcome under an alternative exposure x' can be computed via: Y. fr (a', (M' 2). 2), where we simulate new mediator value M = fM (x', 2) based on the counterfactual exposure z' and observed covariate 2. Then, we predict the counterfactual outcome Y,, using the counterfactual exposure x', the simulated mediator Myr, and the same covariate z.
Example As exemplified in Figure (b)), student with LOW-INCOME socioeconomic status did not receive tutoring (x = 0), studied hours per week (m = 5), and scored 78 on the exam (y- 78).
We now ask: What would the student's 's score have been if they had received tutoring (z' = 1)? We compute: (i) Simulated study time: m' fM(x' = 1, LOW-INCOME) 9 and (ii) counterfactual score: Y =fy (x' = m' = 9, z LOW-INCOME) 85.
We conclude: The lutoring would have increused the student's score from 78 to 85 (Figure (c)).
3.2 DECOMPOSITIONAL EVALUATION TASK
Counterfactual reasoning is often described structured chain of analysis from identifying vari- ables to modeling causal relations, specifying interventions, and simulating outcomes (Pearl) 2009 Bareinboim al. 2022). Recent work in MI. and LLMs further emphasizes the importance of
4

---

Published as a conference paper at ICLR 2026
Table I: Summary of counterfactual benchmarks including data source, use case, presence of causal variables present, C: partially present), the definition of counterfactual condition, included modalities, and number of instances. Concrete examples are shown in Appendix A
Causal Variable
Counterfactual
Data
Use Case
Modality
Num
M Y
Condition
Frohhery Binder
Question inwwering
"What if conditine
Text
CLOMO (fimnn ce bile
Text logic parsing
New premise for textual statement
Text
1,100
RNN-Ty myN-oyyoloyy Ris hovel ef 2019
Text syntax parsing
New syntactic stricture of sentence
Text
584
CVQA-Bool Taan
Question anowering
Hypathetical hehavioml pattem
CVQA Count Veali FE
Namerical reasoning
Hypothetical numerical pattern
Text.Image
2,011
COCO LL a. 2023
Text-image matching
"What ..." condition
Text_
17,410
Arihhmtt Wo Win #021
Mathematical reasoning
Change number base
Symbol
6,000
MalAlgoQ
2024
Question Answering
"What ..." condition
Text, Symbal
807
HumanEvall-Be
Execution simulation
Hypothetical coding
Text, Code
Open Crilic Vozora 2021
Code gencration
Hypochetical descriptive functions
Text.Code
8,910
CodePrrlkeeaoc' Veaum 2024
Code sommarization
Hypothetical code structures
Todee
9,389
disentangling these steps to evaluate reasoning capacity (Kiciman et al., 2023 Chi et al., 2024).
Motivated by the need for decomposition, we design four evaluation tasks that reflect the full pipeline of counterfactual reasoning, with each task targeting distinct capability in counterfactual analysis.
Task I: Causal Variable Identification. Given inputs containing factual information, model is required to identify the values of the causal variables (X, Z, M, Y). This step serves as the foundation for subsequent causal modeling and counterfactual reasoning.
Task II: Causal Graph Construction. Given the identified variables, the model is tasked with constructing DAG that captures the causal relationships among them. This step evaluates the model's s ability to discover causal dependencies.
Task II: Counterfactual Identification. Given a counterfactual query (e.g.. "What if vari- able X had been different??) the LLM must identify the new value of (i.e., the intervention). This task evaluates whether the model can detect intervention in the counterfactual condition.
Task IV: Outcome Reasoning, Based on the constructed causal graph and identified intervention, the model is prompted to predict the counterfactual outcome. This step measures whether the model can simulate the hypothetical scenario while respecting the underlying causal mechanisms.
3.3 BENCHMARKING COUNTERFACTUALS
Next, we introduce the datasets we leverage for the decompositional evaluations:
Data Sources and Use Cases. As shown in Table I we collect diverse set of datasets to ensure broad coverage across various NLP tasks and modalities. The included use cases are: (1) Question Answering, evaluated using CRASS (Frohberg & Binder 2021). CVQA-Bool (Zhang al. 2024b),
and MalAlgoQA (Sonkar et al. 2024). which involve answering general-purpose textual or visually grounded questions; (2) Text Parsing, using CLOMO (Huang et al.] 2024) for logical structure reconstruction and RNN-Typolog (Ravfogel et al. 2019) for syntactic structure understanding, (3) Reasoning Tasks, with CVQA-Coum (Zhang et 2024b) for numerical reasoning and Arithmetic (Wu et al. 2024) for symbolic arithmetic computation; (4) Multimodal Matching, represented by the COCO dataset (Le et al., 2023) for image-text alignment; (5) Code-based Tasks, includ- ing HumanEval-Ess (Chen et al. (021) for execution simulation, Open-Critic (Vezoral (0224) for generation, and Pede--reeeeeece (Vezoral 2024) for summarization. These datasets are therefore intentionally positioned as prerequisite stage support safe and informed downstream application.
These datasets span four modalities, natural language text, images, mathematical symbols, and code, that encompass diverse definitions of counterfactual interventions tailored to each task. Collectively, they support a comprehensive multimodal evaluation of LLMs' abilitics to reason under varied counterfactual settings and data types.
Our Preprocessing. To support our decompositional evaluations, we curate those datasets to augment each instance with three additional aspects of information relevant to Tasks I-III (Section (3.2). Specifically, we begin by identifying and annotating the causal variables (X, Z, M, Y) from the original data, questions, descriptions. Using these annotations, we construct DAG represent
5

---

Published as a conference paper at ICLR 2026
Table 2: LLMS' performance in causal variable identification, report of FI across all instances for each variable. Each value is scaled to 100%. The standard deviation is in Table
GPT-5
GPT-04
Qwen
Llama4-8
Ilama4-M
Gemini2.5
DeepSeek
Dataset
02
V2
(Exposure), Z (Covariate)
CRASS
92.3
89.2
87.3
85.4
88.5
86.9
90.6
89.1
86.2
89.5
87.1
CLOMO
89.8
87.6
88.1
85.6
83,5
81.9
87.1
85.3
88.8
87.0
84.3
82.8
86.4
84,2
RNN-Topo
87.9
85.4
85.9
83.7
80.4
78.6
84.3
82.6
85.7
84.2
81.7
80.1
83.8
82.0
CVQA-Bool
79.4
76.2
79.8
76.5
72.3
70.5
77.9
75.8
79.1
76.9
68.5
66.9
70.7
68.3
EVQA-Count
74.7
72.3
74.3
72.9
68.9
67.2
73.6
71.9
74.4
72.5
65.3
63.7
67.4
65.1
COCO
72.8
73.2
71.1
65.4
72.5
73.6
71.7
60.9
65.9
Arithmetic
88.2
86.5
84.9
82.8
75.7
73.8
80.3
78.6
81.6
79.8
76.9
74.5
78.3
76.1
MalAIgoQA
84.1
81.3
81.5
78.9
72.9
70.6
79.5
77.1
80.6
78.2
73.5
71.2
75.9
73.4
HumanEval
69.3
66.9
71.4
69.2
61.9
67.8
68.9
66.7
$7.3
62.1
59.8
Open-Critic
71.7
69.4
70.1
67.3
61.8
59.7
66.5
64.7
67.6
65.8
57.3
55.9
60.4
58.1
Code-Preeeeence
49.6
68.4
80.2
69.0
72.9
60.5
73.6
61.9
75.2
63.4
68.4
66.5
61.2
01
(Mediator), 12
(Outcome)
CRASS
87.4
91.7
84.1
89.3
72.8
79.9
81.2
87.4
82.4
88.6
74.1
81.6
76.2
83.5
CLOMO
83.1
81.3
87.9
77.4
77.2
84.9
78.6
86.0
71.2
78.9
73.5
RNN-Topp
81.7
86.3
79.4
85.6
67.1
75.8
75.3
83.1
76.6
84.4
69.3
76.4
71.5
78.9
CVQA-Bool
73.5
79.3
73.9
80.1
59.7
66.9
71.8
72.9
79.4
63.2
60.5
CVQA-Count
69.6
75.2
70.1
76.2
56.8
63.6
68.9
74.8
70.2
75.9
54.7
60.4
57.9
COCO
67.3
73.4
68.0
74.4
54.2
61.8
66.2
72.1
67.4
73.5
52.8
58.3
55.7
62.6
Arithmetic
82.1
82.3
73.4
63.6
71.9
79.1
80.3
86.0
73.5
65.7
74.9
MalAlgoQA
79.2
83.4
76.8
80.4
61.5
69.3
76.6
81.2
77.8
82.4
60.2
70.5
63.8
72.3
HumanEval-Eii
63.2
67.4
66.0
70.3
51.9
59.7
61.9
66.3
63.0
67.5
49.7
57.0
53.6
60.5
Open- Cein-Ciitcc
66.3
70.6
64.1
68.9
57.2
64.7
69.0
65.6
70.0
47.3
54.8
51.3
58.7
Code Pode-Preeeeeeee
65.9
75.3
66.3
77.0
50.4
78.6
64.5
79.3
65.7
80.4
48.2
76.1
52.5
59.8
the underlying causal structure of each data instance, which enrich original instances with causal and counterfactual structures. A running example in Figure 2
Preprocessing Feasibility, Notably, all datasets are built upon "what-if" conditions or hypothetical scenarios (as outlined in Table [T and intervention-style narratives, thus naturally supporting counter- factual interventions. For each instance, we parse and extract the intervened variables and, guided by the previously constructed DAG, annotate the corresponding counterfactual outcomes and construct matched counterfactual graph. We provide the curated instances in Appendix A
4 EXPERIMENT
We aim to empirically answer two research questions: RQ:: How well do LLMS perform when their counterfactual reasoning is decomposed into distinct reasoning tasks? RQ,: What auxiliary techniques can improve LLMs' counterfactual reasoning? We defer the experimental settings into Appendix and additional results into Appendix B.2
LLMs. We evaluate reasoning-centric and multimodal LLMs due to the nature of counterfactual reasoning tasks. Specifically, we leverage GPT-5, GPT-o4-mini- miniiiihihhh Qwen3-VL VL-223B--222BB Thinking, Llama-4-Scout - L 10B-1111820 128E, Gemini2.5-PP 5-or. DeepSeek-VLL
Metrics. We use the Fl score for Tasks I, II, and IV, as they involve multiple instances (e.g., M, Z. or graph edges) that require set-level evaluation. For Task III, which typically involves a single intervention on X, we use accuracy to assess whether the LLM correctly identifies the intervened X.
4.1 LLM PERFORMANCE ON DECOMPOSITIONAL TASKS (RQ1)
Setting, We evaluate LLMs independently each decompositional task. For each task, explicitly provide the ground-truth outputs from the preceding tasks to isolate and measure LLMs' capabilities specific to that task. For example, when assessing models' ability to construct causal graphs, supply the original inputs along with the ground -tuth causal variables.
Task I: Causal Variable Identification. Table presents the performance of LLMs identifying causal variables (X, Z, M, Y). We observe that model performance is strongly influenced by the modality of the dataset. Specifically, datasets involving more complex modalities (e.g., images,

---

Published as a conference paper at ICLR 2026
GPTet
Elamat-M
Gemini2.5
DeeoSadk
100
14
04
X-M 2-48 X 244
X-M Z-M 2-X X-Y 2
04 X-M Z-+8 2-xx M-YY X-Y Z-y
on
X-M Z-1M 2-x K-Y X-Y 2-Y
160 CRASS
(b) COCO
(cl MalAlgo
Open Critic
Figure 3: Evaluation causal graph construction. We evaluate F1 score balance (i) whether the constructed edges under one category (e.g., X + MM is correctly constructed if the (X, 2, M, Y) are already given. Additional results for all other datasets Figure
GFT-04
QWen-Vl
Llama4-M
DeegBock
1.0
0.9
0.8
CRASS
COCO
MalAlgoQA
Open-Critic
Figure Evaluation LLMs' accuracy in identifying the correct intervention (i.c., the counterfactual value of X). Additional results for all other datasets are provided in Figure 6
mathematical symbols, codes) tend to reduce LLM accuracy (e.g., <0.7 Fl Open-Critic), even when variables like X, Z, Y are explicitly present in the context.
Interestingly, even within the text modality, LLMs show notable difficulty in identifying the implicit mediator M, which often requires reasoning about the underlying causal pathways connecting X, Z. and Y. This suggests that the challenge lies not only in the complexity of the input modality but also in the abstractness and inferential nature of the variable type itself. Together, these findings highlight the need for improved methods that enhance LLMs' capacity to handle both modal complexity and deeper causal reasoning.
Task I: Causal Graph Construction. As described in our experimental setup, isolate each decompositional step by providing the ground-truth outputs of preceding steps as inputs. For causal graph construction, we supply the identified variables X, M, Y and prompt LLMs construct the corresponding counterfactual graph. The results are presented in Figures 3 and 5 Notably, the overall performance mostly exceeds 0.9 F1 scores, indicating that LLMS can accurately construet graph edges. Moreover, the impact of dataset modality and variable types (e.g., explicit Z vs. implicit M) appears to be minimal in this step. We attribute this to the rule-based nature causal graph construction: since causal graph structures are well-defined (as shown in Figure 2 it is relatively less challenging for LLMs to apply construction rules and generate the correct causal relationships.
Insights from causality modeling.
The major challenge in causal modeling lies in causal variable identification, where (1) LLMss highly sensitive to the complexity and structure the input modality, and (2) implicit variables (i.c., the mediator M) reveal a critical gap in LLMs' causal reasoning capabilities.
Task III: Counterfactual Identification. Next, we evaluate LLMs' capability in identifying inter- ventions, i.c., determining the counterfactual values of X (i.e., X'). As shown in Figures 4 and 6 the experimental results indicate that LLMs generally effective recognizing the counterfactual values of across most datasets and modalities. This demonstrates that LLMs have solid grasp of pinpointing intervention points within the context. However, note that the task remains relatively
7

---

Published as a conference paper at ICLR 2026
Table L.LM performance (FI mean) in reasoning the counterfactual mediator (M') and outcome
(Y'). Standard deviation is in Table
GPT-5
GPT-04
Qwen3
Llama4-5 $
Llama4 M
Gemini2.5
DeepSeek
Dataset
M
Y"
M'
Y"
M
Y"
Ms'
Y"
M'
Y"
M²
Y"
M
Y"
CRASS
92.1
88.0
90.5
86.2
80.5
73.9
70.1
63.5
84.9
79.5
81.7
75.2
82.9
77.1
CLOMO
85.3
88.7
83.9
77.8
71.6
67.2
60.9
82.9
77.2
79.3
72.8
80.5
74.3
RNN-Topp
88.9
83.4
87.9
81.6
75.6
69.4
65.2
58.7
80.5
75.0
77.1
70.6
78.3
72.0
CVQA-Boo
81.2
74.5
77.1
70.2
65.4
58.6
54,8
48.1
70.9
64.3
63.2
56.8
66.7
59.8
AVQA-Count
79.2
72.0
76.2
69.3
62.2
$5.7
51.7
45.2
67.5
61.0
60.1
53.8
63.5
$6.9
COCO
77.8
70.1
75.3
66.9
60.1
53.4
49.3
42.7
65.4
58.7
57.8
51.5
61.3
$4.6
Arithmetic
87.8
82.7
85.8
80.9
69.8
63.2
59.1
52.4
76.3
70.6
72.1
65.4
74.0
67.5
MalAlgoQA
85.1
79.6
83.6
77.8
67.5
60.9
57.4
50.7
74.0
68.2
69.6
62.9
71.8
65.1
HumanFvvll Exe
75.7
71.5
73.4
66.5
58.2
$1.5
47.7
41.2
63.6
56.9
55.8
49.4
99.4
$2.7
Open Critic
75.3
69.4
73.8
67.5
56.0
49.4
45.8
39.2
61.5
54.7
53.7
47.3
57.2
50.6
Code-Preference
77.0
71.0
74.4
66.8
57.1
50.4
46.6
40.0
62.7
55.9
54.7
48.3
58.3
51.6
Table 4: Improvement in LLM performance (comparing with Table for identifying explicit causal variables. Results are reported on six representative datasets spanning all major modalities.
GPT-o4
Qwen3
Llama4-$
Gemini2.5
Dataset
X
Z
y
X
CRASS
+6.0
+6.6
+3.5
+
+10.8
49.4
+9.7
+15.2
+11.
+11.9
+6.6
+5.3
+7.1
CLOMO
+11.3
+16.8
CVQA-Count
+17.7
+13.9
+18.2
+21.9
+21.4
+22.7
+32.0
+26.1
+24.1
+18.1
+15.7
+19.3
47.2
+6.0
+3.8
MalAlgoQA
+42
+5.9
-2.6
+9.6
+8.1
+6.8
+12.5
+10,2
+6.5
+12.4
+8.5
+8.3
Open-Critic
+128
+7.0
+109
+5.1
isolated and does not challenge the model's ability to propagate the effects of the intervention through downstream variables (e.g., M', Y"). which we address Task IV.
Task IV: Outcome Reasoning. In this final task, evaluate LLMs' ability to infer the mediator (M') and outcome (Y') under a counterfactual intervention. As shown in Table 3 LLMs consistently exhibit insufficient performance in inferring these implicit variables since both M' and Y' are implicit under the counterfactual condition (whereas only M is implicit in the factual condition), this result suggests that LLMs lack sufficient capacity to reason over causal chains, even when the underlying structure is explicitly provided.
all datasets. Notably,
Insights from counterfactual reasoning.
Regardless of whether the setting is factual or counterfactual, the primary challenge lies in identifying causal variables and performing causal reasoning. In particular, the complex input modality and the implicit nature of mediation hinder effective reasoning through causal pathways.
4.2 AUXILIARY TECHNIQUES TO IMPROVE COUNTERFACTUAL REASONING (RQ2)
Given the insights from previous evaluations, we aim to correspondingly address the limitations arising from multimodal complexity and intermediate reasoning, we propose augmenting LLMS with two auxiliary techniques: (i) tool-auugmeted execution and (ii) advanced elicitation strategies.
4.2.1 AVOL-AUGDNNTED EXECUTION IN EXPLICIT VARIABLE IDENTIHCATION
Settings. To enhance L.M performance in identifying explicit variables (X, Z, Y) across different modalities, we adopt tool-augmented approach, where the LLM dynamically calls additional specialized tools to assist entity identification, identical to a named entity recognition (NER)
paradigm. We leverage several pretrained models tailored to the multimodality of datasets: (1) Text- based NER: We use BERT- BASE-NT BASE-NER (Devlin et al. 2018 Tjong Kim Sang & De Meulder 2003) to identify candidate entities in text- bxt-based data, including mathematical symbols.(2) Vision-based NER: We employ GROUNDING-DE -INGG--IOO-OOE DINO-BASE (Liu et al. (023) to detect all relevant objects in images and generate focused regions by masking out irrelevant backgrounds. (3) Code analysis: We adopt
8

---

Published as a conference paper at ICLR 2026
Table 5: Improvement of LLM performance (F1 score) in reasoning implicit variables.
GPT-o4
Qwen3
Llama4-S
Gemini2.5
Dataset
Elicitation
M
M
y
M'
y
M'
M
M'
y
CoT
+5.6
44,0
+3.1
+7.4
+5.6
+4.1
+7.9
+5.8
+4.2
+6.8
+4.5
+3.2
CRASS
CoT-SC
+8.6
+1.3
+6.0
+8.2
+5.7
ToT
+6.8
+5.2
+-41
+8.9
+6.7
+5.0
+8.9
+7.4
+4.7
+8.2
+5.9
+4.3
Col
+6.4
+4.9
+3.6
+8.3
+6.55
+4,8
+8.2
+6.9
+5.0
+7.6
+3.3
+3.9
CLOMO
CC
+7.0
+5.2
+7.66
+5.44
+9.9
+7.7
+5.55
48.9
ToT
+10.1
+3388
+.2
+88.7
4-5.2
+3.5
+77.2
+-5.1
+3.9
+-2.44
4442
+3.0
CoI
+3.7
+4.4
+3.55
+7.9
+-6.1
+44.4
+8222
+6.4
+4.5
+7.3
+5.4
+3.8
CVQA-Count
CcT-SC
44.1
44.2
+-2.4
477.2
4-6.6
44.88
4-5.9
+-6.8
45.77
+-2.22
47.6
ToT
+4.9
+3.8
+3.1
+6.9
+5.33
+4,00
+6.5
+5.2
+4.0
+6.1
+4.5
+3.1
CoT
+3.8
+2,9
+2.1
+6.0
+-16
+3.2
+5.4
+4.0
+2.6
+4,9
+3.4
+2.3
COCO
CCT-SC
+5.4
+4.1
+-5.7
+4.33
+7.6
+-5.9
+3.88
+669
+5.1
Tol
+4.5
+3.5
+2.7
+6.8
+5.1
+3.6
+6.2
+5.0
+3.5
+5.6
+4.3
+3.0
CoT
+4.6
+3.5
+2.7
+7.0
+55.5
+44.0
+6.6
+-5.0
+3.3
+5.8
+4.0
+2.7
MalAlgoQA
45.5
44.3
+-3.4
47.7
4-6.1
44.6
47,9
+-6.3
44.1
47.2
+5.2
43.7
Tol
+6.2
+4.8
+3.8
+8.55
+6.77
+5.0
+8.6
+7.1
+4.6
+8.1
+6.0
+4.3
CoT
+3,5
+2.7
+2.0
+5,3
44.2
43.0
+5.0
+3.7
+2.5
44.5
+3.1
+2.1
Open-Critic
CoI-SC
44.2
43.3
+226
+6.1
+4.9
43.6
+5.8
+4.5
43.1
45.0
+3.5
+2,7
TuT
+5.0
+388
+2.9
+6.7
+5.2
+3.8
+6.1
+5.3
+3.6
+6.3
+4.7
+3.4
GRAPHCODEBERT (Guo
2020 to extract functions, variables, and control structures for
programming tasks.
After identifying candidate entities across each modality, we prompt the LLM to refine and filter the final set of explicit variables (X, Z, Y) according to the formal definitions provided in Section 3.1
Experimental Results. We randomly select three representative LLMs and conduct experiments across multiple datasets. As shown in Table tool-augmented execution consistently improves LLM performance in identifying explicit causal variables (X, Z, Y) across all modalities. For example, by leveraging GRAPHCODEBERT to parse code structures and forward the results to Llama, which gains a clearer understanding of programming logic and achieves an Fl improvement up to 0.189. Similarly, in the vision modality, the object detector GROUNDING-DING BASE assists by generating a set of candidate visual objects, which GPT-o4 can then contextualize and compose into a coherent factual variable. For instance, detected objects like "Woman," " "Knife," " and "Apple" " can be effectively integrated by GPT-o4 into the causal expression: "A woman cuts an apple with a knife."
These results demonstrate that tool-augmented learning effectively mitigates modality- bottle- necks by offloading low-level entity recognition to specialized models, allowing the LLM to focus on higher-level reasoning. Looking forward, there potential explore alternative tool configurations that may yield comparable or even superior performance. Additionally, future work may explore multi-agent frameworks with specialized agents collaboratively handle different variable categories.
4.2.2 ADVANCED ELICITATION STRATEGIES FOR REASONING OVER IMPLICIT VARIABLES
Settings. To enhance LLM reasoning over implicit variables, particularly factual M and counter- factual M', Y', implement advanced elicitation strategies that guide the model through more structured reasoning. Specifically, we apply Srain-of-Thougee (CoT) (Wei et al., (022), CoT with Self-Consistency (CoT-SC) (Wang al. 2022), and Tree-of-Thougee (ToT) (aao al. 2023):
CoT: given pre-determined explicit variables, LLMs are encouraged to infer intermediate variables step-by-step.
CoT-SC:
LLMs are prompted to generate multiple reasoning paths and select the final answer via majority voting consensus.
LLMs are prompted to explore multiple parallel reasoning paths in a branching structure and evaluate candidate outputs based on intermediate criteria.
ToT:

---

Published as a conference paper at ICLR 2026
Table 6: Overall change of LLM performance (F1 score) with different improvement strategies.
The performance change is compared with Table 3 where results of Y' demonstrate the final counterfactual reasoning outcomes.
Improve Explicit Variable ($1.211
Improve Implicit Variable ($4.2.2)
Improve Both
Dataset
G5
QW3
LM4S
GM2.5
G5
QW3
LM4S
GM2.5
G5
QW3
LM4S
GM2.5
CRASS
+188
+2.3
+3.55
+2.0
+6.2
+74
+7,9
668
+9.0
+10.1
+10.5
+9.3
CLOMO
+2.1
43.1
44.2
+2.6
5.7
16.3
46.1
192
+10.4
411.0
19.8
COCO
+12
+1.5
+1,9
+1,4
+4,1
+5.1
+3,9
-5.1
+5.8
+6.4
+6.7
+6.2
Open-Critic
+1.6
42.3
+2.7
42.0
+3.3
48.3
+11.2
-4.3
45.3
411.5
414.2
+5.5
In implementation, both CoT-SC and ToT are executed with k=5 sampled reasoning paths. ToT further evaluates candidate outputs by scoring their textual similarity using BERTScore (Zhang et al. 2019), specifically assessing how well the intermediate results align with the original task statement.
Experimental Results. Our experimental results in Table 3 show that advanced elicitation strategies generally lead to improved performance in reasoning factual or counterfactual cases. However, we also observe that more complex prompting strategies (e.g., CoT-SC and ToT can sometimes perform slightly worse than simpler approaches (e.g.. CoT). While these advanced methods encourage more exhaustive exploration of reasoning paths, they may also induce overthinking behavior in LLMs, leading the model introduce unnecessary causal links or misinterpret the underlying problem structure. For instance, consider the following context "A person is running a marathon and collapses." " with expected mediator "Dehydration" While CoT and CoT-SC strategies correctly identify the mediator, ToT leads LLMs to overanalyze and identify "lack of training" or "overexertion" " as the mediator. These choices, although related, are not directly supported by the input data and reflect an over-extension of the reasoning process.
implicit variables (M, M', Y') despite
The over-qualification of elicitation strategies (e.g., ToT) highlights that, while advanced prompting techniques can improve reasoning capabilities, they may also introduce complexities that divert the model from the most straightforward and contextually supported causal pathways. Therefore, it's crucial to balance the reasoning depth while maintaining alignment with the input data.
4.2.3 OVERALI PERFORMANCE IMPROVEMENT
In end-to-end setting, we incorporate the prior improvements for explicit variable identification (Section I.2.T). implicit variable reasoning (Section 4.2.2), and their combinatorial strategy to evaluate the overall change in counterfactual reasoning performance. Table presents the results across representative datasets and LLMs.
We have to observations: (i) In general, improving implicit variable reasoning (i.e., for mediators and outcomes) yields more substantial gains in end-to-end performance, as these variables directly influence the final counterfactual predictions (Task IV). In contrast, improvements in explicit variable identification (c.g., for X, Z, Y) primarily strengthen the early stages of the pipeline, offering moderate but necessary support. (ii) Notably, the combined strategy achieves the highest overall improvement, although the gains are not strictly additive. This is due to accumulative reasoning errors across the decompositional steps, where inaccuracies in earlier predictions may propagate and compound through the pipeline. These findings highlight both the opportunities and challenges of modular improvements.
5 CONCLUSION
This work provides a decompositional framework for evaluating counterfactual reasoning in LLMs. We collect a set datasets across multimodalities, and then curate them with reference causal variables, structured graphs, and counterfactual intervention. Next, we our curated dataset to study the reasoning process into distinct stages from causal variable identification to outcome inference. We uncover the qualifications of current LLMs in counterfactual reasoning and where they fall short. Based on experimental insights, we further propose improvements to offer actionable insights for enhancing I.LM reasoning, particularly in multimodality and implicit reasoning settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Science Foundation (NSF) under Grants 2046590, 2419831, and 2444759, as well as the SUNY Empire Innovation Program (EIP) awarded to N.L.G.
REFERENCES
Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. Probabilistic and causal inference: the works of judea pearl, pp. 507-556. 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Haoang Chi, He Li, Wenjing Yang, Feng Liu, Long Lan, Xiaoguang Ren, Tongliang Liu, and Bo Han. Unveiling causal reasoning in large language models: Reality mirage? Advances in Neural Information Processing Systems, 37:96640-96670, 2024.
Pratyush Desai, Luoxi Tang, Yuqiao Meng, and Zhaohan Xi. Safegpt: Preventing data leakage and unethical outputs in enterprise Ilm use. arXiv preprint arXiv:2601.06366 2026.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, toss/880,00405, 2018. URL
http::/arxivv wwg. abs/1810s11111 1810000005 04805
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Fuli Feng, Jizhi Zhang, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. Empowering language understanding with counterfactual reasoning. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2226-2236, 2236, Online, August 2021. Association for Computational Linguisties. doi: 10.18653/ v1/2021 1//202..inddins--o Andings-acl..19) URL https: htt.://ts///aaaattoooo..o///oo.. findinggss acl 1967
Jörg Frohberg and Frank Binder. Crass: A novel data set and benchmark to test counterfactual reasoning large language models. arXiv preprint arXiv:2112. 1941, 2021.
Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. Faithful explanations black-bor box nlp models using Ilm- rengeeeratted counterfactuals. The Twelfth International Conference Learning Representations, 2024.
Ian D Gow, David F Larcker, and Peter C Reiss. Causal inference in accounting research. Journal of Accounting Research, 54(2):477-523 2016.
Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint 2009 08366, 2020.
Emilia Gvozdenović, Lucio Malvisi, Elisa Cinconze, Stijn Vansteelandt, Phoebe Nakanwagi, Em- manuel Aris, and Dominique Rosillon. Causal inference concepts applied three observational studies in the context of vaccine development: from theory to practice. BMC Medical Research Methodology, 21:1-10 2021.
Ronghang Hu. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference computer vision, pp. 804-813, 2017.
11

---

Published as a conference paper at ICLR 2026
Yinya Huang, Ruixin Hong, Hongming Zhang. Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang, and Linqi Song. Clomo: Counterfactual logical modification with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), 2024.
Licheng Jiao, Yuhan Wang, Xu Liu, Lingling Li, Fang Liu, Wenping Ma, Yuwei Guo, Puhua Chen, Shuyuan Yang, and Biao Hou. Causal inference meets deep learning: A comprehensive survey. Research, 7:0467, 2024.
Sriharshini Kalvakuntla, Luoxi Tang, Yugiao Meng, and Zhaohan Xi. Smart privacy policy as- sistant: An 11m-pover ain-powered system for transparent and actionable privacy notices. arXiv preprint arXiv:2601.06357, 2026.
Atoosa Kasirzadeh and Andrew Smart. The use and misuse of counterfactuals in ethical machine learn- ing. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 228 236, 2021.
Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. Transactions Machine Learning Research, 2023.
Lisa Koonce, Karen K Nelson, and Catherine M Shakespeare. Judging the relevance of fair value for financial instruments. The Accounting Review, 86(6):2075-20) 2011.
Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with milm in visual question answering: A survey, ACM Compul. Surv. 57(8), March 2025. ISSN 0360-0300. doi: 10.1145/3711688 Itttttts::///o1 1tt.o/////11111111111111
Evangelia Kyrimi, Somayyeh Mossadegh, Jared M Wohlgemut, Rebecca S Stoner, Nigel RM Tai, and William Marsh. Counterfactual reasoning using causal bayesian networks healthcare governance tool. International Journal of Medical Informatics, 193:105681, 2025.
Tiep Le, Vasudev Lal, and Phillip Howard. Coco-counterfactuall. Automatically constructed counterfactual examples for image-text pairs. Advances in Neural Information Processing Systems, 36:71195-71221 2023.
Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 805-815, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v a2co020....-soorrt7 URL https: //acle/cclcee http://acclaatoooog.....ooo org/
3073.acl-short 707
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
Xiaoqun Liu, Jiacheng Liang, Qiben Yan, Jiyong Jang, Sicheng Mao, Muchao Ye, Jinyuan Jia, and Zhaohan Xi. Cylens: Towards reinventing cyber threat intelligence in the paradigm of agentic large language models. arXiv preprint arXiv:2502.2077 2025.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2522 2022.
Yuqiao Meng. Luoxi Tang, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, and Zhaohan Xi. Benchmark- ing llm-assisted blue teaming via standardized threat hunting. arXiv preprint arXiv:2509.23571, 2025.
12

---

Published as a conference paper at ICLR 2026
Van Bach Nguyen, Paul Youssef, Christin Seifert, and Jörg Schlötterer. LLMs for generating and evaluating counterfactuals: A comprehensive study. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (cds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. -1809-14824, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 102118655/1122224
https////////2..mfddnnnmemmmmmm findingg--mmllp.8770
URL
org /
Findings 1tr.igssemm......77..
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
Zhaozhi Qian, Yao Zhang, Ioana Bica, Angela Wood, and Mihaela van der Schaar. Synctwin: Treatment effect estimation with longitudinal outcomes. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3178-3190. Curran Associates, Inc.,
2021. URL http:::: wwwplinngnneeurrrae newrips. cc/paper files//paper/2021/ pdf
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. Studying the inductive biases of rnns with synthetic variations of natural languages. arXiv preprint
2019.
Aryan Shrivastava and Paula Akemi Aoyagui. Dice: A framework for dimensional and contextual evaluation of language models. arXiv preprint arXiv:2504. 10359, 2025.
Shashank Sonkar, Naiming Liu, MyCo Le, and Richard Baraniuk. Malalgoqa: Pedagogical evaluation of counterfactual reasoning in large language models and implications for in education. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.
2024.
Luoxi Tang, Yuqiao Meng, Ankita Patra, Weicheng Ma, Muchao Ye, and Zhaohan Xi. Polar: Automat- ing cyber threat prioritization through Ilm powered assessment. arXiv preprint arXiv2251000155.
2025a.
Luoxi Tang, Tharunya Sundar, Shuai Yang, Ankita Patra, Manohar Chippada, Giqi Zhao, Yi Li, Riteng Zhang, Tunan Zhao, Ting Yang, et al. Are ilms ready for english standardized tests? a benchmarking and elicitation perspective. arXiv preprint arXiv:2505. 100005/77000 70056, 2025b.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CONLL--2033 shared task: inguuag-inddeeedeent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at IILT-NACCL 2003, pp. 142-147, 2003. URL https://wwz
aclweb. www.eb.org//ntthoooooo -
Vezora. Code preference pairs.
https///ccgiigaae.c///taassssss
Code Pele-Prreeereeee-Paaa 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint Xrviv:2203.111 11171, 2022.
Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng. Yuhong Feng, and Zhiqi Shen. A survey on natural language counterfactual generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4798-4988888 188--8188 8, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.11863/v11/222 http:///........nnnnneneeeee www.iings-emlll.2255...
URL https:://ccaantteett org 2024
findings emm 2767
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 244377, 2022.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1819-1862, 2024.
13

---

Published as a conference paper at ICLR 2026
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 11338,,,.1.,,,, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 233, Red Hook, NY, USA, 2023. Curran Associates Inc.
Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. Working memory iden- tifies reasoning limits in language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 16896-16922, Miami, Florida, USA, November 2024a. Associa- tion for Computational Linguistics. doi: 10. 18633v http//11/200 mmn. caammlppmainn93.. URL https:
//aclanntholoy..rrgg// 2024 emnlp-main. 938/
Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong. Xin Wen, and Bingchen Zhao. What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21853-21862, 2024b.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint 1904.0904.0900 2019.
Yuan Zhou, Peng Zhang, Mengya Song, Alice Zheng. Yiwen Lu, Zhiheng Liu, Yong Chen, and Zhaohan Xi. Zodiac: A cardiologist-lveel Ilm framework for multi-agent diagnostics. arXiv preprint arXiv:2410.02000 2024.
A COMPLEMENTARY INFORMATION CAUSALITY AND CAUSAL GRAPH
This appendix presents concise overview of each dataset, followed by its causal structure and graphical representation, alongside a concrete example. For each dataset, we identify the four variable types Trees-Fxposure, Covariate, Mediator, and Outcome-and distinguish their roles in both factual and counterfactual scenarios, illustrating each with directed edges in the corresponding causal graph. In addition, include a sample prompt used to generate responses simple text-parsing Q&A task dataset.
In evaluation, we rely on universal prompt template as shown below, wherein only the specifce contents are replaced for each dataset.
Prompt Template.
**Task Description**
You are asked to perform [Task / Task II / Task III / Task IV] in decompositional counterfactual reasoning. Follow the definitions of causal variables and causal relations strictly.
IIInuu Context+*
Here is the factual instance from the dataset: [Insert factual context or multimodal description]
IIItermediate Outputs**
If applicable, use the following ground-truth results from previous tasks
Exposure (X) [...]
Covariate (2) [...]
Mediator(S) (M): [...] Outcome (Y): [...]
Insstructioon for the Current Task**
14

---

Published as a conference paper at ICLR 2026
Task I (Variable Identification) [description about domain specific meanings causal variables x, 2, M, and and the identification task]
Task II (Graph Construction): [instruction about constructing the causal graph by listing all directed edges among X, 2, M, Y.
Task III (Intervention Identificatien) [instruction about identifying which variable is intervened on the counterfactual query.]
Task IV (Outcome Reasoning): [instruction about inferring] the counterfactusl mediater M' and outcome Y' under the specified intervention.
**Output Format**
Provide the answer using the follewing structure:
Exposure (X) [...]
(ovariate (2) [...]
Mediator(s) (M) [...]
Outcome (Y): [...]
Causal Eéges: [...]
Interventicn: [...]
Counterfactual Mediator (M'): [...]
Counterfactual Outcome (Y'): [...]
CRASS Example. The Counterfactual Reasoning Assessment for Structured Scenarios (CRASS) dataset is designed to evaluate whether language models can reason about hypothetical alternatives to factual events. Each example in CRASS presents a factual scenario (e.g., "A woman opens treasure chest") followed by counterfactual question (e.g.. "What would have happened if the woman had not opened the treasure chest?"). Models are asked to select the most logically consistent outcome from multiple-choice options, such as "The treasure chest would have remained closed", which labeled correct. The following displays full example:
'inputt: "A woman opens treasure chest. what would have happened if the woman had not opened the treasure chest?", "target_scoren"
"The treasure chest would have been open.": ": 0,
"That is not possible.": 0, "The treasure chest would have remained closed. "; 1, "I ton't know,", "; 0
CRASS Causality & Causal Graph.
"factua_roles""
"Exposure": ["act of opening treasure chest"),
"Covariate:; ["key possessicn", "physical capability"),
"Mediator": ["lock mechanism release"),
"Outcome": I"chest opened"|
"counterfactual :
"Exposure: ["omission of opening action"),
"Covariate:; ["key possession", "physical capability"), "Mediator"; ["lock state preservation"),
"Outcome": ["chest remains clcsed"]
"causal_graph";
"factual_edges"
["key possession", "act of opening treasure chest"), ["key possession", "lock mechanism release"), ["key possession", "chest opened" ["physical capability", "act of opening treasure chest"), ("physical capability", "1ock mechani release"],
15

---

Published as a conference paper at ICLR 2026
l"physical capability", "chest opened"), [*act of opening treasure chest", "lock mechanism release"), ["lock mechanism release", "chest opened"l, [*act of opening treasure chest", "chest opened"),
paanntterfaatulledeeee
("key possession", "lock state preservation"),
""ke possession", "chest remains closed"],
"physical capability", "lock state preservation"),
"physical capability" "chest remains closed"),
Momission of opening acticn", "lock state preservation"),
"lock state preservation", "chest remains closed"],
("omission of opening acticn", "chest remains closed"),
CLOMO Example. The Counterfactual Logical Modification (CLOMO) dataset designed to evaluate whether large language models can perform controlled, counterfactual edits to natural language arguments in logically coherent way. Each example presents base argument and two premises: Premise 1 has logical sensitivity with the original argument, while Premise 2 does not. The model instructed modify the argument such that Premise 2 has a logical sensitivity with the original argument, while Premise no longer is. For instance, given argument attributing the rise in gasoline prices fully to government policies, the model must produce revised version (c.g., changing "fully responsible" to "partly leads") of argument that shifts logical sensitivity from one premise to another without introducing new claims. The following displays a full example:
"i *instruction": "In the follcwing, you will see an argument and 2 premises, where Premise provides a necessary assumption to the Argument. Please modify the Statements in the Argunent until Premise 2 provides a necessary assumption to the Argument instead, wnile Premise 1 fails to provide necessazy assuaption to the Argument. Note that additional statement should added. "input": "Argument: Statementl: Consumer advocate there is no doubt that the government is responsible for the increased cost of gascline, because the government's policies have significantly increased consumer demand for fuel, and demand, the price of gasoline has risen steadily.Premiseel The government can bear responsibility for that which it indirectly causcs.Premies Premise2; Consumer demand for gasoline cannot increase without causing gasoline prices increase. write the modified argument below: ",
result of increasing Premisel:
"output": "Statementl: Consumer advocate there no doubt that the government. partly leads to the increased cost of gasoline, because the gevernment's policies have significantly increased consumer demand for fuel, and result of increasing demand, the price of gasoline risen steadily undoubtedly."
CLOMO Causality & Causal Graph.
" "factual_roles""
"Exposure"; ["Premise as necessary assumption"), "Covariate";
"Government's s policy impact on demand", "Demand-price relationship assumption"
"Mediator"; ("Causal attributicn mechanism (direct vs indirect) "I,
"Outcome"; ["Full responsibility attribution to government")
Loaanterfactualrrlles
Txxposure": ["Premise necessary assumption"),
16

---

Published as a conference paper at ICLR 2026
"Covariate":
"Government's policy impact en demand",
"Demand-price relationship assumption"
"Mediator": ("Responsibility attribution modifier (partial f]
"Outceme": ("Partial responsibility attribution to government")
"aausal_graph""
"Factual_edges"
["Government's policy impact en demand", "premise as necessary assurption"),
[
("Government's policy impact en demand", CCassal attribution mechanism"),
("Govezmment's policy impact en demand", "Full responsibility
attribution to goveznment"),
"Demand-price relationship assumption", "Premise as necessazy assumption"),
"Demand-price relationship assumption", "Causal attribution
mechanism"),
(DDemandprrce relationship assumption", "Full responsibility
attribution to government"),
"Premise 1 as necessary assumption", "Causal attribution mechanism "I,
"Causal attribution mechanism", "Full responsibility attribution to government"!,
["Premise 1 as necessary assumption", "Full responsibility attribution to goveznment"]
}, "oounterfactual_ddeee
("Government's policy impact on demand", "Responsibility
attribution modificr"), ("Govenmmett's policy impact demand", "Partial responsibility
attribution to govezament"), "Demmnndpricc relationship assumption", "Responsibility attribution modifier"].
(DDmmnddprrce relationship assumption", "Partial responsibility
attribution to government"),
"Premise 2 as necessary assumption", "Responsibility attribution modifier"),
"Respensibility attribution modifier", "Partial responsibility
attribution to government"), ("Premise 2 necessary assumption", "Partial respensibility attribution to goverament")
RNN-Typology Example. This is a synthetic dataset contains sentence pairs that reflect syntactic alterations to word orders (e.g., converting English from subject-verb-objeat (SVO) to subject-object- verb (SOV) order). For example, the factual sentence "Tim saw Lucas." (SVO) is transformed to its SOV equivalent "Tim Lucas saw.".
"tim saw Lucas.", "; "tim lucas saw.
RNN-Typology Causality & Causal Graph.
factual_roles
"Exposure"; ["subjett-err--bjject order"),
"Covariate"; ["syntactic rule", "Lexical items (Tim, saw, Lucas) "], "Meciator"; | ["SOV reordering operation"),
"Outcome": ["tim saw lucas."]
17

---

Published as a conference paper at ICLR 2026
"counterfactual roles":
"Exposure"; ["subjeet-bbjectveer order"),
"Covariate"; ("syntactic rule", "Lexical items (Tim, saw, Lucas) "I, "Mediator"; ["svo restoration operation"),
"Outceme": ["tim lucas saw."]
"causal_graphi { "factual_edges":
lssyntactic rule", "suoject-verb-objee order"!,
"syntactic rule", "SOV reordering operation"),
ssyntactic rule", "tim
lucas."),
order"!, ["Lexical items (Tim, saw, Lucas)", "SOV reordering operation"),
Lexiiilll items (Tim, saw, Lucas)", verjject-erbb-beeee
"Lexical items (Tim, saw, Lucas)", "tim saw lucas. "), subject-verb-objee order", "SCV reordering operation"), sov reordering operation", "tim saw lucas.") subjcct-verb-objjes order", "tim saw lucas."]
},
"contterfactualeddee"
(ssyntacticc rule", "$vO restcration operation"),
syntactic rule", "tim lucas saw.",,
Lexical items (Tim, saw, lucas)", "svo restoration in operation"!,
Lexical items (Tim, saw, Lucas)", "tim lucas saw."), subject-object-veet order", "SVO restoration operation"),
svo restoration operation", "tim lucas sax.")
veeb order", "tim lucas saw."]
CVQA-Bool Example. Counterfactual Visual Question Answering(CVQA. is designed to assess the ability of vision-language models to perform counterfactual reasoning over images. Each example presents factual visual query-answer answer pair (e.g.. "Is there red sandal here?" > yes) grounded in COCO image, along with a corresponding counterfactual query that modifies a key visual condition (e.g., "Would there be red sandal here all shoes
removed?" no). The task requires the model to infer changes object presence or relationships under hypothetical alterations to the scene. The dataset focuses boolean query type. The following displays an example:
image
query
query
type
there red sandal bere?
Would there be red sandal here if shoes were removed!
boolean
eVQA-Bool Causality & Causal Graph.
" "factual_roless
"Exposure"; I"presence of red sandal"), "Covariate"; ("original shoe collection cart", "visual recognition capability"),
"Mediator"; ["sandal-as-shoe categorical inclusicn"),
"Outcome"; ["yes"]
"counterfactualrollss
"Exposure: ["removal of all shces"),
Covariatte: ["original shoe collection in cart", "visual recognition capability"),
"Mediator;; ["sandal-shoe categorical dependency"),
"Outcome"; ["no"]
"causal_grapp *accuull "_ttuaallddee"
18

---

Published as a conference paper at ICLR 2026
("oigiinal shoe collection in cart", "presence red sandal"), ("original shoe collection cart", "sandal-as-shoe categorical inclusion"),
["original shoc collection in cart", "yes"), ("visual recognition capability", "presence of red sandal"), ("visual recognition capability", "sandal-as-shon categorical inclusion"),
[ ("visual recognition capability", "yes"!, '"presence of red sandal", "sandal-as-shoe categorical inclusion"), "sandal-as-shoo categorical inclusion", "yes"), ("presence of red sandal", "yes"]
"counterfactualledges
("original shoe collection in cart", "sandal-shce categorical
dependency"),
("original shoe collection cart", "no"),
["visual recognition capability" "sandal-shoe categorical dopendency"),
(vvisua recognition capability", "no"],
(rremvall of all shoes", "sandal-shor categorical dependency"
"sandal-shoe categorical dependency", "no"],
["removal of all shoes", "nc"]
CVQA-Count Example. Visual Counterfactual Query Dataset (CVQA) also evaluates whether language models can perform a direct indirect numerical counterfactual reasoning grounded in visual inputs. Each example consists of a factual visual question (c.g.. "How many plates are there?" + paired with corresponding counterfactual query that modifies the quantity in clearly defined way (e.g., "How many plates would there be if more plates were added?" 3). The model must integrate visual perception (e.g., detecting single white plate in image) with numerical logic (e.g., adding 2) to produce the correct answer. The dataset focuses on counting query type. The following displays example:
real image
Query
answer
new query
new answer
type
How plates there
How many plates would there be more plates were added?
3
direct counting
CVQA-Count Causality & Causal Graph.
"factual_roless
"Exposure": ["current plate presence (1 unit) "I,
"Covariate"; ["original plate count. (1) ", "visual counting capabi fity "I,
"Meciator"; "vvisual plate detection mechanism"),
"Outcome": ["1"]
"counterfactual_roe
"Exposure; ("addition of plates"),
"Cvariatee; ("original plate count (1)", "visual counting capability "],
"Mediator"; ["numerical addition operation"),
"Outcome": ["3"]
"aausal_graph""
"factual_edges""
["original plate count (1)", ", "current plate presence (1 unit)
["original plate count (1)", "visual plate detection mechanism");
19

---

Published as a conference paper at ICLR 2026
["eriginal plate count (1)", "I"I,
["visual counting capability", "current plate presence "n ["visual counting capability", "visual plate detection mechanisn"] ["visual counting capability", "1",, [*current plate presence (1 unit)", "visual plate detection mechanism"),
["visual plate detection mechanism", "1"), ["current plate presence (1 unit)", "1"]
"counterfactual _eteraaccttue__ee=:: edges":
["original plate count (1)", "numerical addition operation"), ("original plate count (1)", "3"1,
["visual counting capability", "numerical addition operation"!, l'visual counting capability", "3"I, "addition of 2 plates", "numerical addition operation"),
"numerical addition operaticn", "3"],
"addition of 2 plates", "3"]
COCO Example. Common Objects in ConCext(COCOO) dataset provides automatically constructed counterfactual examples for evaluating multimodal reasoning in image-text pairs. Each instance contains two images and two near-idenee identical captions that differ only in a key noun (e.g., "A big burly grizzly bear is shown with grass in the background" vs. "A big burly grizzly bear is shown with deer in the background"). The dataset is designed to test whether models can detect minimal semantic changes and determine whether the new image visually aligns with the counterfactual caption. The goal is to assess visual-textual consistency and a model's sensitivity to causal or identity-bazt -ased alterations in structured multimodal contexts. The following displays an example:
Factual Caption
Image
Counterfacinal Caption
Image
big burly grizzly bear shown with the background.
big burly grizzly bear shown with deer the background.
COCO Causality & Causal Graph.
porctuaalroles"""
"Exposure"; ["original image (bear with grass)
"Covariate"; ["bear presence", "background context"),
"Mediator"; ["grass visual detection"), "Outceme": ("caption with 'grass""]
"Exposure" ("modified image (bear with deer) "I,
"Covariate"; ('bear presence", "background context" "Meciator": I'deer-grass substitution mechanism"),
"Outcome"; ("caption with "]
"causal_graph"; factual_edges": [
["bear presence", "original image (bear with grass) ["bear presence", "grass visual detection"), ["bear presence", "caption with 'grass'"),
("background context", "original image (bear with grass)
("background context", "grass visual detection"), ["background context", "caption with 'grass'"), [ ["original image (bear with grass) ", "grass visual detection"), ["grass visual detection", "caption with 'grass""),
20

---

Published as a conference paper at ICLR 2026
("riginnal image (bear with grass) "caption with 'grass'
counterfactualeooees: [
["bear presence", "deer-grass substitution mechanisn"), ("bear presence", "caption with 'deer'", ("background context", "deer-grass substitution mechanism"), "bbackground context", "caption with 'deer'",, "modified image (bear with deer)", "deer-grass substitution mechanism"),
"ceer-grass substitution mechanism", "caption with 'deer" ("modified image (bear with deer)", " "caption with 'deer!")
Arithmetic Example. Base-computation Arithmetic dataset evaluates counterfactual numerical reasoning by lesting arithmetic operations across multiple numeral systems(e.g, Base-8, 10, 11, 16). Each example pairs factual base- 10 calculation with counterfactual alternate-base computation (e.g., base-8: 14g + 57g = 73g, base- 16: EC1s + DD16 1C911). The dataset includes inputs (num1, num2), the numeral system (e.g., "g" for octal, "1 6 for hexadecimal), and the base-specific result (addrst)). assesses models' ability to adapt numeral system transitions and consistency in counterfactual reasoning. The following display base-8 computation example and base- 16
example:
"8"
"num1" ": "14",
"num2": "57"
"acdrst"; "73"
"16":
"numl ": "EC",
"num2": "DD"
"acorst"; "IC9"
Base-8 Arithmetic Causality Causal Graph.
" "factual roles": :
"Exposure": ["10-based system"),
"Covariate": 1"14", "57"],
"Mediator"; ("baso-10 arithmetic operation"), "Outcome"; ["71"]
"connterfactual "aaarrrfaaattaarrrlss"" roles":
"Exposure"; ["8-bases system"), "Covariate": ["14", "57"],
"Meciator";
"base-8 base-10 conversica",
"base-10 sum conversion to base-8" },
"Outcome"; ["73"]
}
"causal_graph": {
paacuaal_ddges [
["14", "10-based system"),
["14", "base-10 arithmetic operation"),
["14", "?1"I,
["57", "10-based system"),
["57", "base-10 arithmetic operation"),
["57", "71"],
[*10-based system", "base-10 arithmetic operation"),
21

---

Published as a conference paper at ICLR 2026
1'base-10 arithmetic operaticn", "71"], (*10-based system", "71"]
"oonttrrfattua_cdgee=
["14", "base-8 to base-10 conversion"), ["14", "73"],
["57", "base-8 to base-10 conversion"), ["57", "?3"I,
("8-based system", "base-8 to base-10 conversion"),
1"base-8 to base-10 conversicn", "base-10 conversion to base-9" 1,
("base-10 sum conversion to base-8", "73"],
["8-based system", "73"]
Base-16 Arithmetic Causality & Causal Graph.
" "factual_roles "Exposure"; ["10-based system"], "Covariate"; ["RC", "bb"], "Meciator"; [",.A.",,
"Outcome": ["N.A."]
"counterfactual_roles""
"Exposure: ("16-based system"), *Covariate"; ("EC", "DD"],
"Mediator":
"hex-to-decimal conversion", "decimal-to-hex reversion"
"Outcome"; ["1C9"]
"causal_graph": "acctualedgee"; [ ["xC", "10-based system"], ["DD", "10-based system"]
"conterffatuaaleegee""
["EC", "hox-to-decimal conversion"), ("hex-to-decimal conversicn", "decimal-to-hex reversion"), ["EC", "1C9"], ["DD", "hex-to-decimal conversion"),
["DD", "IC9"], ["16-based system", "hex-to-decimal conversion"), "dacimal-to-hex reversicn", "1C9"I, "16-based system", "1C9"]
MalAlgoQA Example (Malformed Algorithmic Question Answering (MalAlgoQA) dataset is designed intentionally including factual and counterfactual rationales between multiple-choice question answering to validate language model's s ability discern sound reasoning the presence of rationales(factual or counterfactual). Each question is presented alongside factual rationale that supports the correct answer (e.g., "Correctly ordered the values from greatest to least: 276, 254, 237, 235." + C), and is paired with counterfactual rationales (e.g.. "Ordered least to greatest") that correspond to plausible but incorrect or altered answers (e.g., A). Each example is decomposed into factual and counterfactual role pairs, allowing researchers to assess how changes in reasoning paths (rationales) lead to different answer choices. The following display an example of raw data and its decomposed data points:
22

---

Published as a conference paper at ICLR 2026
"Question: "Which "Khiich 1ist shows the following number in order from highest to lowest?", "Answer"""c"
"Choice_A":"
254
"
"Choice_B":" 237
276
235
254
"Choice_C":" 276
254
237
235 ",
"Choice_D":" 276
254
235
237
"Rationalo__""""rrder""" least greatest",
"Fat.cnalc__""."rrerro greatest least by ones place.", "Rat
ordered the values from greatest to least: 276, 254, 237, 235.",
"Raticnale_D" "Switcheed last 2 numbers."
""usstio":"Whie""" list shows the following number in order from highest to lowest?", "Answer"; "C", "Counterfactual Answer":"A"
"Cheice_A"" 235 237
254
276 ",
"Chcice B":" 237
276
235
254 "
"Chcice_c":" 276
254
237
235 "
"Cheice_D";" "ationale"""Ordert""""
254
235
237 "
"Counterfactual
least to greatest",
"Raticnale_G"s "":""""rcttlly ordered the values from greatest to least: 276, 254, 237, 235."
"'ucstion"""hiiee"" list shows the following namber in order from highest to lowest?",
"Answer":""" "C",
"Counterfactual
Answer";"""
"Chcice_A"" 235
237
254
276 ",
"Chcice_Bu B":" 237
276
235
254 "
"Choice C":" 276
254
237
235 "
"Cheice_D":"
254
235
237 "
"Counterfactual "triionaee="Orderee""" greatest to least by ones
place", "Rationale_c": "Correctly ordered the values from greatest to least: 276, 254, 237, 235."
"Question": "Which list shows the following number in order from highest 1owest?",
Annswer"""C"
"Counterfactuua Answer": "b""
"Chcice A";" 235
237 254
276 "
"Choice B":" 237
276
235
254 "
"Cheice C"," 276
254
237
235
"Cheice_D";" 276
254
235
237 "
"Rat ictalee___"" "Correctly crdered the values from greatest to least: 276, 254, 237, 235.",
"Counterfactusl Rationale": "Switchhedd last numbers." "
Take the first decomposed sample as example showing MalAlgoQA Causality Causal Graph.
23

---

Published as a conference paper at ICLR 2026
" "factual Practualrroess; roles":
"Exposure"; ["Ordered from greatest to least"), "Covariate"; ("Number set (276, 254, 237, 235) "I, "Mediator"; ["Descending comparison logic"], "Outceme": ["Choice c*]
"coutterfccuull
_rrraacccttarrros roles":
"Exposure": ("Ordered least to greatest"), "Covariate"; "Nuumber set (276, 254, 237, 235) "I, "Meciator:: |"Ascending comparison logic"), "Outcome": ["Choice A"]
"causal_graph":
pooctual_edgese
["Number set (276, 254, 237, 235)", "Ordered from greatest to least
"],
[ "Number set (276, 254, 237, 235)", ", "Descending comparison logic"l,
[ ("Number set (276, 254, 237, 235) ", "Choice c"l, "Ordered from greatest to least", "Descending comparison logic"I, (DDsccending comparison legic", "Choice c*), ("Ordered from greatest te least", "Choice C"]
},
*counterfactual_edge""
"Number set (276, 254, 237, 235)", "Ordered Least to greatest")
"Number set (276, 254, 237, 235)", "Ascending comparison logie"), "Number set (276, 254, 237, 235) ", "Choice A"1,
"Ordered least to greatest", "Ascending comparison logic"),
"Ascending comparison logic", "Choice A"],
"OOdereed least greatest", "Choice A")
HumanEval-Exx Example This dataset performs a programming-relatee task: code execution. It is designed to probe the ability of code-execution language models to perform counterfactual reasoning in the context of program behavior. Each example consists of function definition and test case input, and the model asked to predict the output under a factual assumption (e.g., Python's default O-based indexing). The example is paired with counterfactual version of the same test case, where hypothetical condition is introduced such as switching to I-based indexing. The model must then predict the corresponding counterfactual output. For instance, given a function that checks for close taating--oimm elements in list, the model is expected to reason whether the list and threshold would yield different outcome if indexing conventions altered. The full example is shown follows:
"instruction" "from typing import List\nin has_classe cecloss_eeeeeeeeeeee elements nambers: List [ficat], thresheld: float) -> bool:\n \"\"!"
Check if in given iist of numbers, any numbers closer to each other than given threshold,
>>> has wwwse elements
([I.0, 2.0, 3.0], 0.5) Falsein 9, 2.8, 3.0, 4.0, 5.0, 2.01, 0.3) Truein \"\"!""
>>> has_close_eleeeeeeeee ([1.
"input": "[1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3", "output": "True", "counterfactual outpott "True"
HumanEval-Eww Causality & Causal Graph.
" "factual_roles
"Exposure"; ("O-based indexing"), "Covariate";
"List values [1.0, 2.0, 3.9, 4.0, 5.0, 2.2)", "Threshold 0.3", "pairwise comparison al algorithm"
24

---

Published as a conference paper at ICLR 2026
1, "Mediator"; ["Range iteration logic (0 <= len (numbers) "Outccmme; ["True"]
Counterfactual_rollss
"Exposure"; ["1-based indexing"), "Covariate":
"List values 11.0, 2.0, 3.9, 4.0, 5.0, 2.2)",
"Threshold 0.3",
"Pairwise comparison algorithm"
1,
"Mediator"; ["Range iteration logic (1 <= < j len (numbers) "!,
"Outcome"; ["True"]
"causal "causal_graph";
"factual_edges";
["List values I...]", "0-based indexing"),
("Threshold 0.3", "0-based indexing"), ["Pairwise comparison algerithn", "O-based indexing"),
["0-based indexing", "Range iteration logic <= len (
numbers))
[ ["Range iteration logic (0 <- < < len (numbers)) ), "True"),
lLLit values I...I", "True"I,
"TThrehhlld 0.3", "True"I
"contterfattulleeeee"::
["List values [. [...]", "1-based indexing"),
["Threshold 0.3", "1-based indexing"),
("Fairwise comparison algerithm", "1-based indexing"),
["i-based indexing", "Range iteration logic (1 < <= len ))
"RRange iteration logic (1 <= i J <= len (numbers)) "True"l,
"List values [... ....,, "True"), ["Threshold 0.3", "True")
Open-Critt Example This dataset performs a programminng-rlated task: code generation. It is a synthetic code editing benchmark designed to train and evaluate large language models on their ability identify and fix bugs in code. Each example consists of natural language task description, a correct code solution that satisfies the task, and a counterfactual explanation that describes bugs introduced into similar but faulty implementation. The objective for the model learn to generate or repair the bugged version of the code based on the bug descriptions as counterfactual scenario. For instance, given a correct nested loop that prints all valid digit combinations excluding the number 5 and repetitions, the counterfactual explanation points out altered range values (e.g.. using range (1 (100) and range (1) ) ), leading to an incomplete iteration logic. The full example is shown as follows:
"task": "Create a nested loop to print every combination of numbers between 0-9, excluding any combination that contains the number Additionally, exclude any combination that contains repeating digit. implement the solution without using any built- in functions or libraries to check for repeating digits.",
"correct_code"; "for i in range (10): # First digitIn for j in (10) Second digitin for k in (10)e(10); and j
Third digit\ and k
Checking for the conditions\n := and i != and 1 != k and j != k:ln print (1, j, k)", "correct_expp exprect_expllaaaiioo". "This cede will generate and print every combination of three di digits between 0-9 that do not contain the number 5 and do not have any repeating digits.",
25

---

Published as a conference paper at ICLR 2026
"bugged code": "for i in range An for j in (100) in in range
if !- and != 5 and != and j and i != and j != ki (rint J, k)", wat.utorfaattallopplaaattttt
"Reading through this code, I notice that: \n\ns The second loop is set to 'range (100) instead of vange(10) which from to 99. This will cause the script to miss some combinations. \nw The third loop is set to 'range (1) which means it will only iterate over the number 0. This will cause the script to only print combinations with three. AnAnThese bugs will prevent the script from generating and printing all possible combinations of three digits between 0-9 that de not contain the number 5 and do not have any repeating AnkaTips for avoiding these mistakes:\n Double-check the range values in each loop sure the loops iterate correctly over the desired range of values."
will only iterate
the numbers
digit, instead of
they correct. \ne Make
Open-Critte Causality & Causal Graph.
"factual
"Exposure"; ["Correct explanation (valid ranges and checks) "I, "Covariate";
"Task requirements (0-9 digits)
"Exclusion logic (no 5/repests)
"Nested loop structure"
1,
"Mediator"; ["Proper range initialization (range (10) "3)
"Outcome"; ["Correct triple-nested loop code"]
*counterfactual __ar__rrools=_=___ {
"Exposure"; ("Counterfactual explanation (invalid ranges) "),
"Covariate":
"Task requirements (0-9 digits)",
"Exclusion logie (no 5/repeats)",
"Nested loop structure"
1,
"Mediator":
"Flawed range parameters (range (100) /range " "Incomplete digit iteratien"
"Outcome": ["Bugged code with limited iterations")
b.
"causal_graph"
"factual_odges: [
["Task requirements", "Correct explanation"), ["Exclusion logic", "Correct explanation"], ["Nested loop structure", "Correct explanation"), "TTask requirements", "Preper range initialization") "EExclusion logie", "Proper range initialization"); ''Nested loop structure", "Proper range initialization") ["Correct explanation", "Correct triple-nested loop code"), ["Task requitsment"", "Correct triple-nested loop code"), ("Exclusion logic", "Correct triple-nested loop code"), ["Correct explanation", "Proper range initialization"), ("Proper range initialization", "Correct triple-nested loop code"],
["Correct explanation", "Correct triple-nested loop code")
ccanterfaatual__ddees""
[TTask requirements", "Flawed range parameters"), ["Exclusion logic", "Flawed range parameters"), ["Nested 1oop structure", "Flawed range parameters"),
26

---

Published as a conference paper at ICLR 2026
""Task requirements" "Bugged code with limited iterations"), ("Exclusion logic", "Bugged code with limited iterations" ("Nested loop structure", "Bugged code with limited iterations"
1,
("Counterfactual explanation", "Flawed range paramaters"), ["Flawed range parameters", "Incomplete digit iteration"), ("Incomplete digit iteration", "Bugged code with limited iterations"!,
I"Counterfactual explanation", "Bugged code with limited iterations".
Code-Preference Example This dataset performs a programming-related task: code summarization. It contains pairs of duplicate code examples, with the only difference being the bugged code example has the bugged code 'surgically transplanted in' while the corrected code is left the same. Each example consists of natural language instruction, correct code solution that satisfies the instruction, and a hug explanation that describes bugs. The objective is for the model to learn to summarize and generate the bug descriptions as a counterfactual scenario. For instance, given a correct nested loop that prints all valid digit combinations excluding the number 5 and repetitions compared with a bugged loop, the bug description generated by the model will be able to point out altered range incomplete iteration logic in its
values (e.g.. using range (100) and range (1) leading summarized response bug explanation. The full example is shown as follows:
"Instruction": 'Create nested loop print every combination of nambers between 0-9, excluding any combination that contains the number 5. Additionally, exclude any combination that contains repeating digit. Implement the solution without using any built- in functions or librarics to check for repeating digits.',
"bugged_coder code": "What nfer in rang)(10): \n for in range (100): (100) range (1): \n if !- 5 and !- and != 5 and !- and != and i != k:ln print (i, 3, k) kn
the problems with this code? " "nnpythoon)
for in
"bug "buu__eppii exp.aaatiiinnn 'Reading through this code, I notice that: AnAn= "range(1CC) instead of 'range (10)
The second loop set which means it will only iterate over the numbers from to 99. This will cause the script to miss sone combinations Ane The third loop is set to 'range which means it will only iterate over the number 0. This will cause the script combinations with one digit, instead of three. Tn'ne bugs will prevent the script from generating and printing all possible combinations of three digits between C-9 that do not contain the number 5 and do not have any repeating digits. AnAnTips for aveiding these mistakes: AnAn Double-check the range values in each 1oop to ensure they are correct. \n+ Make sure the loops iterate correctly over the desired range of values. \n\nfere is the corrected "nde:\nnn""" digit for in range (10): # Third digitin != and i != and k != and i != and i != k and j != k:ln print (i, i, k) Nw "correct_code"; 'Here is example of nested loop in Python to print every combination of numbers between 0-9, excluding any combination that contains the number 5 or repeating digits:\nnn "'pytoonnnfor i in range (10): # Second digitin for k in range Checking for the conditions\n and != and aThis code will generate and print every combination of three digits between 0-9 that do not contain the number 5 and do not have any repeating digita.'
only print
First for in range # Checking for the conditions if
i in range (10): Second digitin
for i in range Third digitAn # if != and != 5 and k != and j k:An print (i, J. An Mnn
# First digit
27

---

Published as a conference paper at ICLR 2026
Code-Preferencee Causality & Causal Graph.
" "factual_roles";
"Exposure"; ["Correct code (range loops) "], "Covariate":
"Task requirements (0-9 digits)",
"Exclusion logiz 5/repeats))
"Nested loop structure"
"Meciator"; ["Proper range initialization (range (10) x3)
"Outcome": 1"Correct explanation (valid ranges and checks) "]
coonteerfactualrroles.
"Exposure; ("Bugged code (range (100) /range (1) "l,
Covariate":
"Task requirements (0-9 digits)",
"Exclusion logic (no 5/repeats)",
"Nested loop structure"
"Meciator"; ['Flawed range parameters", "Incomplete digit iteration")
"Outceme"; ("Bug explanation (incorrect ranges analysis)
"causal "
"factual "factual_edges"" :dges":
["Task requirements (0-9 digits)", "Correct code (range loops)
1,
("Exclusion logis (no 5/repeats)", "Correct code (range (10) loops)
1,
("Nested loop structure", "Correct code (range (10) lcops) ("Task requirements (0-9 digits)", "Proper range initialization range (10) x3)"i,
("Exclusion logic (no 5/repeats)", "Proper range initialization
range (10) x3)"
"Nested loop structure", "Proper range initialization (range((0)
x3)
[ ["Takk requirements (0-9 digits)", "Correct explanation (valid ranges and checks) "],
(EExclusion logic (no 5/repeats)", "Correct explanation (valid
ranges and checks) "I,
["Correct code (range Icaps)", "Proper range initislization range (10) x3)"
["Proper range initializaticn (range (10) "3)", "Correct explanation
(valid ranges and checks)
["Cerrect code (range lceps)", "Correct explanation (valid
ranges and checks)
},
"counterfactual
"Task requirements (0-9 digits)", "Flawed range parameters")
"Task requirements (0-9 digits)", "Incomplete digit iteration");
[ ["Task requirements (0-9 digits) "Bug explanation (incorrect
ranges an
['Exclusion logic (no 5/repeats)", "Flawed range parameters"), [Exxclusion logic (no 5/repeats)", "Incomplete digit iteration"!, "Exclusion logic (no 5/repeats)", "Bug explanation (incorrect ranges analysis)
["Nested loop structure", "Flawed range parameters"), "Nested loop structure", "Incomplete digit iteratien"), "Nested loop structure", "Bug explanation (incorrect ranges analysis) "),
[Buugged code (range (100) (range(1))",, "Flawed range parameters"), [#Flawed range parameters", "incomplete digit iteration"),
28

---

Published as a conference paper at ICLR 2026
Incemplete digit iteraticn", "Bug explanation (incerrect ranges analysis) "], "Bugged code (rango (100) /range (1))", "Bug explanation (incorrect ranges analysis)"
B COMPLEMENTARY EXPERIMENT
B.1 SETTING
This section lists the experimental settings used in this study.
Table 7: LLM query hyperparameters used during all experiments.
Hyperparameter
Value
Description
Temperature
Controls randomness generation Probability mass for sampling
Top-p (nucleus sampling)
0.95
Max tokens
2048
Maximum number of tokens to generate
Stop sequences
[" ["\n" ", "Q: "
Used truncate responses
Prompt format
CoT, CoT-SC, ToT Enabled (Selective)
Prompting strategy used Section 4.2.2
Tool-calling API
Used tool- aogmengeentedd experiments
Computational Resources. All experiments were conducted on a high-performance computing server equipped with six NVIDIA RTX 6000 Ada Generation GPUs, each with 49 GB of dedicated VRAM. The system utilized CUDA version 12.8 and NVIDIA driver version 570.1244 174.0.. These GPUs supported parallel execution model querying, evaluation, and rool-augmented tasks across benchmark datasets. The hardware configuration ensured sufficient memory bandwidth and processing capability to accommodate large-scale inference, particularly for multimodal tasks and multi-sample prompting strategies such as CoT-SC and ToT. No resource-related constraints were encountered during experimentation.
B.2 ADDITIONAL. RESULT AND DISCUSSION
This part presents additional experimental results that complement the main evaluation in the body of the paper. These supplementary findings, together with what has been presented in previous sections, offer comprehensive insights into LLMs capabilities across different decompositional tasks.
To deepen our understanding of how LLMs handle counterfactual reasoning, we analyze representative datasets that cover text, multimodal, symbolic, and code-based modalities. We aim to uncover impediments each decompositional stage. Below, highlight dataset-level characteristics that either enable or hinder performance.
Textual Question-Answertii and Logic Parsing, Datasets built on natural language (e.g., textual QA and logical modification tasks) generally facilitate the recognition of explicit causal variables such exposures and outcomes. Models can easily link stated intervention ('if tutoring was provided") to its corresponding variable. However, mediators are often described abstractly (through latent constructs like "trust," "motivation." "belief states")) rather than explicit entities. This makes Task disproportionately difficult for mediators compared to exposures or outcomes. Errors at this stage then propagate Task IV, where the model must simulate how such mediators would affect outcomes under intervention. Even when causal graph construction (Task II) is near-perfect due the structured nature of logical relations, the absence explicit mediators limits downstream reasoning fidelity.
Vision-Language Counterfactuals. Multimodal datasets combining images with text pose unique challenges. When asked identify causal variables, LLMs must ground textual descriptions in visual objects. For example, distinguishing "presence of a ball" (exposure) from "action of kicking" (mediator) requires fine-grained alignment of object attributes with causal semantics. This grounding
29

---

Published as a conference paper at ICLR 2026
Table 8: LLMS' performance (FI standard deviations, scaled to 100%) in causal variable identification (reordered; adjusted variability).
GPT-5
GPT-o4
Qwen3
Llama4-S
Llama4-M
Gemini2.5
DeepSeck
Datasel
v2
22
V1
12
92
81
U/
Vi
(Exposure). U2 (Covariate)
CRASS
4.6
4,9
9.1
9.9
5.6
6.2
8.3
7.9
3.5
4.4
4.5
5.3
4.8
5.0
CLOMO
4.8
5.1
9.3
10.0
7.2
9.6
8.9
5.0
5.9
6.5
5.3
5.7
RNN-Topo
9.5
10.2
9.5
5.8
CVQA-Bool
5,4
5.7
9.8
10.4
10.2
12.4
13.1
8.9
10.8
11.5
9.4
10.1
CVOA-Count
5.6
5.9
9.9
10.5
10.6
11.3
13.7
14.2
9.8
10.5
11.5
12.3
10.3
11.2
COCO
5.8
6.1
10.0
10.6
11.9
12.6
15.2
15.8
11.1
11.8
12.9
13.7
11.6
12.5
Arithmetic
5.2
5.5
9.4
10.1
8.9
9.5
11.3
12.0
7.5
8.1
8.4
9.1
7.9
8.6
MalAlgoQA
9.6
10.1
12.2
12.9
8.2
9.7
8.5
9.2
HumanEval-Eix
6.0
6.3
12.8
13.6
16.3
17.1
12.4
13.9
14.7
12.5
Open-Critic
6.4
10.3
10.9
13.5
14.2
17.1
17.8
12.6
13.3
14.6
15.3
13.7
14.5
Code- Preeerefeeeece
6.0
6.2
10.1
10.7
13.1
13.9
16.7
17.4
12.2
12.9
14.3
15.0
13.1
13,9
M (Mediator),
U2
Y (Outcome)
CRASS
4.9
5.2
9.2
10.0
9.6
7.4
12.1
9.7
8.5
9.1
6.8
8.7
6.2
CLOMO
9.4
12,9
7.5
9.3
RNN-Topo
5.1
5.4
9.6
10.3
11.1
8.5
13.6
10.8
9.6
10.5
7.9
9.8
7.2
CVQA-Bool
5.7
6.0
10.0
10.6
13.8
11.3
16.9
14.2
12.4
10.0
14.1
12.5
13.2
10.7
CVQA-Count
5.9
6.2
10.1
10.7
14.6
12.1
17.8
15.1
13.3
10.8
15.0
13.3
14.1
11.6
COCO
6.0
6.3
10.2
10.8
15.3
12.8
18.5
15.9
14.0
11.5
15.7
14.1
14.8
12.4
Arithmetic
9.5
12.5
9.8
15.4
13.1
9.2
11.3
8.7
MalAlgoQA
5.6
5.9
9.7
13.1
10.4
16.1
13.2
13.8
10.0
12.0
9.3
HumanEval-EE Exe
6.2
6.5
10.4
11.0
16.2
13.6
19.3
16.7
14.8
12.3
16.5
14.8
15.5
13.1
Open-Critic Crit-rriicc
6.3
6.6
10.5
11.1
16.9
14,4
8.1
17.5
15.5
13.0
17.2
15.5
16.1
13.9
6.1
6.4
10.3
10.9
16.5
14.0
19.7
17.0
15.2
12.6
16.9
15.1
15.8
13.5
Canin2.5
DeesSock
1.04
0
2-M Z M
X-M Z-M 2 N-- x
0n
X-M 2+x X 2
2-18 24x M X
@ CLOMO
(b) RNN Topology
(c) CVQA Bool
(d) CVGA Count
03
X- 2 2-4X M x Z-Y
X- Z-H 2-x MaY
X Z
X-MM Z-M M-X--A R4Y X Z-Y
(4) Artthmatic
10 EvalnEnallEsse
[g] Code-Pruference
Figure 5: Additional evaluation on causal graph construction, complementing to Figure
step introduces errors in Task especially when visual scenes are cluttered or ambiguous. Even when interventions (Task III) are identified correctly, outcome reasoning (Task IV) suffers because models struggle to propagate visual changes into numerical behavioral predictions. For instance, recognizing that "removing a ball" should reduce the count of possible goals involves chaining visual detection, counting logic, and causal propagation- steps that current LLMs rarely integrate coherently.
Symbolic and Mathematical Reasoning. Datasets built on arithmetic or algorithmic transformations highlight another bottleneck: reliance memorized patterns instead of causal mechanisms. In variable identification (Task I), explicit quantities are correctly recognized. In causal graph con- struction (Task II), rules linking operations (e.g., "base conversion influences the final number")
30

---

Published as a conference paper at ICLR 2026
GPT-5
GPT-o4
@Wen-VL
Llama4-S
Llamz4-M
Gemmi22.5
DeepSook
1.0
0.8
CIOMO
RNN-Topology
CVQA-Bool
CVQA-Count
0.9
0.8
Arithmetic
HumanEval-Exe
Code-Preference
Figure 6: Additional evaluation LLMs' intervention identification, complementing to Figure
Table 9: LLM performance (FI standard deviation) in reasoning the counterfactual mediator (M') and outcome (Y").
GPT-S
GPT-o4
Qwen3
Llama4-S
Llama4-M M
Gemini2.5
DeepSeek
Dataset
M'
y
M'
Y'
M'
Yr
M'
y'
M
y
M'
Y'
M'
CRASS
4.2
5.0
4.8
5.6
6.7
8.5
10.0
12.5
5.0
7.1
6.3
8.1
5.5
7.4
CLOMO
4.4
5.2
5.0
5.9
7.2
9.0
10.6
13.0
5.4
7.5
6.7
8.6
6.0
7.9
RNN-Topo
4.6
5.4
5.1
6.0
7.8
9.5
11.0
13.5
5.6
7.7
7.2
9.1
6,5
844
CVQA-Boo Bool
4.8
5.6
5.2
11.2
13.3
14.5
16.8
9.0
11.1
12.1
14.0
10.4
CVQA-Count
5.0
5.8
12.0
14.1
15.2
17.6
9.8
11.9
12.9
14.9
COCO
5.1
12.7
15.9
18.3
10.5
12.7
13.6
Arithmetic
4.5
5.2
5.1
6.0
11.6
12.8
15.3
7.4
9.5
9.0
11.1
8.3
10.4
MalAlgoQA
4.7
5.3
5.2
6.1
10.1
12.2
13.4
15.9
7.9
10.0
9.7
11.8
8.9
11.0
HumanEvall-e Eval-Exe
5.2
6.2
5.5
6.5
13.3
15.4
16.5
18.9
11.0
13.2
14.2
16.2
12.5
14.7
Open-Critic
5.3
6.3
5.6
6.6
13.9
16.0
17.0
19.6
11.6
13.8
14.8
16.9
13.1
15.3
Code-Preference
5.1
6.1
5.5
6.5
13.6
15.7
16.7
19.2
11.3
13.5
14.5
16.5
12.8
15.0
applied consistently. However, in outcome reasoning (Task IV), models frequently fail to simulate the correct causal pathway, often defaulting to template-based responses rather than computing the actual counterfactual result. This suggests that while symbolic data supports high precision in explicit structure, it exposes the weakness of LLMs in mechanistic simulation of causal processes.
Code-Basse Reasoning. Programming-orientn orienttd datasets such as code execution, code generation, or preference tasks are particularly difficult across all stages. In Task I, identifying exposures and outcomes is hindered by the abstractness of programming constructs (e.g.. "function signature" as exposure, "program output" as outcome). Mediators (such as intermediate execution states) are even harder to capture, as they are explicitly represented in the code text but must be inferred from semantics. In Task II, while models can generate plausible causal graphs describing dependencies among variables or functions, these graphs often overgeneralize or miss critical execution details. Task IV is especially challenging: even when interventions like "changing loop to recursion" are recognized, LLMs often fail to simulate downstream program behavior, producing outcomes that logically plausible but incorrect. This reflects persistent gap between syntactic recognition and semantic reasoning.
Cross-Cutting Observations. Across modalities, two key impediments recur:
(1) Complex modalities impede variable identification. Images and code introduce higher error rates in Task since grounding semantic parsing must precede causal reasoning.
(2) Implicit mediators bottleneck outcome reasoning. Regardless modality, when mediators are abstract not explicitly present, performance in Task IV drops substantially. LLMs can identify interventions reliably, but they fail to propagate their effects along causal chains to yield consistent counterfactual outcomes.
31

---

Published as a conference paper at ICLR 2026
These findings suggest that LLMs "eligible" for decompositional reasoning in structured settings with explicit causal variables (e.g., clean text or arithmetic). However, when confronted with modality complexity or implicit causal pathways, their reasoning capacity is significantly impeded.
B.3 WORKING MEMORY PERSPECTIVE TO INTEPRET COUNTERFACTUAL REASONING
Prior research (Zhang et al. 2024a) has demonstrated that language models exhibit notable difficulty temporally storing and manipulating information even in n-back tasks that are cognitively simpler than explicit reasoning. This underlying limitation in working memory capacity poses constraints on long-term and multi-step reasoning. To explore the connection between memory bottlenecks and mediator identification challenges, conducted additional experiments in more depth.
Experiments on Working Memory. To examine how working memory affects mediator reason- ing, designed controlled n-back Mediator Recall task. While most benchmarks involve only single-step mediation, the Open- Critic dataset (code modality) includes examples of multi-step causal mediation. For instance, adjusting code inputs requires reasoning over prior inputs and transforma- tions. In this task, the mediator must be inferred from causal variables presented steps earlier in the input. We vary n from and report F1 scores consistent with Table T0
Table 10: LLM performance (F1)
n-back mediator recall
Model
1-hop
2-hop
3-hop
GPT-o4 72.2%
63.5%
9.7%
Qwen
58.3%
39.6%
12.1%
Gemini
66.4%
26.1%
7.5%
LLaMA4-Scout
45.5%
47.2%
3.6%
Findings and insights. These results reveal a sharp performance drop as the number of intermediate steps increases. From working memory perspective, this suggests that current LLMMs struggle to retain or reconstruct causal paths to mediators when they are separated by multiple reasoning hops. This degradation highlights key constraint in long-horizon causal reasoning.
These findings align with our earlier observation that mediator reasoning is consistent bottleneck in decompositional analysis. By framing this terms of working memory capacity, we offer mechanistic explanation for why LLMs falter on such tasks and why enhanced memory mechanisms (e.g., intermediate supervision or tool-assisted retrieval) may be necessary for progress.
B.4
INFLUENCE OF MODEL SCALE
Table 11: LLM performance in reasoning variables (X, Z, M,M, M,M',,,,, YY).
Qwen- VL-2BB
Qwen-VLL.4p VL.4B
Qwen-VL VL.SB
Dataset
M'
Y'
X
CRASS
44.2
27.1
20.3
37.2
18.2
16.5
47.0
29.5
22.1
39.4
20.3
18.7
52.8
34,4
26.7
44,5
25.8
23.6
CLOMO
23.8
18.4
11.6
15.3
8.3
5.1
26.0
20.2
13.0
17.0
9.6
6.4
30.5
24,0
20.8
12.5
9.3
CVQA-Count
25.4
21.3
16.1
MalAlgoQA
17.2
14.0
10.5
14.6
8.5
6.2
19.1
15.6
12.0
16.3
10.2
7.6
23.4
19.3
15.7
19.7
13.6
10.5
Besides comparing GPT and Llama families, we further conduct a controlled scaling study over the Qwen-VL series to examine how model size influences both causal variable identification and down- stream counterfactual reasoning. Table [T summarizes the performance of Qwen-VIL-2B. 4B, and 8B across factual variables (X, Z, M,, Y and counterfactual targets (M' Y'). Overall, we observe a clear but non-uniform scaling trend. Moving from 2B to 4B yields modest and consistent gains, particularly on explicit variables (X, Z, Y), suggesting that moderate scaling primarily improves surface-lewe grounding and extraction. In contrast, the 8B model shows more noticeable improve- ments across both explicit and implicit variables, including the more challenging mediators (M, M') and counterfactual outcomes (Y'). These gains indicate that larger Qwen models are better able to propagate interventions through the underlying causal structure rather than solely memorizing lexical
32

---

Published as a conference paper at ICLR 2026
associations. However, even the 8B model retains substantial gaps tasks requiring multi-hop causal reasoning, especially when mediators are implicit or visually grounded. Taken together, the scaling analysis suggests that increased model capacity helps alleviate some of the bottlenecks identified in smaller models, but does not by itself resolve the fundamental challenges of counterfactual mediation and outcome inference.
33


Reviews:
[{"review_id": "KtGwcjz5Du", "reviewer": "Reviewer_aJkt", "summary": "The paper proposes a decomposition of counterfactual reasoning into smaller sub-tasks: causal variable identification, causal graph construction, counterfactual identification, and outcome reasoning. The authors evaluate the performance of different LLMs in performing these sub-tasks across a variety of datasets, identify variable identification and counterfactual identification as a weak point, and propose strategies to improve it.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It has been observed repeatedly that LLMs perform worse when answering counterfactual queries relative to factual ones. The paper’s attempt at understanding why in a more fine-grained fashion is a significant problem.\n\nExperimental evaluations are comprehensive in terms of the number of models and the variety of datasets they consider.", "weaknesses": "It is not clear how the performance on the four sub-tasks relate to the end-to-end performance. Establishing this relation is important when interpreting performance on these sub-tasks as decomposition.\n\nFor instance, the paper concludes that LLMs are generally better at Task 1 than Task 2 conditioned on correct results from Task 1 (which is supported by experiments and indeed seems to be the case). However, it could still be the case that starting from the inputs of Task 1 and directly querying for the outputs of Task 2 is as easy as performing Task 2 conditioned on correct results from Task 1. If, for instance, processing X, Y, Z variables implicitly is somehow easier than outputting them explicitly.\n\nWhile the above is probably unlikely to be the case, it is a crucial validation to perform in order to confirm that Task 1 and Task 2 (conditioned on Task 1 results) behave as intended. Evidence provided in the paper does not provide support for attributing poor end-to-end Task-1 inputs to Task-2 outputs performance to poor Task 1 performance. \n\nI want to emphasize that this is a general consideration for all sub-tasks, the entire chain, not just Tasks 1 and 2 specifically.\n\nThe paper doesn’t discuss the related work to a sufficient depth. In particular, other methods of evaluating counterfactual reasoning and whether there have been attempts at decomposing this process into smaller steps in evaluation is not discussed.\n\nThe four sub-tasks that are identified, while generally sound, are arbitrary. Their choice is not motivated.\n\nWhile evaluations in Section 4.1 are interesting, they are not exactly actionable. They help identify which sub-tasks are the weak points to be improved (which arguably all a decomposition is supposed to do) and the paper suggests improvement for these weak points. However, the suggestions take the decomposition as a given. When the goal is to answer a question end-to-end, starting from the given of Task 1 and providing the outputs of Task 4, the paper does not explore the impact of going through Tasks 1, 2, 3, and 4 as a pipeline rather than direct queries. Moreover, it does not explore whether the task specific improvements in Section 4.2 lead to a comparable end-to-end gain as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iYa8xAk9Q9", "reviewer": "Reviewer_Ndd6", "summary": "This paper examines counterfactual reasoning in LLMs across a range of modalities and models.\nThe paper shows that by decomposing the reasoning scenario to causality construction and counterfactual intervention, the reasoning ability of LLMs under counterfactual scenarios can be greatly improved.\nBy leveraging tools that can help the model identify the causal variable, the performance can be further improved.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall methodology of decomposing the counterfactual reasoning process is novel and the experiments show this really helps.\n2. The experiments cover a wide range of dataset design specifically for counterfactual reasoning.\n3. The final proposed method seems to be easy to adopt for any LLM for reasoning.", "weaknesses": "1. The experiments covers many datasets, but it lacks comparison on model scale, for example, Qwen 3 provides models across different scales, it could make the paper stronger if some results are shown there.\n2. The NER tools are designed to use Bert like models, however, would it be possible that the tools are instantiated by another model using different prompts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wuo67Bp2W", "reviewer": "Reviewer_BrYZ", "summary": "This paper makes three key contributions:\n\n- It introduces a **decompositional framework** for evaluating counterfactual reasoning in LLMs, breaking it into four stages: causal variable identification, causal graph construction, counterfactual intervention identification, and outcome reasoning.\n- It constructs a **comprehensive multimodal benchmark** using 11 datasets across text, image, code, and symbolic modalities, enriched with causal annotations and graphs.\n- It provides an **in-depth evaluation and improvement strategy**, identifying key bottlenecks (especially in mediator reasoning and cross-modal tasks) and proposing tool augmentation and advanced prompting techniques to enhance LLM performance.", "questions": "The benchmark relies on curated causal structures. To what extent do you believe the performance on these structured tasks generalizes to real-world scenarios where causal graphs are not provided and must be inferred from raw data? Did you run any experiments on \"raw\" data without this scaffolding?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Systematic and Granular Evaluation Framework:** The paper's primary strength is its decompositional approach, which breaks down the complex task of counterfactual reasoning into four distinct, measurable stages. This allows for a much more precise diagnosis of *where* and *why* LLMs fail, moving beyond a monolithic \"pass/fail\" assessment to identify specific bottlenecks, such as the particular difficulty with implicit mediators.\n\n2.  **Comprehensive and Multimodal Benchmark:** The authors construct a substantial and diverse benchmark by curating 11 datasets spanning various modalities (text, image, code, symbols) and tasks. This breadth ensures that the findings are not limited to a single domain and provides a robust testbed for evaluating the generalizability of LLMs' counterfactual reasoning abilities.\n\n3.  **Actionable Insights and Proposed Solutions:** The paper goes beyond mere identification of problems by proposing and evaluating concrete strategies for improvement. The demonstration that tool-augmentation can alleviate modality-specific issues and that advanced prompting can help with implicit reasoning provides valuable, actionable directions for future research aimed at enhancing LLM reasoning capabilities.", "weaknesses": "I think there are 2 brief weaknesses of the paper:\n\n*   **Potentially Artificial Evaluation:** The benchmark relies on pre-annotated causal structures, which may not reflect the challenge of inferring causality from raw, unstructured data.\n\n*   **Surface-Level Diagnosis:** The analysis identifies performance bottlenecks but offers a high-level explanation (e.g., \"working memory\") without deeply investigating the underlying architectural mechanisms in LLMs that cause these failures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y58uSFR7bi", "reviewer": "Reviewer_fFFZ", "summary": "Large language models typically struggle on counterfactual reasoning tasks, however it is unclear *why* they struggle, particularly at which parts of such reasoning. This submission proposes a benchmark to evaluate different aspects in LLMs' causal reasoning.They isolate that LLMs struggle to properly identify causal variables and find the counterfactual mediator and outcome even when given the variables, their relationships and the intervention to perform.", "questions": "- Can you answer/address the above weaknesses?\n\n- In the datasets, are the examples *only* made up of the $X,T,M,Y$ variables, or are any other variables present such that the model must extract the relevant variables *among other options*?\n\n- Did humans or LLMs annotate the paper's benchmark dataset? Which of these scored LLMs' answers, or was this done through another method?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The approach is original; I have never seen any attempt at explaining the difficulties of LLMs in causal reasoning.\n\n- The division into subtasks looks sound to me.\n\n- Diversified source of tasks.", "weaknesses": "- The benchmark seems to impose the strict presence of four variables : covariates, treatment, mediator, outcome. However, the mediator could be absent from the context, as in the example:  “A person is running a marathon and collapses.” l.467-469. It is unclear why LLMs should explicitly,identify “Dehydration” as a mediator, and not other sources.\n\n- It is unclear what prompts are given to LLMs : from my understanding, it is the \"CRASS Prompt\" given in Appendix A, but it seems to apply to all datasets?\n\n- Typo? \"the primary challenge lies not in\" l.390-391 : it should be \"lies in\", right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)