Ice Cream Doesn’t Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference
Authors: Jin Du
Li Chen
Xun Xian
An Luo
Fangqiao Tian
Ganghua Wang
Charles Doss
Xiaotong Shen
Jie Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Causal inference
Primary area: datasets and benchmarks
TL;DR: We present CausalPitfalls, a benchmark that systematically evaluates LLMs' reliability in causal inference, revealing how models can fall into statistical traps like Simpson’s or Berkson's paradoxes, even when their answers appear accurate.

Abstract: Reliable causal inference is essential for making decisions in high-stakes areas like medicine, economics, and public policy. However, it remains unclear whether large language models (LLMs) can handle rigorous and trustworthy \textit{statistical causal inference}. Current benchmarks usually involve simplified tasks. For example, these tasks might only ask LLMs to identify semantic causal relationships or draw conclusions directly from raw data. As a result, models may overlook important statistical pitfalls, such as Simpson’s paradox or selection bias. This oversight limits the applicability of LLMs in the real world.
To address these limitations, we propose \textbf{CausalPitfalls}, a comprehensive benchmark designed to rigorously evaluate the capability of LLMs in overcoming common causal inference pitfalls. Our benchmark features structured challenges across multiple difficulty levels, each paired with grading rubrics. This approach allows us to quantitatively measure both causal reasoning capabilities and the reliability of LLMs' responses. We evaluate models using two protocols: (1) direct prompting, which assesses intrinsic causal reasoning, and (2) code-assisted prompting, where models generate executable code for explicit statistical analysis. Additionally, we validate the effectiveness of this judge by comparing its scoring with assessments from human experts.
Our results reveal significant limitations in current LLMs when performing statistical causal inference. The CausalPitfalls benchmark provides essential guidance and quantitative metrics to advance the development of trustworthy causal reasoning systems. Our code is publicly available at   \href{https://github.com/dudududuu/CausalPitfalls}{CausalPitfalls}}.

Published as a conference paper at ICLR 2026
ICE CREAM DOESN'T CAUSE DROWNING: BENCH- MARKING LLMs AGAINST STATISTICAL PITFALLS IN CAUSAL INFERENCE
Jin Du', Li Chen', Xun Xian', An Luo', Fangqiao Tian', Ganghua Wang", Charles Doss', Xiaotong Shen', Jie Ding'
Schooo of Statistics, University of Minnesota, Minneapolis, MN 55455 "Data Science Institute, University of Chicago, Chicago, IL 60637
(du000142, chen7019, xian0044, luc00318, tian0257eemmn edu [cdoss, xshen, dingi] Bumn. edu, ganghua@uchicago.com
ABSTRACT
Reliable causal inference essential for making decisions in high-stakes areas like medicine, economics, and public policy. However, it remains unclear whether large language models (LLMs) can handle rigorous and trustworthy statistical causal inference. Current benchmarks usually involve simplified tasks. For example, these tasks might only ask LLMs to identify semantic causal relationships or draw conclusions directly from raw data. As result, models may overlook important statistical pitfalls, such as Simpson's 's paradox or selection bias. This oversight limits the applicability of LLMs in the real world. To address these limitations, we propose CausalPitfalls, a comprehensive benchmark designed to rigorously evaluate the capability of LLMs in overcoming common causal inference pitfalls. Our benchmark features structured challenges across multiple difficulty levels, each paired with grading rubrics. This approach allows us to quantitatively measure both causal reasoning capabilities and the reliability of LLMs' responses. We evaluate models using two protocols: (1) direct prompting, which assesses intrinsic causal reasoning, and (2) code-assisted prompting, where models generate executable code for statistical analysis. Additionally, we validate the effectiveness of this judge by comparing its scoring with assessments from human experts. Our results reveal significant limitations in current LLMs when performing statistical causal inference. The CausalPitfalls benchmark provides essential guidance and quantitative metrics to advance the development of trustworthy causal reasoning systems. Our code is publicly available at CausalPitfallss
INTRODUCTION
Causal inference (Pearl 2009 Imbens & Rubin 2015) is fundamental to decision-making across diverse fields. For instance, accurately determining the effectiveness and safety of a vaccine is pivotal in public health decisions (Voysey et al. 2021). However, identifying causal relationships with both reliability and interpretability remains challenging. In practice, individuals without formal statistical training frequently fall into subtle pitfalls, leading to plausible yet incorrect conclusions. A classic illustration is the erroneous conclusion that ice cream sales cause drowning incidents overlooking the hidden confounder of hot weather causing both events (Pearl 2009 Greenland & Robins 1986
Rosenbaum 1987).
Given these complexities, automated tools like large language models (LLMss present promising avenues, demonstrated by their effectiveness scientific problem-solving (Lewkowycz et al. 2022
Achiam et al. 2023) and clinical reasoning (Singhal et al., 2023). Recent studies (Wang 2024
Dhawan et al. 2024 Liu et al., 2024) have evaluated LLMs' abilities to evaluate accuracy in causal- effect estimation. but these benchmarks often neglect crucial aspects like robustness, interpretability.
and susceptibility to common causal pitfalls. As a result, LLMs can produce seemingly convincing yet misleading causal conclusions.

---

Published as a conference paper at ICLR 2026
To illustrate why reliability assessment is crucial, highlight representative failure modes (detailed in Section B). First, LLMs can ignore strong data evidence, but in favor of superficial semantic cues: in synthetic health scenario with identical datasets, LLMs concluded the drink was beneficial when labeled "HealthPlus" and harmful when labeled "UltraSugar," even when the data indicated the opposite. Second, L.MS mistake random variation for genuine causal structure: when tested on research funding data from the Netherlands, LLMs attributed differences in success rates to gender bias Simpson's paradox, despite statistical analyses showing that neither claim is supported. These cases demonstrate that LLMs may produce confident causal claims directly contradicted by the data, showing the need for benchmarks that assess causal inference reliability.
Direct Prompting
Code-assisted Prompting
Cisude 3.5 sonnet
Deepseek-chal
Confounding
Confounding
GFT-04-mini
Gemini-2 D-flsh
Interventions
External
External
80
Interventions
Laama3.1-706
80
40
40
Counterfactual
Counterfactual
Causal Discovery
Causal
Discovery
Mediation Effect
Mediation Effect
Our results reveal clear reliability gap in causal inference when LLMs rely only on direct prompting, with all models struggling most on mediation and external validity questions. Introducing code-assisted prompting leads substantial gains across every task and brings all models closer together in performance. This shows that executable analysis is essential for large language models to handle complex statistical challenges and deliver trustworthy causal conclusions. Full results for all evaluated LLMs are provided in Table 4
Figure 1: Overall Message:
1.1 MAIN CONTRIBUTIONS
First, we introduce CausalPitfalls, a novel comprehensive benchmark specifically designed to evaluate the reliability of large language models (LLMs) in statistical causal inference. Unlike existing benchmarks primarily focused on accuracy. our benchmark targets model susceptibility to common causal pitfalls shown in Figure 1 including (1) Confounding Biases and Spurious Associations, (2) Interventions and Experimental Reasoning, (3) Counterfactual Reasoning and Hypotheticals, (4) Mediation and Indirect Causal Effects, (5) Causal Discovery and Structure Learning, and (6) Causal Generalization and External Validity (Pearl et al. 2003 Peters et al., 2017). These categories are structured into 15 distinct challenges, encompassing a total of 75 evaluation questions and 75 carefully constructed datasets that systematically test the robustness of LLM causal reasoning capabilities.
Second, we comprehensively evaluate the reliability of ten LLMs under distinct evaluation protocols: (1) direct prompting, assessing intrinsic causal reasoning from raw data, and (2) code- assisted prompting, where models generate executable code to perform statistical analyses before responding. This
approach provides detailed quantitative assessment, highlighting areas where computational assistance significantly improves causal reasoning and where intuitive reasoning could suffice.
Third, we introduce a quantitative metric termed causal reliability, calculated as the average normal- ized score across all benchmark challenges, enabling standardized comparisons of LLM reliability in causal reasoning tasks. By systematically quantifying reliability, this metric provides a crucial framework for future research aimed at developing more robust and trustworthy causal inference capabilities in Al systems.
2

---

Published as a conference paper at ICLR 2026
a
b
Q: Does this treatment improve recovery?
Q: Does this treatment improve recovery?
0
(1) Direct Prompt
(2) Code-assisted Prompt
Question: (O) Data: (raw data)
Question: (Q)
Data samples (10 rows)
Recovary Rate
Helpful!
</>
A: given data, we conclude that......
Treatment Consumptions
A: From the statistical results, conclude that......
Older
Recovery Rate
Grading Rubric
33
Harmful!
Younger
Grader
Treatment Consumptions
Figure 2: High-level overview of the CausalPitfalls benchmark. (a) An illustrative real-world pitfall (Simpson's 's paradox): when data on treatment consumption and recovery are pooled (top), a naive analysis finds positive effect ("Helpful!"",, but stratifying by age reveals negative effect within both younger and older subgroups ("Harmful!"". (b) Benchmark workflow: LLMs are evaluated data, assessing intrinsic causal reasoning, and (2) Code-Assisted Prompting on sampled data, assessing computationally grounded inference. both cases, model answers automatically scored against hidden grading rubric by independent grader to quantify each model's causal reliability.
under two protocols: (1) Direct Prompting
1.2 RELATED WORK
1. Causal Inference and Statistical Pitfalls. Causal inference from observational data is inherently challenging because counterfactuals are unobservable and confounding ubiquitous (Pearl 2009 [m-
bens & Rubin 2015). Causal inference methods for addressing confounders, whether the confounders
are measured Chan et al. 2016 Lin et al., 2023] Doss et al., 2024) or latent (Kang et al.) 2016 Guo
et al. al. 2022), depend restrictive model assumptions that often difficult to verify empirically.
Inferring causal direction similarly hinges on stringent structural equation assumptions (Peters
Bühlmann 2014 Li al. 2024) auxiliary information such valid instruments (Chen al. 2024). Mediation analysis (MacKinnon) 2012] Yang et al. 2021). which targets specific causal pathways, demands careful adjustment for intermediate variables to avoid post-treatment bias. Fi- nally, transporting causal conclusions across different domains requires justification of Nource-target invariances and methodologies for causal knowledge transfer (Wei al. 2023 Chen al. 2025).
Rigorously confronting each of these challenges is essential to conduct reliable causal ana analysis.
LLMs for Causal Reasoning, Recent studies have extensively investigated the causal reasoning capabilities of LLMs (Willig et al. 2022; Zečević et al. [2023] Qin et al. (019)). For example, Kiciman et al. (2023) demonstrated that L.MSs can infer causal relationships only from variable names, outperforming traditional statistical approaches (Peters et al.,, 2017). However, these evaluations focus on scenarios involving commonsense causality. To bridge this gap, Jin al. (2023) introduced synthetic datasets generated from causal graphs, thereby enabling the assessment of LLMs' causal reasoning performance in contexts extending beyond commonsense knowledge. Additionally, recent works (Long et al.) 2023 Zhou al. 2024) have assessed LLMs' capabilities in data-driven causal inference tasks, focusing on accuracy in estimating causal effects and recovering DAG structures from observational data.
3

---

Published as a conference paper at ICLR 2026
2 BENCHMARK CURATION
2.1 PITFALL CATEGORIES AND CHALLENGES
To evaluate the reliability of causal inference performed by LLMss, introduce the benchmark CausalPitfalls to assess model performance across common statistical pitfalls. Specifically, our benchmark addresses six major categories of causal inference pitfalls, consisting of 15 distinct challenges. Each challenge includes five questions across difficulty levels ranging from "very easy" "very hard." Table summarizes the categories and their respective challenges:
Table 1: CausalPitfalls benchmark categories and challenges
Confounding blases and spurious associations Simpson's & paradox
Interventions and experimental reasoning
Ohservational vs experimental reasoning Causal effect estimation
Selection hias (Berkson's 's paradox)
Counterfactual reasoning and hypotheticals
Mediation and indirect causal effects
Counterfactual outcome prediction
Mediatur-outcomi confounding Sequential mediators
Causal necessity and sufficiency
Treatment-medde interaction effects
Causal discovery and structure learning Cause-effect direction inference Handling uncertainty in causal structures
Causal generalization and external validity
Population shift and transferahility
Temporal stability of causal effects Contextual interaction and moderation effects
Domain shift and transportability of causal knowledge
Below brief overview of the six major categories in the CausalPitfalls benchmark:
Confounding biases and spurious associations: Covers scenarios where misleading correlations arise due to hidden variables or biased conditioning.
Interventions and experimental reasoning:
Focuses on distinguishing correlation from causation
through randomized experiments or interventional data, and quantifying treatment effects.
Counterfactual reasoning and hypotheticals: Evaluates LLMs' ability answer "what if" questions by reasoning about alternate outcomes under hypothetical changes.
Mediation and indirect causal effects: Assesses whether models can identify and reason through intermediate causal pathways, including complex mediation structures.
Causal discovery and structure learning: Tests the ability to infer causal directionality or relationships from data without pre-defined causal graphs.
Causal generalization and external validity: Examines whether causal conclusions generalize across different contexts, populations, and environments.
Many these scenarios can also be framed as purely statistical association problems. However, throughout this benchmark, adopt causal interpretation grounded in either the Neyman-Rubin po- tential outcomes framework (Splawa-Neyman et al., 1990. Rubin 1974) or Pearl's do-calculus (Pearl 2009). This distinction ensures that tasks target causal reasoning, rather than surface-level correlations.
One illustrative example of these pitfalls Simpson's paradox (Simpson 1951 (Fig 24), commonly misunderstood statistical phenomenon. Simpson's 's paradox occurs when overall data seem to suggest one conclusion: for example, patients recover better with a particular medical treatment. At first glance, the treatment might appear beneficial. However, once the patients are divided into subgroups based on age, the same data show that the treatment is harmful within each age group. An incorrect analysis would overlook the importance of these subgroups, mistakenly suggesting the treatment is effective. Importantly, the stratification variable in Simpson's paradox is assumed to be a confounder, not a collider.
2.2 EVALUATION PROTOCOLS AND DATA
LLM-based Causal Inference Protocols. In this paper, we consider two unified protocols to evaluate LLM reliability (Fig. Zp): (1) Direct Prompting: LLMS directly answer causal inference
4

---

Published as a conference paper at ICLR 2026
Table 2: Questions by Difficulty Level for "Simpson's Paradox"
Difficulty
Question
Very Easy
Evaluate whether [TREATMENT] } causally affects {OUTCOME). Adjust for the known confounder (1 ((CONOUNNEE)) using stratification or regression. State if Simpson's puradox is present, and provide adjusted rates with 95% confidence intervals along with a recommendation.
Easy
Evaluate whether (TREATMENT) causally affects {CUTCOKE). Consider the potential influence of the confounder ( ([CONOOUDER]). adjust accordingly, and assess Simpson's 's paradox. Provide adjusted effect sizes with uncertainty estimates and recommendation. Evaluate the causal impact of {TREATMeNT] {OUTCOME). Account for relevant confounders. Provide adjusted effects with uncertainty measures and justify your recom- mendation.
Medium
Hard
Assess the causal relationship between (IREATMENT) and (OUTCOME). considering potential confounders.
Very Hard
Evaluate whether (TREATMENT) causally affects {OUTCOME) without additional hints.
questions based the provided raw data. This approach tests the models' intrinsic capability to perform causal inference without additional computational tools or external support. (2) Code- Assisted Prompting: LLMs generate executable code perform statistical analysis relevant the questions, then interpret the results to answer the questions. This method assesses the LLMs' ability translate causal reasoning tasks into accurate computational procedures and use analytical results to avoid common pitfalls.
Questions. Each challenge includes five versions of the core question, ranging from very easy to very hard (Table The easier versions give the model more guidance. For example, pointing out the confounder to adjust for asking directly about Simpson's 's paradox. As the difficulty increases, these hints are gradually removed. This setup lets us test whether models can still recognize and handle causal pitfalls when less direction is given (Fig. 2)
Datasets. To construct datasets tailored to each challenge, utilize causal graphs following Pearl et al.] (2003) and Peters et al. (2017). For every statistical pitfall, we select causal graphs that capture its unique complexities and characteristics. Each challenge is accompanied by five distinct datasets, each containing over 500 samples for comprehensive evaluation. Our simulation approach uses structural causal models based on directed acyclic graphs (DAGs), where each structural equation represents a causal mechanism rather than merely statistical association. The coefficients in these equations directly encode the causal effects, allowing us to define the ground truth against which inference methods can be evaluated. This approach is mathematically equivalent to simulating potential outcomes under the specified causal structure. The structural equations include both linear and non-lines linear forms (c.g., non- linear link functions and interaction terms), so the evaluation is not limited to purely linear relationships.
2.3 EVALUATION METRICS
To evaluate the reliability of LLMs for causal inference, we developed detailed grading rubrics for each causal pitfall, informed by guidelines from Sterne et al. (2016); Vandenbrouckel al. (2007). Each benchmark challenge includes multiple questions, each assigned points based on how effectively the model addresses the specific pitfall (see Appendix for detailed rubric). The total score for challenge is the sum of points obtained across these questions, and max_score is the maximum achievable score. To enable fair comparisons across challenges, we compute a normalized score:
score
Normalized Score (%) max_score
X 100%.
We evaluate LLM responses automatically using independent GPT-4o model (Achiam al. 2023) to minimize potential biases. To validate the accuracy of this automated evaluation, we additionally engaged three statisticians to manually grade 150 randomly selected responses. We
5

---

Published as a conference paper at ICLR 2026
measure consistency between automated and human scores using the gap metric:
1
150
SCOTE score score
Gap
human
E [0, 1},
150
smax,
where Smax,i is the maximum score of corresponding challenge, and scoreTim: score N+ scores from automated and human evaluations, respectively. Here, the gap metric ranges from to 1, where 0 indicates perfect agreement.
(8)
Finally, to provide a summary metric, we define causal reliability as the average normalized score across all benchmark challenges. This measure captures the overall trustworthiness and reliability of LLMs in statistical causal inference tasks.
3 ILLUSTRATIVE PITFALLS IN CAUSAL REASONING
When evaluating causal inference with LLMs, surface-level answers can create an illusion of compe- tence. Statistical causal inference requires grounding conclusions in evidence, checking assumptions, and ruling out alternatives. LLMs, however, may produce confident but flawed outputs that rely on irrelevant cues or statistical artifacts, giving a false sense robustness.
We illustrate this problem with two failure cases. The first shows how models can base conclusions on superficial semantic cues instead of data, while the second shows how they may misinterpret random variation causal structure.
Branding Bias: Adversarial Sensitivity to Branding and Semantic Manipulation. As an illustrative example, examined whether LLMs be misled by superficial cues when drawing causal conclusions. We constructed a synthetic scenario in which beverage consumption affected health outcomes. In the setting, other factors, lifestyle and health awareness, acted confounders, but the brand label itself had no causal role (Fig.
Health Awareness
Lifestyle
Consumption
Heallh
Figure 3: Causal DAG illustrating how beverage consumption, health awareness, and lifestyle affect health outcomes. The beverage's brand name ("HealthPlus" "UltraSugar") does not causally influence outcomes.
Despite identical data, simply changing the brand name from healthy-sounding label (HealthPlus) to carmful-sounding one (Ultrauugar) induced changes in LLMs' conclusions (Table 3). GPT-4o and Gemini-2. (aiiii--.--ffasss 2.0-flash, for example, made a conclusion aligning purely with the brand semanties. rather than the provided data.
The branding bias example shows that LLMs may rely on superficial semantic cues, attributing causal effects to labels even when the underlying data provide evidence to the contrary.
Table 3: Branding bias evaluation. Each row represents combination of the beverage's label ("HealthPlus" or "UltraSugar") and the true effect the given data (beneficial or harmful), checkmark (v) indicates beneficial conclusion and cross (x) ) harmful conclusion; correct inferences are those that match the true effect.
Brand Label
True Effect (Data) GPT-4o Gemini-2.0-flash
Claude-3.5-sonnet
HealthPlus
UltraSugar
HealthPlus
x
V
UltraSugar
x
X
X

---

Published as a conference paper at ICLR 2026
Spurious Causal Inference from Random Patterns. As second illustrative example, we show how LLMs can mistake random variation in real-world data for genuine causal structure. Specifically, evaluated LLMs real data from PNAS study (Van der Lee & Ellemers 2013) that analyzed funding success rates across academic disciplines in the Netherlands. When asked if the data reveal gender bias favoring male applicants, all of the tested L.MS drew incorrect conclusions, either attributing the differences in raw percentages directly to gender or considering Simpson's 's paradox.
However, a careful statistical analysis illustrated in the Appendix following Irizarry (2019) shows that women are not disproportionately applying to more competitive disciplines, so Simpson's 's paradox does not apply. Furthermore, computing log-odds ratios divided by their standard errors across disciplines and adjusting for multiple comparisons reveals no statistical evidence of gender bias within any department. None of the tested L.MMs recognized these key insights.
The spurious inference example shows that LLMs may mistake random variation in observational data for genuine causal structure, failing to apply the statistical checks needed to rule out noise.
4 RESULTS AND ANALYSIS
In this section, we present the key findings from our experiments, examining the causal inference capabilities of ten large language models. Our evaluation spans both closed source and open source LLMs, with the complete list of models provided in the Appendix. All models were evaluated using two protocols: direct prompting and code assisted prompting. The assessment covered six categories of causal inference pitfalls and fifteen challenges, each built from five datasets and five questions of varying difficulty.
Table 4: Causal reliability across causal pitfalls, comparing direct and code-assisted prompting. Values represent averages normalized scores, defined in equation (T). across five questions per pitfall category; higher scores indicate better performance.
LLM (Direct Prompting)
Conf
Interv
Counter
Med
Disc
Ext
Average
Gemma2-9b
14.00
30.72
8.00
15.89
9.49
4.05
13.69
Llama3,1-8 1-8b
18.46
34.78
10.86
29.33
7.71
6.05
17.86
Llama3.1 1-70b
17.60
35.92
10.29
33.67
11.31
9.92
19.78
Mistral-7h
17.31
29.83
5.71
19.17
8.40
6.18
14,43
Mixtral-8x22b
16.40
32.08
8.00
27.50
9.20
8.75
16.99
Claude-3.5-sonnet
18.74
47.60
12.00
40.50
18.63
19.84
26.22
Gemini-2.0- flash
20.06
37.56
13.43
46.67
13.48
14.93
24.36
Deepseek-chat
25.89
52.42
12.86
53.83
20.83
28.74
32.43
GPT-4.1
17.26
33.57
6.57
53.27
16.43
24.31
25.24
GPT-o4-mini
41.43
45.21
18.57
57.67
36.97
44.48
40.72
LLM (Code-Assisted Prompting)
Conf
Interv
Counter
Med
Disc
Ext
Average
Gemma2-9b
7.89
18.50
4.86
23.04
6.63
18.50
13.24
Llama3.1
11.93
15.99
7.67
18.06
6.23
17.29
12.86
Hamma3.1-70b
23.54
22.83
8.21
25.17
14.41
25.17
19.89
Mistral-7h
4,68
13.15
1,43
11.11
6.43
9.09
7.65
Mixtral-8x221
22.50
30.53
5.44
26.49
14.29
24.31
20.59
2laude-3.5-sonnet
32,91
47.33
11.71
41.73
16.12
26.47
29.38
Gemini-2.0-naa
37.20
42.96
14.29
42.17
16.19
37.98
31,80
Deepseek-chat
38.63
48.70
10.86
47.13
25.79
45.63
36.12
GPT-4.1
47.14
42.65
12.29
49.40
23.87
48.58
37.32
GPT-o4-mini
62.00
51.86
16.96
50.00
26.67
50.71
43.03
Conf: Confounding biases and spurious associations: Interv: Interventions and experimental reasoning: Counter: Counterfactual reasoning and hypotheticals; Med: Mediation and indirect causal effects; Disc: Causal discovery and structure learning; Ext: Causal generalization and external validity.
Overall Performance. Across models, GPT-o4-mini demonstrated the highest overall reliability, achieving average causal reliability of 40.72% under direct prompting and 43. 43.03% under code-
7

---

Published as a conference paper at ICLR 2026
assisted prompting (Table 4. Deepseek-chate although lower in aggregate (average 32.43% under direct prompting and 36. 12% under code- assistss prompting), obtained the strongest performance in interventions and experimental reasoning (52.42% under direct prompting and 48. 70% under code-assisted prompting). Taken together, these results indicate that optimized mid-scale models can, in certain contexts, outperform larger frontier systems causal reasoning tasks.
Benefits of Code-Assisted Prompting. Although code-assisted prompting improved several strong models, its benefits were universal. For example, GPT-o4-mini and Gemini-2.0-flas gained from code execution, increasing their averages from 40.72% to 43.03% and 24.36% to 31.80 31.80% respectively. GPT-4.1 also improved significantly (25.24% 37 37.32%). Under code-assisted prompting, models first convert raw data into summary statistics through generated code, then reason over those statistics. This separates low-level data parsing from causal reasoning, which partly explains why stronger models benefit more: they produce correct analysis code and reason on clean numerical outputs rather than raw tables. In contrast, some small open-source LLMs showed little benefit: Llama3. 1-8B decreased from 17.86% to 12.86%, Gemma2-9B remained unchanged (13.69% to 13. 13.24%). and Mistral-7B even dropped from 14.43% to 7.65%. These results indicate that computational results increase the strengths of adready-capabll LLMs, but does not provide uniform advantage across the LLMs. Models with high code-error rates (c.g., Mistral-7B, Llama-8B) have worse scores under
prompting. Allowing debugging attempt can bring their performance back to direct-prompting levels. We discuss this further in Appendix
Impact of Difficulty Levels. Causal reliability consistently decreases the question becomes harder. As shown in Table [] all models performed best on very easy and easy questions, with average scores declining as task difficulty increased, where the question contains fewer hints (see Table 2 For example, GPT-o4-mini achieved 60. 90.72% under direct prompting and 56.73% under code-assisted prompting on very easy items, but dropped 17.75% and 32. 32.84%, respectively, very hard questions. Deepseek-chal and Gemini-2. 2.0-faah showed similar trends, maintaining moderate performance on medium and hard levels but falling quickly on the very hard questions. These results indicate that code assistance is particularly beneficial for challenging tasks, helping stronger models recover some performance higher difficulty levels.
Table 5: Causal reliability by difficulty levels of questions, comparing direct and code-assisted prompting. Values represent averages of normalized score, defined equation challenges; higher scores indicate better performance.
across 15
LLM (Direct Prompting)
Very Easy
Easy
Medium
Hard
Very Hard
Gemma2-9h
20.50
15.64
14.22
6.72
5.46
Llama3. 1-8b
27.99
20.85
19.90
10.71
5.82
Llama3.1 1-70b
31.04
27.38
22.04
11.29
5.23
Mistral-7b
20.04
18.88
18.02
7.49
3.80
Mixtral-xx22b
27.80
21.20
20.31
8.41
5.23
Claude-3.5-sonnee
40.27
36.63
29.69
13.42
8.30
Gemini-2.0-ie
37.83
33.59
27.59
15.72
8.22
Deepseck-chat
46.37
38.92
37.92
28.79
16,36
GPT-4.11
38.01
33.94
33.59
17.31
7.67
GPT-o4-mini
60.72
54.05
48.13
30.72
17.75
LLM (Code-Assisted Prompting)
Very Easy
Easy
Medium
Hard
Very Hard
Gemma2-96
19.09
18.35
16.43
10.56
7.94
Llama3.1 1-8b
26.04
14.41
12.77
8.84
7.90
Llama3.1-700 1-70b
32.21
27.88
24.77
14.08
7.58
Mistral-7b
9.44
12.03
9.59
6.63
2.73
Mixtral-8x22b
33.14
29.77
25.64
14.59
7.77
Claude-3.5-sonnee
39.70
35.06
33.55
20.92
18.34
Gemini-22 2.0.ii-.-fffaar... 0-flssh
46.01
39,83
32.17
28.00
21.67
Deepseek-chie chat
55.56
48.23
43.69
28.46
16.50
GPT-4.1
57.80
46.49
46.48
24.33
19.65
GPT-o4-mini
56.73
49.66
49.37
34.31
32,84
8

---

Published as a conference paper at ICLR 2026
Persistent Reliability Gaps. Despite variation across models and prompting strategies, significant reliability gaps remain in all settings. Among the tested LLMs, GPT-o4-mini achieved the highest causal reliability score 43.03% average under rode-assisted prompting, while most other models remained well below this level. Performance on difficult causal reasoning challenges was especially limited, with very hard questions rarely exceeding 30% even for the best I.LMs. Taken together, the results suggest that existing LLMs, when used without finetuning or specialized architectures, remain unreliable apply high-stakes causal inference.
Human-LLM Grading Alignment. To validate the fidelity of our automated GPT-4o scoring against expert judgments, we conducted human validation study on a stratified sample of 150 model responses, with equal representation across the six pitfall categories and five difficulty levels (see Appendix). Three PhD students statistics independently graded each response using our detailed rubries. We then compared the resulting human scores with those produced by GPT-4o via the gap metric, which yielded a mean value of 0.11. This close agreement confirms that our GPT-4o evaluator reliably mirrors expert assessments, justifying its use for large-scale, reproducible evaluation of LLM performance in causal inference without the need for extensive human oversight.
Code-Assisted Execution Errors. As shown in Figure 4 code-execution failures peak in the "mediation effects" and "interventions and experimental reasoning" categories, where implementing correct stratification transportability routines is most demanding. Interestingly, "very easy" questions produce the highest failure rates, whereas "very hard" questions yield lower rates.
b
Claude-3 Sauude-3S-eormee S-eornet
Confounding
Very easy
Deepseek chat
Interventions
GPT-o4-miri
External
Gemini
80
Very banry
Liama3: 1 1-70b
40
Counterfactual
Causal Discovery
Medium
Hard
Mediation Effect
Figure 4: Code execution failure rates (%) in code-assisted prompting protocol across causal inference challenges and question difficulty. Failure rate is defined as the percentage of code- generation attempts that either raise execution errors produce invalid analytical outputs, computed only for the code-assisted prompting protocol. (a) Average failure rate for each of the six causal- inference pitfall categories. (b) Average failure rate by question difficulty level, increasing from very easy through very hard tasks.
5 CONCLUSION
We introduced CausalPitfalls, a benchmark designed to rigorously evaluate the reliability of LLMs in performing statistical causal inference. Unlike existing benchmarks that focus primarily on accuracy. our benchmark reveals how LLMs can produce confident yet flawed conclusions by falling into classical statistical pitfalls. Our results indicate substantial gaps in reliability across all models and settings. Even state-of- tate-off-hhe-rt models exhibit systematic vulnerabilities to confounding, semantic bias, and difficulties generalizing causal knowledge across contexts. These findings highlight an urgent need for targeted interventions to improve LLMs trustworthiness in scientific and policy domains. Future directions include expanding the benchmark to cover more nuanced forms of causal reasoning. such instrumental variable analysis, latent confounding, and policy evaluation. Additionally, envision CausalPitfalls as a platform to guide training or fine-tuning strategies that aim to instill causal robustness in LLMss.

---

Published as a conference paper at ICLR 2026
More detailed descriptions of our benchmark pitfall categories, challenges, and implementation details are included the Appendix inside the supplementary material.
ACKNOWLEDGMENT
The work was supported in part by the National Science Foundation CAREER Program under grant number 2338506.
ETHICS STATEMENT
All authors have read and adhered to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have made every effort ensure the reproducibility of our results. Detailed descriptions of datasets, preprocessing steps, and experimental settings are provided in the main text and appendix. The CausalPitfalls benchmark, including datasets, questions, grading rubrics, and evaluation scripts, is publicly available at //tttbb/ comm//occududuu//csssstsaat
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 20377:23003.08774 2023.
Kwun Chuen Gary Chan, Sheung Chi Phillip Yam, and Zheng Zhang. Globally efficient non- parametric inference of average treatment effects by empirical balancing calibration weighting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(3):673-700, 2016.
Li Chen, Chunlin Li, Xiaotong Shen, and Wei Pan. Discovery and inference of causal network with hidden confounding. Journal of the American Statistical Association, 119(548):57222584, 2024.
Li Chen, Xiaotong Shen, and Wei Pan. Enhancing causal effect estimation with diffusion--eneratee data. arXiv preprint arXiv:2504.03633 2025.
Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul G Krishnan, and Chris Maddison. End-to-end causal effect estimation from unstructured natural language data. Advances in Neural Information Processing Systems, 2024.
Charles R Doss, Guangwei Weng, Lan Wang, Ira Moscovice, and Tongtan Chantarat, A nonparametric doubly robust test for continuous treatment effect. The Annals of Statistics, 5244):1592-1615, 2024.
Sander Greenland and James Robins. Identifiability, exchangeability. and epidemiological con- founding. International journal of epidemiology. 15(3):113449, 1986.
Zijian Guo, Domagoj Čevid, and Peter Bühlmann. Doubly debiased lasso: High-dimension
inference under hidden confounding. Annals of statistics, 50(3):1320, 2022.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences.
Cambridge university press. 2015.
Rafael A Irizarry. Introduction to data science: Data analysis and prediction algorithms with R.
Chapman and Hal//CRC, 2019.
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez. Adauto, Max Kleiman-Weines Mrinmaya Sachan, al. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems, 36:31038- 31065, 2023.
Hyunseung Kang, Anru Zhang, T Tony Cai, and Dylan S Small. Instrumental variables estimation with some invalid instruments and its application to mendelian randomization. Journal of the American statistical Association, 11 111(513):322-14, 2016.
10

---

Published as a conference paper at ICLR 2026
Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. Transactions on Machine Learning Research, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Chunlin Li, Xiaotong Shen, and Wei Pan. Nonlinear causal discovery with confounders. Journal of the American Statisticul Association, 105544)) 2024.
Zhexiao Lin, Peng Ding, and Fang Han. Estimation based on nearest neighbor matching: from
density ratio to average treatment effect. Econometrica, 91(6):2187-2217, 2023.
Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are Ilms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. Findings of the Association for Computational Linguistics ACL, 2024.
Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin. Causal discovery with language models as imperfect experts. In International Conference on Machine Learning Workshop. 2023.
David MacKinnon. Introduction to statistical mediation analysis. Routledge, 2012.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl et Causality: Models, reasoning and inference. Econometric Theory, 19, 2003.
Jonas Peters and Peter Bühlmann. Identifiability of gaussian structural equation models with equal error variances. Biometrika, 101(1):219- 2288 2014.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. Counterfactual story reasoning and generation. In Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP), pp. 5043-5053, 2019.
Paul R Rosenbaum. Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika, 74(1):13-26, 1987.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series (Methodological), 1 13(2):238-241, 1951.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis. Stephen Pfohl, et Large language models encode 2023.
clinical knowledge. Nature, 620(7972):
Jerzy Splawa-Neyman, Dorota M Dabrowska, and Terrence P Speed. On the application of probability theory to agricultural experiments, essay principles. section 9. Statistical Science, pp. 465-472, 1990.
Jonathan AC Sterne, Miguel A Hernán, Barnaby C Reeves, Jelena Savović, Nancy D Berkman, Meera Viswanathan, David Henry, Douglas G Altman, Mohammed T Ansari, Isabelle Boutron, al. Robins-i: tool for assessing risk of bias in non non-randomised studies of interventions. bmj, 355, 2016.
Romy Van der Lee and Naomi Ellemers. Gender contributes to personal research funding success in the netherlands. Proceedings of the National Academy of Sciences, 1 12((4)): 1231111232222 2015.
11

---

Published as a conference paper at ICLR 2026
Jan P Vandenbrouckel, Erik von Elm, Douglas G Altman, Peter C Gotzsche, Cynthia D Mulrow, Stuart J Pocock, Charles Poole, James Schlesselman, and Matthias Egger. Strengthening the reporting of observational studies in epidemiology (strobe): explanation and elaboration. PLoS 2007.
Medicine,
Merryn Voysey, Sue Ann Costa Clemens, Shabir A Madhi, Lily Y Weckx, Pedro M Folegatti, Parvinder K Aley, Brian Angus, Vicky L Baillie, Shaun L Barnabas, Qasim E Bhorat, et al. Safety and efficacy of the chadox ncov-19 vaccine (azd1222) against sars-cov-2: an interim analysis of four randomised controlled trials in brazil, south africa, and the uk. The Lancet, 397(10269):
99-111, 2021.
Zeyu Wang. Causalbench: A comprchensive benchmark for evaluating causal reasoning capabilities of large language models. In Advances in Neural Information Processing Systems Workshop on Mathematical Reasoning and Al (MATH-AI), 2024.
Song Wei, Ronald Moore, Hanyu Zhang, Yao Xie, and Rishikesan Kamaleswaran. Transfer causal learning: Causal effect estimation with knowledge transfer. In ICML 3rd Workshop on Interpretable Machine Learning Healthcare (IMLH), 2023.
Moritz Willig, Matej Zežević, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality? In UAI Workshop. 2022.
Tianzhong Yang, Jingho Niu, Han Chen, and Peng Wei. Estimation of total mediation effect for high-dimensional omics mediators. BMC bioinformatics, 22:1-17, 2021.
Matej Zečevíć, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting, Causal parrots: Large language models may talk causality but are not causal. Transactions on Machine Learning Research, 2023.
Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. Causalbench: A comprehensive benchmark for causal learning capability of llms. arXiv preprint arX1:2404.00339, 2024.
12


Reviews:
[{"review_id": "fshEOBE4VF", "reviewer": "Reviewer_fhn1", "summary": "The paper introduces CausalPitfalls, a benchmark designed to evaluate large language models (LLMs) on statistical causal inference tasks while specifically testing their susceptibility to classical causal reasoning errors. The benchmark comprises 75 datasets and 15 challenges across six categories, including confounding, interventions, counterfactual reasoning, mediation, causal discovery, and external validity. It assesses two protocols, direct prompting and code-assisted prompting, and proposes a quantitative metric called causal reliability to standardize comparison across models. Results show that all evaluated LLMs exhibit systematic reliability gaps, particularly on tasks involving mediation and generalization, with moderate improvements observed when code execution is permitted.", "questions": "Refer to the weaknesses section above, additionally, How prevalent are the main pitfalls highlighted in the benchmark within real-world datasets, and to what extent do they typically affect practical causal analyses?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes a timely and significant contribution by systematically targeting the reliability rather than accuracy of LLMs in causal inference, an aspect underexplored in previous benchmarks such as CausalBench and CLADDER. The structured coverage of classical statistical pitfalls like Simpson’s and Berkson’s paradoxes, mediation confounding, and counterfactual reasoning reflects a solid understanding of the causal inference literature and provides a rigorous stress test for LLMs. The benchmark’s dual evaluation protocol is well-motivated and demonstrates a clear methodological insight: executable statistical reasoning through code-assisted prompting substantially mitigates causal errors that stem from linguistic biases. The proposed causal reliability metric is also a valuable addition that enables consistent quantitative comparisons across diverse causal reasoning challenges.", "weaknesses": "Although overall a good contribution, The benchmark lacks a bit in scope and interpretation. Although the benchmark convincingly demonstrates single LLM limitations, the current coverage only includes very basic prompting stratagies. Current causal inference methods don't use LLMs directly anyways, benchmarking more architectures like finetuned models for causal inference, or better prompting stratagies like React would be more helpful. Moreover, while the paper positions code-assisted prompting as a key improvement, it does not sufficiently analyze why certain models benefit more than others or what kinds of causal reasoning errors persist even after code execution. Without this diagnostic breakdown, the benchmark feels more descriptive than explanatory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egbXaM8ytg", "reviewer": "Reviewer_Z3sS", "summary": "The paper introduces CausalPitfalls, a benchmark designed to evaluate the capability of LLMs in recognizing and thus overcoming common causal inference pitfalls. The authors claim this will improve LLMs' capability of handling statistical causal inference. They show this by getting LLM generated text and code evaluated by human experts.", "questions": "1. Do the authors fear that benchmarks for LLMs could be a losing cause? For example, is it possible that this benchmark becomes \"training data\" - in other words can LLMs be gamed to do well on benchmarks but actually still be bad at causal inference?\n2. Having said that, do the authors have any plans to extend their work to include more types of pitfalls in the future?\n3. What if you tell LLMs that they were wrong and they should try again. Do you expect these second chance answers to improve?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Advancing causal inference research is a welcome topic in the field of LLMs\n2. The structure of the benchmark and the design aspect of it seems well-thought out and illustrated\n3. Having human evaluation for LLM generated content is key", "weaknesses": "1. The datasets are at the end of the day still synthetic and not drawn from real observational/experimental studies\n2. It is not clear how well this will generalize to real world scenarios\n3. It would be helpful if the authors expanded a bit more on what is an acceptable success rate at these tasks. How well do humans do?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6p9t5VIjHo", "reviewer": "Reviewer_iHLz", "summary": "This paper introduces CausalPitfalls, a benchmark to test whether LLMs can do statistical causal inference without falling into classic traps like Simpson’s paradox, selection/collider bias, et.c. It shows that current models often give confident but statistically wrong causal conclusions that are highly sensitive to phrasing, and that letting them run code helps but doesn’t solve the problem overall.\n\nContributions:\n\n* 6 causal-pitfall families, 15 challenges, 75 questions + 75 SCM-based datasets, each with 5 difficulty levels, targeting: confounding/spurious association, interventions vs observation, counterfactuals, mediation, causal discovery, and external validity/transport.\n* (1) Direct prompting to measure intrinsic causal reasoning from raw data; (2) Code-assisted prompting where the model generates and runs statistical code before answering. This separates questions of “can the model spot the pitfall?” and “can the model reason its way out of it?”\n* A rubric-based, normalized metric for causal reliability validated against human statisticians to compare models on trustworthiness, not just accuracy.\n* Illustrative failure cases including models flipping answers based on branding (“HealthPlus” vs “UltraSugar”) on identical data; falsely detecting gender bias / Simpson’s paradox with age stratification in synthetic medical data, falsely treating Berkson’s/collider bias in hospital data as a real effect, over-attributing effects in mediator–outcome confounding setups, fail to detect non-identifiability/Markov equivalence in causal discovery.\n* Empirically findings: the best model reaches only ~40–43% causal reliability, mediation and external validity are especially difficult", "questions": "Could you have converted the raw data into sufficient statistics sufficient to resolve the causal inference problems (e.g., counts, proportions, rates, descriptions that imply monotonicity in case of probabilities of causation, etc.), then construct prompts based on these sufficient stats? One could do this without giving cues to the LLM about the right way to solve a problem (e.g., providing additional summary statistics that not needed to correctly do the inference). This would have better evaluated a models' ability to \"talk\" its way through the inference problem and less dependent on its ability to parse raw data.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In my view, the main strength of the paper is demonstrating cases where the LLM fails to resolve common pitfalls in causal inference across a broad set of causal inference problems, using generative causal graphical model ground truth. The paper then extends this analysis into a formal benchmark with a useful metric focused on evaluating LLMs for how reliable they are in answering these types of questions.", "weaknesses": "The benchmark relies heavily of the LLMs ability to parse raw data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7QFSzJjoF", "reviewer": "Reviewer_gZXP", "summary": "LLMs have commonly been assessed for causal reasoning and inference tasks; however typically only a subset of these tasks - or plain commonsense reasoning - is evaluated in any benchmark. This paper proposes an exhaustive summary of pitfalls in causal inference to be verified, and proposes an automated pipeline with a LLM to rate answers - this LLM is shown to be in agreement with a large pool of human graders.", "questions": "- Can you address/answer the above weaknesses?\n\n- To double check that the discrepancy of 0.11 between gpt-4o and human scores is low, can you also compute it in these cases: a) scores are all zero, b) scores are all maximal, c) scores are uniformly sampled between zero and the maximal possible value (averaged over several seeds).\n\n- Did/Can you test the evaluation pipeline on statistical or causal libraries (eg DoWhy), or on humans?\n\nNote: I am ready to increase my score up to Accept levels if my concerns are addressed.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Original approach, and first paper I see that extensively reviews all causal inference skills in LLMs. The list of tasks looks exhaustive, and the taxonomy by difficulty looks relevant and consistent with results.\n\n- The automatic grading with gpt-4o is a welcome innovation and seems validated by external graders", "weaknesses": "- It is not detailed where exactly the previous evaluations in the literature are missing wrt the challenges in Table 1. A table detailing this would be welcome.\n\n- The process to generate DAGs in l.245-253 is a bit unclear. No precise equations are given. In the sentences, \"For every statistical pitfall, we select causal graphs that capture its unique complexities and characteristics. Each challenge is accompanied by five distinct datasets, each containing over 500 samples for comprehensive evaluation.\", it is unclear whether five daatsets are generated by graph, or five graphs are chosen by task? Also, the sentence \"For every statistical pitfall, we select causal graphs that capture its unique complexities and characteristics. Each challenge is accompanied by five distinct datasets, each containing over 500 samples for comprehensive evaluation.\" suggests that only linear SCMs are evaluated, which would be restrictive.\n\n- It is unclear how scores are computed exactlty. I understand that points are assigned to questions, but what's the exact assignment? And why assign more than 1 point to each question (scores could then be normalized on a 0-100% scale)? It is also unclear how gpt-4o is prompted to evaluate models, or is evaluation is done exactly with it. Does it choose points itself? Or is it Correct/Incorrect, with Correct giving all points and Incorrect giving none?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)