Title: Counterfactual reasoning: an analysis of in-context emergence
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-context learning; transformers; reasoning; causality
Abstract: Large-scale neural language models exhibit remarkable performance in in-context learning: the ability to learn and reason about the input context on the fly. This work studies in-context counterfactual reasoning in language models, that is, the ability to predict consequences of a hypothetical scenario. We focus on a well-defined, synthetic linear regression task that requires noise abduction. Accurate prediction is based on (1) inferring an unobserved latent concept and (2) copying contextual noise from factual observations. We show that language models are capable of counterfactual reasoning. Further, we enhance existing identifiability results and reduce counterfactual reasoning for a broad class of functions to a transformation on in-context observations. In Transformers, we find that self-attention, model depth and pre-training data diversity drive performance. Moreover, we provide mechanistic evidence that the latent concept is linearly represented in the residual stream and we introduce designated *noise abduction heads* central to performing counterfactual reasoning. Lastly, our findings extend to counterfactual reasoning under SDE dynamics and reflect that Transformers can perform noise abduction on sequential data, providing preliminary evidence on the potential for counterfactual story generation. Our code is available under *https://github.com/mrtzmllr/iccr*.


Peer reviews:
comments: SUMMARY:
The authors present a study of counterfactual reasoning by in-context learning of large language models. They provide synthetic experiments that show, based on MSE performance, that language models are capable of performing the task.

STRENGTHS AND WEAKNESSES:
Strengths:
- Rigorous theoretical underpinning of counterfactual reasoning
- Controllable study on synthetic data

Weaknesses: 
- Due to the synthetic setup, it is very hard to understand what the experimental hypothesis is, and how the described experiment related to natural language generation. The example given in appendix G does not help since it does not represent "what if" questions nor does it explain the task of noise abduction in natural language.
- What is Figure 2 supposed to show: 1) Attention is required for optimal performance (GPT-2)? 2) Attention is not required since GRUs are a close second? 
- The evaluation of the "reasoning" capabilities is based on the MSE results, not on an investigation of the reasoning process itself. It is well-known that the correctness of  the final conclusion does not necessarily reflect the validity of a model's reasoning process (see https://arxiv.org/abs/2404.01869, https://arxiv.org/abs/2305.18654)

QUESTIONS:
1. How does the example given in appendix G represent "what if" questions or explain the task of noise abduction in natural language? Please provide an instantiation of the variables used in the synthetic setup on a concrete example.
2. What is Figure 2 supposed to show: 1) Attention is required for optimal performance (GPT-2)? 2) Attention is not required since GRUs are a close second? 
3. How would you evaluate the  "reasoning" capabilities of the model itself, not just the results?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper intends to answer the question whether LLMs can take a sequence of factual examples, imagine a “what-if” change, and generate the output without gradient updates. It examines the problem with a synthetic linear regression task requiring noise abduction.  A large transformer-based model is trained to fit the data from a mixture of sequences with different distributions but the same causal structure. Other models with different architectures, such as GRU and LSTM are trained on the same data for comparison. The paper concludes that large transformer models have the capability of in-context counterfactual reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nS1: Examining the in-context counterfactual output generation capability of large transformers is an novel idea. This is an interesting direction considering the importance of counterfactual reasoning in decision making. \n\nS2: A controlled study with linear regression is helpful to understand the transformer behaviour on counterfactual output generation. \n\nW1: The training objective at line 116 minimises the counterfactual loss without considering the factual loss. Does the training have negative impact on factual output prediction? With such a single objective optimisation, the formulation is not much different to approaches studying emergence phenomenon of in-context learning itself, such as the following:\n\nGarg, S., Tsipras, D., Liang, P.S. and Valiant, G., 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35, pp.30583-30598.\n\nIn a sense, the paper reaches a similar conclusion as the above paper due to that the training overfits counterfactual examples. I might have missed something here.\n\nW2: In addition to transformers, the evaluation also shows results from other neural network architecture, e.g., GRU and LSTM also show counterfactual generation capability. I am wondering how the claim that counterfactual reasoning emerges in self-attention is justified. \n\nW3: Line 41. it is better to describe \\mu_y clearly here. In line 48, \\mu_y is referred as context, but in Fig.1, it is referred as noise. Can you please also explain why the \\mu_y is indexed by y not x?\n\nQUESTIONS:\n1. The paper assumes that a sequence of random variables is exchangeable. However, the order of words in an utterance matters in language model training. What is the implication of this assumption on LLM-generated counterfactuals?\n\n2. Line 109-110 are not clear to me. How \\theta is defined and how does it parameterise $U_X$ and $U_Y$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies in-context counterfactual reasoning using sequential models. It first introduces an in-context learning setup for counterfactual inference, aiming to answer queries of the form $P(y'_{x'} \\mid x, y)$ given a causal graph $X \\rightarrow Y$. The paper shows that this counterfactual query can be reduced to transformations on in-context observations. It then empirically demonstrates that transformers are capable of inferring such counterfactual queries in both linear and exchangeable sequence settings. Extensive empirical studies are also conducted to identify which components of training transformers or sequential models are critical for successful counterfactual inference. Finally, the experiments are extended to nonlinear and more dynamic sequential settings.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nS1. The paper is well-structured and clearly written. The problem statement and conclusions are presented clearly. \n\nS2. The problem setting is well formalized and original, which may provide valuable insights for future work on in-context learning and counterfactual reasoning. \n\nS3. The main conclusion, that language models can perform in-context counterfactual reasoning, is interesting and well supported by the experiments. This finding could guide future research in both causal inference and language modeling.\n\n\n# Weaknesses\n\n**W1. The example about John in the introduction (lines 30–35) is not well aligned with the actual problem setting of in-context counterfactual reasoning.** In the formal problem setup, the pairs $(x_i, y_i)$ are sampled from a family of SCMs sharing the same causal graph. However, the example, \"John got the flu and is recovering in bed. Had he not attended the get-together, he would now visit his grandmother for cake.\", is just a counterfactual statement and does not reflect the in-context formulation. It would be more helpful if the example directly illustrated the actual problem setting.\n\n**W2. The theoretical insight (Lemma 1) it provides does not seem directly relevant to the core empirical claim of the paper.** The paper states that it \"provides insights into how in-context counterfactual reasoning is equivalent to transformations on in-context observations (Lemma 1),\" but this does not clearly support the conclusion that transformers are capable of performing counterfactual reasoning.\n\nIn particular, Lemma 1 suggests that $y^{\\mathrm{CF}}$ can be expressed as a transformation of $x,y, x^{\\mathrm{CF}}$ (Equation 2). However, this transformation may not be unique. If the true transformation underlying the data does not match the learned model's transformation, then the counterfactual prediction could be entirely incorrect. This corresponds to the identification issue mentioned later in the paper. Similar identifiability issues for invertible SCMs are discussed in prior work [1]. I believe combining such identification results with Lemma 1 would significantly strengthen the theoretical support for the paper’s main claim.\n\n[1] Nasr-Esfahany, Arash, Mohammad Alizadeh, and Devavrat Shah. \"Counterfactual identifiability of bijective causal models.\" International Conference on Machine Learning. PMLR, 2023.\n\nPS: I am happy to raise the score if the concerns are solved but may lower score if the concerns are not solved.\n\nQUESTIONS:\n**Q1.**\n Is there a more closely aligned example that directly illustrates the in-context counterfactual reasoning problem as formalized in the paper? (Please see the discussion in **W1**.)\n\n**Q2.**\n Could the authors briefly connect the identifiability results in invertible structural causal models to Lemma 1? This connection seems highly relevant and, in my opinion, could strengthen the theoretical foundation of the paper.\n (See **W2** for further discussion.)\n\nLIMITATIONS:\n**L1.** The paper suggests that data diversity in pre-training, self-attention, and model depth are key for the performance of ctf reasoning with experiments. However, it lacks theoretical explanations or intuitive insights to support these observations. Including such a discussion would strengthen the paper and help guide future research.\n\n**L2.** Additionally, as acknowledged by the authors, the experiments are conducted on small-scale synthetic datasets, which may limit the generalizability of the results. There could be a gap between these controlled settings and real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes LLMs’ capabilities to perform in-context counterfactual reasoning. To this end, the authors prompt a model with what-if questions based on a numeric synthetic dataset using a function-learning probe. The authors focus on a linear SCM (Structural Causal Model) setting, working in the latent space, where they add noise to various stages of token representations.\nIn a GPT-2–sized Transformer, they show that counterfactual prediction copies the noise from the factual pair and applies a learned transformation. They empirically demonstrate that it uses fewer in-context examples than other architectures (e.g., LSTMs). The authors further show that performance degrades when self-attention is removed (using only the MLP), and that the mixture and number of SCMs influence pre-training. They argue that their findings also generalize to data mimicking counterfactual story generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\nThe authors focus on a very timely topic: counterfactual reasoning. The paper is generally well-structured and well-written.\n\nThe task and synthetic dataset are well designed for the research question and thoroughly developed: the task is methodologically well-formalized, and the experiments follow this design closely, answering relevant questions and presenting meaningful ablation studies. In this way, the authors present a sound theoretical framework to test SCMs in an in-context counterfactual reasoning setting.\n\nThrough this approach, the authors provide several insights—such as the importance of self-attention—which constitute a meaningful contribution to the growing body of work on reasoning in LLMs.\n\nWeaknesses:\n\nSince the approach is only tested on a synthetic dataset, its general applicability to real-world tasks remains uncertain. It would have been valuable to include an evaluation on at least one real-world dataset.\n\nFurthermore, the method is evaluated only on a comparatively small model.\n\nThe approach primarily builds on existing function-learning probing techniques, applying them to in-context counterfactual reasoning. Moreover, the task of answering \"what-if\" questions has been addressed in prior work. While the paper tackles a meaningful problem in a well-controlled setting, I find that the methodological novelty is rather modest.\n\nQUESTIONS:\nHave you conducted any preliminary experiments on real-world tasks to assess the generalizability of your approach?\n\nBeyond analyzing attention scores, have you explored other interpretability techniques, such as methods from mechanistic interpretability (e.g., probing the residual stream)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}