Title: Causal Discovery and Inference through Next-Token Prediction
Venue: NeurIPS 2025
Accepted: yes
Keywords: causal inference

causal discovery

transformers

large language models

language modeling

mechanistic interpretability

structural causal models

neural probing

decoding

causal intervention

next-token prediction

counterfactual reasoning

causal representation learning

Pearl's causal hierarchy

ladder of causation

emergent representations

world models
Abstract: Deep neural networks have been criticized as fundamentally *statistical* systems that fail to capture causal structure and perform causal reasoning. Here we demonstrate that a GPT-style transformer trained for next-token prediction can simultaneously discover instances of linear Gaussian structural causal models (SCMs) and learn to answer counterfactual queries about those SCMs. First, we show that the network generalizes to counterfactual queries about SCMs for which it has seen interventional data but not any examples of counterfactual inference. The network must, thus, have successfully composed discovered causal structures with a learned counterfactual inference algorithm. Second, we decode the implicit “mental” SCM from the network's residual stream activations and manipulate it using gradient descent with predictable effects on the network's output. Our results suggest that statistical prediction may be sufficient to drive the emergence of internal causal models and causal inference capacities in deep neural networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors start with Pearl’s Ladder of Causation, arguing that this framework suggests DNNs trained in a “statistical mode” — that is, to predict passive observations — can only capture associations (L1) and are inherently limited from reasoning about actions, interventions, and explanations. They then show that a GPT-style transformer trained on next-token prediction can, in fact, simultaneously discover examples of linear Gaussian structural causal models and learn to answer counterfactual queries about them. This leads them to propose that neural networks trained with standard statistical prediction objectives on passively observed data may still be able to uncover and leverage causal models of the world.\n\nSTRENGTHS AND WEAKNESSES:\nThe claim that a “statistical mode”—that is, predicting passive observations—can only capture associations (L1) and is inherently limited from reasoning about actions, interventions, and explanations reflects a misunderstanding of the relationship between causality and statistics. It also represents a misinterpretation, or at best an outdated reading, of Pearl’s Ladder of Causation. In fact, many works have demonstrated how statistical models can be designed to encode and decode causal relationships.\n\nIt has also been widely observed that large language models (LLMs), trained solely on next-token prediction, exhibit a degree of causal reasoning ability. Most studies along this line are empirical, though there are a few theoretical works that explicitly link next-token prediction with causality.\n\nStrengths: The authors have clearly invested effort into this investigation.\n\nWeaknesses: However, the paper stems from an outdated perspective, explores a problem that has already been thoroughly examined, and offers observations that have been extensively discussed in the community over the past several years. I would recommend that the authors update their understanding of the current literature in this area.\n\nQUESTIONS:\nI would recommend that the authors update their understanding of the current literature in this area in order to make a meaningful contribution to the field.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether a transformer model, trained exclusively on a next-token prediction objective, can learn to perform causal discovery and inference. The method involves training a GPT-style model on a synthetic dataset generated from a constrained class of linear Gaussian structural causal models (SCMs).\n\nThe workflow and evaluation are structured as follows:\n\n* Data Generation: The authors generate a large, finite set of unique SCMs. From these SCMs, they create text strings in a simple, artificial language that fall into two categories: DATA strings, which describe the results of noisy interventions on an SCM, and INFERENCE strings, which provide the answers to counterfactual queries about an SCM.\n\n* Generalization Challenge: To test for true causal understanding rather than memorization, the model is trained on a dataset where a specific subset of SCMs (D_test) is only ever described using DATA strings. The core of the evaluation is to assess whether the model can still perform correct counterfactual inference for this D_test set, despite never having seen an INFERENCE string for any of its \n\n* Evaluation & Probing: The primary evaluation metric is the model's accuracy on counterfactual queries for the held-out D_test set. The authors supplement this behavioral test with mechanistic interpretability techniques. They probe the model’s internal activations to determine if a representation of the underlying SCM can be decoded. They further test the functional relevance of this representation by directly intervening on the activations via gradient descent and observing whether this manipulation produces predictable changes in the model's output for causal queries.\n\nThe authors report that the model successfully generalizes to the D_test set, achieving near-optimal performance on counterfactual inference tasks. Their probing analysis reveals that the SCM's structure can be decoded from the model's residual stream, and their interventional experiments demonstrate that manipulating this internal representation causally influences the model's reasoning in a predictable manner. The results suggest that a standard statistical training objective can be sufficient for a neural network to discover and learn to use a causal model of its data-generating process.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n * Addresses a Foundational Question: The paper addresses the question of whether causal reasoning can emerge from statistical training objectives. Its approach, which uses a controlled environment to study a small transformer, offers a method for investigating mechanisms that may be relevant to larger models.\n\n* Targeted Experimental Design: The \"generalization challenge\" is designed to distinguish causal learning from pattern memorization. By evaluating the model on counterfactual queries for SCMs presented only through interventional data, the experimental setup directly tests the model's ability to generalize causal rules.\n\n* Insight into Internal Representations: The analysis of the model's internal states is a notable strength. Probing experiments indicate that the model's latent space contains a decodable representation of the true SCM. Furthermore, interventional experiments confirm that the model actively uses this internal representation to generate its causal inferences, providing evidence that the learned structure is functional.\n\n* High-Accuracy Results: The reported results show that the model achieves high accuracy on the generalization task, approaching the optimal performance bound. This result is presented as evidence that a next-token prediction objective can be sufficient for inducing causal inference capabilities in this specific setting.\n\n## Weaknesses\n\n* Highly Constrained Experimental Setting: The paper's claims are derived from a very restrictive setting involving linear Gaussian SCMs with only four variables and ternary weights (-1, 0, 1). While this simplicity is necessary for the detailed analysis performed, it raises significant questions about scalability and generalizability. It remains unclear whether the same mechanisms would hold for SCMs with more variables, continuous causal strengths, or non-linear relationships.\n\n* Use of a Simplified Artificial Language: The model is trained on a simple, unambiguous artificial language where causal relationships are explicitly described. Natural language, in contrast, is noisy, ambiguous, and context-dependent. The degree to which these findings depend on the clean, structured nature of the input data is a major open question that limits the direct applicability of the conclusions to real-world LLMs.\n\n* Limited Discussion of Implications for LLMs: The discussion section is brief and does not sufficiently bridge the gap between the findings in this toy model and the behavior of large-scale language models. While the paper serves as an interesting “existence proof”, it offers little speculation on how these mechanisms might manifest, or be identified, in models trained on vast, unstructured web-text. A more thorough discussion of these implications would strengthen the paper's contribution.\n\nQUESTIONS:\n1.The study successfully demonstrates causal discovery in SCMs with four variables. How is the model's performance expected to scale as the number of variables (and thus the complexity of the causal graph) increases? Is there a theoretical or practical limit where the next-token prediction signal becomes insufficient?\n\n2.The findings rely on a simple, unambiguous artificial language. How robust do the authors expect these results to be if the language included more naturalistic complexities, such as ambiguity, paraphrasing, or implicit causal statements?\n\n3.The probing and intervention techniques are central to the paper's analysis. Do the authors believe these methods, particularly the gradient-based interventions, could be practically adapted to analyze causal representations in much larger, pre-trained foundation models?\n\n4.The ternary weight system makes faithfulness violations (e.g., pathways cancelling out) plausible. Could the authors comment on whether the model successfully learns the true SCM structure in such cases, or if it is misled by the resulting statistical independencies in the data?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates that next-token prediction alone can induce genuine causal reasoning: the authors construct a synthetic language that interleaves observational samples, interventional statements and counterfactual queries drawn from thousands of four-variable linear-Gaussian structural causal models (SCMs), train a 12-layer GPT-style transformer solely to predict the next token in these sequences, and find that on held-out SCMs it answers unseen counterfactual questions with near-optimal accuracy—evidence that it has inferred the latent graph rather than memorised responses. Linear and MLP probes decode each SCM edge weight from the network’s residual-stream activations, and gradient-based edits to those activations flip the model’s counterfactual predictions exactly as the modified SCM would, revealing a manipulable internal causal representation . Layer-wise intervention analysis shows this representation persists deep into the model even when later layers stop consulting it, warning that decodability need not imply functional use.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is well-written and easy to follow.\n* Uncovering structures from LMs is not a new research topic. There are even harder tasks that are linguistically driven, e.g., uncovering syntactic trees from synthetic languages generated using PCFGs. However, it is still quite interesting to see that simple interventions could uncover an SCM's weights.\n* The setup is simple but clean. However, whether it is extensible to more real-world data remains a question.\n\n**Weaknesses**\n\nI think the experimental setup is solid. My main concern resides in the high-level goal of this work and how it relates to some other interpretability research in the field.\n\n* **Missing related works in uncovering trees or structural models from LMs.** There are existing works that come from the more classic NLP interpretability camp (see \\[1] and its cited works) which are trying to (1) uncover tree structures (i.e., syntactic ones) from LMs; (2) or even induce trees in LMs. I think it would be worth discussing the main differences between this work and them. In particular, focusing on what additional insights can be gained by using SCM-induced datasets from this paper would be interesting. At a high level, both lines of work are trying to interpret very similar things. In \\[1], a PCFG model is used to generate synthetic data that are then used to train LMs, and interpretability methods are used to assess whether we can uncover syntactic tree structures (i.e., a PCFG graph with edge weights of 1) from the LM.\n\n* **Needs to be better motivated.** Although it is interesting to see some causal structures emerge in the LMs for this very specific synthetic dataset, I think it would be beneficial to say more about the motivation behind this research in a precise way. For instance, what is the additional gain from your setup compared to previous ones (including the line of work I mentioned above)? With your methods, are you finding additional interpretability signals that inform us how transformers work beyond what people already know (e.g., transformers generalize, transformers learn structures of the dataset generation process)?\n\n[1] Murty et al., *Grokking of Hierarchical Structure in Vanilla Transformers*, [https://arxiv.org/pdf/2305.18741](https://arxiv.org/pdf/2305.18741)\n\nQUESTIONS:\nSee my comments above.\n\nLIMITATIONS:\nSee my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores whether transformers trained for next-token prediction can discover and infer causal relationships within a constrained class of linear Gaussian structural causal models (SCMs). The authors demonstrate that the model generalizes to counterfactual queries for SCMs it only encountered through interventional data strings, suggesting it learns a causal inference engine. They also decode SCM weights from the model's residual stream activations and manipulate these representations to influence predictions. However, the approach lacks theoretical guarantees for consistency in causal discovery/inference, and its reliance on SCM indices limits generalizability to unseen SCMs. Additionally, the experiments are not benchmarked against established causal discovery algorithms like PC or FGES, raising questions about comparative performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths  \n\n1. **Innovative Approach**:  \n   - The paper presents a novel integration of causal inference with next-token prediction, demonstrating that transformers can implicitly learn and utilize causal structures from text-like data. \n   - The use of a constrained class of linear Gaussian SCMs provides a clean experimental setup, allowing the authors to rigorously test the model's ability to generalize beyond memorization.  \n\n2. **Probing and Interventions**:  \n   - The authors employ linear probing and multi-layer perceptron (MLP) probes to decode SCM weights from the model's residual stream activations. This provides partial evidence that the model internally represents causal structures, supporting the claim that it performs causal discovery.  \n   - Gradient-based interventions on the residual stream activations demonstrate that the model's predictions can be controllably altered by modifying its implicit SCM representation. \n\n3. **Generalization Challenge**:  \n   - The experimental design includes a \"generalization challenge\" where the model must answer counterfactual queries for SCMs it has only seen through interventional data. This effectively tests whether the model has learned a generalizable causal inference engine rather than simply memorizing training examples.  \n\n4. **Behavioral and Mechanistic Analysis**:  \n   - The paper combines behavioral metrics (e.g., prediction accuracy) with mechanistic analyses (e.g., probing and interventions), providing a comprehensive evaluation of the model's causal capabilities. This dual approach strengthens the validity of the conclusions.  \n\n---\n\n### Weaknesses  \n\n1. **Lack of Consistency Guarantees**:  \n   - The paper does not provide theoretical guarantees that the model's causal discovery process is consistent (i.e., that it converges to the true SCM as training data increases). Consistency is a fundamental requirement for reliable causal discovery, and its absence limits the method's applicability to real-world problems where correctness is critical.  \n   - Without such guarantees, it is unclear whether the model's success is due to the specific simplicity of the linear Gaussian SCM class or a more general capability.  \n\n2. **SCM Index Dependency**:  \n   - The model relies on predefined SCM indices to reference causal structures, which may act as a crutch, allowing the model to \"look up\" SCMs rather than infer them from raw data. This raises concerns about whether the approach can generalize to entirely new SCMs not seen during training.  \n   - In real-world settings, causal structures are not pre-indexed, so the model's ability to discover causal relationships from scratch remains unproven.  \n\n3. **Weak Experimental Comparisons**:  \n   - The paper does not compare the model's performance against established causal discovery algorithms (e.g., PC, FGES, LiNGAM) or state-of-the-art neural causal methods (e.g., DECI, DCDI). Such comparisons are necessary to assess whether the transformer-based approach offers any advantages in accuracy, scalability, or robustness.  \n   - The lack of benchmarks on standard causal discovery datasets (e.g., synthetic graphs, real-world cause-effect pairs) further limits the ability to evaluate the method's practical utility.  \n\n4. **Limited Scope of SCMs**:  \n   - The experiments are restricted to linear Gaussian SCMs with at most four variables. While this simplifies analysis, it leaves open whether the method can handle more complex, non-linear, or high-dimensional causal structures.  \n   - The reliance on analytical solutions for counterfactual queries (which may not exist for non-linear SCMs) suggests the approach may not easily generalize beyond the tested class.  \n\n5. **Ambiguity in Probe Interpretation**:  \n   - While probes can decode SCM weights, the paper acknowledges that decoding accuracy does not necessarily imply the model uses these representations for inference (Section 4.4). This weakens the claim that the model truly \"discovers\" causal structure, as opposed to merely encoding it incidentally.  \n   - The intervention experiments, while compelling, are limited to a small set of SCMs and weight modifications. Broader validation is needed to confirm that the model's behavior is systematically controlled by the manipulated representations.\n\nQUESTIONS:\n1. How does the model's causal discovery accuracy compare to traditional constraint-based (e.g., PC) or score-based (e.g., FGES) methods on the same SCM class?  \n2. Can the approach scale to more complex, non-linear SCMs, and how does its performance degrade compared to specialized causal discovery tools?  \n3. Does the method outperform existing algorithms in settings where only interventional data (no counterfactual examples) are available during training?  \n4. What are the computational and sample complexity trade-offs between this approach and classical causal discovery methods?\n\nLIMITATIONS:\nThe requirements of SCM index may restrict the proposed approach to be really useful.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts an empirical study designed to test and inspect the causal inference abilities of LLMs by training a GPT style model from scratch via next token prediction on interventional and counterfactual data. \n\n### Study setup\nThe paper considers the class of Markovian linear Gaussian structural causal models (SCMs) over 4 variables *with a fixed causal order* (important, see below); the intercepts and coefficients are sampled from {-1,0,1}. This results in 3^10  distinct causal models of which 1000 are used for testing and the rest for training. The LLM is trained on interventional data from both training and test SCMs, as well as on counterfactual data from the training SCMs, and evaluated on counterfactual predictions on the test SCMs. The data is simulated from the respective SCM and then encoded as strings, containing a dummy encoding of the SCM ID (random characters), of which variables are intervened upon or observed (DO vs OBS), and whether the input is interventional data or a counterfactual inference example (DATA vs INFERENCE). \n\n### Main findings\nThrough various experiments, it is shown that the trained LLM:\n- reaches near optimal performance for counterfactual inference on the test SCMs (given only interventional test data, and counterfactual data from the training SCMs) for most combinations of number of intervened and observed variables (Figs. 5 & 6) \n- learns an internal representation in its later layers from which the SCM parameters, a ternary 10-dim. vector in this case, can be decoded with medium to high accuracy using linear and MLP probes (Fig. 7)\n- appears to learn an algorithm for counterfactual inference (within the considered model class) in the sense that manipulating the model's internal representation to change some weights in the SCM leads to changed results that agree with what would be expected analytically (Figs. 8 & 9)\n- is only controllable up to (and incl.) its sixth layer after which controllability is lost (despite still containing information about the SCM parameters (Fig. 10, cf. Fig. 7)\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- the paper is very well written and easy to follow\n- the study is (for the most part---see below) well-designed\n- the experiments, which are carried out, are sensible, extremely well-described and augmented with very helpful figures\n- apart from the main limitation listed below, this study was a pleasure to read and is a prime example of good empirical research\n\n### Weaknesses\n- Unfortunately, it is a main flaw/limitation of the study design that the causal structure (i.e., the causal order) is fixed across all 59k+ SCMs. As a result, it is sufficient to memorise the correct (partial) causal order  $V_1 \\prec V_2 \\prec V_3 \\prec V_4$ globally. That is, the LLM does not have to learn how to perform causal discovery (rendering the title highly misleading), but only needs to solve the simpler, purely statistical problem of learning the partial correlations among variables within each SCM. In other words, **what the study in its current form demonstrates (and does so very well) is the following: given a *known/fixed causal order*, an LLM trained via next token prediction can learn the path coefficients within each SCM, and, given sufficiently many examples, can learn the abduction-action-prediction algorithm for counterfactual inference.** Hence, either the study design needs to be updated or the claims (and title) of the paper adjusted to reflect this.\n- [more minor] Pearl's claims in the introduction and their relation to the present manuscript appear slightly misleading or inaccurately portrayed. As far as I know, his claims are about a system trained on purely observational ($\\mathcal{L}_1$) data. While I think the point that natural language often conveys interventional or counterfactual knowledge is a good and valid one, this information is typically implicit in most datasets. In contrast, the interventional nature is made very explicit in the constructed dataset via the DO/OBS distinction. I don't think the data generating process needs to modified (beyond also randomizing the causal order as explained above), but more work is needed to convey this mismatch/these nuances in the Introduction and Discussion.\n\nQUESTIONS:\n- Why did you decide to fix the causal order (effectively removing the causal discovery aspect from the study)? Can you rerun the experiments (or a smaller version thereof, e.g., with only 3 variables) for a scenario in which the causal order is randomised? (e.g., you could randomly permute the label of variables separately for each SCM so that it is also possible that $V_2$ causes $V_1$) Without this (I realise this may not be feasible within the rebuttal period), the main claims of the paper should be adjusted accordingly and a clarification added. \n- For the current design (with fixed causal order), I hypothesise that even observational data should be enough (since no per-SCM causal graph needs to be learnt), provided that counterfactual inference statements are still provided. It could be interesting to try how the model performs when only given counterfactual inference queries (if sufficiently many queries are given, it should be possible to learn the path coefficients) or when additionally given observational instead of interventional data. \n- [minor] in Fig. 3 (top), why does the sample from the interventional distribution contain two different values for $V_3$? (V3 -0.9, V3 -1.5) Is this a typo and the first one should be V2? (else, please explain what is going on here)\n\nLIMITATIONS:\nWhat I view as the main methodological limitation of the work---using the same fixed causal order among all training and test SCMs (see above for details)---is not discussed at all. In light of this, I do not think that the study provides convincing evidence that LLMs can reliably perform causal discovery (as may be inferred from the title), beyond learning an algorithm for inference within the narrow model class considered (that much is acknowledged by the authors). Readers should not conclude that LLMs outputs necessarily support causal claims, which may lead to false overconfidence in using them for high-stakes causal inference tasks. I think adding a short discussion to clarify this would be a useful addition.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}