Causal Discovery and Inference through Next-Token Prediction
Authors: Eivinas Butkus
Nikolaus Kriegeskorte
Venue: NeurIPS 2025 poster ()
Keywords: causal inference
causal discovery
transformers
large language models
language modeling
mechanistic interpretability
structural causal models
neural probing
decoding
causal intervention
next-token prediction
counterfactual reasoning
causal representation learning
Pearl's causal hierarchy
ladder of causation
emergent representations
world models
Primary area: deep_learning
TL;DR: Statistical prediction may be sufficient to drive the emergence of internal causal models and causal inference capacities in deep neural networks.

Abstract: Deep neural networks have been criticized as fundamentally *statistical* systems that fail to capture causal structure and perform causal reasoning. Here we demonstrate that a GPT-style transformer trained for next-token prediction can simultaneously discover instances of linear Gaussian structural causal models (SCMs) and learn to answer counterfactual queries about those SCMs. First, we show that the network generalizes to counterfactual queries about SCMs for which it has seen interventional data but not any examples of counterfactual inference. The network must, thus, have successfully composed discovered causal structures with a learned counterfactual inference algorithm. Second, we decode the implicit “mental” SCM from the network's residual stream activations and manipulate it using gradient descent with predictable effects on the network's output. Our results suggest that statistical prediction may be sufficient to drive the emergence of internal causal models and causal inference capacities in deep neural networks.

Causal Discovery and Inference through Next-Token Prediction
Eivinas Butkus
Nikolaus Kriegeskorte
eivinas eivinas. www..uuuuuuuuomubaaocoooo edu
wwc.iiggeeootteeollmbbbbi...... edu
'Columbia University
NSF Al Institute for Artificial and Natural Intelligence
Abstract
Deep neural networks have been criticized as fundamentally statistical systems that fail to capture causal structure and perform causal reasoning. Here demonstrate that GPT-style transformer trained for next-token prediction can simultaneously discover instances of linear Gaussian structural causal models (SCMs) and learn to answer counterfactual queries about those SCMs. First, we show that the network generalizes counterfactual queries about SCMs for which it has seen interventional data but not any examples of counterfactual inference. The network must, thus, have successfully composed discovered causal structures with learned counterfactual inference algorithm. Second, we decode the implicit "mental" SCM from the network's residual stream activations and manipulate it using gradient descent with predictable effects on the network's s output. Our results suggest that statistical prediction may be sufficient drive the emergence of internal causal models and causal inference capacities in deep neural networks.
Introduction
How can an AI system discover and learn to reason about causes and effects-the mechanisms that remain invariant under local interventions 43 35
Pearl [3] has argued that deep neural intrinsically limited in their causal reasoning capacities. His argument rests on Pearl's Causal Hierarchy (PCH) (2). also known as the "Ladder of Causation" [38. PCH describes three levels of causal capabilities capabilites--asociattion
networks (DNNs) trained using prediction objectives
(C1), interventional (L2), and counterfactual (La)-and implies that answers to higher level queries are generally under- determined by data or information from lower levels. According Pearl [36], this hierarchy implies that DNNs trained a "statistical mode" to predict passive observations can only master associations (f1) and prevented from reasoning about actions, experiments, and explanations (L2 and Lz).
true, Pearl's claim would have important theoretical and practical implications regarding the causal abilities of large language models (LLMs) and other foundation models. Such models are typically pretrained using statistical objectives to predict held-out portions of passive streams of 6 5). For example, LLMs are trained to predict the next token in snippets drawn from a diverse corpus of text. Following Pearl's reasoning (36), at least initial pretrained versions of foundation models should be limited to the level of associations (L1).
data [8 40 39
While PCH is an extremely valuable theoretical framework with wide-reaching implications, we do not think that Pearl's 's claim about DNNs directly follows from PCH. Training data for foundation models contains rich information about the causal structure the world. Natural language in particular contains many descriptions of interventions and causal inferences (Fig. D. "Passive" data, then, is not equivalent to "observational" (Li) data. Notably, in a more recent interview, Pearl [37 acknowledges that text does contain Lz information. Standard PCH logic, therefore, cannot rule out the emergence of causal models in LLMs and other kinds of foundation models.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

"even my head would go through," " thought poor Alice, "it would be of very little use without my shoulders".
This bottle was not marked "poison." so Alice ventured to taste ... "I must be shutting up like a telescope." she was now only ten inches high
"Well, I'll eat [the cake]." said Alice, "and If it makes me grow larger. I can reach the key; and if it makes me grow smaller, can creep under the door; so either way I'll gel into the garden
[She] soon finished off the cake, "now I'm opening out like the largest telescope that ever was! Good-bye, feet!"
Figure I: Natural language includes descriptions of interventions and causal inference. Examples from Alice's 's Adventures in Wonderland [7]. illustrated by John Tenniel. The drink makes Alice shrink, while the cake causes her grow; she also reasons counterfactually about fitting through the tiny door. LLMs may discover causal structure- the mechanisms that remain invariant under local and learn causal inference engines to predict the next token in such strings.
interventions 43 35
However, Pearl 37 and others [51] have maintained that LLMs cannot possess causal models. The underlying assumption seems to be that systems operating in "statistical" or "model-free" mode regarddess of the data they operate on mn-cannot give rise to genuine internal causal models. This reasoning emphasizes the nature of the training objective (prediction) while neglecting the content of the training data (which includes causal information). Even when applied to text containing causal information, the argument goes, prediction objectives can only capture statistical associations (however richly and abstractly structured they may be) rather than causal understanding that supports generalization to novel scenarios (37]. In other words, LLMs are "causal parrots" (51).
We propose an alternative hypothesis and provide an existence proof: The next-token prediction objective, at least in some contexts, may drive LLMs to acquire real causal models and causal reasoning capacities at the interventional (C2) and counterfactual (Cg) levels. We test this hypothesis empirically in controlled setting. We generate text (in simple artificial language) describing interventional data and counterfactual inferences from a constrained class of linear Gaussian structural causal models (SCMs). Generated text strings fall into one of two classes (Fig. 3 (1) DATA strings describe the behavior of the referenced SCM under interventions; (2) INFERENCE strings describe counterfactual inferences, given the referenced SCM. The strings reference the underlying SCMs via arbitrary indices, and the SCM structure is never explicitly provided. We train a GPT-style transformer model to predict the next token in these strings.
To reduce next-token prediction loss in our task, the model could adopt one of two strategies: memorize the training examples, or discover the structure of the underlying SCMs and learn more general causal inference algorithm. To distinguish these two possibilities, we devise a "generalization challenge" where the trained model has generalize to counterfactual queries about a separate set of test SCMs (Dres) for which the model has only been trained with interventional data, and not with any counterfactual inference strings. The only way the model might correctly complete counterfactual inference strings about these SCMs is if it had both (1) inferred the structure of the test SCMs from the interventional data and (2) learned how perform counterfactual inference for our class of SCMs. We find that the trained model generalizes to the test SCMs, indicating it has not memorized the answers, but has instead learned counterfactual inference engine for our class of SCMS.
In addition to demonstrating generalization of counterfactual inference behavior, we use mechanistic interpretability tools to probe the network's 's representations. First, we show that we can decode the referenced SCM from the transformer's 's residual activations using linear probes. Second, we use probes to manipulate the SCM in the "network's 's mind" mid-computation using gradient descent on the activations. These interventions have predictable effects on the network's 's output, suggesting that our probes capture a real SCM representation used by the model.
Existing critiques of L.MS' causal capabilities do not clearly define what it would take for an LLM to possess an internal causal model [37 51]. We establish three pieces of evidence that our trained
2

---

network possesses internal causal models: (1) it generalizes to unseen query--trucctn combinations, (2) learns decodable representations of causal structure, and (3) these representations can be causally manipulated with predictable effects on the output. Together, our results provide an existence proof of a DNN trained to predict passive streams of data that can nonetheless discover and use causal models.
2 Related work
Our work contributes to the broader discussion about the extent to which LLMs and other foundation
models understand the world 4 49 28 55]. Pearl [35 36) and others [12 20 38
have argued that human understanding derives from building and using powerful causal abstractions of the world. Causal capacity of LLMs thus plays an important role in this larger debate about understanding.
Some have been skeptical about the causal abilities of LLMs. Referencing Pearl's hierarchy, Zežević et al. 15 argue that LLMs are "causal parrots" that occasionally answer causal questions correctly by capturing the "correlation of causal facts", We agree that LLMs may learn to recite often repeated causal claims or exploit correlations between certain words to answer causal queries. However, here we present evidence that LLMs may also discover true causal structure from text and learn causal inference engines that generalize to unseen structure and query combinations.
17). Several datasets and benchmarks have been created to make progress on this question [5
Empirical results on LLM causal capacity have been mixed 18
52
42 In contrast to evaluating causal abilities of pre-trained L.LMs, here we consider a relatively simple causal task (discovery and inference within constrained set of linear Gaussian SCMs) and train a small transformer from scratch with full control over the training data. We then study not only the behavior of the model, but also its internal representations. We are inspired by other probing and mechanistic interpretability work 33 10 31, particularly on emergent world representations in transformers [22 23 24
16
and knowledge localization/editing [26 [4].
We use the structural causal model (SCM) formalism 35 to synthesize our training data. Prior work
has shown that transformers trained on synthetic data can uncover hierarchical or compositional structure. For instance, Murty al. 20 show that transformers can learn hierarchical syntactie rules through extended training, while Lake and Baroni [19 demonstrate systematic generalization to novel combinations through meta-learning on algebraic reasoning tasks with compositional structure. SCMs are distinct in that they encode mechanistic causal relationships that support interventional (C2) and counterfactual (L3) reasoning within Pearl's Causal Hierarchy 2 Training on SCM generated data
allows us to put Pearl's theoretical claims about the limitations of deep neural networks [38 36 37
to direct test.
It is also worth mentioning neural-causee models 34 equations with differentiable neural components. Instead, we train standard transformers end-to-end on next-token prediction on data generated from the SCMs, demonstrating that causal reasoning can emerge from purely language modeling objective.
47
48
46
that replace SCM functional
Finally, our work complements a finding from Lampinen et al. 21 that RL agents can learn "causal strategies" from purely passive offline data, which the agents can then exploit by intervening at test time to uncover causal structure. Interestingly, our results suggest that purely "passive" training on next-token prediction without any interventions carried out by the model may be sufficient to discover causal structure and learn a causal inference engine.
Methods
Our setup involves: (1) generating instances from a constrained class of linear Gaussian structural causal models (SCMs) (section B.I): (2) generating text using those SCMs in a simple artificial language that describes interventional data and counterfactual inferences (section 3.2); (3) setting up a generalization challenge for the transformer by providing only interventional data strings for
some of the SCMs during training (section 3.3): (4) training a GPT-style transformer to predict the next. token in generated text (section 3.4. We then study the emergent behavior of the model and its internal representations.
3

---

SCM 12002
SCM 39124
SCM 57123
U.
V1
V22
U.
Ux
Figure 2: Some examples of the 59,049 unique SCMs. Bias weights wa are displayed top of the variables. Non-zero effect weights Wij are displayed on the edges between the variables.
3.1 Linear Gaussian structural causal models (SCMs)
We consider a constrained class of linear Gaussian structural causal models (SCMs) with 4 variables V1, V2, V3, Vi:
U,~ ~ N/0, o2) with v0.1 0.32 V where Vi,j wiy E {-1,0,1]
(1)
Vy= U, Øグ +
(2)
Background variables U = U2. U3, U4} (sometimes referred to as "exogenous" or "noise" variables) are sampled independently from a fixed Gaussian distribution. Each endogenous variable VG is a linear function of the corresponding background variable Uj. bias term Wjj. and a weighted combination of parent values Σies Wiy V,, where why represents the effect of variable Vú on Vj. Note that each SCM instance within our class is fully defined by weight vector w with 10 ternary values:
w [WI1, W12, W13, W14, W22; W22; W24, W33, W34, W44]
This weight vector w can be interpreted as the symbolic causal generative program for particular SCM within our SCM class. Foreshadowing the results, it turns out that can linearly decode w from the transformer's residual stream activations and manipulate it using gradient descent with predictable effects model's output.
We generate all possible weight combinations (3 possible values for 10 weights), resulting in 310 59,049 unique SCM instances. Some example SCMs can be seen in Fig. 2
Answers counterfactual queries within the linear Gaussian SCM class can be computed analytically (details provided in appendix D.I). Efficient analytical estimates allow us to implement a text- generative model that dynamically samples queries and computes answers for each training batch.
3.2 Generating text (training data)
Natural language includes descriptions of interventions, outcomes under those interventions, and various causal inferences (Fig. []. Our text generation setup using simple artificial language is intended to emulate this aspect of natural language.
More concretely, we generate two types of strings (Fig. 3 Each string begins with token that describes its type (DATA or INFERENCE), followed by an SCM index encoded using letter tokens (c.g. R Q corresponds to SCM index 12002). Crucially, the SCM structure is never explicitly provided the SCM indices are assigned randomly and, by themselves, carry no information about the underlying SCM. Therefore, the model has to discover the structure of the associated SCM from the data. We use OBS [Vi] [value] token sequence to represent an observation and DO [vi] [value] to represent with one decimal point precision (e.g. 0.3, -7 -7.5). Numbers that fall outside of that range are encoded using IINF and +INF tokens. All strings end with EDS token.
intervention. Numbers within [010, 10] are encoded using numerical tokens
DATA strings answer: What happens if we intervene on X? (C2 information within PCH). They provide noisy interventional data about the underlying SCMs. To construct a DATA string, sample up to

---

vocabulary
DATA INFERENCE
DQ
OBS
V1
v4
INF
-10.0
9.9
10.0
+INF
BOS end sequence
string type
SCM Index encoding
Intervention/ observation
variables
numbers encoded using one decimal point precision
example strings
DATA
-0
string
SCM index
0-2 interventions
sample from the interventional distribution
end of sequence BOS end sequence
INFERENCE
OBS
V2
DC
V1
2
V
6 0
2.0
string type
SCM index
observations and 0-2 interventions
counterfactual means and standard deviations given observations and interventions
next-token prediction
internal SCM representation
target: -1.3
transformer
input:
DATA
DO
V3
1.5
V2
0
V3
1.
V4
Tokens in the vocabulary of the artificial language. Middle: Example strings from the training data. We generate two types of strings using the SCMs. DATA strings provide noisy samples from the SCM under interventions. INFERENCE strings provide examples of counterfactual inference. Bottom: We hypothesized that the GPT-style transformer may learn interpretable internal SCM representations predict the next token.
Figure 3: Top:
two interventions with values drawn uniformly from 51 ensuring that the intervened variables do not repeat in the same string. Given the referenced SCM and interventions, we compute the analytical interventional distribution, and sample one realization from that distribution. We then record the sampled variable values in a random order using [Vi] [value] token pairs. Thus, each DATA string provides one sample "rollout" from the interventional distribution- single realization of what could happen under the intervention.
INFERENCE strings answer: In this specific world where we observed Y, what would have happened if we had intervened on X? (L3 information within PCH). They provide examples of counterfactual inference within our SCM class. To construct INFERENCE string, sample 1-2 observations (also ~ 51 5)) and (-2 interventions. We then compute the analytical counterfactual distribution (details in Appendix D.Ip. In simple terms, counterfactual inference can be decomposed into three steps: "abduction", " "action", and "prediction" (35). We first condition on the observations Yous and find the posterior over the background variables p(U Y yotss (abduction). We then alter the factual SCM according to the interventions by fixing the intervened variables to constant values := Zini (action). Finally, we compute the counterfactual posterior distribution P(V Y Yous). given the new background variables and the intervened SCM (prediction). We then use random order to record the counterfactual mean and standard deviation for each variable using [Vi] [mean] [std] token sequence. Unlike DATA strings which provide single samples, INFERENCE strings require computing the counterfactual posterior mean and standard deviation for a specific scenario.
Finally, we consider two variable naming schemes (fixed and shuffled) that map variable names (V1, V2, V3, V4) to ground truth variables (V1, 122 V3. V4). In the fixed naming scheme, each name refers to the same variable across SCMs: V1 + V1, V2 + V22 V3 + V3. V4 + V4. In the shuffled naming scheme, for each SCM we sample permutation uniformly from all 4! 24 possible permutations, which then maps names to variables: Vi V=(a)- The shuffled naming scheme is challenging since mapping for each SCM cannot assume that, say, the variable named V1 precedes V2 in the causal order. We use the shuffled naming scheme for the main behavioral results (section 4.1 to demonstrate that the model can indeed discover the underlying SCM structures even without global fixed naming scheme. However, probing models trained with shuffled names is challenging (see appendix (D.2 for further explanation). so we used the fixed naming scheme for the mechanistic interpretability analyses (sections
the model has discover the name-to-variatee
5

---

SCM set
number of SCMs
INFERENCE strings
DATA strings
(1) learning counterfactual
(2) building a shared representation
Isin
58,049
inference engine
generalization
discovering causal structure from interventional data
D lest
1,000
challenge!
X
strings seen during training? / X
Figure 4: Generalization challenge. To generalize to unseen INFERENCE queries for SCMs with DATA strings only (Dres) ) the model has to: (1) learn a counterfactual inference engine from descriptions of inferences with the Dunin SCM set, (2) build representation for counterfactual inference and interventional data strings that exploits the shared structure, (3) discover the structure of Deest SCMs from interventional data.
3.3 Generalization challenge
We wanted to distinguish between two possibilities: (1) The model memorizes answers to causal queries; (2) The model learns a causal inference engine for our SCM class. For this purpose, devised "generalization challenge" by randomly choosing held-out (denoted Diest ) for which the model only saw DATA strings during training (Fig. 4). If the trained model answer counterfactual queries about this test set, it means that has (I) learned more general counterfactual inference engine, (2) built shared representation for interventional data and counterfactual inference, and (3) discovered the causal structure of SCMs within the Diest from interventional data strings. In other words, it can compose the learned counterfactual inference engine from Duain strings with the discovered causal structure from interventional data strings in D
of 1,000 SCMs
3.4 Model architecture and training
We use the TransformerLens
implementation of a GPT 2-style decoder-only transformer. Trans- formerLens is a mechanistic interpretability library that exposes the model's 's internal activations for reading and editing purposes during the forward pass. We utilize these features of TransformerLens post-training when we decode SCM weights from model's residual stream activations and intervene on the SCM representation mid-computation.
Our transformer model has 12 layers, hidden size 512, 8 attention heads size 64, MLP size 2048, GELU activation function, and "Pre-LN" type layer normalization. We use AdamW (25) with learning rate 10-5, betas [0.9, 0.999], eps 10-8 and 0.001 weight decay. We set batch size to 128 and train for different number of epochs depending on whether the variable naming scheme is fixed or shuffled (see 3.2). When variable names are fixed, we train for 300 epochs, reducing learning rate 10-6 for the last 10 epochs. When variable names are shuffled, the model takes longer converge, so we train for 1,500 epochs, reducing learning rate to 10 for the last 100 epochs. In each epoch, we draw 10 DATA and 10 INFERENCE strings per SCM from our text generative model, resulting in around 1.2 million strings per epoch. Note that there are O INFERENCE strings for Dress SCMs in the training data.
Results
We show the following: (1) The trained model successfully generalizes to counterfactual queries about those SCMs that only had interventional data strings (section (2) We can decode the SCM weights from residual stream activations within the model using linear and multi-layer perceptron (MLP) probes (section 4.2). (3) We can use these probes to manipulate the underlying SCM representation within residual activations using gradient descent with predictable effects on model's and
output (sections and 4.4). Our results suggest that the next-token prediction objective drives the model discover SCMs and learn an algorithm for counterfactual inference that generalizes within our SCM class.

---

Loss
DATA sample prediction
INFERENCE
INFERENCE
mean prediction
std prediction
aaselne
0.10
less
2.0
absotute
1.0
absolute
1.5
0.05
0.5
0.5
вp ar performancce
optimal performance
0.00
8ptimar performance
1000
1500
500 opech
1000
1500
500
1000
1500
500
1000
1500
epoch
epoch
epach
SCM set
Dtrain
Dtest
DATA and INFERENCE strings
DATA strings only
Figure 5: We track model loss and absolute emror while predicting interventional DATA samples, counterfactual INFERENCE means and standard deviations for both Dirain and Dies SCM sets. The model eventually reaches near-optimal prediction performance. Naive baseline represents predicting the average value. Shaded regions show 95% bootstrapped confidence intervals for the mean across evaluation examples.
4.1 Transformer generalizes counterfactual inference to SCMs for which it has seen only interventional data
Our main behavioral test is generalization challenge (3.3 The core intuition behind the challenge that SCMs from the Dust set are referenced only within DATA strings during training. If the trained model can correctly complete strings with counterfactual queries about Diest SCMs, it means that: (1) the model has learnt a counterfactual inference engine (not just memorized the answers to counterfactual queries for the Dimin set), and (2) it has discovered the causal structure of Diest set SCMs from noisy interventional data.
We evaluate the model by providing causal query (string type, SCM index, observa- tions/interventions) within its context window and letting the model predict the next-token. We then calculate the absolute error between the predicted and the ground truth values (see appendix E, for further details). For the main behavioral results we consider the more challenging shuffled variable naming scheme, where randomly permute the names across SCMs (see 3.2 However, behavioral results also hold when variable names are fixed (see appendix E.3)
Fig. 5 shows the training trajectory for few key metrics: the loss, absolute errors on interventional
data samples (evaluated on DATA strings), and absolute errors for counterfactual means and standard deviations (evaluated on INFERENCE strings). The model eventually reaches near-optimal prediction performance for both Darin and Dres SCMs.
Fig. [6 zooms in on the last epoch and shows the trained model's performance as function of number of interventions and observations provided in the query string. Overall, the network performs near optimally on all accuracy metrics for all conditions. Most importantly, there is no performance gap
DATA prediction
INFERENCE
INFERENCE prediction
prediction
0.05
0.005
0.38
Dowa Deatt
obs
1bbb
absolute eftor
0.36
optimal
absolute eftor
0.04
abssoluze ettor
apt
0.03
Dewh
abs
0.003
2bb
ma.mak...
apt.ak....
0.32
0.02
0.002
0.30
0.01
0.001
0.000
0.00
number of interventions
number of interventions
number of interventions
Figure 6: Trained model's errors as function of number of interventions and observations in the query. The model performs near optimally on all prediction tasks. Importantly, performance on the Dress SCM set does not systematically differ from D model passes the generalization challenge. Shaded regions indicate 95% bootstrapped confidence intervals for the mean across evaluation examples.
7

---

between Durin and Diest SCM sets model generalizes successfully to counterfactual queries about Des SCMs. We conclude that the network passes the generalization challenge, i.c. it does not simply memorize strings in the training set, but learns a counterfactual inference engine for our SCM class, discovers Dee SCMs from strings describing interventional data, and successfully composes the discovered structure with the learnt inference engine.
4.2 SCM weights can be decoded from residual stream activations
In the previous section, we established that the model generalize to unseen query- combinations. But what supports this behavior internally? Since the weight vector w (discussed in 3.1) fully defines an SCM within our class, we hypothesized that the model may map SCM indices to an internal representation that is closely related to this vector. To test this hypothesis, we trained "probes" [1 3 [
that map model's residual activations to elements of the SCM weight vector u. Namely, we train a separate probe for each of the 12 transformer layers to each of the 10 SCM weights, resulting in 120 separate classifiers. We decode from residual activations the last SCM index position as that position had the highest decoding accuracy-consisteen consistent with the idea that the model "pulls up" the SCM representation given the full SCM index for downstream computation.
Inspired by [23 32). we train both linear and multi- perceptron (MLP) probes. Linear probes map the 5i2-dimensional layer I residual stream activations directly to 3-way softmax output for each weight way E {-1, 0, 1}, while MLP probes include a 256-dimensional hidden layer:
linear (x1) softmax (WTa)) TTx 21);
W E R
(4)
Paiy (zi) softmax ? ReLU(WT x))
W1 E 5112225
W2 R
(5)
We split Drrain set (Fig. into Drrobe (57,049 SCMs) and D (1,000 SCMs) sets. We train the probes on D Drroee obtaining last SCM index position residual activations using INFERENCE [SCM index] input strings (see appendix |E.4 details on probe training). We then test probe accuracy on the Drest SCM set (Fig. 7 SCM weights can be decoded above chance (33%) using both linear and MLP probes, suggesting that the transformer maps the arbitrary SCM index into a meaningful structured representation of the underlying SCM.
4.3
We can manipulate SCM representation in the residual stream
Probing results should be interpreted carefully, especially for non-linear probes Although successful decoding establishes the presence of information, we do not know whether or how this information used by downstream computation. We hypothesized that probes do in fact recover at least parts of the model's representation of the underlying SCMs. If our hypothesis is correct, we should be able to control model's behavior by overwriting the SCM representation the "network's mind" [10
Linear probe
MLP probe
decoding accuracy on Dani SCM
decoding accuracy Dun set
2.36
0.79
955
6.43
8.63
0.66
1,67
2
8.59
2.16
2,38
9.39
9.35
8:55
0.21
5:35
1.1
2,80
6.59
243
11.
0.56
9.23
0.55
8.52
0.52
0.08
1.63
3/43
5.58
3.45
31-
0.55
0.54
0.55
5.97
0.92
0.45
1.93
3.87
5.42
2.02
0.75
0.37
0:38
5.93
9,32
9.52
9.32
58A
9.52
9.73
3.68
2.58
8.58
2.46
1.38
0.22
9.38
5.25
0.22
0:38
1.74
285
9.89
2,0
9.22
9.62
3.83
2.54
0.32
0:15
5.7%
Layer
1.28
9.25
8:42
9.73
1.35
2.35
2.52
1:08
0.25
5.83
0.22
1:93
2.50
5.59
1.92
9.38
9,53
1.41
9.52
1..0
2.53
2.2k
2.45
9,12
9.355
0,/2
5:52
18:33
5:43
2.51
1,89
3.41
2,13
0.53
9.55
9:58
0.53
0.45
9:81
2,25
9.48
2:50
2.53
0.59
0.28
5.53
0.52
0:45
5:85
2.58
9:68
2.55
0.25
5.31
0.33
5.34
2.35
2.15
0.15
9:16
9.35
2.35
9.36
0
0.15
6:31
0.31
0.30
8:31
2,21
6:71
2:75
2
0.34
0.30
0.15
5:31
0.33
0.32
5:33
2.30
5:32
2.31
2,40
0.54
9.33
8.35
0.53
9.53
2,55
2,53
0.53
2.18
2.34
0.51
0,15
0.33
0.51
0.15
5.33
2,8%
1.58
2,81
W33
W1z
W954
Wea
W13
M1.
W/4
Wy5
Wee
welgitt
SCM
weight
Figure 7: Average SCM weight decoding accuracy. Each cell corresponds to accuracy of the classifier that takes residual stream activations of a single layer and predicts SCM weight Whj E (-1,0, 1). Values in smaller font below indicate 95% bootstrapped confidence interval for the mean across the SCM set. Chance performance 33% By layer 5-6, most SCM weights can be decoded by linear or MLP probes above chance.
8

---

intervening on the internal SCM representation
2. change make a hypothetical change the SCM weights
[0, 0 -1,0 -1..
1. read
3. write
decode SCM weights using probe w [0, 0, -1,0,
update activations to make the probe predict the hypothetical change aVs)
Vx
predicted
mean
queried
input:
IMPERENCE
variable
linear probe intervention example
Query: INPERENCE
OBS V4 -0.3
[Vs] [nean prediction]
mean
V2 mean
V3 mean
V4 mean
2
token Probbblity
0.8
token
0.6
0
0
0,2
-2
-2
-2
gradient descent steps on ros dual stream activations
analytical result
intervention
after intervention
(Wi2 D)
(Ws2 1)
Figure 8: Top: Intervening the internal SCM representation. Bottom: Linear probe intervention example (W12 = 0 + W12 = 1). To test whether our intervention successfully modifies the representation, we: (1) specify causal query, (2) compute analytical estimates for the counterfactual mean before (W12 = 0) and after (W12 = 1) the corresponding intervention in the ground truth SCM, and (3) check whether the post-intervention model predictions match analytical estimates. Model predictions flip for V22 and V4, aligning with analytical estimates.
To test our hypothesis, we use the gradient signal from the probes to intervene on the SCM rep- resentation mid-computation (Fig. 8). We successfully manipulate the representation. the model predictions post-intervention on the residual stream align with analytical estimates for the corresponding change in the ground truth SCM, suggesting that the probes do in fact recover actual representation of the referenced SCM. Further details on our intervention setup and an intervention example using MLP probe be found in appendix R.5
4.4 Decoding accuracy does not imply controllability
We introduce quantitative metric (the "intervention score")) to measure how well we can control the network's 's behavior by changing residual stream activations. Given an SCM, weight change (my w/y residual stream intervention at layer I (xz - xx)) and queried variable Viueried:
intervention
score
3
P
- model (token V 2y -
queried
Pmodel (token( V Wiy w (D)
queried
(6)
probability assigned to the correct token after residual stream intervention
probability already as assigned to the correct tuken before residual stream intervention
Note that this score can be negative when the model assigns non-zero probability to the correct post-intervention token before the intervention, but the intervention on activations (1 (1++1 1') lowers that probability. Moreover, the model generally hedges its bets, normalize the quantity in the

---

big brackets by the average max probability & model assigns during normal operation (< 0.89). An average intervention score of 1 then corresponds to "perfect" interventions the model gives correct answers with the same average confidence as during normal operation. Further details on computing intervention scores can be found in Appendix E.6
Linear probe
MLP probe
average intervention
12
2.38
0.03
0.02
2.51
9:58
9:35
0,59
11-
2,88
0.03
0.38
5.85
0.02
0.55
1.42
2.02
0,59
2.52
2.85
0.90
0.58
5.05
0.02
0.38
1.82
2.02
5.99
2
10
9.88
0.01
0.38
8.88
0.02
0.88
3.83
2.00
5.59
2.53
9.28
0.90
0.50
5.85
0.02
0.38
5.8
2,00
0,60
2.88
0.85
9.88
6.6
0.03
9.55
1.83
2.00
5.59
2.53
2.88
0.08
9.58
686
0.02
9:38
888
2.00
5,99
0.58
0,03
1.58
5.65
0,02
5,56
5.63
2,00
5,00
3,53
8-
9.35
0,00
0.50
5.85
0.02
0.30
5,83
2,02
5,00
2.2
Layer
9.38
0.02
5.02
0.52
0.23
0.01
2.38
0.92
9:55
8A6
0.05
0,36
3.04
2.53
1:52
9.55
0.39
9:48
8.55
9.33
9:48
5,94
2.44
6:09
2A1
9.31
1/24
01-3
0.12
0.26
9.00
1.02
3.01
2.32
2.52
9.23
0.53
5:03
0:25
5.03
5.02
2.92
2,53
9,02
1.52
3.94
0.32
1.4
2.55
5,59
2.51
3-
0.52
9.85
0.56
5.86
9.52
1:23
5.85
2
8:58
2
9.12
0.02
13.05
19.05
2.52
0.88
1933
8.81
0.31
3.05
[5.8g
0.12
0.01
10.33
35.1
0.05
5,.00
3,05
3,85
15.65
2.56
0.33
0.00
0823
3.00
0.02
0.03
3.00
3,02
5.00
2,01
14y1
5CM weight
SCM weight
Figure 9: Intervention scores are not coupled to decoding accuracy beyond layer 7 (compare to Fig. 7. Each cell corresponds to the average intervention score for weight Wi and layer l. Values in smaller font indicate 95% bootstrapped confidence interval for the mean across queries, queried variables V, and weight changes wij - w//--
Quantitative intervention analysis allows us to localize where the SCM representation is used (Fig. ( Note that the decoding accuracy is decoupled from the intervention scores beyond layer 7-changing residual stream activations (at the last SCM index position) beyond layer 7 does not have an effect on model output. Furtermore, while the linear probe has lower overall decoding accuracy, it achieves higher intervention scores for some layer-SCM weight pairs (particularly, for W111 Fig. 14 for contrast between linear and MLP intervention scores).
5
Discussion
By demonstrating that transformers can discover causal structure and learn causal inference engines solely through next-token prediction, we challenge fundamental assumptions about the limitations of neural networks trained to predict "passive" streams of data [36 38 37 51. Although the learning is passive in the sense that the model cannot perform actions in the world, the data is often rich with causal information, reflecting interventions and causal inferences performed by other agents. Even a purely predictive model may discover the underlying causal structure and learn causal inference to meet its prediction objective.
This raises broader questions that apply to all foundation models. For instance, do video generation models (c.g. OpenAl's Sora (5) learn simulators of the world- correppodin,, at least informally, to La of Pearl's Causal Hierarchy? Can we find abstract representations of physical objects and agents within their activations that we can intervene upon? If so, can we augment these models with causal reasoning capacities, so as to enable them to reason counterfactually (Cg)? We are excited to explore these questions in future work.
The primary limitations of our work are the artificial language setup and the constrained SCM class. Our setup, while designed to emulate how natural language conveys causal information about interventions and inferences, does not capture the full richness of real-world text and causal structures. However, this constrained setting was crucial for providing a clear and unambiguous "existence proof" that statistical prediction can, in principle, give rise to causal models and reasoning. Future work could explore to what extent our results generalize to natural language descriptions of the causal scenarios and more complex causal model classes, including non-linear SCMs.
Conclusion. Our work provides conerete existence proof that statistical prediction objectives can drive the emergence of causal models and causal reasoning. Through careful behavioral tests and mechanistic interpretability analyses, we demonstrated that next-token prediction can yield models that discover causal structure, build internal causal representations, and perform counterfactual inference. While our study focuses on a constrained setting, it opens new questions about the causal capabilities that may be emerging in today's 's foundation models trained on vast datasets.
10

---

Acknowledgments
We thank Yushu Pan and Zhuofan Josh Ying for helpful suggestions during the early stages of the project, and Elias Bareinboim and members of the Causal Artificial Intelligence Lab at Columbia University for feedback on how to situate the work in the broader context of causality research. We also thank reviewers at NeurlPS 2025 and the Mechanistic Interpretability Workshop at NeurIPS 2025 for insightful comments that substantially improved the quality of the paper. Initial ideas for this project emerged during book club between Eivinas Butkus and Cecilia M. Crews while reading The Book of Why (38).
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 1rxiv:160000666 2016.
[2] Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. On Pearl's hicrarchy and the foundations of causal inferenec. In Probabilistic and causal inference: the works of judea pearl, pages 507-556. 2022.
[3] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics. 48(1):207-219) 2022. Publisher: MIT Press One Broadway. 12th Floor, Cambridge, Massachusetts 02142, USA
[4] Emily M Bender, Timnit Gebru, Angelina McMillan-Maayy Major, and Shmargaret Shmitchell. the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610 623, 2021.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Erie Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Vidco generation models as world simulators. 2024. URL httpps:////ppeaaai http:p/://pppaaa..cc///eeeeeee/e com//researcc//iiie peneerationmodeels-.e models-
orld- simulators
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and others. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[7] Lewis Carroll. Alice's 's Adventures in Wonderland. Macmillan, London, 1865.
[8] Jacob Devlin, Ming-Wis Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[9] Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield- Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, A Mathematical Framework for Transforme Circuits. Transformer Circuits Thread, 2021.
[10] Atticus Geiger, Hanson Lu, Thomas lcard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-998 9986,, 2021.
[11] Alison Gopnik and Henry M Wellman. Reconstructing constructivism: causal models, Bayesian learning mechanisms, and the theory theory. Psychological bulletin, 138(6)10085, 2012. Publisher: American Psychological Association.
[12] Alison Gopnik, Clark Glymour, David M Sobel, Laura Schulz, Tamar Kushnir, and David Danks. theory of causal learning children: causal maps and Bayes nets. Psychological review, 111(1):3, 2004. Publisher: American Psychological Association.
[13] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.0161 2023.
[14] Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differencee in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36:17643-17668 2023.

---

[15] Zhijing Jin. Yuen Chen, Felix Leeb, Luigi Gresele, Ojasy Kamal, Zhiheng Lyu. Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiin Mrinmaya Sachan, and others. Cladder: Assessing causal reasoning in language models, Advances Neural Information Processing Systems, 36:3 36:31038-3105,, 2023.
[16] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? arXiv preprint arXiv:2306..00 0068336 2023.
[17] Nitish Joshi, Abulhair Saparov, Yixin Wang, and He He. Lims are prone to fallacies in causal inference. arXiv preprint arXiv:2406, 12158, 2024.
[18] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint NXXv:2305.000550 2023.
[19] Brenden M. Lake and Marco Baroni. Human-like systematic generalization through meta-learning neural network. Nature, 623((985):115--2 November 2023. ISSN 0028-0836, 1476-4687. doi: 10. 11888/441556--22-066666
URL https: //www/wwwwwoo nature. com
[20] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017. Publisher: Cambridge University Press.
[21] Andrew Lampinen, Stephanie Chan, Ishita Dasgupta, Andrew Nam, and Jane Wang. Passive learning of active causal strategies in agents and language models. Advances in Neural Information Processing Systems, 36: 12831199, 2023.
[22] Belinda Z Li. Maxwell Nye, and Jacob Andreas. Implicit Representations of Meaning in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1813-1827, 2021.
[23] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenherg. Emergent world representations: Exploring a sequence model trained on synthetic task, ICI.R, 2023. Publisher: ICLR.
[24] Zichao Li, Yanshuai Cao, and Jackie CK Cheung. Do LLMs build world representations? Probing through the lens of state abstraction. Advances in Neural Information Processing Systems, 37:98009-99800 2024.
[25] Ilya Loshchilov and Frank Hutter. Decoupled 17Xiv:1711.05101,22 2017.
weight decay regularization.
arXiv preprint
[26] Kevin Meng, David Bau, Alex Andonian. and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:1 7359-17372, 2022.
[27] Raphaël Millière and Cameron Buckner. Interventionist Methods for Interpreting Deep Neural Networks. In Gualtiero Piccinini, editor, Neurocognitive Foundations of Mind. Routledge.
[28] Melanie Mitchell and David C. Krakauer. The debate over understanding in Al's large language models. Proceedings of the National Academy of Sciences, 12e( March 2023. ISSN 0027-8424, URL htttttttp/:/paaat org/ httg//ooo/11... 1073/pma..... 22159071
1091-6490. doi: 10. 22007//pa...21590077777.
20
[29] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Grokking of Hierar- chical Structure in Vanilla Transformers, May 2023. URL http://aaxii..rrraaa//////.... 1874 arXiv:2305. 18741 (es).
[30] Neel Nanda and Joseph Bloom. Transformerl.ens, 2022. htt.hhtpa://// /tt.http:://gihuubbo nsOrg/Transf ormerLens
[31] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, October 2023. URL http://arxiv. httt:///axxv.orr/abbbst abs//301/2211.... 25217 arXiv:2301.022 [es].
[32] Neel Nanda, Andrew Lee. and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00%. 2023.
[33] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):000024 001, 2020.
12

---

[34] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in neural information processing systems, 33:857-869, 2020.
[35] Judea Pearl. Causality. Cambridge university press, 2009.
[36] Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. arXiv preprint
2018.
[37] Judea Pearl. Judea pearl, AI, and causality: What role do statisticians play? Interviewed by D. Mackenzie). Amstar News (AI Special Issue), 555:6-9, 2023.
[38] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[39] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, and others. Improving language understanding by generative pre-training. 2018. Publisher: San Francisco, CA, USA.
[40] Aditya Ramesh, Mikhail Paylov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation, February 2021. URL http://///////ii org/ abs/ arXiv:2102. 12092 [es].
02. 12092
[41] J. Brendan Ritehic, David Michael Kaplan, and Colin Klein. Decoding the Brain: Neural Representation and the Limits of Multivariate Pattern Analysis Cognitive Neuroscience. The British Journal for the Philosophy of Science, 70(2):81-607, June 2019. ISSN 0007-0882, 1464-3537. doi: 10. 00.1033/ URL BtRlhhtpp:// //www. journals........................ wwwicago........ooo aaau/dooi/1 1093/bjps/axxx023
[42] Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut. CRAB: Assessing the Strength of Causal Relationships Between Real-world Events, 2023. https
// //arxiv.. aerpii.orr//bbbu.e/ arg/abs/23111 2311. 04284
eprint: 231 1.04284.
[43] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5); 612-634, 2021. Publisher: IEEE.
[44] Steven Sloman and Steven A Sloman. Causal models: How people think about the world and its alternatives. Oxford University Press, 2009.
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of machine learning rexearch, 9(11), 2008.
[46] Kevin Xia and Elias Bareinboim. Neural Causal Abstractions. Proceedings of the AAAI Conference Artificial Intelligence. 38(18):20585 20595, March 2024. doi: 10.1609/aai..v www.aa......i........ 3.0044 URL https://oj/ aaai org/ index. http/AAAl/arttice///ieee/////
[47] Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expres- siveness, learnability, and inference, Advances in Neural Information Processing Syxtems, 34: 10823-108866 2021.
[48] Kevin Xia. Yushu Pan, and Elias Bareinboim. Neural causal models counterfactual identification and estimation. arXiv preprini arXiv:2210.00035 2022.
[49] Blaise Agūera y Arcas. Do large language models understand us? Daedalus, 151(2)118-117, 2022. Publisher: MIT Press One Rogers Street, Cambridge, MA 22142-1209, USA journals-nfoo j
[50] llker Yildirim and LA Paul. From task structures to world models: what LLMs know? Trends in Cognitive Sciences, 2024. Publisher: Elsevier.
[51] Matej 7ečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal, August 2023. http:p/: org/ ab 5/2308.1300 ar Xiv::2300.. 13067 [es].
[52] Cheng Zhang, Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, and others. Understanding causality with large language models: Feasibility and opportunities. arXiv preprint 2:554:2304.05524, 2023.
13

---

A
Code
and-inference inference--hhouughh-----
t-token-prediction
B
Compute Resources
Our model requires approximately 3GB VRAM and can be trained consumer- gradde hardware. Each epoch processes approximately 1.2 million examples in 10 minutes. We used one NVIDIA L40 GPU to train the final models within university cluster using PyTorch 2.3. Evaluations, probe training, and interventional analyses were performed using a desktop machine with NVIDIA GeForce RTX 2080 Ti (1OGB VRAM).
C Broader Impacts
Our work provides evidence that statistical prediction objectives can drive emer- gence of causal models in neural networks. This advances scientific understanding of the relationship between statistical learning and causal reasoning, and puts less weight on the hypothesis that LLMs merely "stochastic parrots" or "causal parrots" that can never, in principle, build causal abstrac- tions through pure prediction objectives. Our mechanistic interpretability approach could inform future methods for auditing whether deployed Al systems have learned appropriate causal models for their domains.
Positive impacts:
Potential risks: Our results demonstrate an existence proof in controlled setting (linear Gaussian SCMs with artificial language), which should not be misinterpreted as evidence that current LLMs have robust causal reasoning across all real-world domains. Misunderstanding our findings general guarantee could lead to overconfidence in deploying LLMs for high-stakes applications (medical diagnosis, policy decisions) where erroneous causal inferences could cause significant harm. Extensive domain-specific validation remains necessary before deploying Al systems that make causal judgments.
D Methods Appendix
D.1 Analytic counterfactual inference in linear Gaussian SCMs
We implement analytic counterfactual inference for linear Gaussian SCMs, computing the quantity P(V m. Y Yo where Y Yobs observed values in the factual world, and Fim represents the intervention do X Fin)) the counterfactual world. Counterfactual inference involves three steps
35
1. Abduction finding the posterior over the background variables given observed values p(U|Y Yohs).
2. Action modifying the SCM according to the intervention; in our case, this corresponds to setting subset of the endogenous variables constant values in the structural equations
X ==
Zint and setting incoming weights to zero for intervened variables w - Wa..
3. Prediction given the posterior over background variables and the modified SCM, computing the counterfactual distribution over the endogenous variables V.
To analytically compute answers in steps (1) and (3) for a Gaussian linear SCM, it is useful to define the "total effects" matrix Let Tj: denote the total effect of the background variable U, on the endogenous variable V,, where i, E {1,...,m, .m) and m is the number of variables (in our case, m 4). The entries of the total effects matrix T are defined recursively as follows:
1. Base Cases: For each target variable V, where j = 1,...., m.
Ti = 1
(Direct effect U, Vý).
Tj 0 for all i > j (No effect from later noise variables).
14

---

2. Recursive Step: For each target variable Vj and each noise source U, where i <
T3
Tki Wkj
Note that T is a lower triangular matrix with ones on its diagonal. Let b b= [W11. W22, Wa3, Waal denote the vector of bias terms. We can then use T and b to define the structural equations in matrix form:
TU +
Let Yobs E R" denote the vector of n observed variable values, and let Tohs S (l...., mj be the set of indices of observed variables.
The observation matrix H R=x= is formed by selecting the rows of the total effects matrix T corresponding to the observed variables;
Tobss :]
This gives us the linear relationship Yobs = HU for posterior inference. Note that bias terms here are treated as implicit in the observations rather than explicitly subtracted, which is slight simplification.
The posterior over the background variables given the observations p(U yonbs) N(U pooa, Epoa can be computed from the prior mean Mgriior and covariance Eprior using standard Gaussian condition- ing:
K E pricr H² (HH (H H")
(9)
Mpss Pprior Post
K (yoons
H peproo)
(10)
Σ prior
K KHEprion
(11)
where K' is the Kalman gain matrix.
Let the counterfactual mean Muuuteer and covariance matrix Ecounter parameterize the counterfactual distribution P( P(Va.. |Y = Yobs) = N(V Meounter: Excunner)
To compute the counterfactual mean Menutter compute each variable in topological order as: variable j is intervened upon
(Zint) (Hupoaa + Wis + Eies Wiy (Mooonter)
(Jeconner)) =
(12)
otherwise
To compute the covariance matrix Ecounter we first construct modified posterior covariance matrix Epost by setting all rows and columns corresponding intervened variables to zero. We also compute the total effects matrix Tri using the modified weight matrix Wa.. (with incoming edges to intervened variables removed). Then:
Σ Exumter T
Fx post T
(13)
For DATA strings, take one sample from the counterfactual distribution. For INFERENCE strings, provide counterfactual means and standard deviations (square root of Ecouner diagonal values).
D.2 Motivating variable names across SCMs for mechanistic interpretability analyses
We present behavioral results for both variable naming schemes: shuffled 4.1 and fixed E.3 We find that the model passes our generalization challenge both cases. However, we found that the probe decoding accuracy (mapping from activations to the SCM weights) generally lower for the shuffled naming scheme, so we focused the mechanistic interpretability analyses on the case when variable names fixed.
One potential reason why decoding accuracy is lower when variable names are shuffled is that there a fundamental inherent ambiguity in how variables get mapped to the model's internal representation. Even if the model maintains consistent ordered internal representation (e.g. four representational "slots" for the four variables), for certain SCMs there are multiple equivalent ways to assign variables
15

---

to these slots. This poses challenge for our simplistic probes that assumee a fixed canonical mapping between variables and internal representational positions.
Consider an SCM that has no effects between variables, i.e. all the weights between variables are zero. For such an SCM, there are 4! = 24 equivalent ways assign the four variables to the four internal representational slots. Since there are no causal dependencies, any assignment produces identical behavior. When variable names are shuffled randomly across SCMs, least for certain variables within certain SCMs, the training data does not determine unique assignment-the model free choose any of the equally valid internal assignments. Our probes, trained assuming fixed mapping, struggle to decode weights consistently across these varying assignments.
In contrast, when variable names are a fixed across all SCMs, this ambiguity is resolved- the model can use the variable names themselves as anchors to establish a consistent mapping. For instance, the model can learn to always represent the variable named V1 in the first internal slot, V2 in the second slot, and so on, regardless the causal structure. This provides a canonical mapping that our probes can reliably decode.
Future work could consider more sophisticated probes (e.g. transformer-based) that can be conditioned on variable names without assuming any fixed mapping. For instance, query for such probe could take the form lactivations [Vi] [Vj], where the target is the SCM weight (effect) between vari- ables named Vi and Vj. Bias weights could be similarly queried by [activations] [vi] [Vi].
E
Results Appendix
E.1
Details on absolute error calculation
We chose mean absolute error (MAB) to measure model accuracy mostly for interpretability and robustness reasons. Unlike mean squared error (MSE), MAE is in the same units as the values being predicted and treats all prediction errors proportionally. MAE is less sensitive to occasional large outliers than MSE or RMSE (which heavily penalize large errors due to squaring), making it a more robust metric when occasional large deviations occur in the predictions.
To calculate absolute error when predicting DATA samples, we prompt the model with:
DATA [SCM index] [interventions]
[query variable] (+ [sample prediction])
and convert the highest probability next token to numerical value. We then calculate MAEdata = Ei= [r; 2,), where x; is the ground-truth sampled value for example i, 3, is the model's predicted value, and n is the total number of evaluation examples.
To calculate absolute error for INFERENCE mean and standard deviation, we prompt the model with:
INFERENCE [SCM index] [observations & interventions] [query variable] [std prediction]
(+ [mean prediction]
and auto-regressively predict two tokens. We interpret the first as the counterfactual mean prediction and the second the standard deviation prediction. We convert both tokens to numerical values and calculate MAE using the same formula, where ground truth values are the analytically derived counterfactual means and standard deviations.
Note that optimal absolute error is non-zero for two reasons. First, DATA strings provide noisy samples from interventional distributions, which cannot be predicted perfectly even with perfect knowledge of the SCM. Second, both observation//ineeeventios in the query and predictions are encoded with one decimal point precision in the text. We compute the optimal prediction by performing exact causal inference on the encoded (one decimal point) query values, encoding the output, and comparing to the ground truth full precision values (the actual sample for DATA strings; the analytically computed mean/std for INFERENCE strings).
Note that optimal absolute error is non zonozero for two reasons. First, DATA strings provide noisy samples of interventional distributions and is obviously impossible to predict those perfectly. Second, for both DATA and INFERENCE strings, we encode tokens with one decimal point precision. We take that into account to compute the optimal prediction: we calculate the analytical estimate given the one decimal point precision values actually encoded in the string and compare that to the ground truth.
16

---

In Fig. 5 we present model evaluations at every 10 epochs. For each evaluated epoch and metric we sample 1,000 Dinin and 1,000 Diea queries (where a single query a combination of an SCM index, set of interventions, and a set of observations for INFERENCE strings). For the last epoch (Fig. @ we sample 10,000 Dirain and 10,000 Dies queries to obtain more precise estimates. For each query, we run the model with four different queried variables. We exclude intervened queried variables from the analysis since the answer is trivial-the model can achieve perfect accuracy by simply copying the intervened value from the query and predicting zero for standard deviation. Since queries contain varying numbers of interventions (0-2), excluding intervened variables leaves approximately 2.9 variables per query average. This results in around 5,800 evaluations for each epoch (approximately 58,000 for the final epoch).
E.2 Behavioral results hold across instances
As robustness check, we trained two additional model instances (Fig, shows all three instances for comparison). All instances successfully generalize counterfactual queries about Diest SCMs, reaching near-optimal performance despite somewhat idiosyncratic training trajectories.
Loss
DATA
INFERENCE
INFERENCE
sample prediction
prediction
prodiction RaIVE Baseline
rave baseline
2.1
1.5
2.0
absolute error
abselute error
abselute error
Instance
2.0
1.0
1.5
lo5s
1.9
0.05
0.5
0.5
optimal p perforaacce
optimal performancc
вpGInaT pertrimance
1000
1000
1000
1000
epoch
epoch
epoch
epoch
TIVE BaSeliNE
NaIE Baseene
2.1
1.5
abcolute error
havve baséline
absolute enror
0.10
Instance
loss
2.0
1.0
absolute
1.5
0.05
0.5
0.5
1.8
optimal perforaance
optima performance
optimal performance 1000
1000
1000
1000
epach
epoch
epoch
opach
nave BaSaTIA
PSIVE B3SGIIR
2.1
1.5
2.0
absolute errar
absolute error
absolute error
Instance
1.0
1.5
l055
1.9
0.05
0.5
0.5
aptimal perfomannce
1.8
optimaa perforancc
optimal performance
1000
1000
1000
1000
epoch
epoch
epoch
epoch
SCM set
Dtrain
tast
DATA and INFERENCE strings
DATA strings only
Figure 10: All three model instances pass the generalization challenge, reaching near-optimal counterfactual inference performance Diea SCMs (instance from main text shown alongside instances 2 and 3 for comparison). Same metrics as in Fig.
E.3 Behavioral results hold when variable names are fixed across SCMs
Our behavioral results generalize to the somewhat simpler case when variable names are fixed across SCMs, which removes the variable-to-slot mapping ambiguity discussed in Appendix (Fig. [T).
Note that it takes less than 300 epochs to converge near-optimal performance in this case.
17

---

Loss
DATA sample prediction
INFERENCE mean prediction
INFERENCE std prediction
2.0
naive baselin
naive baseline
2.0
1.9
haive BESEINE
absolute emor
absolute eroer
0.10
Instance
169
1.8-
absolute
1.0
0.05
1.7
0.5
spumat performanice
1.6-
0.0
0.0
öptimmat performance
0.00
optnmal perio riancc
200
200
200
200
epoch
epoch
epoch
2.1
1.5
2.0
WaIVE DaSEIIE
naive baseline
absolute emor
haive baSEriE
absolute emor
absolute onor
0.10
Instance
2.0
855
1.9
1.0
0.05
aptimat performance
1.8
0.0
0.0
optimal performance
0.00
optimal performance
200
200
200
200
epoch
epoch
epoch
2.1
naive BASVIIN
naive baseline
1.5
2.0
absolute enor
(aave DaSERIT
absolute enor
absolute enor
0.10
Instance
1055
1.0
1.9
1.0
0.05
optimal parformance
1.8
0.0
0.0
optimal performanice
0.00
optimal perio rance 200
200
200
200
epach
epoch
epoch
epach
SCM
set
Drain
Dest
DATA INFERENCE
only
Figure 11: Training trajectories for the fixed variable naming scheme. The model converges to near-optimal performance faster than in the shuffled case. Same metrics as in Fig. 5
E.4 Probe training and probe accuracy on Probe SCM set
We probe the residual stream pre-activations in the transformer model. So layer activations correspond token embeddings with positional embeddings, layer activations 22 correspond to the output of layer etc.
We train both linear and MLP probes using AdamW optimizer 25 with batch size of 128, learning rate 0,001, eps 10 8 betas [0.9. 0.99], and weight decay 0.0. Due to different convergence rates, linear probes were trained for 20 epochs, and MLP probes were trained for 40 epochs. We did not observe any signs of overfitting for either of the probes based on the validation set performance.
We trained on prrobe train and tested on Diest SCM sets (see Fig. 7 In Fig. 12 we provide probe decoding accuracy the Drrohe set. This is easier generalization for the probes than the case in the main text since validation set examples come from the same Drrain set used to train the probes.
E.5 Intervention algorithm and MLP intervention example
For all intervention experiments we only consider the INFERENCE counterfactual mean prediction task since that the most challenging for the model (see Fig. 6
Our intervention setup (Algorithm I is directly inspired by Li et al. 23 The basic idea is to update the residual activations using gradient descent by minimizing the divergence between the weight predicted by the probe and the desired weight. While linear probes may allow simpler intervention schemes 32, we wanted to ensure fair comparison between linear and MLP probes.
For the linear intervention example (Fig. g), we intervened on layer with learning rate a 0.08. For the MLP example (Fig. 13), we intervened on layer with learning rate 0.08.
18

---

Linear probe
MLP probe
decoding accuracy Divee SCM set
decoding accuracy on Doge SCM set
12-
0,28
0.22
1855
0.35
1.0
2,51
2.4h
12-
8.32
0.28
0.35
5:51
8.32
0.35
5253
2
0.22
181
0:22
0:58
10
9:38
9.23
9.32
0.92
9.74
1.83
2.2
R38
2:46
8.88
1.30
9.28
6:44
9.22
0.38
1.24
226
9.22
264
0.31
0.55
0.56
2.28
9.23
8:2
1.63
2.28
2:48
0.28
0:25
2.22
0.59
2278
Layer
8J88
18.77
1..07
220
2232
1,88
3.32
1,35
1,12
1223
325
2250
1.59
2.45
2.35
0.52
9.12
8:28
0.52
9.58
E/1
224
6.59
2:12
9.38
0.98
9.53
5.45
9.92
6.75
5.54
2.52
5.69
2.45
0.75
0,455
6.15
0.53
5.29
2.25
2.15
0.40
0.55
0.50
0.48
0.45
0.25
8:31
0.25
2:31
2.21
2:78
0.39
0.32
0.33
9:38
8:35
2.20
0:29
0,36
0.22
9.55
0:31
0.23
221
2.23
2.38
2-
9.55
828
0.23
9:35
8,33
2.66
6:8
8.88
0.11
6.22
W31
W12
W11
Myy
19/3
W94
1964
W13
M/1
Wy1
Mir
19/3
W94
W4c
SCM
weight
SCM
weight
Figure 12: SCM weight decoding validation accuracy Deri set. Probe accuracy is slightly higher than for the Diest set presented main text. This expected because the probe does not need to generalize as font indicate 95% bootstrapped confidence interval for the mean across the SCM set.
prohe and D valid come from the same Durain SCM set. Values in smaller
Algorithm Gradient Based Residual Stream Intervention
Require: Model M, probe P, weight index (i,j). target weight w', layer L, learning rate a, steps b> Extract activations the last SCM index position >> Initialize Adam optimizer for Store initial probe predictions
1: xe|:, ()..e)))))
2: Adam a, a)
3: yprv P(a, E)
4: for step 1 to le do
5: y F(
Current probe predictions
6:
Lurge t- (rossEntropyyyss,. w')
Push target weight probe prediction to w' Push other probes maintain initial predictions
7:
Lathers + (.
8:
L Liargtt + Lothers
9:
Update using gradient descent L
10:
end for
x xel:.4.1 a
12: return Model output logits
E.6 Intervention score details
For intervention score calculation, only consider mean prediction in INFERENCE strings. Note also that we only consider cases where inst-intervention analytic mean is different from the pre- intervention analytic mean (E(v=".--='' + ElV D), i.e. we only consider variables for which the ground truth intervention "does something".
Query: INFERENCE A
OBS V1 -0.8 0BS V4 -0.3 DO V3 0.9 [Vs] [nean prediction]
V1 mean
V2 mean
V3 mean
V4 mean
token grobablig iilliyy
0.8
1
token
0
0.4
-1
0.2
0.0
0
10
20
30
0
10
20
30
0
10
20
30
0
10
20
30
gradient descent steps
analytical result
on residual stream activations
before intervention
intervention
(Wi2 = 0)
(Wip 1)
Figure 13: MLP probe intervention example (W12 + W12 1) for the same query in the main text (Fig.P) g).
19

---

Linear MLP probe
contrast between average intervention scores
A62
0.00
8.00
0.89
0.92
0.00
9.00
0.09
9.92
AR8
9,90
9,99
9.89
RA9
9
9.00
9.00
9.9g
9.98
RAR
2
2.00
9/09
9.92
202
9
0,00
9.00
9.00
0.0g
9.02
Layer
p.05
:9401
9.90
2.40
9.9
0,006
.9.03
0.04
0.00
-0.03
.0.04
-0.03
9.50
9.00
0.01
0.00
-9.02
2.0
0.03
9.93
19,0%
0.03
9:00
8,82
0.03
.000
9.95
9,92
9.69
9.91
9:90
0.00
0.00
-0.06
0.95
.0.03
-0.93
0.92
-9.02
0.00
0.19
0.91
-0.03
-3.08
0.05
0.50
-0.03
0.04
:0.05
9.99
W23
W933
Wha
Wes
SCM weight
Figure 14: Intervention score contrast between linear (blue) and MLP probes (red) (intervention score for MLP probe subtracted from the score for linear probe). While linear probes have lower decoding accuracy (Fig. and [2). MLP probes do strictly dominate them in terms of intervention scores. Values in smaller font indicate 95% bootstrapped confidence interval across queries, queried variables V; and weight changes Waj + w/y.
We set normalization factor 3 = 0.89 in eq. 6 by computing the average max probability the model assigns when predicting the mean.
We first generate set of 200 queries that are fixed across all layers and SCM weights. We then evaluate intervention scores for each query and for each layer-SCM weight combination by changing the weight to all possible values Wij - W/y E {-1, O, 1 with number of steps k = 30 and learning rate a = 0.08. We found these values result in generally good interventions qualitatively and we found quantitatively that intervention scores were quite robust to, say, changing number of steps to k- 20. However, in future work we should consider much more extensive intervention hyperparameter sweep. For instance, it may be that certain layer-SCM weight combinations require completely different number steps or learning rate a.
F Miscellaneous
F.1
Visualizing trained model embeddings
Embedding weights (PCA)
Embedding weights (t-SNE)
2.1
5.4
022
Dimennionn
Dimensiio
2.3
-1.
-0.
-6.4
-62
60
0'2
0.6
0.6
12
-20
-10
Dimension
Dimensian
1
Figure 15: We visualize the trained transformer token embedding weights using PCA and t-SNE 45 (instance 1, shuffled variable names version). The model seems to learn the number line (numerical tokens color-coded based on their absolute value). The model keeps negative and positive versions of the same absolute value nearby, possibly because the prediction task requires sign flipping (when parent value is negative and weight -1).
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim three main things: (1) the model trained for next-token predictino can generalize to counterfactual queries for those SCMs that only had interventional training data strings (demonstrated in section 4.1p: (2) the SCM weights
be decoded from
model's residual stream activations (section 4.2); (3) the SCM weights can be manipulated
with predictable effects model behavior (sections 4.3 and 4.4). These results support
the conclusion that neural networks can discover and learn to use causal models through next-token prediction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See limitations in the Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the technical aspects of the experiments sufficient detail and we provide code for reproducibility (Appendix A).. One could train the models and probes from scratch or use our pre-trained model and probe instances run the analysis scripts to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is available (appendix A The code allows to reproduce the plots, and train the model/probes from scratch.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/pu
Flic/guides/Con http://odSSmmmiiinneeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ inipss.ccppublic///
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details on model architecture and training in section 3.4 details on probing in section 4.2 and Appendix E.4 and details on interventional analyses in sections
4.3
Appendices E.5 E.6
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide bootstrapped confidence intervals for all quantities in the experi- ments and explain which factors of variability the intervals are capturing.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See "Compute Resources" (Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :
Answer: [Yes]
Justification: Our work conforms to every aspect of NeurlPS Code of Ethics. We do not anticipate any harmful consequences of our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?

---

Answer: [Yes]
Justification: See "Broader Impacts" (Appendix C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The material in this paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets g.g code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators of assets are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithoom wwwde.. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code includes a README file with instructions and documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Research did not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Research did not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for important or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should not be described.
27


Reviews:
[{"review_id": "PVOIpcz1Af", "reviewer": "Reviewer_nnXY", "summary": "The paper conducts an empirical study designed to test and inspect the causal inference abilities of LLMs by training a GPT style model from scratch via next token prediction on interventional and counterfactual data. \n\n### Study setup\nThe paper considers the class of Markovian linear Gaussian structural causal models (SCMs) over 4 variables *with a fixed causal order* (important, see below); the intercepts and coefficients are sampled from {-1,0,1}. This results in 3^10  distinct causal models of which 1000 are used for testing and the rest for training. The LLM is trained on interventional data from both training and test SCMs, as well as on counterfactual data from the training SCMs, and evaluated on counterfactual predictions on the test SCMs. The data is simulated from the respective SCM and then encoded as strings, containing a dummy encoding of the SCM ID (random characters), of which variables are intervened upon or observed (DO vs OBS), and whether the input is interventional data or a counterfactual inference example (DATA vs INFERENCE). \n\n### Main findings\nThrough various experiments, it is shown that the trained LLM:\n- reaches near optimal performance for counterfactual inference on the test SCMs (given only interventional test data, and counterfactual data from the training SCMs) for most combinations of number of intervened and observed variables (Figs. 5 & 6) \n- learns an internal representation in its later layers from which the SCM parameters, a ternary 10-dim. vector in this case, can be decoded with medium to high accuracy using linear and MLP probes (Fig. 7)\n- appears to learn an algorithm for counterfactual inference (within the considered model class) in the sense that manipulating the model's internal representation to change some weights in the SCM leads to changed results that agree with what would be expected analytically (Figs. 8 & 9)\n- is only controllable up to (and incl.) its sixth layer after which controllability is lost (despite still containing information about the SCM parameters (Fig. 10, cf. Fig. 7)", "questions": "- Why did you decide to fix the causal order (effectively removing the causal discovery aspect from the study)? Can you rerun the experiments (or a smaller version thereof, e.g., with only 3 variables) for a scenario in which the causal order is randomised? (e.g., you could randomly permute the label of variables separately for each SCM so that it is also possible that $V_2$ causes $V_1$) Without this (I realise this may not be feasible within the rebuttal period), the main claims of the paper should be adjusted accordingly and a clarification added. \n- For the current design (with fixed causal order), I hypothesise that even observational data should be enough (since no per-SCM causal graph needs to be learnt), provided that counterfactual inference statements are still provided. It could be interesting to try how the model performs when only given counterfactual inference queries (if sufficiently many queries are given, it should be possible to learn the path coefficients) or when additionally given observational instead of interventional data. \n- [minor] in Fig. 3 (top), why does the sample from the interventional distribution contain two different values for $V_3$? (V3 -0.9, V3 -1.5) Is this a typo and the first one should be V2? (else, please explain what is going on here)", "limitations": "What I view as the main methodological limitation of the work---using the same fixed causal order among all training and test SCMs (see above for details)---is not discussed at all. In light of this, I do not think that the study provides convincing evidence that LLMs can reliably perform causal discovery (as may be inferred from the title), beyond learning an algorithm for inference within the narrow model class considered (that much is acknowledged by the authors). Readers should not conclude that LLMs outputs necessarily support causal claims, which may lead to false overconfidence in using them for high-stakes causal inference tasks. I think adding a short discussion to clarify this would be a useful addition.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- the paper is very well written and easy to follow\n- the study is (for the most part---see below) well-designed\n- the experiments, which are carried out, are sensible, extremely well-described and augmented with very helpful figures\n- apart from the main limitation listed below, this study was a pleasure to read and is a prime example of good empirical research\n\n### Weaknesses\n- Unfortunately, it is a main flaw/limitation of the study design that the causal structure (i.e., the causal order) is fixed across all 59k+ SCMs. As a result, it is sufficient to memorise the correct (partial) causal order  $V_1 \\prec V_2 \\prec V_3 \\prec V_4$ globally. That is, the LLM does not have to learn how to perform causal discovery (rendering the title highly misleading), but only needs to solve the simpler, purely statistical problem of learning the partial correlations among variables within each SCM. In other words, **what the study in its current form demonstrates (and does so very well) is the following: given a *known/fixed causal order*, an LLM trained via next token prediction can learn the path coefficients within each SCM, and, given sufficiently many examples, can learn the abduction-action-prediction algorithm for counterfactual inference.** Hence, either the study design needs to be updated or the claims (and title) of the paper adjusted to reflect this.\n- [more minor] Pearl's claims in the introduction and their relation to the present manuscript appear slightly misleading or inaccurately portrayed. As far as I know, his claims are about a system trained on purely observational ($\\mathcal{L}_1$) data. While I think the point that natural language often conveys interventional or counterfactual knowledge is a good and valid one, this information is typically implicit in most datasets. In contrast, the interventional nature is made very explicit in the constructed dataset via the DO/OBS distinction. I don't think the data generating process needs to modified (beyond also randomizing the causal order as explained above), but more work is needed to convey this mismatch/these nuances in the Introduction and Discussion.", "was_revised": true, "final_justification": "After extensive discussion with the authors, my main concerns from the initial review (fixed topological) have been adequately addressed by the new set of experiment and results.\n\nThe other contentious issue concerns the positioning of the authors' claims w.r.t. those of Pearl. I think that with the proposed adjustments, this has been clarified. (The positions of some other reviewers on this matter appear a bit too extreme, and slightly oversimplified to me.) \n\nI believe that this paper (in its revised form) will be a valuable addition to the literature on the causal capabilities of LLMs. \n\nMy only worry is that the changes resulting from the rebuttal phase are rather major and may benefit from another review cycle.", "rebuttal": "We thank the reviewer for the close reading and thoughtful critique! Their feedback should improve the quality of the paper.\n\n# Fixed global topological variable order ($V_1 \\prec V_2 \\prec V_3 \\prec V_4$)\n\n> Unfortunately, it is a main flaw/limitation of the study design that the causal structure (i.e., the causal order) is fixed across all 59k+ SCMs. As a result, it is sufficient to memorise the correct (partial) causal order globally. That is, the LLM does not have to learn how to perform causal discovery (rendering the title highly misleading), but only needs to solve the simpler, purely statistical problem of learning the partial correlations among variables within each SCM. […] **what the study in its current form demonstrates (and does so very well) is the following: given a *known/fixed causal order*, an LLM trained via next token prediction can learn the path coefficients within each SCM, and, given sufficiently many examples, can learn the abduction-action-prediction algorithm for counterfactual inference.**\n> \n\n## Model passes generalization challenge without fixed topological order\n\nTo directly address the reviewer’s concerns, we demonstrate that our approach works without fixed variable ordering:\n\n- We **retrained the model without fixed topological variable order** using a smaller setup (due to time constraints):\n    - 3 variable linear Gaussian SCMs\n    - $w_{ij} \\in \\\\{-2, -1, 0, +1, +2\\\\}$ (5 possible weight values)\n    - $|D_{train}| = 5^6 -|D_{test}|= 15625 - 200$  (counterfactual queries + answers)\n    - $|D_{test}| = 200$ (interventional data only)\n    - we randomly permute the variable names within each SCM\n- **The model passes the generalization challenge**. It answers counterfactual queries for SCMs in $D_{test}$ near-optimally, despite only seeing interventional data about those SCMs.\n- We trained a model with fixed order (no permutation) on the exact same SCM set. It converged faster to near-optimal performance (~200 epoch vs. ~350), indicating that the problem is easier with a fixed variable order. But the fixed order is not essential for the model to discover the SCMs.\n\n## Why we chose a fixed topological order\n\nWe initially trained models without a fixed topological order, but then pivoted to a fixed order ($V_1 \\prec V_2 \\prec V_3 \\prec V_4$) because of what could be called an **“internal representation equivalency problem”**:\n\n- Without a fixed topological variable order, for a single ground truth SCM there may be multiple equivalent ways to represent the same SCM internally. For instance, consider a “no edge” SCM (4 variables with biases only) with $4! = 24$ ways to map variable names to internal representations of the variables (e.g. `V1` could be mapped to the first variable or the last variable in the internal representation).\n- Behaviorally, the model can learn any of these equivalent orderings and produce the correct output. However, it is **challenging to set up a probe**. The simple residual stream activation map to a fixed weight vector does not cut it because we do not know the internal order of the variables for a given SCM used by the model.\n- While it may be possible to train more sophisticated probes without a fixed order of the variables (e.g. by iteratively querying the value of the edge between every two variable names), for the sake of simplicity we decided to fix the order globally, which allowed us to train simple linear and MLP probes without worrying about this internal representation equivalency problem.\n\n## Why “causal discovery” in the title is warranted\n\nEven given a fixed topological variable order, the model still has to discover everything from scratch solely through next-token prediction, including:\n\n- The **global structure of the SCMs** (linear SCMs, Markovian normally distributed noise terms, 4 variables, global order $V_1 \\prec V_2 \\prec V_3 \\prec V_4$)\n- **Local** parameters of each SCM.\n- Arguably, the whole problem taken together goes beyond simple “parameter estimation”.\n\n# Pearl’s claims and “implicit interventions”\n\n> Pearl's claims in the introduction and their relation to the present manuscript appear slightly misleading or inaccurately portrayed. As far as I know, his claims are about a system trained on purely observational (L1) data. While I think the point that natural language often conveys interventional or counterfactual knowledge is a good and valid one, this information is typically implicit in most datasets. In contrast, the interventional nature is made very explicit in the constructed dataset via the DO/OBS distinction.\n> \n\n## Regarding Pearl’s claims\n\nThe reviewer raises important points about how we characterize Pearl's position. We think it is useful to distinguish between three claims:\n\n1. **Pearl’s Causal Hierarchy (PCH) and its uncontroversial implications**. Observational (L1) data underdetermines answers about interventions (L2) and counterfactuals (L3), without further assumptions. We agree with that.\n2. **Erroneous conclusion from PCH about deep neural networks.** In the “Theoretical impediments…” (2018) paper and “The Book of Why”, Pearl invokes PCH to argue that deep neural networks trained using statistical prediction objectives will *always* be limited to L1. We disagree with this position.\n3. **In so far as LLMs capture L2 and L3, they are “causal parrots”**. More recent perspectives like Zečević et al. (2023) try to square recent advances of LLM causal abilities while maintaining (2). They claim that LLMs can make correct L2 and L3 statements, but only because they are “causal parrots” that capture the “correlation of causal facts”. A similar position still seems to be maintained by Pearl (e.g. in an interview with Dana MacKenzie, Amstat News, September 2023), where he says that ChatGPT does not possess a causal model, even if it can answer some L2 and L3 queries correctly.\n\nWe hypothesized that LLMs may be building actual causal models and inference engines for the purpose of next-token prediction. Via our artificial language setup, carefully crafted generalization challenge and the mechanistic insights (probing and manipulating model’s learnt representations of the SCMs), we provide an **existence proof** **ruling out (2)** and **questioning** **versions of (3)** that are based solely on a priori reasons like PCH.\n\n## Implicit interventions\n\nInterventions are indeed more implicit in natural language (e.g. “Alice ventured to taste a magical potion…”) compared to our artificial language (”DO V1 2.3…”). The intuition behind our work is that LLMs may learn to detect (implicit) interventions to improve statistical next-token prediction objective (e.g. detecting that Alice drank a magical potion may help the model anticipate that something is about to happen to Alice, improving the ability to predict the next-token).\n\n# Regarding observational data and identifiability\n\n> For the current design (with fixed causal order), I hypothesise that even observational data should be enough (since no per-SCM causal graph needs to be learnt), provided that counterfactual inference statements are still provided. It could be interesting to try how the model performs when only given counterfactual inference queries (if sufficiently many queries are given, it should be possible to learn the path coefficients) or when additionally given observational instead of interventional data.\n> \n- The reviewer appears to be right that, given a fixed causal order and given the particular structure of our SCMs (fixed noise variances, linear Gaussian, etc.), observational data *theoretically* should be enough to identify the parameters. It could be interesting to see whether our model could uncover the SCM structure from observational data only.\n    - However, it may be extremely difficult to identify certain structures in practice. For instance, when paths cancel out ($V_1 \\rightarrow_{+1} V_2 \\rightarrow_{+1} V_3, V_1 \\rightarrow_{-1} V_3$), the only signature in the observational data would be the variance/covariance structure of the variables, which would require precise estimation from finite samples.\n    - Moreover, in the rerun version without a fixed topological order (see above), SCMs are no longer identifiable by observational data only.\n- Counterfactual inference queries + answers will be enough to learn the coefficients because of Pearl’s Causal Hierarchy (PCH). Since counterfactuals are at the top of the hierarchy (L3), that information determines the other levels: interventional (L2) and observational (L1).\n    - While it may be interesting to train on counterfactual inference queries only, we arrived at this particular setup to test the generalization capabilities of the model in the more challenging direction (L2 → L3 of PCH).\n\n# Typo in Fig 3.\n\n> [minor] in Fig. 3 (top), why does the sample from the interventional distribution contain two different values for ? (V3 -0.9, V3 -1.5) Is this a typo and the first one should be V2? (else, please explain what is going on here)\n> \n\nThank you for catching this typo!\n\n# Cautioning against overconfidence\n\n> Readers should not conclude that LLMs outputs necessarily support causal claims, which may lead to false overconfidence in using them for high-stakes causal inference tasks. I think adding a short discussion to clarify this would be a useful addition.\n> \n\nWe see our work as an existence proof against Pearl’s strong claim that it is *impossible* for DNNs trained to predict passive streams of data to build causal models or learn causal reasoning—not that LLMs are good causal reasoners. \n\nWe thank the reviewer for raising these points. We will **(1) include the motivation for the fixed topological variable order**, **(2) incorporate the results without a fixed order**, **(3) clarify our characterization of Pearl's claims and the implicit nature of interventions in natural language**, and **(4) add discussion cautioning against overconfidence in LLM causal abilities based on our results.**"}, {"review_id": "bljOnt50D0", "reviewer": "Reviewer_Y1qU", "summary": "The paper explores whether transformers trained for next-token prediction can discover and infer causal relationships within a constrained class of linear Gaussian structural causal models (SCMs). The authors demonstrate that the model generalizes to counterfactual queries for SCMs it only encountered through interventional data strings, suggesting it learns a causal inference engine. They also decode SCM weights from the model's residual stream activations and manipulate these representations to influence predictions. However, the approach lacks theoretical guarantees for consistency in causal discovery/inference, and its reliance on SCM indices limits generalizability to unseen SCMs. Additionally, the experiments are not benchmarked against established causal discovery algorithms like PC or FGES, raising questions about comparative performance.", "questions": "1. How does the model's causal discovery accuracy compare to traditional constraint-based (e.g., PC) or score-based (e.g., FGES) methods on the same SCM class?  \n2. Can the approach scale to more complex, non-linear SCMs, and how does its performance degrade compared to specialized causal discovery tools?  \n3. Does the method outperform existing algorithms in settings where only interventional data (no counterfactual examples) are available during training?  \n4. What are the computational and sample complexity trade-offs between this approach and classical causal discovery methods?", "limitations": "The requirements of SCM index may restrict the proposed approach to be really useful.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths  \n\n1. **Innovative Approach**:  \n   - The paper presents a novel integration of causal inference with next-token prediction, demonstrating that transformers can implicitly learn and utilize causal structures from text-like data. \n   - The use of a constrained class of linear Gaussian SCMs provides a clean experimental setup, allowing the authors to rigorously test the model's ability to generalize beyond memorization.  \n\n2. **Probing and Interventions**:  \n   - The authors employ linear probing and multi-layer perceptron (MLP) probes to decode SCM weights from the model's residual stream activations. This provides partial evidence that the model internally represents causal structures, supporting the claim that it performs causal discovery.  \n   - Gradient-based interventions on the residual stream activations demonstrate that the model's predictions can be controllably altered by modifying its implicit SCM representation. \n\n3. **Generalization Challenge**:  \n   - The experimental design includes a \"generalization challenge\" where the model must answer counterfactual queries for SCMs it has only seen through interventional data. This effectively tests whether the model has learned a generalizable causal inference engine rather than simply memorizing training examples.  \n\n4. **Behavioral and Mechanistic Analysis**:  \n   - The paper combines behavioral metrics (e.g., prediction accuracy) with mechanistic analyses (e.g., probing and interventions), providing a comprehensive evaluation of the model's causal capabilities. This dual approach strengthens the validity of the conclusions.  \n\n---\n\n### Weaknesses  \n\n1. **Lack of Consistency Guarantees**:  \n   - The paper does not provide theoretical guarantees that the model's causal discovery process is consistent (i.e., that it converges to the true SCM as training data increases). Consistency is a fundamental requirement for reliable causal discovery, and its absence limits the method's applicability to real-world problems where correctness is critical.  \n   - Without such guarantees, it is unclear whether the model's success is due to the specific simplicity of the linear Gaussian SCM class or a more general capability.  \n\n2. **SCM Index Dependency**:  \n   - The model relies on predefined SCM indices to reference causal structures, which may act as a crutch, allowing the model to \"look up\" SCMs rather than infer them from raw data. This raises concerns about whether the approach can generalize to entirely new SCMs not seen during training.  \n   - In real-world settings, causal structures are not pre-indexed, so the model's ability to discover causal relationships from scratch remains unproven.  \n\n3. **Weak Experimental Comparisons**:  \n   - The paper does not compare the model's performance against established causal discovery algorithms (e.g., PC, FGES, LiNGAM) or state-of-the-art neural causal methods (e.g., DECI, DCDI). Such comparisons are necessary to assess whether the transformer-based approach offers any advantages in accuracy, scalability, or robustness.  \n   - The lack of benchmarks on standard causal discovery datasets (e.g., synthetic graphs, real-world cause-effect pairs) further limits the ability to evaluate the method's practical utility.  \n\n4. **Limited Scope of SCMs**:  \n   - The experiments are restricted to linear Gaussian SCMs with at most four variables. While this simplifies analysis, it leaves open whether the method can handle more complex, non-linear, or high-dimensional causal structures.  \n   - The reliance on analytical solutions for counterfactual queries (which may not exist for non-linear SCMs) suggests the approach may not easily generalize beyond the tested class.  \n\n5. **Ambiguity in Probe Interpretation**:  \n   - While probes can decode SCM weights, the paper acknowledges that decoding accuracy does not necessarily imply the model uses these representations for inference (Section 4.4). This weakens the claim that the model truly \"discovers\" causal structure, as opposed to merely encoding it incidentally.  \n   - The intervention experiments, while compelling, are limited to a small set of SCMs and weight modifications. Broader validation is needed to confirm that the model's behavior is systematically controlled by the manipulated representations.", "was_revised": true, "final_justification": "After rebuttal, I believe the experimental design of the paper is problemitic.", "rebuttal": "We thank the reviewer for their feedback. However, the review appears to misunderstand the fundamental goal of our paper. Our aim is not to propose a method that outperforms existing causal discovery algorithms. Rather, we explore a fundamental question: Can deep neural networks trained on passive data streams build causal models and learn causal inference engines?\n\nPearl and others have argued this should be theoretically impossible: that such networks will be limited to associations (L1) and cannot reach interventional (L2) or counterfactual (L3) reasoning. Through carefully crafted experiments, we provide an empirical existence proof that this is possible.\n\n# Weaknesses\n\n### **1. Lack of Consistency Guarantees**\n\n> The paper does not provide theoretical guarantees that the model's causal discovery process is consistent (i.e., that it converges to the true SCM as training data increases). Consistency is a fundamental requirement for reliable causal discovery, and its absence limits the method's applicability to real-world problems where correctness is critical.\n> \n\n> Without such guarantees, it is unclear whether the model's success is due to the specific simplicity of the linear Gaussian SCM class or a more general capability.\n> \n- Our primary goal is an **existence proof** against Pearl’s strong claim, not an improvement over existing methods. Theoretical convergence guarantees therefore lie beyond the scope of this paper.\n\n### 2. **SCM Index Dependency**\n\n> The model relies on predefined SCM indices to reference causal structures, which may act as a crutch, allowing the model to \"look up\" SCMs rather than infer them from raw data.\n> \n- SCM indices are random by design and have no inherent meaning at the beginning of training. The model then discovers the underlying SCM associated with a particular SCM index *from raw data* by predicting text strings. Any model needs some data about an SCM to generalize to it. Without any information, generalization is impossible.\n\n> In real-world settings, causal structures are not pre-indexed, so the model's ability to discover causal relationships from scratch remains unproven.\n> \n- In real-world settings, causal contexts are identified through variable names (e.g. “smoking”, “cancer”), domain descriptions, etc. Since variable names are the same within every SCM in our setup, our SCM indices serve this same function.\n\n### 3. **Weak Experimental Comparisons**\n\n> The paper does not compare the model's performance against established causal discovery algorithms (e.g., PC, FGES, LiNGAM) or state-of-the-art neural causal methods (e.g., DECI, DCDI). Such comparisons are necessary to assess whether the transformer-based approach offers any advantages in accuracy, scalability, or robustness.\n> \n\n> The lack of benchmarks on standard causal discovery datasets (e.g., synthetic graphs, real-world cause-effect pairs) further limits the ability to evaluate the method's practical utility.\n> \n- Our goal is **not** to introduce another causal‐discovery algorithm, but to answer a deeper question: Can a transformer, trained solely via next‑token prediction, both discover the causal models *and* build its own L2/L3 causal inference engine? Classical methods like PC and FGES are designed to recover DAG structure from numerical data. Neither employs a text‑prediction loss nor produces counterfactual answers. In fact:\n    - **structure‑only:** PC FGES only output a graph. They have no mechanism for computing “what if?” counterfactuals, so their counterfactual‑query accuracy is effectively 0%\n    - **orthogonal design:** These tools rely on independence tests or score maximization under explicit SCM assumptions, whereas our model learns everything end‑to‑end via language\n\n### **4. Limited Scope of SCMs**\n\n> The experiments are restricted to linear Gaussian SCMs with at most four variables. While this simplifies analysis, it leaves open whether the method can handle more complex, non-linear, or high-dimensional causal structures.\n> \n- In the discussion, we acknowledge the limited scope of the SCMs tested.\n- However, we would like to note that an existence proof requires just one example.\n- Moreover, our simplified setting was a strategic choice. Previous studies lacked full control of training data (unclear memorization vs. generalization) and did not probe internal representations (unclear if LLMs build causal models vs. parrot surface statistics). Our controlled setting with mechanistic analysis addresses both limitations.\n\n> The reliance on analytical solutions for counterfactual queries (which may not exist for non-linear SCMs) suggests the approach may not easily generalize beyond the tested class.\n> \n- The choice of linear SCMs was a methodological choice to use the analytical solution to efficiently generate training data.\n- We have already explored creating training data for non-linear SCMs by computing answers to counterfactual queries numerically using probabilistic programming (Pyro package). While it takes longer to generate such training data, it is definitely possible to apply our approach to non-linear SCMs.\n\n### **5. Ambiguity in Probe Interpretation**\n\n> While probes can decode SCM weights, the paper acknowledges that decoding accuracy does not necessarily imply the model uses these representations for inference (Section 4.4). This weakens the claim that the model truly \"discovers\" causal structure, as opposed to merely encoding it incidentally.\n> \n- This appears to be a misunderstanding of our analyses. Our successful gradient-based interventions (in sections 4.3 and 4.4) demonstrate that the model actually uses these representations functionally, not just incidentally. When we modify the decoded SCM weights, the model's outputs change predictably—this directly proves the representations are causally relevant to the model's reasoning process.\n\n> The intervention experiments, while compelling, are limited to a small set of SCMs and weight modifications. Broader validation is needed to confirm that the model's behavior is systematically controlled by the manipulated representations.\n> \n- We respectfully disagree with this assessment. To systematically measure whether we can control the representations, we develop a quantitative “intervention score”. We then measure this score across 200 randomly selected causal queries (with different SCMs from both $D_{train}$ and $D_{test}$) for all SCM weights (10) changing them to two possible other weight values (2) across all layers (12) of the model for both probe types (linear and MLP), resulting in 200 * 10 * 2 * 12 * 2 = 96,000 evaluations. So our interventional analyses are quite extensive, and we show that the model’s behavior is indeed systematically controlled by the manipulated representations in section 4.4.\n\n# Questions\n\n> 1. How does the model's causal discovery accuracy compare to traditional constraint-based (e.g., PC) or score-based (e.g., FGES) methods on the same SCM class?\n> \n\nOur model's near-optimal counterfactual performance on the generalization challenge demonstrates successful SCM discovery. In contrast, PC cannot handle interventional data and would struggle with identifiability in our linear Gaussian setting. FGES might recover the causal DAG but would require explicit priors about noise distributions and SCM structure. FGES also cannot perform counterfactual inference and would achieve 0% on our generalization challenge. Our approach uniquely learns both structure discovery and counterfactual reasoning from text, addressing Pearl's fundamental claims about neural network limitations rather than competing with dedicated causal discovery methods.\n\n> 2. Can the approach scale to more complex, non-linear SCMs, and how does its performance degrade compared to specialized causal discovery tools?\n> \n\nOur approach should scale to non-linear SCMs since the methodology (learning from text descriptions) does not depend on linearity assumptions. We have already explored generating training data for non-linear SCMs using numerical methods with probabilistic programming. While we focus on linear Gaussian SCMs here for analytical tractability and interpretability, the key advantage is that our approach learns both structure discovery and counterfactual inference without requiring explicit assumptions about functional forms, noise distributions, or causal mechanisms (unlike specialized tools that need such priors).\n\n> 3. Does the method outperform existing algorithms in settings where only interventional data (no counterfactual examples) are available during training?\n> \n\nWhile performance comparisons could be interesting, our work addresses a different question: Can transformers emergently learn causal reasoning from text? This is a fundamental capability question rather than a methods comparison.\n\n> 4. What are the computational and sample complexity trade-offs between this approach and classical causal discovery methods?\n> \n\nThese are excellent questions for future work. Here, our focus is on demonstrating that causal discovery and counterfactual inference engines can emerge through statistical prediction objectives, addressing Pearl's fundamental claims about neural network limitations.\n\n# Limitations\n\n> The requirements of SCM index may restrict the proposed approach to be really useful.\n> \n- The SCM index requirement appears to be a misunderstanding. SCM indices are arbitrary labels with no inherent meaning—the model must discover the underlying causal structure associated with each index from the raw data itself.\n- More broadly, this criticism misses the flexibility of our approach. In real-world settings, causal contexts are identified through various cues (variable names, domain descriptions, and so on). Our SCM indices serve the same function as these identifiers. The power of next-token prediction is precisely that it can learn from any text format, including natural language descriptions of causal relationships."}, {"review_id": "3BKIZaURLO", "reviewer": "Reviewer_m4HY", "summary": "The paper demonstrates that next-token prediction alone can induce genuine causal reasoning: the authors construct a synthetic language that interleaves observational samples, interventional statements and counterfactual queries drawn from thousands of four-variable linear-Gaussian structural causal models (SCMs), train a 12-layer GPT-style transformer solely to predict the next token in these sequences, and find that on held-out SCMs it answers unseen counterfactual questions with near-optimal accuracy—evidence that it has inferred the latent graph rather than memorised responses. Linear and MLP probes decode each SCM edge weight from the network’s residual-stream activations, and gradient-based edits to those activations flip the model’s counterfactual predictions exactly as the modified SCM would, revealing a manipulable internal causal representation . Layer-wise intervention analysis shows this representation persists deep into the model even when later layers stop consulting it, warning that decodability need not imply functional use.", "questions": "See my comments above.", "limitations": "See my comments above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is well-written and easy to follow.\n* Uncovering structures from LMs is not a new research topic. There are even harder tasks that are linguistically driven, e.g., uncovering syntactic trees from synthetic languages generated using PCFGs. However, it is still quite interesting to see that simple interventions could uncover an SCM's weights.\n* The setup is simple but clean. However, whether it is extensible to more real-world data remains a question.\n\n**Weaknesses**\n\nI think the experimental setup is solid. My main concern resides in the high-level goal of this work and how it relates to some other interpretability research in the field.\n\n* **Missing related works in uncovering trees or structural models from LMs.** There are existing works that come from the more classic NLP interpretability camp (see \\[1] and its cited works) which are trying to (1) uncover tree structures (i.e., syntactic ones) from LMs; (2) or even induce trees in LMs. I think it would be worth discussing the main differences between this work and them. In particular, focusing on what additional insights can be gained by using SCM-induced datasets from this paper would be interesting. At a high level, both lines of work are trying to interpret very similar things. In \\[1], a PCFG model is used to generate synthetic data that are then used to train LMs, and interpretability methods are used to assess whether we can uncover syntactic tree structures (i.e., a PCFG graph with edge weights of 1) from the LM.\n\n* **Needs to be better motivated.** Although it is interesting to see some causal structures emerge in the LMs for this very specific synthetic dataset, I think it would be beneficial to say more about the motivation behind this research in a precise way. For instance, what is the additional gain from your setup compared to previous ones (including the line of work I mentioned above)? With your methods, are you finding additional interpretability signals that inform us how transformers work beyond what people already know (e.g., transformers generalize, transformers learn structures of the dataset generation process)?\n\n[1] Murty et al., *Grokking of Hierarchical Structure in Vanilla Transformers*, [https://arxiv.org/pdf/2305.18741](https://arxiv.org/pdf/2305.18741)", "was_revised": true, "final_justification": "I have read the author response. I am leaning towards accepting this paper.", "rebuttal": "We thank the reviewer for constructive feedback and pointing us to some relevant literature.\n\n> **Missing related works in uncovering trees or structural models from LMs.** There are existing works that come from the more classic NLP interpretability camp (see [1] and its cited works) which are trying to (1) uncover tree structures (i.e., syntactic ones) from LMs; (2) or even induce trees in LMs. I think it would be worth discussing the main differences between this work and them. In particular, focusing on what additional insights can be gained by using SCM-induced datasets from this paper would be interesting. At a high level, both lines of work are trying to interpret very similar things. In [1], a PCFG model is used to generate synthetic data that are then used to train LMs, and interpretability methods are used to assess whether we can uncover syntactic tree structures (i.e., a PCFG graph with edge weights of 1) from the LM.\n> \n\n> **Needs to be better motivated.** Although it is interesting to see some causal structures emerge in the LMs for this very specific synthetic dataset, I think it would be beneficial to say more about the motivation behind this research in a precise way. For instance, what is the additional gain from your setup compared to previous ones (including the line of work I mentioned above)? With your methods, are you finding additional interpretability signals that inform us how transformers work beyond what people already know (e.g., transformers generalize, transformers learn structures of the dataset generation process)?\n> \n- The hierarchical generalization test in Murty et al. (2023) has important parallels to our generalization challenge, as both works test whether models learn hierarchical rules rather than surface patterns in their respective domains. This literature is relevant and we will include it in our related work section.\n- While Murty et al. focus on syntactic structure discovery, our work **addresses a fundamental question in causal inference**: whether neural networks can build causal models and perform counterfactual reasoning via statistical prediction objectives, directly challenging Pearl's theoretical claims about the limitations of deep neural networks.\n- **The key additional gain from our setup is providing an empirical existence proof against a foundational theoretical claim.** Pearl and others have argued it should be theoretically impossible for statistical learning systems to reach interventional (L2) and counterfactual (L3) reasoning. Our controlled SCM setup allows us to test this claim with ground truth causal structures and analytical solutions for validation.\n- The mechanistic analysis in our paper addresses a further critique from causality researchers.\n    - Pearl (Amstat News, Sept 2023) and Zečević et al. (2023) argue that, insofar as LLMs can answer L2/L3 queries, they do so by memorizing and “parroting” causal facts—based on their reading of Pearl’s Causal Hierarchy.\n    - Our mechanistic analysis directly refutes the assumption that statistical learning systems cannot build *true* causal models and reasoning engines through prediction objectives alone. Beyond behavioral generalization, we **decode** SCM weights from the residual stream and **intervene** on those representations. Targeted edits reliably flip counterfactual predictions in the same direction as the modified ground-truth SCM—demonstrating that the model has constructed an internal causal model that is interpretable and functionally used in inference."}, {"review_id": "mRKKFJmGtS", "reviewer": "Reviewer_d6ju", "summary": "This paper investigates whether a transformer model, trained exclusively on a next-token prediction objective, can learn to perform causal discovery and inference. The method involves training a GPT-style model on a synthetic dataset generated from a constrained class of linear Gaussian structural causal models (SCMs).\n\nThe workflow and evaluation are structured as follows:\n\n* Data Generation: The authors generate a large, finite set of unique SCMs. From these SCMs, they create text strings in a simple, artificial language that fall into two categories: DATA strings, which describe the results of noisy interventions on an SCM, and INFERENCE strings, which provide the answers to counterfactual queries about an SCM.\n\n* Generalization Challenge: To test for true causal understanding rather than memorization, the model is trained on a dataset where a specific subset of SCMs (D_test) is only ever described using DATA strings. The core of the evaluation is to assess whether the model can still perform correct counterfactual inference for this D_test set, despite never having seen an INFERENCE string for any of its \n\n* Evaluation & Probing: The primary evaluation metric is the model's accuracy on counterfactual queries for the held-out D_test set. The authors supplement this behavioral test with mechanistic interpretability techniques. They probe the model’s internal activations to determine if a representation of the underlying SCM can be decoded. They further test the functional relevance of this representation by directly intervening on the activations via gradient descent and observing whether this manipulation produces predictable changes in the model's output for causal queries.\n\nThe authors report that the model successfully generalizes to the D_test set, achieving near-optimal performance on counterfactual inference tasks. Their probing analysis reveals that the SCM's structure can be decoded from the model's residual stream, and their interventional experiments demonstrate that manipulating this internal representation causally influences the model's reasoning in a predictable manner. The results suggest that a standard statistical training objective can be sufficient for a neural network to discover and learn to use a causal model of its data-generating process.", "questions": "1.The study successfully demonstrates causal discovery in SCMs with four variables. How is the model's performance expected to scale as the number of variables (and thus the complexity of the causal graph) increases? Is there a theoretical or practical limit where the next-token prediction signal becomes insufficient?\n\n2.The findings rely on a simple, unambiguous artificial language. How robust do the authors expect these results to be if the language included more naturalistic complexities, such as ambiguity, paraphrasing, or implicit causal statements?\n\n3.The probing and intervention techniques are central to the paper's analysis. Do the authors believe these methods, particularly the gradient-based interventions, could be practically adapted to analyze causal representations in much larger, pre-trained foundation models?\n\n4.The ternary weight system makes faithfulness violations (e.g., pathways cancelling out) plausible. Could the authors comment on whether the model successfully learns the true SCM structure in such cases, or if it is misled by the resulting statistical independencies in the data?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n * Addresses a Foundational Question: The paper addresses the question of whether causal reasoning can emerge from statistical training objectives. Its approach, which uses a controlled environment to study a small transformer, offers a method for investigating mechanisms that may be relevant to larger models.\n\n* Targeted Experimental Design: The \"generalization challenge\" is designed to distinguish causal learning from pattern memorization. By evaluating the model on counterfactual queries for SCMs presented only through interventional data, the experimental setup directly tests the model's ability to generalize causal rules.\n\n* Insight into Internal Representations: The analysis of the model's internal states is a notable strength. Probing experiments indicate that the model's latent space contains a decodable representation of the true SCM. Furthermore, interventional experiments confirm that the model actively uses this internal representation to generate its causal inferences, providing evidence that the learned structure is functional.\n\n* High-Accuracy Results: The reported results show that the model achieves high accuracy on the generalization task, approaching the optimal performance bound. This result is presented as evidence that a next-token prediction objective can be sufficient for inducing causal inference capabilities in this specific setting.\n\n## Weaknesses\n\n* Highly Constrained Experimental Setting: The paper's claims are derived from a very restrictive setting involving linear Gaussian SCMs with only four variables and ternary weights (-1, 0, 1). While this simplicity is necessary for the detailed analysis performed, it raises significant questions about scalability and generalizability. It remains unclear whether the same mechanisms would hold for SCMs with more variables, continuous causal strengths, or non-linear relationships.\n\n* Use of a Simplified Artificial Language: The model is trained on a simple, unambiguous artificial language where causal relationships are explicitly described. Natural language, in contrast, is noisy, ambiguous, and context-dependent. The degree to which these findings depend on the clean, structured nature of the input data is a major open question that limits the direct applicability of the conclusions to real-world LLMs.\n\n* Limited Discussion of Implications for LLMs: The discussion section is brief and does not sufficiently bridge the gap between the findings in this toy model and the behavior of large-scale language models. While the paper serves as an interesting “existence proof”, it offers little speculation on how these mechanisms might manifest, or be identified, in models trained on vast, unstructured web-text. A more thorough discussion of these implications would strengthen the paper's contribution.", "was_revised": true, "final_justification": "My concerns about the current setting of the problem has been answered, by outlining how the work is in fact a theoretical counterexample. The discussion about how relevant this work is regarding LLMs will also be updated, which I trust will be a great addition to the paper.", "rebuttal": "We thank the reviewer for a constructive assessment of our work.\n\n# Weaknesses\n\n> **Highly Constrained Experimental Setting**: The paper's claims are derived from a very restrictive setting involving linear Gaussian SCMs with only four variables and ternary weights (-1, 0, 1). While this simplicity is necessary for the detailed analysis performed, it raises significant questions about scalability and generalizability. It remains unclear whether the same mechanisms would hold for SCMs with more variables, continuous causal strengths, or non-linear relationships.\n> \n- We acknowledge the constrained setting in the discussion. As pointed out by the reviewer, this simplicity was partly dictated by the detailed analysis performed.\n- Note that **Pearl's claim was *categorical and universal***: it is theoretically impossible for any deep neural network trained using statistical prediction objectives to ever reach interventional (L2) or counterfactual (L3) levels of the causal hierarchy. For such sweeping theoretical claims, **a single well-designed counterexample is sufficient to demonstrate that the claim is false**.\n- Pearl (interview in Amstat News, September 2023) and Zečević et al. (2023) have implied that L2 and L3 LLM behavioral competence is due to parroting memorized surface statistics, arguing that LLMs cannot possess an *actual* internal causal model (due to Pearl’s Causal Hierarchy). Our mechanistic analyses (probing and interventions) are particularly valuable to respond to such claims since we show, via a concrete counterexample, how internal causal models may be implemented inside the “black box” LLM.\n- While questions about scalability to more complex SCMs are interesting for future work, they do not diminish the theoretical significance of showing that such learning is possible in principle.\n\n> **Use of a Simplified Artificial Language**: The model is trained on a simple, unambiguous artificial language where causal relationships are explicitly described. Natural language, in contrast, is noisy, ambiguous, and context-dependent. The degree to which these findings depend on the clean, structured nature of the input data is a major open question that limits the direct applicability of the conclusions to real-world LLMs.\n> \n- We acknowledge that there are important differences between our artificial language and natural language. Natural language is indeed more ambiguous and complex.\n- However, we believe the artificial language setup serves our specific research goal: providing a controlled test of Pearl's theoretical claims about neural network limitations. This required a setting where we could have ground truth causal structures for validation, generate unlimited training data with analytical solutions, and perform precise mechanistic analysis of learned representations.\n- Most importantly, **causal relationships are not explicitly described** in our artificial language. The underlying SCMs appear only implicitly through noisy interventional samples and inference examples. The model must discover the hidden causal structure through next-token prediction, as it would need to in natural language.\n- While our artificial setup is simplified, we view this as a necessary first step for establishing that such learning is theoretically possible. Future work can explore how these findings extend to more complex, naturalistic settings.\n\n> **Limited Discussion of Implications for LLMs**: The discussion section is brief and does not sufficiently bridge the gap between the findings in this toy model and the behavior of large-scale language models. While the paper serves as an interesting “existence proof”, it offers little speculation on how these mechanisms might manifest, or be identified, in models trained on vast, unstructured web-text. A more thorough discussion of these implications would strengthen the paper's contribution.\n> \n- We agree with the reviewer that we should extend the discussion on the implications of our results for LLMs.\n- **How would these mechanisms manifest in LLMs?**\n    - Our results suggest that LLMs *may* be building internal causal models and inference engines to improve next-token prediction. However, we hypothesize that the causal models and engines of pre-trained LLMs are likely to be less formal and more intuitive—supporting everyday causal inferences present in natural language rather than formal causal reasoning (analogous to how intuitive physics differs from formal physics).\n- **How could causal models and inference engines be identified in LLMs?**\nSeveral approaches seem promising:\n    - **Behavioral approaches**. Designing evaluation methods and datasets to understand LLM causal reasoning (e.g. by asking LLMs to judge causal vignettes, like in Kıcıman et al. 2023). We cite many studies evaluating LLM causal abilities behaviorally in our related work section. While these studies show mixed results, there is no clear picture when and why LLMs succeed or fail in their causal inferences, and much remains to be done.\n    - **Mechanistic approaches**. To apply similar probing/interventional techniques as in our paper, we would need to train much more sophisticated decoders. One possibility: Train a decoder that maps internal model activations to natural language descriptions (trying to read the “mind” of the model). For instance, the decoded text may say something like “Alice drank the potion…”. We could then intervene on that internal representation using similar gradient based methods as presented in our paper, changing the output of the decoder to something else (”Alice **spilled** the potion”), and observing functional implications on model output.\n- We will expand the discussion on broader implications for causal reasoning in LLMs—and to clarify how our setup lays a foundation for extending mechanistic analysis to real-world models.\n\n# Questions\n\n> 1. The study successfully demonstrates causal discovery in SCMs with four variables. How is the model's performance expected to scale as the number of variables (and thus the complexity of the causal graph) increases? Is there a theoretical or practical limit where the next-token prediction signal becomes insufficient?\n> \n- A priori, there is no theoretical limit for the number of variables or complexity of the SCMs.\n- However, we expect there to be some practical performance limits when we increase the complexity of the SCM:\n    - **computational complexity**: the number of possible SCMs grows exponentially with the number of variables\n    - **sample complexity**: more interventional samples will be needed to identify the SCM, resulting in longer training runs\n    - **representational capacity**: larger SCMs will require higher dimensional representations\n- Nevertheless, multiple factors suggest reasonable scalability: real-world causal structures are often sparse, LLMs show reasonable scaling properties on other tasks, and next-token prediction objective would remain the same, regardless of the complexity of the SCM.\n\n> 2. The findings rely on a simple, unambiguous artificial language. How robust do the authors expect these results to be if the language included more naturalistic complexities, such as ambiguity, paraphrasing, or implicit causal statements?\n> \n- This is an important question for future work. We expect that the findings would be robust: LLMs have demonstrated remarkable robustness to ambiguity and paraphrasing across many domains.\n- Furthermore, our artificial language is already implicit in a crucial sense: the model is never explicitly provided the underlying SCM structures and has to uncover them through next-token prediction, similar to how it would need to extract causal relationships from natural language.\n- **The key insight from our work** is that the fundamental capability—learning causal models through statistical prediction—is possible. While natural language complexities would certainly make the task more challenging, they do not seem to pose fundamental theoretical barriers to this type of learning.\n\n> 3. The probing and intervention techniques are central to the paper's analysis. Do the authors believe these methods, particularly the gradient-based interventions, could be practically adapted to analyze causal representations in much larger, pre-trained foundation models?\n> \n- Yes, this is exactly what we would like to explore in future work!\n- See above for a description how the mechanistic gradient-based interventions may be applied to a pre-trained LLM.\n- We are also interested in applying similar techniques to video generative models (akin to Sora). We hypothesize that video generative models may learn simulators of the world that contain implicit causal structure. To explore this hypothesis, we could train decoders of physical objects and agents and apply similar intervention techniques to the ones presented in our paper. Another path is to train a video generative model ourselves using a game engine (with known ground truth causal structures) and see whether we can decode the causal structures and intervene on them with predictable effects on model’s video generation output.\n\n> 4. The ternary weight system makes faithfulness violations (e.g., pathways cancelling out) plausible. Could the authors comment on whether the model successfully learns the true SCM structure in such cases, or if it is misled by the resulting statistical independencies in the data?\n> \n- It is true that our particular setting presents faithfulness violations. Take the case when paths cancel out: $V_1 \\rightarrow_{+1} V_2 \\rightarrow_{+1} V_3, V_1 \\rightarrow_{-1} V_3$. While in theory the structure could still be identified from the observational covariance structure, it may be difficult to achieve in practice.\n- More importantly, our model relies on *interventional data* to identify the SCM structure, so even when there are faithfulness violations, interventional data would circumvent those issues entirely."}, {"review_id": "vpURZQyWbk", "reviewer": "Reviewer_uXJf", "summary": "The authors start with Pearl’s Ladder of Causation, arguing that this framework suggests DNNs trained in a “statistical mode” — that is, to predict passive observations — can only capture associations (L1) and are inherently limited from reasoning about actions, interventions, and explanations. They then show that a GPT-style transformer trained on next-token prediction can, in fact, simultaneously discover examples of linear Gaussian structural causal models and learn to answer counterfactual queries about them. This leads them to propose that neural networks trained with standard statistical prediction objectives on passively observed data may still be able to uncover and leverage causal models of the world.", "questions": "I would recommend that the authors update their understanding of the current literature in this area in order to make a meaningful contribution to the field.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "The claim that a “statistical mode”—that is, predicting passive observations—can only capture associations (L1) and is inherently limited from reasoning about actions, interventions, and explanations reflects a misunderstanding of the relationship between causality and statistics. It also represents a misinterpretation, or at best an outdated reading, of Pearl’s Ladder of Causation. In fact, many works have demonstrated how statistical models can be designed to encode and decode causal relationships.\n\nIt has also been widely observed that large language models (LLMs), trained solely on next-token prediction, exhibit a degree of causal reasoning ability. Most studies along this line are empirical, though there are a few theoretical works that explicitly link next-token prediction with causality.\n\nStrengths: The authors have clearly invested effort into this investigation.\n\nWeaknesses: However, the paper stems from an outdated perspective, explores a problem that has already been thoroughly examined, and offers observations that have been extensively discussed in the community over the past several years. I would recommend that the authors update their understanding of the current literature in this area.", "was_revised": true, "final_justification": "I have reviewed the authors' response and noted the consensus among reviewers to reject. I am not convinced by the authors' rebuttal to warrant increasing my score.", "rebuttal": "We respectfully disagree with the reviewer’s assessment:\n\n1. The reviewer appears to **fundamentally misunderstand our position**. They attribute to us Pearl’s position:\n    \n    > The authors start with Pearl’s Ladder of Causation, arguing that this framework suggests DNNs trained in a “statistical mode” — that is, to predict passive observations — can only capture associations (L1) […]\n    > \n    \n    We **explicitly argue *against* this position**, as stated in the first paragraph of our paper:\n    \n    > According to Pearl [36], this hierarchy implies that DNNs trained in a “statistical mode” to predict passive observations can only master associations (L1) and are prevented from ever reasoning about actions, experiments, and explanations (L2 and L3). While PCH captures fundamental distinctions among causal concepts—providing an extremely productive theoretical framework with wide-reaching implications—we do *not* think that Pearl’s claim is one of those implications.\n    > \n    \n2. The reviewer further asserts that our contributions revisit ideas that have already been extensively discussed in the literature, but **provides no citations** or specific references to support this claim. In contrast, we have reviewed and cited the relevant body of work in our related work section—including studies of causal capabilities in LLMs and theoretical arguments regarding LLM causal abilities (e.g. Zečević et al., 2023).\n\n3. Our contribution is distinct: we provide a **controlled existence proof** that next-token prediction on passive data can yield a functioning internal causal model and inference engine. We support this with **mechanistic evidence** (via decoding and gradient-based interventions), not just behavioral results. This goes beyond existing observational studies and directly challenges the notion that true causal discovery and causal reasoning engines at interventional (L2) and counterfactual (L3) levels of Pearl’s Causal Hierarchy cannot emerge through statistical prediction objectives."}]

Author rebuttal:

Meta-review: This paper tackles a foundational question, whether statistical learning systems can achieve interventional (L2) and counterfactual (L3) reasoning.
Pearl says no. 
The authors propose a simplified setting, where a large number of 4-variable structural causal models (SCMs) is used to generate L2 sentences and L3 sentences. For some SCMs, L2 and L3 sentences are used in the train; for other (referred as test thereafter), only L2 sentences are used in the train. 
The proof of concept is established by showing that the trained LLM successfully answers L3 queries on test SCMs.
In brief, the authors say yes.

The value of this paper thus lies - besides tackling a hot question - in proposing an innovative artificial methodology, that imo could be applied to investigate other limitations of LLMs, as this methodology avoids the known pitfall that LLMs can find the sought answer in the data they have access to.

The discussion about the merits and value of the paper has been long and thorny.
Rev nnXY spotted an issue in the experimental setting, concerning the fixed causal ordering of the variables. This issue was fixed by the authors. 
Rev Y1qU is wondering whether there might be a leakage between the train and the test, given the restricted experimental setting.
Rev m4HY, though positive, states that related work should have been mentioned and discussed (with a list of references). 
Rev d6ju is concerned by the highly constrained and simplified experimental setting. Authors reply that they only aim to refute Pearl's claim; and a negative example (here, a proof of concept for successful L3 reasoning) is enough. While d6ju globally agrees, Reviewer insists on the fact that the discussion about the limitations of this proof of concept (e.g., when natural language is considered instead of artificial one) be detailed and careful. 
Rev uXJf is concerned with the exact contribution of the paper, as the state of the art already shows that statistical data may be enough to answer causal questions or identify the causal model. Authors state that their goal and their contribution is providing an empirical counterexample to Pearl’s impossibility claim.
Decision: Accept (poster)