A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment
Authors: Raanan Yehezkel Rohekar
Yaniv Gurwicz
Sungduk Yu
Estelle Aflalo
Vasudev Lal
Venue: ICML 2025 poster (2025)
Keywords: structural causal model
causal reasoning
GPT
mechanistic interpretability
explainable AI
causal discovery
Primary area: general_machine_learning->causality
TL;DR: We examine whether GPT models are trained only to predict the next token, or do they implicitly learn a world model from which a sequence is generated one token at a time.

Abstract: Are generative pre-trained transformer (GPT) models, trained only to predict the next token, implicitly learning a world model from which sequences are generated one token at a time? We address this question by deriving a causal interpretation of the attention mechanism in GPT and presenting a causal world model that arises from this interpretation. Furthermore, we propose that GPT models, at inference time, can be utilized for zero-shot causal structure learning for input sequences, and introduce a corresponding confidence score. Empirical tests were conducted in controlled environments using the setups of the Othello and Chess strategy games. A GPT, pre-trained on real-world games played with the intention of winning, was tested on out-of-distribution synthetic data consisting of sequences of random legal moves. We find that the GPT model is likely to generate legal next moves for out-of-distribution sequences for which a causal structure is encoded in the attention mechanism with high confidence. In cases where it generates illegal moves, it also fails to capture a causal structure.

A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment
Raanan Y. Rohekar" 1
Yaniv Gurwicz"
Sungduk Yu?
Estelle Aflalo Vasudev Lal
Abstract
In this paper, we suggest that there is restriction in the GPT architecture that prevents it from learning conditional independence (CI) relations between tokens in a sequence. Moreover, under certain assumptions, a causal structure is directly entailed from these CI relations. One may ask whether this lack of restriction results in implicitly learning causal model of the world during the pre-training procedure of GPT. Assuming that both causal world model and model based on surface statistics are sufficient solutions, one possibility is that a causal world model is more compact and more likely to be learned during pre- pre-training, in line with Occam's is razor. For example, if weights are distributed from a uniform distribution in the surface statistics model, then a causal structure limits the range of their distribution. If so, what assumptions underlie this causal world model?
Are generative pre-trained transformer (GPT) models, trained only to predict the next token, implicitly learning a world model from which se- quences generated one token at time? We address this question by deriving causal interpre- tation of the attention mechanism in GPT and pre- senting a causal world model that arises from this interpretation. Furthermore, we propose that GPT models, at inference time, can be utilized for zero- shot causal structure learning for input sequences, and introduce corresponding confidence score. Empirical tests were conducted in controlled en- vironments using the setups of the Othello and Chess strategy games. A GPT, pre-trained on real- world games played with the intention of winning. was tested on dustif-distribution synthetic data consisting of sequences of random legal moves. We find that the GPT model is likely generate legal next moves for -iist-riiitiiit sequences for which a causal structure is encoded in the at- tention mechanism with high confidence. In cases where it generates illegal moves, it also fails to capture a causal structure.
Rohekar et al. (2024) recently proposed ABCD, a method for causal interpretation of unmasked self-attention in BERT models (Devlin et al., 2019), demonstrating its use in ex- plaining movie recommendations (Nisimov et al., 2022). We take similar approach, with key differences, and propose a causal interpretation of GPT's masked attention mechanism. Furthermore, we define corresponding causal world model. ABCD is adapted to learn causal structures, where the in- duced dependency relations are encoded in GPT's attention matrices. We then ask whether errors generated by GPT are correlated with the uncertainty in representing the causal structure by the attention matrices. To this end, we define a metric based on the entropy of p-values from CI tests used for inferring the causal structures.
1. Introduction
In recent years, the generative pre-trained transformer (GPT) model (Radford et al., 2018) has demonstrated high-quality generative capabilities, as perceived by humans. Although this model is trained to generate one token time, it has been demonstrated to perform range of tasks beyond next- token prediction, such as visual understanding and symbolic reasoning (Liiu et al., 2024; Team et al., 2023; Chowdhery et al., 2023). Are these emergent abilities (Li et al., 2023) or are they merely a 'mirage' resulting from the choice of metric and task (Schaeffer et al., 2024)?
2. Related Work
Recent work has examined the internal process of large lan- guage models and investigated whether a world model implicitly learned using a well-defined and constrained set- ting, such in Chess (Toshniwal et al., 2022) and Othello (Li et al., 2023) games environments. For the Othello board game setting, Li et al. (2023) demonstrated that the board state can be inferred from attention matrices in GPT, and Nanda et al. (2023) showed that a linear classifier suffices to reconstruct the board state from these attention matri- ces. They claim the emergence of world model in GPT. Nevertheless, they do not explain how the board game is
"qqual contribution 'Intel Labs. Correspondence to: Raanan Rohckar <raanann.cchzkee intel.com>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
(I) Leurned causal gruph
(1I) Initial state
(III) After move
(IV) After move
(V) After move 2
(VI) After move
Figure 1. An example of a real Othello game sequence and the corresponding causal structure recovered using the proposed method. Red numbering (0,1,2,3) on the causal graph nodes and game board discs corresponds the indices of the game moves. The blueish letters (a,b,c,d} indicate the discs in the initial state of the board game. (I) The causal graph learned by our method given the sequence of moves described hereafter. (II) The initial state of the board. (III) After move O: Black plays and flips disc 'd' to black. (IV) After move 1: White plays and flips disc "c' to white. This move does not depend on the previous move 0, and it aligns with the learned causal graph where node "1" found independent of node *0", (V) After move 2: Black plays and flips disc 'a" black. This was made possible since disc 'd' had being flipped to black in the earlier move 0 (this causal link is indicated by yellow arrow). Correspondingly, this causal link is also revealed in the learned causal graph by node "O" being the sole parent of node '2'. (VI) After move 3: White plays and flips disc 'd' to white. This was made possible because disc '1" was white (due to move 1), and disc 'd" was black (as mentioned before, it was flipped to black earlier in move 0). Therefore we expect both moves O and to be the causes move (as indicated by yellow arrows). This exactly revealed by the Iearned causal graph.
(T) Learned consal graph
After O
(IT) After move
(IV) After move
(V) After move
(VT) After move 4
WIW
A
2
Figure 2.
An example of Chess game sequence and the corresponding casual structure recovered using the proposed method. It evident that first move (Move 0), played by White, enables playing Move (a directed edge from node 0 to node 2). In addition playing Move led Black to play Move 3 (a directed edge from node 0 to node 3). These moves led to Move 4 (directed edges from nodes 0 and 3 into 4.
encoded within the attention matrices, nor why the attention mechanism represent the board state. In essence, they do not provide an explanation for the apparent emergence of the world model. Furthermore, their reconstructed world model (the board game state) applies only to the domain for which the GPT model was trained and lacks the generative mechanism underlying the token sequences.
tural causal models. Matrices are written in bold, vectors in bold-italic, and models in calligraphic font. A summary of the main symbols used in this paper is provided in Table 1.
3.1. Attention in GPT
Attention is a mechanism that estimates network weights with respect to the context in an input sequence of to- kens (Schmidhuber, 1992). In a GPT model, which is based on the decoder part of the Transformer architecture (Vaswani et al., 2017), an attention layer estimates an n x n trian-tri (masked) attention matrix A given an in- put sequence of tokens. The input sequence in the form of an n X d matrix Y. where the i-th row vector Y (i, .) is an embedding (representation) of the i-th to- ken in d dimensions. The attention matrix is estimated by A = sof soot Y YT). where A is lower triangular and each row sums to I'. In addition to the attention weights,
In this paper, we consider the structural causal model general-purpose world model that describes the generative process that is applicable across various domains (not spe- cific to a single task, such as the board state in Othello or Chess). We explore whether GPT is capable of capturing properties of this world model, which may help explain its apparent emergence. See an example for Othello in Figure and for Chess in Figure? 2.
3. Preliminaries
In this section, we provide the notations and descriptions for self-attention in the GPT architecture, as well as for struc-
W 'ee = W.wkk/ // k//a,, where We and W k are typically Iearned explicitly, and dx is the number of columns.
2

---

A Causal World Model Underlying Next Token Prediction; Exploring GPT in Controlled Environment
Table Main notations used in the analogy hetween attention in GPT and SCM. The first of symbols represents entities in GPT, and the second set represents entities in SCM.
Symbol
Description
Z
output embedding of input symbol i, Z, Z(i, .), in attention layer
Vi
value vector corresponding input i, V, = V(i, -), in attention layer attention matrix
A
T
Transformer neural network
WV.
W QK
learnable weight matrices GPT
X;
random variable representing node i in an SCM latent exogenous random variable in an SCM weighted adjacency matrix of an SCM causal graph structure
U;
G
g
the attention layer calculates value matrix, V YWv. where row V(i, is the value vector of the i-th token. Then, the output embeddings are
to an SCM consists of one node per variable, and directed edges representing direct causal relations evident from F.
In this paper, we relate the linear inter-token relations in GPT attention (Equation 1) to a corresponding linear- Gaussian SCM having directed acyclic graphs (DAG). In these SCM models, each variable is determined by a linear combination of its direct causes and an independently dis- tributed additive noise represented by corresponding nor- mally distributed exogenous variable. For a Gaucs-G SCM, let G be a weight matrix, where G(i,j) is the weight of the parent (direct cause) node X, linearly determining the child (direct effect) node X, Node X is not parent of X; if and only if G(i, k) = O. In addition, ~ N(Hus Cv), where in this paper. we assume Cu is diagonal matrix. The set of functions F is defined such that Vi E [1....,n]. n]]
Z AV.
(1)
where the i-th row, Z, is the embedding of the i-th output token. In a GPT, several attention layers are stacked and pre-trained such that the i-th output embedding the last layer predicts the (i + 1) input token. That is, predicts the next token in the sequence.
It is important to note that, in the GPT architecture, the embedding of one token is influenced by another token only by the attention matrix, A. In addition, note that an attention matrix A is estimated uniquely for each input sequence of tokens, using weight matrices (Wok QK: Wy) that are learned commonly for all in-distribution input sequences.
X₂ < G .XX U,.
Assuming DAG and causally sorted nodes (ancestors pre- cede their descendants), G is strictly lower triangular (zeros on the diagonal). Given the assignment, we can write in matrix form X = G. + U, and
3.2. Structural Causal Model
structural causal model (SCM) is a model that can en- code causal mechanisms domain (Pearl, 2009; Spirtes al., 2000; Peters et al., 2017) and explain data samples generated from these causal mechanisms (Pearl & Macken- zie, 2018). An SCM is tuple (U,X, F, P(U)), where U = (U,,...... } is a set of latent exogenous random variables, = (X,,... X,, is a set of endogenous ran- dom variables. FF = (fi..... fm)) is a set of deterministic functions describing the values X given their direct causes, and P(U) is the distribution over U. Moreover, each en- dogenous variable X; has exactly one unique exogenous cause U; (m n). The value of endogenous variable X,, Vi E [1.....7] n] is determined by
x = -I-G)
(4)
As is a strictly lower- twerr-trangullr weight matrix, (I- G) is a lower uni-triangular matrix (ones on the diagonal). Note that this is equal to the sum of geometric series
n-1 (I G-G---
G
(5)
It can be seen that element (i.j) represents the cumulative effect of X, on X; via all directed paths of length up to n - 1. The equivalent weight of directed path from X, to X; is the product of the weights of all edges along that path. The cumulative effect is the sum of the equivalent weights of distinct directed paths from X, to X,. Note that even some of the nodes are latent confounders, (I -1 is still
X + (Pa, U.),
where Pa, is the set of direct causes (parents in the causal graph) of X₁, and left-arrow indicates assignment resulting from the cause-effect relation. graph G corresponding
3

---

A Causal World Model Undertying Next Token Prediction: Exploring GPT in Controlled Environment
triangular because, by definition, latent confounders have no ancestors and precede other nodes in causal ordering. Equation 4 represents a system with input U, output X, and weights (I- - [-G) The covariance matrix of the output is
is, given an sequence of tokens, {le, -.. 1-11) GPT predicts tokens (fi,.... , f. corresponding values matrix V have rows corresponding to input tokens f fx-1): 1. and the output embeddings of these tokens are the rows of matrix Z = AV. Note that V = YWv. where Wy is a weight matrix fixed for all input sequences, and Y the input embedding of the tokens in a specific sequence. Each column of Wv can be viewed as an independent vector onto which the input embeddings are projected. That is, V(i,j) is the projection of the input embedding of token 4, Yii, +), onto the vector Wv Wv(+.)) which is common to all in-distribution sequences. At inference, each attention matrix of the last attention layer, A, is extracted and lower uni-triangular matrix calculated, D- A,, where D = diag(A). Then the covariance matrix estimated
An attention matrix A and the
Cx = E[(X- - xx) - -xx]
B[(1- - G) úū (I-
(6)
[II - G) BÜÜ ||
(1 G) "] Cu ()-
where U = U Pu and Fx = =I (I-G)-
In this paper, we employ a constraint- causal discovery approach (Spirtes al., 2000) that uses conditional indepen- dence (CI) tests to learn the underlying causal graph. This approach generally requires assuming the causal Markov property and faithfulness.
C [
(7)
Definition 3.1 (Causal Markov). In causally Markov graph, a variable is independent of all other variables, except its effects, conditional on all its direct causes.
Note that, unlike Rohekar et al. (2024), who proposed C= T for unmasked self-attention, we utilize the trian- gular form of masked attention in GPT to revert the attention normalization performed by the softmax and obtain a uni- triangular form. Thus, this covariance matrix allows us to treat properties calculated from different attention matrices in similar manner. In this paper (Section 4.2 and Sec- tion 4.3), the properties we calculate are based on p-values from tests of conditional independence between tokens, esti- mated from the covariance matrix. Next, following Rohekar et al. (2024), we relate each token to an endogenous node in an SCM, and assume CU = I from the central limit theorem. Thus, we equate the covariance C Cu:
Definition 3.2 (Faithfulness). A distribution is faithful to a graph if and only if every independence relation true in the distribution is entailed by the graph.
4. A Causal Interpretation of GPT
We first describe a relation between GPT and SCM. Then, we present an efficient method for zero-shot causal struc- ture learning-in the presence of latent confounders-for a given input sequence, using a modified version of the ICD algorithm (Rohekar et al., 2021). Finally, we introduce confidence scoring function for learned causal structures that uses p-values computed during causal discovery.
[D = (I- [[I (I- G)- (8)
where both D A and G) are lower uni-triangular matrices. The (i.j) elements, Vi > j. of these matri- ces have the same meaning: influence of token/node j on token/node Finally, since GPT is pre-trained pre- dict tokens (1,,.... (n) given input tokens (lo,.... ., Ln- ). and since the only cross-token influence embeddings is through the attention matrix, the last attention layer captures the causal structure underlying the output tokens. Earlier attention layers transform embeddings of (lo...., (o.....,n-)) 1 to values, V, which are equivalent to instantiations of the ex- ogenous variables, U, in SCM. This follows from equating Equation and Equation 4, where D = (I-G) That is, we equate the outputs: tokens' embeddings and SCM nodes' values. If some of the nodes are hidden con- founders, then the corresponding rows and columns are removed,
4.1. A Relation between GPT and SCM World Model
Rohekar et al. (2024) derived a causal interpretation of BERT (Devlin et al., 2019). We follow similar approach, with several important modifications and extensions, to de- rive an SCM-based causal interpretation of GPT. The de- rived relation between GPT and SCM is threefold (Figure 3):
1. 'Values Matrix' as instances of SCM exogenous nodes,
2. output embeddings observations of SCM endoge- nous nodes, and
3. attention matrix as a transitive closure of the SCM graph.
First, unlike BERT-based models, which are pre-trained to predict masked tokens within the input sequence using the surrounding tokens (Devlin et al., 2019). GPT is pre- trained to predict the next tokens in the sequence. That
(9) where denotes the indices of nodes hidden in the world model (i denotes the omission of the corresponding rows and columns).
D D- (I-G)) -
4

---

A Causal World Model Underlying Next Token Prediction; Exploring GPT in Controlled Environment
Masked Self Attention
Structural Causa Model (SCM)
wy
V u
Context (exogenous):
U2
U/
token 4 embeddding
GyG.
V2
G(2.1)
G(3,1)
Output: Z
Ug
Observation
X2
G(1.3)
X3
AV
Wg
(endogenous):
G(4.2)
[2
0
Es
Z
Z, X
X4
So softmas fomttax QK vā
G5544)
Wx
K
Graph weights:
G(2.1)
0
X5
Attention
G(5.1)
G(5.4)
D (1 G)-
abstract causal relations world model
Figure 3. Relations between GPT (left) and SCM (right), derived in Section 4.1. 'Values matrix' as exogenous variables in SCM: In the attention mechanism, the input embeddings matrix is multiplied by the column vectors (.) of the weight matrix Wy to form the column vectors V; of the values matrix V. Each values vector V; is treated as an instantiation u of the exogenous nodes in SCM, where element j in V; is an instantiation of node Uj- Output embeddings as observed nodes: Each values vector V, multiplied by the attention matrix, resulting in column vector Z, of the output embedding. This corresponds to observation of the endogenous nodes in SCM. Attention matrix as a transitive closure of causal graph: An element A (i, in the attention matrix reflects the "attention" given token j when computing the embedding of token i This corresponds the influence that node j has node through all directed paths in the causal graph, as estimated by (I - G)- = E3 G If some nodes in SCM are hidden confounders, then the "] after removing the rows and columns corresponding the hidden nodes.
attention matrix reflects [(T G)-
In light of the causal interpretation of GPT, one important question is what causal world model the GPT architecture supports. Note that GPT's non-linear transformations do not affect inter-token relations. Often, single causal structure is assumed to govern a domain. In contrast, the causal world model entailed by the causal interpretation of GPT assumes a distinct SCM for each sequence. Specifically, in causal world model supported by a GPT with & heads in the last attention layer, each sequence is assumed to be generated by an ensemble of & SCMs.
by the underlying causal structure. Although this supports only rung-1 inference in the ladder of causation (Pearl & Mackenzie, 2018) many of the underlying causal relations can be extracted under certain assumptions- even in the presence of latent confounders and selection bias (Spirtes et al., 2000). These relations are generally represented in a type of causal structure known as a partial ancestral graph (PAG) (Richardson & Spirtes, 2002). We follow a procedure called ABCD, proposed by Rohekar et al. (2024), with sev- eral modifications. First, since the causal (topological) order is given (restricted by the masked attention in GPT), apply causal discovery recursively to efficiently learn the causal structure. To this end, we slightly modify the iterative causal discovery (ICD) algorithm (Rohekar et al., 2021), as described in Appendix B, to reconstruct a causal structure at each recursive iteration. The procedure is outlined in Algorithm 2. The input is sequence of tokens over which we construct the graph. The output is PAG structure. In line 2, exit condition corresponding to the base case (a single-node graph) is tested. In line 3, the last token is popped from the sequence and assigned to b. resulting in a shorter sequence S'. Then, a recursive call is made in line 4 to learn the structure over the tokens in S'. Note that since it is ensured that t, is not an ancestor of any token in S' the skeleton and v-structure relations of G' are guaranteed
In addition, for given head, the causal structure over quence of tokens (4,,..,44] (n is identical to the correspond- ing subgraph over these tokens in all in-distribution exten- sions of the sequence. That is, given a sequence of tokens (4..... } and a corresponding graph structure 9n. ob- serving any next token f,+1, such that (fI...., ., fn. fn+1] is in-distribution, should not violate the causal relations in Gn and may only reveal relations between tokens (t.....,t., t.) and token fn+1-
4.2. GPT for Zero-Shot Causal Structure Learning
The causal interpretation presented in this paper leads to view in which each attention module captures associa- tions (correlations) between input tokens that are induced
5

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
not to change when 4 is added back the graph (Spirtes et al., 2000). In lines 5-7, token in is connected to every node in g'. Finally, in line 8, edges between in and the rest of the graph are learned (removed if conditional indepen- dence is found) using the ICD algorithm (Rohekar et al., 2021) and the graph is oriented (Zhang, 2008), Although we use ICD, other constraint-basee causal discovery algorithms (Colombo et al., 2012; Claassen et al., 2013: Yehezkel & Lerner, 2009; Spirtes et al., 2000; Rohekar et al., 2018; Nisi- mov et al., 2021), differing in their underlying assumptions, can also be used.
determined significance threshold (a). It is important to note that causal structure be uniquely represented by set of CI tests and their results. Hence, we propose scor- ing function based on the distribution of these p-values to evaluate the confidence in a structure learned from a given attention matrix. A complete undirected graph corresponds to a lack of knowledge about causal relations. Generally, causal trructre-learming algorithms prune edges from this graph based statistical CI tests between pairs of variables (tokens, in our case). The removal of edges between inde- pendent variables may then entail causal relations between other variables (Zhang, 2008).
Algorithm 1: Recursive Causal Discovery for GPT
Let p= (p...... pt} be the set of all p-values computed as part of causal structure learning. The null hypothesis cor- responds independence, where p-values greater than the significance threshold, a, correspond to edges removed from the complete graph. We define Pnnd = {o E p p 2 a)) and Pdep {o E p p < a). Since p-values are uniformly dis- tributed under the null hypothesis, we expect the entropy of p-values corresponding to independence, Hind, to be higher for matrices that correspond to a structure than for those that do not. Conversely, we expect the distribution of Puep to be weighted toward zero. Hence, the entropy of p-values corresponding to dependence relations, Hasp- is expected to be lower for matrices that correspond to a structure com- pared to those that do not. We therefore define the following confidence score, given an attention matrix A:
Input: S: a sequence of tokens (t2....,th) .t.]
Output: G: a partial ancestral graph (PAG)
Function Lear ructure (S)
S|| = then return a graph with the single node
in S
tn. 5' + pS)(S)
g' < Leaasstrucuuuuee(S
g'
set E to the set of edges (circle edge-marks) between in and every node in G'
connect E in G
test CI for edges E and orient G using Algorithm 3 (Appendix B)
return g
R(A) = Hind Hasp:
(10)
which captures the contrast between dependence and inde- pendence relations entailed by the learned causal graph.
Thus, a causal structure for a particular output sequence can be inferred in a zero-shot manner directly from the attention matrix in the last layer. In multi-head attention, the final attention layer, having heads, is the last layer in which tokens may affect one another. Hence, Algorithm 2 is invoked independently for each head, returning a set of k structures.
5. Experiments and Results
We use an experimental framework in which the world lay- out and rules governing the generation of sequences well defined and known, but are not utilized during training. We measure how well attention in the trained GPT model repre- sents a causal world model and whether this representation is correlated with the ability to generate tokens that adhere to the world rules.
4.3. Causal Structure Confidence
In this section, we derive metric that describes how com- patible a sequence is with the causal model implicitly en- coded by GPT. Given an output sequence of tokens, S, and a causal structure G recovered from the last attention layer A, can we score the confidence in this causal structure? Recall that in the proposed world model, each sequence has its own causal structure, and each causal structure may include latent variables. Since it is unclear how to calculate likelihood PS G), we propose the following approach.
5.1. Setup
We used two controlled environments: Othello and Chess strategy games. For Othello, we examined a GPT model trained by Li al. (2023) on 132 thousand real-world sequences, and for Chess, we examined GPT model trained by Toshniwal et al. (2022) on 2.9 million real- world games.
A causal muucture-learning algorithm performs multiple sta- tistical tests of conditional independence (CI) using the co- variance matrix estimated from the attention matrix. These CI tests calculate p- values and compare them against a pre-
For both environments, no information about the game board layout or game rules was used during their training process, and the training data consisted of games in which
6

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
(a) CI Conditioning Size
(b) CI Conditioning Size
(c)
Conditioing Size or
(d) All CI Conditioning Sizes
0.3
0.3
0.25
0.25
025
0.15
0.15
0.15
0.4
0.2
0.05
0.05
0.05
-0.06
-0.05
-0.05
-0.1
-0.1
-0.1
-0.1
0.15
-0.15
-06
-1.2
-0.2
-0.2
10
18 26
10
24
28
50.32
10 12 14 16 28 30
10
26
Figure 4. Average difference in structural confidence between legal and illegal move generation (vertical axis) for different ieput-sequence lengths (horizontal axis). Error hars represent the 95% confidence interval calculated using t-test. Confidence scores are calculated from p-values of: (a) all unconditional (marginal) independence tests, (b) all CI tests having exactly one conditioning node, (c) only tests from both cases (a) and (b), and (d) only CI tests without limiting the conditioning set sizes, needed to reconstruct a causal structure.
players played with the intention of winning, For example, positional encoding was not used.
structure without limiting conditioning set sizes. The results are shown in Figure 4. Let Riegat be the average structural confidence score of sequences for which a legal token was generated, and Rillegal be the average structural confidence score of sequences for which an illegal token was gener- ated. The vertical axis represents the difference in structural confidence scores Rigaa Rillgaal Error bars indicate 95% confidence intervals (unpaired t-test). The horizontal axis indicates sequence length.
In all our experiments, we used test sets that are out-of- distribution with respect the training set, consisting of sequences of randomly sampled legal moves (not by the GPT models), lacking the objective of winning. In other words, the support of the test distribution is not subset of the support of the training distribution, where supp(Ptrain) C supp( Ptest). ) See Appendix A for an em- pirical comparison between the sets. This enables evaluat- ing whether the model implicitly encodes the game rules. For both Chess and Othello, test sets consisted of 1,000 ran- domly generated sequences of legal moves. See Appendix A for more details.
It is evident that when relying solely on raw attention values, case (a), the difference between legal and illegal generated tokens not statistically significant, except for sequence length 20. Relying solely on Cl-test with exactly one node in the conditioning set, case (b), the difference in structural confidence is positive for all tested sequence lengths, but statistically significant only for sequence length 17. When employing pairwise correlations and CI tests with exactly one node the conditioning tests, case (c), the result statistically significant for both sequence lengths 17 and 20, implying that these two types of tests are complementary. Finally, using all Cl-tests needed to learn the causal graph, without limiting the conditioning set sizes, case (d), pro- vides the best results: sequence lengths in range [15, 22] are statistically significant, and the difference between legal and illegal scores positive (Riegal Rillega) for all tested sequence lengths.
For causal discovery implementation and empirical evalua- tion we used the Causality Lab repository: github. com/ Intellabe https//asslllltylaaaaaa
5.2. Ablation Study
We examine legal move generation with respect to 1) limit- ing the condition set sizes in the CI tests used to learn causal structures, and 2) pruning attention heads based on the con- fidence scores of their corresponding causal structures.
5.2.1. CONTRIBUTION OF CI TESTS
We examine whether conditional independence (CI) tests from which the causal structure is entailed provide an ad- vantage over pairwise correlations directly represented by elements in the attention matrix. To this end, we calculate the confidence score (Equation 10) using p-values from: a) all pairwise marginal independence relations (from raw iitention-mattiss elements)- CI conditioning size 0; b) CI tests having exactly one node in the conditioning set; c) all CI tests having either an empty or single-node condition- ing set; and d) all CI tests used to reconstruct the causal
5.2.2. ATTENTION HEADS PRUNING BASED ON CONFIDENCE SCORE
In this experiment, we examine the importance of each attention head (in multi-head attention) for legal-move gen- eration. We evaluate the importance a head by the degree of confidence with which it represents a causal structure. This is different from the experiments in Section 5.3 and Section 5.2.1 where the average structural confidence score of the heads was associated with each test sequence.
7

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
Sequence Length 15
Sequence Length 17
Sequence Length 20
0.9
0.9
0.9
0.8
0.8
0.8
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.3
0.3
02
0.2
0.2
0.1
0.2
0.5
0.2
0.4
0.6
0.8
0.2
0.6
Sequence Length 22
Sequence Length 25
Sequence Length
0.9
0.9
0.8
0.8
0.8
0.7
0.
0.5
0.5
0.55
0.4
0.4
02
02
0.2
0.1
0.1
0.1
02
0.4
0.6
0.8
0.2
0.4
0.6
0.28
0.2
0.4
0.6
0.8
Figure 5. Normalized accuracy of legal-move generation (vertical axis) as a function of the percentage of heads pruned (horizontal axis) based on structural confidence. solid blue curve represents pruning percentage heads having lowest structural confidence, while a dotted orange curve represents pruning in the reverse order (pruning a percentage of heads having the highest structural confidence).
Here, structural confidence score is caleulated for each attention head for each sequence in the test set. That is, for 1,000 test sequences and 8 heads in the last attention layer, there is a set of 8,000 scores. This set, denoted R, is sorted in ascending order. From this sorted set, nine equally spaced values are selected as thresholds, denoted th (th,,...,thg, corresponding to the 10%, 20%. 90% percentiles. Given threshold th,, for each test sequence the attention heads that have structural confidence scores lower than the threshold are pruned (skipped in the forward pass) and the next token is generated without those heads. Hence, the number of pruned heads may vary from sequence to sequence. We then calculate the legal-move generation accuracy for each threshold, that is, accuracy per pruning percentile. Note that retraining the model after pruning is not required (Voita et al., 2019).
(aceuracy function of pruming percentile) than in the reverse-order pruning process.
In Figure 5, it is evident that pruning heads with lower structural confidence scores (solid blue curve) results in higher legal-move generation accuracy and greater area un- der curve, compared to removing heads with higher struc- tural confidence scores (dotted orange curve). This demon- strates the importance of individual attention heads that encode structural information for generating legal moves.
5.3. Legal Move Generation vs. Structural Confidence
Is there a relation between generating legal tokens (moves) and how well attention matrices implicitly represent causal structures? Recall that the model was not trained explicitly to generate legal game moves but rather to predict the next move played by a human with the intention of winning the game. Moreover, information about the game, such as the existence of board game and rules, were not provided to the model (Li et al., 2023; Toshniwal al., 2022).
In our case, it is expected that pruning heads with low struc- tural confidence will have limited impact on the accuracy. To examine this, we compare the accuracy to that of a reverse- order praning process. In this process, we prune heads hav- ing high structural confidence scores while keeping those with lower scores. Specifically, we sort the set of scores, R. in a descending order, and for each threshold, prune the heads that have higher structural confidence scores. Un- der the assumption that GPT implicitly uses a causal world model to generate the next tokens, we expect that pruning heads having low structural confidence scores will result in higher legal-move accuracy and larger area under curve
In this experiment, we examine whether the cases in which the model generates illegal tokens are also cases where the causal structure less distinctive, as measured by the struc- tural confidence score, (Equation 10). Here, the score for a given sequence is the average of structural confidence scores calculated for the attention heads in the last layer. Recall that structural confidence is not objective in GPT pretraining.
8

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
Sequences Lengths
Sequences Lengths 17
Sequence Length
Sequence Length 15
0.96
0.995
0.94
0.90
0.92
0.95
0.96
0.85
0.55
0.9
0.98
0.92
0.9
0.84
0.86
0.55
0.97
0.88
0.86
0.82
0.3
0.965
0.2
0.78
6.5
8:5
10.3
125
7.75
9.75
10.75
11.25 12.75
0.5
1.5
0.75
185
135
1.9
Sequences Lengths
Sequences Longths 22
Sequence Length
Sequence Length 25
0.9
0.95
0.9
0.97
0.99
0.98
0.9
0.94
0.95
0.97
0.7
0.91
0.85
0.6
0.92
0.95
0.94
0.5
0.9
0.93
0.75
0.59
9.5
11.5
12.5
855
10.5
11.5
1.4
1.54
1.68
1.66
2.1
1.98
Sequences Lengths
Sequences Lengths 30
Sequence Length
Sequence Length
0.88
0.96
0.9
0.97
0.95
0.8
0.9
0.96
a-
0.94
0.95
0.6
0.92
0.93
0.5
0.2
0.91
10.5
12.5
105
125
1.35
1.65
1.75
1.85
195
2.05
1.7
Othello
Chess
Figure 6. Legal move generation accuracy (vertical axis) a function of structural confidence score It (horizontal axis) for Othello (left two columns) and Chess (right two columns). Horizontal limits for each point indicate interval of R in which accuracy was averaged. Horizontal dotted red line represents average accuracy. Accuracy inercases with the structural confidence score.
From Figure 6, for Othello and Chess, is evident that the legal move generation accuracy (vertical axis) increases with the structural confidence score R (horizontal axis). That is. GPT is more likely to generate legal tokens for out-of- distribution inputs when a causal structure can be learned more confidently from its attention maps.
Othello and Chess strategy games, we demonstrated that GPT implicitly learns to represent causal structures in at- tention heads. Specifically, in cases where the confidence in recovering structures from the attention matrices is low, GPT generally fails to generate a token that adheres to the game rules. In future work, these results may provide in- sights into the sources of hallucination in GPT-based models and methods for detecting them.
6. Conclusions
We presented a causal interpretation of GPT that may clarify the apparent emergence of world models in recent studies and extend their findings. Following this interpretation, we described method that utilizes the triangular form of the attention matrices in GPT to recover the covariance ma- trix of SCM endogenous nodes, and efficiently learn the causal graphs for input sequences in a zero-shot manner. Furthermore, we introduced& confidence scoring function for the learned graphs, based on the difference in entropy between the dependence and independence populations of y-values. Finally, using the controlled environments of the
Impact statement
We propose a link between the internal mechanism of the GPT model and its ability to implicitly encode the world model of a given domain. As GPT models become increas- ingly widespread, it is crucial to understand the reasoning be- hind their outputs relation to domain-specific rules. This understanding enables better human supervision and over- sight of these complex automated models. We believe our work has positive societal implications by fostering trans- parency and accountability in Al-driven decision-making

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
References
Richardson, T. and Spirtes, P. Ancestral graph markov models. The Annals of Statistics, 30(4)992-1030, 2002.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240)11-13,, 2023.
Rohekar, R. Y., Gurwicz, Y., Nisimov, S., Koren, G., and Novik, G. Bayesian structure learning by recursive boot- strap. Advances in Neural Information Processing Sys- lems, 31, 2018.
Claassen, T., Mooij, J. M., and Heskes, T. Learning sparse causal models is not NP-hard. Uncertainty in Artificial Intelligence, pp. 172. Citeseer, 2013.
Rohekar, R. Y., Nisimov, S., Gurwiez, Y., and Novik, G. Iterative causal discovery in the possible presence of la- tent confounders and selection bias. Advances in Neural Information Processing Systems, 34:2454-2465, 2021.
Colombo, D., Maathuis, M. H., Kalisch, M., and Richard- son, T. S. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pp. 294-321, 2012.
Rohekar, R. Y., Gurwiez, Y., and Nisimov, S. Causal in- terpretation of self-attention in pre-trained transformers. Advances in Neural Information Processing Systems, 36, 2024.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of NAACL-HILT, pp. 4171-4186, 2019.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024.
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent world representations: Exploring sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023.
Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Spirtes, P., Glymour, C., and Scheines, R. Causation, Pre- diction and Search. MIT Press, 2nd edition, 2000.
Liu, H., Li, C., Wu, Q.. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., kehhllkyyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Nanda, N., Lee, A., and Wattenberg, M. Emergent lin- ear representations in world models of self-supervised sequence models. EMNLP 2023, pp. 16, 2023.
Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as a testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11385-11393, 2022.
Nisimov, S., Gurwicz, Y.. Rohekar, R. Y., and Novik, G. Improving efficiency and accuracy of causal discovery using a hierarchical wrapper. In Uncertainty Artificial Intelligence (UAI 2021), the 4th Workshop on Tractable Probabilistic Modeling, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Nisimov, S., Rohekar, R. Y., Gurwiez, Y., Koren, G., and Novik, G. Clear: Causal explanations from attention in neural recommenders. arXiv preprint arXiv:2210.1066 2022.
Voita, E., Talbot, D., Moiseev, F,, Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1-113, 2019.
Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge university press, second edition, 2009.
Pearl, J. and Mackenzie, D. The book of why: the new science of cause and effect. Basic books, 2018.
Yehezkel, R. and Lerner, B. Bayesian network structure learning by recursive autonomy identification. Journal of Machine Learning Research (JMLR), 100u1):1527-1550, 2009.
Peters, J., Janzing, D., and Schölkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
Zhang, J. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16-17): 1873-1896, 2008
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
10

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
A. Comparison between Training and Test Data
The data used to train GPT model consisted of real-world sequences of game moves (Li et al., 2023). These moves were played strategically with the intention of winning the game. In contrast, the experiments in the paper were conducted using test data consisting of randomly generated sequences of moves that adhered to the game rules, without considering the outcome of the game.
A.1. Accuracy in Predicting the Legal Next Move for Test Sequences
In Figure 7, we plot the accuracy the model in generating legal next move (vertical axis) in Othello and Chess as function of the number of moves (sequence length) in the test input sequences (horizontal axis). Note that the test sequences were not generated by the GPT model. Instead, each move in test sequence is sampled uniformly from the set of next legal moves, according to the game rules.
For Othello, note that length-n sequences are test sequences that are trimmed to keep only the first n tokens, such that the same 1,000 sequences are used for all evaluated lengths. Although the average accuracy of the model is 95% (dashed red line), it is not uniformly distributed across different sequence lengths. For example, given sequence of 15 moves, GPT generates a legal 16th move 88% of the time (adhering to the game board state and rules). It is evident that the accuracy significantly lower for input sequence lengths in the range [10, 30] (below the average of 95%). From the Othello game rules, at the beginning of game there are only four legal moves, and as the game unfolds, the number of possible legal moves generally increases before finally decreasing again as the number of vacant spaces on the board diminishes. It might be that memorization of surface-level statistics can take place at the beginning the game. We therefore report experimental results for input sequences with sizes in the range [10, 30] (gray area), where the accuracy is lower than average. Throughout the experiments, we employ Algorithm 2 for causal discovery using partial correlation with a significance level of a 0.01 for testing conditional independence (CI tests).
For Chess, to avoid the possibility of memorization, the model constantly decreases with the sequence length, we use sequences up to 40 moves. Longer sequences generally lead to game termination before the full sequence length is reached. Due to the small error rate for shorter sequences, in our experiment we used a test set of 10,000 samples for evaluating sequence lengths up to moves.
sequences having least 10 moves. Then, since the accuracy of
A.2. Difference between Train and Test Datasets
Recall that the test sequences were synthesized by sampling each move uniformly from the set of legal next moves according to the game rules. We measure the difference between the distributions of sequences in the training dataset, train and the test dataset, test by estimating n n-gram frequencies. For a given sequence, (fo....,tee tt-1). we extract the last n tokens, assuming that the probability of the next generated token to depends only on these n tokens,
totelto..... ,e-... = P(telte-m.....
(11)
legal-mon maal-move--eee male-mooeeeeeeee generiton acourscy
00
0.38
0..99
000
0.98
generation
0.97
96
0.52
0.96
0
0.9
000
0.23
0.98
0.86
0.93
10
20
30
50
50
10
15
20
25
35 40
sequence longth
sequence Length
Figure 7. Baseline model accuracy of generating legal Othello (left) and Chess (right) game moves. Models were trained by Li et al. (2023) for Othello and by Toshniwal et al. (2022) for Chess real-world games predict the move. The test consists of randomly generated sequences of legal moves. Mcasured accuracy: the percentage of generated moves that are legal according to the game rules. The gray area Othello highlights input sequenees with sizes in the range [10, 30), where the accuracy is lower than the average 95% (red dashed line). For Chess, the input sequences that are considered have 10 or more moves (gray line threshold).

---

A Causal World Model Underlying Next Token Prediction; Exploring GPT in Controlled Environment
For the i-th sequence the test set, trimmed length é, we count the number of occurrences, A (fe-ns...y te-1) of the test sequence in the training data sequences, trimmed to length t. We then divide this count by the number of training sequences, Dtraing and estimate the mean tes ( test is the number of test sequences),
(t), of the n-gram
Artest
test fest//rraiti P4
Σ
(12)
D seeat
D
Dussin
train]train the percentage of occurrences of Similarly, using sequences excluded from the training data, we estimate Pn
n-grams of training sequences in the training data sequences. For each sequence length evaluated in the paper, (15, 17, 20, 22, 25, 30), we calculate the percentage of n-gram occurrences for n ( 6]. We then compare the percentage of occurrences Pnn real-world sequences played with the intention of winning (Dtrain) is different from that of randomly generated sequences (Dtest) used in the paper to examine the trained GPT model.
test/train and pa
train]train in Figure 8, This evaluation clearly shows that the distribution of
Sequence Length 10
Sequence Length
Sequence Length 17
Sequence Length
0101
0.0023
8.11
0.0038
0.0015
0.00%6
0,0004
0.00
0,0010
0.0000
Soquence Length
Sequence Length
Sequence Length
0101
0:001
0.0099
0.5016
0100
0.0006
0.0005
0,0055
0.0004
0 0008
0.001
Figure Percentage occurrences (vertical axis) of n-grams from test and training sequences the training data for (2,..., (horizontal axis). Light blue columns are yin train/srain and dark blue are fin Pn train|train which indicates clear difference hetween the distributions of real-world sequences used train the GPT model and randomly generated sequences used for evaluation.
test/train values. The clear difference between pl testjttaan and
B. Recursive Causal Discovery from GPT Attention
We describe our method in Algorithm 2, where, given an input sequence, a causal structure is learned from an attention matrix in the last layer. In this section, we provide a more detailed explanation of line 8, where the ICD algorithm (Rohekar al., 2021), modified learn only a given set of edges, called. The operations in line 8 are largely similar to those the ABCD algorithm (Rohekar al., 2024). The main difference is that this step refines a partially learned causal structure by testing conditional independence between pairs of nodes connected by edges in a given list E.
The operations in line 8 of Algorithm 2 are as follows. First, covariance is estimated from an attention matrix A,
C = D [
(13)
where D Aiag(A) is diagonal matrix consisting of elements the diagonal of A, such that D-1 A is uni-triangular matrix. Then, correlation matrix is estimated,
diag( diag(C)) -12 C ddag(C))
(14)
12

---

A Causal World Model Underlying Next Token Prediction: Exploring GPT in Controlled Environment
Conditional independence between two variables X and Y, conditioned correlation from R. Then, let Ind (X, 2) denote CI test based partial correlation, where p-values are estimated using Fisher z-transform. Finally, ICD is called to learn set of edges using Ind.
Z, estimated calculating the partial
In Algorithm& 3, we provide a simple modification of ICD such that it learns only the edges in E' and uses given initial graph. In red we strike out parts of the ICD and in blue are our additions. The rest of the pseudo-code exactly as given by Rohekar et al. (2021). As input, we add the initial graph to be used and further refined, and add the set of edges E to be learned (remove edges connecting conditionally independent nodes). In line 1, we remove the initialization of a complete graph, since the initial graph given as input. In line and line iteration function. Lastly, in line 8, only edges in E, rather than all edges in G, tested.
add the of edges E be tested within the ICD
Overall, utilizing the causal order enforced by the triangular form of the GPT attention matrix, each recursive call assumes that the current graph is the final learned graph, except for the edges connecting the newly added node to the rest of the graph nodes (edge list E). Note that this does not violate the ICD-Sep conditions (Rohekar et al., 2021), which constitute a sufficient set for ensuring sound and complete causal discovery algorithm. By considering only the edges connecting node its predecessors in the given causal order, a significantly lower number of CI tests are required for learning the causal graph compared the unmodified ICD algorithm.
Algorithm 2: Causal Discovery for GPT
Input: S: a sequence tokens (fı,..., (x)
Output: G: a partial ancestral graph (PAG)
Function LearnSt :
S then return graph with the single node in S
bn, S" PS)(S)
g' + LearnStructure (S")
g g' + (t,)
set to the set of edges (circle edge-marks) between t, and every node in g'
connect E in G
test CI for edges E and orient & using ICD (Rohekar al., 2021)
return g
13

---

A Causal World Model Underlying Next Token Prediction; Exploring GPT in Controlled Environment
Algorithm 3: Modified ICD (Rohekar et al., 2021) algorithm
Input:
Ind: a conditional independence oracle
G: initial PAG
E: set of edges be learned
Output:
G: PAG
initialize: +- 0, 9 + grompleeeegraph-iittee 'o" edge-marks, and done t- False
while (r S/Sm n) & (done Falsc) do
(G, Iteration (E, g, r 11
refine g using conditioning sets of size
return g
Function Iteration (E, r):
done +- True for edge (X,Y)
(gg do {Z,k-1 & PDSepRange Y. if l then
Z, complies with ICD-Sep conditions
done False for i < 1 to & do if Ind(X, Y Z then remove edge (X, Y) from 9 record Z, as separating set for (X, Y)
14
15
break
orient edges in G return (G, done)
14


Reviews:
[{"review_id": "xuVN49LrBe", "reviewer": "Reviewer_EPcN", "summary": "The work explores whether or not GPT style models learn a causal world model implicitly without explicitly being trained to do so by using the predict the next token objective. This is done in Othello and chess, and the theoretical formalization paired with the empirical results strongly suggest that GPT style models do learn a causal world model. The strong results in Figures 4, 5 support this, and they even show this extends to out of distribution outputs (Figure 7).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and suggestions for improvement. Your suggestions will improve the clarity and emphasize the significance of the proposed approach and findings presented in the paper.\nWe also thank you for the many suggestions for future work.\n\n**Re Other Comments or Suggestions**\n\nUpon your interesting suggestions, in our future work we will extend the analysis for OOD data in different domains. Specifically, we plan to explore improvements for pre-training relying on the findings in the paper. For example, we plan to explore the effect of OOD generalization after calibrating token-generation accuracy with respect to causal confidence (Equation 10).\nFinally, we will improve the clarity of figures' captions as requested and correct grammatical errors and typos.\n\n**Re Questions for Authors**\n\n* The full set of CI tests having all possible conditioning set sizes has a one-to-one correspondence with a causal graph (up to Markov equivalence). A causal-discovery algorithm uses a sub-set of this exhaustive set of CI tests. A sound and complete algorithm, like the one we use in the paper, creates a one-to-one correspondence between this subset and a Markov equivalence class. We will add this clarification along with a citation (Spirtes et al., 2009).\n\n* Yes. Positional encoding specifically for the board was not used (in accordance with the work of Li et al., 2023). Positional encoding was (also) not used by Li et al., 2023 who trained a GPT without domain information, such as the presence of a board and its 2D alignment on which the game moves take place. We will clarify this.", "claims_and_evidence": "The claims in the submission are supported by strong and clear evidence. The authors support their claim regarding causality both with a theoretical formalization and then empirical results.", "methods_and_evaluation_criteria": "The benchmark datasets, consisting of Othello and Chess, make sense for the problem of determining whether world models learn a causal structure.", "theoretical_claims": "I briefly checked through all of the proofs for theoretical claims and did not find any issues", "experimental_designs_or_analyses": "I thoroughly checked the validity/soundness of all experiments regarding Figures 3-5, 7-8. I did not see any issues and believe the experimental design strongly support the claims.", "relation_to_broader_scientific_literature": "There are two main ways this is related to the broad scientific literature. First, is the relation to world models and autoregressive (GPT) style models. Second, is the relation to causality/causal inference. The results bring link two domains together, and effectively demonstrate that GPT style autoregressive models can learn a world model which understands causal factors at play and can reason about what this causality implies.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- Very important work to formalize theoretically that GPTs can learn a causal world model\n- strong theoretical analysis of causality in attention and generally strong results (Figures 4, 5) to support them\n\n## Weaknesses\n- It's still unclear the extent to which generalization occurs, even though we know it worked for the OOD test set\n- Although in Figure 3 the results are statistically significant, they were not statistically significant very often, especially since the p value was not very high at 0.05%.\n- It's unclear whether these results would hold for language models not trained solely on chess/othello data, or data that is noisy/imperfect", "other_comments_or_suggestions": "- Figure out a better way other than n grams to show the test and train distributions are different\n- Do more experiments regarding the extent to which OOD generalization occurs\n- The figures were not very clear to me, the captions could have better encapsulated what was being done.\n- It would have been nice to see non game benchmarks, i.e. math benchmarks or some other problems with causal structure, but the benchmarks were sufficient for the claims.", "questions_for_authors": "- The paper states \"determined threshold of significance level (α). It is important to note that there is a one-to-one correspondence between the results of these CI test and the entailed causal structure. That is, a causal structure can be represented uniquely by a set of CI tests and their results.\"\n\t- How? This needs a citation, or is there a mistake?\n- Why is positional encoding not used?\n\t- Do they mean positional encoding specifically for the board?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0tM8BwbSXR", "reviewer": "Reviewer_v1kX", "summary": "The paper investigates whether a GPT trained for next-token prediction implicitly learns a causal world model, using an interpretation of the attention matrix as encoding a linear Gaussian SCM, first proposed in Rohekar et al. (2024). The introduce a causal discovery method for learning partially oriented causal graphs from the attention matrix using conditional independence testing. To indicate that the network is learning to represent a causal graph for a given sequence, they introduce a `structural confidence score’ R(A) which is the entropy difference between the conditional independence test p-values for detected dependencies and independencies. Experimental results show that sequences with higher structural confidence scores correlate with correct legal move predictions, and that pruning low-confidence attention heads does not affect performance, whereas pruning high-confidence heads does.\n\n\n## update after rebuttal\n\nFollowing the authors rebuttal, I have updated my score and am leaning towards acceptance, but with a low confidence due to my own lack of familiarity with the mechanistic interpretability aspects of the paper and how faithfully they map over to the causal claims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback, insights, and important questions. Addressing your review improves the over clarity of the paper and emphasizes the significance of the contribution. \n\n# Re Questions for Authors\n\n1. The relation $\\mathbf{D}^{-1}\\mathbf{A} = (\\mathbf{I}-\\mathbf{G})^{-1}$ is not an assumption, but rather a result by considering each token as a node in an SCM (the sentence just before equation 8). Since attention calculates $\\mathbf{Z}=\\mathbf{A}\\mathbf{V}$ (Equation 1) and the SCM calculates $\\boldsymbol{X}=(\\mathbf{I}-\\mathbf{G})^{-1}\\boldsymbol{U}$ (Equation 4), we equate the outputs: token embedding and SCM node value. This is described two sentences before Equation 9, but we will clarify this point more explicitly in the paper.\n\n2. A ground truth of causal graphs is not available for most domains. In the strategy games of Othello and Chess, since there are multiple next legal moves, a causal graph over a given set of game moves may contain information about the strategy of the player (in case of a real game) in addition to the game rules. Moreover, it is unclear how to modify the game rules such that a sequence of game moves is legal for both the original rules and the modified rules, while making sure the set of next legal moves entailed by the modified rules does not overlap with that entailed by the original moves. That is, for a fair comparison the input sequence should be legal for both the original and intervened rules, and for correctly testing, the next legal move should be different. Nevertheless, at the request of Reviewer g9Sr, we evaluated the accuracy of legal move generation as the function of confidence score (Section 4.3) calculated for correlation-based methods they suggested. See also Figure 3a vs. Figure 3d in the ablation study (Section 5.2).\nWe believe your concern is additionally addressed in our reply to their Questions 1 and 2. We did not find an increase in legal move generation accuracy as a function of the confidence score for correlation-based methods (as found for the learned causal graph).\n\n3. For a given sequence of tokens and a legal next token generation, there may be multiple, equally minimizing the loss, attention matrices. During pre-training, these attention matrices were not restricted to have high confidence for determining high-order conditional independence (CI) relations between tokens (in comparison to marginal pair-wise correlations). We examined confidence of attention matrices generating legal vs illegal token generation. In all our experiments we found a correlation between the causal confidence score, which was calculated during causal structure learning, and the accuracy of generating legal moves. Confidence decreases when uncertainty increases in CI tests decisions used for constructing the causal graph structure (e.g., an edge is removed if conditional independence is found, and edge orientation is determined based on the conditioning set that disconnected two nodes). We will add a corresponding clarification this in the paper.\n\n4. We provided a qualitative examples of causal graphs learned for the first few moves in Othello and Chess games in Figure 1 and Figure 6, respectively. These graphs are actual results. After these few moves it becomes challenging to follow the true causal relations. For example, during the Othello game, previously placed pieces may flip their color multiple times affecting the set of legal moves. Next, we regard the possibility of having high $R(\\mathbf{A})$ due to the last few moves effecting the next move. In our Ablation study in Figure 3 we compare (a) marginal (in)dependence relations to (d) conditional independence relations that are required to construct causal graph. It is demonstrated that the difference in $R(\\mathbf{A})$ between legal and illegal token generation is more prominent when considering independence relations required for constructing a causal graph. As mentioned in our reply to your Question 2, we also examined the experiment in Figure 5 for correlation-based methods.\n\n5. The main difference between the causal interpretation and other interpretation is the incorporation of conditional independence relations having various conditioning set sizes rather that treating the attention values as weights. As mentioned in our answer in Question 1, by relating output embeddings computed using the attention matrix to the node values in an SCM we obtain the relation  in Equation 9. By equating the covariance matrices (Equation 8) we can employ a constrain-based causal discovery algorithm. In our experiments we demonstrated that a confidence score computed from conditional independence relations provides a better differentiation between legal and illegal token generation compared to marginal independence relations.\n\nFinally, we found the papers you suggested to include the the introduction to strengthen the significance of this paper's results. As suggested, we will add a related discussion.", "claims_and_evidence": "The main claims are supported by correlating high R(A) with legal move accuracy. However, it is not immediately clear to me why this implies the causal interpretation is valid, when this correlation could have other explanations. It is also not immediately clear to me why this was the experiment they chose to validate the interpretation, rather than the more obvious ones of comparing the learned causal structure to the true causal structure, or making interventions and seeing how the learned causal structure changes (see questions below). \n\nOverall, I find the proposal quite compelling but am confused by the way they have gone about validating it.", "methods_and_evaluation_criteria": "The choice of domains is reasonable and has been studied before in this context", "theoretical_claims": "The key theoretical assumption, that D^-1 A \\simeq (I-G)^-1, is plausible but lacks rigorous derivation beyond conceptual arguments. It would be beneficial to validate this claim on synthetic data with a known causal structure.", "experimental_designs_or_analyses": "I have not assessed these in depth, but they appear to be rigorous, confidence intervals are given, etc.", "relation_to_broader_scientific_literature": "The work aligns with recent findings that transformers learn implicit world models (Li et al., 2023; Nanda et al., 2023) and extends these ideas by applying an adapted version of Rohekar et al. (2024) for GPT models. The paper does not cite alternative interpretations of attention (e.g., information flow, memory retrieval, statistical smoothing), and it may be worth discussing how the causal interpretation of attention relates to these other interpretations.", "essential_references_not_discussed": "There are some papers the authors could discuss below, that motivate for the papers results (i.e. why should we even expect the network to have learned causal structure), [1] showing that this is required for out of distribution generalization, and [2] giving examples where causal world models are detected using mechanistic interpretability. These are not essential, but could strengthen the argument. \n\n[1] Robust agents learn causal world models (Richens et. al). \n[2]Transformers Use Causal World Models in Maze-Solving Tasks (Spies et. al.)", "other_strengths_and_weaknesses": "I find the approach compelling and the overall quality of the paper good, but it is perhaps somewhat incremental in extending Rohekar et al. (2024) to GPT models?", "other_comments_or_suggestions": "There are lots of grammar errors in the paper, which made some key parts of it hard to follow. \nAbstract “Are generative pre-trained transformer (GPT) models, trained only to predict the next token, implicitly learn a world model from which a sequence is generated one token at a time?”, suggest replacing “Are” with “Do”. “suggesting a causal world model that arises from this interpretation.” this sentence doesnt make sense to me,that the causal world model arises from your interpretation? \n“Note that even if some of the nodes are latent confounders is still (I − G) −1 triangular”\nMany more, would recommend some polishing", "questions_for_authors": "1. Can you provide a rigorous justification for the claim that D^-1 A \\simeq (I-G)^-1 ? Have you tested this on synthetic data with a known causal graph, and found that this correspondence approximately holds? \n2. Im confused why you don’t include an interventional study, where you change the game rules (e.g., modifying Othello transition dynamics) to verify if the causal structure you recover using your interpretation changes according to this intervention. This feels like much more direct evidence for the causal interpretation than correlating the models ability to predict legal moves for a given sequence with the certainty in the causal structure returned by your causal discovery algorithm.\n3. “in cases where the confidence in recovering structures from the attention matrices is low, GPT generally fails to generate a token that adheres to the game rules”. This doesnt obviously imply that “GPT implicitly learns to represent causal structures in attention heads”. Can you explain this connection more? \n4. How does the learned causal structure compare to the actual game rules? Could you provide qualitative examples of inferred causal graphs? It is not clear from R(A), which measures only how concentrated the p-values are for the conditional independence tests, that the model is learning the correct causal graph. Are there other reasons that a high R(A) could correlate with legal moves? E.g. move legality is determined by a few most recent states, resulting in a high localised attention pattern, which could perhaps show up as a high R(A)?\n5. Given that multiple interpretations of attention heads exist (e.g., memory retrieval, information diffusion), what makes the causal interpretation the most compelling? (note [1] could give some justification). It could be beneficial to include in the paper a discussion of other interpretations, and how these could explain the observed correlation between R(A) and performance? (e.g. consider in your analysis alternative hypotheses beyond the causal structure learning hypothesis you are proposing).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dk9utwc7Zn", "reviewer": "Reviewer_B5pU", "summary": "N/A", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your review and your perspective on the paper. We value your feedback and believe the following answers your concerns.\n\n* Re first point. We would like to clarify that autoregressive generation is not inherently causal. Often the attention in GPT is called 'causal' but it only means that a token is a function of previous tokens in the sequence. That is, the upper triangular part of the attention matrix is masked (zero). However, this is an oversimplification of the broader term \"causal\". In the paper we show that an attention value $\\mathbf{A}_{i,j}$ describes the sum of all directed paths from node $j$ to node $i$, that is the effect node $j$ has on node $i$ accounting for all directed paths. Another difference between the attention and causal graphs is the set of encoded conditional independence relations. For example, consider the causal relations $X$ causes $Z$ and $Y$ causes $Z$, and no other relations: a graph: $X$ --> $Z$ <-- $Y$. The graph entails that $X$ and $Y$ are marginally independent, but dependent conditioned on $Z$. However, note that several causal graphs may entail the exact same independence relations (e.g., a hidden confounders may be present between $X$ & $Z$ and $Y$ & $Z$ instead of causal relations). The attention matrix only represents the correlation between $X,Y$ and $Z$ (the level of 'attention' $Z$ gives to $X$ and $Y$) and does not represent causality. The causal interpretation in this paper demonstrates that a causal graph, including hidden confounders, could be extracted, where the attention matrix represents the total effect a node has on another through all directed causal paths. That is, the $(i,j)$ element in the attention matrix $\\mathbf{A}$ represents the sum of all the directed paths from node $j$ to node $i$ in the causal graph. We also introduce a novel confidence score for the learned graph using entropy of p-values of statistical tests used during the causal structure learning.\n* Re second point. We will clarify in the paper that a 'causal world model' describes the causal mechanism underlying the observations as well as probability distribution of hidden variables, as often used in the causal inference literature. Specifically, we will note that it is assumed that underlying each input sequence there exists a corresponding structural causal model (SCM, Section 3.2). That is, the causal world model consists of the causal mechanism that generates tokens.\n* Re third point. We discuss implications of the findings in Section 6 (Conclusions) and in Section 'Impact statement\" (after Section 6. Conclusions; part of ICML 2025 template which does not count towards page limit). The implications include a) the ability for zero-shot causal discovery which can be beneficial in various scientific domains (e.g., understanding the mechanism by which a GPT trained on the protein space generates novel protein sequences [1]), b) measuring uncertainty of attention heads per input sequence, c) calibration during training (accuracy vs. causal confidence), and d) better human supervision by examining the causal mechanism by which a token is generated. \n\n[1] Ferruz, N., Schmidt, S., & Höcker, B. (2022). ProtGPT2 is a deep unsupervised language model for protein design. Nature communications, 13(1), 4348.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "While I acknowledge my limited expertise in this domain and therefore express low confidence in this review, I have several concerns that lead me to recommend rejection. My concerns are as follows:\n\n- First, I struggle to identify the core contribution of this paper. The authors explain the relationship between GPT and causal learning, but this connection seems rather obvious. GPT's autoregressive generation is inherently causal, and its architectural design enables efficient generation through causal mechanisms. The fact that GPT can learn causal knowledge about the real world is well-established. The attention mechanism learns token similarities through its similarity matrix, from which causal structures can naturally be extracted. None of this is surprising.\n\n- Second, the paper's treatment of \"world model\" lacks precision. While this term is currently popular across various domains including robotics, autonomous driving, video generation, and 3D generation, it lacks a clear definition. \n\n- Third, the practical implications of these findings are unclear. Understanding the relationships between GPT, causal learning, and world models, but how does this knowledge advance the field? Can it guide the development of better GPT models for real-world applications? The paper lacks substantial discussion of these practical considerations. \n\nDue to my limited knowledge in this field, my review may be biased, and I welcome corrections from the authors and other reviewers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBS8MYzxU2", "reviewer": "Reviewer_g9Sr", "summary": "This paper explores whether GPT models, designed for next-token prediction, implicitly learn a causal world model from which sequences are generated. The authors derive a causal interpretation of the GPT attention mechanism and suggest that GPT models can be used for zero-shot causal structure learning with a proposed confidence score. The study is conducted in a controlled environment using Othello and Chess games, where a GPT model trained on real-world games is tested on synthetic data sequences of random legal moves. Results indicate that the GPT model can often generate legal next moves for out-of-distribution sequences by capturing causal structures encoded in the attention mechanism, but struggles and fails to capture causal structures when generating illegal moves. The introduction elaborates on GPT's unexpected capabilities beyond next-token predictions, proposing that these abilities might arise from implicit learning of causal structures during pre-training due to Occam's razor, which favors simpler, more compact solutions. The paper builds on recent methods for causal interpretation in transformer models, adapting them to GPT, and investigates the correlation between GPT-generated errors and the uncertainty in representing causal structures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thorough review and for the clear and detailed suggestions. We believe these suggestions further highlight the significance of the causality-based approach presented in the paper compared to correlation-based approaches.\n\n# Answer to questions\n## Re Question 1 and 2\n\nFirstly, under the assumption made in the paper that hidden confounders may be present, a correlation-based method may not be applicable. The method we presented in the paper employs a causal discovery algorithm to extract a graph, accounting for the possible presence of latent confounders, from the attention matrix.\n\nAs described below and empirically demonstrated, accuracy in generating legal tokens is more strongly associated with the causality-based approach presented in the paper than to the correlation-based methods, as measured by the confidence score (Section 4.3). \n\nYou suggested 2 correlation-based methods to derive an adjacency matrix. The first is thresholding the attention matrix, and the second is thresholding the precision matrix. The first method is related to testing marginal independence (empty conditioning set), which was examined in Section 5.2.1, Figure 3a. In the second method the precision matrix represents pair-wise independence conditioned on all other nodes (only the largest possible conditioning set size, $n-2$). In relation to these methods, constraint-based causal discovery tests independence conditioned on a range of conditioning set sizes from 0 to the maximal possible, as needed. In Section 5.2.1 Figure 3 we demonstrate that statistical significance results obtained using empty conditioning sets are different than those obtained using size 1 (Figure 3a-3c). The aggregation of results from CI tests required by causal discovery improves the number of statistically significant cases (Figure 3d) differentiating legal from illegal move generation.\nFurthermore, given a graph $\\mathbf{G}$, nodes' values are $\\boldsymbol{X}=\\boldsymbol{X}\\mathbf{G}+\\boldsymbol{U} \\implies  \\boldsymbol{X}=(\\mathbf{I}-\\mathbf{G})^{-1}\\boldsymbol{U}$ (Equation 4). Treating attention as $\\mathbf{G}$, instead of $(\\mathbf{I}-\\mathbf{G})^{-1}$ imposes restrictions on the graph.\n\n### Empirical Evaluation\nFor each game length we calculated confidence (Section 4.3). As in Figure 5, we binned confidence values and calculated mean accuracy. Results for Chess are in the tables below. \nIt is evident that, except for a few cases, for both correlation-based methods there is no clear trend of accuracy with respect to confidence, as clearly evident for causality-based approach (Figure 5).\n\n**Attention thresholding**\n|Len|Bin 1|Bin 2|Bin 3|Bin 4|Bin 5|Bin 6|Bin 7|\n|-|-|-|-|-|-|-|-|\n|10|0.994|0.992|0.987|0.993|0.996|0.933|0.933|\n|15|0.998|0.987|0.977|0.967|0.978|0.988|1.000|\n|20|0.982|0.974|0.975|0.978|0.979|0.967|0.973|\n|25|0.889|0.960|0.964|0.957|0.969|0.986|1.000|\n|30|0.867|0.954|0.942|0.950|0.953|0.950|0.963|\n|40|0.667|0.980|0.970|0.943|0.938|0.928|0.938|\n\n**Precision-matrix thresholding**\n|Len|Bin 1|Bin 2|Bin 3|Bin 4|Bin 5|Bin 6|Bin 7|\n|-|-|-|-|-|-|-|-|\n|10|1.000|0.988|0.988|0.989|0.994|0.990|0.988|\n|15|1.000|0.982|0.977|0.980|0.982|0.979|0.975|\n|20|0.961|0.967|0.971|0.967|0.973|0.969|0.963|\n|25|0.941|0.956|0.964|0.963|0.966|0.967|0.980|\n|30|0.961|0.953|0.949|0.963|0.967|0.920|0.961|\n|40|0.945|0.940|0.936|0.934|0.925|0.882|0.933|\n\n## Re Question 3\nA graph constructed using the last layer attention represents a graph over the output tokens.\nThe last layer's attention output represents tokens (pre-training loss compares output tokens to input tokens), up to an MLP non-linear mapping, where MLP has no effect on inter-token relations and therefore does not influence causal structure learning. Earlier layers represent context (such as exogenous nodes in the SCM) for their following layer. See also the explanation following Equation 8. For example, in Othello intermediate layers may represent the board (as exhaustively tested and found by Li et al., 2023) which serve as context for the causal graph over game moves represented by the last layer. Furthermore, recent studies empirically show that the last layers have more weight in determining the next token, specifically in Othello GPT [1] and in LLMs in general [2,3,4].\n\n[1] How GPT learns layer by layer. Du et al., 2025.\n[2] Adaptive Large Language Models by Layerwise Attention Shortcuts. Verma et al., 2024.\n[3] The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights. Aljaafari et al., 2024.\n[4] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers? Jin et al., 2024.\n\n# Re Other comments or suggestions:\n* In Eq 9 both RHS and LHS are matrices. A hat over index $i$ represents the omission of the $i$-th row/column, as implicitly mentioned in the sentence before the equation. We will explicitly describe this notation.\n* We will clearly define the computation of the entropy values $H_\\textrm{ind}$ and $H_\\textrm{dep}$.", "claims_and_evidence": "Partly. Please see questions.", "methods_and_evaluation_criteria": "The proposed method is simple yet effective. However, I have a question about whether the results can only be obtained through causality or if they can also be achieved using correlation-based methods. Please see questions.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "I have checked the soundness of experiments.", "relation_to_broader_scientific_literature": "To the best of my knowledge, this is the first paper introducing causality to study the world model in a controlled environment. If the authors can demonstrate that causality is necessary, I believe it will inspire follow-up research and have a significant impact in this area.", "essential_references_not_discussed": "No to the best of my knowledge.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. This paper is the first to introduce causality into the controlled study of the world model.\n  \n2. It is interesting to see that attention weights pruning can improve the legal move, and that legal moves are correlated with confidence.\n\n**Weaknesses:**\n\n1. For the results discussed in Sections 5.2 and 5.3, I would like to know whether a causality-based method is necessary. I will improve the score if the authors can address this point. (Please refer to Questions 1 and 2)\n\n2. The authors state that \"they do not explain how the board game is encoded within the attention matrices and why the attention mechanism can represent the board state\". But the authors do not explain these questions clearly. (Please refer to Question 3)", "other_comments_or_suggestions": "In Eq (9), the LHS is a matrix while the RHS is a value. \n\nThe computation of $H_{ind}$ and $H_{dep}$ should be clearly defined in the main text.", "questions_for_authors": "It would be interesting to explore whether correlation-based methods can reproduce the results presented in Sections 5.2 and 5.3. Some simple correlation-based methods include:\n\n1. **Attention as the \"causal structure\" $\\mathcal{G}$**:\n   1. Let $G$ be the adjacency matrix of $\\mathcal{G}$. Set $G_{ij} = 1$ if $A_{ij} > \\frac{\\alpha}{i}$.\n   2. Repeat the algorithm in Section 4.3 to compute the confidence.\n   3. Check whether the results in Sections 5.2 and 5.3 can be reproduced.\n \n2. **Precision matrix as the \"causal structure\" $\\mathcal{G}$**:\n   1. Compute the covariance matrix based on attention $C = [D^{-1}A][D^{-1}A]^T$.\n   2. Recover the precision matrix $\\Theta$ using an existing library (e.g., [Ref. A]), and set $G_{ij} = 1$ if $\\Theta_{ij} > \\alpha$. If $C$ is well-conditioned, one can also use $\\Theta = C^{-1}$.\n   3. Repeat the algorithm in Section 4.3 to compute the confidence.\n   4. Check whether the results in Sections 5.2 and 5.3 can be reproduced.\n \n3. The authors use the last layer attention. Why is only the last layer useful? The authors should explain the functions of the previous layers' attentions and MLPs.\n\n[Ref. A] https://github.com/choltz95/sparse-precision-matrix-estimation/tree/master", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates an important and interesting research question: Can GPT-style models trained only to predict the next token implicitly learn a world model? The authors provide an interpretation of the attention mechanism in GPT from a causal perspective to validate this issue. In addition, the experimental results are strong, which can support the authors' claim.

In the rebuttal period, the authors have made substantial efforts to address the reviewers' concerns and provided a detailed response, including additional experimental results to further verify the effectiveness of the proposed approach, as well as a clear explanation of concepts, definitions, and empirical findings. Most of the major concerns raised by the reviewers have been addressed.

Given the positive feedback from all reviewers and the improvements made, I recommend accepting the paper for publication.
Decision: Accept (poster)