Verification of the Implicit World Model in a Generative Model via Adversarial Sequences
Authors: András Balogh
Márk Jelasity
Venue: ICLR 2026 Poster (2026)
Keywords: generative sequence model
implicit world model
adversarial sequences
chess
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Generative sequence models are typically trained on sample sequences from natural or formal languages. It is a crucial question whether&mdash;or to what extent&mdash;sample-based training is able to capture the true structure of these languages, often referred to as the "world model". Theoretical results indicate that we can hope for soundness at best, that is, generating valid sequences, but not necessarily all of them. However, it is still important to have practical tools that are able to verify whether a given sequence model is sound. In this study, we focus on chess, as it is a domain that provides enough complexity while having a simple rule-based world model. We propose adversarial sequence generation for verifying the soundness of the sequence model. Our adversaries generate valid sequences so as to force the sequence model to generate an invalid next move prediction. Apart from the falsification of soundness, this method is also suitable for a more fine-grained analysis of the failure modes and the effects of different choices during training. To demonstrate this, we propose a number of methods for adversarial sequence generation and evaluate the approach on a large set of chess models. We train models on random as well as high-quality chess games, using several training recipes. We find that none of the models are sound, but some training techniques and dataset choices are able to improve soundness remarkably. We also investigate the potential application of board state probes in both our training and attack methods. Our findings indicate that the extracted board states have no causal role in next token prediction in most of the models.

Published as a conference paper at ICLR 2026
VERIFICATION OF THE IMPLICIT WORLD MODEL IN A GENERATIVE MODEL VIA ADVERSARIAL SEQUENCES
András Balogh' and Márk Jelasity
I 'University of Szeged, Hungary "HUN-REN-SZTE Research Group on AI, Hungary (abalogh, jel jelasity]@inf www.zu-dzz.erree
ABSTRACT
Generative sequence models are typically trained on sample sequences from nat- ural or formal languages. It is a crucial question whether-or to what extent- sample-based training able to capture the true structure of these languages, of- ten referred to as the "world model". Theoretical results indicate that we can hope for soundness at best, that is, generating valid sequences, but not necessarily all of them. However, it is still important to have practical tools that are able to ver- ify whether a given sequence model sound. In this study, we focus on chess, it is a domain that provides enough complexity while having a simple rule- based world model. We propose adversarial sequence generation for verifying the soundness of the sequence model. Our adversaries generate valid sequences to force the sequence model to generate an invalid next move prediction. Apart from the falsification of soundness, this method is also suitable for grained analysis of the failure modes and the effects of different choices during training. To demonstrate this, propose number of methods for adversarial sequence generation and evaluate the approach on large set of chess models. We train models on random as well as high-quality chess games, using several training recipes. We find that none of the models are sound, but some training techniques and dataset choices are able to improve soundness remarkably. We also investi- gate the potential application of board state probes in both our training and attack methods. Our findings indicate that the extracted board states have no causal role in next token prediction in most of the models.
fine-
INTRODUCTION
Generative sequence models like large language models see increasingly more use in areas where a solid understanding of complex concepts and interactions is critical for their success (Lin et al.) 2023 Nijkamp al., 2023) et 2022b). Recent findings suggest that such important capabilities might naturally emerge during training, yet our understanding of how this knowledge is represented and used by models is still rather limited (Zheng et al., 2024 Schaeffer et al.) (023).
An interesting aspect of this problem is whether the emergent capabilities of generative models are based on some representation of a system of world-states and transitions, and so, whether this implicit world model is consistent with reality (Vafa et al., 2024). In order to study implicit world models, recent works proposed the use of synthetic tasks like board games that can described by formal languages, where the world model is explicitly known (Li et al. [023)] Toshniwal et al.) 2022). We can then compare the behavior generative models to the true world model.
However, it is difficult to test if the implicit world model of the generative model is sound, that is, whether adheres to the true world model. To tackle this problem, we propose a novel methodology based adversarial sequence generation, where adversary generates valid sequences with the aim of forcing the model to break the formal rules of the true world model.
We examine generative sequence models in the domain of chess, using diverse datasets and various training recipes that facilitate learning the true world model. Our methodology reveals a low level of soundness across the board, along with numerous novel insights into the (lack of) causality of world

---

Published as a conference paper at ICLR 2026
state probes, the roles of different training objectives, and the impact of dataset choice, such size and semantics.
Our contributions are as follows
We present a novel adversarial framework for measuring the soundness of implicit world models and evaluate it in the domain of chess.
We perform a large-scale empirical study, We introduce several training schemes that fa- cilitate learning the true world model over datasets of varying sizes and qualities.
We analyse the models with our adversarial methodology. and show that none of them are sound, but the choice of training recipe, dataset, and adversary has significant effects.
We examine the role of linear board state probes during training and evaluation, and find that they have limited causal connection with the predictions of the models.
1.1 RELATED WORK
Instead of explicit generative world models (Ha & Schmidhuber 2018 Zeng et al. 2023), our paper focuses implicit world models learned by generative models (Li al. 2023 Vata et al. 2024).
One way to verify the soundness of these internal world models to extract them via mechanistic- (Bereska & Gavves 2024 Nikankin et al.) 2025), or conceptual interpretability methods (Patel &
Pavlick 2022). Of the latter, linear probing (Alain & Bengio 2017 Hewitt & Manning/ 2019) has been used extensively to decode learned concepts from feature representations (Abdou al. 2021
[] et al. 2021: Hewitt& & Liang. 2019 Feng et al. 2025). However, the causal role of board states extracted with probes is not evident, as the main goal of probes is analysing if, and how information is encoded, rather than how it is used (Belinkov & Glass 20019 Belinkov] 20222). Therefore, probes might incorrectly indicate the soundness of implicit world models (Vata et al. 20244.
Another approach to verification is to compare the outputs of the model with formal structure that defines the true world model, such as automata (Liu al.] 2023] Laufer & Kleinberg 2025). or formal rules (Sun et al. 2024] Wolfram & Schein (2025). Vala et al. (2024) propose a framework based on sequence-level distinctions to evaluate whether a language model learned the automaton of
the true world model, and show that generative models fail to do so. We extend this line of work with
a novel approach to implicit model verification that does not rely on sensitive threshold parameters to define the generated language.
Board games have been extensively used evaluating the emergent capabilities of language models (Krvvnenn et al., 2024). Li et al. (2023) successfully train language models on Othello transcripts, and they, along with Nanda et al. (2023) and Hazineh et al. (2023) argue that linear board state probes have causal connections to the model's function, while jylin04 al. (2024) show that the implicit world model of OthelloGPT is fragmented. Toshniwal et al. (2022) and Karvonen (2024) train language models chess transcripts and argue through output- based and probing methods that these models have emergent world models that are consistent with the true world model.
2 PRELIMINARIES AND NOTATION
Informally speaking, we assume that there is ground truth world model, and we train a sequence model based only on action sequences generated by this world model. Starting from a (hidden) initial state, an action sequence is recorded by following legal state transitions allowed by the possible actions in the world model. We then ask whether the implicit world model learned from a set of action sequences is consistent with the ground truth world model. Let us elaborate on this setup more formally and present an application as well: the game of chess.
2.1 WORLD MODELS AND GENER ATIVE MODEL.
ag be an action sequence, where A > 0 and V,a; E Σ. Let the set of all possible action sequences be denoted by Σ".
Let Σ be the finite set of all actions in a world model. Let ag..a
'Our code, models, and datasets are availahle at http://gtthhccammaraaaaa/araaaat
2

---

lokens, in which Published as a conference paper at ICLR 2026
We assume that the true world model is given through the function W, where ((a1..0)) C E is the set of valid continuations of the sequence 01..0g- We say that a sequence a1..0g is valid and only if E ((a.-00-1)) for all 0 < i S (by definition, the empty sequence (i.e. lo 0) is valid).
Given a set of valid sequences, we can train a generative model M > - A(Σ), which is a model that predicts probability distribution over Σ, given action sequence. Let ((as) denote the conditional probability assigned to a E E by the model, given $ E E" When generating a sequence, we need decoding policy m : Σ E. For example, the greedy decoding policy is mms) arg max, M (a]s).
Note that it is possible that one action is represented by sequence of two case model training and prediction should be understood at the token level.
Definition 2.1. A generative model M with decoding policy m is sound with respect to the true world model W if and only if for any sequence s that is valid in and (s) # 0, we have m(s) E (s).
Focusing on soundness. Our problem formulation focuses on the verification of soundness, that is, examining whether the generative model generates only valid sequences. Our method will be able to disprove soundness by searching for counterexamples in the form of valid sequences, for which the sequence model predicts invalid continuations. We note that sound and complete generative models (ones that are identical the world model) are theoretically impossible learn from samples, even for regular languages (Gold 1967) while sound models are at least theoretically possible under reasonable assumptions (Kleinberg & Mullainathan 2024).
Scope. In our formulation, we define (s) the valid continuations of without any restrictions on the complexity the true world model question. As result, our framework generalizes to settings where the true world model is more complex (e.g., pushdown automaton), as opposed to the framework of (Vafa et al. 2024), which requires the true world model to be a deterministic finite automaton.
2.2 CHESS NOTATION
We focus the game of chess due to its clear and deterministic set of rules that form a ground truth world model of the type introduced above.
Like Toshniwal et al. (2022). we use the Universal Chess Interface (UCI) notation to represent actions (moves). This notation combines the starting and destination squares to represent a move. For example, the notation e2e4 means the player moved the piece on e2 to eé. Special moves and events (e.g., castling, check, and checkmate) are not explicitly encoded, with the exception of promotion, where the piece type the pawn is promoted to is indicated the end of the move. For example, the notation a a means the pawn on a 7 was moved to a 3 and got promoted to queen.
2.3 BOARD STATE DECODERS
To analyze the soundness of implicit world models, some of our algorithms rely on a board state decoder B that is implemented predict the current board state B(M, s) from a hidden representation within M after a sequence of moves Most often, the decoder is simple linear probe Alain & Bengio (2017) that solves a 13-class classification problem for each of the 64 squares on the board independently, where the classes represent the six piece-types for the two sides, and the empty square.
extra head added to a generative model M, and trained to
We will also use the loss function (B( s) in some of our algorithms that measures the error between the true board state after move sequence and the predicted board state B(M, s).
ADVERSARIAL VERIFICATION OF SOUNDNESS
Our goal is to evaluate whether generative model generates only valid sequences (i.e., its implicit world model is sound) and, if not, we also interested in the extent of the inconsistency.
3

---

Published as a conference paper at ICLR 2026
Sequence-level evaluation is essential. It has been argued by Vafa et al. (2024) that simple metrics like next-token prediction accuracy are misleading because even completely wrong models might have high accuracy. Therefore, there is need for sequence-level analysis and metrics. Our approach based on generating valid but adversarial sequences such that the generative model predicts an invalid continuation for the sequence.
A Advantages of adversarial verification. While Vafa et al. (2024) propose a theoretically motivated methodology verify and quantitatively characterize soundness, their approach requires the defi- nition of the formal language generated by the generative model, which in turn requires an ad hoc probability threshold parameter. At the same time, the adversarial sequences simply seek provide existential proof that the generative model is incorrect, avoiding the need for defining the gener- ated formal language exactly. However, as we will demonstrate, the method still offers quantitative metrics and detailed insight into the failure modes of different models through the fine-grained analysis of successful attacks.
3.1 THE ABSTRACT ADVERSARY
The key component in our framework is the adversary, whose goal is to force the generative model to generate an invalid next action. It is very important that the adversary itself will always produce valid sequences, but in a way so that the next action predicted by the attacked model is invalid.
While an adversary could check all valid sequence prefixes in W up some length to disprove the soundness of the generative model, this would not be efficient, or even plausible in most cases. Instead, given a sequence prefix a1..ag that valid in W, our adversary extends the sequence with 1 based on solving
0
max max f(M, ((M.01.020x+1). a641 EW
where f is an auxiliary function that attempts capture, for example, the uncertainty or incorrect- ness of the sequence model before or after the sequence aj..ag is extended with d:-1- Note that the maximization is done only over valid actions, so the function f itself does not capture validity, only an order of preference. We will describe multiple design choices for f in Section 3.2
The attack successful for some index j > the adversary can force an invalid next action. That is, the adversary finds a sequence s* a0.00 x+ where ((s") A 0 and m(s") ) & W(s*).
Two-player games. Since chess a two-player game, we adapt our attack framework accordingly by having the adversary play against the sequence model. The attacker always plays with white, so for all i 2 1, the move a2--1 given by Equation and d2; = m(a1...2x-1). The attack successful if, for some i 1, 02i is an illegal move.
3.2 ADVERSARY IMPLEMENTATIONS
First, we present three attacks, that is, three different implementations of in Equation T We then add two non-adversarial baselines as well for comparison.
Illegal Move Oracle (IMO). Our first attack is based on a very natural idea: the attacker picks the legal move that maximizes the conditional probability of an invalid continuation by the opponent. Formally.
fiMo 01.4444411))
=
max M(a++2lq1.0041)
(2)
6242 (011)
Board State Oracle (BSO). The attacker picks the legal move that maximizes the error of the board state predicted by a given probe B compared to the true board state. This attack is motivated by the hypothesis that the predicted board state has a functional role (a causal effect) on next-token Karvonen, 2024). To be more precise, we maximize the loss of the
prediction (Nanda et al. 2023
board state predictor:
Inso (M, (1-001111) L.(M, (1-a0))))))
(3)
4

---

Published as a conference paper at ICLR 2026
where (M a1.044))+1) is the classification loss of the probe's prediction after the moves a1, 0k+1-
Adversarial Detours (AD) by Vafa al. (2024). We include this attack for comparison with lated work. Here, the attacker picks the legal move with the lowest conditional probability according the sequence model:
(AD 01.00.0k+1) - (ax+1la1.00z)
Note that this attack is not directed explicitly towards forcing an error; instead, it attempts to guide the generation toward out-of-distribution (OOD) regions.
Random Move (RM). As simple baseline, the adversary randomly selects legal move in each attack step. That is, fRM random and independent its parameters.
Sequence Model Move (SMM). The attacker picks the legal move with the highest conditional probability according to the sequence model:
fMMM (M, a1.0010-11)) M(ast)/o...ae))
(5)
In practice, this has the effect of simply letting the sequence model generate the sequence, but correcting any incorrect moves by white. That is, the "attacker" is more of a benevolent oracle here.
4 OUR SET OF MODELS: ATTEMPTING TO LEARN THE WORLD MODEL
We train a number of models using different training recipes and datasets in order to evaluate the effect of number of design choices the quality of the implicit world model.
4.1 DATASET CHOICE
The curated datasets we used were the following: (1) MB-500k with 500k games from the Million- base dataset, consisting of high-quality games, used also by Toohniwal al. (2022); (2) Stockfish- 8M with 8M games generated by Karvonen] (2024), where the superhuman chess engine Stockfish played as white against engines of varying strength; and (3) Lichess- 16M with 16M human games obtained from the public Lichess database, also used in Karvonen 2024).
Random datasets. Motivated by the findings of [i et al. (002)) and Vafa al. (2024), who show that models trained random games learn the true world model better than those that were trained on curated datasets, we use random datasets as well. These contain 500K, 2M, and 10M valid random games, respectively, none of which end due resignation agreeing to draw.
Similar to Toshniwal et al. (022), we limit the length of every game in the training sets to 150 moves. Longer games removed from the datasets, and the dataset sizes are given after filtering. For more information about the datasets, please refer to Appendix
4.2 TRAINING OBJECTIVES
Tokenization. We use the tokenizer of Toshniwal al. (2022), where all squares (e.g. e2), and the four possible piece types in promotion (q, r, b and n) are represented as single tokens. Thus, all moves are encoded with either and end of the game, respectively, and separate PAD token for efficient training.
tokens. We also use BCS and EOS tokens to indicate the start
The next token (NT) prediction objective aims at predicting the next token after any prefix of any training sequence. While next token prediction is the usual choice, we introduce two additional training objectives that capture certain aspects of the true world model more directly.
The first is the probability distribution (PD) objective that aims capturing all the legal moves simultaneously. as opposed to training only on a single legal target token. This approach is motivated by Vafa et al. (2024), who explicitly define the generated language with the help of the predicted distribution. In the case of our tokenization choice, we need target distributions for move-starting and move-ending tokens. For move-starting tokens, the uniform distribution is used over squares
5

---

Published as a conference paper at ICLR 2026
Success rate of each attack strategy over all models. Bold and italic represent the highest and lowest success rates for model, respectively.
Table 1:
Random-500k,
Random-2M
Random- 10M
NT
PD
NT+JP
PD+JP
NT
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.954
0.975
0.954
0.984
0.854
0.931
0.848
0,930
0.673
0,874
0.699
0.839
SMM
0.419
0.881
0.493
0.810
0.408
0.943
0,476
0.954
0.172
0,900
0.192
0.845
IMO
0.996
0.999
0.996
1.000
0.999
1.000
0.997
0.999
0.972
0.992
0.976
0.988
BSO
0.886
0.875
0.816
0.872
0.779
0.858
0.745
0.901
0.541
0.811
0.528
0.757
AD
0.946
0.970
0.918
0.985
0.841
0.947
0.824
0,939
0.516
0.902
0.394
0.878
Millionbase-500k
Stockfish-8M
Lichess-166
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.823
0.994
0.818
0.998
0.149
0.913
0.176
0.931
0.107
0.914
0.075
0.852
SMM
0.513
0.794
0.494
0.846
0.190
0.911
0.267
0.931
0.287
0.897
0,211
0.859
IMO
0,999
1,000
0.995
1.000
0.631
1.000
0.662
0.998
0.387
0.995
0.349
0.987
BSO
0.524
0.885
0,547
0.938
0.105
0,753
0,147
0.882
0.057
0.764
0.043
0.795
AD
0.806
0.989
0.803
0.994
0.142
0.893
0.150
0.915
0.066
0.898
0.067
0.883
where the player has a movable piece. For move-ending tokens, the target is the uniform distribution over the possible destination squares for the selected piece specified by the previous token. case of third promotion token, the uniform distribution is used over the four possible piece type tokens.
The PD objective can also be seen as an explicit way of learning the transition rules of the true world model. This is particularly important for models that are trained non-random datasets of high-quality chess games, where the next token objective is highly biased by a strategic value function. That is, the next token objective does not allow the model to distinguish between illegal and strategically bad moves (Li et al. 2023 Vafa et al. 2024).
The second is the joint probe (+JP)) objective, motivated by recent advances in deep supervision, where training multiple heads on related auxiliary tasks has been used to achieve better performance and consistency (Li et al. 2022a; Zahorodnii] [2025] Huo al., 2025). We add linear board state probe to the model, and perform joint training by minimizing the combined loss the next-token predictor head and the board state probe. This method be seen as learning track the world
state explicitly.
Four objectives. We will use four training objectives, namely standard next-token prediction (NT), probability distribution prediction (PD), next-token prediction combined with board state prediction (NT+JP), and probability distribution prediction with board state prediction (PD+JP).
4.3 ARCHITECTURE AND HYPERPARAMETERS
Our models follow the GPT-2 architecture Radford et al. (2019) with 12 hidden layers, 768 hidden dimensions, and 12 attention heads, and total of 86M parameters. All models were trained for epochs with identical parameters, detailed in Appendix
Every model has an associated board state probe. Similar to Vafa et al. (2024), our board state probes take the transformer's 's last layer representation input. We only train and evaluate probes on move-ending tokens. If a joint probe was included in the training of a model, we use this probe in our probing experiments. Otherwise, train probe for the frozen generative model over 50K games from the model's training set. Further details are presented in Appendix [
5 EXPERIMENTAL RESULTS: ARE IMPLICIT WORLD MODELS SOUND?
Let us first consider the quality of our set of 24 models. Detailed measurements are provided in Appendix D Here, highlight that, although models trained smaller datasets (Random-500k and Millionbase-5000. achieve relatively low legal move ratios between 94.65% and 96.71% on their test sets, the models trained on large datasets (Random- Stockfish-8M, and Lichess- Lichess-16M)

---

Published as a conference paper at ICLR 2026
achieve ratio between 99.75% and 99.98%, so these models could be considered high-quality if one focused on this metric.
We evaluated every model using our various adversaries. We applied the greedy decoding policy, that is, m(ss = arg max, M (als), for all the models. With this policy, having the sequence model play against any non-random adversary results in deterministic sequence depending only the starting position. Thus, to evaluate the soundness of a model, we selected 1000 unique prefixes of 10 moves from the training dataset of the model, and performed our adversarial evaluation after each of these warmup sequences, which allowed us to collect statistics and gain fine-grained insights.
The results of the experiments shown in Table and Figure [ of the 5 adversaries against our 24 models, while the figure also shows the cumulative attack success rate as function of the number of moves after the warmup sequence.
The table shows the success rate
Clearly, the implicit world models are not sound. For most models, at least one adversary achieves close to 100% success rate, indicating severe inconsistencies between the implicit and the true world models. In the following, we make number of more fine-gradd tee-grained observations based on the results.
5.1 ADVERSARIES
IMO is always the strongest adversary, usually by a wide margin. Given that IMO directly steers the generative model towards an illegal move, this is not surprising; however, what is surprising is that the other two adversaries, namely AD and BSO, are rather weak. This showcases the need for strong udversaries in order to reliably verify generative models.
BSO shows a mixed performance, but sometimes it is weaker than even the most benign baseline SMM. This implies weak causal link between the correctness of the board state predicted by the probe and the legality of the move predicted by the generative model. We further investigate this phenomenon in Section 6
AD by Vata et al. (2024) consistently achieves success rates similar to Random Move (RM). An explanation could be that most moves with low conditional probabilities essentially random from the generative model's perspective. This also shows that a more aggressive attack, such as IMO, is essential for evaluation.
5.2 EFFECT OF TRAINING SETUP
Dataset size matters. According to Figure I] it is clear that increasing the dataset size very reliably increases the robustness our attacks. That is, large datasets increase the level of soundness. This is true independently of dataset type and training objective.
Random and curated datasets differ mainly when the next token objective is used for training. With the next token (NT) objective, models trained on curated datasets seem to be very robust, especially when large dataset used. At the same time, models trained on random datasets seem to be less robust under the NT objective, compared to the distribution objective PD, sometimes significantly so (see also Appendix E on this topic). However, in Section 7 we demonstrate that when executing the attacks using an out-of- diitof--ditribttion warmup sequence, the curated models are much less sound. We discuss the possible reasons Section 7
Multi-task learning does not help. Adding joint probe to the training scheme has negligible effect on the soundness of the implicit world model. We also investigate this in Section 6
Models overfit sequence length. Figure also reveals that many models, especially those trained on large datasets with the PD objective, strongly overfit the sequence length. This is evident from the fact that after exceeding the sequence lengths available in the dataset (up to 150), the models suddenly become extremely unreliable. This alarming finding suggests that the models do not use abstract board state representations internally that would be independent of sequence length.
6 ON THE CAUSALITY OF BOARD STATE PROBES
Here, we investigate the connection between the board state probes and the next-token predictor heads. As observed in Section 5 attacking the board state probe is not an effective strategy, and
7

---

Published as a conference paper at ICLR 2026
NT
PD
NT+JP
PD+JPP
Randem- 500K
0.5
0.5
100
120
160
120
0.0
120
140
0.0
100
120
140
1.0
1.0
1.0
1.0
Random-
0.5
0.5
0.5
0.5
20
60
60
120
1.0
80
80
80
1.0
10
60
69
60
100
120
1.0
1,0
1.0
1.0
Random-10n
0.0
100
120
0.0
199
110
0.0
120
0.0
100
140
11B-50KK
0.5
0.5
0.5
0.5
100
120
140
10
60
100
120
160
40
60
80
160
120
140
0.0
60
60
100
140
1.0
1.0
1.0
1.0
Stockdish-Bi
0.0
30
80
200
0.0
60
120
1.0
0.0
0.0
69
60
69
Lichess-160
850
SHN
0.0
0.0
0.0
0.0
000
-20
1.0
49
69
3,0
60
100
110
-00
19
6
100 20 ,00
Figure 1: Attack dynamics demonstrated by the move-wise attack success rate (ASR) for each dataset (row) and model (column). On each plot, the X-axis shows the move number, and the Y-axis shows the ASR attained by the attacks. Stronger attacks increase ASR more quickly, All lines stop the move when the attack reached its final ASR reported in Table [
Table 2: Mean sample-wise cosine distance of the gradients of the next-token head and the board state probe w.r.t. their input embeddings.
Table 3: Ratio of illegal moves under the BSO attack where the predicted illegal move is legal in the board state obtained via probing.
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
R-500K
0.989
0.986
0.982
0.979
R-500K
0.386
0.477
0.365
0.460
R-2M
0.990
0.989
0.988
0.992
R-2M
0.279
0.328
0.107
0.259
R-10M
0,989
0.989
0.992
1.000
R-10M
0.144
0.105
0.051
0.044
MB-500K
0.984
0.982
0.975
0.975
MB-500K
0,305
0,514
0.258
0.575
SF-8M
0.978
0.989
0.989
1.000
SF-8M
0.181
0.124
0.034
0.184
LC-16M
0.966
0.988
0.990
1.000
LC-16M
0.193
0.093
0.023
0.142
multi-task training with a board state probe (+JP) achieves negligible improvements in soundness. The former observation suggests weak causal link between the obtained board state and the pre- diction of the model, and the latter one provides additional evidence that the probe functions inde- pendently of the next-token predictor head.
Representation gradients. We investigate these hypotheses further using gradient- maadeent-based alignment analysis. Let assume that z(s) is the last token of the final-layer representation of M over action sequence s. In our setup, z(s) is also the input of both the next-token predictor head and the board state probe. We will consider the gradient of the loss terms according to a(s). namely
8

---

Published as a conference paper at ICLR 2026
OB Va(s)L((M, s) and 9NT Va(s) CNT (M, Depending on the model in question, LNT is either the hard next-token loss or the soft PD loss.
The heads are independent. We calculate the average cosine distance between 9NT and 9B 10,000 games from each model's training set and present them Table In all the cases-including the joint probe objectives- the gradient of the board state probe is almost orthogonal to that of the next-token head, which indicates that the two tasks rely on independent subspaces of the represen- tation. This finding is the exact opposite of the hypothesis that motivated the use of joint probe, namely that training a board state probe will encourage a better representation of the board state, thereby increasing the soundness of the implicit world model as well.
BSO attack success is mostly independent of probe. shows the ratio of those illegal moves enforced by the BSO attack that are also illegal according the board state probe. Especially for large datasets, this ratio is very low, indicating that even when the BSO attack is successful, it is not due misleading the board state predictor. This indicates that the probe more aligned with the ground truth than the model's prediction, further supporting limited causal link between the predicted board state and the model's prediction.
7
ARE SEEMINGLY SOUND MODELS REALLY SOUND?
Our results in Section indicated a surprising level of soundness for the models trained on the larger curated datasets, and especially on Lichess-16M with next-token prediction (NT). Here, we argue that these models are not tually sound. We hypothesize that the appar- ent soundness of models trained large rated datasets has to do with strong gravita- tion in-distribution trajectories. This, in turn, is most likely due to predicting not only legal, but also strategically good moves, resulting in a much more focused distribution that assigns high probability to far fewer moves than other models.
NT
NT+JP
DODDoMo Moves
1.0
1.0
0.5
0.5
10
A9
69
80
199
120
10
A9
60
80
100
OOD Moves
1.0
1.0
0.0
10
A0
10000MMa Moviis
1.0
0.0
20
0.0
50
60
30
To test this hypothesis, we applied random but valid warmup sequences that are not in the training set. We evaluated 1000 such out- of of-distribution warmup sequences of different lengths: 10, 20, and 30 moves per player. Ta- ble 4l shows the success rates of each attack, along with the difference from the original eval- uation with in-distribution warmup sequences,
Number
Move Number
Figure 2: Attack dynamics demonstrated similarly to Figure for NT (top row) and NT+JP (bottom row) models trained on the Lichess-16X dataset, evaluated with -ui-s--distiiuutionn (OOD) warmup sequences of varying lengths.
and Figure 2 shows the corresponding attack dynamics.
The attack success rate significantly increases make the initial board state and more out-of-distribuution This indicates that the model does not capture the true abstract transition rules.
8 ON THE IMPACT OF DECODING POLICY
In this section, investigate the impact that different, sampling-based decoding policies have on our analysis framework. Here, we focus on top-k sampling (Fan et al., 2018) and we further in- vestigate top-p sampling (Holtzman et al.) 2020) in Appendix E where we observed highly similar results.
Experimental setup. We use top-k sampling with & = 4. order to remain consistent with our earlier experiments, used the same 1000 warmup sequences in our evaluations. Since the move sequence is now non-deterministic, we perform three sets of evaluations with different random seeds and report the average ASR achieved by our attacks over these evaluations.

---

Published as a conference paper at ICLR 2026
Table Success rate of each attack strategy over the NT and NT+JP models trained on the Lichess-
16M dataset, when evaluated with out-of-distribution (OOD) warmup prefixes of varying lengths. The increases in ASR compared to evaluations with in-distribution prefixes (as seen in Table I are in brackets.
20 OOD Moves
40 OOD Moves
60 OOD Moves
NT
NT+JP
NT
NT+JP
NT
NT+JP
RM
0.244 (+0.14)
0.221 (+0.15)
0.360 (+0.25)
0.272 (+0.20)
0.495 (+0.39)
0.343 (+0.27)
SMM
0.203 (-0.08)
0.145 (-0.07)
0.244 (-0.04)
0.200 (-0.01)
0.351 (+0.06)
0.217 (+0.01)
IMO
0.667 (+0.28)
0.659 (+0.31)
0.785 (+0.40)
0.735 (+0.39)
0.864 (+0.48)
0.797 (+0.45)
BSO
0.080 (+0.02)
0.097 (+0.05)
0.148 (+0.09)
0.124 (+0.08)
0.245 (+0.19)
0.154 (+0.11)
AD
0.165 (+0.10)
0.183 (+0.12)
0.270 (+0.20)
0.225 (+0.16)
0.374 (+0.31)
0.294 (+0.23)
Table 5: Success rate of each attack strategy over all models with the top-k decoding strategy (I: 4). Results are averaged over three separate evaluations over the same set of warmup sequences. Bold and italic represent the highest and lowest success rates for a model, respectively.
Random-500k
Random-21 2M
Random- 10M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.963
0.990
0.969
0.993
0.886
0.971
0.902
0.976
0.703
0.903
0.750
0.908
SMM
0.937
0.994
0.937
0.997
0.745
0.973
0.819
0.984
0.315
0.913
0.386
0.926
IMO
0.998
1.000
0.999
0.999
0.995
0.997
0.998
0.999
0,958
0.974
0,959
0.959
BSO
0,961
0.979
0.960
0,978
0.895
0.954
0,903
0.968
0,706
0.885
0,750
0.884
AD
0.982
0.991
0.985
0.993
0.969
0.975
0.977
0.979
0.944
0.938
0.955
0.919
Millionbase-S0e
Stockfish-8M
Lichess
16M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.954
0,998
0.951
0.996
0.326
0.926
0.365
0,930
0.246
0.912
0.189
0.922
SMM
0.952
0.999
0.968
1.000
0.263
0.943
0.281
0.949
0.623
0.910
0.607
0.922
IMO
0.998
1,000
0.996
1.000
0.787
0.981
0.815
0.985
0.654
0.979
0.616
0.977
BSO
0.938
0.993
0.927
0.990
0.366
0.909
0.394
0.913
0.240
0.871
0.194
0.902
AD
0.943
0.996
0.936
0.998
0.322
0.935
0.344
0.943
0.192
0.932
0.156
0.933
Results. 5 shows the average ASR of attacks when our models use the top-k decoding policy. All attacks achieve a higher ASR compared to the results against the greedy decoding policy, but otherwise their relative performance is similar, showeasing that our results are robust to the decoding policy used to generate sequences. This is particularly interesting in the case of the IMO attack, which assumes a greedy decoding policy. These results imply that steering the model towards states that maximize the probability of the top-1 error will also maximize the overall probability of error, suggesting that IMO able to uncover vulnerabe state-regions, that is, gaps in the model's knowledge opposed to just one-off errors.
CONCLUSIONS AND LIMITATIONS
We proposed adversarial sequence generation to test the soundness of implicit world models. The most successful attack was IMO based on an explicit lookahead search for illegal moves. Our methodology allowed us not only to prove that none of the training setups resulted in sound models, but also to observe interesting patterns, such as the importance of using large dataset, the mislead- ing appearance of soundness in the case of high-quality, large gameplay dataset, and the positive effect of using a probability distribution objective. At the same time, we found that board state probes do not help much in any form we tried, and seem to be mostly independent of generation.
Our main limitation is that, similar to other seminal works in the field (Vafa et al.) 2024 Li et al. 2023). we rely one generative sequence model architecture due to the expensive training and evaluation. Although this study provides compelling arguments behind our proposed methodology, the effect of different architectures would certainly be interesting to analyse in the future.
10

---

Published as a conference paper at ICLR 2026
10 ACKNOWLEDGEEMENTS
This work was supported by the University Research Grant Program of the Ministry for Culture and Innovation from the source of the National Research, Development and Innovation Fund, and by project Ministry of Culture and Innovation Hungary from the National Research, Development and novation Fund, financed under the 1224-12.3-HU-RRIZON funding scheme.
implemented with the support provided by the
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellic Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? A case study in color. In Arianma Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference on Computational Natural Language Learning, CoNLL 2021, Online, November 10-11, 2021, pp- 102-132. Association for Computationnl URL http::://bll.orr/1111166633 http:///11///2211m111111111 conl1-1.9
LinnapttttiionalLiigiiittttsss Linguistics, 2021. doi: 10. 1863////2221..ONLLLI
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics. 48(1):207-219, 2022. doi: 10.1162/COLII,A) _00422 URL hwtp:httss/// //ettps://di...o http://///ee.../1111111111 .otg/10.1122222
coli 00422
Yonatan Belinkov and James R. Glass. Analysis methods in neural language processing: A survey. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 3348-3354, Association for Computational Linguisties, 2019. URL https:
//aclanthology. wttp:/91gggNN1111113313
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety a review. Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856. htthhttps///peeete Survey Certification, Expert Certification.
net/ htte/forummiccc wttormmiideUUUteePPP
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 889-898, 2018
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL /tttps://genneeiiew wwwpeenwweeeeeeeeeeeeeeeeeeeeeee http://ww..etttfoomommoeoyyy/n/a
E. Mark Gold. Language identification in the limit. Inf. Control., 1065):447-47, 1967. doi: 10. (01G50019-999536679111655 91165-5]
URL httppss:/dol..eg/110.00000
(67)
David Ha and Jürgen Schmidhuher. World models. CoRR, 00b./1003.1112.. 2018. URL http:
wwwp:i..rrg/abb///ee 10122
Dean Hazinch, Zechen Zhang, and Jeffrey Chiu. Linear latent world models in simple transformers: A case study on othello-GPT. In Socially Responsible Language Modelling Research, 2023, URL
net https:://penreviiee http:/lwwwweee/ooruueeeee www.rooiiooooooooooo SmreYNKLKv
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 7733-2743.. Association for Computational Linguistics, 2019. doi: 10. 18.18863/V 1/75 URL httpt::tp//............ /tt.:/........0..33///11/1111110
11

---

Published as a conference paper at ICLR 2026
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word repre- sentations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, NAACI-HIIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4129 4138. Association for Computational I Linguistics, 2019. doi: http0////....1.//////////11
10.18653/V 1/N19-149, URL
1419
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https:
//openreview net. fert//formmicccoo wtt.://mmmddddyyyr....v..
Wenshuai Huo, Xiaocheng Feng. Yichong Huang. Chengpeng Fu, Baohang Li, Yangfan Ye, Zhirui Zhang, Dandan Tu, Duyu Tang, Yunfei Lu, Hui Wang, and Bing Qin. Enhancing non-english capabilities of english-centric large language models through deep supervision fine-tuning. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 29(23):24185-22199) Apr. 2025. doi: (0.16609aaaai.v39223.3455 axticle/view/4552
URL httts:://oss..... www.oss.aaaa...mmm wwwpaa.o.giiidex.pppppp AAAI/
jylin04, JackS, Adam Karvonen, and Can. OthelloGPT learned tics. - Sttallogg---aarreeee---oooo-o--r
bag of heuris-
https://www. http://wwwwwlsrrrgg..cogoo coop wotpstts/ggpuuEZzaaaaaaaaa
2024. [Accessed 19-09-2025).
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. CoRR, 15492, 2024. doi: 10.48550AARXV.2400..855
URL https: //
doi. org. 10.
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictio- nary learning for language model interpretability with board game models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 83091-831 Curran Associates, Inc., neuris cc/ wwpppprpppppppppcpppppppeeee...e laceer___lless__ www.s//appeee/2/222// pdf
2024. URL
fi
Jon M. Kleinberg and Sendhil Mullainathan. Language generation in the limit. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December URL httpt:::pppppprs..ot . nips cc wttppssscccppppppee feneer_fflesse fiaaperfille//pppe///000 -ossttttte..ttttttttrtaa-t-a---e weweaactttoooooeeneeeecec htm
10 15, 2024, 2024.
hash/
Benjamin Laufer and Jon Kleinberg. Measuring rule-following in language models. In JCML
Workshop on Assessing World Models, 2025. URL https:://penrett net. nwt. www.:/oreettfffmmmeee forum?
id=JjhpELEtCG
Belinda Z. Li, Maxwell I. Nye, and Jacob Andreas. Implicit representations of meaning in neu- ral language models. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pp. 1813-1827. Association for Computational Linguisties, 2021, doi:
/AW.:///22221ALLLLOOOO 7/doi. orw.l.rgg110 100 18633vvll// /2021212222 acl 1acclllonge long. 143
URL https:
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wat- tenberg. Emergent world representationss: Exploring sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023. URL
https:// /ttp:///pperviieeee net/
TcTvT
Renjie Li, Xinyi Wang, Guan Huang, Wenli Yang, Kaining Zhang, Xiaotong Gu, Son N. Tran, Saurabh Garg, Jane E. Alty, and Quan Bai. A comprehensive review deep supervision: Theo- ries and upplications. CoRR, abwp/2277.02366 2022a. doi: 10.48550/ARXIV2207.00m 02376.. URL https://dds http:s:://oo..r/11...oo
2207. 02376
12

---

Published as a conference paper at ICLR 2026
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624)11092-19999 2022b.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam ZaraadZarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 37966637:1113- 2023, doi: 10.1 126/ science.ade2574. URL httpt:tt///// fictps://vee http:///wwwwwsseneeeerrrroee wwtp:///gg//oooeeeeeoooo http///i//bbb.../// 1 126/ sci /0t/////cinnci ence. ade2574
Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representa- tions, 2023. URL URL
net wwwpeeeoroiiddeeeeeeeeeee (ttpforrruuniiieee//yyyyyaaaeeee
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OvenReview.net, 2019. URUhtttss://ppeeeviiee wwt.:/ett/otteoee - Bkg6RicqY7
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world mod- els of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Nerworks for NLP. BlackboxNLP@ EMNLP 2023, Singapore, December 7, 2023, pp. 16-30. Association for Computational Linguistics. 2023. doi: 10. 20.18853//1/22023.MP HTT.://002...AAAKXXXXXXXP.. 2023.bl www.aaabbbxxoppppccccc
URL https::/doi. ottp://///oo..o//1..o. 18633/1.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(1 1):9668997777 2023.
Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algo- rithms: Language models solve math with a bag of heuristics. In The Thirteenth International Conference on Learning Representations, 2025.
net www//forrutteeeeeeeeeeeeeeeeoeot forum?
Id=09YTt26r2F
Roma Patel and Ellie Pavlick. Mapping language models to conceptual spaces. In Interna- tional Conference on Learning Representations 2022. URI. htt.:htpsss//ppeeeviiieeee net/
forum? drgum? sxHK.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners. OpenAl, 2019. URL https:
7/con. wwt.cc////btbeee http///btttrrraaguugggeeeeee modelso setoodessllanuueeen Islleagageeeemmmeeee _ www.miiitttsteeeerrrrsseeee
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Thirty-seventhh Conference on Newral Information Processing Systems,
2023. htl/tt/ppe//peeenneeeeeeeeeeeeee
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10. /001////EELUC...23.22776666 2023. 127063
| phtttps://do/ http//00...///..../////// 1016//100001111 neucom.
Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evaluating rule following of large language models. CoRR, abs/2407 abs/2407.08440, 2024. doi: 10.45550/ARXX1.2200..882 https://do. org/l0. 48550/ atxiv.2407.... 08440
URL
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as a testbed for language model state tracking. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual AAAI Press, 2022. doi: 10. AAALL
Event. February 22- March 1. 2022, pp. 1 11385- 11393,
V36110.21390. URL WwRL.ttpss:0.. / .rg 1609/a888 weeaaa.vvv6.. v3611
13

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. doi: 10.4555/ARXIV.23222.333 https://dot http://ddo..rgg////// 48550/arxiv 2302 13971
URL.
Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, and Sendhil Mullainathan. Evalu- ating the world model implicit in generative model. In The Thirty-eighth Anmull Conference on Neural Information Processing Systems, 2024. URL [tt.htttps://openeee/e nttp://.eeeooooo forum? id=aVK4JFpegy
Thomas Wolf, Lysandre Debut. Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface'
transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. URL
Christopher Wolfram and Aaron Schein. World models and consistent mistakes in LLMs. In ICML Workshop on Assessing World Models, 2025. URL PRLLtttpss//peeneeiiee
Forum?id=Xinn YYDvv33
Andrii Zahorodnii. Improving world models using supervision with co-evolving linear probes. In ICLR 2025 Workshop on World Models: Understanding. Modelling and Scaling, 2025. URL
net/ forum?id=xsAJaxx00
Siliang Zeng, Chenliang Li, Alfredo García, and Mingyi Hong.
When demonstrations meet genera- tive world models: A maximum likelihood framework for offline inverse reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.). Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL ott https///eccpppppeeaaaaaaapprr//p
ce 01t.
Danna Zheng, Mirella Lapata, and Jeff Z. Pan. Large language models as reliable knowledge bases? CoRR, abs/2407.13578, 2024. URL https: htt::://dcc..ooo. 485500 arXiv.
2407 13578
14

---

Published as a conference paper at ICLR 2026
Table 6: Number of tokens and moves for each dataset, along with the average game lengths (stan- dard deviation indicated in brackets).
Dataset
Number of Tokens
Number of Moves
97,974,729
48,389,735
Random-5000
Avg. per game: 195.95 (+/-72.20)
Avg. per game: 96.78 (+/-35.97)
392,296,887
193,756,892
Random-2M
Avg. per game: 196.15 (+/-72.06)
Avg. per game: 96.88 (+/-35.89)
1,962,204,706
969,142,950
Random-100 10M
Avg. per game: 196.22 (+/-71.93)
Avg. per game: 96.91 (+/-35.84)
75,964,111
37,469,929
Millionbase-500K
Avg. per game: 151.93 (+/-57.52)
Avg. per game: 74.94 (+/-28.73)
1,302,918,935
641,501,357
Stockfish-8M
Avg. per game: 163.97 (+/-66.97)
Avg. per game: 80.73 (+/-33.29)
2,3 2,311,92,,519
1, 1,138,275,036
Michess-16M
Avg. per game: 142.57 (+/-53.74)
Avg. per game: 70.19 (+/-26.75)
A DATASET DETAILS
In our evaluations, we used three randomly generated datasets and three curated datasets. All datasets contain only legal game sequences. We rounded the sizes of the Stockfish-SM and Lichess- 16M datasets (Karvonen 2024). they contain 7, 7,946,145 149, and 16,216,625 games after filtering, respectively. The number of moves and tokens in each dataset is shown in Table and the distribu- tion of game lengths is shown in Figure 3
All games in the random datasets, as well as the StockFish-8M dataset, end according to the rules (i.e., by checkmate, stalemate, draw by repetition, or draw by insufficient material). However, human games in the Millionbase-Sin resigning, both players agreeing tokenized game sequences, this phenomenon shows up as the EOF token which always used to indicate the end of the game being the end of sequence where the game is not over according to the rules.
and Lichess- 16M datasets can end prematurely (i.e., by one player
draw, in rare cases, player running out of time). In the
While 70.22% of games in the Lichess- 16M dataset, and staggering 94.37% of games in the
Millionbase-5000 dataset, end prematurely, usually immediately after a player makes a strategic blunder, found this have little effect the soundness of the implicit world models. We detail these findings in Appendix B
B MODEL TRAINING DETAILS
We used the GPT-2 implementation of the t t rs library (Wolf al. 2019). Our hyper-
parameter setting closely follows that Toshniwal et al. (2022). All our models were trained for
epochs using the AdamW optimizer (Loshchilov & Hutter 2019 with learning rate of 3 x 10
and an Le weight decay of 0.01. The learning rate is warmed up linearly over the first 10% of training, followed by linear decay. We used a batch size of 128 and accumulated gradients over 4 batches before each optimizer step. We did not use mixed-precision training. Depending on the dataset size, training a model took between 70 minutes and 37 hours on a single Nvidia H100 GPU.
For the joint probe (+JP) training objective, we experimented with various scaling factors for the loss of the board state probe in our initial exploration phase, but found no meaningful difference between
SSpecifically, version 4.55.3, compatibility with other versions guaranteed.
15

---

Published as a conference paper at ICLR 2026
Random-5000 500K
Random22
Random-LoM
1734
200K
326
403
Game Length
Game Langtt
Game Langth
Millorbaww-5000
Stockflsh-aM
Ucche-11M
2034
17504
500K
10304
12394
236
10334
2014
7504
203K
5924
2O3K
2324
25
44
60
100
120
140
23
60
66
03
120
124
1.90
20
40
40
80
1.30
120
140
Figure 3: Distribution of game lengths in our datasets.
different settings. We don't apply any scaling to any loss term and note that the joint probe's loss is typically fifth of the next token predictor's ' loss.
B.1 ILLUSTRATING THE PROBABILITY DISTRIBUTION OBJECTIVE
Figure 4 illustrates the probability distribution (PD) objective for the first three moves of a game. After a game prefix, the model is trained to learn the probability distribution of valid single-token continuations.
C PROBE TRAINING DETAILS
Our linear board state prohes are trained to predict the board state at the end of move sequence from the final-layer representation of the language model. We only train and evaluate probes on move- ending tokens, i.c., for moves comprised of two tokens (e.g. e2e4), the destination square token, and for moves comprised of three tokens (e.g. e 7788q), the promotion piece type token's 's representation is used. This is motivated by the fact that only after processing the last token of the move should the move be completed the language model's internal model. We rely a separate oracle to know which tokens move-ending, not the language model itself.
use the representation of
In formulating the targets for the board state classification problem, we use an absolute encoding just like Li al. (2023), where a piece's 's label is always the same, regardless of which player's turn is. In contrast, Karvonen (2024) and Nanda et al. (2023) use side-specific encoding, where the labels of the pieces depend on which player is move. Nanda et al. (2023) show that absolute en- coding is harder for probes to learn, but our probes achieve comparable (and in some cases superior)
accuracies to those in Karvonen (2024). as showcased in Section D
When probes are not jointly trained with the language model, we train them after the model is trained and frozen. Our training parameters are inspired by Karvonen (2024). We train our probes on 50,000 games from the model's 's own training set for 1 epoch using the Adam W optimizer with betas (0.9, 0.99), an initial learning rate 10-3 and L2 weight decay 0.01. The batch size, i.e., the number of move-ending token representations per optimization step, was 4096, and we decayed the learning rate to 10 after 1000 optimization steps.
D PERFORMANCE METRICS
Table shows the perplexities of our models, evaluated over 15,000 test games that were unseen by either the model or the probe during training. While perplexity does not measure the soundness of the implicit world model, the values show that the joint probe (+JP) objective fails to achieve meaningful (or, in some cases, any) improvement in the model's performance.
16

---

Published as a conference paper at ICLR 2026
Game Prefix:
P
W
Valid next-token probability distribution: a2: 1/10
b1: 1/10 g1: 1/10
b2: 1/10 c2: 1/10
d2: 1/10
e2: 1/10
#2: 1/10
s2: 1/10 b2: 1/10
Game Prefix: BOS e2
Valid next-token probability distribution:
1/2
A
A
A
A
d5: 1/2
Game Prefix: BOS e2e4
Valid next-token probability distribution:
b8: 1/10 g8: 1/10
a?: 1/10 b7: 1/10 o7: 1/10
47: 1/10
e7: 1/10
27: 1/10 97: 1/10 b7: 1/10
Game Prefix: BOS e2e4 d7
2
Valid next taken probability distribution:
A
A
1/2
A
A
A
d5: 1/2
Game Prefix: BOS e2e4 d7d5
Velid next-token probability distribution: b1: 1/13 d1: 1/13
e1: 1/13 f1: 1/13 g1: 1/13
a2: 1/13
b2: 1/13
c2: 1/13
d2: 1/13 #2: 1/13
1/3
b2: 1/1
e4: 1/13
Game Prefix: BOS e2e4 d7d5 #1
8
Valid next-token probablity distribution: 1/5
2
2
4
4
b5: 1/5 a6: 1/5
Figure 4: Illustration of the probability distribution (PD) objective using the first three moves of game. Boards the left highlight movable pieces after sequence of completed moves, indicating the possible move-starting squares. A uniform probability distribution assigned to the tokens corresponding to these squares. Boards on the right highlight the possible destination squares in red, once a starting square (highlighted with green) is available. A uniform probability distribution is assigned these possible move-ending squares as well.
Table 7: Model perplexities. We report the standard, losen-wise perplexity, as opposed to canonical (move-wise) perplexity reported by Toshniwal et 2022))
R500k
R2M
R10M
MB500K
SF8M
LC8M
NT 5.9478
5,7139
5.7574
3.1347
2.3756
2.1577
PD
6.5096
1893
6.2446
6.6601
5.0957
5.9161
NT+JP
6.0369
5,7428
5.7812
3.1480
2.3820
2.1581
PD+JP
6.7879
6,2740
6,3030
6.9640
5.0999
5.9257
The perplexities of models trained with the probability distribution (PD) objective naturally lower, as the model is trained not to assign a high probability to the actual next token in the sequence but to approximate the probability distribution of valid single-token continuations. As result, the model's confidence for the actual next token will be lower, which turn increases perplexity.
Table shows the ratio of legal moves played by models in 10,000 games that unseen by each model during training. While models trained on smaller datasets (Random- 500k and Millionbase-500k achieve relatively low legal move ratios between 94.65% and 96.71 %, models
17

---

Published as a conference paper at ICLR 2026
Table 8: Ratio of legal moves of our models on 10,000 test games that were unseen by the models during training.
R500k
R2M
RIOM
MB500K
SF8M
LC8M
NT 0.9634
0.9914
0.9986
0.9640
0.9977
0.9985
PD
0.9539
0.9862
0.9985
0.9671
0.9991
0.9998
NT+JP
0.9612
0.9883
0.9985
0.9638
0.9975
0.9983
PD+JP
0.9465
0.9772
0.9989
0.9597
0.9990
0.9997
trained on large datasets (Random-10M, Stockfish-8M, and Lichess-166) 16M) achieve high legal move ratios between 99.75% and 99.98%.
However, as argued by Vafa al. (2024) and demonstrated by our results, legality ratio is only surface-level metric and does not reflect on the soundness of the implicit world model.
Tables show the move-wise mean accuracies and piece accuracies of our board state probes, evaluated over 15,000 test games that were unseen by either the model or the probe during training. Piece accuracy is defined accuracy over squares that either contain pieces (i.e., they are not empty) or are predicted by the probe to contain picces.
While most probes achieve remarkably high accuracies (on par with, even higher than, the probing accuracy reported in Karvonen (2024), ) it must be noted that probes, especially those that were not jointly trained with the model, are biased towards empty squares. As shown in Figure towards the later parts of the game, probes get progressively worse at predicting pieces on the board, but their accuracies stay high due to the large number of empty squares that the probe is able correctly guess.
To correct this bias, we experimented with weighting the loss term of the board state probe per square, based on whether it is a "piece square" (i.e., a square that either contains a piece or is predicted by the probe to contain a piece) or an empty square. Our goal was to apply an increased weight to piece squares, thereby forcing the probe to learn to track the pieces better. We applied weights between and 20 to piece squares preliminary experiments, the results of which showed minor improvements in piece accuracy at a minor cost of overall accuracy, but these probes showed no difference compared to the standard probes when used as the basis of the BSO adversary in our framework.
While we believe this bias towards empty squares represents fundamental issue, its relevance to our findings is minimal, especially in light of the aforementioned weighting experiments. We leave it up to future work to create linear probe training methods that properly address this challenge.
Table 9: Move-wise average accuracies of our board state probes.
R500k
R2M
R1OM
MB500K
SF8M
LC8M
NT
0.8416
0.9178
0.9554
0.9014
0.9640
0.9698
PD
0.8237
0.8754
0.9410
0.8849
0.9584
0.9732
NT+JP
0.9831
0.9996
1.0000
0.9879
0.9999
1.0000
PD+JP
0.9786
0.9982
1.0000
0.9851
1.0000
1.0000
E FURTHER EXPERIMENTAL RESULTS
In this section, we present further experimental results supporting our claims.
18

---

Published as a conference paper at ICLR 2026
Table 10: Move-wise average piece accuracies of our board state probes.
R500k
R2M
R1OM
MB500K
SF8M
LC8M
NT
0.6005
0.7856
0.8812
0.7473
0.8858
0,9182
PD
0.5634
0.6840
0.8445
0.7041
0.8671
0.9264
NT+JP
0.9546
0.9989
1.0000
0.9675
0.9998
0.9999
PD+JP
0.9427
0.9951
1.0000
0.9600
1.0000
1.0000
NT
PD
NT+JP
PD+JP
Random- Sootom-D0K 500K
1.00
1.0
1.0
1.0
Accuracy
0.8
0.6
0.0
150
150
150
150
Random-2M 2M
Accuracy
1.00
1.00
0.8
0.75
0.95
0.50
0.90
0.25
50
100
150
50
100
130
50
100
150
50
100
150
1.0
1.0
1.000
1.000
Accurayy ex
Random-
0.8
0.6
0.4
0.985
0.990
50
50
50
Accuracy
0.6
100
150
100
150
100
150
160
150
Stockfich-BM
1.0
1.00
1.00
Accuracy
0.8
0.4
50
200
150
59
100
150
59
100
150
90
100
150
1.0
1.0
1.00
1.00
Accuragy
Lichess-166
0.8
0.6
0.99
0.99
0.8
Place Accuracy
100
100
150
100
160
150
Move Number
Move Number
Move Number
Move Number
Figure 5: Move-wise mean board state probe accuracies and piece accuracies. Error bars represent one standard deviation.
E.1 MODEL RESILIENCE TO OUR ADVERSARIES
Table 11 shows the average lengths of the sequences generated by the various adversaries playing against all models, without counting the length of the warmup sequence, regardless of whether the adversary succeeds. These results complement Table D as stronger attacks yield shorter sequences, while weaker attacks result in longer sequences. From different point of view, longer sequences for the same adversary show increase in resilience by the models.
Interestingly, models trained with the probability distribution (PD) objective are harder to attack than regular next-token (NT) models. This is especially true for weaker adversaries. where PD can achieve nearly 3x increase in sequence lengths. This supports the notion that PD a more explicit way of learning the rules, while NTT models learn inconsistent and possibly fragmented rules. On the
19

---

Published as a conference paper at ICLR 2026
Table 11: Average sequence length under adversarial conditions.
Random- -aod
Random 2M
Random 10M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
79.0
80.6
77.8
72.3
92.9
112.2
92.1
105.2
106.5
131.3
106.9
131.8
SMM
45.3
78.1
47.8
78.1
52.6
105.0
61.0
96.5
46.7
125.5
48.6
128.4
IMO
20.9
18.7
20.1
19.0
27.9
34.6
27.9
29.7
51.5
81.4
BSO
46.1
54.9
57.3
57.7
35.4
71.9
57.6
85.2
43.9
113.2
67.4
116.6
AD
73.0
70.2
72.1
62.4
84.5
93.2
83.4
86.7
96.7
124.3
91.6
129.1
Millionbase-500
Stockfish-8M
Lichess-16N)
NT
PD
NT+JP
PD+JP
NT
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
65.4
49.4
62.7
43.7
51.8
129.1
53.9
129.9
50.4
131.7
52.6
130.8
SMM
48.8
59.1
50.7
57.3
72.4
120.6
73.5
124.5
84.8
120.4
82,9
125,4
IMO
13.5
10.8
16.0
8.4
49.0
65.9
51.7
74.5
65.3
70.2
70.7
80.0
BSO
48.7
51.5
53.2
47.5
46.7
103.6
48.9
120.7
41.4
117.7
44.9
119.4
AD
61.6
40.9
62.1
35.9
46.5
118.6
52.6
125.0
45.4
123.9
47.0
128.3
Table 12: Ratio of games that end in either checkmate stalemate where the model correctly identifies the end of the game by predicting the EOS token after the final move and nowhere else.
R500k
R2M
R1OM
MB500K
SF8M
LC8M
NT
0.1865
0.3330
0.6806
0.0388
0.2510
0.3122
PD
0.1686
0.2948
0.6955
0.0000
0.2744
0.6139
NT+JP
0.1846
0.3236
0.6503
0.0397
0.2413
0.3280
PD+JP
0.1533
0.2528
0.7617
0.0000
0.2926
0.5635
other hand, the joint probe (+JP) objective has minimal impact on the models' resilience, furthering our claim that the board state probe is largely independent of the next-token predictor head.
E.2 THE IMPACT OF PREMATURELY ENDED GAMES
As mentioned in Appendix A two datasets of human games contain a high ratio of games that end prematurely. Here, we investigate if this has any effect on the models and the adversarial evaluation.
We evaluate the models' ability to correctly predict the end of the game, on 10000 games that end in checkmate and 1000 games that end in stalemate. All games were unseen by all models. We say a model is able to accurately identify the end of the game if, when processing the entire sequence, it predicts the EOS token after the final move, and nowhere before.
Table 12 shows the accuracies of all our models in predicting the end of the game. It is clear that the nature the dataset (random or curated) has more impact on the models' ability identify the end of the game than the ratio of prematurely ended games. Models trained on the Stockfish-8M dataset, a dataset without prematurely ended games, still perform poorly, while models trained on the largest random dataset (which is only slightly larger than Stockfish-8M are significantly better at predicting the end of the game.
However, it is still possible that the mistake the adversaries force the models make is incorrectly predicting the end of the game. One could assume that, for models whose training data has a very high ratio of prematurely ended games, this type of error would dominate the adversarial evaluation. While this would not mean the implicit world models are sound, a phenomenon like this would still cast shade results by suggesting that simply identified overfitting in models.
20

---

Published as a conference paper at ICLR 2026
The success rate of our attacks against our models broken down into its two possible sources of success: forcing the model to predict an illegal move (top half), and forcing the model to incorrectly predict the end of the game (bottom half).
Table 13:
Attack Success Rate due to Illegal Move
Random-50kl
Random-2h 2M
Random- 10M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.609
0.762
0.612
0.831
0.298
0.346
0.311
0,511
0.093
0.062
0.106
0.047
SMM
0.272
0.769
0.349
0.731
0.188
0.483
0.242
0.660
0.060
0. 0.166
0.054
0.107
IMO
0.842
0.855
0.851
0.868
0.695
0.821
0.752
0.878
0.445
0.507
0.452
0.361
BSO
0.700
0.763
0.660
0.791
0.385
0.498
0.502
0.655
0.134
0.148
0.145
0.119
AD
0.803
0.845
0.813
0.891
0.611
0.604
0.612
0.726
0.256
0.184
0.157
0.134
Millionbase-500
Stockfish-SM
Lichess- 16M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.724
0.989
0.766
0.993
0.121
0.178
0.147
0.197
0.049
0.151
0.029
0.121
SMM
0.484
0.791
0.478
0.843
0.045
0.382
0.054
0.407
0.030
0.341
0.030
0.247
IMO
0.995
0.998
0.990
1.000
0.552
0.701
0.561
0.610
0.173
0.724
0.155
0.683
BSO
0.512
0.883
0.544
0.938
0.079
0.231
0.129
0.349
0.042
0.176
0.037
0.257
AD
0.732
0.989
0.758
0.994
0.121
0.330
0.114
0.319
0.049
0.300
0.041
0.215
Attack Success Rate due to Incorrectly Predicted Game Ending
Random-5000
Random-2h 2M
Random 10M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.251
0.092
0.237
0.096
0.356
0.180
0.307
0.067
0.164
0,056
0.166
0.037
SMM
0.141
0.041
0.136
0.032
0.208
0.190
0.195
0.084
0.092
0.163
0.112
0.071
IMO
0.153
0.143
0.144
0.132
0.303
0.179
0.245
0.119
0.502
0.331
0.501
0.351
BSO
0.181
0,104
0.139
0.061
0.390
0.269
0.180
0.076
0.382
0.199
0.267
0.097
AD
0.094
0.090
0.065
0.089
0.120
0.189
0.088
0,106
0.037
0.132
0.040
0.063
Millionbase-500k
Stockfish-8M
Lichess-16M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.035
0.000
0.013
0.000
0.011
0.023
0.009
0.047
0.024
0.016
0,010
0.021
SMM
0.029
0.000
0.016
0.000
0.136
0.047
0.201
0.037
0.212
0.024
0.155
0.035
IMO
0.004
0.002
0.005
0.000
0.065
0.186
0.082
0,243
0.101
0.116
0.038
0.101
BSO
0.012
0.000
0.002
0.000
0.026
0.152
0.016
0.044
0.012
0.053
0.001
0.016
AD
0.035
0.000
0.012
0.000
0.006
0.101
0.010
0.074
0.010
0.017
0.003
0.019
Table 13 breaks down the attack success rate (ASR) achieved by every adversary against our models into two components: ASR due to forcing the model to predict an illegal move, and ASR due to forcing the model to incorrectly predict the end of the game. In almost all cases, the vast majority of successful attacks force the model to predict illegal moves, even when the models were trained on datasets that contain many prematurely ended games. Among the few exceptions, the IMO adversary against the Stockfish-PD and Random OM-NT models cannot be explained by the ratio of prematurely ended games, because there are none in these datasets (and, in the former case, PD eliminates premature game ends well). The other notable exception is the sequence model move (SMM) baseline adversary against the Lichess-NT models, which suggests a degree of overfitting to the errors present in the dataset.
While incorrectly predicting the end of the game is still a rule violation and is enough to show that the implicit world models are not sound, our findings reveal that our adversaries do not solely rely on this error type. Furthermore, even if the adversaries succeed this way, it is not a result of the models overfitting to this type of error in the dataset.
21

---

Published as a conference paper at ICLR 2026
Table 14: Attack success rates achieved by simply letting the models generate the sequence after processing the warmup prefix.
R500k
R2M
RIOM
MB500K
SF8M
LC8M
NT
0.616
0.563
0.263
0.762
0.316
0.427
PD
0.947
0.966
0.921
0.935
0.936
0.916
NT+JP
0.696
0.609
0.273
0.740
0.395
0.349
PD+JP
0.918
0.975
0.866
0.955
0.956
0.891
E.3 SEQUENCE MODELS FAIL BY THEMSELVES
further possible baseline adversary against sequence models can be implemented by letting the sequence model simply generate the sequence until it 'fails by itself". While this method does not conform to our definition of an adversary, probably the easiest way to verify the soundness of the implicit world model.
Table 14 presents the adversarial success rates achieved by this simple method. Impressively, this method achieves high ASRs against Lichess-NT models; however, it is generally among the weaker adversaries.
E.4 DOES THE BSO ADVERSARY SUCCEED?
The goal of the board state oracle (BSO) adversary to cause the sequence model's associated board state probe to have as many errors possible when predicting the board state. One could assume that the reason behind the weakness of the BSO adversary is that fails to cause the probe to have significant errors.
Figure 6 shows the move-wise mean accuracies and piece accuracies under non-adversarial condi- tions (evaluated on unseen test games), well when the BSO adversary is used generate moves for white. The BSO adversary is able to guide the game towards regions where the probe's accuracy significantly higher than its
non-adversarial test games.
Despite its success inducing errors in the probed board state, BSO still fails to be an effective adversary against the rule-following capabilities of sequence models. This further shows the limited causal connection between the generative model's function and the board state probe's out- put.
E.5 AGREEMENT BETWEEN MODELS AND PROBES
Here, we delve into the agreement between the ground truth board state, the output of the board state probe, and the implicit board state representation of the sequence models. For move sequence sequence model AM Ws((s)
V+, let define the implicit world state representation {a E Σ (als) 2 <), i.e. the set of actions with at least conditional probability. Given world state probe B, let us denote the set of legal actions in B (MM, s) (i.e., the world state predicted by the probe) as (x) C Σ. As introduced in the main text, Wss) C Σ represents the set of legal actions the true world model after the action sequence 8.
Let us use the intersection over union (IoU) metric to quantify the agreement between the true world model, the world state probe, and the implicit world state. Formally,
(s)) n (s) (W(s) (s)
loU
(6)
denotes the agreement between the true world state and the implicit world state of M,
IoUw.e(s) = ((s) n (s(s)
(7)
(W(s) (+()))
22

---

Published as a conference paper at ICLR 2026
NT
PD
NT+JPP
PD+|P
Randoi--500 -00m
1.0
1.0
1.0
Accurasy
0.8
0.8
0.5
0.6
0.6
0.0
0.4
150
100
150
100
150
Random--2 2M
Ac Accuracy
1.0
0.75
0.9
0.9
0.5
0.50
0.25
0.7
50
150
150
50
109
150
50
100
1.0
50
100
150
1.00
1.00
2.00
1.00
Accuracy
Random-
0.75
0.75
0.95
Accuracy
Rece Accurarry
0.50
0.90
BSD Accuracy
BSD Piece Accuraty
0.25
0.85
50
100
50
100
100
1.50
100
150
1.0
1.0
1.0
1.00
MB-500K
Accuracy
0.75
0.5
0.5
0.6
0.0
0.0
0.4
0.25
150
150
100
150
100
150
Stockrich BM
1.00
1.0
1.0
Accuraty
0.75
0.75
0.9
0.9
0.50
0.50
0.25
0.8
0.8
100
150
100
1.9
100
150
Lichocs-15M
Accuracy
1.00
1.00
3.00
1.00
0.75
0.85
0.50
0.50
0.90
0.50
0.25
0.25
0.85
50
100
150
50
100
150
50
100
150
Move Number
Move Number
Move Number
Move Number
Figure 6: Move-wise mean board state probe accuracies and piece accuracies (similar to Figure 5 along with accuracies and piece accuracies when the BSO adversary generates the moves for white. Note that the BSO adversary takes effect after move 10, which is the end of the warmup sequence. Error bars indicate one standard deviation.
denotes the agreement between the true world state and the state recovered by the world state probe, and
iOUMBB(5 War n (s)
(M (B)
denotes the agreement between the implicit world state of M and the state recovered by the world state probe.
Figure 7 shows the move-wise agreements between the true world model, the board state probes, and the models' implicit world state, evaluated over 15,000 test games that were unseen by either the models or the probes during training. Inspired by Vafa et al. (2024), we used e = 0.01.
Our findings show stark differences between dataset types and training objectives as well. It is clear that models trained on random datasets agree more with the true world state than models trained on curated datasets, as also shown in Li et al. (2023) and Vafa et al. 2024). However, the probability distribution (PD) objective mitigates the probable fragmentation of the NT models throughout all phases of the game, again showing that it is more effective tool for learning the rules.
More strikingly, there is always a significant difference between the IoUw.M and IoUw.B. indicat- ing that there is a significant disagreement between the models' next-token predictor heads, and the board states extracted by the probes. This phenomenon is most striking when the next-token predic- tion and joint probe objectives combined (NT+JP), where the probes always agree with the true
23

---

Published as a conference paper at ICLR 2026
NT
PO
NT+JP
PD+jP
Randem- 500K
1.00
1.00
1.00
0.75
0.75
0.75
18
0.50
0.50
0.25
0.25
100
150
100
150
100
150
100
150
1.0
1.0
1.00
1.00
Random-
0.75
(0)
0.5
0.50
0.50
0.0
0.25
0.25
0.0
50
1.90
150
90
1.00
150
50
100
50
100
150
Random 10M
1.0
1.0
1.0
1.0
0.5
0.5
100
100
150
100
150
0.0
100
150
0.75
0.5
0.5
0.5
0.0
0.25
0.0
0.0
50
150
150
50
100
150
50
100
150
59
1000
150
Stookt -8M
1.0
1.0
1.0
1.0
100
0.5
0.5
0.5
50
100
150
150
100
150
150
1.00
0.75
Lichese-
101
0.3
0.5
0.0
0.25
180l
60
100
150
109
150
56
300
150
g
309
150
Move
Number
Figure 7: Move-wise mean IoUs between board state probes, model outputs, and the true world state. Error bars represent one standard deviation.
world model, but the agreement between the models and probes, as well as the models and the true world model, significantly lower.
These results cast further doubt over the causality of probes, as well as the generally accepted prob- ing paradigm, where the probes are trained extract the ground truth. We believe would more beneficial to create probes that directly represent the "knowledge' of the sequence models, but leave this up to future work.
E.6 THE COMPUTATIONAL COST OF OUR ADVERSARIES
Table [3] shows the computational costs of our adversaries against every model. We report these costs in seconds per sequence when using single H100 GPU, averaged over 1000 sequences used in our evaluation against the top-k sampling strategy. Note that longer sequences yield longer evaluation times.
15
The cheapest adversary RMM, as it does not require model inference. The computational costs of SMM and AD are similar as they both require one model inference at each attack step. Interestingly, BSO computationally inefficient due to the rather costly evaluation of the board state probe, but we admit that our implementation has room for optimization. On the other hand, IMO uses optimized implementation that predicts the probabilities of single-move continuations using an internal batch size of 128. As expected, IMO is the slowest attack, showing 10-20x increase in computational cost compared to single-inferent
attacks like SMM and AD, which is in line with the cost of standard adversarial attacks in other domains.
24

---

Published as a conference paper at ICLR 2026
Table 15:
Computational cost of each of our attacks against all models in seconds per sequence,
averaged over 1000 sequences.
Random-500k,
Random-2M
Random- 10M
NT
PD
NT+JP
PD+JP
NT
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.359
0.457
0.327
0.326
0.692
0.700
0.630
0,504
0.913
0,996
0.869
0.984
SMM
0.519
0.716
0.438
0.557
0.846
0.984
0.722
0.763
0.854
1.445
0.878
1,516
IMO
7.949
6.775
6.880
6.733
14.031
12.384
12.099
9.614
28.727
32.496
25.371
34.828
BSO
10.916
8.959
5.265
4.179
9.296
8.147
7.681
7.137
15.945
38.180
12.986
13.902
AD
0.655
0.590
0.604
0,514
1.281
0.866
1.097
0.777
1.513
1.650
1.586
1.640
Millionbase-500k
Stockfish-8M
Lichess-166
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.314
0.238
0.402
0.205
0.469
1.086
0.507
0.932
0.365
1.035
0.372
0.978
SMM
0.493
0.427
0.582
0.388
0.482
1.533
0.443
1.426
0.798
1.439
0.788
1.425
IMO
6,258
4.824
7.926
3,984
18.339
31,363
17.668
34.055
13.037
33.389
13.391
34.174
BSO
11.228
11.200
5.007
4.471
7,800
17.158
6.549
17.024
4,512
23.387
2.976
18.685
AD
0.567
0.341
0.673
0.287
0.796
1.524
0.766
1.535
0.609
1.644
0.579
1.549
Table 16: Success rate each attack strategy over all models with the top-p decoding strategy (p = 0.9). Results are averaged over three separate evaluations over the same set of warmup sequences. Bold and italic represent the highest and lowest success rates for a model, respectively.
Random-5000
Random-2h
Random 10M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.983
0.993
0.990
0.996
0,936
0.969
0.955
0.980
0.804
0.911
0.833
0.908
SMM
0.95/
0.997
0.966
0.995
0.805
0.968
0.861
0.984
0.408
0.921
0.451
0.925
IMO
1.000
1,000
1.000
1.000
0.998
0.998
0.998
1.000
0.978
0.965
0.975
0.962
BSO
0.974
0.982
0.975
0.980
0.928
0.964
0.933
0.964
0.784
0.878
0.816
0.873
AD
0.990
0.995
0.993
0.993
0.986
0.975
0.987
0.979
0.973
0.933
0.978
0.924
Millionbase-SM0
Stockfish-SM
Lichess- 16M
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.972
0.998
0.967
0.998
0.383
0.931
0.379
0.918
0.254
0.917
0.183
0.913
SMM
0.967
0.999
0.970
0.999
0.226
0.946
0.248
0.943
0.507
0.933
0.459
0.915
IMO
0.999
1.000
0.998
1.000
0.832
0.981
0.836
0.980
0.667
0.974
0.649
0.976
BSO
0.953
0.992
0.944
0.990
0.404
0.909
0.441
0.919
0.226
0.873
0.178
0.893
AD
0.963
0,997
0,951
0.997
0.360
0.939
0.366
0,938
0.194
0.926
0,150
0.931
E.7
RESULTS AGAINST TOP-p SAMPLING
Table 16 shows the ASR of our attacks against models with the top-p sampling policy (p 0.9).
These results echo our findings with the top-k sampling policy in Section g The success rates of each attack is higher than the ASR against the greedy decoding policy, giving further evidence to the generalizability of our method.
E.8 TOWARDS ADAPTIVE ADVERSARIES
In this section we present a modification of the Illegal Move Oracle (IMO) adversary that can be adaptive variant of the IMO variant we used in the main text. As opposed to selecting the move that maximizes the conditional probability of an illegal continuation, this variant aims to find the move that maximizes the sum of the conditional probabilities of all illegal continuations.
In practice, our implementation only analyzes single-move continuations that are reachable by top-k sampling. When set l to be the size of the vocabulary, the attack is equivalent to the original idea
25

---

Published as a conference paper at ICLR 2026
Table 17: ASR of out adaptive IMO attacks against our models with the greedy decoding strategy (left), and the top-k sampling decoding strategy (right).
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
R-500K
1,000
1.000
0.998
1.000
R-500K
1.000
0.999
1.000
1.000
R-2M
0.999
0.999
0.995
0.998
R-2M
0.998
0.995
0.997
0.999
R-10M
0.994
0.967
0.995
0.911
R-10M
0.979
0.904
0.979
0.899
MB-500K
1,000
1.000
0.998
1.000
MB-500K
0.999
1.000
0.994
1.000
SF-8M
0.644
0.985
0.661
0.990
SF-8M
0.670
0.941
0.680
0.953
LC-16M
0.481
0.951
0.417
0.947
LC-16M
0.561
0.923
0.458
0.903
above. As result of this modification, this can be seen an adaptive attack against top-k sampling, although sampling is done on the token level, and the attack analyzes moves (that are made of 2 or 3 tokens).
Table 17 shows the results of this attack against our models with both the greedy decoding strategy and the top-k sampling strategy. Surprisingly, this attack achieved marginally higher ASR against models with greedy decoding (compared to that of of the original IMO in Table [ and somewhat lower ASR against models with top-k decoding (compared the success rates in Table (5). This surprising finding hints at a disconnect between the token-level decoding strategies of the models and the move-level analysis of the attacks.
F BREAKING DOWN How OUR MODELS BREAK DOWN
Here, investigate the types of errors our models made result of our adversarial evaluation. We first provide a taxonomy of possible errors, analyze their frequencies, and provide further fine- grained insights into some of the more complex errors.
F.1 A TAXONOMY OF ERRORS
Let us start by introducing seven еттоr categories:
Nonexistent Piece: The model tries to move piece that does not exist. other words, the starting square predicted by the model is empty.
(2) Opponent's Piece: The model tries to move a piece that belongs to its opponent. other words, the starting square predicted by the model contains the opponent's piece.
(3) Immovable Piece: The model tries move piece that cannot be moved for some reason,
e.g., it is blocked, or the model has to block a check and the selected piece unable to do so, etc.
(4) Invalid Direction:
The model picks movable piece, but moves it in an invalid direction, e.g., moving rook diagonally bishop horizontally.
(5) Erroneous Move: The error made by the model cannot be categorized into the previous categories, e.g. jumping over pieces, capturing the opponent's 's king, invalid castling, incor- promotion, moving the king next to the opponent's king, etc.
(6) Structural Error: The move predicted by the model is not in the UCI notation, e.g., the model predicts e8q as move.
The model incorrectly predicts the end of the game. This error type was analyzed in Appendix E.2
Incorrect End Prediction:
Note that our taxonomy by no means a complete breakdown all possible error types in chess, but it serves sensible grouping of the possible failure modes. In addition, not all failure modes can be attributed to an atomic deficiency the model. Only error types (1) and (2) can be clearly attributed to the model having incorrect understanding of the board state, but error types (3), (4), (5), and (7) can all arise from an incorrect board state representation, a lack of understanding the rules, even an incorrect representation of the game history well.
26

---

Published as a conference paper at ICLR 2026
NT
FD
KTTpP
P0
7D
WT+ep
CIRRONI PO
Sandee-2M
Frequencies of different eπror types made by our models against all adversaries, with models trained on random datasets being shown on the left, and models trained curated datasets on the right.
Figure 8:
Results. Figure 8 shows the frequencies of different eπror types. Note that our attack does not differentiate between error types; an error type being prevalent in our evaluation does mean the model is guaranteed to not make that error, only that other errors are easier to cause.
For all models, Immovable piece (type 3) and erroneous move (type 5) errors are always among the most prevalent. Incorrect end prediction (type 7) is more common for models trained on random datasets, as also demonstrated in Appendix E.2
The difference between the NT and PD objectives is relatively small when random datasets are used training, but remarkable when curated datasets are used instead. The PD objective leads more uniform error distribution which, when combined with our earlier analysis on model resilience, suggests that PD models fundamentally break down towards the end the game.
When it comes to attacks, the four weaker attacks (RM, SMM, BSO, and AD) almost always yield similar error distributions. The only exception is SMM against models trained on curated datasets with the NT objective, where it achieves high ASR by causing the model incorrectly predict the end the game. However, IMO is clearly different, as achieves errors related to rule knowledge (types 3, 4, and 5) more frequently.
27

---

Published as a conference paper at ICLR 2026
NT
FD
KTap
7B
WT+ep
70+P
TD
TD
11
Figure 9: Frequencies illegally moved pieces grouped by piece type for complex, rule-based errors. Results of models trained on random datasets are the left, and that of models trained on are on the right.
F.2 THE IMPACT OF PIECE TYPES IN COMPLEX ERRORS
We further analyze the impact of piece types in complex errors, namely immovable piece (type 3), invalid direction (type 4), and erroneous move (type 5) from our previous taxonomy. Here, we investigate which pieces the model tries move, but moves illegally.
Figure 2 shows the results for every model and attack. The trends are largely similar to our earlier analysis on general error types. Models trained on random datasets, as well those trained with the PD objective overwhelmingly struggle with king moves, while models trained on large curated datasets with the NT objective predominantly struggle with pawn moves.
G RESULTS ON LLAMA MODELS
We trained LLaMA models Touvron et al. 2023 with the settings described Section 4 resulting in further 24 models. We used the same architecture size with the GPT-2 architecture, as described in Section 4.3 We then evaluated them using our adversarial framework, adhering to the settings described in Section
28

---

Published as a conference paper at ICLR 2026
Table 18: Success rate each attack strategy against LLaMA models. Bold and italic represent the highest and lowest success rates for model, respectively.
Random-500k
Random-2M
Random- 10M
NT
PD
NT+JP
PD+JP
NT
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.944
0.987
0.961
0.990
0.899
0.935
0.885
0,910
0.703
0.839
0.680
0.811
SMM
0.777
0.892
0.792
0.950
0.702
0.940
0.645
0.930
0.285
0.863
0.286
0.877
IMO
0.999
1,000
0.999
1.000
0.999
1.000
0.999
1.000
0.979
1.000
0.980
0.990
BSO
0.885
0.953
0.896
0.951
0.826
0.875
0.814
0.873
0.525
0.699
0.581
0.772
0.973
0.986
0.968
0.990
0.930
0.960
0.889
0,958
0.726
0.901
0.672
0.868
Millionbase-500k
Stockfish-8M
Lichess-160
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
NT
PD
NT+JP
PD+JP
RM
0.853
0.994
0.849
0.993
0.303
0.917
0.267
0.84/
0.229
0.888
0.230
0.848
SMM
0.767
0.867
0.745
0.915
0.178
0.928
0.196
0.895
0.280
0.883
0.331
0.859
IMO
0.998
1.000
0.999
1.000
0.838
1.000
0.780
0.994
0.659
0.996
0.646
0.995
BSO
0.754
0.921
0.756
0.959
0.264
0.904
0.273
0,861
0.096
0.766
0,148
0,806
AD
0.846
0.999
0.843
0.992
0.298
0.925
0.232
0.892
0.194
0.863
0.179
0.852
Table 18 shows the success rates of each attack against all 24 LLaMA models, and Figure shows the dynamics of each attack. Notably, all our findings hold true for the LLaMA architecture as well, showcasing that the errors we found with our methodology are not architecture-specitt
Notably, LLaMA models are even less sound than GPT-2 models, with most errors occuring before the 150-move mark. However, shown in Figure 1O these models also exhibit a substantial bias to- wards the 150-move sequence length, showcasing that the models pick up irrelevant patterns when comes to rule learning. Interestingly, LLaMA models can predict legal moves beyond the 150-move mark, which is most notable with models that were trained with the probability distribution (PD) objective, further showcasing that PD facilitates rule learning better than the next-token prediction (NT) objective. We suspect this capability is result the LLaMA architecture replacing absolute positional embedding with rotary positional embeddings (Su et al. 2024).
29

---

Published as a conference paper at ICLR 2026
NT+JP
PD+|P
Random-
0.5
0.5
0.5
90
80
30
120
150
180
30
60
90
170
150
30
69
90
150
30
67
90
120
159
1.0
10
1.0
Random-21
0.0
120
150
160
0.0
129
210
0.0
120
110
0.0
150
210
Random- 10M
0.5
0.5
0.5
8
100
120
340
39
69
99
920,00
21000
2000
/10
30
90
170
350
389
y10
240
99
520
880
200
1.0
1.0
1.0
1.0
ASR
69
90
0.0
80
90
80
Stockfich-aW
RR.
191
0.5
120
550
180
210 10,
120
150
180
120
150
160
210
Lichest 16M
1.0
IMO
RM
10
1.0
1.0
AD
30
60
80
500
y50
180
30 60 30009,00
30
50
68
80
300
140
30 60 90 5,,,00 300.0..00 300010 250,00 ,00
Move Number
Move Number
Move Number
Move Number
Figure 10: Attack dynamics demonstrated by the move-wise attack success rate (ASR) for each dataset (row) and model (column) using the LLaMA architecture. On each plot, the X-axis shows the move number, and the Y-axis shows the ASR attained by the attacks, Stronger attacks increase ASR more quickly
30


Reviews:
[{"review_id": "BX8CKe7Pzr", "reviewer": "Reviewer_c2oN", "summary": "The paper investigates whether sequence models trained on chess possess a sound implicit world model by searching for legal prefixes that cause the model to predict an illegal move.\nSeveral adversaries are introduced: Random Move (RM), Sequential Mimic (SMM), Approximate Decoder (AD), Board State Oracle (BSO), and Illegal Move Oracle (IMO).\nAcross 24 GPT-2-style models trained with varied datasets and objectives, all models fail the soundness test, although larger datasets and probability-distribution training improve robustness.", "questions": "1. How competent are the models at playing chess overall? For example, have you measured legality rates or gameplay strength relative to an engine or other chess language models?\n2. Do you terminate evaluation at the first illegal move or continue generation afterwards?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an interesting and clearly defined problem, formulating world-model verification as falsification through adversarial legal sequences.\n2. The chosen adversaries (RM, SMM, AD, BSO, IMO) form a meaningful distribution of strength and show how more powerful attacks reveal hidden inconsistencies in the model’s behaviour.\n3. The paper is generally clearly written and has a fairly detailed analysis of performance patterns.", "weaknesses": "1. It is difficult to interpret the results without understanding how competent these GPT-2-based models are as chess players. A comparison to specialised chess language models or an Elo-style strength estimate would help determine whether the observed unsoundness is surprising or simply reflects limited gameplay skill.\n2. The strong dependence on sequence length suggests that the models struggle with long-range reasoning. This is an interesting finding, but it mostly reflects the limitations of the training setup rather than providing deep insight into world-model soundness.\n3. Games longer than 150 moves are removed rather than truncated. Truncation could have preserved more board diversity while controlling length. A short justification for this choice would improve clarity.\n4. The Illegal Move Oracle (IMO) is predictably the strongest adversary because it has access to a legality oracle. The statement in line 342 that this “showcases the need for strong adversaries to reliably verify generative models” feels too general. In more complex domains where such oracles cannot be built, this approach may not scale.\n5. In figure 1, not all plots are complete. \n6. Line 139 states that only some algorithms use a board-state decoder, whereas line 304 says every model has a board-state probe. This inconsistency is minor but worth clarifying so readers understand that the probe is used for post-hoc analysis rather than during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rr7EO2RSiU", "reviewer": "Reviewer_oiiD", "summary": "This paper belongs to a line of work evaluating the fidelity of a language model in uncovering the hard rules of its underlying environment. The specific strategy in this paper is to adversarially design sequences where the model will incorrectly predict the next token (for instance, the max probability next token is not a valid move in the underlying game). This differentiates itself from other methods which directly measure next-token prediction, are based on linear probes or other white-box methods, or recent work which measures similarity between the model and the DFA-structure of the underlying world model.\n\nThis paper focuses on the game of chess, where the adversary is literally playing with the model and attempting to force the model to make an illegal move.\nThe attacker is greedy and chooses its acception at each timestep according to one of the following rules:\n - IMO: Maximize the conditional probability of the model on an incorrect token.\n - BSO: Maximize the error the board state as given by a linear probe.\n - AD: Choose the token with minimum probability under the model.\n - RM: Random token.\n - SMM: Choose the token with maximum probability under the model. This is standard argmax decoding.\n\n Three training strategies are employed:\n  - NT: next-token prediction, self-supervised over the given dataset\n  - PD: also self-supervised, but the target distribution is uniform over all valid moves\n  - JP: the model is jointly trained to produce a good board state via a linear probe\n\nAcross datasets, training strategies, and attack strategies, the attack success rate can vary significantly. For instance, on a dataset of 500k random moves, RM, IMO, and AD all achieve >90% attack success across all training strategies. On the other hand, on a dataset of 10M real games, the best attack (IMO) only achieves <40% success. Regardless, the attacks, especially IMO, are commonly capable of finding invalid sequences. Interestingly, next-token prediction (NT) is often the best strategy, with PD and JP not adding much benefit or making attacks easier.\n\nThe IMO attack is the most robust: all other attacks reduce to small success rates in some settings. Both in terms of attack and training procedure, introducing linear probes does not lead to better performance.\n\nInterestingly, the attacks performed worse on curated datasets than purely random datasets. Prior works showed random datasets have better world model recovery according to their metrics. The authors demonstrate that injecting an out-of-distribution prefix before running the attack significantly increases the attack success rate, demonstrating a limitation of the proposed attacks.", "questions": "- What is the success rate averaged over? As both the model decoding and the adversary are deterministic, how is a percentage success rate achieved?\n\n - Why isn't IMO defined as the sum over all illegal tokens of their conditional probabilities? Perhaps there is a board state where a single illegal move is unlikely but in aggregate, the model places high probability on illegal moves. I would guess this is related to argmax decoding.\n\n - On random data, isn't it the case that NT and PD are essentially equivalent (at least in expectation)? This is not fully realized in the numbers, and I do not understand why.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper provides a clean, new method of evaluating the ability of the a model to recover the underlying rules of its environment. The attack-based evaluation can be performed on any black-box model and is easy to understand. It avoids having to deal with many subtleties of model inference or choosing a background distribution that appear in other works.\n- The experimental setup evaluates several natural and interesting choices of attack, training objective, and training data.\n- The paper is clearly written. The takeaways are convincing and laid out well.", "weaknesses": "- It is not clear how the success rate is calculated, see the first question.\n\n- While easy to implement and generally applicable, notions like success rate of attacks are not very interpretable: they are a loose proxy. Does a model which has attack success rate 40% rather than 50% understand the underlying world model better? It is probably not that informative at that scale as the success rate depends heavily on the specific attack that is chosen.\n\n- The setup and experiments are all ran with deterministic argmax decoding. While this is a reasonable choice for finding invalid sequences, it would be interesting to investigate more realistic settings where tokens are sampled from the model.\n\n- It would be interesting to consider attacks which are not completely greedy. For instance, they could unravel $k$ steps of the game tree for small $k$. The takeaway from Section 7 suggests that an attack like AD should succeed but perhaps it does not work well in the purely greedy setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbNNgRe5k0", "reviewer": "Reviewer_9qiV", "summary": "This papers introduces an adversarial framework for verifying the soundness of implicit world models in generative sequence models, building on the work of, for example, Li et al. (2023) and Vafa et al. (2024) and using chess as a testbed. Soundness is defined as whether a model generates only valid sequences according to the true world model's rules --- that is, given any sequence, the model's continuation (generated by greedy decoding) yields a valid sequence under the true world model. The authors key contribution is to construct adversaries that generate valid moves (following a sequence generated by the model) to force the model to predict illegal continuations. \nThe authors propose several different adversaries to evaluate the soundness of the model. The paper conducts a thorough empirical study training 24 generative sequence models --- exploring the effects of alternative dataset (random games vs. curated datasets) and different training objectives. The authors main findings are: \n(1) None of the models are sound—all can be forced to make illegal moves, though the Illegal Move Oracle (IMO) adversary is far more effective than alternatives. IMO picks the legal move that maximizes the conditional probability of an invalid continuation by the opponent.\n(2) Dataset size appears to affect soundness --- larger datasets tend to have more sound models across training objectives. \n(3) Alternative objectives appears to matter --- the authors argue that the PD objective improves soundness. \n(4) The authors argue that board state probes (i.e., the ability to predict board state using the final representation) has limited causal influence on next token predictions.", "questions": "Please see my earlier discussion of weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper has many nice strengths: \n\n1) The idea of generating adversarial continuations is clever, and a natural way to assess the implicit world model of generative sequence models. The approach provides an ``existence'' proof of unsoundness. \n2) The experimental scope of the paper is impressive --- by my read, 24 models trained across 6 datasets of varying sizes and qualities, evaluated with 5 different adversaries. The systematic experiments enable the authors to make interesting statements about how design choices in training might affect soundness (i.e., dataset size, quality and training objectives). \n3) The results make interesting progress in reconciling the results in Vafa et al. (2024) (taking a more functional approach to world model evaluation) and well-known works using probes. The results showing that board state probes have limited influence on model predictions is quite nice. \n4) The paper is very well-written and easy to follow.", "weaknesses": "I use this box to describe weaknesses and make comments that I'd like to see the authors address. \n\n1) The authors criticize Vafa et al. (2024) for using an ad hoc probability threshold to define the generated language by the model, and claiming that their focus on adversaries avoids this. But there are analogous choices that must be made here. The authors focus exclusively on greedy decoding --- why not other choices? Would they perform differently? \n\n2) The definition of world models as valid continuations is presented with little discussion. It would be useful to point out that this is more general than the DFA framework used in Vafa et al. (2024) --- for example, it applies to context-free grammars, pushdown automata. So the adversary framework is potentially more directly applicable than the Myhill-Nerode based metrics in Vafa et al. \n\n3) It was surprising to me how effective RM and SMM were at generating failures. This should merit more investigation, but the authors somewhat gloss over this. For example, SMM achieves 0.943 against Random-2M-PD, meaning the model frequently fails even when generating its own preferred continuations (with only white moves corrected). These results seem to indicate the models are fundamentally poor at chess, not just vulnerable to adversarial attacks. This undermines the interpretation of IMO's success—are we testing world model soundness or just confirming the models don't play chess well?\n\n4) I found the comparison between PD and NT to be insufficiently visible in the main results. Table 1 looks like it shows mixed results to me, and the clearer evidence is buried in the appendix. If this is main result, it would be nice to bring it into the main text. \n\n5) It would be worthwhile to discuss the computational costs of the adversaries. The IMO adversary requires enumerating all illegal continuations and evaluating their probabilities. This is potentially expensive for large action spaces. \n\n6) While the paper establishes that models are unsound, it provides little insight into how and why they fail. What types of illegal moves do models make? Could the illegal moves generated by categorized? E.g., Do models violate basic piece movement rules (e.g., moving pawns backward, bishops diagonally incorrectly) in particular board positions? A taxonomy of errors would reveal whether models learn some aspects of chess better than others. Analogously,  does IMO consistently exploit particular weaknesses (e.g., rare board configurations, endgame positions, complex tactical situations)? Does BSO create specific types of confusion? Understanding which aspects of the world model are fragile would be more informative than aggregate success rates. These sorts of results would allow the authors to connect the paper to recent work emphasizing that generative models learn heuristics/shortcuts. \n\n7) The paper uses only linear probes on the final layer with absolute encoding. Could it be that chess requires different and slightly more complex probes that Othello (in particular, chess has more piece types and special rules so its a more complex game). It would be nice to see that the authors results on probes are not sensitive to the specific linear probes considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sjjl4Gt6Hy", "reviewer": "Reviewer_ggS8", "summary": "A new method based on adversary attack is proposed as a way of verifying a generative sequential model's world understanding. Unlike previous methods, this method uses an adversary that plays only legal moves but aims to trigger an illegal next move from the model. The authors test several adversary strategies on gpt2 style chess models trained on different datasets, and find that all models can be broken, implying none are fully sound. Even larger datasets and improved training objectives only delay, not prevent, failure. After all, the result suggests that generative models like GPT still lack understanding of the underlying world rules, despite appearing so.", "questions": "See the above weaknesses.\n\nBesides, it might be interesting to explore whether causal inference/discovery ideas could be applied to this setup (although not necessarily straightforward or even relevant). For instance, one could imagine treating the model’s internal states as potential mediators between the world state and its next prediction, or performing small interventions on hidden features to test causal dependence. If such analyses are possible, they could offer a complementary way to examine whether rule-following behavior arises from genuine causal reasoning or just statistical association.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper is well written. Even as someone not fully familiar with the research topic, I could follow it quite well.\n- Their proposed verification framework, which focuses on model behaviors, complements previous approaches based on probing, which are more retrospective and analytical.\n- The method is effective while remaining easy to implement.\n- Their experiments include multiple adversary strategies and a broad range of datasets, giving stronger support to their conclusions.\n- The paper also presents insightful diagnostics (eg, probe gradient alignment, OOD warmup tests) that help explain why surface-level rule-following can be misleading.", "weaknesses": "- Experiments are based on a rather outdated model (gpt2 with 86M parameters). This leaves open the question of whether stronger world understanding might emerge as models get larger. While the authors mention that their choice was constrained by compute in the last section, I find this justification unconvincing. Given access to h100 GPUs, training newer but still relatively small models (e.g., 2B or 7B) should not pose excessive computational burdens.\n- The authors use greedy decoding without much justification. Although greedy decoding is appealing as it maximizes the likelihood of producing a legal move, real chess players (and possibly more human-like models) do not necessarily reason this way. It would be interesting to see whether different sampling strategies lead to different types or frequencies of illegal moves.\n- This is not necessarily a weakness, but more of a suggestion: an analysis of common failure modes would have been valuable. For example, does the chance of failure depend on the sequence length, the type of chess pieces involved, or the board state complexity? Identifying such patterns could deepen understanding of how and why the models fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)