Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models
Authors: Adam Karvonen
Venue: COLM (2024)
Keywords: GPT, large language model, interpretability, world model
TL;DR: We train a GPT model from scratch to play chess and  find that it learns to compute board state and estimate player Elo. We use these representations to edit the GPT's internal board state and increase or decrease its chess-playing ability.

Abstract: Language models have shown unprecedented capabilities, sparking debate over the source of their performance. Is it merely the outcome of learning syntactic patterns and surface level statistics, or do they extract semantics and a world model from the text? Prior work by Li et al. investigated this by training a GPT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model's internal representations using linear probes and contrastive activations. The model is given no a priori knowledge of the game and is solely trained on next character prediction, yet we find evidence of internal representations of board state. We validate these internal representations by using them to make interventions on the model's activations and edit its internal board state. Unlike Li et al's prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character. We derive a player skill vector and add it to the model, improving the model's win rate by up to 2.6 times.

Published as a conference paper at COLM 2024
Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models
Adam Karvonen Independent adam www.onnnegmmiil.cccmmmm com
Abstract
arXiv:2403. 20 15403.1 [cs.LG] 14Jul 2024
Language models have shown unprecedented capabilities, sparking debate over the source of their performance. Is it merely the outcome of learning syntactic patterns and surface level statistics, or do they extract semantics and a world model from the text? Prior work by Li et al. investigated this by training GPTT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model's 's internal representations using linear probes and contrastive activations. The model is given a priori knowledge of the game and is solely trained on next character prediction, yet we find evidence of internal representations of board state. We validate these internal representations by using them to make interventions on the model's activations and edit its internal board state. Unlike Li et al's prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character. We derive player skill vector and add it to the model, improving the model's win rate by up to 2.6 times.
1 Introduction
Large language models (LLMs) have shown unprecedented capabilities that extend far beyond their initial design for natural language processing. Models trained with a next word prediction task can write code, translate between languages, and solve logic problems. However, these models are also black boxes trained on massive datasets that are typically private. Because of this opacity, there has been debate over how these capabilities emerge. Some have argued that LLMs' apparent competence is deceptive, and is largely due to the models learning spurious surface statistics (Bender et al. 2021). Others have argued that, to extent, LLMs are developing an internal representation of the world (Bowman) 2023).
One approach to study this problem is by training a general language model architecture on narrow, constrained tasks. [i et al. (2023) used this approach and trained language model on the game of Othello. The model OthelloGPT was trained on a next token prediction task dataset of Othello game sequences. Despite having a priori knowledge of the game Othello, the model learned to play legal Othello moves. Using non-linear probes (classifiers that predict board state given the model's activations), the authors were able to recover the model's 's internal state of the board. They were further able to perform interventions using these probes, by editing the model's internal board state and confirming that it plays legal moves under the new state of the board.
The authors were only able to recover the model's internal board state using non-linear probes, and were unsuccessful when using linear probes. In follow-up work, Nanda et al. (2023) found that the model had a linear representation that could be recovered using linear probes. Nanda et al. were also able to successfully intervene on the model's activations using the linear probes.
Code htt.tp///ggthib.ommmmmmmarrreere Ilm interpretability

---

Published as a conference paper at COLM 2024
However, the authors were only successful when examining model trained synthetic dataset games uniformly sampled from the Othello game tree. They tried the same techniques model trained using games played by humans and had poor results, A natural question arises from this: do language models only form robust world models when trained on large synthetic datasets?
We build on OthelloGPT to investigate this question more complex setting by training language model on chess game transcripts. We find that language model can also learn to play legal and strategic chess moves, and we train linear probes that recover the internal board state and perform successful interventions using these probes (Figure [ Thus, we demonstrate that language models can also form world models when trained on non-synthetic datasets.
Our model is trained on real human games, not synthetic games. As result, there are interesting latent variables we can probe for. We find that to better predict the next character, the model learns to estimate the Elo rating of the players in the game. We validate this representation by using it to both improve and decrease the model's chess playing ability.
Ground truth white pawn positions
Predicted white pawn positions
Confidence gradient white positions
-5
-10
Ground black king position
Predicted black king position
Confidence gradient black king position
-15
-2
-30
Figure 1: Heat maps of the model's internal board state derived from the probe outputs, which have been trained probabilities for the 13 different piece types at every square, which we can use to construct heat map for any piece type. The left heat maps display ground truth piece locations. The right heat maps display gradient of model confidence on piece locations. To view more binary heat map, we can clip these values to between -2 and 0, which can be seen in the center heat map. The model has reasonable representations. It is very confident that the black king is not on the white side of the board.
one-hot classification objective. The probes output log
2 Chess Model Training
Our goal is to investigate the internal representations of language models in more con- strained setting. Thus we train a standard language model to play the game, instead more targeted system like AlphaZero (Silver et al. 2018), which is built to win competitive games. We intentionally give the model no a priori knowledge of chess to mimic the setting of training on natural language.
2

---

Published as a conference paper at COLM 2024
2.1 Dataset
The authors in Li et al. (2023a) trained two models one on 16 million game dataset of synthetic games unitormly sampled from an Othello game tree, and one on 132,921 games from online Othello games played by humans. We hypothesized that the poor performance of the model trained on human games was due to the small size of the dataset. Chess is much more popular game and there are billions of games available online. We download 16 million games from the public Lichess chess games database, 120 times the size of the original Othello human game dataset.
2.2 Model Training
Average Chess-GPP GPT Win Rate by Opponent Skill over 1000 rounds
16 layers
0.6
0.5
Averoge Win Rate
0.4-
0.3-
Steckffsh
Stockfsh
Stockfis 2
Stockfish 3
Stockfis 4
Stockfih 5
Stecffsh
Stockfis
Stockfish 8
Stockfish Level (tim 0.1 seconds per move)
Figure 2: Model win rate versus range of Stockfish 16 levels. The 16 layer and 8 layer mod- els are trained on identical datasels for an identical number of epochs. The 16 layer model consistently has higher win rate than the 8 layer model. While precise Elo measurements for Stockfish are complex, a reasonable approximation for level 0 is around 1300 Elo. Details in Appendix
We train two models, an and 16 layer GPT model (Radford et al. 2018 with 512 dimensional hidden space with respective parameter counts of 25 million and 50 million. Each layer consists of 8 attention heads. The input to the model is a chess PGN string (1.e4 e5 2.. ...) of a maximum length of 1023 characters, with each character representing an input token. The model's vocabulary consists of the 32 characters necessary to construct chess PGN strings.
The model is initialized with random weights and is given no a priori knowledge of the game or its rules. It is solely trained on an autoregressive next token prediction task. For given sequence of characters, the model must predict the next character.
The models' performance against Stockfish is detailed in Figure 2 The 16 layer model has legal move rate of 99.8%, and the 8 layer model of 99.6%. After training, the 8 layer model achieved win rate of 46% against Stockfish 16 level 0 (~1300 Elo), while the 16 layer model had win rate of 64% (details in Appendix E Thus a standard GPT, orders of magnitude smaller than modern LLMs like Llama or GPT-4, will learn to play strategic chess if given dataset of millions of chess games.
We also checked if it was playing unique games not found in its training dataset, rather than regurgitating memorized game transcripts. Because we had access to the training dataset, we could easily examine this question. In random sample of 100 games, every game was unique and not found in the training dataset by the 10th turn (20 total moves). How is the model successfully playing legal moves? A plausible answer is that it learns to
3

---

Published as a conference paper at COLM 2024
track the state of the board. To examine this question, probe the model's internal representations.
3 Probing Internal Model Representations
3.1 An Internal Board State Representation
standard tool for measuring a model's internal knowledge is a linear probe (Alain & Bengiio 2016 Belinkov 2022). We can take the internal activations of a model it is predicting the next token, and train a linear model to take the model's activations as inputs and predict board state output. A linear probe has little capacity and the state of the board is not a linear function of the input. If the linear probe accurately predicts the state of the board, then it demonstrates that the model transforms the input into a linear representation of the board state in its activations.
Given a chess board with 64 squares, we aim to predict the state of each square, i, using linear probe. For each square, the probe is designed to classify it into one of 13 possible states (blank, white or black pawn, rook, bishop, knight, king, queen). The linear probe for square at layer of the model is formulated as follows:
pi = Softmax Ar W1,)
Where:
Pt is the probability distribution over the 13 classes for square i, as predicted by the probe at layer
Wi is 512 x 13 weight matrix for the linear probe associated with square i at layer
L,
A the 512-dimensional activation vector from layer I of the model,
The Softmax function is applied convert the logits into probabilities.
This method involves training separate linear probes for each of the model's layers and each individual square on the board, where each probe at layer takes the activations after layer I as input to predict the state of square We follow Nanda et al. (2023 's finding that the model represents squares as containing (Mine, Yours, Empty) rather than (Black, White, Empty). To use this objective, for every layer, we must train one group of 64 linear probes for predicting the board state at every white move and a second group of 64 linear probes for every black move, rather than single group of 64 linear probes for every move.
We train our probes on 10,000 games not found in the model's training dataset. An additional set of 10,000 games, also excluded from the model's training dataset, serves as our held out test set. As detailed in Table [1 the most accurate probe achieves a 99.6% accuracy in classifying the state of each square across the test games. Further insights into the model's performance across different layers are detailed in Figure 3 Most notably, the 8 layer model obtains peak accuracy at layer 6, while the 16 layer model doesn't t obtain comparable accuracy until layer 12. Intuitively, one might expect that the most efficient approach would be calcu calculating the board state as soon as possible and then using this information to calculate the next move. Instead, we see the deeper model using twice as many layers to calculate its ultimately more accurate board state.
We can visualize the model's internal representation of the board by creating heat maps. These heat maps were derived from the probe outputs, which had been trained on a one-hot objective to predict whether chess piece, such the black king, was present on a given square (1 if present, 0 if not). As seen in Figure [ the model's internal representation reflects reasonable facts about the board state. For example, the internal representation is very confident that the black king is not the white side of the board.
4

---

Published as a conference paper at COLM 2024
Model
Square
Elo
layer trained
99.6
90.5
8 layer trained
99.1
88.6
16 layer randomized
74.7
69.3
8 layer randomized
75.0
70.2
Table 1: Comparison the best square and skill classification accuracies (%) obtained for the most accurate probe for each model. Appendix
F
contains F1 scores per piece type.
Probe Square Classification Accuracy per Layer
Probe Classification Accuracy per Layer
1.0
0.9
Accuracy
0.7
16 layer model
8 layer model
0.6
8 layer randomized
16 layer randomized
10
12 14 16
30 12
14 16
Layer
Layer
Figure We linear probes for player Elo classification and board square state classifica- tion on every layer of each model. The 8 layer model computes an accurate board state by layer 6, yet the 16 layer model doesn't obtain similar accuracy until layer 12. Oddly, the skill probes trained on randomized models become more accurate on deeper layers.
3.2 Probing For Latent Variables
Because the model learned to predict the next move in competitive game, rather than game uniformly sampled from a game tree, there are interesting latent variables we can probe for. In particular, our dataset of chess games has the Elo ratings of the players for every game, which we can use to supervise the training of linear probe.
Initially, trained the probe the regression task of predicting the Elo rating of the white player between turns 25 and 35, as it would be difficult to predict player skill early in the game. Despite achieving promising average error of 150 Elo points, this approach was complicated by the narrow Elo range prevalent in our dataset (the majority of games are between 1550 and 1900 Elo). Distinguishing between players of closely matched skill levels, such as between a 1700 and 1900 Elo player, is intrinsically difficult. This narrow range makes baseline methods appear deceptively effective, as linear probe trained on randomly initialized model had an average error of 215 Elo.
To better evaluate the model's understanding of player skill, we trained a probe on classification task, where it had to classify players as below Elo of 1550 or above 2050. As detailed in Table [ a probe trained on the real model performs significantly better than probe trained on randomly initialized model, suggesting that the trained model learns to estimate player skill order to better predict the next character. is further evidence of the ability of neural networks to lear unsupervised representations of high level concepts using next token prediction, especially notable as our models are orders of magnitude smaller than today's state of the art models.
The layer-by-layer performance of our probes is depicted in Figure 3 Peak accuracy for skill classification is obtained at the final layer of the model, unlike board state classification

---

Published as a conference paper at COLM 2024
where peak accuracy is obtained in the mid to late layers. Based on the observed differences in layers for peak accuracy in skill and board state classification, we hypothesize that the model computes board state and range of possible next moves in parallel to estimating player skill. At the final layers, the player skill estimate influences which move is selected. The observation that skill probes trained randomized models become more accurate on deeper layers is peculiar and warrants further investigation into potential unexpected patterns within model architectures.
Model Interventions
Our probes suggest that the model computes internal representation of the board. To validate the information found by probes, it is recommended to perform interventions and establish a causal relationship between the internal board state representations and the model outputs (Belinkov) 2022). For our GPT, we use the probes to causally intervene on the model's activations as is predicting the next token. If there between the model's 'ss internal board state representations and the model's predictions, we should able to influence the model's chess playing ability and edit its internal state of the board.
causal relationship
4.1 Intervention Technique
Linear probes enable a simple intervention approach of vector addition. In this case, our probes are trained on the model's residual stream, which is a 512 dimensional intermediate state vector output by each layer before being fed to the next layer. To intervene on the model's activations, we can simply add or subtract vectors derived from our linear probe from the model's residual stream. Alternatively, we can add or subtract our vector to or from the transformer layer's 's final Multilayer Perceptron (MLP) bias term, which gives an equivalent intervention that has zero additional inference cost.
x + apd
Here, d represents a specific direction of a probe p, such as (High Skill, Low Skill) derived from the skill probe, or one of the 13 different possible square states derived from the board state probe. is a scaling factor that modulates the intervention's 's magnitude. denotes the model's intermediate activations or a layer's final bias term both 512-dimensional vectors. We can apply this intervention at a single layer or sequentially to range of layers. For detailed discussion on the selection of intervention hyperparameters, see Appendix C
Additionally, we explore the technique of contrastive activations for skill interventions (Rimsky et al. 2023 Zou et al., )023). In this case, we obtain the average model activation in 2,000 high skill games and 2,000 low skill games between turns 25 and 35. We subtract the low skill activation from the high skill activation to obtain a skill vector, which we can again add subtract from the model's residual stream MLP bias term.
4.2 Board State Interventions
Our board state intervention is detailed in Figure 4 We test our intervention on 5,000 different board states. Intuitively, we find a piece the model intends to move and remove the piece from the model's 's internal board state by modifying the model's activations (we do not alter the input PGN string), then sample additional moves from the model. our intervention is successful, the model's moves will be legal under the new hypothetical board state.
To evaluate our intervention, we sample five probabilistic moves from both the original and modified model at a temperature of 1.0. The original model provides a baseline legal move rate with no intervention applied, and the modified model's 's legal move rate measures the success the intervention. Our intervention significantly outperforms our baseline as detailed in Table 2 However, the best intervention success rate obtained is 92%, indicating

---

Published as a conference paper at COLM 2024
1
W
M
M
Original Model
Intervention
Modified Model Modified Board
Original Board
Figure 4: Board State Intervention Process. We first sample the model's next move prediction, identifying strategically relevant piece the model intends move (e.g., white pawn from C2 to C3). We then delete this piece from both the original board and the model's internal representation by subtracting the corresponding vector (the 512-dimensional "my pawn" vector from the C2 square linear probe) from the model's s residual stream. Using the unmodified PGN string input, we generate 5 new moves from the modified model. A successful intervention results in all moves being legal under the hypothetical modified board state, despite the model receiving no explicit information about the piece removal.
Model
Original Board
Modified Board
16 layer with intervention
85.4
92.0
8 layer with intervention
81.8
90.4
16 layer no intervention
99.9
40.5
8 layer intervention 99.8
41.0
Table 2: Legal move rates (%) for models with and without the board state intervention on the original and hypothetical modified boards (see Figure 4 Results based on 5,000 test cases, sampling moves per case at temperature 1.0. The baseline is the original model on the modified board (bottom right quadrant). This has only 41% legal moves, typically result of the original model moving the deleted piece. By applying our intervention (top right quadrant), the legal move rate improves significantly to 92%. The original model on the original board is in the bottom left quadrant. The modified model's legal move rate decreases on the original board when moves pieces into occupied squares (top left quadrant). An example cause would be moving the D3 Queen to C2 in This is legal on the modified board but illegal on the original board.
that there is potential for improved model intervention strategies. As seen in Figure 5 the intervention can have unintended side effects and make the positions of other pieces less distinct.
4.3 Model Skill Interventions
We obtain one skill vector from contrastive activations and the second by subtracting the linear probe's ss low skill vector from its high skill vector. We can add the skill vector to the model's activations to increase its skill. We can flip the sign of the intervention to decrease the model's skill. An excessively large intervention will cause the model to output illegal moves and random characters. When testing our intervention to both increase and decrease model skill, ensure that the model still makes over 98% legal moves after the intervention. Although the skill probe was trained to predict the Elo ratings of players specifically between turns 25 and 35, we add the skill intervention vector to the model's
7

---

Published as a conference paper at COLM 2024
Model and Board
No Intervention
Positive
Negative
16 layer standard board
69.6
72.3
11.9
8 layer standard board
49.6
54.8
5.5
16 layer random board
16.7
43.2
5.9
8 layer random board
15.7
31.3
3.6
Table Model win rate (%) a Stockfish level over 5,000 games with positive, negative, and interventions. We can add (positive intervention) or subtract (negative intervention) the skill vector from the model's activations to increase and decrease its chess playing ability. We test the model on both standard chess board and on boards initialized with 20 random moves (random board). In the case of the random board, we can improve the model's win rate by up 2.6 times by adding our positive skill intervention.
Pawn Position Heat Map Pre-intervention
Pawn Position Heat Map Post-intervention
2
2
Figure In this intervention, we delete the pawn described in Figure 4 After the inter- vention, the C2 pawn has been erased from the model's internal representation. However, the other pawns are less distinct, indicating that the intervention has had unintended side effects.
MLP bias term, which means that is applied at every token in the input PGN string. We observe that it maintains effective performance across all stages of the game.
We perform this intervention two settings. the first, we have the model play 5000 games against Stockfish 16 beginning from standard chess board. In the second, the model plays 5000 games against Stockfish 16, but every board is initialized with 20 randomly selected moves. The results are contained in Table 3 In the standard board, we notice slight but measurable increase in skill when adding the skill vector, and significant decrease in skill when subtracting the skill vector. possible explanation that the average skill level per game in the model's training dataset is 1,690 Elo, higher than the model's capability and limiting the room for performance improvement in typical scenarios. In the randomly initialized board, both models perform significantly better with positive skill intervention.
The layer model's win rate improves by 2x, and the 16 layer model's win rate improves by 2.6x. The 16 layer model's larger positive increase is possibly because it has more latent ability. More details are contained in Appendix E
One potential explanation is model predicting the next character in game with 20 random moves predicts the next character as if the players have low skill. By adding a positive skill intervention, the model can restore significant fraction of its ability. To investigate this

---

Published as a conference paper at COLM 2024
hypothesis, we tested our Elo probe on 4,000 randomly initialized games. The Elo probe classified 99.4% of these games as low skill, adding evidence favor of this explanation.
Related Work
This work was follow-on to work done by [ al. (2023b) and Nanda et al. (2023) where they trained an LLM on synthetically generated games of Othello. Early models such as AlphaZero were trained play competitive chess and Go, and given built in knowledge of the board and game rules (Silver et al. 2018). In these models, evidence has been found of familiar chess concepts (McGrath et al. 2021). Toshniwal et (021) trained language model to play chess and found it could play legal moves, although they did not investigate its internal representations.
There have been several examples of researchers finding semantics extraction in Large Language Models trained on text. In 2017, researchers trained a long short-term memory network (LSTM) to predict the next character in Amazon reviews and found that linear probes trained on the model exceeded the state of the art in sentiment classification (Radford et al. More recently, researchers have found evidence of concepts such truthtulness and emotion in the Llama LLM (Li et al. 2023b Zou et al. 2023).
A growing body of work has looked at intervening language models using these repre- sentations, such as by changing the sentiment or truthfulness of the model's response by modifying its activations. [Li et (2023a) used gradient descent change Othello-GPT's ac-
tivations. Other approaches pertorm linear arithmetic with "truth" vectors [i et al. (2023b),
"task vectors" Ilharco al. (2022), or a variety of steering vectors (Zou et 2023 Rimsky
ee al. 2023
Subramani et al.
2022
6
Conclusion
We provide evidence that an LLM trained on a next token prediction task can develop world model of complex systems such chess, including the ability to estimate latent vari- ables such as player skill. We validate these representations by making causal interventions the model to increase and decrease its skill and edit its infernal state of the board.
Chess a constrained setting compared to the domain of natural language. This constrained setting makes it possible to perform these nuanced measurements and interventions and provide valuable insights into the underlying mechanisms of steering vectors and other concepts in large language models. In addition, also enables exploration and comparisons of a variety of interpretability and intervention methods. There is room for improvement in our intervention approaches, and we can easily measure our ability to make fine-grained changes to the model's 's activations, such as by deleting single piece from the board.
However, chess is ultimately well-defined, rule-based system that does not capture the complexity and ambiguity present in less structured environments like natural language. Accurate understanding of LLM world models could enable many applications, such as detecting and reducing hallucinations. Our hope is that these techniques can become practical in these domains.
Acknowledgments
We thank Walt Woods, Alexander Grushin, Martin Wattenberg, Kenneth Li, Nikola Jurkovic, Jannik Brinkman, and Austin Davis for feedback and advice.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint ar Xr11::1610.0164,, 2016.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 4811)2207-219, 04 2022. ISSN 0891-2017. doi: 10.1162/coll 00422.

---

Published as a conference paper at COLM 2024
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Sam Bowman. Eight things to know about large language models. arXiv preprint ar arXiv:2304.00612, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arviv preprint ar arXiv:2212.04089, 2022.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint ar (rio:2206.00344/ 2023b.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Xiv preprint ar arXiv:2111 11.1111.999 2021.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30. Association for Computational Linguistics, 2023.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint ar arXic:1704.01444 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Ilama via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(611)) 1144, 2018
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Compu- fational Linguistics: ACL 2022, Pp. 566-581, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (0.18653/v1/200 wwwp://2..ininngs-add....o. findings-acl.4s. URL https //aclanthology 2022.022.02 findings-acl.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Learning chess blindfolded: Evaluating language models on state tracking. arXio preprint arXiv:2102.13249, 2021.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. ar Xiv preprint arXio:2310.01400 2023.
10

---

Published as a conference paper at COLM 2024
A Hyperparameters
Hyperparameter
Value
Optimizer
A
Learning Rate
0.001
Weight Decay
0.01
Betas
0.9, 0.99
Training games Epochs
10,000
5
Table 4: Hyperparameters used for training linear probes.
Hyperparameter
Value
Optimizer
AdamW
Learning Rate Schedule
Cosine
Max Learning Rate
3e-4
Min Learning Rate
3e-5
Weight Decay
0.1
Betas
0.9, 0.95
Batch Size
100
Block Size
1023
Training Iterations
600,000
Dataset Characters
6,5B
Dropout
0.0
model
512
heads
8
n_layers
8, 16
Parameters
25M, 50M
Table 5: Hyperparameters used for training our GPT models.
The 25M parameter model took hours to train on one RTX 3090 GPU. The 50M parameter model took 38 hours to train on four RTX 3090 GPUs.
B Reproducibility
All code, models, and datasets are available on GitHub and HuggingFace. The code is located at: http:/bggthuu..om///m com adamkarvonen/
wwwerveonnn//esss...eeee chess Ilm interpretabbility
Links to the HuggingFace models and datasets are located in the GitHub repo
C Intervention Technique Comparisons
When performing interventions, there are two primary prrper-parametes:: the scale of the intervention and the selection of layers for intervention. In general, we find that the scale parameter should be decreased the number of layers is increased. In addition, when performing Elo interventions, we had the choice between using vectors derived from the probes and contrastive activations. We observed that the intervention process was successful across a range of parameters, and report the best results obtained for each grouping.
We also explored the following probe based intervention techniques. For board state interventions: both subtracting the moved piece vector and adding the blank square vector. For skill interventions: subtracting the low skill vector XOR adding the high skill vector. All
11

---

Published as a conference paper at COLM 2024
techniques worked, but the techniques that obtained the best results were detailed in the paper. That is, only subtracting the moved piece vector for board state interventions, and subtracting the low skill vector and adding the high skill vector for skill interventions.
When using probe based interventions, we normalize the probe derived vectors before multiplying them by the scale coefficient. We find slightly better performance by not normalizing for contrastive activation based interventions.
The following are the hyperparameters used for the best result obtained for each configura- tion. We did not perform an exhaustive hyperparameter search. In all skill intervention tests, find that contrastive activations more successful than probe derived interventions.
Model and Intervention
Layers
Scale
Intervention Type
8 layer board intervention
4-7
1.5
Probe derived
16 layer board intervention
range(5,15,2)
2.0
Probe derived
8 layer positive Elo intervention
4-7
0.1
Contrastive Activations
16 layer positive Elo intervention
11-15
0.1
Contrastive Activations
layer negative Elo intervention
3-7
-0.2
Contrastive Activations
16 layer negative Elo intervention
1-15
-0.1
Contrastive Activations
Table 6: The hyper-parameter used for the most successful intervention in each configura- tion.
Model and Intervention
Layers
Scale
Win Rate
8 layer positive Elo intervention
5-7
0.6
24.7
16 layer positive Elo intervention
13-15
2.5
42.2
8 layer negative Elo intervention
5-7
-1.2
3.8
16 layer negative Elo intervention
7-15
-1.1
7.9
Table 7: The best performing probe derived skill intervention win rate (%) and hyperpa- rameters. The positive interventions are performed on random boards, and the negative interventions on standard boards.
We find that a single layer intervention can work almost well as multiple layer interven- tion. We had the following board intervention success rates per single layer intervention.
Layer
1
2
3
4
5
6
7
8
Success Rate
7.7
23.0
47.6
73.4
85.5
77.2
56.8
42.5
Table 8: Success rates single layer board state interventions our 8 layer model. Notably, the intervention on layer 4 is 85.5% successful, within 5% of the multi-layer intervention success rate. However, this intervention very sensitive the choice of layer and is thus less robust than multi-layer intervention. For single layer interventions, it's crucial to increase the scale parameter compared to multi-layer interventions.
D Dynamic Scale Experiments
To estimate the upper bound of the success rate for board state interventions, we explored varying the scale of intervention across a range of values for each move, seeking to identify scale that resulted in the model generating legal moves. We found scale that resulted in the model outputting legal moves approximately 98% of the time.
We considered d dynamically setting the scale parameter by targeting specific output logit value for the linear probe of interest, In this case, the target value was another parameter to sweep, and -10 seemed to work well.
12

---

Published as a conference paper at COLM 2024
Let represent the scale factor applied to the intervention vector, A denote the model activation vector, P the linear probe vector for the piece interest, and the intervention vector. All vectors are 512 dimensional. Our goal was to adjust such that the linear probe's output for the modified activations equals a predetermined target logit value, t. The formula for calculating is given by:
(A.P- P-) (V.P)
With some algebra, the equation can be rewritten to solve for the target logit value, f, as follows:
= A. P (V.
This calculation ensures that, after applying the scaled intervention s V to the model activations, the output logit of the linear probe approximates in this case chosen be 100. It worked well, but was consistently around 1-2% worse than simply using single scale value.
E
Model evaluations against Stockfish
For the win rate, win counts as 1 point, a draw as 0.5, and a loss as 0.
One limitation of our GPT models is the maximum input length of 1,023 characters, which limits the PGN string input to a maximum of approximately 92 turns or 184 moves. After that point, the model can continue to make legal moves if we crop the input block size, but it doesn't tt have the full context of the game history. When playing against Stockfish, we stop the game 90 turns and use Stockfish to evaluate the centipawn advantage at the current board state. If either player has an advantage greater than 100 centipawns, they are assigned the win. Otherwise,
counted as draw.
In the case of performing model skill interventions, we limit Stockfish to searching 100,000 nodes per move instead of limiting its search time per move. This uses significantly less search time per move, aiding hyperrparameter sweeps, and reduces variability that may be introduced by performing the comparison on different processors or processors under variable loads.
Elo estimation of Stockfish levels complex with several variables at play. An official estimate of Stockfish 16 Elos can be obtained in commit the official Stockfish repo at commit a08b8d42 In this case, Stockfish 16 level 0 is Elo 1,320. The official Stockfish testing was performed using 120 seconds per move, which is prohibitively expensive for tens of thousands of games on GPU machines. The particular processor being used per machine is another variable.
To estimate the Elo of Stockfish level 0.1 seconds per move, we had play 1,000 games as white against Stockfish level 0 with 0.1, 1.0 and 10.0 seconds per move. The win rate of 0.1 vs 0.1 seconds was 51%, 0.1 vs 1.0 seconds was 45%, and 0.1 vs 10.0 seconds was 45%. At low Stockfish levels, a 100x decrease in search time makes little difference in Stockfish's win rate. In contrast, at higher Stockfish levels, decreasing search time makes a significant difference in win rate. Thus, reasonable approximation of Stockfish 16 level 0 with 0.1 seconds per move is Elo 1300.
F Probe F1 Scores by Piece Type
Table provides a detailed breakdown of the linear probe performance for each piece type and board state. The trained models (8-layer and 16-layer) show consistently high Fl scores
httpss//ggtha..ccmmofffffitttthh fisosstcciiiittttsssssssss/ htt////mmmm//////////e/o
13

---

Published as a conference paper at COLM 2024
Piece Type
8 Layer Model
Layer Model
8 Layer Random Init
16 Layer Random Init
Opp. King
0.985
0.983
0.658
0.667
Opp. Queen
0.986
0.986
0.573
0.576
Opp. Rook
0.973
0.987
0.638
0.647
Opp. Bishop
0.985
0.996
0.480
0.480
Opp. Knight
0.986
0.993
0.494
0.473
Opp. Pawn
0.980
0.997
0.604
0.602
Blank
0.993
0.997
0.821
0.818
My Pawn
0.997
0.994
0.641
0.631
My Knight
0.991
0.997
0.519
0.499
My Bishop
0.994
0.998
0.489
0.475
My Rook
0.986
0.995
0.635
0.646
My Queen
0.992
0.998
0.569
0.570
My King
0.998
0.997
0.684
0.644
Table 9: Linear probe scores for predicting various piece types per model. The probes were evaluated on layer 6 of the layer model and layer 12 of the 16 layer model.
across all piece types, with most scores above 0.98. In contrast, the randomly initialized models show much lower Fl scores, typically in the 0.5-0.6 range, with better performance for blank squares.
14


Reviews:
[{"review_id": "IQsR2CyrEP", "reviewer": "Reviewer_mgGk", "summary": "### Summary of Paper\nThis paper shows that a model trained on human chess gameplay learns to track position information and models player skill level (ELO). Causal experiments suggest that these found representations play a part in how the model makes its decisions. \n\n### Summary of Review\nOverall, I think this work requires 1) additional experiments, 2) further depth for existing sections, and 3) some clarifications of current experiments and what the results imply. For example, they find that causally intervening on a skill vector can improve performance, but its unclear how that intervention is actually doing this or qualitatively what that change in behavior amounts to. As is, I think its hard to know what to conclude.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and feedback. We have addressed your key points below.\n\n**Significance of Results**\n\nOur research contributes to the debate on robust world models in OthelloGPT, which only emerged with large synthetic datasets. There were questions if this was due to size or synthetic nature of the data.\n\nWe emphasize the significance of the ELO intervention, demonstrating chess as a testbed for studying steering vector mechanisms, which are also applicable to latent variables like personality and style in LLMs. Chess provides measurable ground truth unlike natural language, making study of these mechanisms tractable.\n\nWhile OthelloGPT and ChessGPT can track board state, the how is not understood. ChessGPT enables deeper study with greater strategic complexity while maintaining an available ground truth. In general, interpretability is greatly hampered by lack of an existing ground truth in natural language.\n\n**Elo Probe Range**\nThe Elo probe works outside its training range. In 4.3, it accurately classifies games with 20 random initial moves as low skill on turn 10 (below the 25-35 training range).\n\n**Board State Intervention**\n\nWe edit the model's internal state (not PGN input). Thus the input PGN is invalid relative to the desired board state. If successful, the model's move is legal under the new \"make-believe\" state.\n\nWe subtract the steering vector for piece P to \"delete\" it in the model's internal representation. We sample 5 subsequent moves, which should be legal under the “make-believe” board. We verify that the modified board is valid and has at least one legal move before intervening.\n\n**Table 2 bottom-right** The original model tries to move the deleted piece, serving as an intervention effectiveness baseline.\n\n**Table 2 top-left** 92% of the time a non-ablated piece is selected. Most errors are attempts to move the ablated piece. The 85.4% quadrant errors are from trying to move a piece into the \"empty\" ablated square, which is not empty on the original board.\n\n**Addressing Other Questions**\n\n- We represent squares as (Mine, Yours, Empty) per Nanda+2023, training separate probe groups from each player's perspective. We will clarify this.\n- Detailed analysis of Elo intervention impact would be valuable but is not possible during the discussion phase due to conference review guidelines.\n- We did not measure per-move prediction accuracy. The character-level tokenizer loss cannot be straightforwardly converted to move accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The ELO intervention finding is interesting.\n* With some clarification, the fact that interventions are all relatively simpl/linear (but still effective) is telling.\n* This work supports a growing body of research, replicating findings observed for othello", "reasons_to_reject": "### Overview\n* Most of the papers' results would be the assumption given prior work (mcgrath and other probing over game papers like tomlin/forde et al + othello gpt, showing that models track game state.) Some readers would be unsure what to takeaway from reading this paper.\n* The paper doesn't uncover more on how such models are working under the hood; The exception here is that they do raise the possibility that the model would try to track multiple possible actions and then use the predicted player skill level to choose an appropriate move. Further testing this hypothesis would make this work stronger.\n* There are times in the paper where the setup is unclear. The hypothesis/alternatives are not always clearly defined nor how to interpret the results/metrics. \n\n### What to conclude?\nIt is hard to know what to conclude from reading this paper.\n\nW.r.t. to the positions of the pieces, I think the field understood that a model would track this information. Table 2 (for me) doesn't adjudicate on whether the found representations capture something meaningful. Spelling out the significance of the experiments and their conclusions would help. I think that the results say that 85.4% of the time the ablated piece is successfully removed and ignored simply by a linear intervention. Is that the case? I'm not perfectly confident that is how legal move rates should be interpreted here.\n\nW.r.t. to ELO, I think this is the most exciting part of the paper. Pushing this forward with more study and analysis would make this paper much stronger. (One could almost entirely for-go the position probing, or push it to the appendix.) An in-depth qualitative study of one (a few?) games and how the elo intervention changes the model's specific choices would be illuminating. How does the ranking of moves change with measured ELO? How does the ranking of moves change with intervened ELO? \n\n### Reviewing Dimensions of the Work\nOkay ``Empiricism, Data, and Evaluation``\n\nOkay ``Understanding Depth, Principled Approach``\n* I think this paper requires more depth; more diving into the details and further evaluation. Ultimately, this approach is empirical and I think empirically evaluating and understanding what is going on would be helpful. See some of my questions/notes for specifics.\n\nOkay ``Clarity, Honesty, and Trust``\n* Though I found some parts unclear, I think they positioned their findings with honesty/trust-worthiness.\n\nLimited ``Ambition, Vision, Forward-outlook``\n\nLimited ``Technological Impact``\n* Overall, this paper studied previously explored domains (chess board representations) without making a large contribution to further understanding that domain (chess board representations); I think their work does sketch some interesting hypotheses, but require further follow-up experiments.", "questions_to_authors": "### Questions\n* How do the moves change when you increase the \"elo\" for the standard-board games? A qualitative or expert or in-depth analysis of a game on a standard board (with the model boosted to perform even better, getting +3% to +5% win rate, and comparing the moves before and after might (while potentially misleading), would be exciting.\n* How do move probabilities and rankings change given the ELO/intervened-ELO?\n* How good at predicting the lichess games did the models get? Did the error correlate with game ELO? I guess at some point low-ELO games are harder to predict than mid/high level games. For the first, the moves are more random; for the second, its \"harder\" to find the good moves the players find.\n* How do you remove piece (pawn at c2?) and have a valid PGN/board-state?\n* In table 2, bottom-right quadrant: This result says that giving the modified board to the model w/o intervention has a high error rate? What makes the model error here?\n* In table 2, top-left quadrant: does this mean that 85.4% of the time it picks a new non-ablated piece to move? Does it ever try to move the ablated piece? What happens in that remaining 10%?\n* Does the ELO probe work for ranges that it wasn't trained on?\n* Table 3: If you subtract random noise of equal magnitude to the elo-vector does the model reach a similar win-rate (11.9)?\n* Do you have results for probing for \"my pieces\"/\"your pieces\" instead of black/white? [Following https://arxiv.org/pdf/2309.0094] \n\n### Notes\n* I find the move-board-state intervention logic confusing; the point is that you modify the state and then board and check that the model's move post the modified board and intervention state are the same? Writing more explicitly what you expect and what the performances mean would be helpful.\n* (suggestion) Adding board positions or labels to the axes of Figure 5 would help the reader track what C2 means. You could also add additional manual highlighting to the squares.\n* (nitpick) \"it is recommended\" in the first paragraph of \"4 Model Interventions\" sounds funny. I'd say, \"Following Belnikov, 2022, we...\" or directly argue for your case.", "extra_scores": {}}, {"review_id": "FPhd2UEc3n", "reviewer": "Reviewer_cUDn", "summary": "The authors study the ability of a chess playing language model to encode board state and player skill level.  The paper includes layer-wise linear probing experiments, finding that both 8- and 16-layer models can predict board state very accurately at an intermediate layer and player skill classification with increasing accuracy in each layer.  The paper also includes linear interventions, finding that the quality of play can be increased/decreased by adding/subtracting \"skill vectors\".\n\nThis work extends earlier analyses of the capabilities of chess language models, and also adds to the literature on world modeling in language models (at least in highly structured \"worlds\") more broadly.  Although Toshniwal et al. 2022 showed some world state tracking abilities of chess LMs (without probing the models layers), and Li et al. 2023 and Nanda et al. 2023 performed layer-wise probing experiments on Othello-playing LMs, I believe this paper provides the first analyses of layer-wise behavior of chess models.  I find the results about player skill to be particularly interesting; the ability to judge and adjust player skill is less intuitive than board state tracking, which had already been studied previously.  The paper is generally well written.  The contribution is somewhat limited since only a single model (of two sizes) was studied, and since the chess task itself is naturally limited.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and positive assessment of our paper. We appreciate your feedback and the opportunity to clarify a few points.\n\n**Implications for other models**\n\nCrafting steering vectors as demonstrated in the paper has broad implications for the NLP community. Steering vectors are interesting and have powerful effects on LLM behavior, such as increasing truthfulness or reducing hallucinations, but it is difficult to mechanistically study how they are modifying the LLM’s behavior as there is no underlying measurable ground truth in natural language. This is why we believe chess could serve as a valuable testbed for studying the mechanistic interpretability of steering vectors in LLMs and for mechanistic interpretability in general. In chess, there is an underlying ground truth we can measure for the world state, and Stockfish can be used as an oracle to grade the quality of moves after an intervention. The existence of this ground truth makes it much more tractable to understand what is going on internally.\n\n**Clarification of board state intervention**\n\nIntuitively, for the board state intervention, we edit the model’s internal board state (we do not alter the input PGN string) and if our intervention is successful, the model’s move will be legal under the new make-believe board state.\n\nWe begin with the original model’s deterministic next move based on a fixed PGN string. We identify the piece P the model intends to move during this first forward pass. Next, we perform another forward pass through the model, but this time we subtract the steering vector for P from the model’s activations. Conceptually, this deletes the piece from the model’s internal world representation. We sample from this intervened forward pass for 5 subsequent moves. If successful, the five moves should all be legal despite our intervention creating a make-believe state within the model’s internal world representation. Finally, we check that the moves are all legal using a separate chess library which we initialized from the FEN representation of the make-believe board where P was removed.\n\nWhen we delete the piece P above, we check that the modified board is still valid and has at least one legal move available before sampling from the model.\n\nThank you for pointing out the minor wording issues. We will clarify the linear representation sentence and revise the 2017 research sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Interesting results\n+ Generally well written", "reasons_to_reject": "- Limited setting\n- Unclear implications for other models", "questions_to_authors": "- I didn't entirely follow the board state intervention experiment.  I wasn't quite sure how the board is modified, or what is the difference between the board intervention (original vs. modified board) and the model intervention.  I can imagine how it may have been done, but could you provide more details?\n\n- In the case of board state intervention, it seems that the modified board state could be an impossible state.  Is that correct?  If so, can you comment on how this might impact the results?\n\n- A more expansive discussion/conclusion would be helpful, describing how the authors think their work could impact a broader range of models/settings.\n\nMinor wording issues:\n\n- I don't quite follow \"the model had a linear representation that could be recovered using linear probes.\"  Could you clarify?\n- \"In 2017, researchers trained a long short-term memory ...\" -- this sentence doesn't quite parse.  Maybe it's missing \"and\" after \"reviews\"?", "extra_scores": {}}, {"review_id": "2ZZdWgAWDU", "reviewer": "Reviewer_oZhN", "summary": "This study revisits the debate about whether language models (LM) extract semantics and a world model from text, or only surface level statistics through learning to play chess games. Although previous work shows that LM can only learn a world model from synthetic data, the authors hypothesize that the root cause is from the small amount of human behavior data being used. The authors empirically prove this hypothesis by gathering a large amount of online chess game records played by humans, and train a LM to demonstrate it. The authors further attest this by conducting linear probing based on the LM’s internal activations to predict chess board states and player skills, which clearly aligns with the hypothesis.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful questions. We appreciate your recognition of the paper's quality, clarity, and significant contribution to understanding what language models learn beyond surface-level statistics.\n\n**Elo probe range**\n\nWe chose the range of 25-35 turns as it was far enough along in the game to have sufficient information from which player skill could be inferred. If we use all turns (such as turns 0-35), the linear probe's accuracy decreases. This is because there is little information in early game moves that can reliably predict player skill. For example, it is nearly impossible to predict player skill after the first turn. If we use fewer turns or a constant number, such as only using data from turn 25, we would need approximately 10 times as many games for the linear probe to obtain the same amount of training data as using the 25-35 turn range.\n\n**Layer performance difference**\n\nOne hypothesis is that the model computes a board state and probability distribution over next moves, and then uses its skill estimate to shift the probability distribution. In appendix C, skill interventions are typically most successful in later layers compared to board state interventions, offering tentative evidence towards this hypothesis. A more careful examination of intervention success by intervention type and layer number would be helpful. It would be best to use mechanistic interpretability techniques to further understand how the skill estimation and board state interact, but we leave this to future work.\n\n**Figure 3, jump at layer 7**\n\nOur hypothesis is that the model has to process the full sequence of moves and game history in order to compute the board state. It may be able to first estimate player Elo while processing the game history, before board state is fully computed. We notice that in the 8-layer model result, it has a jump in elo classification at layer 5, before peak board state accuracy is achieved at layer 6.\n\nThank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, high-quality, and easy to follow.\n2. The experiment design and results are well-positioned and convincing, rendering little space to challenge the idea.\n3. The paper has a significant contribution to this line of work by clearly demonstrating that what is learned by LM is beyond surface statistics.", "reasons_to_reject": "1. A few analyses can be added to support the claim. Check my questions.", "questions_to_authors": "1. P5: for predicting the Elo, the current experiment uses the first 25-35 turns. Do you have studies on the difference between using less turns and more turns in this experiment? Also, why is the number of turns a range here, rather than a constant number?\n2. Figure 3: The hypothesis to explain the layer performance difference is that the model learns the chess board state first and then be able to infer the player Elo and next move. How can we design an experiment to dive deeper into this hypothesis?\n3. Figure 3: looking at the 16-layer model result, it seems that the model fully understands the chess board at 12th layer, while has a big performance jump in elo classification at 7th layer. Any potential hypothesis to explain this observation?", "extra_scores": {}}, {"review_id": "jZwvwaNEZm", "reviewer": "Reviewer_nmzh", "summary": "Builds up on previous works from [Li et al (2023)](https://arxiv.org/pdf/2210.13382) and [Nanda et al. (2023)](https://arxiv.org/pdf/2309.00941) to show that autoregressive transformer decoder models, when trained on chess moves, can learn to capture the board-state in its internal representations (latents). The authors also show that the model is capable of guaging the still of the player that the model is trying to imitate. And it is possible to improve the model's performance in certain settings with a simple intervention by adding a skill vector.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful comments.\n\n**Incremental nature**\n\nOur research contributes to the debate surrounding the robust world models found in OthelloGPT. OthelloGPT only had a robust world model when trained on a large synthetic dataset and did not find a robust world model when trained on human games, and there was debate on if this was due to the size or synthetic nature of the dataset.\n\nWe emphasize the significance of using chess as a testbed for studying steering vector mechanisms, as chess provides measurable ground truth unlike natural language. Mechanistic interpretability is severely hampered by the lack of a ground truth in language models. We aim to further this promising direction.\n\n\n**Question on linear probes.**\n\n We agree that clarification is needed. We propose modifying the paragraph as follows:\n\"To use this objective, for every layer, we must train one group of 64 linear probes for predicting the board state at every white move and a second group of 64 linear probes at every black move, rather than a single group of 64 linear probes at every move.\"\n\n**F1 score for probe outputs** We agree that measuring F1 score on a per-class basis would be a valuable addition to the paper. Although we cannot submit revisions during the discussion process, we have ran the experiments and will incorporate this analysis in the final version of the paper.\n\n\n**Gradient descent** We appreciate your thoughts on the gradient descent approach used by Li et al. (2023). In our initial experiments, we found that linear probe addition outperformed gradient descent by several percent, similar to Nanda et al's results.\n\n**Skill vector effects** Your interpretation of the model imitating a player according to the estimated skill level is spot-on. In the case of a game initialized with random moves, a good next-token predictor would predict legal random moves, which is what we find. By adding the skill vector, we can fool the model to imitate a stronger player, even if the game begins with random moves. We appreciate the connection to Andreas.\n\n**Board state interventions, Table 2, no intevention - Modified Board**\n\n Your understanding is correct. No intervention - modified board serves as a baseline in this case. We are evaluating the original model on the make-believe modified board, which has one piece deleted from it. In this case, 60% of the original model’s moves are illegal because it is attempting to move a deleted piece.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Provides further evidence for the claim that large transformer models do not just memorize surface-level patterns but also capture the world-model in their latents and use it to make certain decisions.\n\n* Well written paper with strong experiments.", "reasons_to_reject": "The paper is incremental in nature, directly building up on previous works. A *task* vector that makes the model perform a certain task, is also discussed in previously in the works ([Li et al (2023)](https://arxiv.org/abs/2306.03341), [Illharco et al (2023)](https://arxiv.org/abs/2212.04089), [Todd et al (2023)](https://arxiv.org/pdf/2310.15213)).", "questions_to_authors": "1. \"... To use this objective, we must train separate linear probes for predicting the board state at every white move and every black move ...\" - I was a little lost here. Correct me if I am wrong, but this is how I understand it: A probe $P_{i, l, m}$ is trained for a square $i$, at a layer $l$, for a move $m$; where $m$ is either *white turn* or *black turn*? Or $m \\in$ {1st white move, 1st black move, 2nd white move, 2nd black move, ...}?\n\nEither way (or I am missing the point completely) this should be clarified in the next revision. And if it is the latter, I am not sure how it follows the insights from Nanada et al. (2023).\n\n2. Data for training the probes is not balanced - at any time you would expect there to be more blank or pawns on the board. How did you ensure that class imbalance is not skewing the accuracy? Consider also reporting the class-wise accuracy or F1 scores. Maybe also add a confusion matrix reflecting the quality of all the probes (combined for all configs) trained for some of the (best performing) layers.\n\n3. On board state interventions, Li et al (2023)'s strategy using gradient descent is complicated, but it gives you enough flexibility to make sure that the intervention doesn't unintentionally effect the state of other squares. How does the simple addition/subtraction intervention ensure that? On Figure 5 you show that this intervention indeed has unintended side effects. Maybe gradients are the proper way to do this intervention? What's your take on this?\n\n4. Altough not explicitly discussed in the paper, it seemed to me that the model is trying to imitate a player according to the estimated skill level. This reminded me of [Andreas (2022)](https://arxiv.org/abs/2212.01681). Maybe by adding the skill vector you can fool the model into thinking that it is imitating a stronger player, and vice versa. Is this a reasonable assumption?\n\n\nSome minor questions/comments:\n\n - Missing reference in the abstract \"Li et al\"?\n\n - Whenever you draw chessboards the x-axis should always be ticked from $a$ to $h$. This will help following the discussion.\n\n - \"... do language models only form robust world models when trained on large synthetic datasets?\" - I found this question a bit odd. Why the dataset has to be *synthetic*? [Li et al (2023)](https://arxiv.org/pdf/2210.13382) used synthetic data since they didn't have access to enough real data for training. It can be argued that it is the size of the dataset that matters, not whether it is synthetic or not.\n\n - Board state interventions, Table 2 => does the *no intevention - Modified Board* mean that you didn't do an intervention yet, but you are evaluating the legality of next moves on the board state you intend to achieve after the intervention?\n\n - A glossary of terms/phrases such as; \"Elo Rating\", \"Stockfish\" would be helpful.\n\n**EDIT**: Rating increased to **7** after rebuttal", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers are enthusiastic about this work.

Strengths: The consensus is that this analysis of the linear representations within chess-game-trained LM transformers is interesting and clearly presented, with solid experimental findings that are a welcome complement and extension to previous similar work on OthelloGPT. The submission differs from that previous Othello work since Chess is different (and more complex) game; unlike the previous work, the probes here are able to achieve high accuracy on models trained on real gameplay data instead of synthetic data. Reviewers appreciated the experiment design including the causal steering-vector interventions for board state, and they highlighted the novelty and significance of the finding of an ELO skill latent which one reviewer described as the "most exciting" finding of the paper.

Weaknesses: The main weaknesses identified by reviewers indicate the reviewers' wish that the paper explored its findings in even more depth, for example, suggesting further analysis such as expert human analysis of elo interventions. And further analysis of the mechanisms that connect probed states to behavior.

Summary - Chess is one of the most venerable and well-studied classic testbeds for AI, and this paper's development of Chess as a testbed and demonstration for transformer LM interpretation methods will be a real value to the community. The authors should consider adding more in-depth analyses such as those suggested by the reviewers; a full-fleshed analysis that includes human evaluation and a few deeper investigations could make this solid paper into a potentially seminal paper.
Decision: Accept