Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

Authors: adam  karvonen
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 17:34:54

Abstract: Language models have shown unprecedented capabilities, sparking debate over the source of their performance. Is it merely the outcome of learning syntactic patterns and surface level statistics, or do they extract semantics and a world model from the text? Prior work by Li et al. investigated this by training a GPT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model's internal representations using linear probes and contrastive activations. The model is given no a priori knowledge of the game and is solely trained on next character prediction, yet we find evidence of internal representations of board state. We validate these internal representations by using them to make interventions on the model's activations and edit its internal board state. Unlike Li et al's prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character. We derive a player skill vector and add it to the model, improving the model's win rate by up to 2.6 times.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown unprecedented capabilities that extend far beyond their initial design for natural language processing. Models trained with a next word prediction task can write code, translate between languages, and solve logic problems. However, these models are also black boxes trained on massive datasets that are typically private. Because of this opacity, there has been debate over how these capabilities emerge. Some have argued that LLMs' apparent competence is deceptive, and is largely due to the models learning spurious surface statistics (Bender et al., 2021) . Others have argued that, to an extent, LLMs are developing an internal representation of the world (Bowman, 2023) .One approach to study this problem is by training a general language model architecture on narrow, constrained tasks. Li et al. (2023a) used this approach and trained a language model on the game of Othello. The model OthelloGPT was trained on a next token prediction task on a dataset of Othello game sequences. Despite having no a priori knowledge of the game Othello, the model learned to play legal Othello moves. Using non-linear probes (classifiers that predict board state given the model's activations), the authors were able to recover the model's internal state of the board. They were further able to perform interventions using these probes, by editing the model's internal board state and confirming that it plays legal moves under the new state of the board.The authors were only able to recover the model's internal board state using non-linear probes, and were unsuccessful when using linear probes. In follow-up work, Nanda et al. (2023) found that the model had a linear representation that could be recovered using linear probes. Nanda et al. were also able to successfully intervene on the model's activations using the linear probes.However, the authors were only successful when examining a model trained on a synthetic dataset of games uniformly sampled from the Othello game tree. They tried the same techniques on a model trained using games played by humans and had poor results. A natural question arises from this: do language models only form robust world models when trained on large synthetic datasets?We build on OthelloGPT to investigate this question in a more complex setting by training a language model on chess game transcripts. We find that a language model can also learn to play legal and strategic chess moves, and we train linear probes that recover the internal board state and perform successful interventions using these probes (Figure 1 ). Thus, we demonstrate that language models can also form world models when trained on non-synthetic datasets.Our model is trained on real human games, not synthetic games. As a result, there are interesting latent variables we can probe for. We find that to better predict the next character, the model learns to estimate the Elo rating of the players in the game. We validate this representation by using it to both improve and decrease the model's chess playing ability.Figure 1 : Heat maps of the model's internal board state derived from the probe outputs, which have been trained on a one-hot classification objective. The probes output log probabilities for the 13 different piece types at every square, which we can use to construct a heat map for any piece type. The left heat maps display ground truth piece locations. The right heat maps display a gradient of model confidence on piece locations. To view a more binary heat map, we can clip these values to be between -2 and 0, which can be seen in the center heat map. The model has reasonable representations. It is very confident that the black king is not on the white side of the board. 
chess model training: 
 Our goal is to investigate the internal representations of language models in a more constrained setting. Thus we train a standard language model to play the game, instead of a more targeted system like AlphaZero (Silver et al., 2018) , which is built to win competitive games. We intentionally give the model no a priori knowledge of chess to mimic the setting of training on natural language.The authors in Li et al. (2023a) trained two models -one on a 16 million game dataset of synthetic games uniformly sampled from an Othello game tree, and one on 132,921 games from online Othello games played by humans. We hypothesized that the poor performance of the model trained on human games was due to the small size of the dataset. Chess is a much more popular game and there are billions of games available online. We download 16 million games from the public Lichess chess games database, 120 times the size of the original Othello human game dataset.Figure 2 : Model win rate versus a range of Stockfish 16 levels. The 16 layer and 8 layer models are trained on identical datasets for an identical number of epochs. The 16 layer model consistently has a higher win rate than the 8 layer model. While precise Elo measurements for Stockfish are complex, a reasonable approximation for level 0 is around 1300 Elo. Details in Appendix E We train two models, an 8 and a 16 layer GPT model (Radford et al., 2018) with a 512 dimensional hidden space with respective parameter counts of 25 million and 50 million. Each layer consists of 8 attention heads. The input to the model is a chess PGN string (1.e4 e5 2.Nf3 ...) of a maximum length of 1023 characters, with each character representing an input token. The model's vocabulary consists of the 32 characters necessary to construct chess PGN strings.The model is initialized with random weights and is given no a priori knowledge of the game or its rules. It is solely trained on an autoregressive next token prediction task. For a given sequence of characters, the model must predict the next character.The models' performance against Stockfish is detailed in Figure 2 . The 16 layer model has a legal move rate of 99.8%, and the 8 layer model of 99.6%. After training, the 8 layer model achieved a win rate of 46% against Stockfish 16 level 0 (∼1300 Elo), while the 16 layer model had a win rate of 64% (details in Appendix E). Thus a standard GPT, orders of magnitude smaller than modern LLMs like Llama or GPT-4, will learn to play strategic chess if given a dataset of millions of chess games.We also checked if it was playing unique games not found in its training dataset, rather than regurgitating memorized game transcripts. Because we had access to the training dataset, we could easily examine this question. In a random sample of 100 games, every game was unique and not found in the training dataset by the 10th turn (20 total moves). How is the model successfully playing legal moves? A plausible answer is that it learns to track the state of the board. To examine this question, we can probe the model's internal representations. 
an internal board state representation: 
 A standard tool for measuring a model's internal knowledge is a linear probe (Alain & Bengio, 2016; Belinkov, 2022) . We can take the internal activations of a model as it is predicting the next token, and train a linear model to take the model's activations as inputs and predict board state as output. A linear probe has little capacity and the state of the board is not a linear function of the input. If the linear probe accurately predicts the state of the board, then it demonstrates that the model transforms the input into a linear representation of the board state in its activations.Given a chess board with 64 squares, we aim to predict the state of each square, i, using a linear probe. For each square, the probe is designed to classify it into one of 13 possible states (blank, white or black pawn, rook, bishop, knight, king, queen). The linear probe for square i at layer l of the model is formulated as follows:P i,l = Softmax(A l • W i,l )Where:• P i,l is the probability distribution over the 13 classes for square i, as predicted by the probe at layer l,• W i,l is a 512 × 13 weight matrix for the linear probe associated with square i at layer l,• A l is the 512-dimensional activation vector from layer l of the model,• The Softmax function is applied to convert the logits into probabilities.This method involves training separate linear probes for each of the model's layers and each individual square on the board, where each probe at layer l takes the activations after layer l as input to predict the state of square i. We follow Nanda et al. (2023) 's finding that the model represents squares as containing (Mine, Yours, Empty) rather than (Black, White, Empty). To use this objective, for every layer, we must train one group of 64 linear probes for predicting the board state at every white move and a second group of 64 linear probes for every black move, rather than a single group of 64 linear probes for every move.We train our probes on 10,000 games not found in the model's training dataset. An additional set of 10,000 games, also excluded from the model's training dataset, serves as our held out test set. As detailed in Table 1 , the most accurate probe achieves a 99.6% accuracy in classifying the state of each square across the test games. Further insights into the model's performance across different layers are detailed in Figure 3 . Most notably, the 8 layer model obtains peak accuracy at layer 6, while the 16 layer model doesn't obtain comparable accuracy until layer 12. Intuitively, one might expect that the most efficient approach would be calculating the board state as soon as possible and then using this information to calculate the next move. Instead, we see the deeper model using twice as many layers to calculate its ultimately more accurate board state.We can visualize the model's internal representation of the board by creating heat maps. These heat maps were derived from the probe outputs, which had been trained on a one-hot objective to predict whether a chess piece, such the black king, was present on a given square (1 if present, 0 if not). As seen in Figure 1 , the model's internal representation reflects reasonable facts about the board state. For example, the internal representation is very confident that the black king is not on the white side of the board. Because the model learned to predict the next move in a competitive game, rather than a game uniformly sampled from a game tree, there are interesting latent variables we can probe for. In particular, our dataset of chess games has the Elo ratings of the players for every game, which we can use to supervise the training of a linear probe.Initially, we trained the probe on the regression task of predicting the Elo rating of the white player between turns 25 and 35, as it would be difficult to predict player skill early in the game. Despite achieving a promising average error of 150 Elo points, this approach was complicated by the narrow Elo range prevalent in our dataset (the majority of games are between 1550 and 1900 Elo). Distinguishing between players of closely matched skill levels, such as between a 1700 and 1900 Elo player, is intrinsically difficult. This narrow range makes baseline methods appear deceptively effective, as a linear probe trained on a randomly initialized model had an average error of 215 Elo.To better evaluate the model's understanding of player skill, we trained a probe on a classification task, where it had to classify players as below Elo of 1550 or above 2050. As detailed in Table 1 , a probe trained on the real model performs significantly better than a probe trained on a randomly initialized model, suggesting that the trained model learns to estimate player skill in order to better predict the next character. It is further evidence of the ability of neural networks to learn unsupervised representations of high level concepts using next token prediction, especially notable as our models are orders of magnitude smaller than today's state of the art models.The layer-by-layer performance of our probes is depicted in Figure 3 . Peak accuracy for skill classification is obtained at the final layer of the model, unlike board state classification where peak accuracy is obtained in the mid to late layers. Based on the observed differences in layers for peak accuracy in skill and board state classification, we hypothesize that the model computes a board state and a range of possible next moves in parallel to estimating player skill. At the final layers, the player skill estimate influences which move is selected.The observation that skill probes trained on randomized models become more accurate on deeper layers is peculiar and warrants further investigation into potential unexpected patterns within model architectures. 
model interventions: 
 Our probes suggest that the model computes an internal representation of the board. To validate the information found by probes, it is recommended to perform interventions and establish a causal relationship between the internal board state representations and the model outputs (Belinkov, 2022) . For our GPT, we use the probes to causally intervene on the model's activations as it is predicting the next token. If there is a causal relationship between the model's internal board state representations and the model's predictions, we should be able to influence the model's chess playing ability and edit its internal state of the board.Linear probes enable a simple intervention approach of vector addition. In this case, our probes are trained on the model's residual stream, which is a 512 dimensional intermediate state vector output by each layer before being fed to the next layer. To intervene on the model's activations, we can simply add or subtract vectors derived from our linear probe from the model's residual stream. Alternatively, we can add or subtract our vector to or from the transformer layer's final Multilayer Perceptron (MLP) bias term, which gives an equivalent intervention that has zero additional inference cost.x ′ = x + αp dHere, d represents a specific direction of a probe p, such as (High Skill, Low Skill) derived from the skill probe, or one of the 13 different possible square states derived from the board state probe. α is a scaling factor that modulates the intervention's magnitude. x denotes the model's intermediate activations or a layer's final bias term -both 512-dimensional vectors. We can apply this intervention at a single layer or sequentially to a range of layers. For a detailed discussion on the selection of intervention hyperparameters, see Appendix C.Additionally, we explore the technique of contrastive activations for skill interventions (Rimsky et al., 2023; Zou et al., 2023) . In this case, we obtain the average model activation in 2,000 high skill games and 2,000 low skill games between turns 25 and 35. We subtract the low skill activation from the high skill activation to obtain a skill vector, which we can again add or subtract from the model's residual stream or MLP bias term.Our board state intervention is detailed in Figure 4 . We test our intervention on 5,000 different board states. Intuitively, we find a piece the model intends to move and remove the piece from the model's internal board state by modifying the model's activations (we do not alter the input PGN string), then sample additional moves from the model. If our intervention is successful, the model's moves will be legal under the new hypothetical board state.To evaluate our intervention, we sample five probabilistic moves from both the original and modified model at a temperature of 1.0. The original model provides a baseline legal move rate with no intervention applied, and the modified model's legal move rate measures the success of the intervention. Our intervention significantly outperforms our baseline as detailed in Table 2 . However, the best intervention success rate obtained is 92%, indicating 4 ). Results based on 5,000 test cases, sampling 5 moves per case at temperature 1.0. The baseline is the original model on the modified board (bottom right quadrant). This has only 41% legal moves, typically a result of the original model moving the deleted piece. By applying our intervention (top right quadrant), the legal move rate improves significantly to 92%. The original model on the original board is in the bottom left quadrant. The modified model's legal move rate decreases on the original board when it moves pieces into occupied squares (top left quadrant). An example cause would be moving the D3 Queen to C2 in 4. This is legal on the modified board but illegal on the original board.that there is potential for improved model intervention strategies. As seen in Figure 5 , the intervention can have unintended side effects and make the positions of other pieces less distinct.We obtain one skill vector from contrastive activations and the second by subtracting the linear probe's low skill vector from its high skill vector. We can add the skill vector to the model's activations to increase its skill. We can flip the sign of the intervention to decrease the model's skill. An excessively large intervention will cause the model to output illegal moves and random characters. When testing our intervention to both increase and decrease model skill, we ensure that the model still makes over 98% legal moves after the intervention. Although the skill probe was trained to predict the Elo ratings of players specifically between turns 25 and 35, we add the skill intervention vector to the model's Table 3 : Model win rate (%) against Stockfish level 0 over 5,000 games with positive, negative, and no interventions. We can add (positive intervention) or subtract (negative intervention) the skill vector from the model's activations to increase and decrease its chess playing ability. We test the model on both a standard chess board and on boards initialized with 20 random moves (random board). In the case of the random board, we can improve the model's win rate by up to 2.6 times by adding our positive skill intervention.Figure 5 : In this intervention, we delete the C2 pawn described in Figure 4 . After the intervention, the C2 pawn has been erased from the model's internal representation. However, the other pawns are less distinct, indicating that the intervention has had unintended side effects.MLP bias term, which means that it is applied at every token in the input PGN string. We observe that it maintains effective performance across all stages of the game.We perform this intervention in two settings. In the first, we have the model play 5000 games against Stockfish 16 beginning from a standard chess board. In the second, the model plays 5000 games against Stockfish 16, but every board is initialized with 20 randomly selected moves. The results are contained in Table 3 . In the standard board, we notice a slight but measurable increase in skill when adding the skill vector, and a significant decrease in skill when subtracting the skill vector. A possible explanation is that the average skill level per game in the model's training dataset is 1,690 Elo, higher than the model's capability and limiting the room for performance improvement in typical scenarios. In the randomly initialized board, both models perform significantly better with a positive skill intervention. One potential explanation is a model predicting the next character in a game with 20 random moves predicts the next character as if the players have low skill. By adding a positive skill intervention, the model can restore a significant fraction of its ability. To investigate this hypothesis, we tested our Elo probe on 4,000 randomly initialized games. The Elo probe classified 99.4% of these games as low skill, adding evidence in favor of this explanation. 
related work: 
 This work was a follow-on to work done by Li et al. (2023b) and Nanda et al. (2023) where they trained an LLM on synthetically generated games of Othello. Early models such as AlphaZero were trained to play competitive chess and Go, and given built in knowledge of the board and game rules (Silver et al., 2018) . In these models, evidence has been found of familiar chess concepts (McGrath et al., 2021) . Toshniwal et al. (2021) trained a language model to play chess and found it could play legal moves, although they did not investigate its internal representations.There have been several examples of researchers finding semantics extraction in Large Language Models trained on text. In 2017, researchers trained a long short-term memory network (LSTM) to predict the next character in Amazon reviews and found that linear probes trained on the model exceeded the state of the art in sentiment classification (Radford et al., 2017) . More recently, researchers have found evidence of concepts such as truthfulness and emotion in the Llama LLM (Li et al., 2023b; Zou et al., 2023) .A growing body of work has looked at intervening on language models using these representations, such as by changing the sentiment or truthfulness of the model's response by modifying its activations. Li et al. (2023a) used gradient descent to change Othello-GPT's activations. Other approaches perform linear arithmetic with "truth" vectors Li et al. (2023b) , "task vectors" Ilharco et al. (2022) , or a variety of steering vectors (Zou et al., 2023; Rimsky et al., 2023; Subramani et al., 2022) . 
conclusion: 
 We provide evidence that an LLM trained on a next token prediction task can develop a world model of complex systems such as chess, including the ability to estimate latent variables such as player skill. We validate these representations by making causal interventions on the model to increase and decrease its skill and edit its internal state of the board.Chess is a constrained setting compared to the domain of natural language. This constrained setting makes it possible to perform these nuanced measurements and interventions and provide valuable insights into the underlying mechanisms of steering vectors and other concepts in large language models. In addition, it also enables exploration and comparisons of a variety of interpretability and intervention methods. There is room for improvement in our intervention approaches, and we can easily measure our ability to make fine-grained changes to the model's activations, such as by deleting a single piece from the board.However, chess is ultimately a well-defined, rule-based system that does not capture the complexity and ambiguity present in less structured environments like natural language. Accurate understanding of LLM world models could enable many applications, such as detecting and reducing hallucinations. Our hope is that these techniques can become practical in these domains.Let s represent the scale factor applied to the intervention vector, A denote the model activation vector, P the linear probe vector for the piece of interest, and V the intervention vector. All vectors are 512 dimensional. Our goal was to adjust s such that the linear probe's output for the modified activations equals a predetermined target logit value, t. The formula for calculating s is given by: s= (A • P -t) (V • P)With some algebra, the equation can be rewritten to solve for the target logit value, t, as follows:t = A • P -s • (V • P)This calculation ensures that, after applying the scaled intervention s • V to the model activations, the output logit of the linear probe approximates t, in this case chosen to be -10.It worked well, but was consistently around 1-2% worse than simply using a single scale value. 
 
 Appendix: For the win rate, a win counts as 1 point, a draw as 0.5, and a loss as 0.One limitation of our GPT models is the maximum input length of 1,023 characters, which limits the PGN string input to a maximum of approximately 92 turns or 184 moves. After that point, the model can continue to make legal moves if we crop the input to block size, but it doesn't have the full context of the game history. When playing against Stockfish, we stop the game at 90 turns and use Stockfish to evaluate the centipawn advantage at the current board state. If either player has an advantage greater than 100 centipawns, they are assigned the win. Otherwise, it is counted as a draw.In the case of performing model skill interventions, we limit Stockfish to searching 100,000 nodes per move instead of limiting its search time per move. This uses significantly less search time per move, aiding hyper-parameter sweeps, and reduces variability that may be introduced by performing the comparison on different processors or processors under variable loads.Elo estimation of Stockfish levels is complex with several variables at play. An official estimate of Stockfish 16 Elos can be obtained in a commit on the official Stockfish repo at commit a08b8d42 . In this case, Stockfish 16 level 0 is Elo 1,320. The official Stockfish testing was performed using 120 seconds per move, which is prohibitively expensive for tens of thousands of games on GPU machines. The particular processor being used per machine is another variable.To estimate the Elo of Stockfish level 0 at 0.1 seconds per move, we had it play 1,000 games as white against Stockfish level 0 with 0.1, 1.0 and 10.0 seconds per move. The win rate of 0.1 vs 0.1 seconds was 51%, 0.1 vs 1.0 seconds was 45%, and 0.1 vs 10.0 seconds was 45%. At low Stockfish levels, a 100x decrease in search time makes little difference in Stockfish's win rate. In contrast, at higher Stockfish levels, decreasing search time makes a significant difference in win rate. Thus, a reasonable approximation of Stockfish 16 level 0 with 0.1 seconds per move is Elo 1300. Code at https://github.com/adamkarvonen/chess llm interpretabilityhttps://github.com/official-stockfish/Stockfish/commit/a08b8d4 


Decision: Reviewers are enthusiastic about this work.

Strengths: The consensus is that this analysis of the linear representations within chess-game-trained LM transformers is interesting and clearly presented, with solid experimental findings that are a welcome complement and extension to previous similar work on OthelloGPT. The submission differs from that previous Othello work since Chess is different (and more complex) game; unlike the previous work, the probes here are able to achieve high accuracy on models trained on real gameplay data instead of synthetic data. Reviewers appreciated the experiment design including the causal steering-vector interventions for board state, and they highlighted the novelty and significance of the finding of an ELO skill latent which one reviewer described as the "most exciting" finding of the paper.

Weaknesses: The main weaknesses identified by reviewers indicate the reviewers' wish that the paper explored its findings in even more depth, for example, suggesting further analysis such as expert human analysis of elo interventions. And further analysis of the mechanisms that connect probed states to behavior.

Summary - Chess is one of the most venerable and well-studied classic testbeds for AI, and this paper's development of Chess as a testbed and demonstration for transformer LM interpretation methods will be a real value to the community. The authors should consider adding more in-depth analyses such as those suggested by the reviewers; a full-fleshed analysis that includes human evaluation and a few deeper investigations could make this solid paper into a potentially seminal paper.

Reviews: review: paper_summary: paper_summary: ### Summary of Paper
This paper shows that a model trained on human chess gameplay learns to track position information and models player skill level (ELO). Causal experiments suggest that these found representations play a part in how the model makes its decisions. 

### Summary of Review
Overall, I think this work requires 1) additional experiments, 2) further depth for existing sections, and 3) some clarifications of current experiments and what the results imply. For example, they find that causally intervening on a skill vector can improve performance, but its unclear how that intervention is actually doing this or qualitatively what that change in behavior amounts to. As is, I think its hard to know what to conclude.
strength_weakness: strength_weakness: ### Overview
* Most of the papers' results would be the assumption given prior work (mcgrath and other probing over game papers like tomlin/forde et al + othello gpt, showing that models track game state.) Some readers would be unsure what to takeaway from reading this paper.
* The paper doesn't uncover more on how such models are working under the hood; The exception here is that they do raise the possibility that the model would try to track multiple possible actions and then use the predicted player skill level to choose an appropriate move. Further testing this hypothesis would make this work stronger.
* There are times in the paper where the setup is unclear. The hypothesis/alternatives are not always clearly defined nor how to interpret the results/metrics. 

### What to conclude?
It is hard to know what to conclude from reading this paper.

W.r.t. to the positions of the pieces, I think the field understood that a model would track this information. Table 2 (for me) doesn't adjudicate on whether the found representations capture something meaningful. Spelling out the significance of the experiments and their conclusions would help. I think that the results say that 85.4% of the time the ablated piece is successfully removed and ignored simply by a linear intervention. Is that the case? I'm not perfectly confident that is how legal move rates should be interpreted here.

W.r.t. to ELO, I think this is the most exciting part of the paper. Pushing this forward with more study and analysis would make this paper much stronger. (One could almost entirely for-go the position probing, or push it to the appendix.) An in-depth qualitative study of one (a few?) games and how the elo intervention changes the model's specific choices would be illuminating. How does the ranking of moves change with measured ELO? How does the ranking of moves change with intervened ELO? 

### Reviewing Dimensions of the Work
Okay ``Empiricism, Data, and Evaluation``

Okay ``Understanding Depth, Principled Approach``
* I think this paper requires more depth; more diving into the details and further evaluation. Ultimately, this approach is empirical and I think empirically evaluating and understanding what is going on would be helpful. See some of my questions/notes for specifics.

Okay ``Clarity, Honesty, and Trust``
* Though I found some parts unclear, I think they positioned their findings with honesty/trust-worthiness.

Limited ``Ambition, Vision, Forward-outlook``

Limited ``Technological Impact``
* Overall, this paper studied previously explored domains (chess board representations) without making a large contribution to further understanding that domain (chess board representations); I think their work does sketch some interesting hypotheses, but require further follow-up experiments.
questions: questions: ### Questions
* How do the moves change when you increase the "elo" for the standard-board games? A qualitative or expert or in-depth analysis of a game on a standard board (with the model boosted to perform even better, getting +3% to +5% win rate, and comparing the moves before and after might (while potentially misleading), would be exciting.
* How do move probabilities and rankings change given the ELO/intervened-ELO?
* How good at predicting the lichess games did the models get? Did the error correlate with game ELO? I guess at some point low-ELO games are harder to predict than mid/high level games. For the first, the moves are more random; for the second, its "harder" to find the good moves the players find.
* How do you remove piece (pawn at c2?) and have a valid PGN/board-state?
* In table 2, bottom-right quadrant: This result says that giving the modified board to the model w/o intervention has a high error rate? What makes the model error here?
* In table 2, top-left quadrant: does this mean that 85.4% of the time it picks a new non-ablated piece to move? Does it ever try to move the ablated piece? What happens in that remaining 10%?
* Does the ELO probe work for ranges that it wasn't trained on?
* Table 3: If you subtract random noise of equal magnitude to the elo-vector does the model reach a similar win-rate (11.9)?
* Do you have results for probing for "my pieces"/"your pieces" instead of black/white? [Following https://arxiv.org/pdf/2309.0094] 

### Notes
* I find the move-board-state intervention logic confusing; the point is that you modify the state and then board and check that the model's move post the modified board and intervention state are the same? Writing more explicitly what you expect and what the performances mean would be helpful.
* (suggestion) Adding board positions or labels to the axes of Figure 5 would help the reader track what C2 means. You could also add additional manual highlighting to the squares.
* (nitpick) "it is recommended" in the first paragraph of "4 Model Interventions" sounds funny. I'd say, "Following Belnikov, 2022, we..." or directly argue for your case.
paper_summary: paper_summary: The authors study the ability of a chess playing language model to encode board state and player skill level.  The paper includes layer-wise linear probing experiments, finding that both 8- and 16-layer models can predict board state very accurately at an intermediate layer and player skill classification with increasing accuracy in each layer.  The paper also includes linear interventions, finding that the quality of play can be increased/decreased by adding/subtracting "skill vectors".

This work extends earlier analyses of the capabilities of chess language models, and also adds to the literature on world modeling in language models (at least in highly structured "worlds") more broadly.  Although Toshniwal et al. 2022 showed some world state tracking abilities of chess LMs (without probing the models layers), and Li et al. 2023 and Nanda et al. 2023 performed layer-wise probing experiments on Othello-playing LMs, I believe this paper provides the first analyses of layer-wise behavior of chess models.  I find the results about player skill to be particularly interesting; the ability to judge and adjust player skill is less intuitive than board state tracking, which had already been studied previously.  The paper is generally well written.  The contribution is somewhat limited since only a single model (of two sizes) was studied, and since the chess task itself is naturally limited.
strength_weakness: strength_weakness: - Limited setting
- Unclear implications for other models
questions: questions: - I didn't entirely follow the board state intervention experiment.  I wasn't quite sure how the board is modified, or what is the difference between the board intervention (original vs. modified board) and the model intervention.  I can imagine how it may have been done, but could you provide more details?

- In the case of board state intervention, it seems that the modified board state could be an impossible state.  Is that correct?  If so, can you comment on how this might impact the results?

- A more expansive discussion/conclusion would be helpful, describing how the authors think their work could impact a broader range of models/settings.

Minor wording issues:

- I don't quite follow "the model had a linear representation that could be recovered using linear probes."  Could you clarify?
- "In 2017, researchers trained a long short-term memory ..." -- this sentence doesn't quite parse.  Maybe it's missing "and" after "reviews"?
paper_summary: paper_summary: This study revisits the debate about whether language models (LM) extract semantics and a world model from text, or only surface level statistics through learning to play chess games. Although previous work shows that LM can only learn a world model from synthetic data, the authors hypothesize that the root cause is from the small amount of human behavior data being used. The authors empirically prove this hypothesis by gathering a large amount of online chess game records played by humans, and train a LM to demonstrate it. The authors further attest this by conducting linear probing based on the LM’s internal activations to predict chess board states and player skills, which clearly aligns with the hypothesis.
strength_weakness: strength_weakness: 1. A few analyses can be added to support the claim. Check my questions.
questions: questions: 1. P5: for predicting the Elo, the current experiment uses the first 25-35 turns. Do you have studies on the difference between using less turns and more turns in this experiment? Also, why is the number of turns a range here, rather than a constant number?
2. Figure 3: The hypothesis to explain the layer performance difference is that the model learns the chess board state first and then be able to infer the player Elo and next move. How can we design an experiment to dive deeper into this hypothesis?
3. Figure 3: looking at the 16-layer model result, it seems that the model fully understands the chess board at 12th layer, while has a big performance jump in elo classification at 7th layer. Any potential hypothesis to explain this observation?
paper_summary: paper_summary: Builds up on previous works from [Li et al (2023)](https://arxiv.org/pdf/2210.13382) and [Nanda et al. (2023)](https://arxiv.org/pdf/2309.00941) to show that autoregressive transformer decoder models, when trained on chess moves, can learn to capture the board-state in its internal representations (latents). The authors also show that the model is capable of guaging the still of the player that the model is trying to imitate. And it is possible to improve the model's performance in certain settings with a simple intervention by adding a skill vector.
strength_weakness: strength_weakness: The paper is incremental in nature, directly building up on previous works. A *task* vector that makes the model perform a certain task, is also discussed in previously in the works ([Li et al (2023)](https://arxiv.org/abs/2306.03341), [Illharco et al (2023)](https://arxiv.org/abs/2212.04089), [Todd et al (2023)](https://arxiv.org/pdf/2310.15213)).
questions: questions: 1. "... To use this objective, we must train separate linear probes for predicting the board state at every white move and every black move ..." - I was a little lost here. Correct me if I am wrong, but this is how I understand it: A probe $P_{i, l, m}$ is trained for a square $i$, at a layer $l$, for a move $m$; where $m$ is either *white turn* or *black turn*? Or $m \in$ {1st white move, 1st black move, 2nd white move, 2nd black move, ...}?

Either way (or I am missing the point completely) this should be clarified in the next revision. And if it is the latter, I am not sure how it follows the insights from Nanada et al. (2023).

2. Data for training the probes is not balanced - at any time you would expect there to be more blank or pawns on the board. How did you ensure that class imbalance is not skewing the accuracy? Consider also reporting the class-wise accuracy or F1 scores. Maybe also add a confusion matrix reflecting the quality of all the probes (combined for all configs) trained for some of the (best performing) layers.

3. On board state interventions, Li et al (2023)'s strategy using gradient descent is complicated, but it gives you enough flexibility to make sure that the intervention doesn't unintentionally effect the state of other squares. How does the simple addition/subtraction intervention ensure that? On Figure 5 you show that this intervention indeed has unintended side effects. Maybe gradients are the proper way to do this intervention? What's your take on this?

4. Altough not explicitly discussed in the paper, it seemed to me that the model is trying to imitate a player according to the estimated skill level. This reminded me of [Andreas (2022)](https://arxiv.org/abs/2212.01681). Maybe by adding the skill vector you can fool the model into thinking that it is imitating a stronger player, and vice versa. Is this a reasonable assumption?


Some minor questions/comments:

 - Missing reference in the abstract "Li et al"?

 - Whenever you draw chessboards the x-axis should always be ticked from $a$ to $h$. This will help following the discussion.

 - "... do language models only form robust world models when trained on large synthetic datasets?" - I found this question a bit odd. Why the dataset has to be *synthetic*? [Li et al (2023)](https://arxiv.org/pdf/2210.13382) used synthetic data since they didn't have access to enough real data for training. It can be argued that it is the size of the dataset that matters, not whether it is synthetic or not.

 - Board state interventions, Table 2 => does the *no intevention - Modified Board* mean that you didn't do an intervention yet, but you are evaluating the legality of next moves on the board state you intend to achieve after the intervention?

 - A glossary of terms/phrases such as; "Elo Rating", "Stockfish" would be helpful.

**EDIT**: Rating increased to **7** after rebuttal

Comments: comment: make sense to me. Thanks for the clarification.
Thank you for the clarifications, and congrats on your interesting work!
Thanks for the response and clarifications! I'm excited by your findings and to see where this work goes next, and I'm glad that other reviews were quite positive. As raised, your experiments and findings come through. Though I stand by my comments, I think you make a good point highlighting that previous work was done in a synthetic setting. I am raising my score to a to 5: basically I would suggest some revision for clarity and additional experiments to confirm/understand how the elo intervention changes which moves are made. It looks like your work will be accepted regardless (congratulations!)!
I thank the authors for their detailed response. And I do appreciate the effort to compliment and extend previous works in this line of research.

On board state interventions, as far as I understand, the motivation of this experiment was to check if the model refers to its internal representation of the world (board state) to make decisions. Not to squeeze out more performance with a (dare I say) *"less principled"* intervention.

I am satisfied with the authors' response to other questions. I am happy to increase my score to 7. Although it seemed like this paper was gonna get accepted anyways,

Congratulations on your fine work!