Evidence of Learned Look-Ahead in a Chess-Playing Neural Network
Authors: Erik Jenner
Shreyas Kapur
Vasil Georgiev
Cameron Allen
Scott Emmons
Stuart Russell
Venue: NeurIPS 2024 poster (2024)
Keywords: Mechanistic interpretability
Learned search
Chess
Primary area: interpretability_and_explainability
TL;DR: We provide mechanistic evidence of learned look-ahead in Leela Chess Zero, a transformer trained to play chess.

Abstract: Do neural networks learn to implement algorithms such as look-ahead or search "in the wild"? Or do they rely purely on collections of simple heuristics? We present evidence of *learned look-ahead* in the policy and value network of Leela Chess Zero, the currently strongest deep neural chess engine. We find that Leela internally represents future optimal moves and that these representations are crucial for its final output in certain board states. Concretely, we exploit the fact that Leela is a transformer that treats every chessboard square like a token in language models, and give three lines of evidence: (1) activations on certain squares of future moves are unusually important causally; (2) we find attention heads that move important information "forward and backward in time," e.g., from squares of future moves to squares of earlier ones; and (3) we train a simple probe that can predict the optimal move 2 turns ahead with 92% accuracy (in board states where Leela finds a single best line). These findings are clear evidence of learned look-ahead in neural networks and might be a step towards a better understanding of their capabilities.

Evidence of Learned Look-Ahead in a Chess-Playing Neural Network
Erik Jenner UC Berkeley
Shreyas Kapur UC Berkeley
Vasil Georgiev Independent
Cameron Allen UC Berkeley
Scott Emmons UC Berkeley
Stuart Russell
UC Berkeley
Abstract
Do neural networks learn to implement algorithms such as look-ahead or search "in the wild"? Or do they rely purely collections of simple heuristics? We present evidence of learned look-ahead in the policy and value network Leela Chess Zero, the currently strongest deep neural chess engine. We find that Leela internally represents future optimal moves and that these representations are crucial for its final output in certain board states. Concretely, transformer that treats every chessboard square like a token in language models, and give three lines of evidence: (I) activations certain squares of future moves are unusually important causally; (2) we find attention heads that move important information "forward and backward in time," e.g.. from squares of future moves to squares of earlier ones; and (3) we train a simple probe that can predict the optimal move 2 turns ahead with 92% accuracy (in board states where Leela finds single best line). These findings are clear evidence of learned look-ahead in neural networks and might be step towards better understanding of their capabilities.
exploit the fact that Leela is
Introduction
Can neural networks learn use algorithms such as look-ahead or search internally? Or are they better thought of as vast collections of simple heuristics or memorized data? Answering this question might help us anticipate neural networks' future capabilities and give us a better understanding of how they work internally. Recent work has found interesting cases of learned optimization or reasoning in neural networks (von Oswald et al., b023a,b; Akyürek et al., 2023; Brinkmann et al., 2024). However, these works focus simple algorithmic domains, with models that are trained specifically for those research purposes. Instead, we ask: what computations networks learn to perform "in the wild" in more complex domains?
We study this question the microcosm of chess. Neural networks are surprisingly strong at chess, arguably approaching grandmaster level (Ruoss et al., 2024)-22200 how do they achieve that performance? Both reasoning and heuristics are plausible mechanisms. For example, human players and manually designed chess engines perform look-aheud-ther reason about which moves they will make in the future. On the other hand, network might simply learn heuristics based on the current state, such playing knight "fork" titacks-whiel are often advantagcous- look like geometrically. There evidence that residual networks (such as transformers) tend to additively aggregate results from many shallow circuits (Veit al., 2016) and incrementally improve their predictions (nostalgebraist, 2020; Belrose et al., 2023; Din et al., 2023), lending credence to this idea that networks might implement a large collection of heuristies. Since we know how to
based what they
1 Corresponding email: jenmer@berkeley..em edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Clean board state
gighi+
MI
a4
MAA
A
Patch
Cornerted beand clato
1st wove torget square
square
Bha
Clean octivation
Corrupted activotion
mave
Figure 1: Activation patching lets us study where important information stored in Leela. Here, we patch an activation in one particular square and layer from the forward pass on "corrupted" board state (bottom) into the forward pass on "clean" state (top). Each row in the network corresponds to chessboard square, which Leela treats like token in language model. The intervention drastically affects Leela's 's output (right), telling us that the activation on the patched square stores information necessary for Leela's performance in this state. Only patching specific squares has significant effects. See https: /ttp//eeellaaaeee/ee inttarinnrpppppp github. io/ for more (animated) examples.
hand-design chess engines, we know what reasoning to look for in chess-playing networks. Compared to frontier language models, this makes chess good compromise between realism and practicality for investigating whether networks learn reasoning algorithms rely purely on heuristics.
In this work, we look for evidence of look-ahead in the policy and value network of Leela Chess Zero (Leela Chess Zero team), Leela for short. Leela is an MCTS- system (like Alp- haZero (Silver et al., 2018)) and the strongest deep neural chess engine (Haworth and Hernandez, 2021). We use only its policy/value network, without external search, since want study algo- rithms that emerge within the network. Even the policy network, with only one forward pass per state, reaches rating 2600 Lichess (Appendix B) and is at least strong (lepned, 2024) recent model from Ruoss et al. (2024).
Our inroad for interpreting Leela is that it is transformer that treats each chessboard square like a token language model. Thus, we can consider activations on specific squares or attention weights between squares. We consistently find that these activations correspond their squares in meaningful ways; for example, information about a move typically seems to stored activations on the squares involved in that move. This lets apply common interpretability techniques to Leela.
We show that Leela has learned use look-ahead in certain states. Leela internally represents future moves of the optimal line of play. and these representations are causally important for Leela's output. We present three lines cvidence. (1) Activations on the target square (where the piece lands) of certain future moves have a substantially outsized impact on the network's output, determined by activation patching (Fig. and Section 2.3). (2) We identify attention heads that move information "forward and backward in time." For example, one attention head often moves crucial information from the target square future move to the target square an earlier move. (3) A simple, bilinear subset of Leela's ss activations can read off the best move two turss into the future with 92% accuracy.
probe (Hewitt and Liang, 2019)
Our contributions are: (1) We give evidence that neural networks can learn algorithms involving look- ahead "in the wild." (2) We take first steps toward a mechanistic understanding how look-ahead might be implemented in Leela to help it play chess. (3) We introduce techniques that might be useful

---

for interpretability more generally (e.g.. using weaker model to automatically generate corruptions for activation patching, as we will explain in Section 2.3). Our code available on Github.
2 Experimental Setup
This section will describe the model, dataset, and techniques we use. The following section will describe the specific experiments and their results. We ran all experiments on an internal cluster. Each experiment takes at most few hours on a fast GPU (c.g., an A100) and about a day for all experiments combined.
2.1 Leela Chess Zero
Leela is chess engine based on Monte Carlo Tree Search (MCTS), similar to AlphaZero (Silver al., 2018). We focus solely on its policy network, which takes a single board state as input and outputs probability distribution over all legal moves. For ease of exposition, we ignore the value network in the main paper, but we show in Appendix C that our results also apply there (likely because the two networks share common body). When we say "Leela" elsewhere in this paper, we mean the network rather than the full MCTS system.
The key to understanding our analysis is that Leela is a transformer that treats each of the 64 chesshoard squares as one sequence position, analogous to token in language model. This means each square has its own representation in the embedding space, allowing us to analyze activations and attention patterns on specific squares. Unlike a causal language model, attention is bidirectional between squares; there is no autoregressive prediction. Leela has 15 layers and 109M parameters, about the size of GPT2-small (Radford al., 2019).
The network computes a logit for every possible move, corresponding to moving a piece from one square (the source) to another (the target). Each logit is computed using only the final embeddings at the source and target square. There are several peculiarities of Leela's architecture that aren't 't crucial for understanding our results, we discuss them in Appendix A.
2.2 Puzzle dataset
To study look-ahead, we need a dataset of board states where Leela is especially likely to use look- ahead. We are not claiming that Leela uses analyze many states heuristically if there are no complex tactical considerations that require explicit loek- Thus, we focus on complex states that are likely difficult to evaluate heuristically.
in every state. Even human players can
As a starting point, we use 900k puzzles from the Lichess puzzles dataset (Lichess team). Each puzzle has a starting state with single winning move for the player whose turn it is. is also annotated with the principal variation, the optimal sequence of moves for both players from the starting state. All puzzles we consider have at least three moves? in their principal variation, see Fig. for an example.
We discard puzzles that a smaller and weaker version of Leela can solve. This ensures the states in our dataset are challenging and more likely to require look-ahead. We further filter for puzzles that Leela solves correctly, simply so that we can apply our interpretability methods: all our methods check whether Leela internally represents specific future line. In correctly solved puzzles, we look for representations of the correct continuation, but for puzzles that Leela fails to solve, it's unclear which line should look for. Leela may be representing the puzzle), but there are many incorrect lines (vs only one correct one). After this filtering process, 22.5k puzzles remain (mainly because many of the original puzzles are easy enough be solved by the smaller model, which means we discard them). See Appendix D for details.
incorrect continuation (and hence fail
The filtering process leads to overrepresentation of states where the 1st and 2nd move (i.e., the opponent's 's response) have the same target square. The 1st and 2nd target square coincide in 83% of the filtered puzzles, compared to 47% on the original dataset. This is because the starting player often sacrifices a piece that the opponent captures. These puzzles involving sacrifices may he more difficult for the weak model and thus overrepresented in our dataset. Interestingly, the results we present in
Note for chess players: for simplicity, we use the term "move" for what's often called a "ply" or "half-move" We never mean "full move", i.e. move by both white and black.
3

---

Initinl puzzle position
Position after move
Position after 2nd move
Wi
A
A
A
A
A
A
A
A
A
A
A
Egg++
bogh
Whdtt
g8+
812
Figure 2: Top row: An example of the puzzles we use. It white's turn in the starting state, and the only winning action is move the knight to g6. Black's only response taking the knight with the pawn; then white checkmates by moving the rook to h4. We will see the colored squares again: the target square of the move in this principal variation (green) and the target square of the 3rd move (blue). Below: Leela receives each state as a separate input and computes policy in that state.
Section 3 are weaker puzzles where the 1st and 2nd move target square differ and are stronger in cases where they coincide. Perhaps sacrifices are inherently "more interesting" for Leela, or some model components we identify are specialized for dealing with sacrifices. See Appendix H for results on both subsplits of the data; in the main text, we always present results on all puzzles.
2.3 Activation patching
Activation patching' is technique for measuring the causal importance of specific model components. For any given board state and model component (such particular square in particular layer), can tell us how important that component is for producing Leela's output in that state.
To do so, activation patching replaces the activations of the component in question with those from different forward pass (Fig. 1). For a given "clean" board state from our dataset, we use "corrupted" version of that state with small modification, such as a piece being added or removed. In the version of activation patching we use, we run a forward pass on the clean state, but "patch in" activations from forward pass on the corrupted state at the component we're ''re analyzing. We then continue forward pass as normal. If this intervention changes Leela's 's output significantly (compared to clean forward pass without intervention). the patched component must have contained necessary information about the clean state that differed in the corrupted state.
The choice of corruption determines which model mechanisms can study with activation patching. the corrupted state is very different from the clean state, many model components will have different activations, and activation patching won't I tell anything specific about look-ahead. Instead, we want a corrupted state that is similar to the clean one but differs in some key detail that has an outsized effect what move is best. Look-ahead or other sophisticated algorithms should pick up on the importance of this difference, but shallow heuristics should mostly ignore it.
To automatically find such "interesting" corruptions, we again use smaller and weaker version of Leela. We generate many small random corruptions, each modifying only single square or piece position. Then, we select corruptions that have large effect on Leela's 's preferred move but
SSimilar (Heimersheim and Nanda, 2024: Zhang and Nanda, 2024) to causal mediation analysis (Vig al., 2020), causal tracing (Meng et al., 2022), interchange interventions (Geiger et al., 2021)

---

Lo: mast log odds resitcciinn 6,39
L10: mast log odds rediactãon 5.40
L14: mas log odds rediuction 6,10
Con upocd
Eargat
square
A
A
A
3rd move target
A
A
A
A
Patched Square
Logg odldl reduciion
1st move target
3rd move target
Corrupted square(s)
Other squares
12 14
Layer
Figure 3: Results from activation patching in the residual stream. The top row shows results in single example state at three select layers. Darker squares correspond to larger effects from intervening on that square. In the early layer, the effect is strongest when patching the corrupted square h6, then in middle layers, the 3rd move target square becomes important, and finally the 1st move target square dominates late layers. The line plot below shows mean effects over the entire dataset, demonstrating that this pattern holds beyond just this example, The "other squares" line is the maximum effect over all 61 other squares (where the maximum is taken per board state and then averaged). Error bars are two times the standard error of the mean.
small effect on the weaker model's output. This targets mechanisms that explain why Leela gets these puzzles correct while the weaker model does not, making them more likely to be related to look-ahead. The exact algorithm for generating and filtering corruptions described in Appendix
3 Results
Leela uses look-ahead, it needs internal representations future moves. These could located anywhere in the model, but we will argue for a specific hypothesis: that Leela represents future moves on their source or target squares. This hypothesis is motivated by the fact that at the end of the network, the logit of move depends only on its source and target square. We guessing that something similar holds within the network for future moves, and our results bear out this hypothesis.
We present three lines of evidence for this specific look-ahead hypothesis. First, we show that activations on the target square of the move two turns into the future are unusually important for Leela's 's output. Second, we find attention heads that appear to help Leela consider the consequences of future moves, as well as head that moves information "backward time." Third, we demonstrate that simple probe can predict the optimal move two turns into the future with 92% accuracy. The convergence of these three lines of evidence strongly suggests that learned look-ahead an important mechanism behind Leela's 's impressive performance our dataset of puzzles.

---

3.1 Activations on future move squares are unusually important
If Leela uses look-ahead and represents future moves on their source or target squares, then we expect activations on these future squares to be especially important for its output. We test this using activation patching: we corrupt activations one square and layer at time by patching from the corrupted puzzle into the clean forward pass. We measure the causal effect of an intervention by the change log odds assigned to the ground-truth best move. If Leela is using look-ahead, we expect especially big reductions in log odds when we patch on the squares of optimal future moves.
activations
Indeed, we find that patching on the target square of the 3rd move reduces model performance by an unusual amount (Fig. 3). In layer 10, this intervention reduces the log odds of the correct move by an average of 1.88 0.04 (2or standard error of the mean), which corresponds a reduction in probability from e.g. 50% to 13%.
Patching the corrupted square or the 1st move target square also has large effects. This is unsurprising: the corrupted square is the only difference in the input encodings, so in early layers, this square is responsible for any subsequent differences in the forward passes and output. The 1st move target square directly affects the logits of the correct move, so has a big influence in late layers.
Patching any other square has much smaller effects. For each puzzle, consider the biggest effect from patching on any square other than the 1st move target, 3rd move target, or corrupted square, and average those maxima over puzzles (this is the "other squares" line). Even though this is a maximum over 61 squares, the effects are much smaller than those for the 3rd move target. For example, in layer 10, the mean of these maximum log odds reductions is only 0.55 = 0.01, compared to the 1.88 for the 3rd move target. This shows that information stored on the 3rd move target square is unusually important for Leela's 's output, more than information most other squares.
We are unsure why the squares of the 2nd move aren't 't similarly important. This may simply be because the opponent's move is typically "obvious" in our dataset or because suppressing the opponent's 's best response doesn't reduce the quality of the 1st move. We confirm in Appendix H that this is not just for certain why Leela seems to mainly store information on target squares rather than source squares (both for the 3rd move and for the immediate 1st move).
artifact of the overlap between 1st and 2nd move targets. Similarly, we don't 't know
3.2 Attention heads move information forward and backward in time
We have seen that the target square of the 3rd move in the principal variation contains unusually important information. If Leela uses look-ahead, this information must somehow inform its decision for the 1st move. An algorithm involving look-ahead might consider the consequences of making the 3rd move and then propagate that information back to earlier timesteps. In this section, we will find evidence of these processes. We identify an attention head that moves information from the 3rd move target square "backward in time" to the 1st move target square, well as heads that seem to move information "forward in time" to consider the consequences of the 3rd move.
L12H12 moves information "backward in time" In the previous section, we found squares potentially involved in look-ahead just by measuring their importance using activation patching, so we will try the same simple approach for attention heads. When we patch the output of one head at time from the corrupted the clean forward pass, one head stands out: L12H12 (the 12th head in the 1 12th layer) has a much larger average effect than any other attention head (Fig. 4).
So what does this head do? Anecdotally, noticed that the entry of the attention pattern Q" K with the 1st move target as the query and the 3rd move target as the key is often large: in 29.8% of puzzles, it is higher than all 4095 other attention entries in L12H12. In other words, it seems that LI2H12 often moves information from the 3rd move target "backward in time" to the 1st move target, where it is needed for the immediate decision.
We test this hypothesis more directly by ablating this specific attention entry. Remarkably, zeroing out this single entry reduces the log odds of the correct move by more than 1.5 in more than 10% of puzzles (Fig. 5), corresponding to a reduction in probability from, e.g., 50% to 18%. This effect is much larger than simultaneously ablating all 4095 other 1.12112 attention weights.
These results are particularly striking given the scale of the intervention: we are zeroing just one floating point number out of about 1.5 million attention entries and many other types of activations.

---

Mean patching effects
Attention ablation effects
Zoomed in
0.4
Log odds reduction
3rd >st target
0.3
Other
Head
10-
15
0.1
20
0.0
0
10
25
50
75
100
94
96
98
100
Layer
Percentile
Percentile
Figure 4: Mean log odds re- duction from activation patch- ing attention head outputs one head a time. The head that stands out the most is L12H12.
Figure 5: Zero-ablations in the attention pattern in L12H12. Green line: ablation the attention entry with key on the 3rd and query the I st target square. Gray line: ablation of all 4095 other entries at once. The lines show the effect a given percentile puzzles sorted by effect size. Error bars are 95% Cls; see Appendix G.
Piece movement head ablation
Bishop
Log odds reduction
Main ablation
Other piece types
6
Random square
Knight
4
0
Rook
Percentile
Figure 6: Attention patterns of random piece movement heads. Each row is one head, each column a random puzzle. For each puzzle, we plot the attention pattern for fixed random query square and varying key squares. A fixed key and varying query give similar results.
Figure 7: Effect of zero-ablating attention en- tries moving information out of the 3rd move target square, in piece movement heads corre- sponding to the 3rd move piece type. Error bars are 95% CIs, see Appendix for details.
That this has such outsized effect strongly suggests that this specific information pathway is crucial for Leela's ' decision making process in a substantial fraction of states. This is consistent with our residual stream patching results, where the 3rd move target important in earlier layers (around L10 and L11). 1),, but after L.12, the 1st move target dominates. L.12112 seems to one mechanism enabling this shift.
"Piece movement heads" help analyze consequences of future moves In addition to L12H12, which moves information "backward time," we find evidence of attention heads moving information "forward" to analyze the consequences of future moves.
Throughout the network, Leela has several attention heads whose attention patterns closely resemble legal moves for specific piece types (Fig. 6). We call these piece movement heads. For example, there are "knight heads" whose attention focused on squares reachable by knight moves, and similar heads exist for bishops and rooks. By manually inspecting attention patterns in random example states, we identify "knight heads," "bishop heads." " and 29 "rook heads" spread relatively evenly throughout the 15 layers (out of 360 heads total in the network). We also found weaker signs of heads for other piece types but exclude those here. King heads are difficult to study because our dataset contains only few examples where the first move a king move. Pawns move differently for normal

---

moves vs captures, and it seems those functions are captured by different heads, so we ignore them for simplicity. Finally, there seem to be at most handful of queen heads, and we suspect that queens are handled mainly by combining rook and bishop heads.
We hypothesize that these piece movement heads used, least in part, analyze the consequences of future moves, If a knight will be moved on the 3rd move (and thus end up on the 3rd move target square), the network might use knight heads to determine what effects a knight on the 3rd move target square would have.
To test this hypothesis, we again ablate information flow out of the 3rd move target square, but this time in piece movement heads instead of L12H12. For each puzzle, we zero out all attention entries with their key on the 3rd move target square. We do this only in piece movement heads corresponding to the piece type of the 3rd move because we want to see whether the network is attending to the consequences of this move specifically. We also do not ablate the attention entry in piece movement heads between the source and target square of the 3rd move-we still want to let the network consider the 3rd move itself, just not its consequences. In the running example puzzle from Fig. 2, the 3rd move is a rook to h4, so we would ablate information flow out of h4 in all "rook heads," except to the source square d4. This should prevent Leela from "noticing" the effects that the rook would have on h4 (namely delivering checkmate).
For this experiment, we focus on the subset of our puzzles where the principal variation (as given by the Lichess dataset) is longer than 3 moves. This ensures that there are potential consequences to analyze after the 3rd move rather than having reached an easy-to-evaluate state by then.
The ablation typically has a large effect on the network's output (Fig. 7). In 60% of puzzles, the log odds of the top move are reduced by at least 1.5. In contrast, ablating both other piece movement heads (for different piece types) or ablating on a random square has little impact on performance. This suggests our intervention blocks a very important network mechanism rather than simply reducing performance for generic reasons.
While the heads we' ve identified seem involved in analyzing future moves, they are certainly not full explanation of how Leela implements look-ahead. Piece movement heads likely also serve many simpler functions unrelated to look-ahead. Conversely, there might be heads other than L12H12 involved in moving information backward in time, and there are likely many heads involved in look-ahead that don't 't specialize in one piece type.
3.3 Simple probes can predict future moves
We have seen evidence that the 3rd move target square contains information that is unusually im- portant for Leela's ' output and is moved by attention heads in ways consistent with look-ahead. But can we go a step further and show that this square ex- plicitly encodes information about what the 3rd move is?
Third move prediction
1.0
0.8
Accuracy
0.6
Probe on trained model
Probe on random model
0.4
We find that this is indeed possible: probes inspired by our analysis of L12H12 can predict the 3rd move with 92% accuracy. Recall that our dataset only includes puzzles that Leela solves correctly and with a unique principal variation. This makes it possible in principle to achieve such high accuracy. but it is nonetheless remarkable that simple probe can predict a move two steps into the future.
0.2
0.0
0
5
10
Layer
Figure 8: Results of bilinear probe for pre- dicting the 3rd move target square. Errors com- bine standard errors of the mean for five probe training runs with standard errors for accuracy estimates; see Appendix G.
The architecture of our probe is directly motivated by our observations on LI2H12. Recall that the attention pattern of LI2H12 often has a large entry where the 1st move target attends to the 3rd move target. This entry in the attention pattern Q" K can be written as h,2 W Wkhh where hf denotes the residual stream activations after layer I. on square i; ly the target square of the j-th move; and Wo. WK are the query and key weight matrices of L12H12. We could use this attention score directly as logit for predicting t3. but since L12H12
8

---

is not explicitly optimized for this task, we instead train a bilinear probe (Hewitt and Liang, 2019) with an analogous structure from scratch.
Concretely, our probe predicts the 3rd move in two steps:
1. Predicting the target square: Given the target square of the 1st move t1, which we can extract from Leela's 's policy output, we predict the 3rd move target square 13 using
Pr(l3 y/41) softmax
(U"))" + +c).
y
where U, V are learned matrices with shapes matching We and K, and is learned bias. This is exactly analogous to how the attention weight between 83 and t is computed (except for the bias).
2. Predicting the source square: We predict the 3rd move source square 83 conditioned on the predicted target square tz using an analogous bilinear form with separate weights U", V c':
Pr(s3 v|(a) softmax (1h))T U V' - -c)
We train the probes separately, so the second probe uses ground truth values for la during training. At test time, we first predict La and then use that to predict 83. See Appendix F for details on training hyperparameters.
Figure 8 shows the accuracy of our probe for predicting the 3rd move target square after each layer of the network. Accuracy mostly increases through the layers, peaking (92 = 1)% after layer 12. As a simple baseline, probes trained on a randomly initialized copy of Leela achieve only (15 + 2)% accuracy. This shows that the performance can't just be due to the probe "doing all the work."
4 Related Work
Chess-playing neural networks Our work relies on the Leela networks (Leela Chess Zero team; Monroe and Chalmers, 2024). Leela is based on earlier work on Alpha Zero (Silver al., 2018) but is much stronger. Ruoss et al. (2024) recently trained neural network to play chess without any external search (such as MCTS). Their network is similar in architecture and strength (lepned, 2024) to the version of Leela we study, our findings suggest the possibility that their network may have learned to perform internal look-ahead or search. Several works have trained networks to play chess using autoregressive approach motivated by language modelling (Noever et al., 2020; Toshniwal et al., 2021; Feng et al., 2023; Stöckl, 2021; Karvonen, 2024). Unlike Leela or Alpha Zero, these networks don't It get the current board state as input, only sequence of moves, which makes their task more difficult.
Learned look-ahead and search Pal et al. (2023) showed that future tokens decodable from hidden representations relates to our probing results in Section 3.3. However, they don't focus whether representations of future tokens causally influence the prediction of the current token, which is the core focus of our study on look-ahead. Brinkmann et al. (2024) find an interpretable reasoning algorithm in small transformer trained to find paths in trees. This suggests similar algorithms could be present in Leela as well, but Leela and chess-playing are much more complex than this model and synthetic task, and so giving as detailed an interpretation as Brinkmann et al. do would be much more challenging. In concurrent work to ours, Taufeeque et al. (2024) and Bush al. (2024) demonstrate look-ahead in Sokoban- bokoban-playiig RNNs using linear probes. Finally, there is a long line of work showing that neural networks can learn implement learning algorithmss in context, c.g., for regression (Hochreiter et al., 2001; Akyürek et al., 2023; von Oswald et al., 2023b,a) and reinforcement learning (Duan et al., 2016; Wang al., 2016; Lee et al., 2023). This algorithmically quite distinet from look-ahead as we study it, and many of these works focus on behavioral evaluations rather than mechanistic analysis.
some extent language model at earlier token positions, a finding that
(Mechanistic) Interpretability Methodologically, our work relies heavily on mechanistic inter- pretability tools, particularly activation patching (Vig et al., 2020; Geiger et al., 2021; Meng et al., 2022). In terms of results, there have also been few relevant works; most closely connected is

---

Brinkmann et al. (2024) as already discussed. Examples of interpretability applied to game- saae-ig models include McGrath al. (2022) and Schut et al. (2023), who study the internals of AlphaZero. But their specifie experiments are very different from ours and they do not specifically analyze the potential for search or look-ahead. Recent work has also found that game-playing models trained on move sequences can learn to keep track of the current board state in Othello (Li et al., 2023; Nanda et al., 2023) and chess (Karvonen, 2024). This orthogonal to our work: Leela already gets the current state as its input, rather than sequence of moves, and instead of board state tracking, we find evidence of look-ahead to future moves.
5 Conclusion
We have shown correlational and causal evidence of learned look-ahead in Leela. While do not have a good understanding of the exact algorithms Leela has learned, our three lines of evidence strongly suggest that in many tactically complex states, some form of look-ahead plays an important role in determining Leela's policy. Some of the techniques we use are very general and might be useful for mechanistically studying complex behaviors in other networks.
Limitations In our mind, the main limitations of our work are as follows: (1) We do not present a precise description of how look-ahead might be implemented in Leela. Understanding this would be interesting from an interpretability perspective and also provide additional evidence. (2) We focus on look-ahead along single line of play; we do not test whether Leela compares multiple different lines of play (what one might call search). (3) We focus on board states that are unusually complex and thus more likely to require look-ahead. To understand the role of look-ahead across the entire input distribution, we would need to gain a better understanding of how -hedd might be combined with simple heuristics in Leela. (4) Chess a domain might favor look-ahead an unusually strong extent. It would be interesting to study whether language models use similarly principled mechanisms when appropriate. We think all of these questions could be fruitful directions for future work.
Impact We expect our results to inform future research and discussion rather than having direct societal impacts. There has been significant debate about the degree to which frontier neural models, such as large language models (LLMs), internally implement principled algorithms. Our results on chess-playing model certainly don't allow immediate conclusions about LLMs, but they are evidence of complex algorithmic mechanisms in neural networks "in the wild," i.e., not trained specifically to demonstrate such mechanisms. Learned optimization (or mesa- mnesaoppimizzttion could also pose novel risks (Hubinger al., 2019). Leela or similar networks might be promising candidates for test beds to study such potential risks in toy settings.
Author contributions
Erik led the project, including developing ideas, executing experiments, and writing. Shreyas built the infrastructure necessary run interpretability experiments Leela and contributed ideas and technical support throughout the project, in particular using a weaker model to filter puzzles and find corruptions. Vasil ran several exploratory experiments, first noticed L12H12, and conjectured that it was moving information from 3rd to 1st move target. Cam and Scott gave regular feedback and input throughout the project and helped develop the story for the paper; Scott also helped implement the puzzle filtering and I. L12H12 ablations. Stuart advised the project. Stuart, Scott, Shreyas, and Erik were all involved in the original conception of the project
Acknowledgments and Disclosure of Funding
We thank Michael Cohen, Lawrence Chan, Erik Jones, Alex Mallen, Neel Nanda. and everyone else we talked to for feedback various stages of this project. We also thank Discord user masterknió for providing information to help us finetune our version of Leela (Appendix A). This work was supported by funding from Open Philanthropy, the Future of Life Institute, and the AI2050 program at Schmidt Futures (Grant G-22-63471).
10

---

References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. ICLR. 2023.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng. Jiong Gong, Michael Gschwind. Brian Hirsh, Sherlock Huang. Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos. Mario Lezcano, Yanbo Liang. Jason Liang. Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufi,, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Fikan Wang. Xiaodong Wang. William Wen, Shunting Zhang. Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. ACM International Conference on Archilectural Support for Programming Languages and Operating Systems, 2024. URL https: //yytrcc org/ assets/ pyyptttcppppcctcpcccoooo 2. pdf.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of transformer trained on symbolic multi-step reasoning task, 2024.
Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga- Alonso, and David Krueger. Interpreting emergent planning in model-free free reinforcement learning. (forthcoming). 2024.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump conclusions: Short-cutting transformers with linear transformations, 2023.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Picter Abbeel. RL*: fast reinforcement learning via slow reinforcement learning, 2016.
ENOT developers. Igor Kalgin, Arseny Yanchenko, Pyoter Ivanov, and Alexander Goncharenko. onnx2torch, 2021. URL htt.http://// thub. cwt.b...cco///eoooee.eeeeeeee... www//ooTo Autoot/AADDDLL///a conx/torc/..ccccocoooooomo
Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Henry Mguni, Yali Du, and Jun Wang. ChessGPT: Bridging policy learning and language modeling. NeurlPS Datasets and Benchmarks Track, 2023.
Niklas Fickas. python-chess. URL. htttttt////giiii httppp///ggtttub...mo/mlllaa//// wwt..ooo/////.n//eeccme. http:/yythoonhhhhheeeee chess.
Jaden Fiotto- Kaufman. nnsight: The package for interpreting and manipulating the internals deep learned models. URL https::t/// / githu....... hta//JJdenFioto KaaffanKaaa///aaaa nns ansight.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. NeurlPS, 2021.
Causal abstractions of neural networks.
Trevor Graffa. lezero_tools. URL https:://gtthhhie com/ so --c-muchheee mach-metaalczerootoom
Guy Haworth and Nelson Hernandez. The 20th Top Chess Engine Championship. TCEC20. J. Int. Comput. Games Assoc., 2021. URI htttp::/tcce.. chess. www./orticees/TT..C.... 20. .df..
Stefan Heimersheim and Neel Nanda. How to and interpret activation patching, 2024.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. EMNLP, 2019.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. ICANN. 2001.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2019.
John D. Hunter. Matplotii:: A 2D graphics environment. Computing in Science & Engineering, 2007.
IceBerg Contributors, IceBerg compositional graphics & diagramming, 2023. URL https///: com/ revalo/iceberg.
Adam Karvonen. Emergent world models and latent variable estimation in chess-playine language models, 2024.

---

Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum. and Emma Brunskill. Supervised pretraining learn in-context reinforcement learning. NeurIPS, 2023.
Leela Chess Zero team. Leela Chess Zero. URL https://lczero.org org./
lepned.
How well do Lc0 networks
compare to the greatest transformer net- URL
work
from DeepMind?, 2024.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on a synthetic task. ICLR, 2023.
Lichess team. Lichess tactics puzzles dataset. URL httpt::////et //atabas/ lichess..h.....ee sichees.org//ppeeeee #puzzees
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearee, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik, Acquisition of chess knowledge in AlphaZero. Proceedings of the National Academy of Sciences, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. NeurlPS, 2022.
Daniel Monroe and Philip A. Chalmers. Mastering chess with a transformer model, 2024.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self- supervised sequence models, 2023.
David Noever, Matt Ciolino, and Josh Kalin. The chess transformer: Mastering play using generative language models, 2020.
nostalgebraist. Interpreting GPT: the logit lens, 2020. URL httts::/www..esswrrr www.eeeswwon..oooooeeeeeeeeeeeee com/posts/ ackKBwaapaalettrrriiittrrrrrgge gateggg--ttttteeee -ogitte-ees...
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. Future Lens; Anticipating subsequent tokens from single hidden state. Conference on Computational Natural Language Learning (CoNLL), 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners, 2019. URL https://cdn www.dd.....o.iioooooommmoooaaa.. openai language_mosn moceeese__ee______eeeee__e are www.uuuuuuuvieeeee sed, mul ti multitask, learners. pdf.
Maneteer-anngage models/
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with Einstein-like notation. ICLR, 2022.
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, and Tim Genewein. Grandmaster-level chess without search, 2024.
Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, UIrich Paquet, and Been Kim. Bridging the human-Al knowledge gap: Concept discovery and transfer in AlphaZero, 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 2018.
Andreas Stöckl. Watching a language model learning chess. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), 2021.
Mohammad Taufeeque, Philip Quirke, Maximilian Li, Chris Cundy, Aaron David Tucker. Adam Gleave, and Adrià Garriga-Alonso. Planning in recurrent neural network that plays sokoban. Mechanistic Interpretability workshop at ICML, 2024.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as testbed for language model tracking. AAAI Conference Artificial Intelligence, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. NewrIPS, 2016.
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. NewrIPS, 2020.
12

---

Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joāo Sacramento, Alexander Mordvintsev. Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. ICML, 2023a.
Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers, 2023b.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn, 2016.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metries and methods. ICLR, 2024.
13

---

A Details of the Leela architecture
The version of Leela supported model when this project began. (By now, the newer BT3 and BT4 models are even stronger.) Other versions of Leela are CNNs similar AlphaZero, but this model is transformer-based. and we will focus on its architecture. For some training details on a newer iteration of Leela, see Monroe and Chalmers (2024), though note that our version has slight architectural differences to that newer one.
is 182-768x15xx2th-sw--522222
which was the strongest officially
The model is available from https: : //wrrerc....o org/play/eetneeeoo weights themselves don't 't have specified license to the best of our knowledge; the code base is under the GPI-3 license, and the training data is under the Open Database License.
seoorkks/betnetss// The network
Note that this version of Leela wasn't t directly trained using MCTS; instead, it was trained using supervised learning on MCTS roll-outs produced by an earlier model. This is similar objective, however, given that during MCTS, the value and policy nets are also trained to match the output of the entire search process.
High-level description Leela takes a board state as input and produces distribution over moves and value (win/draw/loss probability) for that state. It has a policy and a value head, which share a transformer as the main body of the network. Each square of the chess board is represented as sequence position in this transformer.
Input encoding To feed a board state into Leela, it is tumndd into 12 bitmaps, one for each piece type of either color. For example, there is a bitmap for white pawns, which is an 8 8 boolean array with a on each square with white pawn and a 0 otherwise. In addition to these 12 bitmaps, there are few channels for encoding castling rights and similar information (these channels have a constant value across all 64 squares). The original version of Leela takes in past board state as do without those, as we 'Il discuss. The positional encoding of the model Each square has 64-dimensional embedding with binary values that
well, but we finetuned we use is domain-specine encode which other squares could be reached from the given square one move by some type of piece. The input and positional encodings are concatenated and then fed into a linear map into the 768-dimensional residual stream. Thus, the precise choice of input and positional encoding likely does not matter much; the model can learn how to best embed this information. Note that piece colors are encoded as "player color" and "opponent color", rather than white and black. Additionally, the board is rotated so that the current player's side is always at the same indices. Thus, Leela doesn't 't need to learn the symmetry between white and black; this is already built into the input encoding.
Main network The main part of Leela is a 15-layer transformer with a residual dimension of 768, 24 attention heads per layer, and head dimension of 32. Notably, the MLPs have hidden dimension of only 1024, rather than the common 4 dresidual. Leela is thus unusually heavy attention heads. The LayerNorms in Leela are applied directly to the residual stream rather than to layer inputs. This is what the original transformer paper did ( (Vaswani et al., 2017) but unlike most modern transformers. These non-linearities on the residual stream make certain interpretability techniques more difficult to apply but don't 't present obstacle for any of our experiments.
The biggest way which Leela differs from a typical transformer a method unique to Leela called smolgen. Like other transformers, Leela produces attention scores using Q and K matrices. However, before applying the softmax, smolgen adds the output of an MLP these attention scores. This MLP takes in the entire residual state, which is meant to make it easier for the network to use global information when computing attention scores. From an interpretability perspective, smolgen would likely make it much more difficult to understand how attention patterns are computed. However, we don't It study that question and can mostly ignore smolgen (i.c. we simply use the combined attention pattern without worrying about where it came from). In the case of L12H12, we used the observed attention pattern to motivate our bilinear probe architecture. It turns out that the part of the attention pattern produced by the (bilinear) query/key mechanism is indeed enough to read off the 3rd move target square.
Smolgen
Policy head The policy head of Leela consists of two 2-layer MLPs, which we'll call the source ML. and the target MLP. They share the first linear layer for parameter efficiency but have separate second layers. Both layers have output dimension 768 (matching the residual stream). These MLPs
14

---

are applied separately to each square, yielding outputs with shape 64 x 768 for each. These "source" and "target" oulputs are then matrix multiplied along the 768- f68-dimensional axis to yield 64 64 dimensional output. Each entry in this output represents the logit for the move from the corresponding source square to the target square. Finally, logits for illegal moves are masked out by setting them to negative infinity, then a softmax over the logits yields the distribution over moves. There are a few additional pieces of machinery to deal with pawn promotion, but these aren't It particularly important for our purposes.
Value head We don't It use the value head in the main paper but briefty discuss it here for complete- ness and will present results in Appendix C. It begins by projecting the 768-dimensional residual stream down to 32 dimensions, using a learned linear map that's shared across squares and applied separately to each square. The 64 32-dimensional embeddings are then rehaped get a single 2048-dimensional vector. This vector is passed through small MLP, which produces three logits, for the probability of the current player winning, drawing, and losing.
Finetuning to avoid reliance on past board states Leela originally takes in the past 8 board states, rather than only the current one. This mostly shouldn't be necessary to play chess well, though might help bit given the limited computational capacity. For example, knowing the opponent's 's last move may make it easier to tell what threats (if any) they currently have. For our purposes, passing in history of past board states is very inconvenient since we want automatically generate corrupted states for activation patching. Generating corrupted histories instead would be much more challenging. We experimented with different options, such simply passing in zeros for the history, repeating the current board state, or synthesizing a valid (but not semantically meaningful) history. We found that in most cases, Leela's output didn't t depend significantly on what we passed in history, indicating that Leela (perhaps unsurprisingly) wasn't making too much use of past board states. However, in some cases, the output changed lot when we modified the history, for unclear reasons.
To avoid such confounders when activation patching, we finetuned Leela to behave the same whether or not any history was passed in. Then, for all our experiments, we used this finetuned model and didn't pass in any history. The finetuned model matches the original one in strength despite not using history (both in terms of solving puzzles from our dataset, as well as when playing against the original model directly). Anecdotally, it also seems that finetuning didn't 't change the model's '' mechanism too much (e.g., some attention heads that we had interpreted before finetuning still seemed to perform the same function afterward).
B Leela's playing strength
Leela is usually evaluated with MCTS, in which case it is the strongest MCTS-based chess engine, and competes with Stockfish- a classical chess engine making heavier use of external search- for the title of strongest chess engine in general (Haworth and Hernandez, 2021).
There are fewer evaluations of the strength of only the neural network itself. A Lichess bot using only Leela's policy network has achieved blitz and rapid ratings over 2600, see https: //web. archive. 10w90000/0tttppe..
//lichhese org/ org////azgg////// e/Laayyot/////ee/ee/eeeeae al1. This is clearly below the strength of the best human players but far better than most amateurs. The version we use is slightly larger than the one underlying the Lichess bot and likely somewhat stronger.
Ruoss et al. (2024) recently trained a neural network that can also play chess extremely well without external search. Lepned (2024) compared this network to Leela on the task of solving tactics puzzles and found that the strongest Leela networks outperformed the network by Ruoss et al. (2024), but the version we use (T2) seems roughly evenly matched.
C Value head results
As discussed Appendix A, Leela has a policy head and a value head on top of a shared network body. For simplicity, we only used the policy head in the main paper: the metric we measured there were always the log odds that the policy head assigns to the correct move. But of course, most of the computation going into the network's 's policy output happens in the shared body (since the policy
15

---

Mean patching effects
1.00
Log. oill uninndian nass
5
0.75
Patchkag
Head
10
0.50
15
0.25
0.00
0
10
Layer
Logee
Figure 9: Residual stream patching results (analogous to Fig. 3 but using the win log odds instead).
Figure 10: Attention head patching results (analogous to Fig. 4 but using the win log odds instead).
Attention ablation effects
Zoomed in
Log odds reduction
10.0
3rd->1st target
7.5
Other
5.0
2.5
0.0
50
100
94
96
100
Percentile
Percentile
Figure 11: L12H12 ablation results (analogous to Fig. 5 but using the win log odds instead).
head itself is very small), we'd expect those computations also affect the value head. We should thus expect overall similar results when using the value head to measure effects of interventions. In this appendix, we show that this indeed the case.
Our metric in this appendix are the log odds of the win probability given by the value head, i.e., log
). where Pwin computed by applying a softmax the three output logits of the value head (for win/draw/loss).
Puin
1-Pain
Figures 12 show all the results from the main paper using this win probability metric instead of the probability of the correct move. In other words, the interpretation of these figures is the same as in the main paper, except that "log odds reduction" refers the win probability log odds.
The only new finding is that L14H3 seems very important to the value head (Fig. 10). But note that L12H12 is still just important it was for the policy head (log odds reduction of 0.49 when ablate it), it's only relatively less visible in the heatmap.
The probing results in Section 3.3 only use the shared network body rather than either of the heads, so there no separate value head version of them.
D Creating the puzzle dataset
As described in Section 2.2, we are looking for states where Leela finds the correct move but smaller model does not. The small model we use "Little Demon 2", CNN with only 390k parameters. Here, we describe the exact procedure used to create this dataset.
16

---

Piece movement head ablation
Log odds reduction
Main ablation
Other piece types
Random square
2
25
100
Percentile
Figure 12: Piece movement head ablation results (analogous to Fig. 7 but using the win log odds instead).
We begin with dataset of Lichess tactics puzzles(Lichess team), available at https: //ta///aatbeeeeeeeeee lichess. org/#puzzles under a Creative Commons CCO license. These puzzles have been auto- matically generated from human games using variety of heuristics and computer analysis to select "interesting" states. The player color (i.e. the color whose turn it is in the starting state) always has an advantage, but only move maintains that advantage.
Many of these puzzles are still easy to solve using heuristics. That is where the smaller version of Leela comes in. We discard any puzzle where this smaller version assigns more than 10% probability to any of the player color's moves from the principal variation. other words, we require every move the player needs make to be "difficult to find". Note that we do not apply this restriction to responses by the opponent. Typically, the principal variation has two moves by the player, sometimes three or rarely more.
An alternative would have been use puzzle ratings that Lichess provides to judge difficulty- these are based many human players attempting to solve puzzles. An advantage of our approach is that it is more directly related to what's difficult for neural networks and avoids potential human idiosyncrasies. It is also much more general method that's still applicable if human difficulty ratings are not available.
In addition to making puzzles "difficult", we also want to ensure that the large version of Leela gets them right. Otherwise, there is no interesting behavior for us to study with interpretability. We thus discard any puzzles where Leela assigns less than 50% probability to any of the player's moves. In particular, Leela's 's top choice always coincides with the best move in our dataset.
Finally, for some of our experiments it is useful if the principal variation is forcing, in the sense that there is only ever clearly best move at each step. In particular, otherwise it would be fundamentally impossible to predict the 3rd move well since it would depend on the opponent's response. We thus also discard puzzles where the small model assigns less than 50% probability to the 2nd move in the principal variation, i.e. the opponent's response. For the player's moves, the Lichess dataset already guarantees that there only one winning move at each step.
This filtering procedure and the various thresholds were chosen based on manual inspection a few puzzles; we tried to ensure that the puzzles seemed intuitively "interesting" to us. We did not tune this procedure based on any interpretability results.
E
Automatically generating corrupted states
Generating candidate corruptions Given a board state, we generate candidates for corrupted states by applying each of the following mutations separately (i.e. each mutation yields one candidate corruption; each candidate has only one mutation applied):
1. Add a single pawn of either color empty square.
17

---

2. Remove any pawn from the board.
3. Move any non-pawn piece to any other empty square.
We discard candidates that lead to illegal states (c.g. if the opponent of the current player in check). This produces a few hundred corruption candidates in typical board state. The reason consider only these corruptions and not, for example, moving pawns or adding non-pawn pieces is simply to keep the search space manageable and because almost always find a "good" corruption using only these candidates.
Filtering candidates We apply several corruption filtering steps using the full version of Leela as well as the weaker model we used for filtering puzzles. As with puzzle filtering, we manually designed these filters to lead to intuitively "interesting"" corruptions, but did not tune them in any way on interpretability results. The precise numerical cutoffs used in the filters are somewhat arbitrary.
1. We keep only corruptions that reduce the probability Leela assigns to the previously best to less than 10%-if corruption doesn't 't make the previously best move bad, there's no reason to expect activation patching to have any effect.
2. We discard corrupeed states where the weak model's ' log odds of the previously best move decrease by more than 0.2. Anecdotally, these corruptions often make the best move worse for "obvious" reasons, such as placing an opponent pawn that directly attacks the target square of the move.
3. We discard corruptions that make the board state significantly better according to Leela's value output (increasing the difference between win and loss probability by more than 0.1). These corruptions often make the previously best move worse simply by making some other move extremely good, c.g., putting the opponent's 's queen on square where it can be captured.
Picking the "lowest impact" corruption Out of all the remaining corruptions, we pick the one such that the weak model's 's move distribution changes as little as possible since we are looking for corruptions that change the best move for "subtle" reasons that the weaker model doesn't 't "notice". Concretely, we minimize the Jensen-Shannon divergence between the weak move distribution on the clean and corrupted state.
F
Probe training details
We train one probe for each layer, the residual stream activations after the MLP of that layer. More precisely, we use the activations after the I LayerNorm (recall from Appendix A that Leela applies a LayerNorm to the entire residual stream, rather than module inputs). We use 70% of our puzzle dataset for training and the remaining 30% to evaluate probes.
We found that the choice of hyperparameters has essentially effect probe accuracy; we used Adam with learning rate of le-2, no weight decay or other regularization, a batch size of 64, and trained for epochs.
The probe can be thought of as a low-rank bilinear form, where we parameterize the low-rank matrix as V for U, V matrices of shape k x d, with d = 768 the residual stream dimension. We used k= 32 for the rank, simply to match the attention head dimensions of Leela (each head's attention pattern can analogously be interpreted as the result of low-rank bilinear form). Similarly to other hyperparameters, we found that other values of & work about equally well.
G Details on confidence intervals
All errors we report are 2a or 95% confidence intervals, depending on context. We have three types of еπтог:
For the error of an average, such as the average effects of residual stream activation patching (Fig. 3), we report two times the standard error of the mean. Given that these are means over thousands of data points from i.i.d. samples, this 2a error likely corresponds well to ~95% confidence interval (though we don't report it as such).
18

---

2. For percentile plots that show distribution (Figs. 5 and 7), we report an approximate 95% confidence interval based on the fact that the number of data points below a given percentile is binomially distributed. We will describe the exact procedure for this shortly.
3. For our probes, we consider the error from randomness during probe initialization and training, in addition to the error when estimating the test accuracy from finitely many samples. We compute the standard error of the mean for both and propagate them (i.c. add the square errors) to get an overall error; we then again report twice that error to approximate a 95% confidence interval. See details below.
Note that all of these errors are statistical, and due to our sizable puzzle dataset (22.5k puzzles), they are mostly very small. Naturally, there might be unknown systematic errors that dwarf these statistical ones.
Error bars for probe accuracies We consider two sources of error: (1) randomness during probe initialization and training, and (2) estimating the accuracy of the final trained probe from our finite evaluation dataset. For (1), have five training runs and compute a standard error of the mean over these. For (2), we compute the standard error as well (noting that the empirical accuracy is an estimate of the mean of Bernoulli random variable using the samples from the evaluation dataset). Then we
compute a total error using error propagation as rocal = V + Bouurayy For error bars, we report a 20t interval. Train dominates especially for the randomly initialized model; we could reduce the error bars further by training more than five probes, but they are already negligible compared to the effect size.
Confidence intervals for percentiles Several of our results visualize distribution by plotting a percentile function (the inverse of cumulative distribution function), see Figs. 5 and 7. We approximate a 95% confidence interval for each percentile. Let n be the number of samples we use; in all our cases, these are i.i.d. from an unknown distribution (whose percentile function we want to plot). The number of samples kp) below a given percentile p [0. 1] follows a binomial distribution B(k(p); ne, p.. We find the 2.5% percentile and 97.5% percentile of this binomial distribution k(p) lies between these values with 95% probability. We then sort all our samples by value and find the samples at the 2.5% and 97.5% percentile by value. Those values form the lower and upper bound for our confidence interval of the p-th percentile. This is an approximation since the samples whose values we use will never be exactly at the 2.5%-th and 97.5%-th percentile, but we have at least several thousand samples all cases, so the approximation should be very close to 95% interval.
H Results on subsplits of the puzzle distribution
As discussed in the main text, we noticed that the results of all our experiments are noticeably different on puzzles where the 1st and 2nd move target square are different. In 83% of our overall dataset, these squares coincide because many of these puzzles contain sacrifices by the starting player,
In this appendix, we show results on both subsplits of the data, the one with the same 1st and 2nd move target, and the one with different 1st and 2nd move targets. We find that the effects we observe generally much stronger in cases where 1st and 2nd target square are the same.
In most cases, the effects are qualitatively similar on both subsplits. The one exception is LI2H12, where the baseline ablation has a stronger effect than the one between 3rd move target and 1st move target (though keep in mind that the bascline is ablating 4095 weights, vs only one weight for the main ablation). In fact, the bascline has bigger effects in this setting with different 1st and 2nd target squares than on the full dataset, unlike all other ablation effects we see. This might suggest that L12H12 is performing different functions in these cases (though that function seems less important, given the much lower effect size when activation patching LI2H12 in the "different targets" " setting).
One natural question is whether any of the results we found in the main text involving the 1st move target square instead apply to the 2nd move target square when they are different. This does not seem to be the case to meaningful extent. In the residual stream patching results, the 2nd move target square is automatically included the "other squares" baseline on the "different targets" split, but unsurprisingly, the 1st move target effects are still much larger than this baseline. (Logits are read off
19

---

bd son lager
Adiec Sagal
(a) Different targets
(b) Same targets
Figure 13: Residual stream patching results, analogous to Fig. 3.
Mean patching effects
Mean patching effects
0.15
0.4
5
Head
10
0.10
Head
10
0.2
15-
0.05
15-
20
20
0.0
10
10
Layer
Layer
(a) Different targets
(b) Same targets
Figure 14: Attention head patching results, analogous to Fig. 4.
1st move target squares after all.) We also tested ablating information flow in L12H12 from the 3rd move target to the 2nd move target, but the effect on the "different targets" split is even smaller than that of ablating information flow the 1st move target. thus seems likely that the first, rather than the 2nd target square, is critical for the results presented in the main text, even though they often overlap there.
I Software libraries used
We use onnx2torch (ENOT developers et al., 2021) to convert the official Leela ONNX models to PyTorch (Ansel et al., 2024) for easier instrumentation. This produces auto-generated code for the PyTorch forward pass, which we manually adjust in few ways to support our interpretability experi- ments. We then use unsight cools package (Graffa) for some our instrumentation. We use python-chess (Fickas)
to implement interventions. We also build on the
Atteution ablation effects
Zoomed in
Atteation ablation effects
Zoomed in
odidk roduction
Log odde reduction
Barl-alat
8
Other
Other
4-
2-
2-
0
25
96
100
25
96 Pescentile
100
Percentile
Percentile
Percentile
Different targets
(b) Same targets
Figure 15: Ablations in L12H12, analogous to Fig. 5.
20

---

Piece movement head ablation
Piece movement head ablation
Log odds reduction
Log odds reduction
Main ablation
Main ablation
Other piece types
Other piece types
6
Random square
6
Random square
4
2
2
0
0
25
50
75
100
0
25
50
75
100
Percentile
Percentile
(a) Different targets
Same targets
Figure 16: Ablations in piece movement heads, analogous to Fig. 7,
Third move prediction
Third move prediction
1.0
1.0
Probe trained model
0.8
Probe on random model
0.8
Accuracy
Accuracy
0.6
0.6
Probe trained model
Probe on random model
0.4
0.4
0.2
0.2
0.0
0.0
5
10
0
5
10
Layer
Layer
(a) Different targets
(b) Same targets
Figure 17: Probing results, analogous Fig.
for chess logic, and einops (Rogozhnikov, 2022) for conveniently implementing some of our meth- ods. The figures are produced using iceberg (IceBerg Contributors, 2023) and matplotlib (Hunter, 2007). See https: / /gg thub. cwm/ http///annnompaaillllllleeeeee interp for the full list of pack- ages we use.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The results and claims summarize in the abstract and introduction match those in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss various limitations throughout the paper, and explicitly highlight key limitations in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe most of the experimental setups in the main text, and provide all additional details in appendices (Appendices D to F). We also release all models and data used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
which includes code for running all experiments and reproducing all figures, well as README with instructions for downloading models and datasets and running code.
Justification: See attps://gii ht//gtttubb... com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ public/gun gwwides/CoeeSuunnn cesssiinncccyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
weciiccgguiie..ee CooeeSumiissseeeee wwweiisssioooooooooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe most of the experimental setups in the main text, and provide all additional details in appendices (Appendices to F).
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All our plots and results include either 2a 95% emror bars. We briefly mention which sources of error they are based on in the main text and describe their caleulation in detail in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All our experiments can be run in about day on single GPU, mention in Section 2.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neuril: wwt.iripp..cpppuppieeee cc/public/ httpp/ullc/Ettisseeeeeee cstuidelieie
Answer: [Yes]
Justification: None of the potential concerns discussed in the Code of Ethics apply to research process or results.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As we discuss in Section 5, we expect the impact of our work to be on future research; can't foresee specific eventual impacts society.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models and datasets we release are only lightly modified from existing ones and restricted to the narrow domain of chess.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use the Lichess puzzle dataset and Leela models; we cite both and discuss their licenses in Appendix D and Appendix A respectively.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We describe the dataset and model in the paper as well as in httpp://gggthaae com httpH//aannomppaielllelleeeeeeee innttt
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper did not involve human subject research crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: See above.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "p37oomyQOb", "reviewer": "Reviewer_abVe", "summary": "The paper try to discover and analyze the evidence of learned look-ahead in Chess playing network. They take experiments on filtered Chess puzzle dataset and the policy network of Leela chess policy. The claims are mainly analyzed from three perspectives, including Activation patching, attention head analysis and probing experiments.", "questions": "1. There are several papers about transformer/Chess/AlphaZero that are also related work to this paper, including:\n(1) Zahavy, Tom, et al. \"Diversifying ai: Towards creative chess with alphazero.\" arXiv preprint arXiv:2308.09175 (2023).\n(2) Feng, Xidong, et al. \"Chessgpt: Bridging policy learning and language modeling.\" Advances in Neural Information Processing Systems 36 (2024).\n(3) Noever, David, Matt Ciolino, and Josh Kalin. \"The chess transformer: Mastering play using generative language models.\" arXiv preprint arXiv:2008.04057 (2020).\n(4) Stöckl, Andreas. \"Watching a language model learning chess.\" Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). 2021.\n\n2. Can the author explain more on the activation patching. Is it something like: you replace one neuron in one layer from the neuron of the same place in the forward process of corrupted board? Then I am wondering how you choose this corrupted board. Also in figure 3 why 1st/3rd but no 2nd action?\n\n3. Also I 'd suggest authors to include more illustrative examples/figures in the main paper or appendix to help readers (and those not familiar with chess) to understand the phenomenon.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The finding of the paper is interesting and inspiring. I like the idea of getting inside the network to understand how the policy network or transformer work to predict the optimal action.\n2. The paper is easy to follow and well-written.\n3. Experiments on different perspectives to help verify the conclusion.", "weaknesses": "1. The analysis seems to strongly related with the specific setting: Chess and transformer -- where each grid can be considered as a token. This can somehow limit the generality of the analysis method.\n\n2. The main issue about the weakness comes from the experiments. \n\nSince this is mainly a paper for discovering phenomenon and analysis, with no newly proposed algorithm or showing how this finding can help anything, the standard for NeurIPS paper requires it to have comprehensive experiments to validate the solidness of the finding. I do like this new finding, but the experiment is not enough to make it solid -- for instance, here are several experiments I think necessary for authors to present:\n\n2.1. Present how this phenomenon happens without explicit designing test/evaluation dataset. The authors do a lot of work in section 2.2 on filtering the testing samples to make it more likely to have this phenomenon -- and they use this to claim that the policy network has this phenomenon. This is quite weird since you also somehow optimize the evaluation dataset to help you find the claimed phenomenon. If this is not a general finding or conclusion, it can make the argument pretty weak and may only work on some human created settings -- an extreme case is that I can manually design some test cases that can definitely present this ability but it lacks generality. I agree with the argument that not all boards present this phenomenon -- but at least the authors need to present more things, like in which condition this phenomenon is likely to happen and which will not, and how/why this can happen. Is there any key issue that prevent or support the emergence of this phenomenon? Also the authors filter out all puzzles that Leela Chess may fail. This is really weird and can largely weaken the conclusion since if you have already tested it to solve the puzzle, it is quite normal that the representation of policy network can have something closely related with future optimal action (in the activation patching experiments or probing one).\n\n2.2 Is this conclusion generalizable on other transformer-based Chess policy? For example in \"Grandmaster-Level Chess Without Search\n\" they also have a policy trained by supervised learning with no RL or MCTS like the leela chess does. Will this setting affect the emergence of this behavior?\n\n2.3 Does only the policy network have this phenomenon or the same thing can happen at critic network? For example you can test it on leela-chess/stockfish evaluation function.\n\nThere are also a lot of ablation studies on modules or hyperparameters to make the conclusion solid. So again, solid and comprehensive experiments and ablation studies are necessary for such analysis/discovering paper, while this paper doesn't offer enough evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and suggestions!\n\n**Policy vs value network (2.3):** Great suggestion, thank you! We have now run all our experiments on Leela’s value head and will include these in the paper. We used the logs odds of the win probability as the effect size for patching experiments. (Probing experiments operate only on the shared network body so don’t differ between heads.) All our findings for the policy head still hold for the value head, see the PDF attached to our general response. The only new finding is that L14H3 seems very important to the value head (Fig. 2 in the attached PDF), but L12H12 is still just as important as it was for the policy head (log odds reduction of 0.49 when we ablate it).\n\n**On dataset filtering (2.1):** We appreciate that filtering our dataset can seem strange. However, we think this filtering process is essential to test our hypothesis and doesn’t negatively affect the validity of our results. We clear up potential misunderstandings below.\n\n> This is quite weird since you also somehow optimize the evaluation dataset to help you find the claimed phenomenon.\n\nInvestigating our claimed phenomenon fundamentally requires creating a suitable evaluation dataset *somehow*. As discussed (e.g. line 80), our claim is only that Leela often uses look-ahead *in certain types of states*, namely tactically complex ones. Such hypotheses specific to a certain input distribution are omnipresent in mechanistic interpretability (e.g. [1, 2, 3, 4]). To test this hypothesis, we naturally evaluate it on those states for which we claim Leela uses look-ahead. Our filtering process formalizes the vague notion of “tactically complex states.”\n\n> may only work on some human created settings -- an extreme case is that I can manually design some test cases that can definitely present this ability but it lacks generality\n\nIndeed, manually creating inputs would be very suspect, so we want to emphasize that we start with an existing dataset of inputs and then apply a simple automated filtering step. Importantly, we also do not use internal representations in any way during filtering.\n\n> Also the authors filter out all puzzles that Leela Chess may fail. This is really weird and can largely weaken the conclusion\n\nWe focus on puzzles that Leela solves simply so we can apply our interpretability methods. All our methods check whether Leela represents a specific future line. In correctly solved puzzles, we can look for representations of the *correct* continuation. In puzzles that Leela fails to solve, this wouldn’t tell us as much: Leela may be representing an incorrect continuation (and hence fail the puzzle), but there are many incorrect lines (vs only one correct one). So we can’t test for look-ahead like we can for correctly solved puzzles. We briefly explain this in line 89 but realize that this deserves more space, so we’ll incorporate this explanation in the updated paper.\n\nAs to whether this weakens the conclusion: we want to explain why Leela often solves puzzles correctly and don’t claim to explain why it sometimes fails. We’ve tried to be transparent about this (e.g. line 11 in the abstract) but will do another editing pass to make this clear.\n\n> if you have already tested it to solve the puzzle, it is quite normal that the representation of policy network can have something closely related with future optimal action\n\nCrucially, we don’t think that Leela solving a puzzle necessarily implies the claims we make about internally represented look-ahead. Drawing such mechanistic conclusions from behavioral evidence is dubious, which is why interpretability is needed in the first place. We agree that the model’s ability to solve difficult puzzles might *lead us to expect* that its representations would have something to do with future actions. Our paper's contribution lies in actually testing a more precise version of this hypothesis.\n\n**Testing other transformer-based chess-playing networks (2.2):** This would be an excellent future direction! Unfortunately, the model weights you mention were released only a bit over a month before the NeurIPS deadline. Transferring our experiments to new models would be a non-trivial effort, e.g., since we need to add instrumentation for our patching experiments.\n\nRegarding RL vs. supervised learning, note that the version of Leela we use was, in fact, trained using supervised learning (see line 422 in the appendix for details). The difference is just that it uses data from rollouts of an MCTS-trained Leela, rather than Stockfish evaluations.\n\n**Answers to questions:**\n1. Thank you for the references, we will cite these as additional examples of transformer-based chess models.\n2. That’s right, except we patch entire activation vectors on a square, rather than individual neurons. Our procedure for finding corrupted inputs is described in line 120 and appendix D. In brief, we randomly generate corruptions and then pick one that doesn’t change a weak model’s output, but does change Leela’s output a lot. In Fig. 3, we omit the 2nd move target since it is very often the same as the 1st move target (see line 93). Figure 9a demonstrates that the effect is mainly about the 1st and 3rd rather than 2nd move, which is why show the 1st instead of 2nd move.\n3. Are there specific aspects that you think are difficult to follow and would benefit from additional figures? That would help us a lot!\n\n[1] Wang et al, 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small\n\n[2] Hanna et al, 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model\n\n[3] Lieberum et al, 2023. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla\n\n[4] Nanda et al, 2023. Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4OSWLyEe44", "reviewer": "Reviewer_mWPe", "summary": "This paper closely examines, using various forms of activation patching and probes, the inner workings of the state-of-the-art policy network of Leela Chess Zero (for the game of Chess). The authors find several different pieces of evidence that suggest it is likely that the network has learned to carry out some form of look-ahead search, at least in complex states that require this to arrive at the correct solution.", "questions": "1) Probably a difficult question, but curious to hear your thoughts if you have any: is it definitively possible to draw a hard line between \"heuristics\" and look-ahead? Especially depth-limited look-ahead (like what you seem to be finding evidence for in this paper)?\n\nHere's what I'm thinking. If you only have a very basic heuristic (say, only material count), then you need at least 1 ply on search on top of that to find which moves let you capture something. But you could just add a smarter heuristic that counts the values of pieces your pieces can attack in one move, and then you no longer need one ply for this. With the most basic heuristic, you would need at least 2 plies of search to also consider the responses that the opponent can make. But, you could add a heuristic that counts which pieces defend their own friendly pieces, and that could partially account for what the second ply would typically be searching. Is there a limit to this? Could we conceivably create heuristics that detect the 3-ply situations explored in this paper, without search?\n\n2) As a philosophical follow-up to the above: can we ever really distinguish between look-ahead search and heuristics, if the look-ahead search is depth-limited (I suppose any \"learned look-ahead\" in non-recurrent neural networks would always be depth-limited)?", "limitations": "There is a good discussion of limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I found this to be a very interesting paper, and have little to remark on it. It's always good to see a paper that is actually trying to improve our understanding (and doing a good job at that), rather than just being \"here is our new technique and it has bigger numbers\". It is well written, using good examples and illustrations.", "weaknesses": "My only \"important\" criticism is that I object to the use of \"existence proof\" in both the abstract and the conclusion. I think this is too strong, and would suggest rephrasing to \"evidence\". I do find the evidence provided in the paper to be compelling, but still do not think it can be definitively described as proof rather than evidence.\n\n---\n\nMinor nitpicky comments:\n- line 86 has a random period in the middle of a sentence (right before the footnote)\n- I feel like it would be more natural for the paragraph of lines 222-224 to be moved a bit earlier. It's essentially describing a part of the experiment setup, but after some of the results have already been presented and discussed.\n- Line 225: \"are seem\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and insightful questions!\n\nWe agree that “existence proof” was poor wording on our part and will rephrase that to “evidence” in both places, thank you for the suggestion. Thanks also for the minor comments, which we’ll incorporate.\n\n> is it definitively possible to draw a hard line between \"heuristics\" and look-ahead?\n\nThis is a great question, and as you say, a full answer is probably a difficult philosophical problem. We give a few initial thoughts below.\n\nDrawing a very precise line around “heuristics” might not be possible (in the sense that the space between ad-hoc heuristics and general reasoning algorithms might be relatively continuous and it’s not exactly clear where the boundary is). Even so, we can probably say that certain algorithms do involve look-ahead and certain others don’t (while leaving the question open for some borderline cases).\n\nImplementing look-ahead by simply counting attacks/defenses in the current state, as you describe, is an interesting example of blurring this boundary. But capturing all relevant considerations this way likely only works well for 1- or perhaps 2-ply look-ahead. In the 3-ply case (which the paper focuses on), a piece might be moved twice (and already in the 2-ply case, heuristics would need to be very careful about counting defenders correctly to account for lines closing or opening up as pieces are moved).\n\nSo while in some sense, there is no limit to what heuristics can implement—in the extreme case, a look-up table could implement arbitrary policies—we expect algorithms that explicitly use look-ahead to be far more efficient.\n\n> can we ever really distinguish between look-ahead search and heuristics, if the look-ahead search is depth-limited\n\nThe thoughts outlined above suggest that a depth limit does not preclude a distinction between simple heuristics and look-ahead: while given enough capacity, both might be able to implement the same policies, they can differ in how much capacity that takes (and for sufficient depths, avoiding explicit look-ahead might be infeasible).\n\nOf course, our paper does not rest on these speculative conceptual claims; we aim to give evidence of explicit look-ahead rather than argue for its inevitability. (As one simple example, the method of counting attacks/defenses would not need to explicitly represent which future moves will actually be taken, so it would not predict our probing results.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBtn5RixqW", "reviewer": "Reviewer_SpQ5", "summary": "This paper conducts a mechanistic interpretability analysis of Leela and shows evidence that it learns to look ahead in the network.", "questions": "How often does look-ahead occur?\n\nWhat is the evidence that look-ahead is occurring as opposed to a shallow heuristic designed to pick up on common tactical patterns?\n\nWhy aren't the squares of the 2nd move similarly important?\n\nWhy aren't there pawn or king heads, similar to the other piece types?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper provides a convincing answer to an important question: are networks like Leela's analogues of System 1-style pure intuition, or do they encode some amount of System 2-style calculation? The technique of corrupting the state and using activation patching is clever and effective. It's clear from the experiments that yes, there is some look-ahead being done. The authors are also careful to not overstate their claims: an actual broad search isn't necessarily being done, but at least some 1- or 2-step look-ahead is happening.", "weaknesses": "The paper is essentially an existence proof of lookahead, which is a useful contribution, but it would be nice to know something about the the prevalence of lookahead. The dataset is chosen to maximize the probability that lookahead is happening. How often does it happen?\n\nI thought the claim that \"look-ahead or other sophisticated algorithms should pick up on the importance of this difference, but shallow heuristics should mostly ignore it.\" required more justification. Some of the tactical patterns picked up by the method are so common that it's conceivable that a \"shallow heuristic\" is designed for them, rather than look-ahead.\n\nThe lack of results/explanations regarding why the squares of the 2nd move aren't important weakens the argument somewhat. \n\nWhy aren't there pawn or king heads, similar to the other piece types?\n\nNitpick: the motivation in the first paragraph about chess being different from other domains because in other domains \"models can solve their tasks with a single simple algorithm\" also applies to chess.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and questions! We answer them below and clarify potential misunderstandings.\n\n> the claim that \"look-ahead or other sophisticated algorithms should pick up on the importance of this difference, but shallow heuristics should mostly ignore it.\" required more justification. Some of the tactical patterns picked up by the method are so common that it's conceivable that a \"shallow heuristic\" is designed for them, rather than look-ahead.\n\nWe’d like to clarify that this claim is not a load-bearing assumption for our results. The sentence you quote only motivates why we use corrupted inputs that are similar to clean inputs (in the sense that a weak model gives the same output): if we used corrupted inputs that differ too much from the clean ones, then too many network activations would differ and drown out the evidence of look-ahead.\n\nIf our argument for look-ahead was behavioral, i.e., the fact that Leela can solve these supposedly difficult puzzles, then it would indeed be an issue if some of them are also solvable by shallow heuristics. But our argument instead rests on observations about internal representations, and these observations suggest look-ahead irrespective of what exactly the corrupted inputs are. For example, if we could show that future move target squares are unusually important under random corruptions, this would be just as convincing in our mind.\n\nIn summary, we agree that shallow heuristics could likely pick up on some of the tactics, but we don’t think this weakens our arguments.\n\n> What is the evidence that look-ahead is occurring as opposed to a shallow heuristic designed to pick up on common tactical patterns?\n\nWe think all three lines of evidence we present favor look-ahead over shallow heuristics. For example, a shallow heuristic (which decides on the immediate next move based on matching to common patterns) would not need to explicitly represent which future moves will actually be played. But we find in section 3.3 that such representations of future moves exist, as a look-ahead hypothesis would predict. We similarly think that hypotheses that don’t involve look-ahead would not predict or explain any of our other results, but please let us know if you have specific concerns about this.\n\n> How often does look-ahead occur?\n\nGood question, but unfortunately difficult to answer precisely, for two reasons:\n- It depends on where we’d draw the boundary for “look-ahead” (for example, which effect size would be sufficient in our various experiments). While we think our results are strong enough to conclude that look-ahead is involved in at least some inputs (those with high effect sizes), it’s much less clear where exactly to set the threshold.\n- Our methods can only establish a lower bound on how often look-ahead occurs. For example, throughout the paper, we consider representations on future move target squares. In principle, additional look-ahead mechanisms that use entirely different pathways could exist, which we might not pick up on.\n\nWith those caveats in mind: Our dataset is 2.5% of the initial Lichess dataset (see lines 83 and 91). On this dataset, we think look-ahead is involved in most states (but as discussed, it depends on where we draw the boundary). However, these 2.5% of states were, of course, not directly selected for high effect sizes; rather, we only used a behavior-based filtering procedure. So the true number of states where look-ahead is important is likely significantly larger than this among Lichess puzzles.\n\n> Why aren't the squares of the 2nd move similarly important?\n\nAs discussed in line 163: “We are unsure why the squares of the 2nd move aren’t similarly important. This may simply be because the opponent’s move is typically “obvious” in our dataset or because suppressing the opponent’s best response doesn’t reduce the quality of the 1st move.”\n\nIf the two hypotheses we give there are right, the apparent unimportance of the 2nd move would largely be an artifact of our experimental setup (studying tactics puzzles where the current player is winning). But we can’t rule that there is some interesting deeper reason that would require a more detailed understanding of Leela’s internal mechanisms. \n\n> Why aren't there pawn or king heads, similar to the other piece types?\n\nGreat question! There are, in fact, pawn and king heads, they just involve a few complications, and so we decided to ignore them for simplicity. But we see that not mentioning them was a mistake, thank you for drawing attention to that! We’ll rectify this by adding a few comments to the paper:\n- A few king heads exist (based on attention patterns). But our dataset doesn’t contain many puzzles where the starting player moves their king on the first move, so we wouldn’t have much data for our piece movement head ablation. This is simply because king moves are much rarer in tactics puzzles.\n- Pawns move in different ways depending on context—they usually take one step forward, but they capture diagonally, and they can take two steps at once if they’re in their starting position. Based on attention patterns, it seems that there are distinct heads for these different types of movement, so we decided to ignore pawns to simplify the presentation in the paper. Again, we recognize this omission is also liable to lead to confusion, so we will bring this up in the paper.\n\n> Nitpick: the motivation in the first paragraph about chess being different from other domains because in other domains \"models can solve their tasks with a single simple algorithm\" also applies to chess.\n\nThank you for pointing this out, our phrasing here is unfortunate and we’ll improve it. What we meant to gesture at is simply the difference in complexity between playing chess well and e.g., finding the path from the root of a tree to a specific leaf (which is the task studied in the most similar prior work).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "knXvSjTD1G", "reviewer": "Reviewer_ScDd", "summary": "**Update after rebuttal:** To me personally, the authors' response clarifies all open questions and misunderstandings and adds interesting new results. I remain convinced that the work is ready for publication and interesting to the NeurIPS audience, which means my score remains 'Accept'.\n\nThe paper investigates whether it is possible to reliably identify functional signatures of look-ahead in a Leela Chess Zero (policy) network. This analysis in a complex domain (chess) with a relatively large network is challenging. To do this, the paper uses three interpretability techniques for transformers: activation patching to measure the effect of specific interventions on neuron-activations and the sub-network that they influence, analysis and structured ablation of attention patterns, and training of simple read-out probes to test whether certain information is represented in the internal state or not. Additionally, the paper comes up with a method to automatically collect a large, high-quality dataset of challenging chess situations (that simpler chess engines cannot solve) that have a short and unique solution. This allows reliably identifying candidate-targets to aim the look-ahead investigation at. It also allows automatically constructing highly non-trivial interventions on the board state, which are crucial for meaningful activation patching. The paper finds clear and convincing evidence (from multiple angles and with various control experiments) that at least in these situations, the Leela Chess Zero network performs a one-step look-ahead of its next move (an opponent move is in-between, making it a two-step look-ahead in terms of game steps).", "questions": "1. Are there situations where the weaker Leela gets the puzzle right, but the stronger Leela does not? Or is the stronger Leela strictly better?\n2. Related: how has the weaker Leela been trained? Is it plausible that, e.g. stronger MCTS during training of the weaker Leela means look-ahead is much less important, hence why it does not develop look-ahead (or does look-ahead only develop at sufficiently large and strong networks)? [It is perfectly fine to respond that you do not have the answers for these questions and not spend any more time on this; I am just curious.]\n3. The probe accuracy in Fig 8 is relatively high in early layers - how does this fit with the activation patching results that show the largest effect only in medium-to-late layers 6-11? \n4. As far as I am aware the precise details about the training process of this variant of Leela have not been published (in an academic venue). While this is beyond the scope of this paper, having these details (or as many as possible), e.g, in the appendix, would greatly help the scientific community to use this version of Leela as something that can be reliably reproduced. This may help boost Leela’s popularity for *scientific* research using openly available chess models. Similarly, the model and dataset used in this paper (with the fine-tuning) are a nice contribution.\n5. L225 typo:  “are seem”.", "limitations": "Limitations are nicely discussed in Sec 5.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Very well written paper, with clear intro, and explanation of the fairly involved methodology, and great supporting figures.\n* Very high-quality case-study of interpretability and analysis of a concrete capability/mechanism “in the wild” (meaning in contrast to networks trained on synthetic data to facilitate or simplify the analysis). By its very nature, interpretability work is typically bespoke, and while some general techniques can be developed, I think the field greatly benefits from a body of well executed case studies that provide at least an abstract recipe and approach for others to adapt.\n* Great experimental work, with convincing evidence, important controls, and multiple investigations aimed at the same question to provide strong evidence. Despite the challenging setting, experiments are conducted with care and rigor, and attention to ruling out some obvious alternatives that could lead to the same findings.\n* The filtered dataset is a contribution in itself, as well as the particular model used in this paper, which may easily be picked up by follow-up research.", "weaknesses": "* By the nature of a case study, the paper’s findings are limited to the particular network and the particular domain. The authors have managed to successfully identify and exploit aspects of the architecture (due to the network architecture, the internal state seems to map well onto a 2D chess board) and the data (filtered puzzle set with unique and short solution trajectories). I think this is crucial for the success of the current study and is highly non-trivial and original work, but naturally this means that the method cannot be straightforwardly applied e.g., to a LLM. I do believe though that the work here is highly inspiring to adapt the techniques to other models and domains.\n* (minor) Throughout the paper I was wondering why the policy network and not the value network was used. As the appendix says, both networks share the same torso and simply add relatively shallow heads, meaning the findings in the paper also apply to the value network. I think it would be good to mention this and a few more details in the main paper (such as training via supervised learning, and fine-tuning to remove history-dependence). I will list these below in ‘Improvements’.\n\n**Verdict:**\nThe paper is a great case-study in interpretability and analysis, and I greatly enjoyed reading it. The question tackled in the paper is challenging, and I personally believe that the paper does a great job at producing a number of convincing pieces of evidence, with well chosen control experiments (such as a random network for the probes, and control-ablations for activation-patching and attention-ablations). While there is a challenge to transfer the method to other settings, the paper helps with this by clearly explaining the reasoning behind the various steps. Ultimately, I appreciate a well executed domain-specific case-study over a sloppy execution in a more general setting. Interpretability research is hard, particularly mechanistic interpretability, and I think that the field will be a composition of general techniques (which are also used in the paper) and exemplary case-studies for others to follow and apply to their research questions and domains. Therefore, I think the current paper may well have significant impact beyond the specific findings, which are limited to a particular network and a particular task. I currently recommend acceptance of the paper: all main claims are critically investigated and supported by evidence, the work is novel and very original, and I think is interesting to a wide part of the NeurIPS audience and may be quite impactful for follow-up work.\n\n**Improvements:**\n\n1. I think it would be nice to mention a few more details about Leela in the main paper. In particular that the torso is the same as for the value network (and the value-head is relatively shallow and has no additional attention mechanism, so findings also hold for the value network). Maybe something like L425-428 in the appendix plus a brief description of the training (supervised on a set of high-quality trajectories, plus fine-tuning to remove history-dependency).\n2. It would be interesting to see how the findings regarding look-ahead evolve during training. This is completely optional, and beyond the scope of the paper (and the authors may not have access to the original training procedure / checkpoints). But it would be interesting to see whether attention heads and the activation patching results develop gradually or sharply and whether that can be related to a similar increase in performance on the filtered puzzle dataset used in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and great suggestions!\n\n>  Throughout the paper I was wondering why the policy network and not the value network was used.\n\nGreat question, there was no particular reason for this choice except that we decided to focus on only one head for the purposes of exposition. We have re-run all our experiments with the value head (i.e. using the log odds of the win probability instead of the log odds of the top move as the effect metric). As you suspected, the results are qualitatively the same as for the policy head, see the PDF attachment to the general response. We’ll include these in the paper and also clarify that the network has a shared body and two small heads.\n\n> I think it would be nice to mention a few more details about Leela in the main paper.\n\nThank you for the suggestion, we agree this would help readers and will move some information from the appendix to the paper!\n\n> It would be interesting to see how the findings regarding look-ahead evolve during training.\n\nWe agree this would be very interesting to study, but also that it’s out of scope for this paper. There are public training checkpoints for at least some versions of Leela, so in principle, this would be feasible to study in future work. Applying our exact methods would require finetuning away the history requirement for each checkpoint, which should be possible (albeit somewhat computationally intensive if we want to use many checkpoints in order to even detect potential sudden transitions).\n\n> Are there situations where the weaker Leela gets the puzzle right, but the stronger Leela does not? Or is the stronger Leela strictly better?\n\nThe weaker Leela very occasionally does better than the strong version “by luck.” For example, there could be a move that looks good to the strong Leela but is, in fact, very bad for a subtle reason that even the strong model misses. If the reasons in favor of this move are themselves somewhat subtle, then the weak model might not even consider it, and just play a mediocre move, rather than the strong model’s bad move. But apart from such edge cases, the difference in playing strength is quite large, and we would be surprised if there are any classes of states where the weak model is systematically stronger.\n\n> how has the weaker Leela been trained? Is it plausible that, e.g. stronger MCTS during training of the weaker Leela means look-ahead is much less important, hence why it does not develop look-ahead (or does look-ahead only develop at sufficiently large and strong networks)?\n\nBoth models were trained using supervised learning on MCTS rollouts (so the loss functions are the same as in MCTS training, but the data comes from an existing strong network, rather than the network under training). Given that MCTS trains the network to predict the results of tree rollouts, we don’t think strong MCTS would make look-ahead less important (this would only be true if the network was optimized end-to-end to maximize the playing strength of the overall MCTS process). We’d indeed guess that look-ahead becomes more prevalent for stronger networks, but of course, this would require future work to actually test.\n\n> The probe accuracy in Fig 8 is relatively high in early layers - how does this fit with the activation patching results that show the largest effect only in medium-to-late layers 6-11?\n\nGood question, we are not entirely sure what the answer is. One possibility is that there is a collection of different mechanisms involved in look-ahead, and not all of them activate in the same cases or are placed in the same layers. Probes might learn to exploit look-ahead mechanisms that are present in early layers, whereas the more targeted activation patching experiments only pick up on a subset of look-ahead mechanisms. L12H12 is already an example of a mechanism that is sometimes involved in look-ahead but is important less often than look-ahead mechanisms in general (such as the piece movement heads, contrast figs. 5 and 7). So it’s plausible that there are also some mechanisms that *only* probing picks up on.\n\n> As far as I am aware the precise details about the training process of this variant of Leela have not been published (in an academic venue). While this is beyond the scope of this paper, having these details (or as many as possible), e.g, in the appendix, would greatly help the scientific community to use this version of Leela as something that can be reliably reproduced.\n\nIndeed, most information about Leela is only available on the Leela Discord (which is public but naturally not designed to convey all that information compactly). We’d be happy to extend appendix A with some additional information (but a full description of Leela’s training details would fill its own paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5JFJihN3B", "reviewer": "Reviewer_p7Jw", "summary": "The author examine if a chess NNs encode future moves in their activations. The results suggest that Leela predicts future self-play moves and that the activations can be manipulated to change the predicted moves.", "questions": "Could you give a more formal definition of \" look-ahead in neural networks\" and explain how this work proves it?\n\nDoes the move probe work on human games or non-puzzle positions?\n\nFigure 8, what is the accuracy measuring? There are 64 squares on a chess board, how is random getting above 2% accuracy?\n\nWhat would a falsification look like in this method? The patching for example seems so show that you can disrupt the chosen square, but this assumes the activations are encoding this linearly. If they were non-linear or only partially in the patch how would that changes these results?", "limitations": "Discussed above", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "# originality\n\nLinear probes for difference concepts are well known in the chess literature, but the activation patching and explicitly looking at future moves as concepts are novel.\n\n# quality\n\nThe analysis looks good, but the lack of consideration of alternative hypothesizes or attempts to disprove the results weakens the results.\n\n# clarity\n\nThe paper is clear and flows well.\n\n# significance\n\nI think this work is interesting and presents some good ideas for chess AI and the broader XAI community.", "weaknesses": "My main concerns with this paper is that it does not test the methods outside of a very limited scope. Even looking a puzzles where Leela gets the next move wrong would be a good first step, i.e. what is the accuracy of your move predictor when the model is wrong, or can you patch to get the correct move.\n\nI'm also not that surprised by this result and don't think the main motivation is addressed. The Leela NN is trained to predict the future actions of itself, and those are based on an explicit tree model of the game MCTS. The model internally doing a depth 3 search seems very plausible and I don't think proves the motivating claim of \"look-ahead in neural networks\", as the heuristics could simply be on the depth 3 tree instead of the depth 1 tree. One options for this analysis would be to look at a chess NN that's not trained on selfplay data (Maia, CrazyAra, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback!\n\n> looking a puzzles where Leela gets the next move wrong would be a good first step, i.e. what is the accuracy of your move predictor when the model is wrong, or can you patch to get the correct move.\n\nWe focus on puzzles that Leela gets right simply so we can apply our interpretability methods. For example for the move predictor probes: what should be the “ground truth” for the 3rd move in cases where Leela gets the puzzle wrong? We could see if the probe can still predict the 3rd move of the correct continuation, but given that Leela got the puzzle wrong, there’s no reason to expect this. Leela might still use look-ahead along some incorrect continuation, so we could ask how good the probe is at predicting the 3rd move in this incorrect line. But of course, there are many different incorrect lines (vs one correct line), and we don’t have ground truth for which, if any, Leela is considering. For patching, it will massively depend on how invasive an intervention we perform. For example, patching the right squares in the final layer could just override the move output to be whatever we want.\n\n>  The model internally doing a depth 3 search seems very plausible and I don't think proves the motivating claim of \"look-ahead in neural networks\" \n\nWe don’t follow; if the network is doing a depth 3 search (over possible move sequences), we’d certainly consider that an instance of look-ahead in neural networks. Note that we are not claiming look-ahead arbitrarily many steps into the future, is that the point of confusion?\n\n> Could you give a more formal definition of \" look-ahead in neural networks\" and explain how this work proves it?\n\nBy look-ahead in neural networks, we mean internally representing future moves (of the optimal line of play) and using those representations to decide on the current move (see line 45). Our probing results (section 3.3) are our most straightforward evidence for representations of future moves, though the residual stream patching experiments (section 3.1) also suggest this already. The patching experiments in section 3.2 give evidence of such representations being used the way we’d expect, and in particular to decide on the current move. For example, L12H12 seems to move information from future move target squares back to the target square of the next move, and a very targeted ablation in this head has outsized effects on the output.\n\n> Does the move probe work on human games or non-puzzle positions?\n\nAll puzzles in the Lichess dataset we use come from human games, they are simply selected to be tactically interesting states. In many chessboard states, there are many plausible lines of play, so predicting several moves into the future is fundamentally much less feasible. Using puzzles means that at least there is a clear ground truth correct line, and we can then check whether Leela represents it.\n\n> Figure 8, what is the accuracy measuring? There are 64 squares on a chess board, how is random getting above 2% accuracy?\n\nThe “probe on random model” line in Fig. 8 is a probe trained on a randomly initialized model, which is different from a randomly initialized probe (or random predictions). Even the activations of a randomly initialized model contain information about the input, they just don’t contain more sophisticated features. So this baseline checks how well future moves can be predicted just by the probe itself, without interesting features from Leela.\n\n> What would a falsification look like in this method?\n\nIf the experiments we ran didn’t yield non-trivial effects, we’d consider that evidence against look-ahead. For example, a probe trained to predict future moves could have achieved accuracy not much better than the baseline probe, or patching on future move squares could have had effects no bigger than patching on other relevant squares.\n\n> the lack of consideration of alternative hypothesizes or attempts to disprove the results weakens the results.\n\nWe did implicitly consider alternative hypotheses in every one of our experiments. For example, for the residual stream patching experiment (section 3.1), we wondered whether future move target squares only have big patching effects because these squares tend to be “obviously”/”heuristically” important in the puzzle starting state. That is why we used a very strong baseline, by taking the maximum effect over other squares. While future move target squares might often be important for heuristics, so are many other squares, and under a non-look-ahead hypothesis, it seems likely that the maximum over other squares should typically be larger than the importance of one specific future move square. We have similar baselines to rule out simple alternatives in all our experiments.\n\nWe realize that this reasoning might not always be apparent in the paper, so we’ll include the motivation for baselines more explicitly. Thanks for drawing attention to this!\n\nOf course, we can never fully prove that there's no alternative explanation for our results, we can only rule out specific alternative hypotheses. We don’t know of any concrete hypotheses not involving look-ahead that predict or explain our results, but we would be very curious in case you have a specific one in mind.\n\n> The patching for example seems so show that you can disrupt the chosen square, but this assumes the activations are encoding this linearly. If they were non-linear or only partially in the patch how would that changes these results?\n\nThere seems to be a misunderstanding here. We do not assume any form of linearity for our patching results. We do operate under the hypothesis that information about future moves is localized to their squares, but this is something we argue for with our results rather than an assumption (see line 128).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your detailed reviews and suggestions! We’re encouraged that you found the results *“interesting and inspiring”* (abVe), thought the paper gave a *“convincing answer to an important question”* (SpQ5), and said it has *“great experimental work, with convincing evidence, important controls, and multiple investigations aimed at the same question”* (ScDd).

**Value head experiments:** In response to feedback from reviewers abVe and ScDd, we’ve added versions of all our experiments that use Leela’s value head instead of its policy head to measure the effect of interventions. The results are similar to the policy head results in the paper (likely because most of the network is a shared body). We’ve attached the new results as a PDF, the interpretation of all figures is exactly the same as for the corresponding figures in the paper. The only difference is that we use the log odds of the win probability rather than the log odds assigned to the best move. Note that the probing results (section 3.3) are not specific to value or policy head, since they operate entirely on the shared body.

We’re also making a few minor changes for clarity based on feedback from reviewers. We mention these in response to individual reviews, along with answering questions and clarifying a few misconceptions.

Meta-review: The paper investigates the presence of look-ahead the neural network of Leela. It is fairly difficult to provide reliable evidence either way, but the authors do offer a sensible approach. There is a disagreement amongst the reviewers about how valid the set-up is, and what would be the future impact, but this is not surprising given the unorthodox problem. Considering how much engagement the paper generated from the reviewers, and the strong positive reaction from a subset of them, I feel that the paper could be of interest for the audience of the conference. I would encourage the authors to take into account the comments of the reviewers, especially in clarifying some of the details and claims.
Decision: Accept (poster)