Understanding the learned look-ahead behavior of chess neural networks
Authors: Diogo Cruz
Venue: Accepted by TMLR ()

Abstract: We investigate the look-ahead capabilities of chess-playing neural networks, specifically focusing on the Leela Chess Zero policy network. We build on the work of Jenner et al. (2024) by analyzing the model's ability to consider future moves and alternative sequences beyond the immediate next move. Our findings reveal that the network's look-ahead behavior is highly context-dependent, varying significantly based on the specific chess position. We demonstrate that the model can process information about board states up to seven moves ahead, utilizing similar internal mechanisms across different future time steps. Additionally, we provide evidence that the network considers multiple possible move sequences rather than focusing on a single line of play. These results offer new insights into the emergence of sophisticated look-ahead capabilities in neural networks trained on strategic tasks, contributing to our understanding of AI reasoning in complex domains. Our work also showcases the effectiveness of interpretability techniques in uncovering cognitive-like processes in artificial intelligence systems.

Published in Transactions on Machine Learning Research (09/2025)
Understanding the learned look-ahead behavior of chess neu- ral networks
Diogo Cruz
Pivotal
Reviewed on OpenReview: https: openreview. net/ formm? id np48g2a1ml
Abstract
We investigate the look-ahead capabilities of chess-playing neural networks, specifically fo- cusing the Leela Chess Zero policy network. We build on the work Jenner et al. (2024 by analyzing the model's ability to consider future moves and alternative sequences beyond the immediate next move. Our findings reveal that the network's look-ahead behavior is highly context-dependenl, varying significantly based on the specific chess position. We demonstrate that the model can process information about board states up to seven moves ahead, utilizing similar internal mechanisms across different future time steps. Additionally, we provide evidence that the network considers multiple possible move sequences rather than focusing single line of play. These results offer new insights into the emergence of sophis- ticated look-ahead capabilities in neural networks trained strategic tasks, contributing to our understanding of AI reasoning in complex domains. Our work also showcases the effectiveness of interpretability techniques in uncovering cognilive-like processes in artificial intelligence systems.
1 Introduction
Recent advances in artificial intelligence have produced systems capable of superhuman performance in complex domains like chess and Go (Silver et al., 2018). However, the mechanisms underlying these systems' decision-making processes remain poorly understood. A key question is whether neural networks trained on such tasks learn to implement sophisticated planning algorithms, or if they rely primarily on pattern matching and heuristics.
This paper builds on recent work by Jenner et al. 2024 that found evidence learned look-ahead behavior in chess-playing neural network. We extend their analysis from the third move to the 5th and 7th future moves, reveal the network's use of distinct but consistent internal mechanisms for different move sequences, and demonstrate its ability to consider multiple possible move branches simultaneously.
Understanding the internal reasoning processes of these models is important for several reasons. First, it provides insights into the nature of intelligence that emerges from neural network training, potentially informing our understanding of both artificial and biological cognition (McGrath et al. 2022 Second, it has practical implications for improving AT systems strategic domains, as deeper understanding of their planning mechanisms could lead to more efficient and robust architectures (Czoch et al.] 2024). Finally, it contributes to the broader field of AI interpretability, which is essential for building trustworthy and controllable Al systems (Chattopadhyay et al. 2019
Understanding how look-ahead capabilities scale to longer sequences is particularly revealing. It helps us determine whether models can truly chain logether long sequences of moves or rely primarily on short-term patterns, and whether they use similar or different mechanisms for near-term versus long-teri merg-errm planning. Our findings on specialized attention heads suggest neural networks can develop generalizod planning strategics applicable to novel situations, challenging the view that transformer systems merely memorize patterns without structured reasoning capabilities.

---

Published in Transactions on Machine Learning Research (09/2025)
Recent work in mechanistic interpretability has made significant strides in understanding the internal work- ings of language models (Geva et al. 2023 Wang et al. 2023) and game-playing models (Li et al. 2023a; Nanda et al. 2023). However, most of these studies have focused on relatively simple tasks or isolated com- ponents of larger systems. Our work aims to bridge this gap by analyzing sophisticated planning behavior in state-of-the-arl chess engine.
Chess provides an ideal testbed for this investigation due its well-defined rules, clear strategic elements, and the availability of strong neural network-based models (Ruoss et al., 2024 Unlike language models, where the notion of "correct" behavior is often ambiguous, chess allows for precise evaluation of model performance and decision-making. Additionally, the game's complexity necessitates long-term planning and consideration of multiple possible futures, making it a rich domain for studying advanced cognitive processes in Al systems.
Our key contributions arc:
Demonstrating that the model's look-ahead behavior is highly dependent on the specific type of chess position, with different piece capture and checkmate scenarios being stored differ- ently in the residual strcam, and processed differently by multiple attention heads;
Extending the analysis of Jenner et al. (2024) of look-ahead behavior to the 5th and 7th future moves in chess positions. Specific attention heads seem strongly responsive to longer term future moves, and the model appears to process some future moves using similar concrete internal mechanisms;
Showing that the model considers multiple move sequences, not just single line of play. Moreover, corrupting the board squares relevant to one move sequence often leads the model to pick the alternative move sequence, as expected for look-ahead behavior.
These findings provide new insights into the look-ahead capabilities that can emerge in neural networks trained on strategic planning lasks. They also dernonstrate how interpretability techniques can uncover sophisticated cognitive processes in AI systems. Our work contributes to the growing body of research on Al planning and reasoning (Chen et al. 2021; Hao et al., 2023 Ivanitskiy et al., 2023 Garriga- Alonso et al., 2024), offering a detailed look at how these capabilities manifest in a complex, real-world domain.
To obtaín these results, we construct novel approaches to analyze the model's look-ahead behavior, extending the techniques used in Jenner et al. 20024). We introduce puzzle set notation that disentangles the model's 's behavior for different types of chess positions, and enables a clearer analysis of the model's ahea- behavior for higher move counts. We use activation patching to measure the cansal importance of different board squares in the model's decision-making process, probing to test the prediction accuracy of the model's future moves, and ablation to identify the allention heads that are responsible for the model's look-ahead behavior. By showcasing how these techniques can be used in complementary manner, we expect their usefulness to extend to future mechanistic interpretability studies of other models with potential look-ahead or planning capabilities.
We also adapt the board corruption technique used in Jenner et al. (024) to work for multiple move sequences, and apply it to analyze the model's consideration of alternative moves. This analysis should be suitable for future studies of planning behavior in other domains, by making it easier to produce contrastive pairs for activation patching, thereby enabling a more fine-grained analysis of the model's behavior for different look-ahead strategies.
2 Setup
This section describes the chess model, dataset, analysis techniques, and notation used in our analysis. All experiments were run using an RTX 3070Ti, with a combined runtime of days. For reprodueibility, additional details available in Appendix H
2

---

Published in Transactions on Machine Learning Research (09/2025)
2.1 Chess Model
In this study, we analyze the Leela Chess Zero (Leela) policy network, which is part of a larger MCTS-based chess engine similar to AlphaZero (Silver et al. 2018). Leela is currently the strongest neural network-based chess engine (Haworth & Hernandez 2021). Its policy network takes a single board state as input and outputs a probability distribution over all legal moves.
Loela transformer that treats each of the 64 chessboard squares as one sequence position, analogous token in a language model. This architecture allows to analyze activations and attention patterns on specific squares. The version of Leela use has 15 layers and approximately 109 million parameters. As explained in Jenner et al. [2024), Leela originally takes in past board states in addition to the current one. For our analysis, we use their finetuned version that only considers the current board state, which simplifies the generation of corrupled states for activation patching while maintaining equivalent performance to the original model.
2.2 Dataset
We use Lichess' 4 million puzzle database as starting point. Each puzzle in our dataset has starting state with a single winning move for the player whose turn it is, along with an annotated principal variation (the optimal sequence of moves for both players from the starting state).
In our analysis, we refer to moves in the principal variation as follows: The 1st move the initial move made by the player in the starting position. The 2nd move is the opponent's response the 1st move. The 3rd move is the player's follow-up move after the opponent's response. We extend this notation to refer to subsequent moves (e.g., 5th move, 7th move) when analyzing longer sequences.
The puzzles were curated inlo three datasets: a 22k puzzle dataset used in Jenner et al. 2024 solvable for the Leela model but difficult for weaker models to solve, and used for the 3 and 5-move analysis; a 2.2k dataset of 7-move puzzles; and 609 puzzles for the alternative move analysis. Additional details on the dataset generation, and their difficulty level, can be found in Appendices and H
2.3 Analysis techniques
We employ three main techniques to analyze the internal representations of the model:
Activation Patching. This technique, also known as causal tracing (Meng et al. 2022). allows us to measure the causal importance of specific model components. For a given board state and model component (such as a particular square a particular layer), we replace the clean activations of that component with those from different "corrupted" board state. If this intervention significantly changes the model's output, it suggests that the patched component contained necessary information about the clean state that differed in the corrupted state. In our chess setup, follow Jenner et al. (2024)* 's approach, where "corrupted" board state is created by making minimal modifications to the original position that change the optimal next move while preserving the position's 's lactical complexity. Specifically, we slightly shift piece positions or remove non-essenn non-essential pieces to create a position where the originally correct move no longer optimal. This creates a controlled comparison where most board fcatures remain identical except for critical tactical elements. We quantify the effect of patching using log odds reduction the decrease in log- pro--bbabillty that the model assigns to the correct move after patching. For example, patching the activation at the 3rd move square in the residual stream often produces high log odds reduction, indicating the model uses information about this future move when deciding on the current move.
Probing. We use bilinear probes to decode information from the model's internal representations, following Jenner et al. (2024). These probes predict larget squares of future moves by modeling interactions bet wetween residual stream activations at different board positions. High probe accuracy suggests that the probed information explicitly encoded in the model's representations. In our setup, we use probes to test the prediction accuracy for the puzzles' future moves, based on the model's internal states when given the current board state input (see Fig. 3)
3

---

Published in Transactions on Machine Learning Research (09/202)
1
2
1st
2nd
3rd
3
1st
3
2nd
3rd
2
1
9
Figure 1: Examples of 3-move puzzles in puzzle set 112 (left) and 123 (right). "1stn, "2nd= and "grdn mark the move order, with the green (resp. red) arrow indicating the optimal move of the player (resp. opponent). The board squares the piece moves to are marked in blue. They are listed sequentially starting from 1. The resulting number sequence labels the associated puzzle set, with jet move + square 1, gnd + sq. 1, grd + sq. 2 resulting in the set 112, for example. For these two examples, the optimal move sequence (i.e. principal variation) results in checkmate, which may be marked with the prefix M, these examples additionally belong to the subsets M112 and M123, respectively.
Ablation. We employ zero ablation, particularly when analyzing attention heads. In this technique, we selectively certain activations to zero, effectively removing their contribution to the model's output. By comparing the model's performance before and after ablation, we assess the importance of specific components to the model's decision-making process. This method is particularly useful for identifying key mechanisms involved in look-ahead behavior. Specifically, we apply zero ablation to individual attention pat- tern entries (i.e., specific elements the computed attention matrices) within attention heads, to determine which board squares certain attention heads are mainly attending to.
These techniques allow to investigate how the model represents and processes information about current and future board states, providing complementary insights into its look-ahead capabilities. While activation patching captures direct causal paths but may miss indirect effects distributed across multiple components, probing can identify information that is encoded but not necessarily used for the final move choice. For example, our probing results show that the model encodes information about opponent moves, even when patching does not provide conclusive causal ovidence. Similarly, while patching provides a broad causal view applied across the entire model, ablation provides a fine-grained view of which board squares the model's attention heads are attending to, giving a better qualitative understanding of the model's behavior.
2.4 Puzzle set notation
In Jenner al. (2024); b, it was observed that the Leela model internally treats cases where the player's moved piece is immediately captured by the opponent differently from cases where the opponent piece moves to unrelated square (see Fig. p. When considering more complex future move sequences, the increasing number of different scenarios treated distinctly by the model makes its analysis challenging. To combat this problem, and disentangle the model's behavior for different. cases, we introduce a new labelling approach for each chess puzzle that we analyze.
We start by separating the data into different puzzle sets depending on the similarity between the board squares involved. In particular, for each player and opponent. move, label the move based the square
4

---

Published in Transactions on Machine Learning Research (09/2025)
8
Set 112
Set 11233
Log odds reduction
Corrupted
Corrupted
6
Other
Other
Move
Move
4
Move
Move
Move 3
Move
Move 4
2
Move
8
Set 11234
Set 11223
Log odds reduction
Corrupted
Corrupted
6
Other
Other
Move
Move
Move 2
Move 2
Move
Move
Move
Move
2
Move
Move
0
2
8
10
12
14
0
2
4
6
10 12
14
Layer
Layer
Figure 2: Log odds reduction of the correct move as a result of activation patching, for 5-move puzzle sets of the form 112XY, where Y (i.e. the fifth move square distinct from the first and third move squares). "Corrupted" indicates the square where a piece was (re)moved on the corrupted board, compared to the original board. Higher values indicate greater importance of that square for the model's decision. The label indicates the move square for the i-th move, with solid (resp. dashed) lines indicating the destination square for the player (resp. opponent) piece. "Other" indicates the contributions of the remaining squares. Dashed lines indicate opponent moves. Shaded regions represent two times the standard error of the mean. As expected, for the early layers, the original and corrupted board are differentiated by the content of the corrupted square, so effect dominates. For the final layers, the model has decidod which move to make next, the next move square dominates. For the middle layers, more complex dynamies emerge.
the piece moves to. For the analysis, do not consider the squares the pieces start in, as we verify that this additional complexity does not play significant role in the model's internal behavior (see Appendix previously observed in Jenner et al. (2024).
To handle the large number of move combinations, we developed concise notation system to categorize puzzles into sets, based on the pattern the squares the pieces move to, at each turn. The notation builds on Jenner al. (2024)* 's observation that different sets show distinct behaviors. The notation 8182...8, represents an n-move sequence where cach digit corresponds to square involved in the jth move. For example, "112' represents puzzles where the 1st and 2nd moves go to the same square (labeled '1"), while the 3rd move goes to different square (labeled '2"); and "123" represents puzzles where all three moves go to different squares (labeled '1', '2', and '3' respectively). Fig. 1 illlustrates this notation.
To simplily our analysis, we assign labels sequentially, starting with "1" for the first move square, then assigning new digits only when a move goes to a previously unseen square. This lets us categorize puzzles by their squarc-use patterns rather than specific board locations. For longer move sequences, we may add letters to our notation. Letters A, B, C, and D, specific positions represent distinct digits (e.g., "ABC" means three different squares), while letters X, Y, 7. represent arbitrary digits (e.g., "12X" could match
5

---

Published in Transactions on Machine Learning Research (09/2025)
Probing set 1123456
1.0
0.8
Accuracy
0.6
trained, move
0.4
trained,
trained, move
0.2
random, move
random, 5
random, move 7
0.0
4
8
10 12
14
Layer
Figure 3: Probing the model's residual stream for the puzzle set 1123456. The probc's accuracy decreases as we look into more distant future move squares, with the 7th move square's accuracy being considerably low, but still non-negligible when compared with the probe's accuracy for a random model. The observed accuracy increases as we traverse the model's layers, as the residual stream contains the move information in way that is progressively casier to decode. The sharp dropoff at the last layer likely stems from the model's lack of use of future move information by the policy and value heads, instead relying more strongly the next move information (see Appendix Ch.
121, 122, or 123). We occasionally prefix sequences with M (contains checkmate) N (non-checkmate) to distinguish these important tactical scenarios. No other letters are used.
3 Results
In this section, we verify that the Leela chess model looks ahead into the fifth and seventh moves when solving chess puzzles, and later show evidence that the model is able to consider multiple future branches when choosing the best move to play.
The starting move squares appear to play relatively minimal direct role. While the starting move squares (i.e. the squares the pieces start in before they are moved) are generally important for assessing the right next moves for each player, the results obtained in our analysis are consistent with the conclusion in Jenner et al. (2024) that the starting move squares do not seem to play signíficant direct role the look-ahead behavior of the model. Instead, the model scems to process the board state by dircctly encoding the moves of interest in the associated squares the pieces move to. Consequently, the model responds strongly to corruptions the destination squares of moves, while showing negligible effects for the starting squares (see results in Appendix Ap. Therefore, we only focus our analysis on the squares the pieces move to during each move, and ignore the squares the pieces start in.
While this approach simplifies our analysis, we note that information about starting squares is of course necessary for model to obtain high performance, as otherwise the model would not know the current board state. Our analysis simply shows that the model appears encode this information through mechanisms that do not seem to affect patching techniques, and may instead encode it in more subtle indirect ways.

---

Published in Transactions on Machine Learning Research (09/2025)
3.1 The model considers up to the seventh future move when choosing the best next move
We show probing results for the puzzle set 1123456 in Fig. B and additional results in Appendix C The probe's accuracy decreases the move square becomes increasingly more distant from the present, with the 7th move square's accuracy being considerably low, but still non-negligibl. Activation patching also show higher future move squares playing an important role in the model's performance (see Fig. ann Appendix B We also investigate whether the model can look beyond the 7th move, finding limited evidence for 9th -ove look-ahead (sce Appendix [
The model behavior is highly dependent on the puzzle set. The results of patching the model's residual stream for some 3 and 5-move sets are presented in Fig, 2 with additional results shown in Appendix B considered. We note that patching the fifth move square has
Only sets with more than 50 puzzles non-negligible effect on the log odds of the correct move for most 5-move puzzle sets. The effect is most salient for the set 11223, while not being very significant for set 11233.
The results of patching the attention heads for 3-vo puzzle sets can be seen in Fig. ! with higher move sets in Appendix E. We note marked differences between the sets, with the L12H12 attention head (i.e. head layer 12) being the most important for the set 112, but playing a weaker role the remaining sets. Moreover, the set 111 seems to respond more strongly attention heads L11H10 and L11H13, which do not seem to play significant role in the other sets. Sets 122 and 123 do not respond strongly to patching any of the attention heads. Additionally, in Fig. (with additional results in Appendix F]. we observe that the behavior of some attention heads varies notably depending on whether or not the board position will soon result in a checkmate, indicating that the model behavior is dependent on the near-term possibility of checkmate. The observed differences between checkmate and non con-checkmate scenarios may reflect the model's response to chockmate itself or to correlated tactical features such as king exposure or forcing move sequences, which camnot be ruled out.
Overall, we note that the importance of the future move squares is highly dependent on the puzzle set, suggesting that the Loela model does not treat the sets similarly. Further corroborating results can be found in Appendix
In no puzzle set does a distinct second or fourth move play a significant direct role. Nonetheless, probing results (see Appendix C) suggest that the model does contain information about the second and fourth move squares, but possibly in an indirect way that is not straightforwardly captured by patching techniques.
The model picks up on consistent patterns across different look-ahead depths. In Fig. 2 we note that the saliency of the fifth move square varies significantly between the sets, with the set 11233 barely displaying an effect, followed by 11234, and with the set 11223 displaying strong effect. Based on Fig. 2 and Appendices B and [ we hypothesize that the model may be using similar mechanisms consider the 3rd, 5th and 7th moves. We particular, for move C for puzzle sets of the form A.C.) (.... AAC,, and (..-)AAC, respectively (where ellipsis stands for arbitrary preceding moves). The corresponding puzzle sets shown in Fig. 2 for these three cases are puzzle sets 11233, 11234, and 11223, respectively, and they demonstrate this hierarchy of effects. The temporal invariance suggests the model has learned general pattern-recognition mechanisms for look-ahead that operate independently of the specific move number, though the overall effect magnitude docreases for more distant moves, with the 7th move appearing near the model's effective limit.
note that patching shows weak, moderate, and strong effects
We note that the patterns identify often correlate with chess tactics. Pattern AAC frequently involves opponent captures of just-moved picces (possibly sacrifices), while ACC involves capturing opponent pieces. However, our analysis reveals behavioral differences that transcend simple tactical categories: despite both involving captures, AAC and ACC show markedly different responses across all our techniques, while ABC (which doesn't 't correspond to any specific tactic) exhibits consistent intermodiate behavior. Though we cannot defnitively separate whether the model is responding to these abstract patterns or to correlated tacties, this categorization still proves useful for identifying systematic differences in model behavior.
7

---

Published in Transactions on Machine Learning Research (09/2025)
Set 111
Set. 122
Set 123
Set 112
CV
14
10
21
Head
10
14
16
18
10
15
1
10
10
15
20
14
14
14
14
22
4 8
10
12
14
0
2
6
10
12 0
4
10
12
14
0
2
6 10
122
Layer
Layer
Layer
Layer
Figure 4: Attention head patching results for puzzles with moves. Darker tones indicate higher log odds reduction of the correct move. The letters K, B, and R represent the king, bishop, and rook attention heads, respectively, identified in Jenner et al. (2024). Darker colors mark higher log odds reduction due to patching, with the highest being 0.73, for L12H12 (head 12 in layer 12) in set 112.
3.2 Some attention heads play a crucial role in processing future moves
We note from Fig. (and Figs. 24 to 26 in Appendix Ep that some heads matter a lot for the fifth and seventh move analysis.
In Jenner et al. )024), it was shown that attention head L12H12 moves information "backward in time" from the third to the first move square, for some 3-move puzzles.
In Fig. 5 we show that L12H12 is also important for the 5th and 7th moves. The results are shown in Figs. 16 to 18 in Appendix D We note that, for puzzles with five moves, L12H12 may be responsible not only for moving information backward in time from the third to the first move square, but also from the fifth move square.
We hypothesize that the attention head moves information backward in time from square C to A (or the 1st move square) when the puzzle set has the form (.--)AAC(... ), and to lesser extent when it has the form (...) A.B.(...C...... while not responding to the form (... (.C(..))...... When the set matches the pattern at multiple turns, the later turn often takes precedence (for instance, we would expect 11223, which matches both AAC(...) and AAAC, to mainly move information from the 5th, and not from the 3rd move square). Its behavior mimics that seen from the general activation patching results, discussed in more depth in Appendix B Overall, the specific patterns that L12H12 responds to appear to be time-insensitive- that is, the patterns AAC, ABC, and ACC may apply moves 1-2-3, moves 3-4-5, or moves 5-6-7. This suggests the model has learned some general maternn mechanisms across time rather than timing-spechie heuristics.
We also note that L12H12 strongly responds to moves that may result in checkmate, as previously seen in Fig. 5 We further investigate this behavior in various checkmate scenarios, including puzzles with multiple checkmate options, as detailed in Appendices D and G The model generally appears to have checkmate- specific mechanisms, which are not triggered for non-chockmate scenarios, though we cannot definitively rule out that the model responds to correlated features (e.g., king proximity, piece coordination patterns) rather than checkmate per
Other heads also play a crucial role for complex puzzles. We perform the same detailed analysis for the L12H17, L13H3, LIIHI3, and L11H10 heads, which showed the highest log odds reduction after L12H12 (see Fig. 4 and Appendix Ep. The results and discussion are shown in Appendix D Our analysis reveals distinct roles for these attention heads.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Set M112, L12H112
Zoomed
Set N112, L12H11
Zoomed
Log odds reduction
Log odds reduction
Ist target other
3rd-+lst target
other
2
100 94
96
100
50
100
94
96
988
100
Percentile
Percentile
Percentile
Percentile
Figure 5: Ablation results of the L12H12 head for checkmate (M112, left. and non-cheekmate (N112, right) puzzle set 112. We note that head L12H12 not only appears to mainly move information "backward in time", i.. from the third to the first move square, but it appears to be especially critical in scenarios that explicitly result in checkmate (in this case, in 3 moves).
Head L12H17 appears to move information "backward in time" for puzzle sets of the form AABCD, where C is different from D, and D is preferably equal to A. Notably, in sets of the form AABCA, the model relies more heavily on L12H17 than on L12H12. This head also only seems to play a major role in longer scenarios, as the performance downgrade from patching is not significant for 3-move puzzles. Interestingly, unlike L12H12 (see Figs. and 22 head L12H17 appears to respond more strongly to puzzle sets that do not result in checkmate (see Fig. 23 backward in time.
It may possibly complement head L12H12 in moving information
Attention head L13H3 seems to move information "backward in time" for puzzle sets of the form AABCD, where either C=D or B-C. However, its role less pronounced compared to L12H12 and L12H17. The roles of L11H10 and L11H13 are less clear based on the ablation results alone. While some puzzle sets show responses these heads in the attention patching analysis, the ablation results suggest their contributions may be more subtle or indirect.
Interestingly, our analysis of checkmate non-checkmate scenarios reveals that L12H12 plays more significant role in moving information backward in time in checkmate sccnarios, while L12H17 is more active in non-checkmate scenarios. This disferentiation suggests that the model may process checkmate and non-checkmate positions using distinct mechanisms, highlighting the context-dependent nature of its information processing strategies. The emergence of these specialized components through training, without explicit programming, demonstrates how neural networks can develop sophisticated information processing strategies for planning tasks. This provides valuable insights into how models might learn to handle complex sequential decision-making in other domains.
We note that the observed attention patterns represent meaningful structure rather than cherry-picker examples. We analyzed all puzzle sets with at least 50 examples (to ensure statistical reliability), covering the full range of move patterns in our datasct. The consistency of effects across different puzzle sets with similar patterns (e.g., all sets following pattern AAC showing similar behavior regardless of specific moves) strongly suggests these are general mechanisms rather than artifacts of selective analysis. While limited dataset size prevented analyzing every possible 7-move pattern, the obscrved regularitics across pattern groups provide compelling evidence for the model's systematic processing of future moves.
3.3
The model considers alternative move sequences
We investigate to what extent the model considers alternative moves, focusing on situations where the model exhibits uncertainty between two candidate moves, which we label as the main move branch and the alternative move branch B. To simplify the analysis, for this section, we restrict our attention to 3-move puzzles with two branching sets of moves, each with distinct first and third move squares (for total of 4 distinct squares), such that the Leela model assigns a probability around 1 1/2 of choosing each branch. We

---

Published in Transactions on Machine Learning Research (09/2025)
1,5
Branch A
Branch
Log odds reduction
1.0
Corrupted
Corrupted
Other
Other
0.5
Move 1
Move 1A
Move IB
Move 1B
Move 2A
Move 2A
Move 2B
Move 2B
0.5
Move 3A
Move 3A
Move 3B
Move 3B
1.0
2
10
12
14
Layer
Layer
Figure 6: Patching results of the alternative move analysis, for puzzle 123425 (where the last 3 digits stand for the alternative branch squares). The log odds reduction for the next move for branch A (left) and branch B (right) are shown. Negative log odds reduction for branch A (resp. B) implies that patching the square improves the model's odds of choosing the main (resp. alternative) move branch. Solid (resp. dashed) lines indicate the main (resp. alternative) move squares. Shaded regions mark the standard deviation for the mean of the log odds reduction across the puzzles considered.
note that the original Lichess puzzles are already picked to have a single optimal solution, and no alternative equally good options (see analysis in Appendix J). the alternative branch typically represents an objectively inferior move that the model nonetheless considers plausible. For activation patching, consider corrupted boards which are compatible with both branches A and B. See Fig. 29 for examples and Appendix for details.
We show some results in Fig. G with full results in Appendix Patching the alternative first move square (1B) consistently has a strong positive effect on increasing the model's odds of choosing the main first move (1A), and vice-versa, demonstrating the model's ability to weigh immediate alternatives. Patching the alternative third move square (3B) often improves the model's odds of choosing the main first move, and vico-versa, suggesting the model considers longer-term consequences of alternative moves. We note that the log odds reduction range is smaller than for the one branch case, in large part because the model's odds are spread between different branches.
Furthermore, our analysis of the L12H12 attention head in the context of alternative moves and chockmate scenarios (detailed Appendices and G indicates that L12H12 strongly privileges moving information from the third the first move square the principal variation, even for puzzles where Loela does not choose this the best move, long the puzzle set matches the pattern (- A.A) mentioned above. In the case of two branches with this pattern, head L12H12 appears to move information "backward in time" independently for each branch, without cross-attention behavior (see Fig. 31 In scenarios where multiple chockmates are possible, L12H12 shows less clear attention patterns that span across different move branches, suggesting a sophisticated evaluation of multiple winning lines.
4 Related Work
Mechanistic Interpretability: Recent works have studied mechanistic interpretability in various
texts, including factual associations, in-context lcarning, and arithmetic reasoning tasks (Geva et al. 2023
Wang el al. 2023). Key techniques, used in our work, include activation patching (Vig et al.] 2020
Meng
et al. 2022 Geiger et al.) 2021 Li et al. 2023b), probing (Hewitt & Liang 2019 Gurnee et al. 2023 Dai
et al. 2022). and ablation (McGrath al. 2023). Rescarchers have applied these techniques to pame-playing
models, including Othello (Li et al.) 2023aa] Nanda al. 2023)., chess (Karvonen) 2024). and Blocksworld
Wang al. 2024). Our work extends these approaches to understand look-ahead planning in chess, pro-
10

---

Published in Transactions on Machine Learning Research (09/2025)
viding comprehensive view of how chess-playing transformers represent and manipulate information about future game states.
Chess-playing neural networks: Following the success of AlphaZero (Silver et al. 2018); there has been increased interest in neural network approaches to chess. Recent work has explored more efficient architectures (Czech et al.) 2024) and training procedures (Ruoss et al. 2024 for chess networks, and studied their representations (Karvonen 2024). Our study focuses on understanding the reasoning capabilities of these models, particularly in how they process and utilize information about future game states.
Look-ahead and planning: Several recent papers have investigated whether neural models encode in- formation about future states. In the language domain, studies have examined if models can anticipate
upcoming tokens (Pal et al. 2023 Wu et al.] 2024). In the chess domain, Jenner et al. (024g) provided evidence of learned look-ahead in a policy network. Our work extends this analysis to longer-term planning and alternative move considerations. There is ongoing rescarch into the planning and reasoning capabilities
of large models (Men et al., 2024] Yao et. al., 2023 Hao et al., 20233 Ivanitskiy et al. 2023 Garriga-Alonso et al. 2024). Some work has found evidence of multi-step planning (Chen et al. 2021), while other studies
suggest limitations in systematic reasoning. Our work in chess provides controlled environment to study
these questions, offering insights that may be applicable to language models and other domains.
5 Conclusion
In this study, we have explored the look ahead behavior of the Leela chess model when solving chess puzzles, with a particular focus on understanding how the model processes and utilizes information about future moves.
Our analysis reveals three key findings. First, we demonstrate that the model can process information about board states up to seven moves ahcad, though this capability becomes progressively weaker for more distant moves. The model's look-ahead behavior is highly context-dependentl, varying significantly based on the specific puzzle set and whether the sequence leads to checkmate. Second, we find evidence that the model processes some future moves using similar concrete internal mechanisms, particularly through specialized attention heads like L12H12. These mechanisms appear to be pattern-sensitive rather than timing-specifie, suggesting the model has learned general strategies for processing look-ahead information rather than just position-specific rules. Third, our analysis reveals that the model considers multiple move sequences simultaneeuslly, with different altention heads specializing in processing different types of positions. For instance, L12H12 shows stronger responses in checkmate scenarios, while L12H17 is more active in non- checkmate positions.
Our methodological approach, combining activation patching, probing, and ablation techniques, provides complementary insights into the model's behavior. While patching reveals causally necessary information, probing shows that the model encodes additional information (such opponent moves) that may be used more subtly. This multi-faceter analysis approach could prove valuable for future studies of planning behavior in other domains.
We acknowledge several limitations in our work. First, our activation patching approach captures direct causal paths but may miss indirect effects distributed across multiple components. The model might utilize sophisticated distributed representations not fully captured by single-component interventions. Second, we cannot definitively determine whether the observed behavior represents true planning or sophisticated pattern matching, the model might instead implement complex heuristies that superficially resemble look-ahead behavior. Third, our analysis focuses not generalize to other architectures or planning tasks. Fourth, datasct constraints from the Lichess puzzle database limited our ability to analyze certain move patterns. Some configurations are underrepresented or absent entirely, particularly for longer sequences and alternative branch scenarios. Finally, our techniques may not capture all ways the model processes chess positions; for instance, while activation patching shows minimal direct effects for starting squares, they may influence decisions through pathways our methods don't 't detect.
specific model (Leela) in specific domain (chess) that might
11

---

Published in Transactions on Machine Learning Research (09/2025)
Despite these limitations, our findings have broader implications for understanding how neural networks can develop sophisticated planning capabilities through training. The emergence of specialized components and general pattern-matching mechanisms, without explicit programming, demonstrates the potential for developing AI systems capable of strategic planning in other domains. Our work challenges simplistic views of neural networks as merely statistical pattern matchers, revealing instead a rich internal structure for processing sequential information.
Future work could explore how these look-ahead capabilities transfer to other chess engines trained with dif- ferent architectures objectives, or to modified versions of chess with slightly different rules. In this work, we focused on version of the Loela chess model that solely relics on the current board state. The original model also had access to past board states, and it would be interesting to explore how that additional infor- mation might affect look-ahead behavior. Additionally, investigating whether similar mechanisms emerge in neural networks trained on other strategic games or real-world planning tasks could provide valuable insights into the generality of these findings.
Understauding how models develop look-ahead capabilities and handle complex deci- sion trees could inform the development of AI systems for other strategic tasks. This work may help improve our understanding of how these capabilities generalize, or fail to do so, in novel scenarios, which is necessary for developing robust and trustworthy AI systems in safety-critical applications.
Broader Impacts:
References
Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N. Balasubramanian. Neural net- work attributions: A causal perspective. In Proceedings of the 36th International Conference on Machine Learning, pp. 981-990. PMLR, 2019. URL httpps://proceedings. mlr wttp://www.p.eeee////ettaaoamaa. chtpas//y..//ttttttooohaayaaaaa
html
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Newral Information Processing Systems, pp. 15084-15097, 2021.
Johannes Czech, Jannis Blüml, Kristian Kersting, and Hedinn Steingrimsson. Representation matters for mastering chess: Improved feature representation in alphazero outperforms switching to transformers, 2024. URL htttps://axxvv.or////
14918
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furn Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Länguisties. Association for Computational Linguistics, 2022. URL httpps:/wwwt microsoft. wwtpaasttt..ooo.eeeeooommoooeeee en -nw/eereeeerrcch publication//mooledeeee http://oooooeeeoeoooooooo-sse--e neurons-in www.aaa----pprrrraeedaeataaeeee-
Adrià Garriga-Alonso, Mohammad Taufeeque, and Adam Gleave. Planning behavior in a recurrent neural network that plays Sokoban. arXiv preprint arX15:2407.15421, 2024. URL
2407.15421
Atticus Geiger, Hanson Lu, Thomas leard, and Christopher Potts. Causal abstractions of neural networks. In Aduances in Neural Information Processing Systems, 2021.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12283 Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1 v1/.:///122223..e w0w.....mmllpp--m...m (amnlp-mainn75)) main.751. URL httpp:://claatboooogy/ org/2023. e00.022..mmpppmmmmmm emnlp-main. main.751......1
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. arXie preprint arXiv:2305.016 2023. URL
12

---

Published in Transactions on Machine Learning Research (09/2025)
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. Association for Computational Linguisties. URL pettps://aclanthonnn http://oooooo.rgg/222222 wwg/2023.........oo....o...... emnlp-main.
Guy Haworth and Nelson Hernandez. The 20th top chess engine championship, TCEC20. ICGA Journal, 43(1):62-77) 73, 2021. doi: 10.3233/1CG-2100833
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint pp. 2733 2743, Hong Kong, China, 2019. URL https://aclanthologyy
Conference on Natural Language Processing
Association for Computational Linguisties. doi: 10.18633/v1/D19 1275.
org/D19-1275
Michael Igorevich Ivanitskiy, Alex F. Spies, Tilman Räuker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Structured world representations in maze-solving transformers. arXiv preprint orXiv:2812.02506 2023. URL httts:///axiiv
http://w.worra///////////// http::///arxx.oorr///////////// 02566
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in chess-playing neural network. In Advances in Neural Information Processing Systems, volume 37, pp. 31410-31437, 2024. URL lattps://openreviinn wtt.tt/oooorr//1111220035335
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint 1544: 2024. URL https://arxiv.orgg wtw.//gg//aaaaaaa./////// 15498
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations (ICLR), 2023a. URL https:// net/ forum?id-DeG07
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tine inter- vention: Eliciting truthful answers from language model. In Advances in Neural Information Processing Systems, 2023b. URL httppp///paaaeeeeeeeeeeeeeeerrrs
Lichess. Lichess databasc: Puzzles. 1.0; puzzles file last updated 2025-08-02.
2025. Data under CCO
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47))e2206622111 2022. doi: 10.1073/ /paa///aa..22 2206625129, URL 10t3////..07///paa.a. 2206625119
https: //wp:///wwww... www.wwwwwwwpppppwwww...........p www.a.ogg/a....p.ppp......pp...a org/
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprinl or 1771::1307.15771, 2023. URL
https://arxiv. https://arxiv.orr//bb///000 wrtp:ba////3/ .17771
Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Unlocking the future: Exploring look-ahcad planning mechanistic interprctability in large language models. arXiv preprint arXiv:2406. 20..033,66,633,, 2024. URL htttps//aaxxv..oot wtg/ 16033.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT, 2022. URL. https://arxxv..rrt http://wwwr//aab//220022222
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self- supervised sequence models. In Proceedings of the 6th BlackborNZP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, Singapore, 2023. Association for Computational Linguistics. URL http:://cclaatthooog..oo///// bta.s//o.bbaackooop-p11.22
13

---

Published in Transactions on Machine Learning Research (09/2025)
Koyena. Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace, and David Bau. Future lens: Anticipating subse- quent tokens from single hidden state. In Proceedings of the 27th Conference on Compulational Natural Language Learning (CoNLL), pp. 548 560, 2023. URL https://aclanthology wtt.:/aaaaaooooooogg2220.....00 con110331111-113
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Ellioo Catt, John Reid, and Tim Genewein. Grandmaster-lovel choss without search, 2024. URL https://axxi...rrr /bs// 22002.0440 04494
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanclot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419))1140--1144 2018.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shicber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Informalion Processing Systems, volume 33, 2020. URL Abstaatr html
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9840 9855, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.mmms www.mmmmin.mmeeeeemmmmm URL
URL https://aclaathhology/ org/ 20tg:2223..co0 emn.p-main. 609/
Siwei Wang, Yifei Shen, Shi Feng, Haoran Sun, Shang-Hua Teng, and Wei Chen. Alpine: Unveiling the planning capability of autoregressive learning in language models. arXiv preprint arXi:240550022) 2024.
Wilson Wu, John X Morris, and Lionel Levine. Do language models plan ahead for future tokens? arXiv preprint trXiv:2444.00850 2024. URL httpps://arxiv.org/bbbaa
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neurol Information Processing Systems, volume 36, 2023. URL
14

---

Published in Transactions on Machine Learning Research (09/2025)
8
112 Corrupted
Set 123241
Log odds reduction
Corrupted
6
Other
Other
Move
Move 1
4
Move 2
Move 2
Move 3
Move 3
Move 18
2
Move 25
Move 35
8
Set 123245
Set 123243
Log odds reduction
Corrupted
Corrupted
6
Other
Other
Move
Move 1
Move 2
Move 2
Move 3
Move 3
Move IS
Move 1S
2
Move 2$
Move 2$
Move 3S
Move 35
2
4
8
10
12
14
0
4
10
12
14
Layer
Layer
Figure 7: Residual effects the starting squares for the puzzle set 112. The baseline analysis is replicated in the top left plot, while 3 different subsets of the original set are shown the other plots. Shaded regions represent two times the standard error of the mean. Using our puzzle notation, we have the decoupling 112 - (123245, 123241, 123243) (the underlined digits denote the main move squares, with the other digits marking the starting squares). Not all possible sels are represented. Note that the effect of the starting square is not significant for any of the sets.
A Starting squares
To investigate whether the starting squares play a significant role in the model's behavior, we conducted a detailed analysis of the residual effects of patching these squares. We modified our puzzle set notation account for the starting squares, where for a set ...2 the odd indices represent the squares the piece in play starts in, and the even indices represent the squares the piece moves to.
Our analysis focused on 3-move puzzles (n=6) to maintain consistency with previous studies and simplify the interpretation of results. Figure presents the residual effects for the puzzle set 112, split into subsets based on the similarity between the starting squares.
The results demonstrate that the log odds reduction observed not significantly different for any of the subsets when compared with the baseline results for puzzle 112. This consistency across different starting square configurations suggests that the starting squares do not play critical direct role in the model's Decision-making process for these puzzles.
Based on these findings, we concluded that it was unnecessary to disentangle the effect of different starting square configurations when performing activation patching, probing, or ablation in subsequent analyses. This simplification allows us to focus on the more influential aspects the model's behavior, particularly the squares to which pieces move during the course of play.
15

---

Published in Transactions on Machine Learning Research (09/2025)
8
111 Corrupted
122 Corrupted
Log odds reduction
6
Other
Other
Move
Move 1
4
Move 2
Move 2
Move 3
Move 3
2
8
Set 123
Set 112
Log odds reduction
Corrupted
Corrupted
6
Other
Other
Move
Move 1
Move 2
Move 2
Move 3
Move 3
2
0
2
4
8
10
12
14
0
4
10
12
14
Layer
Layer
Figure 8: Log odds reduction of the correct move as a result of residual stream patching, for puzzles with moves. "Corrupted" indicates the patched square from the corrupted board. The label i indicates the move square for the i-th move. "Other" indicates the contributions of the remaining squares. Dashed lines indicate opponent moves. Shaded regions represent times the standard error of the mcan.
B Residual stream patching for 5-move and 7-move puzzles
In this section, we show the activation patching results for the remaining puzzle sets with 3, 5 and moves. In Fig. g the 112 and 123 set plots reproduce the results from Jenner al. (2024), with the slight change that we do not include puzzles of higher move count. For example, the set 11234 is not counted part of 112, as was the case in Jenner et al. 2024). Nonetheless, the puzzles with more moves have lower count in the puzzle datasct, so the results are not too different from those previously observed.
In most patching plots in Figs. and and Appendix B there a marked uptick in log odds reduction coming from the remaining squares for the last layer. A closer inspection reveals that this usually corresponds the modcl's chosen first move square for the corrupted puzzle version used for patching.
The hypothesis presented in Section that the model's behavior is highly dependent the puzzle set. We reiterate the hypothesis are, follows:
Hypothesis 1. The effect of patching move square is quantilatively different betweee cerlain puzzle sets. In increasing order of effect size, we note puzzle sets of the form (-- (--)ACC(...). (-.. (.BC(.....) and (... (..))A.((...)
In general, we exclude from the hypothesis puzzle sets where the odd move squares are not distinct.
As mentioned Section 3 we hypothesize that the model may be using similar mechanisms as in the third analysis. For the 3-move puzzle sets, note patching effect size, in increasing order, for 122, 123,
16

---

Published in Transactions on Machine Learning Research (09/2025)
Set. 111
Set 11111
Set 11112
Log odds roduction
Corrupted Otber
Cormupted
Corrupted
Other
Other
Move
Movn
Move
Move 3
Move
Move
Move
Move
Move
Set 11123
Set 11122
Log oddds reduction
Cormpied
Cormupted
6
Other
Other
Move
Move
Move
Move
Move
Move
Move 5
Move
10
12
6
10 14
Layer
Layer
Figure 9: Log odds reduction as result of residual stream patching, for puzzle sets of the form 111XY. The 3-move puzzle set 111 is shown on the top left, for comparison. As in Fig. 2 we note that the impact of patching the fifth move square varies considerably from puzzle to puzzle, but is consistent with the hypothesis presented. Shaded regions represent two times the standard error the mean.
Log orids reduction
Set 122 Corrupted
Set. 12223
Corrupted
Other
4-
Move
Move 2
Move 3
Move 3
Move 4
Move 5
12
Layer
Layer
Figure 10: Residual effects the puzzle set 122 and 12223. Shaded regions represent two times the standard error of the mean.
and 112 (111 is excluded, the first and third squares the same). In fact, note the following orderings in patching effect size:
Move 3: 122 < 123 < 112. Hypothesis holds. See Fig.
Move 5: AACC < (...)ABC < (...) AAC (in particular, (11)122 < (11)123 < (11)112, (12)344 < (12)345 < (12)334). Hypothesis holds. See Figs. and
Move 7: (1123)344 < (1123)345 (1123)334, (1123)455 < (1123)456 (1123)445. Hypothesis holds somewhat, conditional on the puzzle sets having the same prefix. See Fig.
Hypothesis seems to hold somewhat for carlier moves when the puzzle set suffix is the same. We have:
Move 123(44) < 112(33), 123(45) < 112(34), 123(33) < 112(22), 123(4567) < 112(3456). The hypothesis does not strongly hold for the case 122(233 123(34) 112(23).
Move 5: (11)123(45) (11)112(34), (11)233(34)
(11)234(45). (11)233(44) (11)234(555)
(11)233(45) ~ (11)234(56).
17

---

Published in Transactions on Machine Learning Research (09/2025)
Sec. 125
Set 12342
Set 12384
Log odds roduction
Corrupted Other
Cormupted
Corrupted
Other
Othwr
Move
Move
Move
Move
Move
Move 3
Move
Move
Move
Move
Set 12345
Set 12944
Set 1200
Log oddds reduction
Carmpitd
Cormuptel
Carmuptea
6
Other
Other
Othec
Move
Move
Move
Move
Move
Move
Move
Move
Move 5
Move
Move
101
12
6
10
14
10
12
14
Layer
Layer
Layer
Figure 11: Residual effects of puzzle sets of the form 123XY. The 3-move puzzle set 123 is shown the top left, for comparison. As in Fig. 2 we note that the impact of patching the fifth move square varies considerably from puzzle to puzzle. Shaded regions represent two times the standard error of the mean.
Additional activation patching results are shown in Figs. 10 and 12 Some of the theoretically possible puzzle are not shown because they are unlikely configurations. The puzzles shown have a sample size of least 50 puzzles.
We showcase puzzle set 1123456 in Fig. 14 patching the seventh move is negliggl but relatively small, indicating that it is likely at the limit of the model's ability to look ahead. Moreover, the probing results in Fig. 3 suggest that the model does contain information about the seventh move square, but the probe's performance only slightly better than for the random chess model. Additional 7-move puzzle sets can be seen in Fig. 13
We note that the model's log odds reduction
result of
18

---

Published in Transactions on Machine Learning Research (09/202))
Set 112
Set 11222
Log odds rediuctin
Corrupte Other
Corrupted
Other
Move
Move
4-
Move
Move 2
Move
Move 3
Move 4
2
Move 5
8-
Set 11231
Set 11221
Log odds reduction
Corrupted
Corrupted
Other
Other
Move
Move
Move
Move
Move
Move 1
Move
Move 5
Layer
Layer
Figure 12: Residual effects for the remaining puzzle sets of the form 112XY, not shown in Fig. 2 The 3-move puzzle set 112 shown on the top left, for comparison. In these puzzle sets, it not possible distinguish the effect of patching the fifth move square directly, it equals either the first or the third move square. Nonetheless, we note that puzzle sets 11231 and 11221 respond differently to patching the third move square. For sets where the st and 5th move square the same, the effect of patching that square is more pronouneed. Shaded regions represent two times the standard error of the mean.
19

---

Published in Transactions on Machine Learning Research (09/2025)
2.0
Set 1123354
Set 1129455
Set 1123345
Log odda reduction
Corrapted
Corrupted
Corrapted
1.5
Other
Mow
1.0
Move
Mave
Move
Move
Move 4
Move
Mow
Move
Move
0.5
2.0
Set 1129445
Set 1112545
Set 1123344
Log odds reduction
Currupted
Currupted
Currapted
1.5
Move
Move
Move 2
Move
1.0
Move
Move 3
Move 3
Move
Move
Meve
Move
0.5
2.0
Sat 1294567
Set 1111251
Sat 1129456
Log odds reduction
1.5
Other
Other
Move
Move
Move
Move
Mave 2
Move
1.0
Move
Move 3
More
Mow
Move
Move
0.5
Mooe
10
12
10
12
14
2
10
12
14
Layer
Layer
Layer
Figure 13: Residual effects for the puzzle sets with moves. The effect of patching the seventh move square is small but not negligible for most the puzzle sets, but its importance varies considerably. Shaded regions represent two times the standard error of the mean.
1.4
Corrupted
Other
Move
Move 2
Log odds reduction
Move
Move
Move
Move
0.8
Move
0.6
0.4
0
2
4
6
10
12
14
Layer
result of residual stream patching, for puzzles with 7 moves. "Corrupted" indicates the patched square from the corrupted board. The label indicates the move square for the i-th move. "Other" indicates the contributions of the remaining squares. Shaded regions represent two times the standard error of the mean.
Figure 14: Log odds reduction of the correct
20

---

Published in Transactions on Machine Learning Research (09/2025)
C
Probing results
Our probing analysis provides additional insights into the model's ability to encode and utilize information about future moves. We conducted probing experiments on various puzzle sets to complement our activation patching results and gain a more comprehensive understanding of the model's internal representations.
Figure presents the probing results for the puzzle 1123456. The probe's accuracy shows clear docreas- ing trend as the move square becomes increasingly distant from the present state. This decline in accuracy is particularly pronounced for the 7th move square, suggesting that while the model does encode some in- formation about very distant future moves, this information becomes increasingly uncertain difficult to extract.
Figure 15 shows the probing results for the puzzle set 12345, offering insights into how the model encodes information about both player and opponent moves. Several key observations can be made:
The probe can find both player and opponent move squares with high accuracy, generally peaking at layer 13. This suggests that the model encodes information about opponent moves, even though activation patching does not show a strong direct response for these squares.
The probe's accuracy decreases the predicted move becomes more distant from the present state, consistent with our observations from activation patching.
Interestingly, the 4th move (an opponent move) seems more difficult to predict than the player's 5th move. This could indicate that the model's representation of opponent moves is less direct more uncertain than its representation of the player's own future moves.
The probe's accuracy for the random chess model is notably higher for the first and second move squares, possibly reflecting some inherent biases or common patterns in chess openings.
These probing results complement our activation patching findings by revealing that the model does encode information about future moves, including opponent moves, even when this information does not have a strong direct effect on the model's output. This suggests that the model's internal representations are rich and multifaceted, capturing various aspects of potential future game states.
The discrepancy between probing and activation patching results, particularly for opponent moves, highlights the complexity of the model''s decision-making process. It suggests that while information about opponent moves is present in the model's representations, it may be utilized in more subtle or indirect ways than information about the player's own moves.
These findings underscore the importance of using multiple analysis techniques to gain a comprehensive understanding of the model's internal workings and decision-making processes.
21

---

Published in Transactions on Machine Learning Research (09/202))
Probing set 12345
1.0
0.8
trained, move
trained, move 2
Accuracy
0.6
trained, move 3
trained, move trained, move
0.4
random, move
random, move
0.2
random, move
random, move random, move
0.0
2
4
6
10
12
Layer
Figure 15: Probing the model for the puzzle set 12345. While activation patching does seem lead to strong response for opponent move squares, the probe can find both the player and opponent move squares with high accuracy, generally peaking at layer 13. These results suggest that the model is encoding information about the opponent's 's moves in a less direct way than the player's moves. We observe the probe's accuracy decreases as the model becomes increasingly more distant from the present. A notable exception is move 4, which seems to be harder to predict than the player's fifth move. The probe's accuracy for the random chess model notably higher for the first and second move squares.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Set 112, L12H112
Zoomed
Set 123, L12H11
Zoomed in
Log odds reduction
3rd-+1st target other
3rd 1st target other
2
25
50
75
100
96
100
25
75
100
94
96
98
100
Percentile
Percentile
Percentile
Percentile
Figure 16: Ablation results of the L12H12 head for the 112 and 123 move analysis.
Ablation results
This section presents detailed analysis of the ablation results for various attention heads, with particular focus L12H12, which appears to play crucial role in the model's look-ahead behavior.
The L12H12 head behavior (Figs.
16 to 18 is consistent with the following observations:
The head moves information from the 3rd the 1 move square for 112, and for all puzzle sets 112XY and 112VWXY, and to a lesser extent for the sets 123XY. The weakest case is the set 11223, as expected, since the 5th move behavior takes precedence.
The head moves information from the 5th directly to the 1st move square for the puzzle set 11223
and 11112 (pattern (... AACC,, and to lesser extent for the set 11234 (pattern (-.. (.-)) For 7-move puzzles, the effect is strongest for sets 11112XY, and to lesser extent for 11234XY.
The head moves information from the 5th to the 3rd move square for the puzzle sets 11223 and 12223 (pattern AAC).
The head moves information from the 7th to the 1st move square for the puzzle sets 1123334, 1111234, and 1123456 (patterns -AAC and (.... ABB).
The head does not directly move information from the 3rd move square for puzzle sets 122 or 122XY. It also does not move information from the 5th move square for puzzle sets (--))AC (such as 11222, 12344).
Nonetheless, the hypothesis not compatible with the set 12334, since would expect to observe behavior in between sets 12223 and 11223, and to mainly move information from the 5th to the 3rd or 1st move square, instead of from the 3rd to the 1st move square. We would also expect some effect from the 3rd to 1st move square for 12344. For 7-move puzzles, we observe no 3rd to 1st effect for 123VWXY sets.
Overall, head L12H122 appears to satisfy the hypothesis presented in Section g and noted for the residual stream patching analysis, where we observed that the model tends to move information from future move squares earlier move squares. Specifically, this head seems to prioritize moving information from the 3rd the 1st move square for patterns like AAB and ABC, from the 5th to the 1 3rd move square for patterns like AAC, and from the 7th to the 1st move square for patterns like AAC and ABC.
For other attention heads:
L12H17 (Fig, 19
appears to move information "backward time" for puzzle sets of the form AABCD, where C is different from D, and is preferably equal to A. sets of the form AABCA, the model relies more heavily on L12H17 than on L12H12.
L13H3 (Fig. 20 scems move information "backward in time" for puzzle sets of the form AABCD, where either C=D B=C.
23

---

Published in Transactions on Machine Learning Research (09/2025)
The roles of L11H10 and L11H13 (Fig. 21 are less clear based the ablation results alone.
Our detailed analysis of these attention heads reveals several important insights inlo how the model processes future move information. First, we find that certain heads specialize in moving information from future move squares the first move square, responding to specific patterns that are round-msensitive- that is, the same pattern may apply to moves 1-2-3, moves 3-4-5, moves 5-6-7. This suggests the model has learned general pattern-matching mechanisms rather than position-specife rules.
Different attention heads appear to specialize in different types of common patterns. For instance, L12H12 is particularly active in checkmate scenarios, while L12H17 shows stronger respouses in non-checkmate positions. This specialization indicates that the model has learned to process different types of tactical situations using distinct mechanisms.
These findings have broader implications beyond this specific chess model. They demonstrate how neural network can learn to develop specialized components for processing look-ahead information through train- ing, without explicit programming of such capabilities. The emergence of these general pattern-matching mechanisms suggests the model may be able to handle novel positions not seen during training. Additionally, this analysis provides case study of how detailed attention head analysis can reveal the development. of sophisticatod information processing strategics in trained models, insights that may extend models other strategic planning domains.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Set 11222, L12H12
Zoomed in
Set 11223, L12H12
Zoomed in
Log odds reduction
3rd-+lst target
Bhrd-lst target
other
5th->lst target
5th-+3rd larget other
25
50
75
100
94
96
98
100
25
50
75
100
94
96
98
100
Set 11233, L12H112
Zoomed in
Set 11112, L12H112
Zoomed in
Log odds reduction
3rd-+1st target
5th-+sst target
6th-+1st target
other
5th-+3rd target
other
25
50
75
100
94
96
98
100
25
50
75
100
94
96
98
100
Set 12334, L12H112
Zoomed in
Set 11234, L12H112
Zoomed in
Log odds reduction
3rd-+1st targel
larget 5th-+lst targel
5th-+1st largel
5th-+3rd target
5th-+3ed target
other
other
25
50
75
100
96
100
100
94
98
100
Set 12223, L12H12
Zoomed in
Set 12344, L12H12
Zoomed in
Log odds reduction
3rd-+1st target
3rd-+1st target
6th-+1st target
Ith target
5th-+3rd target
5th-+3rd target
other
other
25
50
75
100
96
100
25
75
100
94
98
100
Percentile
Percentile
Percentile
Percentile
Figure 17: Ablation results of the L12H12 head for sets with moves. This head's role varies significantly between the sets. For the sets 11223 and 11234, the head plays a significant role moving information from the fifth to the first move square. For the sets 11223 and 12223, it also moves information from the fifth to the third move square. For the sets 11222, 11233, and 12334, it mostly plays the known role of moving information from the third to the first move square. For the set 12344, it seems to be doing something else entirely.
25

---

Published in Transactions on Machine Learning Research (09/2025)
1123086,
Zoomed
1128540, in
odte rechnriee
Jnf-olet
farppt
bazget
7LL. 33rd lezgeh
Pita Hed turged
7ka ided Larget
7t3--dea tanget
7hb-wtth target
7ka-w0th
barget
orber
other
2001
100
Set 17255465, L12H112
Zoomed In
Set 1112546, L12H12
Zoooned
1122164, L12H112
Zoamed in
Logg 01d5 rednccionn
turpet
Carget
Thh-mlet torge
5th-ee Carget
383-=3rd banget
7rh-wbch targe
Sta-aIrd barget
70 larges
orler
7t5-elst Garget
7th-wIed
100
100
100
94
25
96
Set 12345677 L121112
Zoumed
Set 1111231, L121112
Zoomed
1123456, L121112
Zoumed in
Lung odte rutuucioe
Iet--let target
Seh--lat torpe
Ind-elet target
Tol- furpet
Stb-+Let taraet
5th-mded
Trh-wtch
Sch-aded
Larges
71h- target
78h Earget
obbar
Percentile
Percentile
Percentile
Percomslle
Percentlle
Percentile
Ablation results for some puzzle sets with 7 moves, for head L12H12. The results are quite varied. For puzzle sets 1123334, 1111234, and 1123456, the head has small but non-negligible role in moving information from the seventh to the first move square. For the most of the remaining puzzles, it mainly scems to move information from the fifth and third move squares to the first move square. For puzzle sets 1112345 and 1234567, the head appears to move information from and to unknown squares.
Figure 18:
11224, L12H17
Zoomed
Set L12H17
Zecmmed
11221, L12H17
Zoomed in
Log ondi relnction
Bed-- target
Brd - turges brt
ard -ist
Sth-olet tazast
other
50
2001
100
95
Percentile
Perocutile
Peroenttle
Percentile
Percentile
Perocatile
Figure 19: Ablation results for head L12H17, for the puzzle sets with 5 moves that seem to respond more strongly the head being patched (see Fig. [24. In all cases, the head plays a significant and almost exclusive role in moving information from the third to the first move square.
Set 11:22, 1,13H5
Zoomed in
Set 11223, 1,13H3
Zoomenl
Set 11235, 1.1HH
Zoamed in
odde relnciioo
turpet Seh--let turppe
Stb-eler target
Seb- target
Sta-sIrd barpet
obhee
100
100
100
Percentile
Perocntile
Peromtile
Figure 20: Ablation results for head L13H3, for the puzzle sets with 5 moves that seem to respond strongly to the head being patched (see Fig. 24]. When compared to heads L12H12 and L12H17, the head L13H3 plays a less significant role in moving information from the third to the first move square.
26

---

Published in Transactions on Machine Learning Research (09/202))
Set 11111, L11H13
Zoomed in
Set 11111, L11H10
Zoomed in
Log orids reduction
Bth target
5th Ist target other
other
25
50
75
100
94
99
988
100
25
50
75
100
94
96
98
100
Set 11122, L11H13
Zoomed in
Set 11122, L11H10
Zoomed in
Log odds reduction
5th target
5th 1st
Percentile
Percentile
Percentile
Percentile
Figure 21: Ablation results for hcads L11H10 and L11H13, for the puzzle sets with 5 moves that seem to respond more strongly the heads being patched (see Fig, 24). Surprisingly, the heads L11H10 and L11H13 seem to play very minor role in moving information from and squares of interest.
27

---

Published in Transactions on Machine Learning Research (09/2025)
Set M11222, L12H12
Zoomed
Set N11222, L12H11
Zoomed
Log odds reduction
3rd-+1st target other
Log odds reduccton
3rd-+lst target other
1
50
100
94
96
98
100
50
100
94
96
98
100
Set M11233, L12H12
Zoomed
Set N11233, L12H12
Zoomed in
Log odds reduction
Ist target
Log odds reduction
3rd 1st target
5th-+1st target
5th-+1st target
5th-+3rd larget
5th-+3rd target
other
other
50
100
94
96
98
100
50
100
94
96
98
100
Set M11112, L12H12
Zoomed in
Set N11112, L12H12
Zoomed in
Log odds reduction
5th-+Ist 1st target
Log odds reduction
5th target other
50
100
94
96
98
100
50
100
94
96
98
100
Set M11234, L12H12
Zoomed in
Set N11234, L12H12
Zoomed in
Log odds reduction
Sed-+1st target
Log odds reduction
3rd-+1st target
5th-+1st targel
5th-+1st target
5th target
5th-+3rd target
other
other
50
100
94
96
98
100
50
100
94
96
98
100
Percentile
Percentile
Percentile
Percentile
Figure 22: Ablation results for L12H12, for the puzzle sets with 5 moves that seem respond more strongly to the heads being patched (see Fig. (24). When decomposing by checkmate vs non-checkmate scenarios, observe that L12H12 plays more significant role in moving information backward in time in the checkmate scenarios.
28

---

Published in Transactions on Machine Learning Research (09/202))
Set M11234, L12H17
Zoomed
Set N11234, L12H17
Zoomed in
Log odids reduction
3rd-+ target
Log odds reduction
3rd 1st target
1th-+1st target
5th-+lst target
5th-43rd larget
5th-+3rd target
other
50
100
94
96
98
100
50
100
94
96
98
100
Set M11231, L12H17
Zoomed in
Set N11231, L12H17
Zoomed in
Log odds reduction
3rd Ist target
Log odds reduction
3rd 1st target other
Percentile
Percentile
Percentile
Percentile
Figure 23: Ablation results for L12H17, for the puzzle sets with 5 moves that seem to respond more strongly to the heads being patched (see Fig. 24). When decomposing by chockmate vs non-checkmate scenarios, we can observe that L12H17 plays a more significant role moving information backward time in the non-checkmate scenarios.
29

---

Published in Transactions on Machine Learning Research (09/2025)
E
Attention head patching
This section presents a comprehensive analysis of attention head patching results for various puzzle sets, providing insights into how different attention heads contribute to the model's decision-making process.
For 5-move puzzles (Figures Fig. 24 and Fig. 25 we observe distinct patterns:
Strong response to 1.12112 and L13H3: Some puzzle sets (e.g., 11223, 11233, 11234) show strong response to patching these heads, suggesting their crucial role in processing these positions.
Mixed responses: Puzzle set 11234 also responds strongly to L12H17, indicating complex interaction of altention heads for this set.
Strong response L12H17: Some sets (e.g., 11222, 12223) respond strongly to L12H17 patching but hardly to L12H12, suggesting different mechanisms at play for these positions.
Weak or inconsistent responses: Some puzzle sets (e.g, 12233, 12234) do not show strong responses to any particular attention head, which may indicate more distributed processing the involvement of other model components.
Response to L11HI0 and L11H13: Some sets (e.g. 11111, 11112) show responses to these heads, but ablation results suggest their role may be more subtle or indirect.
For 7-move puzzles (Fig. 26 the patterns become more complex, potentially reflecting the increased difficulty in processing longer move sequences.
The analysis of checkmate vs. non-checkmate scenarios (Figures Fig. 27 and Fig. reveals significant differences in attention head responses between these two types of positions. This suggests that the model may employ distinct processing strategies for checkmate and non-checkmate positions, potentially reflecting the different strategic considerations involved in each case.
These results highlight the context-dependent nature of the model's attention mechanisms and the complex interplay between different attention heads in processing chess positions. They also underscore the impor- tance of considering factors like move sequence length and the presence of checkmate possibilities when analyzing the model's behavior.
30

---

Published in Transactions on Machine Learning Research (09/2023)
Set 11222
Set 11223
Set 11233
K
K
15
R
B
B
1C
K
1 11
R1
PC K
11
10
6
R
80
11
Head
10
12
K
R
K
12
K
11
14
16
R
B
18
B
18
13
11.
15
11
15
12
10
20
22
Set 11112 12
Set 12334
Set 11234
0
K
K
K
2
R
K. 11
4
B
18
K
B8
X
B
6
R
Head
10
B R
R.
18
TR
1B
R.
K
12
R
R.
R
K
14
K
K
TB
16
13
K
TB
K
18
11
K
20
TR
FL
22
K
K
R
R.
Set 11231
Sct 11221
2
4
6
15
11
11
10
Head
12
R
15.
14
15
K
16
k
18
R K
K
R. K
20
22
R
X
K
K.
R
8 R. K
K
2
4
6
10
12
14
0
2
4 6
8
10
12
14
Layer
Layer
Figure 24: Attention head patching for some puzzle sets with 5 moves. In the top row, the model responds
strongly to patching L12H12, and to a lesser extent L13H3 (see Figs. 17 and 20 In the middle row, the response is more mixed, with puzzle set 11234 also yesponding to L12H17. In the bottom row, the model
responds strongly to patching L12H17, and hardly responds to L12H12 (see Figs. 19 and 20

---

Published in Transactions on Machine Learning Research (09/2023)
Set 12342
Set 12223
Set 12344
K
1
R
B
B
1C
K
TL
T1
R
PC K
11
10
6
80
11
Head
10
12
K
1
K
12
K
11
11
14
13
16
B
R
B
18
B
18
13
11.
15
K
12
10
20
22
Set 1111 11123
Set 12333
Set 12345
0
K
K
K
2
R
K.
11
4
B
18
K
B
K K
6
R
Head
10
B R
18
R
TR
R.
K
12
R
R.
R
K
RR
1
14
K
K
TB
16
13
K
K
18
K
X
20
T
FL
22
R
K
R:
Set 1111 1111
Sct 11122
2
4
10
6
15
21
10
Head
12
R
18.
R.
14
15
K
16
18
K
18
B
R. K
R. K
K
R. K
20
22
R
R X R. K
K.
R
X. R. K
2
4 6
10
12
14
0
2
4 6 8
10
12
14
Layer
Layer
Figure 25: Attention head patching for the remaining puzzle sets with 5 moves. In the top row, some atlention heads appear to strongly affect the model's behavior, but these do not appear to play a significant in other sets. The puzzle sets in the middle row do ngh seem to respond strongly to any particular attention head. In the bottow row, the model appears to respond somewhat to patching of heads L11H10 and L11H13. However, judging by the ablation results in Fig. 21 these heads do not seem to play significant role in the model's behavior.

---

Published in Transactions on Machine Learning Research (09/2023)
Set. 1123334
Set 1123455
Set 1 1123345
K
K
B
R
B
18
B
K
11
R2
11
11
K
11
K
6
80
11
10
Head
R
12
K
1
11
14
13
16
B
T.
B
18
15
11
13
10
20
22
Set 1123445
Set 11 1112345
Set 1 1123344
0
X
K
K
K
2
13
11
4
R.
B
18
K
B
B
K
6
R.
R
Head
10
BB
B R
-B R
K
1B
R.
R
K
12
R
R
R
K
K
14
K K
K
K
R
K
R
16
K
K
B
18
1
K
20
B
12
B
22
K
Sct 1234367
Set 111 1234
Sct 1 1123456
2
4
10
23
12
11
6
11
10
10
Head
12
R
15
14
15
21. 1t
16
B
K
B
18
B 1 Rt K
14
X
34
13
K
22 20
R R B R. K.
X
R
R.
K.
R
K
K
K
2
4 6
10
12
14
2
4 6
8
10
12
14
0
2
4
8
10
12
14
Layer
Layer
Layer
Figure 26: Attention head patching for some puzzle sets with 7 moves.
33

---

Published in Transactions on Machine Learning Research (09/2025)
E.1
Checkmate and non-checkmate scenarios
MI11
N111
M1
N1
X
K
Hend
11
16
18
20
22
M123
N123
M112
N112
Hend
11 12
11
30
18
10
20
20
1
1
1
4
10
12
14
0
2
6
10
12
14
4
10
12
14
2
6
10
12
14
Layer
Layer
Layer
Layer
Figure 27: Attention head patching for puzzle sets with 3 moves, for both checkmate and non-checkmate scenarios. We can observe that patching the attention heads leads to notably different outcomes in each scenario.
34

---

Published in Transactions on Machine Learning Research (09/202))
MI1III
N11111
M11222
N11222
Head
10
12
14
16
18
20
22
M11231
N11231
M12342
N123422
12 10
Head
14
16
18
20
22
M11233
1233
M12334
N12334
Head
12
14
16
18
10
20
22
18
14
12
14
12
Layer
Layer
Layer
Layer
Figure 28: Attention head patching for some puzzle sets with 5 moves, for both checkiatee and non-checkmate scenarios.
35

---

Published in Transactions on Machine Learning Research (09/2025)
1.00
1761
0.52
0,31
e5.1
A
1814
g414
1714
18h77
g 0.38
glh22 0.35
g- 0.77
glb2
6514
65g7 0.11
112
cle1
0.11
0.73
0
0.19
0.72
0.11
ele8
48ee 0.59
14:11
0.33
F88c
0.81
1.00
d4b2
0.88
0.85
A
2
cTcl
ald1 0.20
0.43
0.27
Figure 29: Instead of considering an arbitrary puzzle (top), whore the number of branching moves can become very large, we focus on puzzles with two distinct branches. The boards correspond to the starting state of the two puzzles, after the zeroth move (top of game tree) is played. The green nodes mark the principal variation. Note that, for the bottom example, the Leela model does not choose the best move, but instead chooses an alternative move.
F Alternative move setup and additional results
This section details our approach analyzing how the model considers alternative moves, focusing puzzles with two distinct branches of play.
In order to study the alternative move analysis, we have to find puzzles that satisfy significant number of constraints:
The puzzle's principal variation (PV) must have length 3. This is in order to simplify the analysis, and remove higher future move squares from consideration.
The puzzle's PV must not be checkmate. In practice, we observe that not only does the model Ireat different puzzle sets differently, but it also seems to have different behavior when a checkmate a likely option, even not the most likely. We impose this constraint in order to remove this confounder from the analysis.
The puzzle must have two distinct branches:
The model must be ambivalent between two first moves. Each move should have a probability of around 1/2. Tn practice, we impose lower bound of p 0.3 for the two moves.
36

---

Published in Transactions on Machine Learning Research (09/2025)
Given first move, the model must be confident in the second move. In practice, we impose lower bound of = 0.7 for the second move.
Given the first two moves, the model must be confident in the third move. practice, we impose lower bound of p 0.7 for the third move.
-One of the branches must correspond to the principal variation. Otherwise, the model cannot be said to be close to solving the puzzle, and it would be unelear to what extent the model's attention due to the alternative move setup
The latter two conditions are mainly to ensure that the model's attention is not too spread out over relatively unlikely future moves. Essentially, we are interested in puzzles like the bottom example in Fig. 29 (but without the checkmate scenario).
The two first and third move squares must all be distinct. Otherwise, would be impossible to distinguish the effects of the 4 squares the analysis.
The two second and third move squares must be pairwise distinct. This way, the 2nd move confounder is removed.
The puzzles should still be hard for the weaker model to solve. The hardness threshold is maintained at 0.05, as in Jenner et al. (2024).
The weaker model should be confident in the second move. The forcing threshold is maintained 0.7, in Jenner al. 2024
The corrupted puzzle versions should be viable for both branches. Previously, we found the corrupted puzzles using only constraints with the PV moves. Here, we also require that the corrupted puzzles are viable for both branches. Otherwise, the corrupted puzzle may treat the branches differently, and lead to unclear results.
These constraints are highlighted Fig. 29 Regrettably, starting with the whole Lichess' puzzle dataset, these constraints reduce the original 4062423 puzzles to around 600 puzzles. In practice, we observe that about half to two thirds of the puzzles have differences between the probabilities assigned the two branches' first moves that non-negligible, and that may explain some of the limited log odds reductions observed in Figs. 30 and
These results, while based on a limited sample size due to our strict criteria, provide evidence that the model does consider alternative moves in its decision-making process. The varying effects across different puzzle sets suggest that this consideration context- deeeeeett.
While Lichess puzzles designate single optimal solution, our filtering ensures Loela assigns 30% probability the alternative branch, indicating it considers hoth viable. The alternative typically represents good but sub-optimal continuation rather than objectively bad move.
37

---

Published in Transactions on Machine Learning Research (09/2025)
1.5
Set 112334
Set 112334 B
Corrupted
Corrupted
Log odds reduction
1.0
Other
Other
Move 1A
Move 1A
0.5
Move 1B
Move 1B
Move 2A
Move 2A
Move 2B
Move
0.0
Move 3A
Move 3A
Move 3B
Move 3B
-0.5
-1.0
1.5
Set 123456 A
Set 123456 B
Corrupted
Corrupted
Log odds reduction
Other
Other
Move A
Move
0.5
Move 1B
Move 1B
Move 2A
Move
Move 2B
Move 2B
0.0
Move 3A
Move 3A
Move 3B
Move 3B
-0.5
-1.0
1.5
Set 123445
123445 B Corrupted
Corrupted
Log odds roduction
1.0
Other
Move 1A
Move 1A
0.5
Move 1B
Move
Move 2A
Move 2A
Move 2B
Move
0.0
Move 3A
Move 3A
Move 3B
Move 3B
-0.5
-1.0
1.5
Set 123425 A
Set 123425 B
Corrupted
Corrupted
Log odds reduction
1.0
Other
Other
Move 1A
Move
0.5
Move IB
Move 1B
Move 2A
Move
Move 2B
Move 2B
0.0
Move 3A
Move
Move 3B
Move 3B
-0.5
1.0
Layer
Layer
Figure 30: Patching results of the alternative move analysis. The log odds reduction for the next move for branch A (left)) and branch B (right) are shown. Negative log odds reduction for branch A (resp. B) implies that patching the square improves the model's odds of choosing the main (resp. alternative) move branch. In this setup, the puzzle set label's first half denotes, most likely branch, and the second half denotes the second most likely branch. The number of examples is highly constrained, due to all the constraints imposed (see Appendix F for details). The model seems to consider alternative moves as one might expect. The effect of patching the alternative move squares (1B, 3B) seems especially pronounced for the puzzle sets

---

Published in Transactions on Machine Learning Research (09/202))
Set 112334 A, L12H12
Zoomed
3A-11A targes
3A-1BB target
3
3B-1A langet
Log odds reduction
3B-1B targel
80
Percentile
Percentile
Set 12334 B, LI2H12
Zoomed in
3A-+A targes
3A-+1B target
3B-+1A target
Log odds reduction
3B >1B
other
20
40
60
80
100
94
96
98
100
Percentile
Percentile
Figure 31: Ablation results for the alternative move analysis. The log odds reductions are shown for the next move when comparing against the branch (top) and branch B (bottom) clean log odds. Negative log odds reduction for branch (resp. B) implies that patching the square improves the model's odds of choosing the main (resp. alternative) move branch. We note that the head L12H12 appears to continue to focus moving information from the third to the first move square, when considering alternative moves. There does not seem to be significant cross- attention between the two branches, with both branches being processed independently.
39

---

Published in Transactions on Machine Learning Research (09/2025)
b
h
init 1.00
10-2
Absointe attribution value
glgō
cle5 0.16
eft7 0.07
c6d7 0.06
0.16
c5d3
g553
g5f7
c5d7 1.00
0.89
0.87
1.00
c1
c1fl
c1f2
cldl
g
clo8 0.61
0.27
0.26
0.73
10
17e6
F766
f7+6
1766
0.51
0.53
0.54
0.51
Rank
Figure 32: Simple puzzle where Lecla fails to pick any of the chockmates in 2 moves. Nonetheless, the future move squares those two game branches are still the main squares that L12H12 attends to, although weakly,
G L12H12 and checkmate
While studying the L12H12 head in the alternative move analysis, we noted that the head seems to strongly privilege moving information from the third to the first move square in the principal variation, even for puzzles where the Loela chess model does not choose the principal variation the best move. The main result be seen in Fig. but here specifically analyze this attention head in the alternative setup.
G.1 Different first moves
Upon further inspection, we noted that L12H12 seems to further prioritize scenarios involving checkmate. As result, in the situation where the principal variation resulted in chockmate, but this was not the modcl's top move, L12H12 still mainly attended to the principal variation squares.
To further investigate this phenomenon, we looked at puzzles of the set 112 where both the first and second top moves resulted in checkmate in 2. Unfortunately, none of Lichess' 4 million puzzles seem to contain such puzzles. As a result, we produced series of handerafted puzzles, and studied the attention of the L12H12 head to each of the squares in both branches. Since this scenario is not present in the Lichess dataset, is possible that this scenario extremely unlikely, and that the Leela model has not encountered such scenarios during training. In fact, even for relatively simple handcrafted puzzles, the model does not always choose the checkmate in 2.
40

---

Published in Transactions on Machine Learning Research (09/2025)
b
init
10-1
e
clal 0.14
glgo
0.26
0.19
0.12
Absonne attribution
10-2
g5fT
c
c
alel
1.00
1.00
0.50
0.89
g1g8
cle8
glel
111g2
h1h2
10-3
0.51
0.48
a7c5
elgl
clc2
0.38
0.85
0.77
Rank
where the rook moves are discouraged, leading Leela to prefer the checkmate options. L12H12 attends to the relevant squares much more strongly in this case. Note the scale difference the attribution plot when compared to Fig. 32
Figure 33:
Modification of the puzzle in Fig. 32
See Figs. and 33 for results. Interestingly, L12H12 not only shows the attention paltern g8-+17 and c8->d7 (corresponding to >1tt expected), but there is also cross-attenting between the 3rd move square and the 1st move square of different branches.
G.2 Different third moves
We may also look at scenarios where puzzles (of the set 112) have two different possible third moves for checkmate in 2, while having the same first and second moves. In the dataset pearrestnn_puzzlee all. ak1, we find 10 puzzles of this type. The puzzle with the highest attribution values shown in Fig. 34
41

---

Published in Transactions on Machine Learning Research (09/202))
b
M
3
3
3
3
8
3
d
h
1.00
b:466
h:15
0.61
0.21
Absointe ottribattion
1
h7g6 1.00
d7d6
0.73
g4h4 0.41
g4h33 0.39
n2a4
15e7
0.22
0.17
#785
c7c6
c8g4 0.94
0.38
0.22
0.16
10'
10'
10
Rank
Figure 34: Puzzle of the set 112 where there are 2 third move options for a checkmate Both options reinforce the choice of the first move square. Nonetheless, the contribution of L12H12 is relatively limited.
42

---

Published in Transactions on Machine Learning Research (09/2025)
H Implementation details
Our implementation is heavily based on the implementation described in Jenner et al. (2024) and previously made available at httpp:///githbb hwwp://aaaaaooooaaaiaaaa///lleea interp For the activation patching, probing, and zero ablation results, modifications were made to account for the case of more than 3 moves. Code for reproducing our results is available at
Analysis Techniques. Our analysis builds on techniques from Jenner et al. (024). which we detail here for completeness. For activation patching, we first run the model on the original position get the "clean" activations. We then create a corrupted position by replacing specific moves in the game history and run the model on this corrupted position. Next, copy activations from specific attention heads in the corrupted run into the corresponding locations in the clean run. Let m. be the correct move, Sp be the patched model state, and Se be the clean model state. The log odds change AL of the target move is then defined as:
AL log odds(me sp) log odds(me se))
(1)
where log odds(me)) s) represents the logarithm of the odds that the model assigns to the correct move me given state A negative AL indicates that patching reduces the model's preference for the correct move, while a positive AL indicates that patching increases it.
For probing, we extract activations from each attention head when rumning the model on chess positions. We then train a bilinear probe to predict the board square associated with the move interest, following the methodology in Jenner et al. (2024).. The probe accuracy serves as a measure what information is encoded by the model. The trained probe's accuracy also compared against random baseline.
Dataset generation. Besides the dataset used by Jenner et al. 2024) we create two additional datasets. The original 3-move dataset (here reused for the 5-move analysis) chess puzzle database (sec hocthsts::///t thub. cwm zles that are solvable by the Leela model but not weaker model. Concretely, the selected puzzles were those where:
created by starting from the Lichess inttee and filtering for puz-
the weaker model assigned less than 5% probability to the optimal first move indicating non- trivial puzzle.
the Leela model assigned at least a 50% probability to the 1st 2nd and 3rd optimal moves indicating would be able to solve the puzzle.
the weaker model assigned more than a 70% probability to the optimal 2nd move indicating that, once the optimal 1st move is played, there is only one feasible move for the opponent. This additional constraint allows us focus on studying the model's look-ahead behavior for its own moves, rather the opponent's, s, which could otherwise confound the analysis.
The 7-move dataset is created by starting from the 4 million puzzle Lichess datasct by filtering for puzzles:
with principal variation (i.e. the optimal sequence of moves) of exactly 7 moves.
where the 7th move square is distinct from the other odd move squares, in order to enable the use of activation patching, probing, and zero ablation techniques.
that are solvable by the Lecla model but not weaker model. In this case, as the number of 7-move puzzles is limited, the criteria used are similar to the 3-move dataset, but the constraints are less stringent, to ensure a larger dataset. The weaker model's probability of the 1st move can now be up to 20% (previously 5%), and no constraint is placed on the weaker model's assessment of the 2nd move. Regardless, the Leela model's probability of the 1st 2nd and 3rd moves must still be at least 50%.
43

---

Published in Transactions on Machine Learning Research (09/2025)
The generation of the alternative move dataset is more complex, and so is described in Appendix F We note that, for the alternative move dataset, the Leela model usually does not assign probability of 50% to the principal variation, since by construction, the model'ss probability is split between the two branches.
Generating corrupted puzzles. For the bulk of the puzzles, we rely on the implementation from Jenner et al. 2024p. For the alternative move dataset, we ensure that the corrupted puzzles are viable for both branches, by applying the constraints described in Appendix of Jenner et al. (2024) both branches.
For activation patching, need to create modified ("corrupted") versions of each chess position where the optimal next move changes. We generate these corrupted positions through minimal modifications to the original board state, following a multi-step procedure:
First, we create candidate corruptions by either: (1) removing a pawn from any position, (2) adding a pawn of either color to any empty square, or (3) moving a non-pawn piece to an empty square. These constitute small changes that preserve the overall structure of the position while potentially altering its lactical nature.
Next, filter these candidates to ensure:
The resulting position valid according to chess rules
The original best move remains legal move (though no longer optimal)
The original move's probability under the model decreases substantially
The position becomes slightly worse in value (suggesting the original best move is no longer optimal)
The overall policy distribution does not change too dramatically (to ensure the position's funda- mental nature is preserved)
Finally, we select the corruption that minimizes the Shasen-Shannon divergence between the original and corrupted policy distributions under the weaker (i.c. non-Lecla) model. This helps ensure the corrupted position is not trivially different from the original, preserving the position's 's tactical complexity while changing the optimal move.
For puzzles with alternative move branches, we apply additional constraints to ensure the corrupted position is compatible with both branches, allowing both candidate first moves to remain legal while being significantly less preferred by the model.
44

---

Published in Transactions on Machine Learning Research (09/2025)
Beyond the seventh future move
Probing set 111234567
Probing set 112345678
1.0
1.0
0.8
0.8
Accuracy
Accuracy
trained, move 3
trained, move 3
trained, movv 5
trained, move 5
0.4
trained, Joove 7 trained, move random, move random, move random,, move random, move
0.4
irained, move 7
trained, move 9
random, move
random, move
random, move
random, move
0.0
0.0
12
2
12
Layer
Layer
Probing set 123345678
Probing set 123456789
1.0
1.0
0.8
0.8
Accuracy
0.6
Avcuracy
0.6
trained; move
trained, move
trained, move trained,
trained, move 5
trained, move 7
trained, move
trained, move 9 random, random, random, random,
0.2
random;
0.2
raudom, random,
2
6
10 12
14
2
10
12
14
Layer
Layer
Figure 35: Probing results display some signal above random chance for the ninth move future. is either similar or slightly lower in strength to the seventh future move.
This signal
To determine whether or not the Leela model also possibly relies on future moves past the seventh, we perform an attention patching and probing analysis on the ninth future move. In order to ensure the robustness of our conclusions, in this section, we focus on 9-move puzzles (i.e. the final move is the ninth move). Additionally, only look at puzzle sets where the ninth move square is distinct from the squares involved in the previous moves, in order to isolate the effect of that move the attention patching and probing analyses. Due to the resulting limited number of puzzles, present the results for the puzzle sels with al least 100 puzzles.
Consider Fig. 35 For the top left puzzle set, the random model's probe performance similar to the accuracy for the seventh move. In general, it is possible, given the low puzzle count, that the probe getting score equivalent to that of a relevant but not necessarily used move square simply overfitting on the training data. Nonetheless, we observe that, while the probe accuracy of the seventh move square generally increases with the layer index (up to the last few layers), the same is not true for the ninth move square, whose accuracy largely holds constant for probes trained on most of the model's layers. These results may be indicative of the probe accuracy overestimate the effect of the ninth move square on the model's performance.
we plot the patching results for the ninth move squares, splitting the puzzle sets into those that involve mate-in-9 moves, versus not. For most puzzle sets, the effect of patching the ninth move square is minimal. For puzzle set M123456789, there slight uptick log odds reduction for the middle layers, possibly attributable the presence of checkmate, though the effect is still minimal.
In fact, in Fig. 36
45

---

Published in Transactions on Machine Learning Research (09/202))
2.0
Set M123456789
Set N123456789
Log odds reduction
Log odds reduction
Corrupted
Corrupted
1.5
Other
Move
Move
Move
Move 2
1.0
Move
Move
Move
Move
Move
Move 5
0.5
Move
Move
Move
Move
Move
Move
Move
Move g
2.0
Set M112345678
Set N112345678
Log odds reduction
Log odds reduction
Corrupted
Corrupted
1.5
Other
Other
Move
Move
Move
Move
1.0
Move
Move
Move 5
Move 5
0.5
Move
Move
Move
Move
Move
Move
Move 9
Move 6
10
12
Layer
Layer
Figure 36: When considering check puzzles with moves, the effect of the ninth move appears be residual, and slightly weaker that observed for the seventh future move in Fig. 13
46

---

Published in Transactions on Machine Learning Research (09/2025)
J Stockfish evaluation of alternative moves
In this section, we study how suboptimal the alternative move sequences in the analysis in Section compared to the optimal principal variation. As indicated on the Lichess website (Lichess) 2025 puzzles included in the database are filtered to not contain any alternative good moves, and instead having only one move (in the principal variation) that is clearly preferable. However, for the subset of puzzles Section 3.3 the Loela model assigns high probability to alternative moves.
3.3 are.
To quantitatively estimate the quality of these moves, looked at portion of the dataset used for Fig. We established baseline evaluation using Stockfish (depth 22, 8 threads, 2GB hash table, NNUE enabled), and, for each puzzle, evaluated up to three candidate moves:
The move associated with the principal variation;
2. The move associated with the alternative move sequence;
3. The move suggested by the weaker model (when different from the previous two).
The results can be seen in Fig. 37 The centipawn evaluation by Stockfish is scaled to set the principal variation at (small deviations result from stochasticity from rums with different seeds). We observe that the alternative moves average value of -700 centipawns, corresponding to difference of 7 fewer pawns the board, which is quite significant. A similar, but slightly bigger difference, is observed for the moves suggested by the weaker model. These results polentially indicate that the Leela model is mistakenly assigning high probability to the alternative moves, more due to poor analysis of the move sequence, and less so due to the alternative move sequence being reasonable. Additionally, no clear pattern observed between the probability assigned to the move and the centipawn score.
0.7
PV
Non-PV Weak Model
0.6
0.5
Model Probability
0.4
0.3
0.2
0.1
0.0
2500
-200
- -5500
-1000
500
0
500
Stockfish Evaluation Delta (centipawns)
Figure 37: Probability assigned move by Leela/weaker weaker model vs relative quality of the move, centipawns, as measured by Stockfish.
47


Reviews:
[{"review_id": "yYP62UXiot", "reviewer": "Reviewer_s9UU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is extremely well written, and the topic of exploration is fascinating. I appreciated the simplification introduced for grouping of puzzle sets, and the visualizations were compelling.\n\nThe biggest weakness in my mind is how niche the work is. Not just for a single domain setting (chess), but for a specific network trained for a specific solver (Leela) for a specific game. Much of the rhetoric surrounds individual attention heads (e.g., all the details on L12H12), and a lot of the findings pertain to just that one network. Some of the results and methods can be seen as general, but a fair portion is not.\n\nTo a lesser extent, some of the figures become simply impossible to read with the variance included (e.g., fig 2 and 7). I think the implication of having such high variance for those statistics should be discussed as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate a particular network trained to serve as the policy network in an MCTS game-playing agent for chess. The work builds on previous research that conducted similar study, and adds the following three additional contributions:\n\n1. Establish that the behaviour is dependent on the context of the situation (e.g., if a checkmate is achievable in limited moves).\n\n2. Extended the observed depth of reasoning evidence from 3 moves to 7.\n\n3. Established that multiple move sequences are considered simultaneously in the learned network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would suggest carefully going through the bibliography to see which entries are not complete. E.g., Jenner et al. (the most cited reference), has nothing but the authors+title+year.\n\n- Perhaps not something needed for the paper, but wouldn't you be able to check the conjecture of depth 7 being the limit of what's learned by creating specific puzzles with an obvious chain of reasoning that must be 8+ steps? In this case, we would expect Leena to perform poorly if the conjecture holds, right?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p9S0CJRN9U", "reviewer": "Reviewer_jeKi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The authors find that the model considers multiple move sequences and show that activation patching can be used to steer the model to prefer different sequences.\n* They extend the analysis of Jenner et al. (2024) from 3 to 7 chess moves and validate that e.g. specific attention heads enable an information flow backward in time.\n* The authors propose a puzzle set notation to better disentangle the model's behavior for different chess positions.\n* They provide a detailed description of their setup and use non-highend GPUs, which simplifies reproducing their results.\n\nWeaknesses:\n* The authors analyze one configuration of the chess model. Ruoss et al. (2024) provide checkpoints of different model configurations (size and training objective). An experiment whether their findings generalize across different configurations would strengthen the paper.\n* They simplify the model to only use the current state. Since specific attention heads seem to move information backward in time (within future move sequences), an analysis whether this extends to past states would be interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyze learned mechanisms of a transformer model trained to play chess. In contrast to many applications of language modeling, chess has well defined rules, which allow for more precise evaluation. The authors leverage this to analyze and probe the learned look-ahead behavior, using activation patching and linear probes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vx5KXQGdZk", "reviewer": "Reviewer_4dXC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the strongest points in this paper are\n1. The demonstration that intervening on future squares associated with the line following move A can *increase* the probability of move B, in cases where Leela is undecided between moves A and B. (Fig. 6) Arguably, one of the weaknesses of Jenner et al.'s work was that all the interventional results focused on *decreasing* the probability of a good move, which is in some sense easy to do by destroying important parts of the activations. So it's cool to see a case where an intervention can increase a good move's probability for understandable reasons.\n2. Extending results to the 5th and 7th move and to opponent's moves. A very natural follow-up question to Jenner et al. is whether the 3rd move is the limit of look-ahead, and whether look-ahead is restricted to the current player's moves. I think the results in this paper regarding those questions aren't particularly surprising, but it's great to have them!\n3. Providing examples of ways in which the cognition of Leela is messy. For example, the analysis of L12H12 and other attention heads, and the conditions under which they seem to matter demonstrates that we certainly don't understand the cognition of Leela very well yet. I think this provides some nice additional color.\n\nIn my mind, the biggest weakness is that the results read like a long list of interesting observations, rather than 2-3 key claims supported by solid evidence. I think some of the paragraph heading claims in section 3 are a bit vague, or it's hard to tell how well the evidence supports them without looking at a lot of figures in the appendix. (For transparency, I skimmed through the appendix and read a few parts in detail, but did not try to verify all the claims precisely. It's possible that some of my critiques below are addressed somewhere in the appendix, in which case I would instead take them as suggestions to modify the writing in order to make the main paper stand by itself better.)\n\nElaborating on the most notable examples:\n* \"The starting move squares do not play a significant direct role\": seems a bit imprecise; for example, I don't think the appendix shows that the starting squares *never* have a very big impact, just that the *average* impact is small. And only if impact is measured with the specific activation patching experiments from this paper / Jenner et al. There's surely a chance that the starting squares matter in some way that isn't captured by these. I think all you really need to justify is that it's fine to restrict yourself to splitting up puzzles by target squares; I agree appendix A shows that well enough, and would avoid making any broader statements (or at least hedge them carefully).\n* \"The model processes 3rd, 5th, and 7th moves similarly\": I found it hard to understand what the precise claim here was. I think a key part of it is that puzzles with the (...)AAC(...) move pattern show the strongest effect sizes under the intervention experiments, and that this is true no matter where the AAC sequence appears? Maybe this could be merged with \"The model behavior is highly dependent on the puzzle set\" into something like \"The model processes sequences that involve captures by the opponent of a piece that just moved differently?\" There's also the checkmate claim, but I'm currently unsure whether this is fundamentally about checkmates, or about some correlate, seems hard to distinguish. (Especially because the puzzle filtering and automatic corruption finding could introduce additional correlations.)\n* I feel like the focus on splitting move sequences by which squares are the same isn't particularly well justified. For example, the (...)AAC(...) pattern where you observe the strongest effects will almost always have the opponent capture a piece that just moved. In a tactics dataset like the Lichess one, many of these cases are probably sacrifices, so maybe what's really happening is that Leela processes sacrifices differently? Or maybe this subset of the dataset makes up a significant fraction of all the times the opponent captures a piece at all, and opponent captures are the central concept? This wouldn't technically make any of the results wrong---you could just stay agnostic about what the \"true\" reason is. But these subsplits feel very central to the current way the paper is presented, with most results divided up this way, and I don't think it's sufficiently clear why this is useful.\n* \"L12H12 is also important for 5th and 7th moves\": more of a writing nit, but maybe have a single section that argues that Leela looks ahead until move 5 and 7, and then consolidate the probing and interventional evidence there? I think this would help against the \"long list of interesting observation\" issue\n\nMisc methodological issues (my best guess is that none of these invalidate the results, but they'd be good to at least mention if nothing else):\n* I don't think it's clear that checkmate, rather than some correlate of it, determines whether a head is important\n* For the alternative move experiments, I think one of the branches is always going to be an objectively bad move, i.e. Leela making a mistake? (whereas the other branch is the correct move) AFAIK, Lichess puzzles always have exactly one move in the initial position that doesn't give away the advantage (except for mate-in-one puzzles). I don't see specific ways in which this would affect the results, but it seems like a slightly weird artifact of where the dataset came from, and seems worth mentioning\n* The probing accuracy for the 7th move (fig. 3) is starting to be scarily close to the one on the randomly initialized model. There might be simple heuristics that a probe could use to predict the 7th move, which are entirely unrelated to the model doing look-ahead; especially look-ahead that far into the future. For example, I'd guess that squares which have opponent pieces on them are all else equal more likely to be target squares for moves (this may be part of how the baseline probe does much better than random performance). If the model represented only where opponent pieces were after the 5th move, and didn't look ahead until the 7th, this could still help the 7th move probe outperform the probe trained on a random model. (And there could be lots of small heuristics like this stacking up.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper builds on a recent work by Jenner et al., which showed evidence that Leela Chess Zero (a chess-playing neural net) sometimes performs \"look-ahead\" in a single forward pass, i.e. considers future moves. This new paper extends the analysis by Jenner et al. in several ways, notably by showing evidence that Leela looks ahead further than previously shown (to the 7th move instead of the 3rd), and by evidence that Leela sometimes considers two different lines of play in parallel.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None of these changes are critical to securing an acceptance recommendation, but I think some of them would significantly improve the paper.\n* I would strongly suggest some writing changes, especially to section 3, in order to focus on making a few key claims (that are new compared to Jenner et al.), and then collecting all the evidence you have for each of those claims. (See my \"Strengths\" list for what I would take as the main claims, though I'm not saying you should just follow that list.)\n* Somewhat less confidently, I would also suggest de-emphasizing the puzzle subsplits a bit. I would definitely highlight that things are messy and that results aren't uniform across the entire input distribution. But I feel like these subsplits add a lot of complexity to the paper and it's not really clear what concrete, interesting takeaways they yield.\n* It would be good (and easy) to at least mention the methodological issues from above\n\nNits:\n* You mention \"linear\" probes in the main text but I think you mean bilinear ones? My impression is that \"linear probe\" is a pretty common term that specifically refers to probes that are linear (in the mathematical sense) in the model's activations, which bilinear probes are not\n* In 2.3, you say \"we apply zero ablation to individual weights in specific attention heads\"---I think you mean that you ablate \"weights\" in the attention pattern, rather than model weights? The phrasing is confusing here if so, since these \"weights\" are activations of the model. (If you ever ablate literal model weights, then I just misunderstood)\n* “Future work could explore how these look-ahead capabilities generalize to chess positions not present in the training data”---isn't this already true for this paper? Leela was presumably not trained on the Lichess dataset, since AFAIK it's just trained on self-play or on offline datasets generated via self-play?", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)