Understanding the learned look-ahead behavior of chess neural networks

Authors: Diogo Cruz
Primary area: interpretability and explainable AI
Keywords: model behavior attribution, look-ahead planning, mechanistic interpretability

Abstract:
We investigate the look-ahead capabilities of chess-playing neural networks, specifically focusing on the Leela Chess Zero policy network. We build on the work of Jenner et al. by analyzing the model's ability to consider future moves and alternative sequences beyond the immediate next move. Our findings reveal that the network's look-ahead behavior is highly context-dependent, varying significantly based on the specific chess position. We demonstrate that the model can process information about board states up to seven moves ahead, utilizing similar internal mechanisms across different future time steps. Additionally, we provide evidence that the network considers multiple possible move sequences rather than focusing on a single line of play. These results offer new insights into the emergence of sophisticated look-ahead capabilities in neural networks trained on strategic tasks, contributing to our understanding of AI reasoning in complex domains. Our work also showcases the effectiveness of interpretability techniques in uncovering cognitive-like processes in artificial intelligence systems.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729852535635

thread_type:
official_review

<thread_object>
creator: reviewer_c1le

<content>
summary:
The paper presents the use of an explainability type of approach to study the depth to which a state of the art deep neural network "analyses" Chess configurations when solving Chess puzzles (and, as a consequence of the network being unaware of this context) as well as during normal play.   The investigation shows that information is used that represents the board state up to 7 moves ahead, with multiple avenues being explored.  The idea is that a similar approach could present insights into the operations or pseudo cognitive processes of neural networks.

strengths:
The paper extends existing work, going beyond previous results in discovered "search" depth and the fact that multiple paths are considered.  The paper also clearly illustrates that not all positions are explored in the same way, but that e.g. depth is situation dependent.  These are nice contributions made.
The paper is well written and easy to read.  The use architecture of the NN investigate is clearly explained, as is the used data.

weaknesses:
The contributions made in the paper are nice and interesting, but limited, both in terms of direct importance (personal view as a non-chess player or chess-AI designer), but also with respect to the wider applicability of the results.  It is unclear how the presented contributions, more than at a very vague conceptual level, would be applicable in other environments.  Chess is really well defined, and the rules are clear and we know how to perform search and lookahead.  In a more complex (and with this I mostly mean less well defined and messy environment) it might be difficult to apply a similar approach.
I understand it is always easy to ask for more, but I think the impact of your work would be greatly improved if you could provide an, even conceptual, study of a less well defined environment.  I also realise this is not easy to do.

questions:
I don't know why the puzzle set notation matters for the outcome.  It is nice information to have to completely reconstruct the work done in the analysis, but it might be information better relegated to the appendix?
I am not sure I fully comprehend the difference between activation patching and ablation.  I understand the difference in approach, but why would both techniques result in different knowledge about the inner workings of the network.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730305895384

thread_type:
official_review

<thread_object>
creator: reviewer_wthr

<content>
summary:
The paper analyses the look-ahead capabilities of Alpha-Zero like chess-playing programs. Specifically, the authors consider Leela Chess Zero and show the program’s ability to perform longer-term planning (up to 7 half-moves). Through a detailed analysis of the role of particular attention layers and heads, specific observations related to different types of solved (chess) problems are reported.

strengths:
1) The paper is well written and relatively easy to follow.
2) The experiments are well-designed and formally correct.
3) The considered topic is of interest to the ICLR community.

weaknesses:
1) The novelty of the paper is limited. The underlying research concept of verifying the Leela’s capabilities of long-term strategic planning directly follows Jenner et al. (2024) paper, only with extended planning horizon (to 7 half-moves)
2) Detailed conclusions regarding the role of concrete layer/head pairs in solving particular types of chess problems (positions) are not especially meaningful, neither from practical nor theoretical point of view.
3) The topic of considering alternative lines of play is not investigated deeply enough in the main paper, even though this path seems to have research potential and may lead to insightful observations.

questions:
1) What is the real (practical or theoretical) gain from extending the research of Janner et al. (2024) to a longer planning horizon?
2) What can the reader learn from detailed observations regarding the role of particular layer/head combinations?
3) The claim about the insignificance of the starting square is disputable. In certain types of problems, e.g. checkmate in 2 / 3 moves, often both the square which a given piece leaves and the one which the piece moves to are important, as leaving a given square has immediate consequences in terms of stopping the attack/defence of specific squares. What is the basis for the above claim?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684174239

thread_type:
official_review

<thread_object>
creator: reviewer_nttj

<content>
summary:
The paper extends recent work by Jenner et al. (2024) and investigates whether a chess-playing transformer network exhibits search-like behavior. The original paper found evidence for lookahead within a single line of play, for up to 3 consecutive moves. This paper considers sequences of up to 7 moves, along with positions with two plausible lines of play, and shows evidence that the network considers moves further in the future as well as moves on alternative lines of play. It also shows evidence that the network's behavior is context dependent, with certain attention heads mainly active in a subset of the positions and not others.

strengths:
The paper extends recent work by Jenner et al. (2024) and investigates whether a chess-playing transformer network exhibits search-like behavior. The original paper found evidence for lookahead within a single line of play, for up to 3 consecutive moves. This paper considers sequences of up to 7 moves, along with positions with two plausible lines of play, and shows evidence that the network considers moves further in the future as well as moves on alternative lines of play. It also shows evidence that the network's behavior is context dependent, with certain attention heads mainly active in a subset of the positions and not others.

weaknesses:
1. I suspect the audience for this paper could be rather limited, as it relies heavily on the interpretability techniques and evaluation methods introduced in the previous paper. Furthermore, it does not introduce new methodology, other than filtering the dataset into more fine-grained subsets. (Though I do think it answers interesting and unresolved questions from the prior work.)

2. The substantial filtering required to find positions that satisfy the constraints for deeper lines and multiple lines means that they use a much smaller dataset, and the results are likely less statistically significant. However, it should be possible to run these experiments on a much larger starting dataset, and it would be interesting to see if these findings hold up.

3. The log odds reduction plots (e.g. Fig 2) use hue and line-style that are quite unintuitive. I would suggest either making the colors distinct for distinct board squares, or use solid/dashed to distinguish the main/alt line with the same hue for the same move. In any case, there should be some simple scheme that's clearly documented in the main text.

4. The core result seems to be the finding that the model considers multiple move sequences, instead of just the main line. This is significant, as it means the model does search and not just look-ahead. However, these results are explained in just a few paragraphs near the end of Sec 3. I would suggest expanding this section.

5. I would also recommend the authors include (perhaps in the appendix) details about the methodology that are derived from the Jenner et al. paper, so that readers unfamiliar with the prior work will be able to understand what's going on.

questions:
1. Have you considered using a larger dataset to verify these findings are statistically significant?

2. Is there an intuitive way to interpret the hue and line-style for the log odds reduction plots?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699037594

thread_type:
official_review

<thread_object>
creator: reviewer_roh3

<content>
summary:
Building upon the previous work of Jenner et al., 2024, this study expands the analysis to examine evidence of learned look-ahead in a powerful chess-playing network. While the original work demonstrated look-ahead capabilities of 1 and 3 moves, this study extends the analysis to consider look-ahead depths of 5 and 7 moves. Additionally, the study presents evidence suggesting that the model can consider multiple lines of move sequences.

strengths:
- Expands on previous mechanistic interpretability results further, which were open questions from that work, which is important to show that these models can perform deeper look-ahead than previously thought, answering more broader questions about what kind of algorithms these networks can learn purely through data and gradient descent.
- Introduces the puzzle set notation which allows the separation of positions where moves either share or have distinct squares, which allow studying what board distributions are treated in what way by Leela in terms of look-ahead, and provide an analysis about what kinds of board positions we should expect to see Leela do 5th and 7th move look-ahead.

weaknesses:
- Apart from the puzzle set notation and separation of chess position input distributions, it is not clear what novel conceptual contributions are made in this work that might be more broadly applicable to interpretability research. For instance, the previous work introduced the idea of using a smaller model to filter positions, a bilinear probe design, finding L12H12, and more.
- The claim about the model considering alternative sequences of moves is not convincing (see questions), since there were no interventions done that would make the model pick the _alternative_ move. If indeed the model was deciding between two variations, patching moves from the principal variation should lead to a _reduction_ in log-odds of the primary move.

questions:
- Primarily if there were any experiments done to make the first move probability reduce in these multiple lines cases, not just increase.
- If the authors want to highlight any novel conceptual techniques that were not prominently highlighted that would be more broadly applicable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733022754405

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the reviewers for their engagement with the paper. We have uploaded an updated paper which incorporates the points raised by the reviewers, by including additional empirical results, and clarifying the motivation and analysis.

We highlight major changes to the updated paper in our responses. We address common questions raised by the reviewers in the global response below, while also responding to the bespoke questions raised by each reviewer in their individual responses.

## 1. Novelty and broader impact:
### Changes made:
- We have included an additional paragraph in the introduction (added lines 046-053), clarifying the broader implications, and how these are distinct from previous work.
- We have also rephrased the key contributions of the paper (modified lines 065-078), to better reflect the novel aspects of our work.
- We have added 2 paragraphs to the end of the introduction (added lines 085-098), discussing the novel methodological contributions of our work.
- We have significantly rewritten the Conclusion section (modified lines 494-519), to better reflect the broader implications of our work.
### Points clarified:
- We show how context-dependent processing emerges naturally, with different mechanisms activated based on position type. This specialization reveals how networks can develop specialized components without explicit architectural constraints.
- We demonstrate that neural networks can learn sophisticated planning algorithms through training alone. The consistency of processing mechanisms across move depths suggests the model learns complex pattern matching rather than memorizing heuristics.
- Our technique for analyzing branching decision processes can be applied to any domain requiring understanding of how models evaluate multiple futures. The branch scaffolding approach could provide a systematic way to study how models internally consider alternative futures in other domains.
- The puzzle set notation provides a framework for categorizing decision contexts in chess, potentially applicable to other chess networks and similar domains.
- The multi-technique approach provides a template for analyzing complex behaviors in other AI systems, revealing different aspects that might be missed by any single approach.

## 2. Clarifying usage of different techniques:
- We have clarified activation patching by adding additional details (added lines 165-167). The technique reveals causal relationships between internal representations and outputs.
- We have clarified the differences between techniques (added lines 185-191). While patching reveals causally necessary information, probing shows what information is encoded but may not be causally used. Ablation identifies critical components.
- We have moved probing results from appendix C to the main paper, including Figure 3 and its discussion (added lines 270-301).
- We have added context motivating the puzzle set notation (added lines 194-205), which enables systematic analysis across position types.

## 3. Improvements on the plot presentation:
- We have rewritten the captions of Figures 2 and 6, and updated Figure 6 to include a second plot.
- We modified Figure 6's plotting style for clarity, using solid/dashed lines for branches A/B and shaded regions for standard deviation.
- We clarify that Figure 6 plots "Log-odds reduction" consistent with Figure 2, where negative reduction means increase.

## 4. Clarifying and extending alternative move analysis:
- We have expanded our analysis to 609 puzzles (up from 41), with results in Figure 6 and Figures 30-31 in the expanded appendix F, with updated analysis at lines 418-460. In Figure 6, we replaced confidence intervals with standard deviation of the mean to show that while 609 examples might seem small, it's sufficient to conclude the log-odds change is not null.
- We now include analysis of both main (branch A) and alternative (branch B) variations:
  - As seen in Figure 6 (left plot), patching branch A move squares (1st or 3rd moves) reduces log-odds for branch A's first move, while patching branch B moves increases them
  - As seen in Figure 6 (right plot), patching branch B move squares reduces log-odds for branch B's first move, while patching branch A moves increases them
  - These effects are consistent across all tested puzzle sets (shown in Figure 30 in Appendix F), though as seen in Figure 2's residual stream patching results, the effect is stronger for certain puzzle types
- Ablation experiments on head 12 in layer 12 (results in Figure 31, Appendix F) show it moves information backward in time from 3rd to 1st moves for both branches simultaneously and independently, strengthening evidence that the model considers multiple sequences.

We believe these revisions have substantially strengthened the paper while maintaining its core contributions to understanding learned look-ahead behavior in chess-playing neural networks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper builds on the work of  Jenner et al.  to analyze the chess-playing neural networks' ability to consider future moves. It extends the prior work to deeper depths and multi-line consideration. It also studies how context influences the network's look-ahead behavior.

While reviewers generally agree that this paper is well written and it extends the analysis of Jenner et al., multiple reviewers are concerned the limited novelty compared to the previous work. Some reviewer is also concerned the narrow scope of this work that may limit its impact to the broader community.

It could have a bigger impact if the authors could show the same method can be applied to broader applications that require strategic multi-step planning.