Interpreting Emergent Planning in Model-Free Reinforcement Learning
Authors: Thomas Bush
Stephen Chung
Usman Anwar
Adrià Garriga-Alonso
David Krueger
Venue: ICLR 2025 Oral (2025)
Keywords: reinforcement learning
interpretability
planning
probes
model-free
mechanistic interpretability
sokoban
Primary area: interpretability and explainable AI
TL;DR: We introduce and utilise a concept-based methodology to provide the first non-behavioural evidence that model-free agents can learn to plan

Abstract: We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by [Guez et al. (2019)](https://arxiv.org/abs/1901.03559), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan formation within the agent's representations, and (3) verifying that discovered plans (in the agent's representations) have a causal effect on the agent's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of a planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning algorithm learned by the agent and discover a strong resemblance to parallelized bidirectional search.  Our findings advance understanding of the internal mechanisms underlying planning behavior in agents, which is important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL.

Published as a conference paper at ICLR 2025
INTERPRETING EMERGENT PLANNING IN MODEL- FREE REINFORCEMENT LEARNING
Thomas Bush', Stephen Chung Usman Anwar't, Adrià Garriga-Alonso David Krueger* 'University of Cambridge, "FAR AI, 'Mila, University of Montreal
@8tbush@gmail. com, {mhc48, 8, ua237, dsk30 } cam ac, ux, adria@far.
'Equal contribution.
ABSTRACT
We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying methodology based on concept-based interpretability to model-free agent in Sokoban commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by Guuzz et al.) (2019), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan forma- tion within the agent's 's representations, and (3) verifying that discovered plans (in the agent's representations) have causal effect on the agent's 's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning al- parallelized gorithm learned by the agent and discover a strong resemblance bidirectional search. Our findings advance understanding of the internal mecha- nisms underlying planning behavior in agents, which important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL.
INTRODUCTION
In reinforcement learning (RL), decision-ting planning that is, the capacity of selecting immediate actions to perform by predicting and evaluating the consequences of future actions - is convention- ally associated with agents that possess explicit world models, like MuZero (Schrittwieser et al., 2020). This naturally raises the question: can model-free reinforcement learning agents that is, agents which lack explicit world models also learn to perform decision-time planning?
In prior work, Guez et al. (019)) introduced Deep Repeated ConvLSTM (DRC) agents. Despite lacking explicit world model, DRC agents behave like they perform decision-time planning. For example, they excel at strategic domains like Sokoban, and perform better if given extra test- time compute (Guez et al. 2019 Taufeeque al. 2024). However, this only partially answers the above question as these behaviors may not be due to internal planning but, rather, other mechanisms behavior in the environments studied. In this paper, we mechanistically analyze a Sokoban plaaoinnn DRC agent and show that it is indeed internally planning. In doing so, we provide the first non-behavioral evidence that model-free RL agents can learn to internally plan.
that generate planning-lin
Using concept-based interpretability (Kim et al. 2018). we provide three types of convergent evi- dence showing that the DRC agent has learned, and making use of, concepts that are instrumen- tally useful for planning. First, we use linear probes (Alain & Bengio 2016) to show that the agent represents specific concepts that predict the long-term effects of its actions on the environment. Then, demonstrate that these concept representations cess by analyzing how the agent uses them to iteratively construct 'plans' at test-time. Finally, we demonstrate that these concept representations causally influence the agent's behavior would be expected if these representations were being used for planning
associated with learned planning pro-
To summarize, this paper makes the following contributions:

---

Published as a conference paper at ICLR 2025
Agent evaluates plan
stop 0, tick
step bck
step 2, tek
step tckk
step 4,
3, bec
(B) Agent adapts its plan
(D) Agent plans forwards from boxes
step
step
step
plans backwards targets
Agent extends parailer
Stop O BCk
Step J, tck
step 2, tCK
step D. BcE
Step d, BCc
Figure 1: Examples of the DRC agent internally forming plans to push boxes to targets. A purple arrow square means that linear probe decodes that the agent plans to push box off of that square in the associated direction. No arrow on a square means that the probe decodes that agent does plan to push a box off of that square. (A) The agent evaluates naively-appealing route, concludes is infeasible, and forms a longer alternate path. (B) The agent adapts its plan and
changes the target plans to push the left-most box to. (C) The agent extends part of its plan backward from a target. extends many parts of its plan in parallel. We provide further examples in Appendices A.2. A.2.5
(D) The agent extends part of its plan forward from a box. (E) The agent
We design procedure, based on concept- based interpretability, for determining if model- free agent performs planning using a hypothesized set of concepts. This procedure involves (I) probing for planning-relevan concepts. (2) investigating plan formation in the agent's internal representations, and (3) verifying the causal effect of plans on the agent's 's behavior.
Using this procedure, we show that, in Sokoban, a DRC agent (Guez et al. 2019) internally forms plans, and that these plans can be altered to steer the agent. We find this agent learns a planning algorithm resembling parallelized bidirectional search, which differs from commonly-usee planning algorithms in RI.
This work aligns with the growing body of research demonstrating that model-free RL agents can learn to plan and even reason. For example, in this study, we show that DRC agents can learn to evaluate and revise plans. Recently, DeepSeek-R1, an LLM with reasoning capabilities primarily trained via RL, has demonstrated similar self- cof-ccoreetton behavior in its reasoning, referred to as 'aha moments' (Guo et al. 2025). As such, we believe that understanding the mechanisms behind these emergent capabilities in RL agents is highly important.
2 BACKGROUND
2.1 PLANNING IN REINFORCEMENT LEARNING
Planning has many meanings in RL, encompassing algorithms utilizing environment models during training (Sutton 1991) or at decision time (Silver et al. [016)] [Chung et al.) 2024a). In this work, we study whether an RL agent is specifically performing decision-time planning. Henceforth, we use 'planning' and 'decision-time planning' interchangeably. In past work, an agent is considered to be planning in this sense engages with an (explicit) world model to select actions associated with the best predicted long-term consequences (Hamrick et al. 2020 Chung al. 2024a). An example MuZero (Schrittwieser et al., 2020). which applies a planning algorithm called Monte Carlo Tree Search (Coulom| 2006) to model of its environment to select actions associated with the best long- run consequences. Other similar agents are VPN (Oh et al., 2017), IBP (Pascanu et al. 2017). I2A (Racanière al. 2017), MCTSNet (Guez et al. 2018b), and Thinker (Chung al. 2024a) agents.
2

---

Published as a conference paper at ICLR 2025
By definition, model-free RL agents lack explicit world model. This makes it difficult to reuse past definitions of planning that presume that an explicit world model is available. Thus, for the pur- poses of this work, provide pragmatic characterization of planning that we as a foundation for investigating whether the model-free agent studied in this paper performs planning.
We consider plans be sequences of potential future actions. We characterize agent planning if it selects actions to perform by considering plans that it formulates and evaluates based on predicted future consequences. This is similar to how planning is understood in neuroscience (Mattar & Lengyel 2022). It also mirrors model-based definitions of planning but relaxes the requirement for explicit world model to the requirement that an agent predict consequences of future actions, regardless of the method used. We discuss our characterization further in Appendix [..]] For an agent to plan under our characterization, it must: (i) form plans, (ii) evaluate plans by predicting their consequences, and (iii) be influenced by these plans when acting.
2.2 SOKOBAN
Sokoban is an episodic, fully-observable, deterministic environ- ment in which an agent moves around walls in an 8x8 grid to push four boxes onto four targets. When an agent moves up/down//eft/right into a square containing a box, the box is pushed up/doww//left/iiht, Sokoban levels let agents perform actions with irreversible, negative, long-run consequences (moving boxes so the puzzle is unsolvable). Sokoban is thus difficult - it is PSPACE- complete (Culberson] 1997) and a common benchmark for study- ing planning (Racaniere al. 2017: Guez al.) 2019 Hamrick et al. 2020). We study a version of Sokoban where the agent ob- serves a symbolic representation ia 8x8887 of the environment. For ease of inspection, all figures are presented as pixel representa-
+
(a) Pixel
(b) Symbolic
Figure 2: Pixel and symbolic
representations of Sokoban board.
tions. Segureecompare these two representations. Appendix E.2
further explains this environment.
2.3 DEEP REPEATED CONVLSTM (DRC) AGENTS
Deep Repeated ConvLSTM (DRC) agents Guez et al. 2019) model-free agents based Con- vLSTMs that perform multiple computational ticks per time step. ConvLSTMs (Shi et al. 2015) are LSTMs (Hochreiter & Schmidhuber 1997) that utilize 3D hidden states and convolutional connec- tions. At each time step I, a DRC agent passes an observation I₁ through a convolutional encoder to generate encoding i IR Ho xG., This is then processed by D ConvLSTM layers. At time the d-th ConvLSTM has cell state of E R xW,xG Unlike standard recurrent networks which perform single tick of recurrent computation per time step. DRC agents perform N ticks of recur- rent computation per step. Guez et al. (2019) show these internal ticks improve the performance and generalization of DRC agents. Appendix provides further architectural details.
DRC agents behave in manner that suggests they internally engage in decision-timg time planning. For instance, DRC agents outperform model-based agents like MuZero (Schritwieser et al. 2020) in Sokoban (Chung et al. 2024b), and exhibit improved performance when given extra test-time com- pute (Taffeqque et al. 2024). This raises a question: do DRC agents genuinely learn to internally perform planning, or is their planning-like behavior mercly a result of complex learned heuristies?
In this paper. we investigate whether a Bokoban-playing DRC agent internally plans. The agent we study has D = 3 ConvLSTM layers and performs N internal ticks per step. The agent's encoder and ConvLSTMs have 32 channels (Ga = 32) and utilize kernels of size with a single layer of input zero padding. Thus, all cell states share Sokoban's 's spatial dimensions (H,, Wa 8). The agent is trained for 250 million transitions on the unfiltered Boxoban training set (Guez et al.) 20T8al (al using similar training setup as Guez al. (209)) explained in Appendix EE.4, Appendix E.5 shows that, consistent with Guez et al. (2019), this agent exhibits planning-like like behavior.
2.4 CONCEPT-BASER INTERPRETABILIT
Concept-based interpretability is an approach to explaining neural network behavior that involves identifying which concepts network internally represents (Kim al. 2018). A concept generally
3

---

Published as a conference paper at ICLR 2025
understood unit of knowledge (Schut et al. 2023). In this paper, specifically consider 'multi-class' concepts, which can formally be defined as mappings from input states (or parts of input states) to some fixed classes. That is, multi-class concepts correspond to interpretable, discrete features, and map inputs to classes of that concept. For instance, a multi-class Sokoban concept might be 'the number of empty targets'. This concept would map any observed Sokoban board to a class in {ONE, TWO, THREE, FOUR} depending on the number of remaining empty targets in Zt.-
We focus on concepts networks represent linearly (Mikolov et al., 2013 To check if a network linearly represents concepts, we use linear probes. These are linear classifiers trained to predict concept classes assigned to imputs using the associated network activations (Alain & Bengio 2016). As linear classifiers, linear probes compute logits 4 = w g for each class K by projecting network activations g E R along a class-specific vector w2 E Belinkovv (0022) explains probes further.
3 METHODOLOGY
3.1 PROCEDURE FOR INVESTIGATING MODEL-FREE PLANNING
In Section 2.1 we characterized planning requiring that agent (i) formulate plans, (ii) evaluate the consequences of these plans, and (ii) be guided by these plans when selecting actions. If an agent learns to plan, expect Mlaninng-relevan concepts emerge in its internal representations to meet the first condition. These concepts ought to reflect the agent's 's plan, and so should corre- spond potential future actions, of plan evaluation - such as avoiding or improving bad plans - should exist to satisfy the second condition. Lastly, to fulfill the third condition, the plan must causally influence the agent's behavior. To determine if an agent exhibits these three properties, we follow the procedure outlined below:
their likely environmental effects. Additionally, evidence
1. Probe for Concept Representations. First, we identify group of environment-speciin concepts that could be instrumentally useful for planning. We then use linear probes to establish whether these concepts are being (linearly) represented by the agent (Section {
2. Investigate Plan Formation. Next, focus on gathering qualitative evidence of the agent forming plans based on the planning-relevan concepts probed for in the previous step, and evidence of the agent evaluating and refining these plans (Section 5)
3. Confirm Behavioral Dependence. Finally, we confirm that these internal plans influence the agent's behavior. For instance, we show that the agent can be steered to form and execute desired plans by intervening plan representations within the network (Section
3.2 PLANNING- CONCEPTS IN SOKOBAN
To apply this procedure, we must specify concepts we expect the agent to plan with. Sokoban has a grid-based structure with localized transition dynamics, i.e., the future state of a square is determined by the current state of its neighbors. This makes spatially local concepts (i.e., concepts related to individual connected squares) more natural for planning than spatially global concepts (i.e., representations of the whole board). We thus claim that an agent that learns to plan in Sokoban may do so by encoding concepts localized to individual squares. We call these 'square-level' concepts. Such concepts seem natural for DRC agents as the 3D structure of ConvLSTMS allows for spatial correspondence between the Sokoban grid and agent hidden states. We focus multi-class square- level concepts which, as explained further in Appendix E.6 map grid squares to concept classes.
We hypothesize that the agent will plan using the following square-level, multi-class concepts:
Agent Approach Direction (Ca): For a given square, this concept encodes whether the agent will move onto the square in the future. If so, it also encodes the direction from which the agent will move onto the square the next time the agent moves onto it.
Box Push Direction (Cn): For given square, this concept encodes whether box will be pushed off the square in the future. If so, it also encodes the direction in which the next box pushed off this square will be pushed.
Figure 3 Dillustrates the classes assigned to each square of a Sokoban board by these concepts over six transitions near the end of episode. Both concepts map each grid square of the agent's observed
4

---

Published as a conference paper at ICLR 2025
(a) Agent Approach Direction CA
(b) Box Push Direction Cb
Figure 3: Examples of the classes assigned to the squares of a Sokoban board over 6 transitions (from left to right) by the concepts 'Agent Approach Direction' (CA) and 'Box Push Direction' (Cp). An arrow corresponds to the assignment of the associated directional class. The lack of an arrow in a square indicates the assignment of the class NEVER.
Sokoban board to the classes (UP, DOWN, LEFT, RIGHT, NEVER). The directional classes corre- spond to the agent's s movement directions. If the next time the agent steps onto specific square, the agent steps onto that square from the left, the concept C^ would map this square to the class LEFT. If the next time the agent pushes box off of specific square, the box is pushed to the left, the concept C3 would map this square to the class LEFT. Finally, the class NEVER corresponds to the agent not stepping onto pushing box off of square again for the remainder of the episode.
Both concepts depend on the agent's behavior: we can only determine the classes these concepts map grid squares to after observing the agent's behavior over the entire episode. Furthermore, as shown in Figure 3 the classes squares are mapped to will change at every transition. Once an agent steps onto square, the classes assigned to that square will update represent the agent's future interactions with that square. We investigate alternate concepts in Appendices) D.4 and D.5
4 PROBING FOR CONCEPT REPRESENTATIONS
We now perform the first step of our analysis: determining whether the agent internally represents the concepts that we hypothesize uses to internally form and evaluate plans.
4.1 EXPERIMENT DETAILS
Specifically, linear probes to determine if the agent represents (a) CA. the agent's future movement onto squares, and (b) Cn. the future directions boxes are pushed off of squares. We train linear probes that take input the agent's cell state activations after the final of the three computational ticks performed each step. We train separate probes for the agent's three layers.
We hypothesize the agent will learn spatial bijection between its cell state and the Sokoban grid. Thus, when predicting CA and Cp. at each location (z, y). our probes receive as input cell state activations centered on (x, y). We train both 1xl probes (which take as input just the activations at (z, v)) and 3x3 probes (which take as input the 3x3 patch of activations around (z, y)). These probes have 160 and 1440 parameters, so are unlikely to overfit. We consider larger probes in Appendix D.3
Each probe trained using logistic regression with the AdamW optimizer, and five unique initializa- tion seeds. The training dataset is generated by running the agent for 3000 episodes on levels from the Boxoban unfiltered training dataset (Guez et al. 2018a). We test probes on a test set of transitions generated by running the agent for 1000 episodes on levels from the Boxoban unfiltered validation dataset. Further probe training details are given in Appendix [.I] We compare the performance of all probes to baseline probes that receive the raw observation 2; input. This comparison aims to assess the extent to which probes' abilities to predict concept classes are due to these concepts being internally represented by the agent rather than the probes learning how do so themselves.
5

---

Published as a conference paper at ICLR 2025
6.-
Probe Type
0.4
0.4
0.2
Layer
Layer
Layer
Layer
Layer
(a) CA
(b) CB
Figure 4: Macro Fls achieved by probes when predicting CA and CB using the cell state at cach layer, or, for the baseline probes, using the observation. Error bars show +1 standard deviation.
CA
Co
CA
Cg
CA
Cp
(a)
(c)
Figure 5: Examples of internal plans computed by the agent. An internal plan corresponds to the agent's 's combined square-level representations of C'A and Cg. That is, an internal plan corresponds to the classes the agent represents these concepts as mapping squares of observed boards to. These internal plans are decoded from the agent's final layer cell state by a 1x1 probe. Teal and purple arrows respectively indicate the agent expects to next step on to, push box off, a square in the associated direction. No arrow indicates the agent does not plan to step onto, or push a box off, square again. Further examples of internal plans given in Figures TO 11 and 12 in Appendix A.
4.2 RESULTS
In many Sokoban boards, the agent will never move onto, nor push box off, large number of squares. As result, many squares assigned the label NEVER for both concepts in probing datasets, leading to class imbalance. We therefore evaluate probe performance using macro scores in place of accuracy. Figure[4shows the macro F1 scores achieved by probes trained predict the classes assigned to Sokoban squares by CA and Cg- The probes that predict these concepts using the agent's cell state activations vastly outperform the baseline, implying the agent linearly represents CA and Cp. This aligns with past work finding linear concept representations in many different networks (Nanda et al. 2023 McGrath al. 2022 Zou al. 2023).
Figure 4 confirms that the agent represents square-level concepts at localized positions of its Con- vLSTM cells opposed distributing representations across adjacent positions. This evidenced by the minimal improvement in performance when moving from a 1x1 probe to 3x3 probe, com- pared the significant improvement in baseline performance. We thus focus on 1x1 probes for the remainder of this paper. Interestingly, Figure 4 perform slightly better than probes at layer there is little variation performance across layers. This indicates that the concepts are represented across all layers. We thus hypothesize that the agent engaged in iterative computation (Jastrzebski et al. 2018), whereby it refines plans across layers.
also shows that while probes at layer 2 generally
INVESTIGATING PLAN FORMATION
In this section, we now provide qualitative evidence that the agent forms plans by searching forward from the boxes and backward from the targets, and that the agent develops, evaluates, and adapts plans parallel. In this section, we primarily focus on descriptive explanations of how the agent forms plans and the general shape of the plans. We defer more conclusive evidence in the form of intervening on the agent's 's plan formation process to steer the agent's 's behavior to the next section.
Previously, demonstrated that the agent encodes (at least) two Maaninggreleeann concepts: CA and Cg. These concepts represent predictions regarding how the agent will act when moving onto given square in the future, and how the environment specifically, the locations of boxes will

---

Published as a conference paper at ICLR 2025
be affected by these actions. We thus posit that the agent's representations of these concepts when looked at holistically, over the entire board - will collectively constitute a plan that the agent forms and adapts. For example, in Figure [ we visualize the agent's 's representations of CB and CA over entire Sokoban boards, as decoded from the agent's cell state by a 1x1 prohe in different levels. Three observations can be made from Figure (a) the arrows, which indicate the direction the agent expects to move or push boxes, tend to be connected and trace path: (b) the arrows tend to connect boxes to specific targets; (c) the arrows collectively form a plan which corresponds to solving the level. In Appendix A we visualize the agent's 's plan across layers, and show that, while the agent's 's plans often contains flaws (like the lack of one necessary arrow in Figure [c).. they usually consist of connected paths for the agent to follow and connected routes linking boxes and targets.
natural question then arises: how does the agent form plans? To answer this, we direct attention to Figure | Figure [ visualizes the agent's plans in terms of CB (e.g. the routes the agent plans to push boxes) over the initial steps (A-C) and internal ticks (D-E) of episodes. As can be seen in Figure [ the agent forms plans iteratively. Interestingly, the agent appears to form plans iteratively by searching forward from boxes as illustrated in Figure [C) and backward from targets - as illustrated in Figure [T). That the agent seems to plan via bidirectional search which is known to be especially efficient when it is applicable (Russell & Norvig, 2010) may explain why [uuez] al. (2019) found DRC agents to rival specialized planning architectures reliant on forward search. Indeed, as shown in Fi), the agent seems to utilize a form of parallelized bidirectional search
whereby extends multiple plans simultancously. Appendices [A2.33 A.2.4 and A.2.5 respectively contain further instances of the agent appearing to utilize forward, backward, and parallel search.
However, recall that, in Section 2.1 we characterized planning as requiring an agent to evaluate the plans it considers. Evidence suggestive of the agent evaluating plans can be seen in Figure [T A)- (B). Figures Figures[[A) [B), show examples in which the agent appears to (1) formulate a naive plan, (2) evaluate it, and then, upon realizing that it is infeasible or could be improved, (3) adapt its plan accordingly. For instance, in Figure [)), the agent changes the targets it plans to push different boxes towards. This is suggestive of the agent using an evaluative search algorithm when forming plans. Appendices I.2.I and A.2.2 contain further examples of the agent seeming to evaluate plans and either plan to push a box longer route, change which boxes it plans to push to which targets.
Further evidence of the agent planning via an itera- tive search algorithm can be seen in Figure 6 For Figure [6] we forced the agent to remain stationary for 5 steps prior to acting in 1000 episodes. These 5 "thinking steps' give the agent 1.5 internal ticks of extra test-time compute. Figure r the macro
0.9
Macro F1
0.8
0.7
Fl when using ixl probes to decode C'A and C3
0.6
from the agent's final layer cell state at each of the 15 extra internal ticks, averaged over 1000 episodes. Clearly, the macro F1 improves with the number of ticks. Since the concepts are predictions of future behavior, we can see the predictions of our probes at any tick as being the agent's internal plan at that tick. We can then see the corresponding macro Fl as reflecting the quality of the agent's plan at that tick. Figure [ shows that, as would be expected if the agent planned via an iterative search, the agent's s plans iteratively improve when given extra compute. Appendix 1 shows test-time plan refinement oc- curs at all layers. Appendix A.3.2 provides evidence that it is a consequence of the agent searching deeper. Appendix C.2 shows that this 'test-time plan refinement capability' arises early in training.
3
6
9
12
15
Internal Tick
CA
CB
Figure 6: Macro Fl when using 1xl probes to decode CA and Cp from the agent's final layer cell state each of the additional 15 internal ticks performed by the agent when the agent is given 5 'thinking steps', aver- aged over 1000 episodes.
When considered with the agent's planning-like behavior, the above evidence indicates the agent uses its representatiions of CA and C3 for search-based planning. Further evidence of this is given in Appendices
show examples of the agent planning in out- caat-f-ddittrbutionn levels, such as levels in which the agent itself not present (Appendix (A2.6)). levels with additional boxes and targets (Appendix A.2.7). and levels in which walls appear and disappea (Appendices A.2.8. A.2.9). These examples suggest the agent's 's ability to adapt and generalize benefits of model-based planning Guez et al. (2019) show DRC agents possess relate to its representations of CA and Cg.
7

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer 3
Trained (9)
Random (%)
Trained (%)
Random (%)
Trained (%)
Random (%)
AS BS
94.6 (+55))
33.7 32.7)
90.1 (#119)
29.8 36.8)
98.8 (=000)
27.8 (#37.9)
56.2 (1:1.4
31.5 (1 13.9)
72.7 (1111)
30.9 (125.8)
80.6 (1:2.4)
4.1 (1:5.4)
Table 1: Success rates (%) when intervening on each layer using representations from trained and randomly initialized probes. AS and BS refer to 'Agent-Shortcut" and 'Box-Shorteut" interventions. Success rates are averaged over interventions performed. We report +:1 standard deviations.
6 INVESTIGATING THE ROLE OF PLANS
So far, we have shown that the DRC agent represents C' and CB (Section 4 and that uses these representations to form internal plans (Section 55. We now conclude our analysis by showing that these representations are causally responsible for the agent's behavior. Specifically, we: (I) use these representations intervene the agent to force it to form and execute specific plans, and (2) show that these representations emerge concurrently with planning-like like behavior during training.
6.1 INTERVENING ON AGENT PLANS
First, we show we can intervene on the agent's activations alter its behavior over entire episodes. Our interventions involve adding concept vectors learned by probes to the agent's 's activations to force represent concepts in specific ways. We then observe the causal effect of our interventions on the agent's 's behavior. Recall that 1xl probe projects activations along vector wa 32 to compute a logit for class of some multi-class concept C. We thus encourage the agent to represent square (zr, y) as class le for concept C by adding We to position (x, y) of the agent's cell state Ja.y.
Lewy 9z.y wk
If the agent indeed uses CA and Cg for planning, altering the agent's square-level representations of these concepts ought to modify its internal plan and, subsequently, its long-term behavior.
We intervene two sets of handcrafted levels: 'Agent-Shorteut" and 'Box-Shorteut' levels. These sets of levels are characterized by, in each level, there existing two plans: short plan and a long plan. The plans are similar, but differ in lengths. The agent by default follows the optimal (short) plan. We show interventions cause it to instead form and execute the suboptimal (long) plan.
In 'Agent-Shorie ShenttShortcut levels all boxes and targets are in one region of the board, and the agent can follow either long short path to this region. In these levels, intervene using vectors learned by probes trained to predict C'A to steer the agent to plan to move along the long path. Our intervention consists of two parts. We add the vector for NEVER cell state positions on the short path. We call this the 'short-route" intervention. We also add the vector for the direction which would lead the agent to move onto the first square of the long path the appropriate cell state position. We call this the 'directional' intervention. An Agent-Shortcut intervention is illustrated in Figure 7b
"Box- Gox-Shortcut' levels are ppecially-desinnee levels in which three boxes are adjacent to targets and a fourth box is not. The final box can be pushed a long or short route to target. In these levels, we intervene using vectors learned by probes trained to predict CB to steer the agent to push this box the long route. Our intervention again consists of two parts. We add the vector for NEVER to cell positions on the short route We also add the directional representation which would encourage the agent to push the box the longer route to the box's 's initial position. We again call these the "short-route' and "directional" interventions. A Box-Shorttee Shortcut intervention illustrated in Figure 8b
We intervenc 200 levels of each type. We created 25 levels of each type and then generated 8 versions of each level by applying vertical reflection and 90", 180%, and 270' rotations. In all levels, repeat the 'short-route' intervention every step but repeat the 'directional' intervention only until the agent moves onto, or pushes the box off, the corresponding square.
We perform our interventions the agent's 's cell state each layer. An intervention is considered successful if it causes the agent to solve the level in the desired suboptimal way. As a baseline, we in- tervene using representations from randomly initialized probes. For comparability, we scale random
8

---

Published as a conference paper at ICLR 2025
(a) Plan without intervention
(b) Intervention
(c) Plan with intervention
Figure 7: An Agent-Shortcut intervention and its effect on the agent's plan formulated in terms of CA: (a) the agent's plan after 4 steps without the intervention, (b) the initial state of the level and the intervention, and (c) the agent's plan after 4 steps with the intervention. The 'short-route" inter- vention adds the representation of NEVER for CA to positions with white crosses. The 'directional' intervention adds the representation of DOWN for CA to the position with the white arrow.
(a) Plan without intervention
(b) Intervention
(c) Plan with intervention
Figure 8: A Box-Shorteu intervention and its effect on the agent's plan as formulated in terms of Cg: (a) the agent's plan after steps without the intervention, (b) the initial state of the level and the intervention, and (c) the agent's 's plan after 4 steps with the intervention. The 'short-route" inter- vention adds the representation of NEVER for CB to positions with white crosses. The 'directional' intervention adds the representation of RIGHT for C3 to the position with the white arrow.
probe representations that the norms of both the random and trained probes are similar. Success rates are averaged over interventions performed with five independently trained or initialized probes.
Table D shows intervention success rates. At all layers, Agent-Shortcut interventions are successful, While the success rate of Box-Shorteut interventions is lower, it remains high relative to the baseline of interventions using random probes. These results in-
Additional Levels Solved
Concept
Ca
dicate that the agent's is representations of C' and CR
Ca
influence its behavior in the way that would be ex-
pected if it used them for planning. Figures 7 and provide examples of the effect interventions on the agent's internal plans. These examples suggest the agent not only behaves differently following the interventions, but does so due to forming a different plan. We show more examples of interventions alter- ing the agent's 's internal plans in Appendix B.I Ap- (endix/B.2Jreportt success rates when using an inter- vention scaling factor and varying the squares inter- vened on. Appendix B.3 reports success rates when intervening to encourage optimal behavior in levels which the agent by default cannot solve. These extra experiments further indicate that the agent's 's repre- sentations of CA and C₁ influence its behavior expected.
0.2
0.4
0.6
0.8
Figure 9: The relationship between the per- centage of extra levels, of medium difficulty, solved when an agent given steps to 'think', and macro F1 score of probes when predicting C'A (blue) and CB (orange) from the agent's 's final layer cell state. Each point corresponds to these quantities calculated for single checkpoint.
6.2 INVESTIGATING THE EMERGENCE OF PLANNING DURING TRAINING
Finally, we show that the emergence of the agent's 's representations of C'A and C3 during training coincides with the agent beginning to exhibit planning-like like behavior. This indicates that the agent indeed uses its representations of C₂ and Cp for planning. Specifically, we show the emergence of these representations coincides with the emergence of the agent's ability benefit from extra test-

---

Published as a conference paper at ICLR 2025
time compute (Guez al. 2019 Taufeeque al. 2024). In particular, collect checkpoints every million transitions for the first 50 million transitions of training. For every checkpoint, we measure two quantities: (i) the macro F1 score of 1xl probes trained to decode the concepts Cx and Cp given the agent's cell state (following the procedure described Section and (ii) the number of additional levels out of 1000 medium-difficulty levels from the Boxoban dataset (Guez et al. (018a) the agent can solve when given extra test-time compute by forcing the agent to remain stationary for the first 5 steps of an episode. Figure plots these quantities against each other and shows strong correlation between them. This implies the agent only reliably begins to exhibit planning- like behavior benefiting from extra test-time compute once its final layer representations of CA and Cp are sufficiently formed. Appendix C.3 shows that this holds for its representations of CA and Cp at layers. Appendix Chosshwws the agent begins to perform better with extra compute at similar point in training as to when it can use this compute to refine its plans.
ADDITIONAL RESULTS
In the Appendix, we include interesting results that we lacked space include in the main text. Appendices F provides evidence of DRC agents planning both without internal ticks, and with ad- ditional internal ticks. Appendix H provides evidence of DRC agents planning in different environment: Mini PacMan. Finally, [ provides evidence of a ResNet (He et al. (016) agent planning Sokoban. However, the question of whether generic agent learn to plan in generic environment remains unanswered.
RELATED WORK
Past work has investigated concept representations learned by game-playing agents (Schut et al.,
2023
McGrath et al. 2022; Hammersborg & Strümke) 2022
2023
Lovering et al. 2022 Mini
et al. 2023) and language models (Li et al. [2023]
et al. 2023
Karvonen 2024 Ivanitskiy
al. 2024). While past work has focused primarily on whether networks internally represent spe-
cific concepts, we study concept representations for the broader purpose of determining if an agent possesses a capability planning. An exception work by Jenner et al. (2024), which finds evidence of look-ahead in a chess-playing agent, but does not investigate wider capacity to 'plan'.
Concept-based interpretability is not the only approach to interpreting agents. An alternative is ittribttionbbase interpretability. This involves determining usually via saliency maps which features an agent's observation influence its behavior (Weitkamp et al., 2019 [yer et al. [018 Puri et al. 2020 Greydanus et al. 2018 Hilton al. 2020). Attribution- based methods
not used here as they can depend on subjective interpretation (Atrey et al.) (020). Another approach, example- based interpretability, explains agent behavior by providing examples of illustrative trajectories or transitions (Rupprecht et al. 2020) Sequeira & Gervasio 2020) Deshmukh et al. 2023) /ahavy et al.) 2016). Due to not studying model internals, example-bar
methods were ill-suited for this paper.
Finally, this paper contributes to recent work investigating the emergence of reasoning capabilities in neural networks (Wei et al. 2022 Kojima al. 2022 Lehnert et al.) 2024 Nye et al. 2021 Wang et al. 2024). However, unlike this paper in which we provide evidence of an agent internally per- forming planning, most work thus far has focused on providing behavioral evidence of reasoning.
An exception to this is work by BBrinkmann et al. (2024) in which an algorithm learned by trans- former trained on simple symbolic reasoning task is reverse-engineered. However, [rinkmaan) et al. (024) focus on much simpler form of reasoning than planning as considered in this paper.
FUTURE WORK
In this paper, we proposed methodology for investigating model-free planning and used it to pro- vide the first non- behavvovvioral evidence of learned planning in model-free agent. Future work may extend our investigation other RL agents, and other environments. In particular, it would be help- ful to better understand the role of different training factors, e.g., model architecture, environment dynamics in the emergence of planning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We are thankful Erik Jenner and Joschka Braun for providing thoughtful feedback on the draft. For much of the duration of this work, TB was supported by the Cambridge Trust and Good Ventures Foundation. UA was supported by OpenPhil AI Fellowship and Vitalik Buterin Fellowship in Al Existential Safety. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (www.csd3.cg ceassdd.cam.c.uu cam.ac.uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EPPT002159/1)) and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.ukk
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.016.44, 2016.
Jacob Andreas. Language models, world models, and human bodidinu 2024. URL https:
//lingo.csal csail mmittedu/// htc////oogggo woog/wwrlllo www.d_rrddooe rd_modee_l__ s
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory explanatory: Counterfactual analysis of saliency maps for deep reinforcement leamionn ing Representations, 2020.
learning. In International Conference on Learn-
net/
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguistics, 48(1):207-219, 2022.
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mech- anistic analysis transformer trained on a symbolic multi-step reasoning task. arXiv preprint arXiv:2402.11917 11917, 2024.
Stephen Chung. Ivan Anokhin, and David Krueger. Thinker: learning to plan and act. Advances in Neural Information Processing Systems, 36, 2024a.
Stephen Chung, Scott Niekum, and David Krueger. Predicting future actions of reinforcement
learning agents. In First Reinforcement Learning Safety Workshop, 2024b. URL https: ///ppn/eview.neeeeeeee wtw.eewwweeeoooroooeeoeeee SohRRnhh780
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Joseph C. Culberson. Sokoban is pspace-complete 1997.
URL https://api.
semant Sommanicscholarre www.llarrrr//oooossSD..... :1:14143:666
Shripad Vilasrao Deshmukh, Arpan Dasgupta, Balaji Krishnamurthy, Nan Jiang, Chirag Agarwal,
Georgios Theocharous, and Jayakumar Subramanian. Explaining RL decisions with trajectories.
In The Eleventh International Conference Learning Representations, 2023. URL https:
7/ //openzeview. http://ww.......mmmmmdeeseeegooe
Ashley Edwards, Laura Downs, and James Davidson. Forward-backward reinforcement learn- ing. arXiv preprint arXiv: 10227 2018
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018
Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. Advances Neural Information Processing Systems, 32, 2019.
Anirudh Goyal. Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for efficient reinforcement learning. arXiv preprint / 003490037, 2018.

---

Published as a conference paper at ICLR 2025
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference on machine learning, Pp. 1792-1801. PMLR, 2018.
Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane We- ber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, Timothy Lillicrap, and Victor Valdes. An investigation of model-free planning: boxoban levels.
http:///tttbboommmeemmmmdddddbdd 2018a
Arthur Guez, Théophane Weber, loannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wier- stra, Rémi Munos, and David Silver. Learning search with metsnets. In International confer- ence on machine learming. pp. 1831-1831. PMLR, 2018b.
Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, et al. An investigation of model-free planning. In International Conference Machine Learning, pp. 2464-2473, PMLR, 2019.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501. 12948, 2025.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems, 31, 2018.
Patrik Hammersborg and Inga Strümke. Reinforcement learning in an adaptable chess environment for detecting human-understandable concepts. arXiv preprint arXiv:2211.05500, 2022.
Patrik Hammersborg and Inga Strümke. Information based explanation methods for deep learning agents-with applications on large open-source chess models. arXiv preprint arXiv:2309.0977 2023.
Jessica Hamrick, Abram Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither- spoon, Thomas Anthony, Lars Buesing, Petar Veličkovié, and Théophane Weber. On the role of planning in model-basee deep reinforcement learning. arXiv preprint arXiv:201 20V::2011.040.11 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
James A Hendler, Austin Tate, and Mark Drummond. Ai planning: Systems and techniques. Al magazine, 11(2):61611, 1990.
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. doi: 1/02stii/istiil.000022.. https://distill http://ssillllu///000ouuuueseans
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Michael Ivanitskiy, Alexander F Spies, Tilman Räuker, Guillaume Corlouer, Christopher Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, et al. Linearly structured world representations in maze-solving transformers. In Proceedings of UniReps: the Först Workshop on Unifying Representations in Neural Models, pp. 133-143, PMLR, 2024.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 144-150, 2018.
Stanistaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evi- dence of learned look-ahead a chess- aless-playing neural network. arXiv preprint arXiv:2406.008: 00877, 2024.
12

---

Published as a conference paper at ICLR 2025
Kristopher T. Jensen, Guillaume Hennequin, and Marcelo G. Mattar. A recurrent network model of planning explains hippocampal replay and human behavior. Nature Neuroscience, 23(7):1340- 1348, Jul 2024. ISSN 1546-1726. doi: 10.10388441533-044-011333 org/ 10.
URL https://doi.
027-7177577
Hermann Kaindl and Gerhard Kainz. Bidirectional heuristic search reconsidered. Journal of Artifi- cial Intelligence Research, 7:283-317, 1997.
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint arXiv:2403. 15498, 2024.
Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors PMLR, 2018.
(tcav). In International conference on machine learning, pp.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA,, USA, 2015.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2223 2022.
Richard E. Korf. Planning as search: A quantitative approach. Artificial Intelligence, 33(1):65-88, 1987. ISSN 0004-3702. doi: https://doi.org/100 htt.://////0.1100000000----03010 1016/00044330008779000555
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, pp. 5618-5627. PMLR, 2020.
Su Young Lee, Choi Sungik, and Sae- Young Chung. Sample-efficient deep reinforcement learning via episodic backward update. Advances in neural information processing systems, 32, 2019.
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mevay, Michael Rabbat, and Yuandong Tian. Be- yond a#: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083, 2024.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence Learning Representations, 2019. URL http::://ppennrreee..eeett net wareeeffoumma
Bkq6RicqY7
Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. Evaluation beyond task performance: analyzing concepts in alphazero in hex. Advances in Neural Informa- fion Processing Systems, 35:25992- 2022.
Marcelo G. Mattar and Máté Lengyel. Planning in the brain. Neuron, 1 110(6):914-334, 2022. ISSN 0896-6273. doi: http://ddaaayyy//uooo/..oooooo.0 URL https::/www:
cec.encedrrett.com//ea.. com http:///cc...........././///////
Thomas McGrath, Andrei Kapishnikov. Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47)ee22666551 19, 2022.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.). Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL URLhttts: htwps: http::///aaaaaatooooooooo
org/ /N1 1- 1090
Kevin J. Miller, Matthew M. Botvinick, and Carlos D. Brody. Dorsal hippocampus contributes to model-based planning. Nature Neuroscience, 20(): 1276, Sep 2017. ISSN 1546-1722 doi: N0.1038/nn.445 4613. URL.ttts: http:pss://ooi..g///1..00..
4613
13

---

Published as a conference paper at ICLR 2025
Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexan- der Matt Turner. Understanding and controlling a maze-solving policy network. arXiv preprint 0rXiv:2310.08043: 2023.
Andrew W. Moore and Christopher G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning,, 13(1):103-130, 1993. ISSN 0885-6125. doi: 10. 10.1023/A:1022235112222
URLhttps::/ /tttpps://do.... http::/:///oii.o///////// www.oo/g//.... 023/:: 0020000000002230111131131 022233662263222
Neel Nanda, Andrew Lee, and Martin Wattenberg, Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NIP,, pp. 16-30, 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 1exiv:21/2.001/ 2021.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. Advances in neural information processing systems, 30, 2017.
Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David Reichert, Théophane Weber, Daan Wierstra, and Peter Battaglia. Learning model-based planning from scratch. arXiv preprint /7Xiv: 2017.
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2020.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination-augmennte agents for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Jonathan Richens and Tom Everitt. Robust agents learn causal world models. arXiv preprint #rXh::2402.10877 2024.
Christian Rupprecht, Cyril Ibrahim, and Christopher J Pal. Finding and visualizing weaknesses of deep reinforcement learning agents. In International Conference Learning Representations, 2020.
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, edition, 2010.
Julian Schrittwesser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588
(7839):604-609, December 2020. ISSN 1476-4687. doi: 10.
URL
http::/dx..oe otw/10..r0110000 1038/1 s 41846-86---- 020-03051-1
Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridg- ing the human-ai knowledge gap: Concept discovery and transfer in alphazero. arXiv preprint arXiv:2310. 164 2023.
Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable reinforcement learn- ing: Understanding agents' capabilities and limitations. Artificial Intelligence, 288: 103367, 2020.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, 28, 2015.
Yaron Shoham and Gal Elidan. Solving sokoban with Borward-backwer reinforcement learning, 2021.
14

---

Published as a conference paper at ICLR 2025
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that mas- ters chess, shogi, and go through self- Science, 362(6119)11440-1144 2018. doi: 10.1126/ science.aar64044. URL. https: / www.ww..ccccc.o................. sci ence wwt. http:///do//bbs//ooog 1126/ ence. aar6404)
Nathan R Sturtevant, Shahaf Shperberg, Ariel Felner, and Jingwei Chen. Predicting the effectiveness of bidirectional heuristic search. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pp. 281-2900 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Mohammad Taufeeque, Philip Quirke, Maximilian Li, Chris Cundy, Aaron David Tucker, Adam Gleave, and Adrià Garriga-Alonso, Planning in recurrent neural network that plays sokoban. arXiv preprint ar arXiv:2407.14421, 2024.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in rein- forcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405. 15001, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07660 2022.
Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep rein- forcement learning for atari games. In Artificial Intelligence: 30th Benelux Conference, BNAIC 2018, 's-Hertogenboshh, The Netherlands, November 8-9, 2018, Revised Selected Papers 30, pp. 151-165. Springer, 2019.
Tom Zahavy, Nir Ben -rihem, and Shie Mannor. Graying the black box: Understanding dqns. In International conference on machine learning, pp. 1899-1908. PMLR, 2016.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01400 2023.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Additional Investigations of Internal Planning
A.I
Further Examples of Internal Plans
17
A.2
Further Examples of Internal Plan Formation
21
A.3
Further Results Regarding Iterative Plan Refinement
35
B
Additional Intervention Results
39
B.1
Additional Examples of Interventions
39
B.2
Additional Intervention Experiments: Further Agent-Shorteun and Box -horttutt
Intervention Experiments
B.3
Additional Intervention Experiments: Intervening in a New Set of Levels To
Encourage Optimal Behavior
48
C
Additional Training- Time Interpretability Results
49
C.1
Investigating the Emergence of Planning-Relevant Relevant Concept Representations Dur-
ing Training
51
Investigating The Emergence of Test- Time Plan Refinement During Training
51
Investigating the Co- Emergence of Planning-Relevant Relevant Concept Representations
and Planning-Liae Like Behavior During Training
31
Investigating the Co-Emergence of Test- Time Plan Refinement and Planning-
Like Behavior During Training
D
Additional Probing Results
54
D.1
Probe Training Details
54
D.2
Additional Probing Metrics
55
D.3
Probing Using Larger Probes
58
D.4
Probing For Alternatve Square- Level Concepts
58
D.5
Probing For Future Actions
60
E
Additional Background Material
61
Decision- Time Planning
62
E.2
Sokoban
................................
63
E.3
Deep Repeated ConvLSTM (DRC) Agent Architecture
63
1.4
DRC Agent Training Details
65
B.5
Behavioral Evidence of Planning Exhibited By The DRC Agent
65
E.6
Operationalizing Concepts
66
7.
Application of Methodology to Other Model-Free Architectures
66
F
Investigating Planning in DRC Agents of Different Sizes
67
Investigating Planning in a DRC(1,9) Agent
67
F2
Investigating Planning in a DRC(9,1) Agent
69
G
Investigating Planning a Different Architecture: ResNet
70
Investigating Planning in a Different Environment: Mini Pacman
76
Mini PacManl
76
H.2
Preliminary Probing Results
77
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL INVESTIGATIONS OF INTERNAL PLANNING
In Section 5 we provide evidence suggestive the agent possessing a search-based internal plan- ning mechanism. In this section, we now provide further complementary evidence regarding the agent's 's internal planning procedure. This section proceeds as follows:
Appendix A.
provides further examples of the agent's internal plan at all layers.
Appendix A
provides additional examples of the agent forming plans in manner sug-
gestive of a search-based planning algorithm.
provides additional investigations of the agent's ' ability to use extra test-time compute improve its plans.
Appendix A.
A.1 FURTHER EXAMPLES OF INTERNAL PLANS
In Figure 5 we provided examples of 'internal plans' formulated by the agent. We understood the agent's 's internal plans to consist of its internal representations, for each square its observed Sokoban board, of CA and CB. In Figure 5 all internal plans were decoded from the agent's final layer cell state. In this section we now provide additional examples of internal plans formulated by the agent decoded from its cell state at each layer.
Figures TO and 12 show internal plans decoded from the agent's 's first, second, and third layer cell states in many different levels. Specifically, Figure shows the agent's internal plans each layer at six transitions where the agent's internal plan as decoded from its first-layer cell state corresponds a complete plan to solve the respective level. Similarly, Figures [] and 12] show the agent's internal plan at each layer at six transitions where the agent's internal plan as respectively decoded from its second- and third-layer cell state correspond to a complete plan to solve the respective levels. We note that the observations we made regarding Figure [ hold here. That is, (1) the arrows tend form connected paths, (2) the agent's plans tend connect specific boxes to specific targets, and (3) the agent often forms complete plans to solve levels very early on in episodes.
Note, however, that the agent's plans in Figures TO 11 and 12often contain mistakes. This despite the illustrated transitions being selected such that the agent's & plan is correct in at least one layer. A few things can be noted about these mistakes. First, the agent's plans for box movements contain, on average, far fewer mistakes than the agent's plans for its own movements. Second, the mistakes in the agent's plan for its own movements are usually minor and consist of e.g. few arrows being wrong, but the overall 'shape' of the plan being correct. Third, the agent's mistakes when planning its own movements in the examples tend to be mistakes regarding how it can move when not pushing boxes. We think these observations suggest that the agent is primarily planning by constructing plans in terms of CB connecting boxes and targets, and then augmenting these plans with planned agent movements where needed.
At a high-level, we suspect that mistakes in the agent's plan are best seen as relating to intermediate steps of the agent's internal planning process. First, this is because many mistakes seems to be plans that the agent considers on its way to arriving at its final plan. This is because mistakes are almost always fixed in future transitions. Second, some mistakes seem to be temporarily added to the agent's 's otherwise-corred plan at specific layers. We believe these mistakes potentially relate to the fact that, as part of planning process, the agent sometimes considers variations on its plan.
17

---

Published as a conference paper at ICLR 2025
CA
Cs
CA
Cs
CA
Ca
(a) Example Layer
(b) Example
Layer
(c) Example 1 Layer
Cs
CA
(d) Example 2 Layer 1
(e) Example 2- Layer
(l) Example 2 Layer 3
CA
Cs
CA
CA
(g) Example 3 Layer
(h) Example Layer 2
ii) Example Layer 3
CΛ
Cg
C
Cn
Cg
(i) Example 4 Layer
(k) Example
Layer
(l) Example Layer 3
CA
C5
CA
C3
CA
C3
(m) Example 5 Layer
(n) Example 5 Layer 2
(o) Example 5 Layer
Ca
CA
(p) Example 6 Layer 1
(q) Example
Layer
(r) Example 6 Layer 3
Figure 10: Internal plans computed by the agent in 6 examples levels as decoded from the agent's
first layer cell state (IOal Iod 10g [0j
10m] and 10p).
second layer cell state {1bb] 10c 10h
10k IOn and 1Oqb third layer cell state (10c
10 10i
101
10o, and [Or) by 1x1 probe. Teal
arrows denote squares that the agent expects next step onto from the associated direction. Blue
arrwws denote squares that the agent expects to push box off in the corresponding direction. These
examples are taken from the first transition the respective episode in which the agent's plan as decoded from its first-layer cell state corresponds to complete plan to solve the level.
18

---

Published as a conference paper at ICLR 2025
CA
Cs
CA
Cs
CA
Ca
(a) Example - Layer
(b) Example - Layer 2
(c) Example
-Layer
Cs
CA
CA
(d) Example 2 Layer 1
(e) Example 2- Layer
(l) Example 2 Layer 3
CA
Cs
CA
CA
C3
(g) Example 3 Layer
(h) Example Layer
ii) Example Layer 3
Cn
Co
CA
Cn
Cg
(i) Example 4 Layer
(k) Example 4 Layer 2
(1) Example
Layer 3
CA
C5
CA
Cg
CA
Ce
(m) Example 5 Layer
(n) Example 5 Layer 2
(o) Example 5 Layer
CA
CA
(p) Example 6 Layer 1
(q) Example Layer
(r) Example 6 Layer 3
Figure 11: Internal plans computed by the agent in 6 examples levels as decoded from the agent's
first layer cell state la Lid [ig I1j Im and Ipp.. second layer cell state {TIb Ie Ih
TTk In and Li third layer cell state [T
Ho, and Hir by 1x1 probe. Teal
arrows denote squares that the agent expects next step onto from the associated direction. Blue arrww denote squares that the agent expects to push box off in the corresponding direction. These
examples are taken from the first transition the respective episode in which the agent's plan as decoded from its second-layer cell state corresponds to complete plan to solve the level.
19

---

Published as a conference paper at ICLR 2025
CA
Cs
CA
Cs
CA
Ca
(a) Example Layer
(b) Example
Layer
(c) Example 1 Layer
CA
(d) Example 2 Layer 1
(e) Example 2- Layer
(l) Example 2 Layer 3
Cs
CA
CA
C3
(g) Example 3 Layer
(h) Example Layer
ii Example Layer 3
CA
Cg
CA
Cg
Cn
Cg
(i) Example 4 Layer
(k) Example 4 Layer 2
(l) Example
Layer 3
CA
C5
CA
C3
CA
C3
(m) Example 5 Layer
(n) Example 5 Layer 2
(o) Example 5 Layer
Ca
CA
(p) Example 6 Layer 1
(q) Example
Layer
(r) Example 6 Layer 3
Figure 12: Internal plans computed by the agent in 6 examples levels as decoded from the agent's
first layer cell state [12a 12d 12g [2
12ml and 12p second layer cell state 12b 12c 12h
12k 12n and 12g third layer cell state [
12o, and 12r) by 1x1 probe. Teal
arrows denote squares that the agent expects next step onto from the associated direction. Blue
arrwws denote squares that the agent expects to push box off in the corresponding direction. These
examples are taken from the first transition the respective episode in which the agent's plan as decoded from its final-layer cell state corresponds to complete plan to solve the level.
20

---

Published as a conference paper at ICLR 2025
A.2 FURTHER EXAMPLES OF INTERNAL PLAN FORMATION
In this section, we provide additional examples of the agent seeming to use its implicit learned environment model internally form plans via an evaluative search. As in the main paper, we focus on the agent's internal plan as formulated in terms of its representations of CB- That is, we focus on the routes the agent expects to push boxes (e.g. 'box plans') rather than the paths the agent expects to follow (e.g. 'agent plans"). This is for two reasons. First, we expect 'box plans' to be determinative of "agent plans' as the primary difficulty of Sokoban regards box movements. Second, 'box plans' are easier to visually inspect as 'agent plans' often intersect such that it is usually ambiguous to tell why a square has been added to an 'agent plan'. All examples use handcrafted Sokoban levels designed to allow for clear plan visualization.
We begin by providing additional examples of the plan formation 'motifs' shown in Figure These motifs are re-occuriing themes in the agent's ' internal plan formation process. In each of the follow- ing five sub-sections we present five examples of the agent formulating internal plans in a way that demonstrates one of the aforementioned motifs. These motif's are:
Evaluative Planning modifying plans based on feasibility (Section 1.2.1)
Adaptive Planning modifying plans based on conflicts (Section A.2.2)
Forward Planning forming plans by searching forward from boxes (Section A.2.3
Backward Planning forming plans by searching backward from targets (Section A.2.4)
Parallel Planning- forming multiple plans in parallel (Section A.2.5
However, these motifs are not the only evidence of evaluative, search-based planning exhibited by the agent's internal plan formation process. Specifically, evidence of this can also be seen when inspecting the manner in which the agent forms plans when confronted with various forms of out- of-distribution Sokoban levels. As such, we provide examples of the agent successfully formulating internal plans under various types of distribution shift. That is, we show examples of:
Blind Planning planning in levels in which the agent itself is not present (Section A.2.6)
Generalized Planning planning in levels with extra boxes and targets (Section A.2.7)
Blocked-Route Planning planning in levels which additional walls appear at later time steps that block obvious routes to push boxes (Section A.2.8)
New-Route Planning planning in levels in which walls disappear at later time steps such that improved routes become available (Section A.2.9
Finally, in Section 1. we will conclude by discussing the implications of the agent's 's apparent learned search-based planning process.
A.2.1 EVALUATIVE PLANNING
Under our characterization of 'planning' in Section 2.1 planning requires that an agent evaluates its plans. That is, merely formulating internal plans is not sufficient to view an agent as engaging in planning. Instead, we understand 'planning' as requiring that an agent arrive at an internal plan by means of evaluating different possible plans. This is because evaluating plans (e.g. in terms of their effects on the environment) allows the agent to formulate plans that predicts will lead to good consequences.
When visualizing the agent's plan formation process, we see evidence indicative of the agent im- plementing some form of evaluative search-based planning. For instance, we see evidence of the agent iteratively constructing planned routes to push boxes to targets, then evaluating these routes. Specifically, the agent seems capable of evaluating routes and determining whether they are feasible. We call this evaluative planning.
Figure 13 shows the development of agent's internal plan in five episodes in which it performs evaluative planning of this sort. For instance, in Figure [3c the agent's initial internal plan (i.e. at the first computational tick) involves the agent pushing the upper-left box down through corridor to the center-most target. On the face of it, this plan is appealing. This is because it is a 'short' plan in terms of the number of squares it would involve pushing box across. However, this plan
21

---

Published as a conference paper at ICLR 2025
is infeasible. This is because the corridor is structured such that, upon pushing the box down into it, the agent would block the corridor off. This would then prevent the agent from (a) navigating to the left of the box would be required to push it right along the corridor to the target, and (b) navigating to the lower-left box and target. Over subsequent ticks, the agent appears to evaluate its plan and realize this. In response, the agent then construct alternative plan for the upper-left box. While less naively appealing it is longer plan in terms of the number of squares the box would need to be pushed this plan would allow the agent to solve the level. The ability of the agent to recognize and avoid such 'bad' plans implies that the agent has learned to evaluate plans required by our characterization of planning.
A.2.2 ADAPTIVE PLANNING
Determining whether single route would allow the agent solve level is not the only type of evaluation that the agent appears to perform when it formulates its internal plans, Additionally, the agent also appears to adapt its plans when conflicts arise between its sub-plans. That is, in cases when the agent's 's internal plan involves pushing two boxes to the same target, the agent adapts its plan by planning for one of these boxes to be pushed top an alternative target. This suggests that the agent is predicting and evaluating the consequences of its actions. We call this form of evaluation adaptive planning.
Instances of the agent performing adaptive planning when formulating its internal plans can be seen in Figure [4] Figure [4 shows the development of agent's internal plan over the initial steps of five episodes in which it performs adaptive planning. For examples, consider the way in which the agent's plans develop in Figure [4c] During the initial three computational ticks, the agent plans to push two separate boxes -i.e. the lower-left box and the upper-left box to the left-most target. Importantly, however, in this level the upper-left box must be pushed to this target. This is because the left-most target is the only target that the upper-left box can feasibly be pushed to (e.g. if the lower-left box was pushed onto the left-most target, the level would be unsolvable). Over the fourth and fifth computational ticks, the agent appears realize this and form an alternate plan for the lower-left target (e.g. plan to push it to one of the top-right targets).
We take the ability of the agent to adaptively plan in this fashion as evidence that, during planning, the agent is capable of predicting and evaluating the (relevant) consequences of its actions. That is, the agent not only internally represents plans formulated in terms of the consequences of its actions on box locations (i.e. its internal plans formulated in terms of CB), but also has learned that a consequence following one these plans and pushing a box onto target is to 'fill' this target. This suggests that the agent has learned that the effect of pushing box onto a target is to stop other boxes being able to be pushed onto that same target.
22

---

Published as a conference paper at ICLR 2025
LO step 3, tick
MAN Step tick
step 9, tick
Step Lik
2, lick
step 5, Bik
(a) The agent initially plans to push box through the circled corridor. However, the agent appears to realize that this is infeasible as it cannot push the box up onto the target as required by this plan. then forms longer plan that involves pushing the box longer route to the same target.
COPP stop 4, tick
step D, tick
stop 1, tickk 3
stop 2, tick
step 3, tick
step 5, tick 3
(b) The agent initially plans to push a box up through the cireled corridor. However, over subsequent time steps, the agent appears to realize that this is infeasible as it cannot get under the box at the corridor entrance as it would need to in order to push this hox up through the corridor. It then modifies its plan that this box is instead pushed longer that avoids the corridor.
S
step O, tick
Stop tick
step 2,
step 3, tick 3
stop 4, tick
step 5,
(c) The agent initially plans to push box down a target along the circled route. However, over subsequent computational ticks, the agent realizes that this plan would prevent it from solving the level as would prevent it from ever reaching the lower- left box and target. The agent then modifies its plan so that this box is instead
pushed a longer route avoiding the corridor.
O +
step lick
Step
step 2, lick
step tick
step
step Bick
(d) At the initial time step, the agent plans to push box down through the circled corridor: However, the agent realizes that this not possible as cannot get above the box at the corridor entrance as would required in order to push it down. The agent then updates its plan so that this box is pushed through the further corridor.
stop 0, tick
step tick
step 2. tick
step 3. tick
step tick
step 5, tick
(c) After the first computational tick, the agent plans to push a box along the circled route (c.g. the shortest route to the respective target). However, at the following tick, the agent appears to realizes that this is not possible as the agent would be unable push the box left as required. The agent then updates its plan so that this box instead pushed longer route to target.
Figure 13: Examples of episodes in which the agent's internal plan initially includes planned routes that would not lead to the agent solving the level. In all these examples, the agent realizes that part of its plan is infeasible and updates its plan accordingly. Blue arrows represent the direction that the agent plans to next push a box off of each square. Yellow circles highlight parts of the agent's plan that are infeasible and later removed. The plans decoded from the agent's cell state at its first (3a). second (3b and 13c) and third [3d and [3e) layer by a 1xl probe. The plans are decoded
from the agent's 's cell states at either the final computational tick of the first six steps of episodes
13a
[3b and l3d). or at each computational tick of the first two steps of episodes (33c and l3c).
23

---

Published as a conference paper at ICLR 2025
1
stop D, tick 3
Stop tick
step 2. tick 3
stop 3, tick 3
stop tick
step 5, tick
(a) The agent initially plans to push two boxes to the lower left target. However, at later time steps, the agent alters its plan to instead push one of these boxes to the top-right target.
step
Step
Step 2, lick
step
step tick
step Bick
(b) The agent again initially plans push two boxes the two lower right targets. Again, later time step, the agent adapts its plan to instead push one these boxes the top-right target.
step 0, tick
stop O, tick
step O, tick 3
step 1, tick
stop 1, tick
step 1, tickk
(c) During the initial ticks, the agent plans to push both the lower-left and upper-left boxes to the left-most target. However, the agent appears to realize that the top-left box must be pushed to this target (i.e. is the only target that is feasible push the top-left box to). The agent then adapts its plan to instead push the lower-left box of the top-most targets.
step tick
stop
step 2,
stop
tick
step
step 5,
(d) Over the first two time steps, agent considers pushing the lower-left box forward one the lower-
right targets. However, this generates a conflict. The agent appears to realize this and then connect this box to a plan it has constructed that links this box to the top-left target.
step D, tick
step tick 2
O, tick
step tick
step tick2
step 1, Bick
(e) After the third tick, the agent plans push two boxes the lower-right target. However, the agent seems realize that the lowest box must be pushed to this target (e.g. the only target the lowest box feasibly be pushed the lower-right target). At the fourth tick, the agent hence plans to instead push the other box that it has associated with the lower-right target to an alternate target.
Examples of episodes in which the agent initially plans to push multiple boxes to the same target before modifying its plan to push one of these boxes to an alternate target. Blue arrows represent the direction that the agent plans to next push box off of each square. Yellow circles highlight parts of the agent's 's plan that involve pushing two boxes to the same target. The plans are decoded from the agent's cell state its first (14a), second (14b and 14ch and third [14d] and 14eh layer by a 1x1 probe. The plans are decoded from the agent's s cell states at either the final computational tick the first six steps of episodes [1a 14h 14d). or at each computational tick of the first two steps of episodes [14c and 14ch.
Figure 14:
24

---

Published as a conference paper at ICLR 2025
A.2.3 FORWARD PLANNING
In the previous two sections, we described forms of evaluation the agent appears perform as part of its iterative plan-construction process. However, this leaves open the question of how the agent iteratively constructs plans. As explained in Section the agent appears to do so by using some form of learned, iterative search process. In this section, we now describe one major form of iterative, search-based plan-construction the agent performs: planning forward from boxes.
Specifically, one of the (two) primary ways the agent appears to construct its internal plans is by iteratively extending its internal plans forward from boxes. That is, the agent seems to frequently 'initializee plans at box locations, and then iteratively extend these plans forwards towards targets over the early computational ticks of episodes. This a form of forward search. Algorithms based on forward search - for instance, Monte Carlo Tree Search (Coulom) 2006) are the predominant form of planning algorithms currently used in model-based planning agents. It is hence notable that the agent appears to have learned planning algorithm that (partially) relies on forward search.
Instances of the agent constructing its internal plans by iteratively searching forward from boxes can be seen in Figure [5] Figure 15 shows the development of agent's 's internal plan over the initial steps of five episodes in which the agent constructs part of its internal plan by planning forward from boxes towards targets. As a specific example, consider Figure [5a] the agent iteratively constructs plan to push the bottom-right box to the top-right target by planning forward from the bottom-right box. Notice that the part of its plan that the agent iteratively constructs forward from this box end up connecting with a partial plan that the agent has constructed by iteratively searching backward from the top-right target. This will be discussed in the following section.
A.2.4 BACKWARD PLANNING
Iteratively searching forward from boxes is not the only form of search-based planning that the agent appears to engage in. Additionally, the agent appears to search backwards from targets to boxes. That is, the agent will frequently initialize the end of a plan (i.e. by initializing plan that ends at specific target), and then iteratively search backwards towards boxes. This is form of backward search. We hence refer to this as backwards planning.
Examples of the agent constructing its internal plans by iteratively searching backwards from targets can be seen in Figure [6] Figure [6 shows the development of the agent's internal plan over the initial steps of five episodes in which the agent constructs part of its internal plan by iteratively planning backwards from targets towards boxes. For instance, consider the manner in which the agent forms its internal plan in the episode shown in Figure [ Between the first and fifth tick in this episode, the agent iteratively extends a planned route backwards from the lower-right target to the lower-right box.
Backward search was long studied in the context of planning in 'classic' RL (Moore Atkeson 1993). However, whilst some recent work has investigated methods relating to backward planning (Goya et al., 2018, Van Hasselt et al.) 2019 Lee et al., 2019), backward-facin icantly less popular than forward- facing planning in modern model-bbsed RL agents. The fact that the agent has learned to (partially) rely upon backwards planning is, therefore, interesting.
planning is signif-
A.2.5 PARALLEL PLANNING
Finally, the agent appears to be capable of extending multiple plans in parallel over a single compu- tational tick. We refer to this parallel planning. Examples of the agent constructing its internal plans in parallel can be seen in Figure 17 Figure 17 shows the development of the agent's internal plan over the initial steps of five episodes in which the agent utilizes parallel planning. Parallel planning is not something that is common in standard planning algorithms. This because, unlike the DRC agent that can plan by applying convolution operations to its spatially-extender cell states, standard planning algorithms must extend a single node at a time. We hypothesize that this parallel planning learned by the agent further increase the efficiency with which it plans.
25

---

Published as a conference paper at ICLR 2025
step 0, tick
Step tick
2. tick
step 3, tick
step
tick
step 5, tick
(a) The agent iteratively extends part of its plan forward from the bottom-right box. The agent extends this part of its plan forward until connects part of the agent's plan that connects it the top-right target.
step
Step
Step O,
step
step
step
(b) The agent constructs part of its plan by iteratively searching forward from the bottom-right box. The agent searches forward until this part of plan connects to part agent's plan that connects box to bottom-left target.
stop O, tick
step tick
step 0, tick 3
stop 1. tick
step tick
step 1, tick
(c) The agent formulates planned route push the top most box by extending its plan forward from this box to the top-right target.
step
Step tick
step
step
step
step
(d) The agent forms plan for the bottom-most box by searching forward from this box to the lower-left target.
step tick
step
step 0,
step
tick
step
step
(c) The agent forms plan for the center-most box by iteratively searching forward from this box the lower- right target.
Figure 15: Examples of episodes in which the agent formulates its internal plan by iteratively tending planned routes forward from boxes. Blue arrows represent the direction that the agent plans push box off of each square. Yellow circles highlight parts of the agent's plan that it has constructed by iteratively searching forward from boxes to targets. The plans are decoded from the agent's cell state its first (15a), second (15b) and [5c) and third [15d|an and (5e] layer by 1x1 probe. The plans are decoded from the agent's ' cell states at either the final computational tick of the first six steps of episodes [15a 15b and 15d), or each computational tick of the first two steps of episodes [1c and 15eh.
26

---

Published as a conference paper at ICLR 2025
step
Step
step 2,
step
step
step
(a) The agent formulates part of its internal plan by iteratively searching backwards from the bottom-right target to the center-most box.
step D, tick
step 1, tick
stop 2, tick 3
step 3, tick 3
stop 4, tick 3
step 5, tick
(b) The agent constructs a plan to push box to the top-most target by iteratively extending a plan backwards from this target. The agent extends this plan backwards until it connects to the top-most box.
step tick
step Lck 2
0, lick
step tick
step tick
step Bick
(c) The agent forms internal plan push box to the top-most target by iteratively searching backwards from this target. The agent formulates this part of its plan by searching backwards until this plan connects the lower-right box.
step 0, tick
stop 1, tick
stop 2, tick
step 3, tick
stop 4, tick
step 5, tickk
(d) The agent formulates a plan to push the left-most box to the left-most target by scarching backwards from the eoot target to the left-most box.
step tick
step tick 2
O, tick
step tick
step tick
step 1, tick
(e) The agent constructs plan to push box to the lower-right target by iteratively constructing plan back- wards from this target. The agent extends this plan backwards until it connects the lower-right box.
Examples of episodes in which the agent formulates its internal plan by iteratively ex- tending planned routes backward from targets. Blue arrows represent the direction that the agent plans to next push a box off of each square. Yellow circles highlight parts of the agent's plan that has constructed by iteratively searching backwards from targets boxes. The plans are decoded from the agent's is cell state at its first (Taa), second (16b) and T6c) and third (T6d] T6e) layer by Ixl probe. The plans are decoded from the agent's 's cell states at either the final computational tick of the first six steps of episodes [I6a 16b and [6d), or at each computational tick of the first two steps of episodes (16c and 16e
Figure 16:
27

---

Published as a conference paper at ICLR 2025
step tick
step
step tick
step tick 1
step
step tick
(a) At the fourth computational tick, the agent extends parts of its plan (e.g. its plans to push boxes to the top-most targets) in parallel.
step D, tick
step 0, tick
step 0,
step 1, tick
step tick
step 1, tick
(b) At the second computational tick, the agent iteratively constructs internal plans for the two top-most targets by searching backwards from these two targets parallel.
stop D, tick
step tick
0, tick
stop 1. tick
step tick
step 1, tick 3
(c) At the third computational tick, the agent iteratively extends its internal plans associated with the two top- most targets by extending these two plans in parallel.
+
step tick
step
step
step
Step
step Bck3
(d) The agent searches backwards from the bottom-most targets in parallel the third computational tick.
step tick
step
step O,
step tick 1
step
step 1,
After the third computational tick, the agent extends two parts of its internal plan parallel.
Figure 17: Examples of episodes in which the agent formulates its internal plan by extending mul- tiple planned routes in parallel over single computational tick. Blue arrows represent the direction that the agent plans to next push box off of each square. Yellow circles highlight parts of the agent's plan that it has constructed in parallel over a single tick. The plans are decoded from the agent's is cell state its first (17a), second (17h and 17cp and third (Td (7e layer by 1x1 probe. The plans are decoded from the agent's s cell states at either the final computational tick of the first six steps of episodes [17a 17b and [7d), or each computational tick of the first steps of episodes [7c and 17e).
28

---

Published as a conference paper at ICLR 2025
A.2.6 BLIND PLANNING
In a discrete, deterministic environment such as Sokoban, natural way for an agent to plan would be for it to search over possible sequences of future actions in search of sequence of actions that achieved some goal. In Section [5] we noted that the agent's internal plans consistently represent routes connecting boxes and targets. This, alongside all previous visualizations of the agent's 's plans, suggests that the 'goal' the agent evaluates sequences of actions in terms of when performing search is whether said actions represent a feasible route to push box along to target.
If the agent formulates internal plans by searching over potential future actions with the goal of connecting boxes and targets, we would expect the agent's planning algorithm to (at least attempt to) search for plans achieving this goal in any Sokoban level so long as that level contained boxes to plan from and targets to plan to. That is, if the agent does indeed formulate internal plans by searching for plans achieving the goal of connecting boxes and targets, we would expect the agent to be able to formulate plans in Sokoban levels drawn from significantly different distributions.
We now provide examples of the agent successfully forming plans in a very different type of level to the levels on which it trained. Specifically, we provide examples of the agent appearing to search for plans in levels in which it is not itself present. These are Sokoban levels in which the agent observes the level, but is not actually positioned on any square. Note that this represents significant distribution shift to the levels the agent was trained on. Indeed, the agent can never actually influence these levels. Crucially, however, this distribution shift should not prevent the agent from attempting to form plans if it did indeed plan in the hypothesized manner.
Figure 18 shows the development of the agent's internal plan in levels in which the agent is not itself present. Clearly, the agent (i) still attempts to form plans and (ii) forms internal plans that success- fully connect boxes and targets. We take the ability of the agent to continue internally forming plans in the face of this radical distribution shift as evidence that the agent indeed possesses some learned search procedure that searches for plans that achieve the goal of connecting boxes and targets. How- plan the
ever, we note that an unexplained curiosity is that, in some such levels, after arriving agent will seemingly completely forget it. That is, in some cases of blind planning, the agent will begin to form a plan and then, after many time steps, proceed to forget the plan and represent no plan at all.
A.2.7 GENERALIZED PLANNING
In the original paper introducing DRC agents, Guez et al. (2019) demonstrated that Sokoban-playing DRC agents trained on the Boxoban dataset of Sokoban levels (i.e. levels containing four boxes and four targets) can successfully generalize to Sokoban levels with additional targets and boxes. Specifically. they showed that such DRC agent can solve Sokoban levels with additional boxes and targets.
Given the discussion thus far, hypothesize that the reason for this is that an agent possessing a planning mechanism of the above sort (i.e. an agent that planned by searching for sequences of actions corresponding to routes between boxes and targets) would be able successfully execute its planning mechanism in such levels. This is because simply introducing a search process that searched for routes connecting boxes and targets could easily generalize to levels in which additional boxes and targets are present.
Figure 19 shows examples of the agent's internal plan the final tick of the initial six time steps in episodes in which there are either five boxes and targets, or six boxes and targets. As implied by the above discussion, the agent (i) still attempts to form plans and (ii) forms internal plans that successfully connect boxes and targets. We take this as additional evidence of the agent searching for plans that achieve the goal of connecting boxes and targets.
29

---

Published as a conference paper at ICLR 2025
step
step
step 0,
step tick
step
step I,
stop 0, tick
stop
stop 2,
stop 3, tick 3
stop
step 5.
step 0, tick
step 0, tick
step 0, tick
step 1, tick
step tick
step 1,
step 0, tick
step 1, tick
step 2, tick
step 3, tick
step 4, tick
step 5, tick
(d)
stsp 0, tick
step tick
step 2, tick 3
step 3, tick
step tick
step 5, tick 3
Figure 18: Examples of episodes in which the agent formulates an internal plan despite not being present in the level. Blue arrows represent the direction that the agent plans next push box off of each square. The plans are decoded from the agent's cell state at its first (8a)), second (18b and I&ch and third (Td [8c) layer by Ixl probe. The plans are decoded from the agent's 's cell states the final computational tick of the first six steps of episodes.
30

---

Published as a conference paper at ICLR 2025
step 0, tick 3
step 1, tick
step 2. tick 3
step 3, tick 3
step 4, tick
step 5. tickk
Step 0, tickk
step 1, tick 3
step 2, tickk
step 3. LiEk 3
step tick
step 5. Bikk
(b)
step tickk
Lick
2, lick
step tick
step tick
5, Bick
tick
step tick
2, tick
step tick
step tick
tick
step
step
step 2,
step
step step
Figure 19: Examples of episodes in which the agent formulates its internal plan despite there being more boxes and more targets than in the levels on which it was trained. Blue arrows represent the direction that the agent plans to next push box off of each square. The plans decoded from the agent's cell state at its first (I9a), second (99 and 19ch and third [9dd and 19e) layer by a Ixl decoded from the agent's cell states the final computational tick of the first six steps of episodes. These episodes take place in levels in which there are five boxes and targets
probe. The plans
19a 19b and T9d) and in levels in which there six boxes and targets (19c and 19c)
31

---

Published as a conference paper at ICLR 2025
A.2.8 BROCKED-ROUTH PLANNING
The previous two sub-sections provided examples of the agent successfully formulating plans in levels that represented significant distribution shifts relative to the training distribution. These pre- vious distribution shifts aimed to induce changes to the agent's s environment that would not impede the planning capabilities of an agent that planned via searching for plans that achieved the implicit goal of connecting boxes and targets. In this sub-section and the following sub-section now con- sider different forms of distribution shift. Namely, we now consider distribution-shifted Sokoban levels that aim to test the ability of the agent to dynamically evaluate and update its internal plan in response to environmental changes.
We begin by considering Sokoban levels in which, at time step following the initial time step, an additional wall square is added to the level. Specifically, this wall square is added to a location that blocks obvious route between box and target. The aim of investigating these levels is to determine whether the agent is capable of evaluating that this additional wall square invalidates its current plan, and whether the agent can dynamically form new plan after doing so. Figure 20 shows the manner in which the agent's 's internal plan develops in levels in which, at time step following initialization, we add wall square block off an obvious route between box and a target. Clearly, the agent is capable of (i) recognizing that the added wall invalidates its initial plan and (ii) dynamically adjusting its plan accordingly. We take this as evidence to support the hypothesis that the agent forms plans via evaluative search process.
A.2.9 NEW-ROUTE PLANNING
Given the ability of the agent to dynamically update its plans in response to the addition of wall square to block off an optimal route to push box, an obvious question to ask is whether the agent can dynamically update its plan in levels representing the reverse type of distribution shift. That is, can the agent dynamically update its plans in levels in which, at some time step following initialization, we remove a wall square to open up an optimal route to push a box to a target that is infeasible prior to the removal of the wall?
Figure 21 shows the development of the agent's internal plan in levels where we, at some time step following initialization, remove a wall square to open up an optimal route to push a box to target that is infeasible prior to the removal of the wall. In some levels for example, in Figure 21a the agent does dynamically respond to this wall-removal by updating its plan to exploit the new, optimal route. However, other levels such as in Figure 216 the agent does not do this. We conjecture that this is potentially due to the agent having notion of a 'completed route' within its internal plan. That is, we conjecture that the agent represents some plans as being complete and requiring no further search, and this is why the agent modifies its plan following the removal of a wall in some cases but not others.
A.2.10 DISCUSSION
In discrete, deterministic, colly-observable environments like Sokoban, an agent with access to perfect environment model can reformulate the problem of 'planning' the problem of searching for a sequence of future actions a plan that achieves a goal (Russell & Norvig 2010). The agent studied in this paper lacks such a perfect world model.
However, we have demonstrated that the agent we study has learned a spatial correspondence be- tween its cell states and the Sokoban grid, such that can represent spatially- corresponding positions of its cell state. This can, perhaps, be seen as a learned 'implicit' model of the environment. Importantly, this learned implicit world model sufficient to (i) represent se- quences of future actions and (ii) predict relevant consequences of these actions the environment. hence appears to be sufficient to allow the agent to plan via search.
concepts at
We believe the examples provided in the previous sections support the hypothesis that the agent indeed plans via applying a learned search algorithm to a learned, 'implicit' model of its environ- it implies that the agent's emergent planning capabilities represent learned analogue to the planning-cat capable agents introduced in Appendix E. that plan via applying an explicit search algorithm an explicit world model.
ment. This is interesting
32

---

Published as a conference paper at ICLR 2025
Step 0, tick
step 1, tick 3
step 2. tick
step 3, tick
step tick
step S, tick
(a) After the first step, a wall is added to block the agent's planned route between center-most box and target. Over the subsequent time steps, the agent realizes this and dynamically forms a new planned route connecting this box and target.
step
Step
step 2,
step
step
step
(b) After the first step, wall is added block the agent's planned between left-most box and target. Over subsequent time steps, agent realizes this and dynamically forms new plan that involves pushing this box an alternate, longer route to this target.
stop O, tick
Step tick
step 2. tick
step 3, tick 3
step tick
step 5, tick
(c) After the first step, wall is added to block the agent's planned route between right-most box and target.
During the steps that follow, the agent realizes that this invalidates its initial plan and dynamically forms new
plan that involves pushing this an alternative route.
step O, lick
Step 1, tick
step 2, tick
step 3, tick
step tick
step 5, tick
(d) Initially, the agent plans to push central box down to central target. After the first step, a wall added to block this route. During the following steps, the agent realizes that this has occurred and dynamically forms new plan that involves pushing this box left, down, and then right, to this target.
step tick
step
2, tick
step tick 3
step tick
step
tick
(c) Initially, the agent plans push the central box down the central target. However, after the first step, wall is added to block off this route. During the following steps, the agent realizes that its initial plan now infeasible dynamically forms new plan that instead involves pushing this box right, down, left, then up, to this target.
Figure 20: Examples of the agent formulating its internal plan in levels in which wall square is added to the environment to block an obvious route between box and target at time step following initialization. Blue arrows represent the direction that the agent plans next push box off of each square. Yellow circles highlight relevant parts of the agent's 's plan before and after the additional wall square is added. The plans are decoded from the agent's cell state at its first (20a), second (20b) and 20ch and third (20d and 20ch layer by Ixl probe. The plans decoded from the agent's 'ss cell states at the final computational tick of the first six steps of episodes.
33

---

Published as a conference paper at ICLR 2025
step D,
tick
step 1, tick 3
step 2, tick
step 3, tick
step 4, tick
step S, tick
(a) The agent begins planning to push the upper-left box up, right, and down to the center-most target. However, after the first step, wall is removed such that this box can instead he pushed straight right to the target. The agent realizes this and updates its plan accordingly.
step
Step
Step 2,
step tick
step
step 5,
(b) Initially, the agent plans push the left-left box right, up, left, and down the left-most target. However, after the third step, a wall is removed such that this box can instead be pushed a shorter route (i.e. left and the up) to the target. The agent does not realize this and does not update its plan in response.
step 0, tick
step tick
step 2. tick 3
step 3, tick 3
step
tick
step 5, tiick
(c) After the second step wall is removed. The removal of this wall means that the optimal route to push the right-moo most box the right-most target is push right and then up. Before the removal of the wall the agent
plans push left, up and then right. However, after the removal, the agent updates its plan to account the
new optimal route.
step D, 3ick
step 1, tick
step 2, tick
step 3, tick
step 4, tick
step 5, Bik3
(d) The agent initially plans to push the right-most box and around the wall that separates it from the right- most target. After the second step, wall is removed such that this box can now be pushed a shorter route to this target. The agent fails respond this.
step 0, tick3
step tickk
step 2,
step 3, tick 3
step
tick
step 5, tick
(c) The agent initially plans push the center-most box around the wall that separates from the center-most target. After the first step, a wall square is removed such that this box can now pushed directly down this target. The agent realizes this and updates its plan to account for the new optimal route to push this box.
Figure 21: Examples the agent formulating internal plans in episodes in which a wall square is removed at a time step after initialization. Removing this wall square opens up a route that it is optimal push box to a target through. Blue arrows represent the direction that the agent plans to next push box off of each square. Yellow circles highlight the relevant parts of the agent's plan (or lack of internal plan) before and after the wall square removed. The plans are decoded from the agent's cell state at its first (2la), second (21b) and 21c) and third 21d and 21c) layer by 1x1 probe. The plans are decoded from the agent's 's cell states at the final computational tick the first six steps of episodes.
34

---

Published as a conference paper at ICLR 2025
This perspective on the agent's concept representations - i.c. that, by enabling the agent form plans and evaluate their consequences, collectively, these representations play a role that can be seen as the role of learned implicit world model, aligns with work that has emphasized the importance of world models for generalization capabilities (Richens & Everiti 2024] Andreas 2024). It also provides new insights regarding learned world models in RI.. Specifically. complements past work that has investigated explicitly training world models (Ha & Schmidhuber, 2018 Freeman et al.) 2019) by showing that world models- or, at least, representations that can play a role traditionally played by world models -can also emerge spontaneously within the representations of a generic agent.
Additionally, it is interesting that the agent constructs its internal plans by simultaneously searching forward from multiple boxes and searching backwards from multiple targets. That is, the agent appears to have learned form of parallelized bidirectional planning. This is very different to the agents introduced in Appendix F. that primarily rely on forward search algorithms.
Whilst some past work has had considerable success in applying bidirectional search to RL (Edwards) ct al., 2018; Lai et al.) 2020), RL agents making use of bidirectional planning are still remarkably rare. This is likely due to the difficulty in specifying which states to plan forwards and backwards from in many environments. Indeed, Sokoban is somewhat unique in that it is especially well-suited for bidirectional search. This is because there are obvious candidates to plan forwards from (boxes) and backwards from (targets). As such, the main takeaway from the emergence of bidirectional search within the agent is likely not that bidirectional search should be applied more widely within RL.
Instead, we believe the main takeaway from this finding be that there are benefits to allowing agents to learn a planning algorithm (and a implicit-world model to apply it to) rather than forcing an agent to use handcrafted planning algorithm. This is because the agent can learn to plan in a way that is well-suited for the enviromment it finds itself in. We suspect this idea of allowing agents to learn search algorithms well-suited for specific domains, rather than forcing them to use a generic, handcrafted search algorithms such as MCTS may become increasingly prevalent.
This to say that we hypothesize that the agent we study has learned form of planning that is especially well-suited to Sokoban relative to other planning algorithms. Bidirectional search is known to be very efficient in certain environments (Kaindl Kainz 1997 Russell & Norvig 2010 Sturtevant et al. 2020). Intuitively, this is because it is more efficient to form plans by searching backwards from goals and forwards from initial states (i.e. because these two plans can 'meet in the middle") > than to form plans by either of these means alone. Furthermore, Sokoban is characterized by actions having negative consequences in the long-run, efficient planning is crucial in Sokoban. This is because forming plans quickly the start of episodes allows the agent to avoid taking actions early on that would make a level unsolvable. Thus the agent studied in this paper appears to have learned domain-specific planning algorithm that works well in the environment finds itself in. Evidence of this can be seen in the fact that one of the nighest-performin Sokoban agents that does not rely on deep learning also uses form of
planning (Shoham & Elidan 2021)
A.3 FURTHER RESULTS REGARDING ITERATIVE PLAN REFINEMENT
In Section 5] we used Figure 6 to demonstrate that the agent can use additional test-time compute at the start of episodes to refine its internal plan. This would be expected if the agent constructed plans using some form of learned search, since the agent would be able to use additional compute to per- form a more thorough search. This additionally helps explain the ability of DRC agents to perform better in Sokoban when given 'thinking time' steps (Guez et al. 2019 Taufeeque al. 2024) we have shown that the extra compute afforded by 'thinking time' facilitates plan refinement. In this section, we further investigate the agent's ability iteratively refine its internal plans when forced to perform 'thinking time' steps - that is, forced stationary steps at the start of episodes prior to acting.
A.3.1 TEST-TIME PLAN REFINEMENT ACROSS LAYERS
In Section 5 Figure 6 only demonstrated that the agent can use 'thinking time' iteratively refine its plan in its final layer. To investigate whether the agent can iteratively refine its plans at all layers when given additional test-time compute prior acting, we thus again forced the agent to perform
35

---

Published as a conference paper at ICLR 2025
1.0
Macro FT
0.8
Macrn F1
0.8
0.6
0.6
0.4
0.4
0.4
Internal T
Internal Tick
Internal Tick
Co
Ca
(a) Layer
(b) Layer
(c) Layer
Figure 22: Macro Fl (averaged over 1000 episodes) when using the agent's ss internal plan at each internal tick during the first steps of an episode predict (a) the agent's future movements (CA) and (b) future box movements (Cp). The 'internal plan' at a layer for tick corresponds to the agent's ss representation of CA and Cp as decoded by a Ixl probe applied to the agent's s cell state at that layer.
(a) Length
(b) Length 6
(c) Length 10
(d) Length 14
Figure 23:
Example of one the levels used to test for behavioral evidence of search when its corridor is of different lengths.
"thinking time' steps before beginning to act in 1000 episodes. As with Figure 6 after each of the 15 internal computational ticks performed by the agent during these steps, we applied 1x1 probes to decode the agent's representations of CA and Cp for each square the observed Sokoban board at that tick. As argued previously, the agent's ' internal representations of the concepts over the entire Sokoban board can seen as its internal plan. We viewed the agent's internal plan at each tick as prediction of its future behavior (CA) and the effect of this future behavior the environment (CB), and measured the correctness these prediction using the macro Fl score. The results can be seen in Figure 22 Clearly, the agent's internal plan gets iteratively refined over the course of 'thinking time' at all layers.
A.3.2 EVIDENCE OF TEST-TIME COMPUTE BEING USED FOR SEARCH
In this paper, have provided qualitative evidence that supports the hypothesis that the agent plans via learned search procedure, and that the agent reason the agent henefits from additional test- time compute because it uses this extra compute to search more thoroughly prior to acting. In this section we now complement this with behavioral evidence of the agent using extra test-time compute for search.
We do this using a dataset of handcrafted levels. These levels all follow a common schematic. Namely, there corridor with single entrance. At the end this corridor a box and target. The entrance square of the corridor also has target on it. There is box adjacent to this target at the entrance the corridor. In these levels, the agent always starts adjacent this box. Thus, at the initial time step, a myopic agent will always push this box on to the target. However, doing so blocks off the corridor (e.g. it prevents the agent from ever reaching the box and target at the end). flanning-capablee agent to realize this, and instead plan push the box out
So, would expect of the way so it can enter the corridor. We create handcrafted levels of this sort. For each level, create copy where its corridor is of length 2, 6, 10 and 14. Figure 23 shows version of one of these levels with these corridor lengths. We then reflect and rotate each level. Thus, we have a dataset of 80 such levels with corridors of each length.
36

---

Published as a conference paper at ICLR 2025
Figure 24 shows the percentage of these levels the agent solves when given between zero and thinking steps prior to acting. If the agent planned via search, we would expect to struggle to solve these levels without additional test-time compute. This is because, plausibly, the agent's search procedure would take multiple steps to extend backward from the end of the corridor to the
100
corridor entrance (i.e. to inform the agent that should the case in Figure 24 Additionally, if the agent planned via search, we would expect more test-time compute to solve levels with longer corridor. This because, if the agent planned via search, it would presumably take longer for the search process to account for the effect of blocking off the corridor (i.c. by my- opically pushing box onto the target at the entrance) in levels with longer corridors. The expected pattern of the agent requiring more 'thinking steps' solve levels with longer ridors can be seen in Figure 24 For instance, the number of 'thinking steps' the agent re-
myopically). This is clearly
Levels Solved (%)
Corridor Length
60
take
40
10
20
14
Number of Th nking Steps
The percentage of the 80 levels in- troduced in Appendix [A.3.2] that the agent solves with different numbers of "thinking steps' (forced stationary steps prior to acting)
Figure 24:
quires to solve at least half of each set of levels increases with the corridor length of these levels. The agent requires thinking steps solve least half of the levels with corridors of length thinking step for levels with corridors of length 4, 2 thinking steps with corridors of length 10, and 3 thinking steps to solve levels with corridors of length 14.
Qualitative evidence of the agent using the additional test-time compute given to it by 'thinking steps' to perform a more thorough search can be seen in Figure [5] Figure 25 we visualize the agent's 's internal plan (as formulated in terms of the squares the agent expects step onto) at the final tick of each 5 additional steps of computation given to the agent when it performs 5 steps of "thinking time' prior acting in levels with corridors of length 14. In all of these levels, the agent at the third tick plans to step directly onto the box, myopically pushing it onto the target and making the level unsolvable. Note that this corresponds the agent's plan without 'thinking steps' and thus explains why the agent fails all of these levels by default. However, over the subsequent steps, the agent iteratively searches backwards from the box the corridor end. Once this search extends backwards onto the target at the corridor entrance, the agent seems to realize that it should myopically push a box onto this target as it needs instead step onto this target enter the corridor. The agent then alters its plan to instead push the box at the corridor entrance out of the way and enter the corridor.
37

---

Published as a conference paper at ICLR 2025
step 0, Bikk
stop 1, tick
stop 2, tick
step 3, tickk
step 4, tikk
(a) The agent initially plans to step up into the circled box, pushing the hox onto the circled target and blocking off the corridor. However, when given additional "thinking time", the agent extends its planned route backwards from the end of the corridor and realizes that it needs to step onto this target. It hence changes its plan so that
it will step onto, and thus push, the box right so that it enter the corridor.
step 0, 3
Step 1, tick
step 2, Hick step J, bck
step 4, tick3
(b) After third computational tick, the agent plans to step right and push the circled box on to the circled target. Over subsequent steps of 'thinking time', the agent plans backwards from the box at the end of the corridor and realizes that it needs to step into the corridor to reach this box. It thus instead plans to step down onto the box, moving it out of the way.
O
step O, BCK 3
step 1 tIck 3 step 2, BIC
step 3, 3
step 4. U
(c) Initially, the agent plans to step up. pushing the circled box onto the circled target. However, the agent extends its plan backwards from the box at the corridor end and realizes that it needs to step onto this target in order to enter the corridor and reach the box at the corridor end. As such, it alters its plan to first push the
cireled box left rather than myopically pushing it on to the target.
COOL step 2, Hick3
step 5ck
step 1 tick
step 3, B
step 4, bck3
(d) Without 'thinking steps', the agent would push the circled box right onto the circled target. However, during "thinking steps', the agent extends a path backwards from the box at the end of the corridor to the circled target. It then plans to instead push the circled box down so that it can follow this path.
1
1
D0
O step 2, tick
step D,
step 1, tick
step 3, Bick
step 4, tickk
(e) Initially, the agent plans pushing the circled box left onto the circled target. However, the agent extends the path it plans to follow to the box at the corridor end backwards, and realizes that needs to step onto the circled target to reach this box. The agent then alters its plan to first push the circled box down.
Examples of the agent's plan (in terms of CA) over extra steps associated with 5 'thinking steps' in levels with corridors of length 14 as introduced in Appendix A.3.2 During thinking steps, the agent searches backwards from the end of the corridor, and realizes that it must not myopically block this corridor off. The agent fails all of these levels when not forced to perform 'thinking steps', but, when given 5 'thinking steps' the agent successfully solves all levels. Yellow circles highlight the box and target for which the agent changes its plans. Teal arrows represent the direction that the agent plans to next move on to each square. The plans are decoded from the agent's 's cell state at its first (55)). second (2b) and 25c) and third (25d) and 25c) layer by Ix1 probe. The plans decoded at the final tick of each extra step performed during steps of 'thinking time'.
Figure 25:
38

---

Published as a conference paper at ICLR 2025
B ADDITIONAL INTERVENTION RESULTS
In Section we provided evidence indicating that the agent's representations of CA and CB were responsible for the agent's ' planning-like behavior. Specifically, in Section outlined the results of experiments in which we used the vector representations of CA and Cr learned by Ixl probes to intervene on the agent's cell state to foree the agent to formulate and execute sub-optimal plans in Agent-Shorten and Box- Box-Shortcut levels. The aim of these experiments was to demonstrate that the plans the agent internally formulated using its representations of C^ and CB causally influenced its behavior in the manner that would be expected of plans.
In this section, we provide further results regarding these experiments.
Appendix B.1 provides additional examples of the qualitative effect of the interventions from Section on the agent's 's internal plan.
Appendix B.2 outlines the results of additional intervention experiments ggent-Shortcut and Box- levels. These additional intervention experiments investigate altering the number of squares intervened upon, and introducing an intervention strength parameter.
Appendix B.3 details alternate intervention experiments in new set of handcrafted lev- els. These levels are designed to test the ability of interventions to force the agent to act optimally when it otherwise would not.
B.1 ADDITIONAL EXAMPLES OF INTERVENTIONS
In Section 6.1 we provided examples of plans as decoded from the agent's final layer cell state by a Ixl probe after the first 4 time steps of Box-Shorteut and Agent-Shortcut episodes both when we did, and when we did not, intervene on the agent's final layer cell state. We noted that, when visualizing the agent's plans as decoded by 1xl probes, we could see that the interventions had the effect of causing the agent to internally formulate and execute the desired type sub-optimal plan (e.g. a plan that involves either following longer-than- a box longer-than-necessnn illustrate this. Figure 27 provides additional example Box-Shorteut interventions and Figure 26 provides additional example Agent-Shortcur interventions.
path, or that involves pushing route to a target). We now provide additional examples to further
39

---

Published as a conference paper at ICLR 2025
(a) Plan without intervention
(b) Intervention
(c) Plan with intervention
(d) Plan without intervention
(c) Intervention
(f) Plan with intervention
G
G
(g) Plan without intervention
(h) Intervention
ii Plan with intervention
ii Plan without intervention
(k) Intervention
(I) Plan with intervention
(m) Plan without intervention
(n) Intervention
(o) Plan with intervention
(p) Plan without intervention
(g) Intervention
(r) Plan with intervention
(s) Plan without intervention
(t) Intervention
(u) Plan with intervention
Figure 26: Examples of Agent- interventions and their effects on the agent's ' internal plan.
Each row shows (1) the agent's internal plan after 4 steps in level without the intervention, (2) the initial state of that level, and the intervention performed, and (3) the agent's internal plan after steps in that level with the intervention. Plans decoded from the agent's final layer cell state by Ixl probe. Teal arrows mean the agent plans to next step onto the associated square in the associated direction. White arrows mark positions which the associated directional representations of CA are added to. White crosses mark positions of the agent's 's cell state that representations of NEVER for CA are added to.
40

---

Published as a conference paper at ICLR 2025
(a) Plan without intervention
(b) Intervention
(c) Plan with intervention
(d) Plan without intervention
(c) Intervention
(f) Plan with intervention
g Plan without intervention
(h) Intervention
ii Plan intervention
Plan without intervention
(k) Intervention
(I) Plan with intervention
(m) Plan without intervention
(n) Intervention
a Plan with intervention
(p) Plan without intervention
(g) Intervention
( Plan with intervention
(s) Plan without intervention
(t) Intervention
(u) Plan with intervention
Examples of Box-Shortch interventions and their effects on the agent's internal plan. Each row shows (1) the agent's internal plan after 4 steps in level without the intervention, (2) the initial state of that level, and the intervention performed, and (3) the agent's internal plan after steps in that level with the intervention. Plans decoded from the agent's final layer cell state by Ixl probe. Blue arrows mean the agent plans to push box of the associated square in the associated direction. White arrows mark positions which the associated directional representations of Cp are added to. White crosses mark positions of the agent's 's cell state that representations of NEVER for Cy are added to.
Figure 27:
41

---

Published as a conference paper at ICLR 2025
Algorithm 1 Agent-Shortcut Intervention
1: ShortRouteSquares -- All positions (I, y) on the short route
2: (ro, yo) + The first square (I, y) of the long route.
3: LongRouteSquaresDDis
'he first p squares (x,, y) that the agent would step onto if following the longer route, and the direction IDR it would step onto them
4: for in 1, Leigtte do
5: for (z, y) in ShortRouteSquares do
>> Short-route intervention
6:
((x,y) < ((x.y) + X CA
7: if Agent has not moved onto (Io. this episode then
8: for ((x, y). DIR) in LongRonteSquaresDiss do
> Directional intervention
CA
9: ((z,)) & C(r.y) + WoR
Algorithm Box-Shorteut Intervention
1: ShoueRoueequures +- All positions (Ir, y) on the short route
2: (zo; Ho) + The initial position (æ:, 3) of the box that is not adjacent to any targets. t- The first squares (x, y) that a box would be pushed off of if pushed the longer route, and the direction D DIR it would be pushed
3: LongRouteSquaresDDn
4: for in 1,2, 2.2.... ... EepisodeLnnghe do
5: for (x, y) in ShorrRouteSquares do
Short-route intervention
Cn
6:
C(R,y) ((x.y) + WNEVER
7: if Agent has not pushed box off of (zo, Mo) this episode then
do 9: C(2.9) & (lay) + +a X woek
8: for ((x, y), DIR)
> Directional intervention
B.2 ADDITIONAL INTERVENTION EXPERIMENTS: FURTHER AGENT-SHORTEE AND BOX-SHORTCUS INTERVENTION EXPERIMENTS
We now consider performing alternate intervention experiments in Box-Shortcut and Agent-Shorteut levels. To reiterate, Agent-Shortcii Shortcut levels are characterized by the agent having to choose to follow either longer or shorter path from its initial position region with boxes and targets. Similarly, Box-Shortcut levels are characterized by there being one box that can be pushed either a shorter or longer route target. In both levels, it optimal for the agent select the shorter option, and this is indeed what the agent does when not intervened upon. Thus, our interventions aimed to force the agent to formulate and execute sub-optimal plan involving choosing the longer option.
Our interventions in Box-Shortcut and Agent-Shortcut levels consisted of two sub-interventions:
Short-Route Interventions. These interventions aim discourage the agent from acting optimally and taking the shorter option. In Agent-Shortcut levels, the short-route interven- consists of adding the representation of NEVER for CA cell state positions along the short path the agent could follow. In Box-Shorteut levels, the short-route intervention consists of adding the representation of NEVER for Cp to cell state positions along the short route the box could be pushed along. This intervention is repeated at every time step.
Directional Interventions. These interventions aim to encourage the agent to act sub- optimally and take the longer option. In Agent-Shortcut levels, the directional intervention consists of adding the appropriate directional representation of CA to the first square the agent would step onto if followed the longer path. In nox-Shorteut levels, the directional intervention consists of adding the appropriate directional representation of C'A to the box s initial position to encourage it to pushed the long route. This intervention repeated at every time step until the agent either pushes the box off the initial squares (Box-Shortcut in- terventions) or steps onto the first square the long-route (Agent-Shortcut Shortcut Interventions).
The experiments in Section 6.1 that performed these two interventions did not investigate three possible axes of variation that could influence the success rate of interventions. First, the previous experiments simply added the un-scaled vector representations learned by 1xl probes the agent's
42

---

Published as a conference paper at ICLR 2025
cell state and did not consider the effect of introducing intervention strength parameter to scale representations by before using them for interventions. Second, the previous experiments did not consider varying the directional intervention specifically, they did not consider whether interventions become more successful if we intervene upon more squares along the longer path. Finally, they did not consider whether the interventions could be successful without the short-route intervention. We now consider the effect of these three factors on the success rate of interventions.
Algorithms [ and respectively provide pseudoscope for general Agent- and Box-Shorteut inter- ventions in which we (1) intervene on the first p squares of the long route as part of the 'directional' intervention, and (2) introduce an intervention strength a. Note that we can also choose not to per- form the 'short-route' intervention. The interventions in Section 6.1 correspond to algorithms and with p and a set to 1.
As with the experiments in Section 6.1] all experiments in this section are repeated with 5 indepen- dently trained and initialized probes, and interventions are considered a success if they cause the agent to solve the level in the desired, sub-optimal way.
B.2.1 VARYING THE NUMBER OF DIRECTIONAL INTERVENTIONS
First, we also consider intervening in Agent-Shorteu and the number of squares intervened upon as part of the directional intervention along the 'long' route. Specifically, we vary the number of squares intervened upon in 'directional' interventions between O squares and 3 squares. When intervening upon an additional square on the 'long route' we intervene on the square following the already-interveneds upon squares. That is, we vary the value of p between O and 3 in algorithms [ and 2 For instance, when we intervene upon two squares in Agent- levels, we intervene upon the first two squares the agent would step onto if it followed the longer path. Then, when intervening upon three squares we would additionally intervene on the third square the agent would step onto if it followed the longer path. We also consider varying
experiments while varying
Figures 28 and 29 show the success rate when intervening on the agent in Agent-Shorteut Shortcut and Box- Shortcut levels when varying the intervention strength and the number of squares intervened upon in the longer route. A few observations can be made from these figures.
First using too high too low of an harms the intervention success rate. This would be expected: when a is too low, interventions will not meaningfully change the agent's 's internal concept represen- tations, while when a is too high, the intervention will cause the agent's internal activations to go off-distribution.
L Likewise, performing additional interventions the long path improves intervention success rate for low a, but harms the success rate for high a. We posit this is because, to alter the agent's 's concept representations for a low a, additional interventions are useful. However, when a is high, these additional interventions cause the agent's 's activations to go further off difriisttibutinn, impeding the ability of the intervention to steer the agent.
B.2.2 REMOVING THE SHORT- ROUTE INTERVENTION
We also considered intervening on Agent-Shorteut and Box-Shortcut levels when not performing any short-route interventions and solely performing directional interventions. These directional- only interventions aimed assess the extent to which the agent's planning mechanism is driven by avoiding planning into squares it represents as having the class NEVER as opposed to extending plans it constructs with the directional concept classes.
Figures 30 and 31 show the success rate when intervening on the agent in Agent- Agent-Shortcct and Box- Shortcut levels when varying the intervention strength and the number of squares intervened upon in on the longer route. Clearly, intervening without the short-route intervention is less successful. However, the reduction in success rate relative to Figures and 29 significantly more pronounced for Agent-SUn Shent-Shortcut interventions than for Box-Shorteut interventions. We thus hypothesize that the agent utilizes a different planning mechanism when planning paths for to follow as opposed to planning routes push boxes. Specifically, the agent's 's path-plas planninlnnn mechanism seems to be more driven by avoiding certain squares (e.g., those it represents using the class NEVER of Cx) than the agent's iss box-route- poatee-lannnng mechanism.
43

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer
100
80
a=
60
0
80
a=0.5
60
40
20
0
Intervention Success Rate (%)
100
80
60
0
100
80
60
40
20
0
100
80
60
20
0
2
Number of Squares Intervened Upon On The Long Path
Probe Type
T
trained
E random
Figure 28: Success rate when intervening in Agent-Shorteut levels when varying the number of cell state positions intervened along the 'long path' during the 'directional' part of the interven- tion. Interventions are performed using the vector representations of CA learned by 1xl probes. Interventions are performed using different intervention strengths on the agent's cell state each of its ConvI STM layers. For each layer, intervention strength, and number of squares intervened upon, we repeat the intervention 5 times using independently trained probes and report the average success rate. We compare interventions performed with trained probes to interventions performed with randomly-inttillzzed probes. Error bars report +1 standard deviations.
44

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer
100
80
a=0.25
40
20
0
100
a=0.5
60
40
20
0
Intervention Success Rate (%)
100
80
40
20
0
100
60
40
20
0
100
80
60
20
0
1
0
2
Number of Squares Intervened Upon On The Long Path
Probe Type
trained
random
Figure 29: Success rate when intervening in Box-Shorteut levels when varying the number of cell state positions intervened on along the 'long path' during the 'directional' part of the intervention. Interventions are performed using the vector representations of C3 learned by 1xl probes. Inter- ventions are performed using different intervention strengths a on the agent's cell state each of its ConvISTTM layers. For each layer, intervention strength, and number of squares intervened upon, we repeat the intervention times using independently trained probes and report the average suc- cess rate. We compare interventions performed with trained probes to interventions performed with randomly-inttialized probes. Error bars report +1 standard deviations.
45

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer 3
100
Q=0.25
40
20
0
100
80
60
40
20
0
Intervention Success Rate (%)
100
80
a=1
40
20
0
80
60
40
20
0
100
80
20
0
3
0
2
Number Squares Intervened Upon On The Long Path
Probe Type
trained
random
Figure 30: Success rate when intervening in Agent-Shorteut levels but not performing the "short- route' part of the intervention. Identical to Figure 28 otherwise.
46

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer 3
100
Q=0.25
60
40
20
100
80
60
20
0
Intervention Success Rate (%)
100
60
a=1
40
20
0
100
80
60
20
0
100
60
40
20
0
3
0
2
Number Squares Intervened Upon On The Long Path
Probe Type
trained
random
Figure 31: Success rate when intervening in Box-Shorteut levels, but performing the "short- route' part of the intervention. Identical to Figure 29 otherwise.
47

---

Published as a conference paper at ICLR 2025
+
(a) 'Agent-One Only' Intervention
(b) -Box-Only' Intervention
(c) Agent-and -nn--oxx-oox Intervention
Figure 32: Examples of (a) 'Agent-Only', and (b) 'Box-Only', and (c) 'Agent-and-He
interven- tions in a Cutoff level. We add the associated directional representation of CA the position with the teal arrow (e.g. in this example the representation of UP for C2). We add the associated directional representation of CB the position with the blue arrow (e.g. in this example the representation of RIGHT for CB).
B.3 ADDITIONAL INTERVENTION EXPERIMENTS: INTERVENING IN A NEW SET OF LEVELS To ENCOURAGE OPTIMAL BEHAVIOR
We also considered performing interventions in a different set of levels. We call these levels 'Cutoff" levels. Cutoff levels follow a schematic very similar the levels introduced in Appendix [A.322 and are designed in such a way that solving them requires the agent to forsee the long-run consequences of its actions. Specifically, these are levels in which a box is adjacent to a target at the entrance of corridor. The agent always begins levels one square removed from this box. At the end of this (variable-length) corridor is another box adjacent to another target. To solve these levels, the agent must not act myopically i.e. it must not immediately push the box at the corridor entrance onto the adjacent target as doing so would block the corridor and make the level unsolvable and must instead push the box out of the way so that it can enter the corridor and reach the box at the corridor end.
We use dataset of 200 Cutoff levels. These were created by designing 25 Cutoff levels (with corri- dors of varying lengths), and then making copies of each by applying vertical reflection and 90", 180", and 270" rotations. Figure 33 shows the percent of these levels the agent solves when given vary- ing number of 'thinking steps'. Recall that 'thinking steps' refer to steps at the start of episodes where the agent forced to remain stationary prior to acting. By default that is, without any thinking time - the agent solves none of the 200 Cutoff levels. However, the agent can solve the vast majority of Cutoff levels when given additional test-time compute to refine its plans. Thus, consider intervening in Cutoff lev- els with the aim of artificially replicating the effect of additional test-time compute. That is, we perform interventions that aim to aid the agent in solving the levels without additional decision-time compute. We investigate these interventions in order to de- termine whether we can intervene on the agent's internal plan to aid it in forming and executing optimal plans. This is in contrast to Section 6. in which we focused on intervening on the agent to encourage it to form and execute sub-optimal plans.
100
Cutoff Levels Solved (%)
80
60
40
20
2
Number of Thinking Steps
Figure 33: The percentage of 200 Cutoff lev- els the forming different numbers of 'thinking steps' prior to acting.
agent solves when per-
We perform three types of interventions in Cutoff levels: 'Agent-and-Brn interventions, 'Agent- Only' interventions, and 'Box-Only' interventions. 'Agent-Only' interventions consist of adding an appropriate directional representation corresponds to the agent stepping into the corridor. An example is shown in Figure 32a The motivation for this intervention is that it should correspond to the agent having extended its plan back fully from the end of the corridor such that the agent knows it should not block the corridor off. If this the case, the agent should then plan to not myopically push the box onto the target at the corridor entrance. 'Box-Only' interventions consist of adding appropriate directional representation of
CA to the target at the entrance of the corridor that
48

---

Published as a conference paper at ICLR 2025
CB to the initial position of the box that could be myopically pushed into the corridor entrance to encourage the agent to instead push this box out of the way of the corridor entrance. An example is shown in Figure 32b] The motivation for this intervention is that it should remove the need for the agent to complete planning backwards from the end of the corridor. Finally, 'Agent-and-Boy interventions consist of the conjunction of 'Agent-Only' and 'Box-Only" interventions. An example
is shown in Figure 32c
In all cases, the interventions are performed by adding the corresponding representations learned by Ixl probes to the agent's cell state at one of its ConvLSTM layers. The interventions are repeated each computational tick until the agent moves the box the corridor entrance which point the interventions cease. As previously, we repeat all interventions with 5 trained and 5 randomly initialized probes. Since the agent cannot solve any Cutoff levels without steps of "thinking time', we consider an intervention to be success if it leads to the agent solving the level.
Figures 34a 34b and 34c respectively show success rates when performing 'Agent-Only' 'Box- Only' and 'Agent-and-Bod tervention strengths a. A few key takeaways can be drawn from these results. First, and most importantly, these interventions can replicate the effect of additional test-time compute and allow the agent to solve levels would otherwise fail. That the concept representations of CA and C3 decoded by our Ixl probes can be used to intervene on the agent in such way that induces be- havioral effect comparable to that of additional test-time compute is further evidence that the plans decoded by the probes causally influence the agent's 's behavior in the way that would be expected.
interventions on the agent's ' cell state at each layer using different in-
Another interesting takeaway from these results is that the pattern of success rates across layers is notably different for 'Agent-Only' (Figure (3a) and 'Box-Only' (Figure (34b) interventions. This suggests potential insights into the role of each ConvLSTM in the planning process. For instance, note that interventions on layer using representations from trained probes are highly successful when performing 'Agent-Only' interventions but are no better than bascline interventions with ran- dom probes when performing 'Box-Only' interventions. This suggests the agent performs some type of 'plan conflict detection' computation either at layer 2, or between layers 2 and 3, such that intervening on layer 2 to encourage the agent to plan to step onto the target causes the agent to detect that acting myopically would conflict with its plans. Likewise, 'Box-Only' interventions are much more successful than 'Agent-Only' interventions at layer 3. This indicates that agent does not perform 'plan conflict detection' computations layer 3 since, if it did so we would expect 'Agent- Only' interventions to be successful but does update the routes it plans to push boxes at this layer.
C ADDITIONAL TRAINING-TIME INTERPRETABILITY RESULTS
In Section 6.2 we briefly investigated the emergence of planning-relevant representations during training. Precisely, we demonstrated the co-occurrence during training of (i) the agent's internal representations of CA and CB in its final layer cell state, and (ii) the ability of the agent to perform better when given additional test-time compute. In this section, we now detail experiments which we further interpret the agent during the early stages of training. Specifically, we show that:
The agent's internal representations of CA and Cn largely emerge at the start of training (Appendix C.I).
The agent's ability to refine its internal plans when given additional test-time compute emerges early on in training (Appendix C.2
As shown for the final layer in Section 6.2 the emergence of the agent's representations of CA and Cp in its cell state all layers coincides with the emergence of its ability to perform better when given additional test-time compute (Appendix C.3
The emergence during training of the agent's ability to improve its internal plans when given extra test-time compute coincides with the emergence its ability to perform better when given this extra compute (Appendix C.4.
49

---

Published as a conference paper at ICLR 2025
100
Success Rate (%)
50
Layer Layer Layer
Layer Layer Layer
Layer Layer Layer
Layer Layer
Layer
(a) 'Agent-Onf Intervention Success Rates
100
Success Rate (%)
0-
Layer Layer Layer
Layer Layer Layer a-0.5
Layer Layer Layer 6-1
Layer Layer Layer 9-2
Layer Layer Layer Q-4
g-0.25
(b) 'Box-Only' Intervention Success Rates
100
Success Pate (%)
Layer Layer Layer a=0.25
Layer Layer Layer a=0.5
Layer Layer Layer o=1
Layer Layer Layer
Layer 1 Layer Layer
Probe Type
Trained
Kandem
(c) AAgntt-and-Bon Intervention Success Rates
Figure 34: Success rate when intervening on the agent in Cutoff levels. We consider 'agent-only', 'box-only' and "agent-and-byl and-b-- interventions'. These interventions are respectively performed using the vector representations of CA. CB. and both CA and CB. as learned by ix1 probes. Interventions are performed using different intervention strengths a on the agent's cell state at each of Con- vLSTM layers. For each layer and intervention strength, we repeat the intervention 5 times using probes and report the average success rate. We compare interventions performed with trained probes to interventions performed with randomly-initialzze probes. Error bars report +1 standard deviation.
0.8
No.ro F1
0.6
Nacro
0.6-
0.4
0.4
0.24
0.2
20
30
Number of Trainings Transitonss (Allloon)
Number Trainings
Transitioss (1A lion)
Layer
Layer
Layer
Layer
Layer
---
Layer
Loyer
Layer
(a) CA
(b) CR
Figure 35: Macro Fl scores achieved when training 1xl probes to predict (a) CA and (b) Cs using the agent's cell state activations at each layer at checkpoints every million transitions over the first 50 million transitions of training.
50

---

Published as a conference paper at ICLR 2025
C.I INVESTIGATING THE EMERGENCE OF PLANNING- RELEVVAT CONCEPT REPRESENTATIONS DURING TRAINING
First, we investigate the emergence of the planning-relevant concepts we study - CA and C3 - over the course of training. We do this by taking checkpoints of the agent every 1 million transitions over the first 50 million (of 250 million total) transitions of training. For each checkpoint, we train Ixl probes to predict C'A and CB using the agent's 's cell state activations at that checkpoint. That is, we train and test probes on datasets generated by collecting the agent's cell state activations when running the agent that checkpoint. Note that re-training probes each checkpoint is important since CA and CB are behavior-dependent concepts in that the classes they assign to squares depends on how the agent behaves (which itself changes during training as the agent's 's parameters update).
Figure 35 shows the macro FI scores achieved when training probes to predict C'A and CB based on the agent's cell state activations over the course of the first 50 million transitions of training- Clearly, the agent's internal representations of these concepts largely emerge early on in training. However, we note that, even after 50 million transitions, the macro Fls achieved by our probes are still somewhat lower than the macro Fls achieved when probing the fully-trained agent as shown in Figure 4 This suggests that the agent does slightly improve its representations of these concepts over the entire course of training.
C.2 INVESTIGATING THE EMERGENCE OF TEST-TIME PLAN REFINEMENT DURING TRAINING
In Section 5 we used Figure to demonstrate that the agent can use additional test-time compute the start of episodes refine its internal plan. Thus, a natural question to ask is when during training does this ability emerge is this an ability that emerges early on in training and that is gradually improved, or is an ability that the agent only develops towards the end of training?
We can investigate the emergence of the ability to refine plans when given additional test-time com- pute by repeating the setup from Figure - i.c. predicting the agent's future actions using its internal plans decoded from its cell state by 1xl probe during steps of "thinking time' whilst the agent is forced to remain stationary prior to moving- but now whilst doing so for checkpoints of the agent taken throughout training.
Specifically, we now use the 1x1 probes from Appendix C. to decode the agent's plan when given additional test-time compute. As before, we use checkpoints of the agent taken every 1 million transitions during the first 50 million transitions of training. We force the agent each checkpoint to perform 5 'thinking steps' prior to acting at the start of 1000 episodes and use 1xl probes to decode the agents internal representations of C' and Cg at each the 15 corresponding internal computational ticks. We view the predictions made by probes at each tick as being the agent's internal plan at each tick. We then measure the average correctness of the agent's plan at each of the 15 ticks by averaging the macro Fl of the probe's predictions at each tick. Finally, we measure the extent to which the agent can utilize the extra test-time compute afforded by 'thinking time' by measuring the increase in average macro Fl when using the probe's 's predictions at the 15th tick relative to at the 1st tick.
Figure 36 shows the increase in macro Fl for different checkpoints of the agent when using probe predictions from the 15th tick of thinking time relative to the 1st tick of thinking time to predict CA for each checkpoint. Figure shows the analogous results when predicting CB for each checkpoint. Clearly, the agent acquires the ability to use additional test-time compute to refine its 'internal plan' (i.e. its internal representations of CA and CB) early on training.
C.3 INVESTIGATING THE CO-EMARGENCE OF PLANNING- RELEVVNT CONCEPT REPRESENTATION AND PLANNING-LIKE BEHAVIOR DURING TRAINING
we illustrated that (i) the agent's internal representations of CA and Op in its final layer cell state and (ii) the ability of the agent to perform better when given additional test-time compute emerge concurrently during training. This naturally leads to the question of whether the emergence of this type of planning- like behavior coincides with the emergence of the agent's 's representations of CA and Cp at all layers. We now provide evidence that this is indeed the case.
In Section 6.2
51

---

Published as a conference paper at ICLR 2025
0.4
0.4
0.3-
N/
p mm
Increase in Maxro F3
0.3-
0.3
0.2
0.2
0.2
Increase in
0.1
0.1
0.1
-0.1
-0.
-0.
10
20
30
40
10
20
30
40
50
30
20
30
40
50
Number of Taining Transtt ons (Million)
Number Traningg Trasitions (Milion)
Humber of Taaning Tranitiooss (Milion)
(a) Layer
(b) Layer
(c) Layer
Figure 36: Increase in macro Fl when predicting the agent's future movements (Cx) using the agent's internal plan at each layer as decoded at the 1st and 15th tick. Each data point corresponds to the increase in macro FI between the first and last tick performed during 5 thinking steps for a checkpoint the agent taken every million transitions during the first 50 million transitions of training. Here, the 'internal plan' layer for tick corresponds to the agent's representation of CA as decoded by a Ixl probe applied to the agent's 's cell state at that layer at that tick.
0.4
0.4
1/
Increase in Macro F1
0.3
0.3
0.3
0.2
0.2
Increase in
0.1
0.1
0.1
-0.1
-0.1
-0.1
1.
20
30
40
10
20
30
50
10
70
30
40
50
Number of Training Transit ons (Million)
Number Trinnng Transtions (Millon)
Number of Taining Transiionns (Millon)
(a) Layer
Layer 2
(c) Layer
Figure 37: Increase in macro Fl when predicting future box movements (Cg) using the agent's s internal plan each layer as decoded the and 15th tick. Each data point corresponds to the increase in macro F1 between the first and last tick performed during thinking steps for checkpoint of the agent taken every million transitions during the first million transitions of training. Here, the 'internal plan' layer for tick corresponds to the agent's representation of Cy as decoded by a Ix probe applied to the agent's cell state at that layer at that tick.
52

---

Published as a conference paper at ICLR 2025
Additional Levels
Additional Levels
Additional Levels Solved (9)
Solved (96)
(86)
4-
2-
408
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Macro
Macro
Macro Fl
Ca
Cp
Ca
Ca
Cp
(a) Layer
(b) Layer
(c) Layer
Figure 38: The relationship between (i) the percentage of extra medium levels solved when agent given steps to 'think', and (ii) macro F1 score of probes when predicting CA (blue) and Cp (orange) from the agent's ' cell state at each layer different checkpoints taken during training. Each point corresponds to these quantities calculated for single checkpoint.
As in Section 6.2 we do this by inspecting both (i) the macro of probes trained to predict CA and CB using the agent's 's cell state activations at all layers, and (ii) benefit from additional test time compute. We measure these quantities at checkpoints of the agent taken ever I million transitions over the first 50 million transitions of training. As in Section 6.2 we measure the ability of the agent to benefit from additional test-time compute by counting the number 1000 medium-difficull levels from the Boxoban dataset (Guez et al. 2018a the agent cannot solve by default, but can solve when given 5 'thinking steps'.
The effect illustrated in Figure 9 namely, that the period of training in which the agent begins to solve additional levels when given extra compute is the same as the period in which its repre- sentations of CΛ and CB become increasingly well-developed can be seen across all layers. This can be seen in Figure 38 in which we plot the relationship between (i) the percentage of additional medium levels solved and (ii) the macro F1 when training probes predict CΛ and Cn using the agent's 's cell state activation at each layer for that checkpoint. As would be expected if the agent's s representations of CA and Cp were used part of an internal planning process, the emergence of these representations at all layers is clearly related to the emergence of the agent's ' ability to benefit from extra test-time compute.
C.4 INVESTIGATING THE Co- NO-EMBRGENCE TEST-TIME PLAN REFINEMENT AND PLANNING-LIKE BEHAVIOR DURING TRAINING
Yet, Figure 38 only shows that the emergence of the agent's 's representations of CA and C3 is related to the emergence of the agent's planning behavior. It hence does not show that the planning process the agent uses these representations as part of is related to the behavioral evidence of planning exhibited by the agent.
However, recall that in Appendix C.2 we showed that we could inspect the emergence the agent's internal planning process by inspecting the manner in which the plans the agent internally repre- sents in terms of CA and C3 become more accurate when given 'thinking steps'. Specifically, we can use probes
decode the agent's internal plan at the 1st and final tick of additional compute given by 5 steps of "thinking time' and measure the correctness of these plans using the macro Fl achieved when viewing these plans as predictions of the agent's future behavior (CA) and its effect on the environment (Cp). We can then measure the extent to which the agent is internally using these representations as part of internal planning process by measuring the increase in macro achieved when using the agent's 's plans after the first tick and after the final tick of 'thinking time'. the agent uses its representations of C'A and C3 as part of an internal planning process, the agent ought to be able to the extra compute associated with 'thinking steps' to form better plan with these representations. That is, if these representations are used for planning, we expect the macro F1 of the agent's internal plan to increase during 'thinking steps'.
Figure 33 shows, for checkpoints of the agent taken over the first 50 million transitions of training, the relationship between (i) the percentage of addition levels the agent solves with 5 'thinking steps',
53

---

Published as a conference paper at ICLR 2025
Additional Levels
Additional Levels
Additional Levels
Solved (96)
(96)
Solved (96)
2-
0-
0800
0.0
0.2
0.4
0.0
0.2
0.0
0.1
0.2
0.3
Increase Macro F1
Increase Macro
Increase in Macro F1
CA
Cp
Ca
Ca
Ca
Gp
(a) Layer
(b) Layer
(c) Layer
The relationship between (i) the percentage of extra medium levels solved when agent given 5 steps to "think', and (ii) the increase in macro Fl when predicting CA (blue) and Cp (orange) from the agent's cell state at each layer between the 1st and 15th computational tick of 5 "thinking steps' at different checkpoints taken during training. Each point corresponds to these quantities calculated for single checkpoint.
Figure 39:
and (ii) the increase macro Fl when using the agent's internal plan the 1st and 15th tick of steps of "thinking time' to predict future agent movements (Ca) and box movements (Cg). As would be expected if the agent used its representations of CA and C3 for planning, checkpoints which the agent can use extra test-time compute to improve its internal plans are also checkpoints at which agent solves more levels when given extra compute.
D ADDITIONAL PROBING RESULTS
Our methodology is underpinned by use of linear probes. In this section, provide further details regarding the probes we use and further relevant experimental results. This section is organized follows:
Appendix provides additional details regarding how we train probes.
Appendix D.2 provides additional metrics specifically, class-specific recalls, precisions and Fls for the Ix1 and 3x3 probes we discuss from Section 4.2
Appendix details the macro scores achieved when using larger probes.
Appendix D 4 outlines the performance of Ixl and 3x3 probes trained to predict alternate square-level concepts to CA and CB-
Dopendix(D..pprvvdde the results of applying "global" probes that receive the agent's entire cell state as input to predict the agent's 's future actions directly.
D.1 PROBE TRAINING DETAILS
In this section, now provide brief overview of the manner in which the probes considered in this paper are trained. All probes are trained for 10 epochs using the AdamW optimizer (Loshchilov Hutter 2019) and implemented as convolutions with batch size of 16, learning rate of 0.001 and a weight decay of 0.001. We train probes to predict the concepts assigned to Sokoban squares using the agent's cell state activations both after training, and at checkpoints taken during training. As concepts we study depend the agent's 's behavior and thus the agent's 's parameters, we train and test probes on different datasets when investigating the agent at different points of training.
All probes trained predict concept classes using the fully-trained agent's 's cell states trained and tested on datasets consisting of 106.6k and 25.7k transitions. The training dataset is generated by running the fully-trained agent for 3000 episodes levels from the Boxoban unfiltered training dataset (Guez et al. 80). The test dataset is generated by running the fully-trained agent for 1000 episodes on levels drawn from Boxoban unfiltered validation dataset.
All probes trained to decode concepts from the agent's 's cell state at checkpoints taken during training trained and tested checkpoint-specifie datasets. For any checkpoint, the training and test
54

---

Published as a conference paper at ICLR 2025
Class NEVER
Metric
Layer
Layer 2
Layer
Baseline
F1
0,9787 + 0.0000
0.9821 0.0001
0,9845 = 0.0000
0.9049 = 0.0000
Precision Recall
0.9795 + 0.0002
0.9838 + 0.0004
0.9826 + 0.0001
0.8279 = 0.0000
0.9779 + 0.0001
0.0003 0.9804
0.9864 + 0.0001
0.9978 = 0.0000
UP
F1 0.7563 + 0.0004
0.8187 + 0.0004
0.8396 + 0.0003
0.0000 = 0.0000
Precision
0.7405 + 0.0026
0.8238 0.0011
0.8237 = 0.0019
1.0000 = 0.0000
Recall
0.7728 + 0.0025
0.8137 + 0.0008
0.8561 + 0.0019
0.0000 = 0.0000
DOWN
F1 0.7548 = 0.0003
0.8278 + 0.0007
0.8255 = 0.0004
0.0000 = 0.0000
Precision
0.7516 - 0.0061
0.8216 0.0039
0.8440 = 0.0021
1.0000 = 0.0000
Recall
0.7581 + 0.0066
0.8342 + 0.0026
0.8078 + 0.0018
0.0000 + 0.0000
LEFT
F1
0.7985 = 0.0004
0.8282 0.0002
0.7955 = 0.0005
0.0000 0.0000
Precision
0.7958 + 0.0019
0.8126 + 0.0020
0.8199 + 0.0019
1.0000 + 0.0000
Recall
0.8013 0.0021
0.0025 0.8445
0.7726 + 0.0010
0.0000 = 0.0000
RIGHT
Fl
0.7238 + 0.0007
0.8228 0.0003
0.8129 + 0.0002
0.2262 - 0.0000
Precision
0.7362 + 0.0029
0.8181 + 0.0029
0.8131 + 0.0010
0.2707 = 0.0000
Recall
0.7119 + 0.0035
0.8277 + 0.0033
0.8128 + 0.0011
0.1943 = 0.0000
Table 2: Average and standard deviation of class-specific performance metrics when probing for CA using Ixl probes.
datasets are created by collecting transitions when running the agent at that checkpoine for 1000 and 500 episodes in the unfiltered training and unfiltered validation Boxoban datasets respectively. These dattasen are of different sizes for each checkpoint as the agent's behavior changes over the course of training
D.2 ADDITIONAL PROBING METRICS
In Section we invessiiieeee training Ixl and 3x3 probes to predict CA and Cg. Figure 4 illus- Seeeuti
trated the macro Fls achieved by these probes. We now provide additional metrics for these probes. Specifically, we provide, for each class, the precision, recall, and F1 achieved by our probes when single negative class. Since train 5 probes with different antiaalization seeds, we report both the mean and standard respectivvll show these metrics for Ix1 and 3x3 probes and respectively despecctivvll show these metries for Ixl and 3x3 probes trained
viewing that class as the positive class and all other classes as belonging
deviation for all metries. Tables 2 and 3 trained to predict CA. Tables 4 to predict Cp.
55

---

Tublishee as a conference paper at ICLR 2025
Class
Metric T
Layer 1
Layer 2
Layer
Tayer's
Baseline
NEVER
Frecisaal
Fl
0.9862 + 0.0001
0.9865
= 0.0003 0366555000
0.9874 + 0.0002
0.9882 + 0.0004
0.9877 = 0.0003 0003
0.9877 + 0.0003
0.8887 = 0.0015 0
0.9294 00294-2
= 0.0005
Recali
0,9860 = 0.0001
0.9866, + 0.0002
0
0.9878 + 0.0005
0.9739
= 0.0008
40004
U
Precision
Fl
0.8614 = 0.0000
0.8619 = 0.0046
0.8614 = 0.0004
0.8568 + 0.0024
0.8735 + 0.0013
0.8669 + 0.0040 0
0.3300 = 0.0110
0.4682 = 0.0092
Recall
0.8610 = 0.005)
0.8660 + 0.00200
0.8802 + 0.0020
0.2553 = 0.0151
0.8662 = 0.0035 60006
DOWN
Peecisaio
Fl
0.8656 = 0.00300 0000000000000
0.8616 + 0.0006
0.8677 08677-1 + 0.0006
0.8734 0.8734 + 0.0005
0.8791 = 0.0033 3
0.3464 = 0.0083
0 = 4O0RS 1
Recall
0.8576 = 0.001
0.8691 = 0.0026
0.8678 = 6.0026
= 0.051 0.2907
0.4297
LEFT
FL Preciscol
0,8606 = 0.0005
0.8654 = 0.0002
0.8728 = 0.0003
0.3528 = 0.0045 8
Recall
0.8570 = 0.003..
0,8641 = 000029
0.8579 = 0.0026
08831 + 0.0028
0,8741 = 0,0017
0,8710 + 0.0016
0.4541 = 0.0020
028852 = 0.0070 0,2885
RIGHTT
e and 0.8491 + 0.0031 1
Fl Eorssoon
0,8536 + 5.0005
0.8626 + 0.0013
0,8713 + 0.0015
0.3524 = 0.0085
Recall
0,8581 = 0.0026
0.8653 + 0.0046
0.8704 + 0.0039
0.8722 = 0.0052
07704
0.2848 = 0.0136 333660
0.4631 = 0.0091
0.8600 + 0.0024
standaan standard deviation deviation of class-specific class-specific performance metrics when prohing for CA corfoormancce
Table Table3 3: Averade
using Asing.. 3x3 probes.
Metrie e
Bassline
Class
Layer 1
Layer 2
Layer
NEVERR
-
-
0.9634 = 0.0000 0 -
Precissal
Fl
0.9907 + 0.0000
0.9906 + 0.0002
0.9943 + 0.0001
0.9943 + 0.0000
0.9913 + 0.0001
0,9913 + 0.0000
0.9311 = 0.0000 0000
Recall
0,9909 = 0.0002
0.9942 + 0.0001
0.9980 - = 0.0000
0.9912 = 0.0001
UP
Precisson Recull
Fl
0.8027 + 0.0008
0.9126 +: 0.0002
0.8805 +: 0.0002
0.0000 = 0.0000
0.7968 = 0.0039
0.9054 + 09054
0.0016
0.8089 = 0.0041 1
0,9200 + 000017
0.9061 = 0.0013
0.8562 + 0.0008
0.0000 = 100000
1.0000 = 0.0000
0.0000 0
DOWN
0.8386 + 0.0003
02257
00,003
0.8590 + 0.0004
Preciscon
F1
0.8407 = 000081
0.9327 = 0.0010
0.0000 + 0.0000
Recall
0.8366 = 000032
09188- + 00.0044
0.8741 = 0.0024
08444= = 40015
0.0000 = 0.0000
1.0000
= 0.0000 -
1000000000
LEFT
Reccision
Fl
0,8968 =
0.8940 = 00000
0,8954 = 0.0000
0.9247
0.9301 + 000016
0.19442 - 000010
000003
0.8144 = 0,0008
0.8058 = 0.0007
0.7955 = 0.0012
0.0000 - = 0.0000
0.0000 = 0.0000
1.0000 = 0000
0000
Recall
0.0000
RIGIT
Fl Preccsson Secall
0.8111 + 0.0008 -
0.9162 + 5
0,8315 +: 0.0005
0.3079 + 0.0000
0.8182 + 0.0020
09194- + 0.0008
0.2707 + 0.0000 0000
0.8321 + 0.0016
0,8041 + 0.0034
0,9131 + 00013
0.8310 = 0
0.3570 = 0.000
Table 4: Average and standard deviation deviation of sassssppehht performance metrics when using 1xl probes.
when probing g for C..
56

---

Published as a conference paper at ICLR 2025
Class NEVERR
Mettic
Layer 1
Layer's
Baseline
Layer 2
Layer
0,9950 + 0.0000
0.9664 = 0.0001 1
0.9436 = 0.0003 -
Freeisson
F1
0.9956 + 0.0001
0.9949 +: 0.0000
Recall
0.9952 = 0.0002
0.9962
0.0001
0.9949 = 0.0002
0.9948 = 0.0002
0.9951 + 0.0001
0.9949 + 0.0002
0.9904 = 0.0004
U
Precisson F
F1
0.9201 = 0.0005
0.9313 = 0.0008
0.9230 = 0.0008
0.3976 = 0.0108
1
0.9127 = 0.0016
0.9219 0.0007
0,9160 = 0.0021
0,5518 = 0.0121
Recall
0.9277 + 0.0012
0.9409 + 0.0012
0.9302 + 0.0014
0.3112 = 0.0166
DOWA
F1
0.9312 + 0.0003 0.0003
0.9347 0.0015
0.9292 + 0.0003
0.4246 = 0.0074
Precision
Recall
0.9248 0.0030
0.9376 + 0.0027
0.9334 0.0022
0,9360 + 0.0038
0,9313 = 0,0017
0.9272 + 0.0014
0.3376 = 0.0144
0.5733 = 0.0153
LOFT
Fl
0.9208 + 0.0006
0.9346 0.0006
0.9160 + 0.0007
0.4068 + 0.0039
Precision
Recall
0,9233 = 0.0019
0.9182 = 0.0020
0.9246 + 0.0021
0.9448 09448= = 0.0019
0.9196 + 0.0026
0.9125 = 0.0038
00125-
0.3109 = 0.0055
0.5884 = 0.0054
RIGHTT
0.4190 + 0.0053
Preeisson
F1
0.9179 + 0.0005
0.9384 + 0.0007
0.9317 +: 0.0004
0.5819 = 0.0125
Recall
0.9206 + 0.0044
0.9153 + 0.0052
0.9378 + 0.0020
0.9391 + 1
0.9338 = 0.0030
0.9297 = 0.0027
00299722
0.3276 = 0.0098
Table 5: Average and standard deviation of class speciiit performaane metrics when probing for Cs using 3x3 probes.
class-spechi
57

---

Published as a conference paper at ICLR 2025
0.4
0.2
0
Layer
Layer
Layer
Baselire
Kernel
1x1
3x3
5x5
7x7
(a) CA
0.8
0.6
0.4
0.2
0
Layer Layer
Layer
Kernel Size
1x1
3x3
5x5
7x7
(b) C
Figure 40: Comparisons of macro F1 achieved when probing the cell state each layer of the agent for 'Agent Approach Direction' (Cx) and 'Box Push Direction' (CB) using 1x1, 3x3, 5x5 and 7x7 probes. Reported Fl scores are averaged over five independent training runs. Error bars report standard deviations.
D.3 PROBING USING LARGER PROBES
In addition probing the agent's ConvLSTM cell states using 1x1 and 3x3 square-level concepts, we now consider using '5x5' and '7x7' probes. These are probes that are identical to the 1xl and 3x3 probes considered previously, but that predict the concept class of square (x, y) using the activations in, respectively, 5x5 and 7x7 grids about the (x, y) position of the agent's ' cell state. We investigate these probes in order investigate whether the agent represents square-level concepts in spatially-localizz way at individual positions of its cell states, or whether it represents square-level concepts in a distributed way across multiple cell state positions. As previously we also consider baseline versions of these probes trained on the raw observation
Figure 40b shows the macro when using 1x1, 3x3, 5x5 and 7x7 probes. This figure illustrates that increasing the probe size leads to minimal gains in performance when probing the agent's 's cell state relative to the increase in the performance of the bascline probes. We take this as evidence that the agent indeed localizes its representations of square-level concepts at individual cell state positions.
D.4 PROBING FOR ALTERNATIVE SQUARE-LEVEL. CONCEPTS
In this section, now investigate the extent which alternative concepts the 'main' concepts
considered in the paper 'agent approach direction' (CA) and 'box push direction' (Cg) - can be successfully decoded from the agent's cell state by Ixl and 3x3 probes. Specifically, we investigate binary simplifications of the 'main' concepts, and versions of the 'main' concepts in which the on-off asymmetry reversed.
D.4.1 PROBING FOR SIMPLIFIED BINARY CONCEPTS
The first set additional concepts we probe for are binary simplifications of the concepts studied in the main paper. These binary simplifications from CA and CB. and just reflect whether an agent will move onto, or push a box off of, square. We call these concepts 'Agent Approach' and 'Box Push'.
concepts that remove the directional components
These are binary multi-class concepts that map Sokoban squares to the class (NEVER, AGAIN]. For instance, 'Agent Approach' maps square NEVER if the agent will never step that square
58

---

Published as a conference paper at ICLR 2025
0.8
0.6
Maaco
0.4
0.4
0.2
Layer
Layer
Layer
Baseline
Layer
Layer 2
Layer 3
Baseline
Concept Probed For Agen: Approach Direction
Concept Probed For Push Direction
Agen: Approach
(a) 1xl Probes, Agent- Based Concepts
(b) 1xl Probes, Box-Based Concepts
Macro
0.6
0.4
Layer 1
Layer 2
Layer
Baseline
Layer 1
Layer 2
Layer 3
Baseline
Concept Probed For Agent Agproach Direction
Concept Probed For Push Push
Agen: Approach
(c) 3x3 Probes, Agent- Based Concepts
(d) 3x3 Probes, Box-Based Concepts
Figure 41: Comparisons of the macro F1 achieved when probing the cell state each layer of the agent for the 'main" concepts ('Agent Approach Direction' and 'Box Push Direction') and the 'bi- nary simplification' concepts ('Agent Approach' and 'Box Push'). Reported scores are averaged over five independent training runs. Error bars report standard deviations.
again in the remainder of the current episode, and maps that square to AGAIN otherwise. Likewise, 'Box Push' maps square to NEVER if the agent will never push a box off of that square again in the remainder of the current episode, and maps that square to AGAIN otherwise. Probing for these simplified concepts allows us to determine the extent which the agent learns the directions will move and push boxes when it visits future squares as opposed to learning simpler concepts merely reflecting which squares will visit and which squares it will push boxes off
Figures 41a and 41c show the Fl scores achieved when using Ixl probes probe for 'Agent Ap- proach' and 'Box Push' respectively. Figures 416 and [Td]show the analogous results when using 3x3 probes. Importantly, probing performance increases only mildly relative to to'Agent Approach Direction' and 'Box Push Direction', suggesting the agent has learned the more complex directional concepts rather than these simpler alternatives. A potential explanation for the minor performance gain when probing for these simpler concepts is that the agent may sometimes know it will visit certain squares but be uncertain what action will perform regarding them.
D.4.2 PROBING FOR REVERSED ASYMMETRICAL CONCEPTS
The next set of additional concepts probe for versions of the concepts studied in the main paper where the on-off asymmetry (i.e. the asymmetry in which C'A captures the direction an agent moves on to square while CB captures the direction in which box is pushed off of square) is reversed. We call these concepts 'Agent Exit Direction' and 'Box Approach Direction'. Probing for these reversed asymmetrical concepts allows us to determine whether we were correct to probe for the asymmetrical concepts focused on in the main paper.
Both 'Agent Exit Direction' and 'Box Approach Direction' are multi-class concepts that map Sokoban squares to the classes {LEFT, RIGHT, UP, DOWN, NEVER). 'Agent Exit Direction' maps squares to the direction which the agent moves the next time it moves off of them (if it ever does in the remainder of the episode). For instance, if the next time the agent moves off of square the agent moves left, 'Agent Exit Direction' maps that square to the class LEFT. 'Box Approach Direction' maps squares to the direction the next box is pushed onto them (if box is pushed onto them
59

---

Published as a conference paper at ICLR 2025
Matro
Layer
Layer 2
Layer 3
Baseline
Layer 1
Layer 2
Layer 3
Baseline
Concept Probed For
Concept Probed For Direction
Agent Approach Direction Agen: Exit Direction
Box Approach Direction
(a) 1x1 Probes, Agent-Based Based Concepts
(b) 1x] Probes, nox-Based Concepts
Macro
Layer
Layer
Layer
Baseline
Layer 1
Layer
Layer 3
Baseline
Concept Probed
Probed For Box Push Direction
Agen: Agproach Direction
Agen: Exit Direction
Box Approach Direction
(c) 3x3 Probes, Agent- Concepts
(d) 3x3 Probes, Box-Based Concepts
Figure 42: Comparisons of macro F1 achieved when probing the cell state at each layer of the agent for 'main' concepts ('Agent Approach Direction' and 'Box Push Direction') and the 'reversed asymmetrical' concepts ('Agent Exit Direction' and 'Box Approach Direction'). Reported FI scores averaged over five independent training runs. Error bars report standard deviations.
again in the remainder of the episode. For example, if the next box pushed onto square is pushed down onto this square, 'Box Approach Direction' maps this square DOWN.
Figures 42a comppar macro F1 scores when using Ix1 probes to predict, respectively, (a) CA as opposed to 'Agent Exit Direction', and (b) 'Box Approach Direction' as opposed to CR. Figures 42a and 22d show the analogous results when using 3x3 probes. The key takeaway from these figures is that there are moderate gains probing performance when probing for 'Agent Approach Direction' rather than 'Agent Exit Direction' when using Ixl probes, and small gains when probing for CB as opposed to 'Box Approach Direction' when using Ixl probes. That the two concepts with the highest performance are 'Agent Approach Direction' and "Box Push Direction' is expected given that these concepts better reflect the transition dynamics Sokoban in which the agent pushes boxes off of squares by moving on to squares.
D.5 PROBING FOR FUTURE ACTIONS
In the main paper, we apply the methodology introduced in Section 3.1 to show that the DRC agent we study plans in the way that we hypothesized it would in Section 3.2 by planning in terms of the square-level concepts CA and Cs. In this section, we now demonstrate how this methodology can be used to falsify a hypothesis regarding how we might expect an agent to plan.
Specifically, we apply the methodology to falsify the hypothesis that the agent plans by determining which actions it expects to take in specific number of time steps. Under this hypothesis, the agent would internally represent concepts such as 'Action To Take in Time Step', 'Action To Take in 2 Time Steps' and so These concepts would, for example, assign the class LEFT if the agent moved left in the relevant number time steps. An agent that represented these concepts would then be able to formulate plans by forming planned action sequences of the form (LEFT, LEFT, UP, RIGHT) that the agent would be able to sequentially execute.
To apply methodology determine whether the agent plans in this way, must first linear probes to determine whether the agent linearly represents these concepts. Unlike the square-level concepts studied in the main paper, these concepts global in the sense that they depend the
60

---

Published as a conference paper at ICLR 2025
70%
60%-
Accuracy
50%
40%
30%
20%
1
2
6
8
10
Steps Ahead
Probe
Layer
Layer
Layer
Baseline
Figure 43: Accuracy achieved when probing the agent's s cell state for the baseline, observation to predict the agent's action specific number of time steps.
entire observed Sokoban board. As such, we can use standard 'global' linear probes, i.e. linear probes that receive the agent's entire cell state at layer as input.
We therefore train global probes to predict the agent's action 1.2. .... 10 time steps into the fu- ture using the agent's cell state activations at each layer. These global probes have 10,240 param- eters These probes are trained using the same set-up as spatially-local probes as described in Appendix D. We also train baseline global probes that receive the agent's entire observation as input. Note that we use accuracy (not macro FI) as a measure of probe performance here as class imbalance is a lesser issue. However, for these results, accuracy is usually higher than macro FI.
Figure shows the accuracies achieved by global probes trained to predict the 'Action Take in n Time Steps' for {1...... 10) into the future. While probes trained the agent's 's cell state activations do outperform the baseline, they do not do so by large margin. The performance of these probes seems especially poor relative to the performance of 1x1 probes (which have 64x fewer parameters) trained to predict CA and CB- The poor performance of these probes implies that these concepts are not linearly represented. We can explain the ability of the global probes to slightly outperform the baseline by noting that global probes can infer some information regarding the agent's future actions based on the agent's internal representations of C'A and Cg. Note, however, that global probes cannot simply read future actions off of the internal plan as it will usually be unclear which planned path the agent will follow.
Since our probes imply that the agent does not linearly represent the concepts 'Action To Take in Time Steps', we have, according to our methodology, falsified the hypothesis that the agent plans in the way suggested at the start of this section. That is, we have falsified the hypothesis that the agent plans by directly forming planned sequences of actions it expects to sequentially execute.
E ADDITIONAL BACKGROUND MATERIAL
In this section, we now provide additional background to the paper that may be of interest to the reader. Specifically, this section is organized as follows:
Appendix E. compares pragmatic characterization of decision-ting time planning to past
definitions of planning.
Appendix
provides further details on the Sokoban environment.
Appendix
outlines the DRC agent architecture in greater detail.
Appendix
details the manner in which the agent we investigate was trained.
Appendix we study.
provides preliminary behavioral evidence of planning exhibited by the agent
61

---

Published as a conference paper at ICLR 2025
Appendix E.6 outlines how we operationalize the kinds of concepts study in this paper.
Appendix E.7 briefly details how our methodology could be applied to alternate agents in alternate environments.
E DECISION-TIME PLANNING
In Section 2.1 we provided a pragmatic characterization of decision-time planning. In this section, we now briefly overview definitions of oecision-time planning other fields. We do so demon- strate that our characterization is very similar to these past definitions. Specifically, we consider approaches to decision-time planning in: (1) classical AI, (2) neuroscience, and (3) reinforcement learning.
In classic AI, or, symbolic AI, planning is viewed as the process of an agent formulating a sequence of actions to perform in order to achieve its goal (Russell & Norvig 2010: Hendler et al. 1990). From this perspective, planning a search problem: it involves an agent searching for a sequence of actions that can be performed to reach goal state from the current state (Korf 1987). This perspective of planning is very similar to our characterization in that it understands planning as re- quiring formulating sequences of actions and evaluating their consequences (i.e. predicting whether performing the sequence of actions will allow the agent to reach the goal state).
Decision-time planning is also topic of interest in neuroscience (Miller et al., 2017 Jensen al. 2024). Within neuroscience, Mattar & Lengyel (2022) have defined planning as 'the process of selecting an action or sequence of actions in terms of the desirability of their outcomes'. This is again very similar to our characterization in that it defines planning as involving formulating sequences of actions and evaluating their consequences.
Finally, in reinforcement learning, decision-time planning is usually taken to refer to the process of an agent interacting with a world model in order to determine which actions, when performed in the current state, will yield positive consequences (Hamrick et al. 2020). As stated in Section 2. this definition very similar to our characterization of planning. The only difference is that this 'model-based' definition requires agent to interact with an explicit world model to predict and evaluate the consequences of actions. Our definition loosens this requirement, only requiring that an agent somehow predict and evaluate the consequences of its actions. Table 6 shows some common approaches to designing RL agents capable of performing decision-time planning. Note that, other than DRC agents as studied in this paper, these approaches all maintain least some dependence on handcrafted artefects in order to plan. Finally, in RL. decision time planning is often contrasted with 'background planning' which refers the process of an agent interacting with world model during training to learn a better policy and/or value function Sutton & Barto (2018). Importantly, our characterization of planning does not aim to capture background planning.
Explicit World Model (Known)
Explicit World Model (Learned)
No Explicit World Model
Explicit Search
Algorithm
Example: AlphaZero
Example: MuZero
Siiver et 2018
Schrittwieser al. 2020
(Handcrafted)
Explicit Starch
Example: MCTSNet
Example: 12A RRcaaière et al. 2017
Algorithm
(Partially Learned)
Guez et al. 2018h
Explicit Search
Algorithm
Example: Thinker
(Chung et al. 2024a
(Fully Learned)
No Explicit
Potential Example: DRC Guuzz et al. 2019
Search Algorithm
Table Summary of common approaches to creating RL agents capable of decision-time planning. Agents are categorized based on the extent to which they rely on handcrafted, explicit world models and search algorithms. A search algorithm is explicit it relies on handcrafted rather than learned elements. A world model is explicit if it depends more on handcrafted elements and less on learnable components. DRC agents are listed as potential examples as, prior to this work, was unclear whether they performed decision-time planning.
62

---

Published as a conference paper at ICLR 2025
E.2 SOKOBAN
This paper investigates planning in the context of Sokoban. As explained in Section 2.2 Sokoban is deterministic, episodic environment in which an agent operating in 8x8 gridworld seeks navigate around walls to push four boxes onto four targets. In this section, we provide a detailed explanation of the transition and reward dynamics of Sokoban, as well as of the symbolic representations of Sokoban boards our agent observes.
Sokoban's 's transition dynamics are as follows. At each time step, a Sokoban agent must choose to either move up, down, left, right not to move. When an agent moves left, right, up down onto a square currently containing a box, that box is respectively pushed on to the square to the left, the right, below, or above. If the move agent attempts perform would involve pushing a box into a non-empty square that is, a square containing either a wall or another box neither the box nor the agent moves. The agent cannot push adjacent boxes simultaneously. The agent not pull boxes. An episode ends either when (a) the agent successfully pushes all boxes onto targets or (b) when an episode length exceeds random number between 115 and 120.
Sokoban's 's reward structure is as follows.
The agent receives a reward of -0.01 at each environment step.
The agent receives reward of +1 when it pushes a box on to target square.
The agent receives reward of -1 when it pushes box off of square
The agent receives a reward of +10 after pushing box onto all four targets.
In this paper, we study version of Sokoban that uses symbolic environment representations. Each square of a Sokoban board is always in one of seven states: it is either wall, an empty square, a box on an otherwise empty square, the agent on an otherwise empty square, a box on a target, the agent on target, or target with nothing on it. Thus, in our symbolic representation, we represent each square of an observed Sokoban board as a seven-dimensional one-hot vector and then combine these vectors into an array to produce the agent's observation Ii E 8 of the environment state at time f. Importantly, however, in all figures in which we show an example of a Sokoban board, we use an RGB pixel representation of Sokoban. We do this to allow the reader to more easily understand visualized levels. Figure 2 compares the pixel and symbolic representation of a Sokoban board, where each color in the symbolic representation denotes which dimension of the one-hot vector is active for each square. should also be noted that our version of Sokoban forgoes the layer of wall squares that sometimes appended the edge of Sokoban boards in previous work.
E.3 DEEP REPEATED CONVLSTM (DRC) AGENT ARCHITECTURE
The agent studied in this paper is Deep Repeated ConvLSTM agent as introduced by Guez et al. (2019) DRC agents are recurrent actor-critic architecture. At each time step t, convolutional encoder e processes the agent's observation of the current environment state, 2t, into an encod- ing E Ho Wox This is then processed by the recurrent backbone of the DRC architecture, which is a stack of ConvLSTM, (Shi et al.) 2015). A ConvLSTM is a modified LSTM (Hochre- litr & Schmidhuber 1997) that include a 3D hidden state and uses convolutional connections. The DRC architecture utilises a stack of D Conv units with untied parameters 0 = (0).... ,0d). These ConvLSTM units performs recurrent computations and, time 6, have an internal state sf (he- gf.) where hy of E I Ha are, respectively, the output and cell state of the d- th ConvLSTM unit. For the agent we study, the encoder and all ConvLSTM units have a hidden dimensionality of G. 32, 0 < d < D and utilise kernels of size 3 with single layer of zero padding appended to convolution inputs. This means that all ConvI states maintain the spatial dimensions of the environment state, so that H2 Wa 8 for all 0 < d < D.
The DRC ConvLSTM stack includes a number of enhancements relative to standard ConvLSTM architectures. These are generic modifications that aim to improve the broad capacity of the archi- tecture as a function approximator. The most notable of these is that, rather than performing a single step of recurrent computation for each time step, the DRC architecture performs N steps of recurrent computation per time step. If the current state of the stack of ConvLSTM units is $;-1, and we denote the operation of this stack input encoding i, as fo(ie, 8t-1): the computation performed
63

---

Pool-and-Inject: to allow spatial information to spread rapidly, each ConvLSTM cell ad- Published as a conference paper at ICLR 2025
T
1
Illustration of DRC(3,3) architecture. For each time step, the architecture encodes the input He as a convolutional encoding it, passes it to a stack of ConvLSTMs which perform three ticks of recurrent computation and then outputs policy logits n. and a value estimate et.
Figure 44:
by the ConvLSTM stack at each time step t can be described by the equations below.
5t,0 5t-1,
(2)
8t.m So St,a 8t.a 1), Ocnsn S
(3)
St,N-
The stack of D ConvLSTM units hence performs N ticks of internal recurrent computation for each single time step t in the environment. DRC agents are referred to as a DRC(D, .) agents to make the choice of hyperparameters D and N explicit. The DRC architecture also includes the following additional modifications relative to a baseline ConvLSTM architecture:
Bottom-Up Skip Connections: To allow information to flow up the ConvLSTM stack, the input encoding i, is provided as an input to all ConvLSTM units in the stack.
Top-Down Skip Connections: To allow information to additionally flow down the Con- vLSTM stack, the output of the final ConvLSTM unit on the current tick provided as an additional input to the bottom ConvLSTM unit on the next tick.
ditionally receives version of its output hg h: on the prior tick that is spatially pooled. Specifically, this pooled output pf is produced by separately mean- and max-pooling h spatially, passing the concatenated pooled vectors through an affine transformation, and then reshaping the result to match the dimensions the hi This is shown below.
m minn11 [MeanPool W. (1), MaxPooll,, (h (I E 2G,
(5)
od Ptn-1
W pa m
tbp E RR H.W.W.G Wps E ERHaW.Cox2GG, bpa E W.W.H.G. (6)
Ptn-1 = Reshape H, (P) E HaHWaaxGa
Finally, the output h. the final ConvLSTM cell at the final tick N is concatenated with the input encoding i, and undergoes an affine transformation followed by ReLU non-linearin to generate vector activations o, which is then fed to a policy head and a value head. The policy head performs an affine transformation o, generate vector of action logits which parameterises categorical
64

---

Published as a conference paper at ICLR 2025
Levels Solved (%)
Levels Solved (%)
62
26
60
24
0
3
Number of 'Thinking Steps'
Number of 'Thinking Steps'
'Medium' Levels
'Hard' Levels
Figure 45: The percentage of 1000 'Medium" and 'Hard' levels that the agent solves when the agent performs 'thinking steps'. Zero thinking steps corresponds to the agent's standard behavior.
distribution from which the next action can be sampled d, ~ my The value head estimates the state- value of the current policy for the current environment state linear combination of o,, The policy and value heads are used as an actor and critic in order to train the agent in an actor-critic fashion. Figure 44 fillustrates the computation performed DRC(3,3) agent single time step.
E.4 DRC AGENT TRAINING DETAILS
This paper focuses on analyzing a Deep Repeated Convi (DRC) agent trained to play Sokohan. The DRC agent investigate is trained 900k levels from the unfiltered training of the Boxoban dataset (Guez et al. 2018a). The agent is trained in an actor-critic setting using IMPALA (Espeholt et al. 2018) for 250 million transitions.
We train the agent using a discount rate of y = 0.97 and V-trace target of À = 0.97. The agent is trained by additionally imposing a L penalty of size 1e-3 on the action logits, L regularisation of strength le-5 on the policy and value heads, and adding an entropy penalty of strength le-2 on the policy. Optimisation is performed using propagation through time with an unroll length of 20. We use the Adam optimiser (Kingma & Ba 2015) with batch size of 16 and a learning rate that decays linearly from 4c-4 to O.
During training, the agent selects actions by sampling from categorical distribution parameterized by its policy head logits. Once trained, the agent acts greedily by always performing the action with the greatest logit. After training, the agent solves 97.3% 3% of unseen levels from the unfiltered test set of the Boxoban dataset (Guez et al. 2018a).
E.5 BEHAVIORAL EVIDENCE OF PLANNING EXHIBITED BY THE DRC AGENT
As explained in Section 2.3 this paper is motivated by the phenomenon of DRC agents behaving in
a way that suggests that they perform planning. For instance, DRC agents have been found in past work solve additional Sokoban levels when forced perform 'thinking steps'. which are steps at the start of episodes where the agent is forced to remain stationary (Guez et al.) 2019 Taufeeque al. 2024). We now confirm that the agent analyze in this paper exhibits this planning-like behavior.
We do this by investigating amount of levels the fully-trained agent solves when given differing num- bers of thinking steps. We investigate this in two datasets of Sokoban levels taken from the Boxoban dataset (Guez al., 20T8a). These are the 'Medium' and 'Hard' subsets of levels. As suggested by their names, these levels are more difficult than the 'unfiltered' subset of levels the agent was trained on. We use subsets consisting 1000 levels taken from each dataset. Figure 455 shows the percentage of these 1000 Medium and Hard levels the agent solves when performing between zero and five thinking steps. Zero thinking steps corresponds to the agent's standard behavior. Clearly, the agent performs better when forced to perform 'thinking steps'. This represents planning -aineng-like behavior. This is because an agent capable of planning would be able to make use the additional test-time compute afforded by "thinking steps' to refine its plan.
65

---

Published as a conference paper at ICLR 2025
E.6 OPERATIONALIZINN CONCEPTS
The concepts we investigate in this paper differ somewhat from the standard operationalization of concepts implicit the concept-basee based interpretability literature. In this section, we thus explain the ways in which the concepts we study are abnormal, and provide a formal operationalization of this type of concept.
A discrete concept C' that takes one of W values in the set Ac = f.i....., model input x E S is typically operationalized as a mapping C: - Ag that maps every input a to the value taken by the concept on that input, C(x) E A. (McGrath et al. 2022). The concepts we investigate differ in the following ways:
for every possible
Behavior Dependence: We believe that defining concepts to be mappings from environ- ment states concept classes is overly restrictive in the context of reinforcement learning. This is because RL agents are situated whereby current actions influence future environment states. Hence, RL agents could plau- sibly learn concepts depending upon their own behavior. An example of such a concept de- pending both the environment state and agent behavior in Sokoban might be 'a Sokoban board that will be solved within five moves'. Importantly for present purposes, we believe that behavior dependent concepts would be natural concepts for planning-capable agent to learn This is because existing model-based planners rely on explicit world models for predicting future behavior when evaluating immediate actions to perform planning. Such predictions are implicit within behavior-dependen concepts. Thus, the concepts we study depend on the agent's s current parameters (since these directly determine agent behavior) and on the past observations encountered by an agent in an episode (since the DRC agent's recurrent architecture allows these to influence future behavior).
continuing interaction with their environment
Spatial Localization: We also believe that, in environments with spatially dy- namics like Sokoban, agents could learn similarly spatially-localizeed concepts. All con- cepts we investigate are hence features of individual squares in Sokoban boards.
We thus propose the following pragmatic operationalization of a discrete concept C' that takes one of K values in the set Ag (. ,CK) for square in Sokoban board. This definition is proposed primarily to characterise the specific concepts we study though could serve as inspiration for future definitions of concepts in RL. Let S denote the state space of Sokoban boards, and let G = (ii be a set of square indexes describing an 8x8 Sokoban board. The index (i.j) refers to the square in the i-th row of the j-th column. Further, let é be the set of all possible parameters for a given DRC agent and let H be the set of all possible sequences of observed past Sokoban boards. A concept C is then defined as a mapping : S 9 x x + Ac from a particular square (i,j) E G of presently-observen Sokoban board da E S, and from agent's current parameters 0 E θ and past episode observations (Io, I1, ... .I--) E H. to the value (i-j)
Cor that the concept takes that square given agent behavior. More compactly, and suppressing dependence on past observations for notational simplicity, the concept value taken on square (i.
of Sokoban board 24 when an agent has parameters 0 can be written as (i.j) C(1,, (1.5),
E.7 APPLICATION OF METHODOLOGY TO OTHER MODEL-FREE ARCUITECTURES
general and could be applied to any model-free agent in any environment. Applying the method to general model-free agent in a general environment would involve three steps. We illustrate these steps using the example of model-free agent trained on Breakout:
We believe that, at a high level, the methodology introduced in Section 3.1
1. Probe For Concept Representations In the first step, we hypothesize concepts the agent could plan with, and then probe for these concepts. For instance, the Breakout agent might plan using concepts corresponding to which bricks it plans to remove over the next 10 hits of the ball.
2. Investigate Plan Formation In the second step, we would inspect the manner in which the agent's s concept representations develop at test-time. For instance, we might investigate whether the Breakout agent's representations of the above concepts developed in way that
66

---

Published as a conference paper at ICLR 2025
corresponded iteratively constructing planned hole to drill through the wall from the bottom to the top of the wall.
3. Confirm Behavioral Dependence the final step, we would investigate whether we could use the vectors from the linear probes to intervene to steer the agent in the expected way. For instance, we could intervene on the Breakout agent to force it to drill a hole a specific location of the wall.
Note, however, that the practical application of each of these steps will depend assumptions made by the researcher in the specific experimental setting. In this paper, we made the following assumptions that informed the application of the methodology:
Spatial Localization We assumed that the DRC agent we investigated represented con- cepts in opatially-localize manner. This informed our choice spatially-loca probes. However, while the assumption of spatially- concept representations may hold in some cases (e.g. CNN-based Atari agents), unlikely to hold for all agents (e.g. MLP- based Mujoco agents). In cases where it doesn't hold, we would have to probe all of the agent's 's activations at a specific layer rather than using spatially-localize probes.
Linear Concept Representations We likewise assumed that any concepts the DRC agent represented, it represented linearly (Mikolov et al. (013). This informed our decision to use linear probes. However, this assumption may be argued to be overly-restrictive. this were the case, we would instead have to use non-linear probes (i.e. probes containing non-linearities).
F INVESTIGATING PLANNING IN DRC AGENTS OF DIFFERENT SIZES
In the main paper, we investigate emergent planning a DRCCagen with D layers that performs N = 3 internal ticks of computation per time step. In this section, now perform preliminary investigation of DRC agents different sizes and provide evidence that they too engage in planning. Specifically, we investigate whether two DRC agents of different sizes internally represent C'A and Cg. Using the terminology from Appendix E.3 in which we referred to DRC agent with D layers that performed N ticks of computation per step as DRC(D,N) agent the agents investigate are:
A DRC(1,9) agent (Appendix F.1
A DRC(9,1) agent (Appendix F22)
Both agents investigated in this section are trained for 100 million transitions using the training scheme described in Appendix [E.4 Likewise, both agents exhibit behavioral evidence of planning. For instance, the DRC(1.9) and DRC(9,1) agents respectively solve additional medium difficulty levels when given five "thinking steps' (i.e. forced stationary steps) prior to acting at the start of episodes.
We use 1xl and 3x3 probes to investigate whether these agents internally represent CA and Cg- As in Section 4 we train probes with 5 different initialization seeds. All probes in this section trained using a training scheme identical to that described in Appendix [D.I except for the fact that the training and test datasets consist of all transitions generated when running the corresponding agent for 500 and 250 episodes respectively.
F.1 INVESTIGATING PLANNING IN A DRC(1,9) AGENT
We first investigate whether the DRC(1.9) agent exhibits evidence of internally planning. To re- iterate, this agent only has single ConvLSTM layer, but performs 9 internal ticks of computation for each environment time step. Due to only having a single layer, this agent represents an interesting case-study of a low-capacity model-free agent. After 100m transitions of training, this agent solves 84.9% of unseen levels, and solves an additional 0.8% of medium -iffluuulttf levels when given five 'thinking steps'. While clearly less capable than the DRC(3,3) agent focus on, the ability of the agent to solve unseen Sokoban levels, and to benefit from additional test-time time compute, represents behavioral evidence of planning.
67

---

Published as a conference paper at ICLR 2025
0.8
0.8
0.6
0.6
Mecro
Probe type 1x1
Mecro
Probe type
1x1
3x3
3x3
0.2
0.2
0
layer
Raseline
layer
Röseline
(a) CA
CB
Figure 46: Macro Fls achieved by probes when predicting (a) CA and (b) CB using the DRC(1,9) agent's cell state, or, for the baseline probes, using the observation. Error bars show standard deviation.
Sucees Rate 180
80
Sucees Rate (N)
80
60
Probe
60
Probe
Trained
Tiained
40-
Random
Leyer
Layer
(a) Agent-SSoone Shortcut
(b) Box-Shortcut
Figure 47: Success rates when intervening the cell state of the DRC(1,9) agent in (a) Agent- Shortcut and (b) Box-Shorteut levels using trained and randomly-initialzze probes. Error bars show + standard deviation.
Probing For Glanning-Relevan Concepts Figures 46a and 46b respectively show the macro F1 scores achieved when probing this agent for CA and Cn- As with the DRC(3,3) agent in- vestigated in the paper, the agent appears to represent these planning-relevant concepts. This can be seen in the strong performance of the 1xl and 3x3 probes relative to the respec- tive baseline. Similarly, concepts in spatially-localized manner. This can be seen in the minimal increase in per- formance when moving from the Ixl probes 3x3 probes relative the baseline. Given that these concepts correspond to predictions of future behavior, and of the impacts of fu- ture behavior on the environment, these results suggest that the DRC(1,9) agent planning.
with the DRC(3,3) agent, the agent appears represent these
Investigating Plan Formation Further evidence of the DRC(1,9) using these concepts to plan can be seen in Figure(48 in which we force the agent to per- form five "thinking steps' prior to acting and mea- sure the average macro Fl when predicting CA and On using the agent's cell state each internal tick the agent performs during these thinking steps. As with the DRC(3.3) agent, the DRC(1,9) agent seems to iteratively refine its internal plan as formulated in terms of CA and C3. Qualitative evidence of the DRC(1.9) agent internally planning can be seen in Figures 52a 53a and [54a in which visualize the agent's s internal representations of Cp over the initial 12 steps of episodes.
0.8
Macro F1
0.6
0.4
27 Internal Tick
18
36
45
Ca
Ca
Figure 48: Macro FI when using Ixl probes to decode CA and CB from the DRC (1,9) eighth-layer agent's cell state at each of the additional 45 internal ticks performed by the DRC (1,9) agent when the agent is given "thinking steps', averaged over 1000 episodes.
Confirming Behavioral Dependence Finally, Fig- ures 47a and 47bb show the success rates when in- tervening with the vectors learned by Ixl probes to steer the behavior of the DRC(1.9) agent in Agent- and Box-Shorteut levels in the manner described in Section 6.I Agent-Shorteun interventions are very successful. While Box-Shorteut somewhat less
68

---

Published as a conference paper at ICLR 2025
0.8
0.6
Mecro
Frobe Type 1x1
Mecro
Probe type
0.4
1x1
3x3
3x3
0.
02
Layer 6-
syer 7-
aaw 9
Naseling
Lavers
Layer
Layen
Layer 8
Layer
Masseing
(a) CA
(b) CB
Figure 49: Macro Fis achieved by probes when predicting (a) CA and (b) Cg using the DRC(9,1) agent's s cell state at each layer, or, for the baseline probes, using the observation. Error bars show + standard deviation.
80
80
Sucees Rate (83)
Sucess Rate 6%:
Probe
Prote
40
Trained
ned
20-
20-
Layer
Layera
Layer a
Layer 5
Layer
wwwn 7
Loyen
Layer 9
Layers
Layer
Layer 3
Layar
Layer S
Layen
Layer
Lor
(a) Agent- Shent-Shortcut
(b) Box-Shorteut
Figure 50: Success rates when intervening on the cell state of the DRC(9,1) agent at each layer in
(a) Agent-Shorteut and (b) Box-Shorteut levels using trained and randomly-inittalized probes. Error bars show standard deviation.
successful than ventions with random probe vectors.
interventions, they still are significantly more successful than inter-
F.2 INVESTIGATING PLANNING IN DRC(9.1) AGENT
We now turn attention to investigating whether the DRC(9,1) exhibits evidence of internally planning. To re-iterate, this agent has 9 ConvLSTM layers but only performs a single recurrent tick of computation time step in the environment. The lack of addi- tional internal ticks means this agent is an instance of generic recurrent, model-free agent. As such, it an interesting case-study for investigating whether generic recurrent agents learn to internally plan. After 100m transitions of training, this agent exhibits behavioral evidence of planning as it solves 94.2% of unseen levels, and solves an additional 5.3% of medium-difficulty levels when given five 'thinking steps'.
0.8
Macro F1
0.7
0.6
1
3
Internal Tick
C
Figure 51: Macro F1 when using 1xl probes to decode C' and CB from the DRC (9,1) agent's eighth-layer cell state each of the additional 5 internal ticks performed by the DRC (9,1) agent when the agent is given "thinking steps', averaged over 1000 episodes.
Probing For Planning-Relevanl Concepts Figures 49a and 49b respectively show the macro Fl scores achieved when probing this agent for CA and Cs. As with the DRC(3,3) agent investigated in the per, the agent appears to represent these planning- relevant concepts, and appears to do in spatially- localized manner. However, the agent only does so at few layers. Namely, the agent only appears robustly represent CA at layers 8 and 9, and to robustly represent Op at layers 6, and 8. Evidence of this can be seen in the fact that these the layers which 1x1 probes strongly outperform the
69

---

Published as a conference paper at ICLR 2025
baseline, and which there only minimal gains in macro Fl when moving from 1x1 probes to 3x3 probes. Given that these concepts correspond to predictions of future actions and their impact on the environment, the fact that the agent represents these concepts provides evidence that it is planning.
Investigating Plan Formation Further evidence of the DRC(9,1) agent planning can be seen in Figure [5]] In Figure 5I] we force the agent to perform five 'thinking steps' prior to acting and measure the average macro Fl when predicting CA and CB using the eighth-layer agent's cell state after each of these thinking steps. We use the agent's s eighth layer as this is the layer at which probes achieve the highest macro F1. Clearly, the agent's 's internal plan, as formulated in terms of C'A and CB. becomes iteratively more accurate when the agent is provided with additional test-time compute. This would be expected if the agent was indeed engaging in iterative planning. Qualitative evidence of the DRC(9,1) 1) agent internally planning can be seen in Figures 52b 53b and [4b] in which we visualize the agent's 's internal representations of Cg over the initial T2 steps of episodes. Note that, as would be expected, the DRC(9,1) agent takes more environment steps to arrive at plans than the DRC(1,9) and DRC(3,3) agents.
Confirming Behavioral Dependence Finally, Figures 50a and 50b show the success rates when intervening with the vectors learned by Ixl probes to steer the behavior of the DRC(L,9) agent in Agent- and Box-Shorter levels in the manner described in Section 6.1 Note that, in the interven- tions detailed in Figures 10gures550laaa 50b) it was found to be necessary to scale probe vectors by a scaling factor of 4. These results indicate that the DRC(9,1) agent does its representations of CA and Cg for planning. This is because, in general, the layers at which interventions are most successful (relative to the baseline) are the layers at which probes achieve the highest macro Fl scores. Note, however, that the success of interventions cannot be fully explained by the success of the probing the respective layer. A notable example of this the much greater success rate of Box-Shorteut in- terventions when intervening at layer 6 rather than layer 8, even though probes are somewhat more accurate at layer 8. We hypothesize that this is a consequence of the DRC(9,1) agent accurately rep- resenting its plans to push boxes at many layers (layers 6-8), but only causally altering these plans at a single layer (layer 6). This aligns with the observation that Agent-Shortcut interventions become more successful at later layers, as it may be that the role of these later layers is to determine which actions the agent needs to perform to execute its plans to push boxes.
G INVESTIGATING PLANNING IN A DIFFERENT ARCHITECTURE: RESNET
As mentioned previously, two questions left open by the main paper are (1) whether an agent with a more generic architecture like ResNet can learn to internally plan, and (2) whether we can use the methodology introduced in Section 3.1 to determine whether such an agent is planning. In this section, we now apply our methodology to a ResNet agent trained to play Sokoban, and provide preliminary evidence suggesting an affirmative answer to both of the aforementioned questions.
The results here regard very simple ResNet agent. This agent is parameterized by a network consisting of 24 simplified residual blocks. At each residual block, the input is passed through a convolution, followed by layer norm, a ReLUU, another convolution and another layer norm. The original input is then added to the result of these operations, and is passed through a final ReLU. These residual blocks perform no down- or up-sampling, and make use of no pooling operations. For consistency with the agents studied in this paper, all residual blocks have 32 channels. After the final residual block, the activations are flattened, passed through an MLP of dimensionality 256, and then passed to policy and value heads. This agent is trained for 250 million transitions using IMPALA with the same training scheme as described in Appendix E.4
Probing For Planning-Relevant Relevant Concepts Figures 55a and 555b respectively show the macro Fl scores achieved when probing the hidden state of this agent for CA and Cp after the final ReLU at each layer using both 1x1 and 3x3 probes. Note that, as with the DRC agents, the ResNet agent appears to possess spatially-localized concepts. Further, note that the Ixl probes become iteratively more accurate over layers until a point at which the reverse trend begins. Specifically, the Ixl probes for CB improve until about layer 10, whilst the probes for CA improve for longer until about layer 16. We hypothesize that this means that the ResNet agent is internally planning using spatially- localized concepts relating to box and agent movements, and that it is doing so by first determining how to move boxes, and then, afterward, reasoning about what that means for its own movements.
70

---

Published as a conference paper at ICLR 2025
step O, tick
step 1, tick
Step 2, tick
step 3, tick
step 1cck 9
step 5, tickk
step 6, tick
bick
tick 9
step 9, tick 9
10, tick
step 11, tick
(a) DRC(1.9)
step O, tick
step 1, bck
step 2, Bck
step 3, tick
step 4, tikk
step 5, tick
step 6, tick
step ttikk
step 8, tick
step 9, tick 1
step 10, tick
step 12, tick
(b) DRC(9,1)
step 0, lick
step
step
step 3, tick
step 4,
step 5,
step 6. tick 3
step 7, b.ckk3 3
step 8, tick
step 9, tick 3
step 10, tick
step 11, tick
(c) DRC(3.3)
Figure 52: The internal plan of (a) DRC(1.9), (b) DRC(9.1) and (c) DRC(3,3) agent after the final internal tick over 12 steps of the same level. Plans are decoded from the agents' (a) first, (b) eighth and (c) third layer using a Ixl probe.
71

---

Published as a conference paper at ICLR 2025
step O, tick 9
step tick 9
step 2, tick 9
step 3, tick 9
step 4, tick 9
step 5, tickkk 9
step 6, tick
tick
tick 9
step 9, tick 9
10, tick
step 11, tick
(a) DRC(1.9
step O, tick
step 1, bcck
step 2, tckk
step 3, tick
step 4, tick 1
Step 5, tick
step 6, tick
step tick
step 8, tick
step 9, tick 1
step 10, tick
step 12, tick
(b) 1RC(9,1)
step O,
step
step tick
step 3,
step 4,
step 5,
step 6. tick 3
step 7, bickk 3
step 8, tick
step 9, tick 3
step 10, tick
step 11, tick 3
(c) DRC(3.3)
Figure 53: The internal plan of a (a) DRC(1.9), (b) DRC(9.1) and (c) DRC(3,3) agent after the final internal tick over 12 steps of the same level. Plans are decoded from the agents' (a) first, (b) eighth and (c) third layer using a Ixl probe.
72

---

Published as a conference paper at ICLR 2025
step O, tick 9
step 1, tick 9
step 2, tick 9
step 3, tick 9
step 4, tick
step 5, tick 9
step
tick
step tick
step 8, tick 9
tick 9
step tick
step 11, tick 9
(a) DRC(1.9)
step tick
step 1, tick
step 2, tick
step 3, tick 1
Step tick
step 5, tick
step 6, tick 1
step 7, tick
step 8, tick 1
step 9, tick 1
step 10, tick 1
step 11, tick 1
(b) DRC(9.1)
-1-1
step O, tick
step 1, tck
step 2, tick 3
Step 3, tick 3
step 4, tick
step 5, tick 3
F
step tick
step lickk
step 8, tick 3
step tick
step
tick
step 11, tick
(c) DRC(3.3)
Figure 54: The internal plan of a (a) DRC(1,9), (b) DRC(9,1) and (c) DRC(3.3) agent after the final internal tick over 12 steps of the same level. Plans are decoded from the agents' (a) first, (b) eighth and (c) third layer using a Ixl probe.
73

---

Published as a conference paper at ICLR 2025
0.8-
0.8
Mecro F1
0.6
Probe Type
Mecro
0.6
Probe Type
0.4
1x1
0.4
1x1
3x3
333
0,2
0.2
Layer per 18
Layer 19
Layi 70
Layen 23
Layer 24
Buseline
Layer 14
1999 16
Layer 20
Beseline
Layer
Layer
Layer
Later
Layar
Layer
Layer
Layer
Layer
Layer
Layer Swrrl
La
Loyer
Layer
LaI Lat
Laye
Lajer
Laver
Layer
Lajer
Layer
1999
Layer
1ayer
Layer
Layer Syeris
Layer
Layer
(a)
Figure 55: Macro Fls achieved by probes when predicting the concepts (a) CA and (b) CB using the ResNet agent's ss hidden state after the final ReLU of the residual block at each layer, or, for the baseline probes, using the raw observation. For each layer and probe type, train five probes with five unique initialization seeds in the manner described in Appendix D.I The reported error bars show + standard deviation.
80
Sucees Kare (%)
80-
Probe
Probe
60
Trained
Surves
40
Random
Random
20
20-
13/6
Layer
Loger
Layer
Layer
Laper
Lapir
18
19
Layar Layer
Loyer
Loyer
Layer
Layer
Layer
Loyer
Sypt
Layer
20
Loyer 2
Leyer
23
Layer
(a) Agent-Shoricut
(b) Box-Shorieut
Figure 56: Success rates when intervening on the hidden state of the ResNet agent after the residual
block each layer in (a) Agent-Shorteut and (b) Box-Sh nox-Shortcut levels using trained and randomly- initialized probes. Error bars show + I standard deviation.
Furthermore, the fact that the probes are becoming iteratively more accurate over layers suggests that the agent has learned perform something akin to iterative planning despite its lack of recurrent connections.
Investigating Plan Formation Qualitative evidence of the agent iteratively planning across its layers can be seen in Fi in which we visualize the agent's 's internal plan (in terms of Cg) over the first of its layers the first time step of four episodes. Of these four episodes, the agent constructs a successful plan by its tenth layer in two episodes (Figures 77 and Figures (56). but does not do so in the other two (Figures 57c and Figures 57d). Beyond suggesting that the ResNet agent is iteratively planning across layers, Figure provides preliminary evidence that the ResNet agent iteratively planning in a manner similar to the DRC(3.3) agent focused on in the paper.
Specifically, like the DRC agent, the ResNet agent appears to possess internal planning mecha- nism with similarities to evaluative bi-directional search. For instance, evidence plan evaluation can be seen in Figure 57b in which the agent initially (e.g. at layer 1) plans to push both left-hand boxes to the left-most target, before realizing that the upper-left box must be pushed to this target (e.g. it is the only target that the upper-left box can feasibly be pushed to) and forming plan to instead push the lower-left box to a further target. Further evidence of the agent adapting a plan in response to apparent evaluation can be seen in iigure(57cin in which the agent appears to realize that cannot push the upper-right box up and left to the nearest target. Similarly, evidence of the agent planning forward from boxes and backwards from targets be in Figure 57b) in which the agent seems to plan forward from the lower-left boxes and backwards from the upper-right targets.
Confirming Behavioral Dependence Finally, Figures [6a] and 56a show the success rates when intervening with the vectors learned by Ixl probes to steer the behavior of the ResNet agent in Agent- and Box-Shortcut levels using the procedure described in Section 6.1 Note that, in the interventions detailed in Figures [6aland 56b it was found to be necessary to scale probe vectors by scaling factor of
74

---

Published as a conference paper at ICLR 2025
Layer22
Layer
Layar
Layer
Layer 6
Layer?
Layor
Layer
Layor 10
Layer
Layer
(b)
Layer
Layer
Layer
Layer's
Layer 3
Layer
Laye
Layer
Layar's
Layer
Layer 5
Layer
Layer?
Layer
Layer
Layer
(d)
Figure 57: Examples of the ResNet agent's 's internal plan (in terms of its square- representations of Cp as decoded by a 1xl probe) over its first ten layer in four levels. In two of these instances (57a and 57b), the agent arrives at a successful plan by layer ten. In the other two (57c and (7d) it does not. The visualized plans are taken from the first step of each episode.
75

---

Published as a conference paper at ICLR 2025
A few things can be noted from these figures. First, interventions very successful in both levels at the first layer. This is consistent with the hypothesis that the agent does use these concepts for planning, Interestingly, however, this is despite it being the case that the probes are more accurate when predicting these concepts at later layers. We hypothesize that this is perhaps because the agent forms its initial plans at its lowest layer and then refines these over later layers, such that the agent's plan at the lowest layer is especially amenable to being intervened upon. It is also interesting that Agent-Shorteut interventions are more successful later layers than Box-Shorteut interventions. This is consistent with the notion that the agent primarily focuses on planning box movements at earlier layers, and then subsequently refines plans for its own actions at later layers such that it is more amenable to changing its planned movements at later layers than to changing planned box movements.
H INVESTIGATING PLANNING IN A DIFFERENT ENVIRONMENT: MINI PACMAN
In the main paper, we use the methodology introduced in Section 3.1 to provide evidence indicating that a DRC agent trained to play Sokoban internally performs planning. However, it is natural to ask the extent to which the finding that model-free agents can learn to internally plan generalizes. This is because the 3D structure of the DRC agent's ConvLSTM cell states means the agent particularly well-suited to learning to plan in environment such as with a grid-based structure and localized transition dynamics. In this section, we now provide preliminary results when investigating whether DRC agent can learn to internally plan in different environment: Mini PacMan.
H.1 MINI PACMAN
Mini PacMan is, like Sokoban, a grid-based environment. In Mini PacMan, an agent must navigate around walls in grid-world and cat food. Initially, each non-wall square has food on, and levels end when the agent eats all food. However, the agent must also avoid ghosts which chase the agent. Ghosts chase the agent using A search. In each level, there are also 'power pills'. When the agent steps onto a square with a power pill, ghosts flee, and the agent eats any ghosts it steps onto for the next 20 turns. The agent gets a reward of +1 for eating food, +2 for eating pill, and +5 for eating a ghost. When the agent eats all food, the level is re-populated with food and new ghosts spawn in. An episode of Mini PacMan ends when the agent is either eventually caten by ghost, or when the agent fails to progress to the next level of an episode within 500 time steps of that level starting. Figure shows an example of a Mini PacMan maze near the start of level.
We study a version of Mini PacMan that is simi- lar to the version studied in Hamrick et al. (2020). The version of Mini PacMan we train our agent on consists of mazes that are randomly generated each episode by (1) generating mazes using Primm's 's al- gorithm, and then (2) randomly removing each wall square with two empty adjacent non-adjacent tiles with a probability of 0.3. Each maze contains 4 pills. The number of ghosts in the initial level of each episode is equal to I plus an integer drawn from a Poisson(1) distribution. The number of ghosts each subsequent level then increases by the floor of a 0.25 plus the level number times a number drawn from Unif[o, 2]. Unlike Hamrick et al. (2020) who use mazes of size 15x19, we use smaller square mazes of size 13x13. Across all levels in a single episode, the same maze is used. We use a version of Mini PacMan where the agent observes a symbolic representation 2, E IR 13x 13x 14 of the environment. However, as with Sokoban, present all visualisa- tions using pixel representations of the Mini PacMan board.
Figure 58: An example of Mini PacMan board. The agent (the yellow pacman) must eat the food (the yellow dots) and avoid the ghosts (red) that are chasing it. When the agent eats pill (the pink circles), the agent is able to eat ghosts over the next few steps, and ghosts change colour (blue) and flee the agent. Levels end when all food is eaten.
76

---

Published as a conference paper at ICLR 2025
0.8
0.8
0.6
0.6
Mecro
Probe type 1x1
Mecro
Probe type 1x1
0.4
3x3
3x3
0.2
0.2
Jayer
layer?
Fayer
Raseline
ayer
layer?
layer
Raseline
(a) 'Agent Approach Direction 16'
(b) 'Agent Approach 16'
Figure 59: Macro Fls achieved by probes when predicting (a) 'Agent Approach Direction 16' and (b) 'Agent Approach 16' using the agent's cell state each layer, or, for the baseline probes, using the observation.
The preliminary results provided in this section regard a DRC(3,3) agent trained for 250 million transitions this version of Mini PacMan. This agent is trained using the same training scheme as the Sokoban agents as described in Appendix EE.4, As with all agents studied in this paper, this DRC agent shares the spatial dimensions of the environment is operating in.
H.2 PRELIMINARY PROBING RESULTS
We now present some very preliminary results regarding the aforementioned DRC agent. We ini- tially tried probing for the concept 'Agent Approach Direction' (Cx) as in Sokoban but found little evidence of the agent representing it. After experimentation, however, found probes to be able to decode the following concept from the agent's cell state:
Agent Approach Direction 16:
This concept tracks which squares the agent will step off of, and which direction it will do in, over the next 16 time steps. That is, this is a variant 'Agent Approach Direction' that only accounts for the agent's actions over the next 16 steps.
Agent Approach 16: This concept tracks which squares the agent will step off of over the next 16 time steps. That is, this is a variant of 'Agent Approach Direction' that only accounts for the agent's actions over the next 16 steps, and ignores the directions that the agent enters squares from.
Figures 59a and 59a show the macro Fls achieved by 1x1 and 3x3 probes when predicting 'Agent Approach Direction 16* and 'Agent Approach 16' respectively. These probes are trained and tested on datasets consisting
23k and 6k transitions respectively. We are unsure whether to interpret these results as indicating that either (1) the agent possesses spatially-localized representations of the concept 'Agent Approach 16', or (2) the agent posseses a representation of the concept 'Agent Approach Direction 16' distributed across adjacent positions of its cell state. This is because, for 'Agent Approach 16', 1x1 probes can accurately predict this concept, and we see minimal improve- ment in performance when moving from 1x to 3x3 probe. In contrast, in performance when moving from Ixl to 3x3 probes when predicting 'Agent Approach Direction 16'. This consistent both with the agent representing 'Agent Approach 16' individual posi- tions of its cell state, and with the agent representing 'Agent Approach Direction 16' across adjacent positions of its cell state.
see large improvements
Figures 60 and 61 shows examples of the predictions made by 1x1 probe trained to predict 'Agent Approach Direction 16' when applied to the agent's 's final-layer cell state at its final internal over the first 24 transitions at different points of 2 example episodes. Similarly, Figures 62 and 63 show the predictions made by a 1x1 probe trained to predict 'Agent Approach Direction 16' when applied to the agent's final-layer cell state at its final internal over the first 24 transitions at different points of 2 (different) example episodes. Note that the ghosts turn blue when edible, and purple on their final two turns of being edible.
These examples indicate that, as in Sokoban, the agent uses its concept representations (of whichever concept does represent) form internal plan. Here, the agent's internal plan consists of the squares it plans to visit in the near-future (and. potentially, the directions it will step on to those squares from). A few observations can be made of the agent's internal plan in these examples. First,
77

---

Published as a conference paper at ICLR 2025
step 0
step 1
step 2
step 3
step 4
step
step
step
step 8
step 9
step 10
step 11
step 12
slep 13
step 14
step 15
step 16
step 17
step 18
step 19
step 20
step 21
step 22
step 23
Figure 60: The DRC agent's 's internal plan (in terms of its square-level representations of 'Agent Approach Direction 16 decoded from its final-layer cell state by 3x3 probe) after its final internal tick over the first 24 steps of an episode. teal arrow corresponds to a probe predicting that the agent expects step onto square from the corresponding direction over the next 16 time steps.
78

---

Published as a conference paper at ICLR 2025
step 0
step 1
step 2
step 3
step
step
step
step
step 8
step 9
step 10
step 11
step 12
step 13
step 14
step 15
step 16
step 17
step 18
step 19
step 20
step 21
step 22
step 23
Figure 61: The DRC agent's 's internal plan (in terms of its square-level representations of 'Agent Approach Direction 16 decoded from its final-layer cell state by 3x3 probe) after its final internal tick over the first 24 steps of an episode. A teal arrow corresponds to a probe predicting that the agent expects step onto square from the corresponding direction over the next 16 time steps.
79

---

Published as a conference paper at ICLR 2025
step 0
step 1
step 2
step 3
step 4
step
step
step
step 8
step 9
step 10
step 11
step 12
step 13
step 14
step 15
step 16
step 17
step 18
step 19
step 20
step 21
step 22
step 23
Figure 62: The DRC agent's internal plan (in terms of its square-level representations of 'Agent Approach 16' as decoded from its final-layer cell state by a Ix1 probe) after its final internal tick over the first 24 steps of an episode. teal cross corresponds to probe predicting that the agent expects to step onto square over the next 16 time steps.
80

---

Published as a conference paper at ICLR 2025
step 0
step 1
step 2
step 3
step
step
step
step
step 8
step 9
step 10
step 11
step 12
step 13
step 14
step 15
step 16
step 17
step 18
step 19
step 20
step 21
step 22
step 23
Figure 63: The DRC agent's 's internal plan (in terms of its square-level representations of 'Agent caption 16' decoded from its final-layer cell state by 1x1 probe) after its final internal tick over the first 24 steps an episode. A teal cross corresponds to a probe predicting that the agent expects step onto square the next 16 time steps.
81

---

Published as a conference paper at ICLR 2025
in Sokoban, the DRC agent's internal plans tend to corresponded to connected paths to follow. Second, again as in Sokoban, the agent's 's internal plans seem to iteratively develop.
However, there are also important respects in which the agent's internal planning in Mini PacMan seems different from the planning of the DRC agent in Sokoban. First, unlike in Sokoban where the DRC agent seems to plan fixed horizon the end of the episode -the agent here does does not seem to have a fixed planning horizon. Rather, it seems to often plan paths towards 'target' such as pill or, when ghosts
edible, edible ghost (the blue/purple sprites). We hypothesise that this may explain why the probing macro Fl scores are relatively low despite the qualitative evidence of planning, since it means the probing target only correlate of what the agent is truly planning in terms of. Similarly, unlike in Sokoban, the agent seems to actively alter large parts of its plans, and considers multiple plans in parallel. Note that these two points imply that the concepts we probe for are mere correlates of the 'true' concepts the agent is planning in terms of.
82

---

Published as a conference paper at ICLR 2025
83


Reviews:
[{"review_id": "2vXSjaf9C8", "reviewer": "Reviewer_CqaJ", "summary": "The paper investigates whether planning -- a typically “model based” ability -- can emerge within the internal representation of “model free” agents. Specifically, the authors apply concept-based interpretability methods to identify planning-relevant concepts, whether they emerge, if they support planning, and if they can be intervened on to change the behavior of the agent. This is done for a “deep repeated ConvLSTM” (DRC) agent architecture trained on Sokoban tasks.", "questions": "1. Figure 4 demonstrates the quality of the probes. However, for completeness, would it be possible to provide additional metrics such as precision, recall, class confusion matrices, etc.? The reason is that the results in this work depend heavily on the probes, and therefore it would be good to be fully transparent about the behavior of the probes and its implication on the results.\n\n2. L447: “we [intervene] by adding the representation of NEVER to cell state position on the short path”. Could the authors describe in more details how the representation is computed for interventions (ideally as equations / pesudocodes)? \n    - To my understanding, the authors train linear probes $f: \\mathbb{R}^d \\rightarrow \\mathbb{R}^{|C|}$, with $d$ being the internal DRC representation dimension, and outputting a $|C|$-dimensional logit over the number of concepts (left right etc.). The idea here is to add a $d$-dimensional vector back into the internal DRC representation to change the agent’s behaviour, but since we are mapping from $|C|$ dimension to $d$ where $|C| << d$ (many $d$ can map onto the same prediction with high probability), how is this mapping done? Would one have to optimize for a $d$ that maximizes each class probability? \n    - Also, why add instead of replace? What would happen if you replace?\n\n3. Can the authors briefly discuss if / how the method can be applied to other model-free methods  that do not have the same architecture as DRC? Is it applicable to all architectures trained with model-free objectives?\n\n4. Spelling: L202 behavior should be capitalized", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**[Originality]**\nThe investigation of emergent planning using a concept-based interpretability approach is original and interesting.\n\n**[Quality]**\nThe paper is of high scientific quality, the experiments are rigorous, hypothesis driven and the conclusion (in the affirmative) is well backed-up.\n\n**[Clarity]**\nThe paper is clearly written with relevant information adequately provided. \n\n**[Significance]**\nProvides a greater understanding about what is possible with model-free training alone.", "weaknesses": "The setting the authors investigate is ultimately restricted to a single agent architecture (DRC) trained using a single type of model-free RL algorithm (IMPALA) in a single environment type (Sokoban). The author is candid about this limitation. Nevertheless, repeating the analysis over more environments could make the general results more convincing, and investigating other architecture and/or learning rules would make the insights more generally applicable (e.g. could it be that some algorithms give rise to planning while others do not?).\n\nFurther, the paper does not address the *consequence* of planning, i.e. why is planning useful at all? For instance, [Guez 2019] investigates *data efficiency* and *generalization* as signs for planning. [Wan 2022] shows that even model-based approaches do not necessarily exhibit adaptability to local change. Thus, while it is nice to see a concept-based notion of planning, investigating whether this *leads* to things such as data efficiency, generalization and adaptability can shed greater light for the usefulness of having these representations at all. \n\n\n[Guez 2019] Guez, Arthur, et al. \"An investigation of model-free planning.\" International Conference on Machine Learning. PMLR, 2019.\n\n[Wan 2022] Wan, Yi, et al. \"Towards evaluating adaptivity of model-based reinforcement learning methods.\" International Conference on Machine Learning. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D7tVDix0B0", "reviewer": "Reviewer_E9uN", "summary": "This paper investigates the internal behavior of a Deep RL algorithm, namely, DRC by Guez et al. (2019). The purpose of this work is to verify whether this model-free RL agent is capable of planning, even if it does not rely on an explicit model of the environment. This hypothesis is tested in a discrete environment called Sokoban. The paper contains three analyses: testing for important concepts in the neural network activations with linear probes, investigating how these concepts evolve during internal RNN iterations and during episode steps, and observing whether the policy can be influenced by providing a bias to these activations using the concepts above.", "questions": "(1) The policy network contains come convolutional layers. Thus, in my understanding, the fact that the activations in the policy network linearly correlate with the future movements of the agent can also be explained from the fact that the CNN can learn to encode the spatial gradients of the value function with respect to actions and neighboring states. If true, this hypothesis can be an alternative explanation for the presence of the concepts in the policy network.\n\n(2) How an intervention is performed in practice? A quick description is given in line 462, but I believe a more precise description is required.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an interesting topic for the RL and planning communities. Understanding whether some form of planning-like behavior is present in model-free RL would greatly help in merging concepts that are now discussed separately and it can help in creating more capable hybrid agents.\n\nThe paper adopts an interesting technique that I did not previously encountered in the RL literature. They apply linear probes to the activations of the policy network with specific concepts that are symbolic representations of the future behavior of the policy. The definition of behavior-dependent concepts is an interesting concept that provides useful insights about the policy. The authors also confirm that forcing the internal representation related to the desired concepts can influence the policy in the expected way.\n\nThe paper is well written and it is easy to follow.", "weaknesses": "1. This paper studies one algorithm, DRC, in only one class of environments, Sokoban. This strongly limits the applicability of some of the insights, which cannot be immediately applied to different algorithms and environments. For example, although the idea of behavior-dependent concepts is interesting, the proposed concepts are strongly dependent on Sokoban (there is one feature for each cardinal direction and for each direction in which the box can be pushed by the agent). Nevertheless, considering that the intended purpose of the paper is to understand whether any model-free RL agent is capable of a planning-like behavior, a positive answer can be answered even by looking at a single agent and environment class. \n\n2. The paper often tends to be inclined towards a positive answer that DRC agents can plan, and omits to discuss alternative explanations for the observed behaviors. For example, an LSTM that is trained for N iterative steps is likely to provide more accurate answers after N interative steps at test time. Moreover, advancing throughout the episodes makes the concepts gradually easier to predict. Therefore, Figure 6 apparently represents the expected behavior of the network and it is not necessarily an instance of \"plan formation\", as suggested. Similarly, the fact that the arrows evolve as in Figure 1 does not necessarily imply that the agent is evaluating then refining an hypothetical policy. In some parts, the paper seems to over-interpret the meaning of a small number of empirical observations. For example, the statements in lines 373-376 or 425-427 are not sharp implications as stated in the text. Question (1) is also related to this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mq6UN8oomL", "reviewer": "Reviewer_LpHt", "summary": "This paper provides the first mechanistic evidence (non-behavioural) that model-free reinforcement learning agents can learn to plan.  The authors do this by studying a Deep Repeated ConvLSTM (DRC) agent playing Sokoban. While previous work showed that DRC agents exhibit planning-like behaviors, this paper demonstrates that they may actually perform internal planning.\n\nThere are three main steps in the methodology: Firstly they use linear probes to probe for planning-relevant concepts in the agent's representations. They then look at how plans form within these representations and finally look at the causal relationship of this planning by intervening on the agent behaviour.\n\nUsing this methodology, they claim that the DRC agent have an internal representation of planning concepts and can form plans through an algorithm that is like a parallel bidirectional search, planning forward and backwards. It then evaluates and adapts its plans. There is further evidence from the fact that the agent develops planning capabilities that correlate with improved performance when given extra \"thinking time\".\n\nAll of this is done within the Sokoban environment.", "questions": "Have you tested whether more conventional architectures (without multiple computational ticks) can learn similar planning capabilities?\nCan you provide statistical significance tests for your key comparisons, particularly where differences appear small relative to standard deviations?\nIn cases where the agent forms \"almost correct\" plans, what prevents it from finding the optimal solution? Is this a systematic failure mode?\nCan you provide more analysis of the correlation between concept representation and additional compute benefit?\nWhat led you to choose the specific concepts (Agent Approach Direction and Box Push Direction) to probe for? Did you investigate other potential planning-relevant concepts?\nWhat happens with stronger or weaker interventions in the causal aspects of the experiments?\nHow computationally expensive is your probing methodology? Would it be feasible to apply this analysis in real-time during training?\nCan you look at how the planning concepts emerge through the training process?\nWill code be made available to reproduce the results shown here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper's main strength is in its rigorous approach to demonstrating mechanistic evidence of planning in model-free agents. The probing experiments are well-motivated and well-designed. The authors then build on this foundation through interventional experiments that demonstrate these representations causally influence the agent's behavior. \n\nThe authors also show that the emergence of these planning capabilities correlates with improved performance when given extra computation time, connecting their mechanistic findings to previously observed behavioral results. The ablation studies also validate their methodological choices and demonstrate the robustness of the findings.", "weaknesses": "The lack of detail in the main part of the paper on what a concept means (which is left to the appendix) means that this important point is hard to follow. If reading the appendix is necessary to understand the paper, then the particular detail should not be in the appendix. The same goes for most sections of the appendix which should not be seen as appendices, but actually necessary parts of the paper to understand it as a whole.\n\nOne significant methodological weakness is the lack of statistical rigor in the empirical evaluation. The authors run only 5 random seeds for their experiments and perform no statistical significance testing, making it difficult to assess the reliability of their results. For example, when comparing performance between different probe types or intervention strategies, it's unclear whether the observed differences are statistically meaningful. The paper would be significantly strengthened by proper statistical analysis, including hypothesis tests, confidence intervals, and effect size calculations. The error bars in figure 4 are also surprisingly small and regular which is itself surprising. However, given that there is no code provided, it is impossible to know how reliable these results are..\n\nAnother major limitation is the narrow scope of the investigation. The paper focuses exclusively on a specific architecture (DRC) in a single environment (Sokoban), which means that it's impossible to know how well these methods or results generalise. The DRC architecture is somewhat atypical, with its multiple computational ticks per timestep, and it's unclear whether more conventional architectures could learn similar planning capabilities. It does not seem all that surprising that an architecture of this type might develop an intrinsic world-model.\n\nAdditionally, while Sokoban is a well-established planning benchmark, it has a very specific structure that may make it particularly amenable to the type of planning discovered. The authors do not discuss how their findings might extend to other environments or architectures. While the causal aspects of the paper strengthen their arguments, it would be particularly interesting to see negative results where planning is not found, using the same techiques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QZskmtht5F", "reviewer": "Reviewer_SDx9", "summary": "The authors conduct a series of experiments that mechanistically interpret learned neural network weights of reinforcement learning (RL) agents using deep repeated convLSTM (DRC) to determine whether they are internally planning on an implicitly learned model. The agent network is probed for specific, predetermined concepts in the Sokoban domain. The results evidence that these agents do learn spatially local concepts and reason about them in a manner resembling parallelized bi-directional search.", "questions": "- Is it possible that algorithms other than DRC that also exhibit emergent planning will be more similar other planning algorithms (as opposed to bi-directional search)?\n- Can you explain or conjecture why parallelized bi-directional search is the algorithm that emerges in this case?\n- What about the size of the model and the agent's capacity to learn the world model? Is there a tradeoff between the model size and the quality of concepts that are learned?\n- It seems like DRC is a perfect fit for problems like Sokoban with spatially local properties.\n- Is it possible to perform many computational ticks to arrive at a final plan and then act blindly according to that plan? would this yield high accuracy? what is the horizon to which such an agent can plan, and does this also have anything to do with the model size?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This is a novel and interesting use of mechanistic interpretability.\n- Provides solid empirical evidence that model-free RL agents can implicitly learn to plan under certain conditions.\n- Performed experiments are highly relevant and results are well analyzed. all points made about the results are clear and it does not seem like any outstanding phenomena were overlooked.\n- The discovery that this is reminiscent of bi-directional search is powerful, and may have implications on the future of model-based RL.\n- visualizations of plans are easily interpretable and highly informative.", "weaknesses": "- Does not dive into the effect of probing different layers, even though such results are displayed.\n- Only tested domain is Sokoban. analyzing second domain that is fundamentally different from grid domains is highly recommended to show that this is not a domain-specific phenomenon.\n- Does not provide any background on probing, even though this is a central part of the experimentation.\n- No theoretical explanation as to why planning behavior emerges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)