Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN
Authors: Mohammad Taufeeque
Aaron David Tucker
Adam Gleave
Adrià Garriga-Alonso
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretability
reinforcement learning
sokoban
Primary area: interpretability and explainable AI
TL;DR: Finding a more precise and causal representation of the plan lets us figure out how it gets constructed in an RNN that plays Sokoban

Abstract: We partially reverse-engineer a convolutional recurrent neural network (RNN) trained with model-free reinforcement learning to play the box-pushing game Sokoban. We find that the RNN stores future moves (plans) as activations in particular channels of the hidden state, which we call *path channels*. A high activation in a particular location means that, when a box is in that location, it will get pushed in the channel's assigned direction.
We examine the convolutional kernels between path channels and find that they encode the change in position resulting from each possible action, thus representing part of a learned *transition model*.
The RNN constructs plans by starting at the boxes and goals.
These kernels, *extend* activations in path channels forwards from boxes and backwards from the goal.
Negative values are placed in channels at obstacles. This causes the extension kernels to propagate the negative value in reverse, thus pruning the last few steps and letting an alternative plan emerge; a form of backtracking. 
Our work shows that, a precise understanding of the plan representation allows us to directly understand the bidirectional planning-like algorithm learned by model-free training in more familiar terms.

Published as a conference paper at ICLR 2026
PATH CHANNELS AND PLAN EXTENSION KERNELS: A MECHANISTIC DESCRIPTION OF PLANNING IN A SOKOBAN RNN
Mohammad Taufeeque, Aaron David Tucker, Adam Gleave & Adrià Garriga-Alonso
FAR.AI
Berkeley, California, United States of America (taufeeque, adria) adrial 0ffar.a
ABSTRACT
We partially reverse-engineer a convolutional recurrent neural network (RNN) trained with model-free reinforcement learning to play the box-pushing game Sokoban. We find that the RNN stores future moves (plans) activations in particular channels of the hidden state, which we call path channels. A high activation in particular location means that, when box is in that location, it will get pushed the channel's assigned direction. We examine the convolutional kernels between path channels and find that they encode the change in position resulting from each possible action, thus representing part of learned transition model. The RNN constructs plans by starting at the boxes and goals. These kernels, extend activations in path channels forwards from boxes and backwards from the goal. Negative values are placed in channels at obstacles. This causes the extension kernels to propagate the negative value in reverse, thus pruning the last few steps and letting an alternative plan emerge; a form of backtracking. Our work shows that, precise understanding of the plan representation allows to directly understand the bidirectional planning- algorithm learned by model-frec training in more familiar terms.
INTRODUCTION
Modern Al systems can accomplish a wide variety of complicated tasks, but neural networks trained using deep learning are often difficult to understand. This may be concerning in the context of agentic behavior, where the AI takes a complicated sequence of actions to accomplish a goal. In such cases, agents may accomplish potentially challenging tasks in ways that are difficult anticipate, making the consequences of their actions hard to foresee. Can the behavior of machine-learne agents be understood? This work answers in the affirmative for the case of an agent trained to play the puzzle game Sokoban using model-free reinforcement learning.
We study Sokoban due to consensus in the literature that particular architecture exhibits planning behavior, forming an internal representation of its anticipated future states and actions which can be extracted using linear probes. Sokoban is grid-based puzzle game with walls floors Li movable boxes and target tiles where the agent's 's goal is to push all boxes onto target tiles. Since boxes can only be pushed (not pulled), wrong moves can make the puzzle unsolvable, making Sokoban a challenging game that is PSPACT-complete (Culberson) 1997). requires long-term planning, and popular planning benchmark (Peters et al.) [2023] Racanière et al. 2017; Hamnicc et al. (021). Guez et al. (2019) introduced the DRC architecture family and showed that DRC(3, 3) achieves sfate-of-the-ae art performance on Sokoban amongst model-free RL approaches and rival model-based agents like MuZero (Schrittwieser et al., 2019 Chung et al.) 2024). They argue that the network exhibits planning behavior since it is data-efficient in training, generalizes to multiple boxes, and benefits from additional compute. Specifically, the solve rate of the DRC improves by 4.7% when the network fed the first observation ten times during inference, giving the network extra thinking logistic regression probes to find causal representation of the plan
time Bush et al. (2025) in the DRC, and qualitatively suggest, based on these representations, that the algorithm performs bidirectional search. Taufeeque et al. (2024) find that the DRC often pauses for few steps before

---

Published as a conference paper at ICLR 2026
Normalized Activation
0.5
Observation
Box right (1,8117)
up (10824)
Agent (1.2H31)
Agent (1.HH18)
(L.H227)
(1.3177)
-0.5
Agent right (1.2H5)
Agent up HL.)29))
PNA (1 2H3))
Target (1 (L0866)
plan (1 (1.2H14)
(1.2H266
(1,033)
Figure I: A level with channel activations for single channel from every group in Table clear activations for the box move channels along the box 's path to the target
|
Note the
acting and that the plan changes more quickly during those steps, indicating that the policy seeks test-time compute when needed.
Our contributions are to substantially reverse-engineet the representation that the DRC agent uses to represent its future states and actions, as well the mechanisms which construct plan. We find that many channels of the DRC(3, 3) network directly represent the propensity move in given direction the short or long term, calling them path channels. This removes the need for linear probes used in prior work (Bush et al. 2025 L'uffequee al. 2024), reducing the analysis simply reading channel representations. We then analyze convolutional kernels, as well as activations' evolution over time, mechanistically understand the agent's planning algorithm. The paper analyses single seed of DRC(3, 3) with Section N showing that the results replicate on four other independently trained networks as well.
THE PLAN REPRESENTATION
In contrast to the linear probes discovered by earlier works, we radically simplify the representation. Hidden state channel activations in the DRC(3, 3) architecture from Guez et al. (2019) directly represent the propensity the agent or box to move in particular direction.
2.1 NETWORK ARCHITECTURE.
We analyze the open-source DRC(3, 3) network trained by Taufeeque al. (2024) solve Sokoban, who closely followed the training setup of Guez et al. (2019). The network consists of convolutional encoder, stack of ConvLSTM layers, and an MLP head for the policy and RL value function prediction. Each ConvLSTM block perform 3 ticks of recurrent computation per in-game timestep. The encoder block consists of two 4 x convolutional layers without nonlinearity, which process the H x W 3 RGB observation v, into H x W x C output with height H, width W. and C channels, at environment step t.
ConvLSTM Layers. Figure visualizes the computation of the ConvLSTM layer. Each of the ConvLSTM layers at depth d and tick in the DRC maintains hidden states h2 < with dimensions H HWW xC and takes as input the encoder output the previous layer's 's hidden state 12 n-1 and its own hidden state 1 from the previous step. The ConvLSTM layer computes four parallel gates i, f, o using convolutional operations with 3 x 3 kernels that are combined to update the hidden state. For the first ConvLSTM layer (d 0), the architecture uses the top-down skip connection from the last ConvLSTM layer (d = 2). This gives the network 3 3 = 9 layers of sequential computation to determine the next action at each step. The final layer's hidden state the last tick processed through an MLP head to predict the next action and value function.
DRC (3,3) architecture. The DRC(3, 3) architecture applies the ConvLSTM modules as shown in Figure encoding the observations ii then applying the STM layers parameterized by 0,, Oz.
2

---

Published as a conference paper at ICLR 2026
MLP
03
03
03
-.. - (A (
Londi
22
82
02
hp
Lanf
fanh
01
Pool
Conv
Conv
Conv
Conv
hg
d/c
(a) DRC(333) architecture
(b) The ConvLSTM block
Figure 2: Left: The DRC(3,3) architecture. There are three layers ConvLSTM modules with all the layers repeatedly applied three times before predicting the next action. Right: The ConvLSTM block in the DRC. Note the use of convolutions instead linear layers, and the last layer of the previous tick (hD-1) input to the first layer. "Pool" refers to weighted combination of mean- and max-pooling.
and 03 in sequence for three layers, then applying the same modules to the previous hidden states (i.e. the hidden states from the previous layer and the previous ticks Figure 2bb three times for three "ticks". The entire network is run three times to choose the action in response to each observation.
Notation. Each DRC tick (n 0, involves three ConvLSTM layers (d 0, 1, 2), each providing six 32-channel tensors (ha, Cd2 dd Jas -. fa, od). Channel of tensor va is denoted LdVe.
Additional details. Additional architecture, training, and dataset details provided in Sections B
and C Unless mentioned otherwise, the dataset levels for all results the medium-difficulty validation set from Boxoban (Guez et al. 2018).
2.2 PATH CHANNELS
With the architecture and notation defined, we now describe how the DRC(3, 3) agent plan is representing in path channels the hidden states ho, hi, and h3.
At each layer, the DRC has C channels, each of which a H x W array. The DRC repeats the same compu- tations convolutionally over each square. This results in subset hidden state channels ha representing the plan where each channel corresponds ment direction. the agent or a box is at position where the channel is activated, this causes the DRC to choose that channel's direction as the action. Sec- tion 2.2 illustrates box path channels, where the box moves down twice then right twice to reach the target Since the path channels are hidden convo- lutional states, each hidden state is repeated for the H W game grid, with the squares extending be- hind the game representing channels C. In the figure, the blue down channels are activated at the two loca- tions where the box moves down, and the purple Figure 3: An idealized path channels diagram. right channels are activated at the two locationns where the box moves right. This figure has only one down channel and one right channel, while the real DRC(3, 3) network has several.
move-
Manual inspection of every channel across all layers revealed that most channels are interpretable (Table T). Detailed labels are in Tables 8 and g of the Appendix. We group the channels:

---

Published as a conference paper at ICLR 2026
Table 1: Channel groups, their definitions and counts for each direction (up, down, left, right).
Group
Definitions
Channels
Box-movement
Path of box (short- and long-term)
20 (3, 5, 6) 10, 2, 1, 4 41, 1,
Agent-movement
Path of agent (short- and long-term)
Grid Next Action (GNA)
Immediate next action, represented at agent square
Pooled Next Action (PNA)
Pools GNA to represent next action in all squares Target, agent, or box locations
4 (1,
Entity
8
Combined path
Aggregate 2+ directions from movement channels
29
No label
Difficult to interpret channels
21
1) Box-movement and 2) Agent-movement channels that, for each cardinal direction, activate highly on a square in the grid if the box or agent moves in that direction from that square at future timestep. The probes from Taufeeque et al. (2024); Bush et al. (2025) aggregated information from these channels.
3) Combined path channels that aggregate directions from the box- and agent- govemet channels.
4) GNA channels that extract the next action from the previous groups of channels.
5) PNA channels that pool the GNA channels to be picked up by the MLP to predict the next action.
6) The entity channels that predominantly represent target locations, with some also representing box and agent locations.
7) 'no label' channels for which we could not discern a pattern.
We define the path channels the set comprising the vox-movement, agent- movem,,, and combined path categories, as they collectively maintain the complete plan of action for the agent. The remaining groups (GNA, PNA, entity, and no-label) are termed non-path channels, storing primarily short-term information, with some state for move selection heuristics.
2.3 QUANTITATIVE EVIDENCE
Ablating the state. First, we tested our classification of channels as being path channels or non- non-path channels by performing a single-step cache ablation. Specifically, we set the relevant hidden state channel's previous value to zero, and recompute its activation based on the previous observation only. This removes long-term information from the ablated channel by zeroing it out, while still preserving information from short-term computations based on the encoder and non-ablated channels. This keeps the ablated channel's 's activations more in line with what the network encounters during training, while removing the ablated channel's ability to store plan information across multiple timesteps. Intervening the 59 path channels caused a substantial 57.6% + 2.8% drop in the solve rate. By contrast, intervening on the 37 non-path channels resulted in a 10.5% + 1.9% performance decrease ((a significantly smaller, yet non-negligible, decrease). Controlling for channel count, intervening on a random subset of 37 path channels still led to 41.3% + 2.4% drop in solve rate. This evidence strongly suggests that the computations essential for long-term planning on difficult levels are primarily carried out within the identified path channels.
Area under the curve analysis. Second, we find that each path channel is predictive of the box or agent's 's future movements. T shows the AUC for using the box or agent path channels for predicting the box or agent movement for different numbers of timesteps in the future. Note that the path channels further decompose into short- and long-term channels (Table g Section K))
In Section N we use the AUC values as the basis for simple automatic labeling method which corroborates our findings on additional random training seeds.
Long-term path channels. The network utilizes the long-term channels to manage spatially overlapping plans for different boxes intended for different times. Figure/Jjillusrraee this: in cases where two boxes pass through the same square sequentially in different directions with the first box moving at 0, the long-term channel for the second move activates well in advance (t. < 0), while
4

---

Published as a conference paper at ICLR 2026
Box up
Box down.
Box left
Box right
LOH2
LAHER
LOIE14
LOJIZ3
LAW117
AUC (9)
LOUBI
60
LOH15
L1H14
LIHL1
L1H15
LI0H24
LIHL7
L1H27
1.299
1,2H56
L1H19
1.222
1,21155
0
20
Steps
Agast up
Agast down
Agent left
Agent right
100
LIH2I
1.1HZZ
L2H3
AUC (%)
60
L1H5
LOHLO
L2HS
L1822
LIHLS
L2H23
L2H25
122928
L2144
L3H27
L2H21
121129
L218
L21131
Steps
Figure 4: AUC scores for predicting box and agent movements from the path channels at different #s of timesteps out. Short-term channels have high AUC for up to 10 steps, while long-term channels show high AUC for predicting actions beyond steps until the end of the episode. The GNA/PNA path channels only exist for the agent, and have high AUC (~100%) for only the next action.
begttees Deven dortore
begtore Miee
Righ law Righ dovr
Up shaticre
Figure 5: Activations of the long- and short-term channels for all directions when different direction action takes place = All directions except the up direction show the long-term channel activations decreasing after the other action takes place at t 0. The mechanism of this transfer of activation from long to short-term is shown in Figure 15
its corresponding short-term channel only becomes active after the first move is completed (t = 0). Figure 15 shows the mechanism of this transfer is primarily mediated through the j-gate.
Causal intervention Finally, tested the causal effect of modifying path channel activations.
Table 2: Causal intervention scores for differ- ent channel groups.
We verify the channel labels by performing causal interventions on the channels. We modify the channel activations based their labels make the agent take a different action than the one originally pre- dicted by the network. We collect dataset of 10,000 transitions by running the network on the Boxoban levels (Guez al., 2018), measuring the fraction of transitions where the intervention succeeds at causing the agent take any alternate target action, follow- ing the approach of Taufeeque et al. (2024). Table shows high intervention scores for every group ex- cept the agent-movement channels, with the PNA
Group
Score ( (%
Pooled Next Action (PNA)
99.7 = 0.2
Grid Next Action (GNA)
98.9 = 0.4
Box- and agent-movement
88.1 1.9
Box-movement
86.3 + 2.1
Agent-movement
53.2 2.1
Probe: box movement
82.5 = 2.5
Probe: agent movement
20.7 = 0.7
interventions achieving state of the art causal intervention scores as compared to the probe-base based approaches in Taufeeque al. (2024) and Bush et al. (2025).
The lower score for agent-movement channels is because they are causally relevant only when the agent is not pushing box, which we did not filter for. Section further validates our channel labels.

---

Published as a conference paper at ICLR 2026
Conclusion. We thus conclude that the network primarily represents its plan in the activations of the identified box- moxemeneentt and agent- agent-movement channels. These plans are then mapped to the next action through the group next action and pooled next action channels, explained in Section
THE PLANNING ALGORITHM
How does the plan get constructed? The plan is made by extending path segments forward from the boxes and agent, and backward from the targets, implementing bidirectional search qualitatively observed by Bush et al. (025). The fact that path channels directly represent paths in individual activations allows to directly examine weight matrices in order to understand key components of the algorithm.
First, the network kernels which respond visual features of the game initialize path segments by activating path channels adjacent to key objects such as the agent E. box M. and target Second, plan extension kernels extend these path segments until encountering obstacles which are represented as negative path channel activations. The plan extension kemnels can propagate these negative activations backward along path segments to prune paths which encounter problems. Finally, winner-takes- mechanism inhibits the activations of conflicting path segments in favor of path segments with stronger activations.
1L0024
-L0024 (up)
LL0000 (down)
-L0020 (down)
+L0023 -L0023 (left)
(L0017 (right)
-L0017 (right)
Figure 6: Visualizations of combined kernels that map from the RGB input to the o-gate of the up, down, left, and right box-movement channels of layer The negative and positive RGB components visualized separately. The kernels activate squares along (for agent and box and against (for target the channel's 's direction. These kernels when applied on the RGB input activate (initialize) few squares around the agent box for forward plan chains and around the target for backward plan chains. The kernel for L0017 (right) initializes plan chains only on the agent and box square.
3.1 INITIALIZING PATH CHANNEL ACTIVATIONS
Analysis of simplified encoder kernels mapping from the game observations to box movement path channels show structures that initialize path channel activations. These kernels detect relevant features (such as targets, boxes, or the agent's 's position) to add activations to initialize path segments, such as moving towards targets away from boxes.
Encoder Simplification. We simplify the encoders for visualization purposes. Individually, the encoder weights have no privileged basis (Elhage et al. 2023). To interpret the weights the encoder, we use the associativity of linear operations to combine the convolution operations of the encoder output e₁ at time to the output gate of for layer d tick n into single convolutional layer. For layers d E {1,2,3) 3} and tick n E {1, 3}
we define the combined kernel Ad. and bias by as:
o = tanh (Wa * et other terms)
LSTM Equation
E, WE, (We. It bg. bg,
LSTM Equation
Wd e, Wd (We, (WE, Iz bz.) be.)
Ad x, + pd
up to edge effects (2)
for Ad = Wd Wr,WW.WW and id = Wd Wb, + bzz).
This results in 9 x convolution kernels mapping observations each gate (Figures and 24

---

Published as a conference paper at ICLR 2026
Foraad
*
62
waru Narnul
Dormoo Left
Lo
Right Right
Hight move
*
Tummrigh
Citact
Up
Dowa
Led
Risht
Lad Down
Rigers by
Dovere Righ
(a) Idealized path extension ker- nels. Convolutions extend the relevant direction.
(b) Empirical linear plan extension kernels. Note ward and backward kernels.
(c) Empirical turn plan extension ker-
between for-
nels. Compare "turn right forward kernel" of subfigure (a).
Figure 7: Empirical plan extension kernels are formed by averaging over all kernels Wa, mapping from the previous hidden state h to each hidden state E (i, for each hidden state
3.2 EXTENDING PATH SEGMENTS
Plan Extension Kernels. While the encoder kernels initialize path channel activations, turning these initial moves into path segments requires an extension mechanism operating on the recurrent hidden states. This is accomplished by specialized "plan-extension kernels" within the recurrent weight matrices (W -1 and W in Section B Linear plan extension kernels (Figure 7b) propagate the plan linearly, extending it one square at Time along the channel direction label. Separate kernels exist to facilitate both forward chaining from boxes and backward chaining from targets. Turn Plan Extension kernels (Figure 7c propagate activations from one channel to another channel representing a different direction. The linear kernels have larger weight magnitudes compared to the turn kernels, only when unable continue in the previous direction.
thus encoding agent's preference
Figures and definitely recurs many kernels, and most have pattern which is not the same, but is only translated or adds square or two. In Section N we find plan extension kernels on four other training seeds.
and 26] show some disaggregated linear and turn extension kernels. The idealized pattern
Weight steering. While the agent is only trained on 10 x 10 boards, in Section we use our knowledge of the path extension kernels to get the agent to solve 40 x 40 level using weight steering. Since plans are represented in path channels, and constructed by the path extension kernels, scaling up the path extension kernels by factor 1.4 allows the agent to stabilize longer paths.
Stopping Plan Extension. Plan extension does not continue indefinitely. It must stop at
appropriate boundaries like targets, squares ad- jacent to boxes, or walls. We observe (Figure R) that this stopping mechanism is implemented via negative contributions to path channels at relevant locations. These stopping signals originate from either the encoder or hidden state channels that represent static environmental fea- tures (such as those in the 'entity' channel group, Table 8), effectively preventing the plan from extending beyond targets or into obstacles. This aspect the transition model prevents the DRC from adding impossible transitions to its path.
Observation LIO13
LOH17
Encoder Target from
Figure 8: Plan stopping mechanism demonstration shown through o-gate contributions of the box- right channel (LIH13). The direct effect shows that convolving the forward and backward right- plan-extension kernel on the converged box-right channel (L0H17) spills into the squares of the box and the target. The encoder and the target channels from layer 0 add a negative contribution to coun- teract the spillover and stop the plan extension.
Conclusion Using the path channel representation allows to directly examine model weights in order to understand model behavior. We find that the encoder initializes path channel activations near the target and boxes, and extends them through forward and backward path extension kernels. Nega- tive path channel activations prevent the path extension kernels from making impossible movements, serving some of the same features of transition model.

---

Published as a conference paper at ICLR 2026
3.3 PRUNING PATH SEGMENTS
Backtracking mechanism. The plan extension kernels serve dual purpose and also allow the algorithm to backtrack from bad paths. As part of its bidirectional planning, the DRC has forward and backward plan extension kernels, so negative activations the end of path are propagated the beginning by the backward kernel, and negative activations at the beginning of a path are propagated the end by the forward kernel. This allows the DRC propagate negative activations along a path, thus pruning unpromising path fragments. See Section [ for an example.
Winner-takes-all mechanism. To select sin- gle path for box when multiple options ex- ist, the network employs Winner-Takes-e (WTA) mechanism among short-term path chan- nels. Excluding the long-term path channels allows the DRC to maintain plans for later execu- tion without inhibiting them. Figure T0 (bottom- left) shows that weights connecting path chan- nels for various directions cause the path chan- nel activations to inhibit each other at the same square. The direction with the strongest activa- tion suppresses activations in alternative direc- tions which, combined with the sigmoid activation, ensures that only one direction's 's path channels remain active for imminent execution. We construct level with equally viable paths causally demonstrate (Figure 9 initially, both paths have similar activations, but the slightly stronger one quickly dominates in steps and 2 and deactivates the other via this inhibitory interaction. Zero- ablating the kernels between the channels of the two directions eliminates the WTA effect, leaving both potential paths simultaneously active. Thus, conclude that kernels connecting various short-term box-movement path channels implement this crucial selection mechanism.
Origino
0.5
LIHI3
ANaated
0.5
Step:
Stepe
Step.
Figure 9: After zero-ablating the kernels connect- ing the box-down and box-right channels, the WTA mechanism cannot suppress the right-down plan.
In Section N find the WTA mechanism four other random training seeds.
3.4 PUTTING IT ALL TOGETHER
We provide an example where the winner takes all mechanism chooses between two paths, with the plan extension kernels propagating the path selection back the box and stabilizing the plan in Figure 10
RELATED WORK
Mechanistic explanations. To the best of our knowledge, our work advances the Pareto frontier between complexity of a network and the detail of its characterization, providing the most detailed description of a neural network of this complexity. Much work focuses the mechanisms of large language models. LLMs are more complex than the DRC, but the algorithms these papers explain simpler measured by the size of the abstract causal graph (Geiger et al., 2021 Chan al.)
2022). Examples include work on GPT-2 small (Wang et al.
2023; Hanna et al. [023] Dunefsky
al.
2024), Gemma 2-2B Markks al., 2024 Nanda al. 2023c). Claude 3.5 Haiku (Lindsey
et al. (2025), which contains many simple explanations whose graphs together would add up graph larger than that of the present work. However, their explanations rely only on empirical causal effects and are local (only valid in their prompt), contrasting with weight-level analysis that applies to all inputs. Pioneering work in understanding vision models (Olah et al., 2020: Schubert et al.) 2021 Voss et al. 2021 is very thorough in labeling features but provides a weight-level explanation for only small part of Inception V1 (Cammarata et al. 2021). Other work focuses on tiny toy models and explains their mechanisms very thoroughly, such as in modular addition (Nanda et al. 2023a
2025 Marks et al. 2025). and others (Zhou et al. 2024). A possible exception is Lindsey et al.
Chughtai et al., 2023] Zhong et al. (2023) Quirke & Barez 2023 Gross et al. 2024 Yip et al. 2025 binary addition (Primozic (003), small language transformers (Olsson et al.
2022 Heimersheim &
Janiak 2023), transformer that finds paths in small binary trees (Brinkmann al. 2024).
8

---

Published as a conference paper at ICLR 2026
WTA at box, target
L1H17
Box right L1H17
& Plan Extension
Plan Extension
Observation
Step tick
Step1 tick 2
Step 2 tick
Winner Takes All (WTA)
Linear Plan Extension (LPE) kernels
Turn Plan Extension (TPE) kernels
Output Channel
de
0.5
Backward Forward
0.4
light LAs Boo
0.2
Rights Down
Rigec
0
V
002
-0.5
-04
Down
UP
Down
Law
Ritet
Down
Lon
toup
Right
Imput Channel
Figure 10: A situation with two equally good paths from the box to the target The sum of box-down (L1H17) and box-right (L1H13) channels shows that the network searches forward from the box and backward from the target. Both paths (down-then-right and right-then-down) are visible step tick (left) due the encoder; and the down and right channels have similar activations on the box square (gray). From step O tick 1 until step 2 tick (Section 2.1 defines 'tick") ' the plans extended the same direction by Linear Plan Extension (LPE) kernels (bottom-middle and extended into switching directions by Turn Plan Extension kernels (bottom-right), stopping (Figure on signals corresponding reaching the target or hitting obstacles. The plan at the box square is resolved at step 1 tick 2 using Winner- Takes-All (WTA) mechanism. The average WTA kernel weights (bottom-left figure, averaging over W.,, and W.,, for E ji o}) subtract each channel from all the others, which through sigmoid approximates argmax. The magnitude of the diagonal entries (stronger for down than right) break ties.
DRC in Sokoban. Taufeeque et al. (024); Bush al. (2025) find internal plan representations the DRC by predicting future box and agent moves from its activations using logistic regression probes. Some of their probes are causal, others be used to generalize the DRC to larger levels; however, further analysis is primarily based on qualitative probe and model behavior rather than mechanisms. Our analysis of bidirectional planning is much more mechanistic, and the representation we uncover is much simpler instead of a probe, it is simply reading off of channel.
5 DISCUSSION AND CONCLUSION
Now we discuss broader takeaways from our analysis the path representation and construction.
Probes find predictive, not causal representation. Prior probes the Sokoban network assign weight to both path channels along with channels that are spuriously correlated. The probes thus had lower causal intervention scores compared the path channels directly identified.
Transition model. Despite training with model- reinforcement learning, we find some compo- nents of transition model. The plan extension kernels place activation on the tiles that would result from moving in particular direction. The model also generates negative activation for invalid moves, such as moving into wall.
Path channel activations value function. So far, we have discussed the path channel activa- tions as representation of the path. We believe that the path channel activations also bear similarities to internal value function, more precisely Q function, evaluating whether to take action at a particular state. Firstly, the plan extension kernels propagate positive and negative activations along path segments propagate reward information forward and backward. Targets anchor positive

---

Published as a conference paper at ICLR 2026
activation for moving in that direction, while obstacles generate negative activation. Negative activa- tion at the end of a path can propagate backwards until it prunes the path segment entirely. Secondly, the winner takes all mechanism uses the path channel activations to choose between conflicting path segments, stabilizing the path to choose (generally) higher activation rewards.
It differs from typical Q function informal reward function in several important ways. Firstly, the activations do not appear to correspond to discounted or penalized reward. In the Sokoban environment, each step costs 0.1 reward, so the reward should decrease over time, with longer paths having a weaker activation. Instead, the activation functions represent the preference for shorter paths implicitly in the dynamics of the planning mechanism. The path extension kernels only extend the path few moves forward per tick, and so shorter paths to the target generally have their path segments reach boxes before longer path segments. In Section [ we exploit this insight to make the agent take the longer of two paths. Secondly, there are various biases in the winner takes all mechanism. An astute reader might look Figure T0 and note that the kernel is biased rather than treating all values symmetrically.
Mesa-optimizers. Hubinger et al. (0199) introduced the concept of a mesa-optimizer, an Al that learns to pursue goals via internal reasoning. Examples of mesa-optimizers did not exist at the time, so subsequent work studied the problem of whether the learned goal could differ from the training signal, reward misgeneralization (Di Langosco et al.) 2022 (hah et al. 2022 Oswald et al. (2023) argued that transformers do in-context linear regression and are thus mesa-optimizing the linear regression loss, but this doesn't 't constitute agentic behavior. Modern Al agents appear reason, but whether they internally optimize goal is unresolved.
This work answers, in the affirmative, the question of whether or not agentic mesa-optimizers exist. We present a model organism of nesa-optimization, then point to its internal planning process and to its learned value function. The value function differs from what should be from the training reward, albeit in benign ways: the training reward has -0.1 per-step term, but the value encoded in the path channels do not capture plan length all. In fact, which path the DRC picks is a function of which one connects to the target first, encoding the preference for shorter paths purely in the LPE and TPE kernels (Section D). To compute the value head (critic), the DRC likely counts how many squares are active in the path channels.
Conelusion In conclusion, we discover a simple representation of the DRC 3) agent's intended path as activations in its path channels. These path channels are initialized by encoder kernels, then extended bidirectionally by forward and backward plan extension kernels, and stabilized by a winner takes all kernel. The agent is able to plan forward and backtrack by increasing decreasing path channel activations via the same plan extension kernels.
LLM USAGE STATEMENT
LLMs were used for basic writing feedback and title brainstorming, but not direct contribution.
REFERENCES
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of transformer trained on a symbolic multi-step reasoning task. arXiv, 2024.
URL
httpg//bs/24222. 91 7v2
Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga-Alonso, and David Krueger. Interpreting emergent planning in model-free reinforcement learning. International Conference on Learning Representations, 2025. URL
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah, Curve Circuits. Distill, 6(1):e00024-066 2021.
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 2019. doi: (0.23915/distill 00015. https://distill http://distillppu///9///aa pptpp2001/acciiiaiioaaallll
10

---

Published as a conference paper at ICLR 2026
Lawrence Chan, Adrià GarrigaAAlnsoo Nicholas Goldowsky-DDe Ryan Greenblatt, Jenny Nitishin- skaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal Scrubbing: A Method for Rigorously Testing Interpretability Hypotheses. In Alignment Forum, 2022.
Chess Programming Wiki. Stockfish NNUE, 2024. URL /wwpp:://www... http:://wwwccessppr.......ammaa.
org/Stockfish NNUE Accessed: 2025-05-16.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engincering how networks learn group operations. arXiv, 2023. URL /tt//ttt///xxxi.or/////bs/
03025v
Stephen Chung, Scott Niekum, and David Krueger. Predicting future actions of reinforcement
learning agents, First Reinforcement Learning Safety Workshop, 2024. URL https://
openreview. (ooorummd--Sohhhhhahh
Joseph C. Culberson. Sokoban is pspace-complete.
1997.
URL https://api
semant irmaanicscholarr. httpaaaroorrorooorrra::11111111.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization deep reinforcement learning. In International Conference on Machine Learning, pp. 12009 PMLR, 2022. URL https: bttps::/prrceeeddnggg mlr www.::iiinnmm................. v162/Langosco22a.bn
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Thirty-igght Annual Conference on Neural Information Processing Systems, 2024.
URL https:///ppneeeveee wttp:/eeeeeooommmmmdddeeeeee
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, pp. 24, 2023.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed Reep-RI. with importance weighted actor-learner architectures. arXiv, 2018.
[tR.httt:://ard vv. org/ abs 802 01561v3
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal Abstractions of Neural Networks. Advances Neural Information Processing Systems, 34:9574-9586, 2021.
Jason Gross, Rajashree Agrawal, Thomas Kwa, Fuan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. CoRR, 2024.
abs 2 06 11779v14
Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver. Timothy Lillicrap, and Victor Valdes. An investigation of model-free planning: boxoban levels, 2018. URL
https::/github. www.huu..oommecco http:///hh.aaaccooeeeeeeeeaaaaaa
Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning. arXiv,, 2019. hhtt:://aaa
org/abs/190 1001. 03559v2
Jessica Hamrick, Abram I.. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither- spoon, Thomas Anthony, Lars Holger Buesing, Petar Veličković, and Theophane Weber. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations, 2021. URL BRLhttpps: //tp://e/eeeeeeeeeeee net/
Michael Hanna, Ollie Liu, and Alexandre Variengien. How Does GPT-2 Compute Greater- Greater-Than. Interpreting Mathematical Abilities in a Pre-Trained Language Model. 2: 11, 2023.
Stefan Heimersheim and Jett Janiak. A Circuit for Python Docstrings in a 4-layer Attention-
Only Transformer. Alignment Forum, 2023. URL https:: al http://wwwww..lggeeeeffoooa
http:://ww..allonneeettttte
Dxc/Dossss/GKKKWNNDDDDXX..O httpa///..ea.aaaaaaaa........oon 1m-a-4--ayyer-atteetiioo
11

---

Published as a conference paper at ICLR 2026
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv, 2019. URL https?
7/arxiv. wtg. abt////a//11000. 01820
Michael Ivanitskiy, Alexander Spies, Tilman Räuker, Guillaume Corlouer, Christopher Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Linearly structured world representations maze-solving transformers. In UniReps: the First Workshop Unifying Representations in Neural Models, 2023. URL https::/oppnzeve//
net wtte://oooummiddddaarr00
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell.
Evidence of learned ahead in a chess-playing neural network. CoRR, 2024. URL http:
//arxiv. //arxiv.org/abbabb www.orggaaaaooo abs/ 2406. 00877v1
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. CoRR, 2024. URL
15498v2
Brandon Knutson, Amandin Chyba Rabeendran, Michael Ivanitskiy, Jordan Pettyjohn, Cecilia Diniz- Behn, Samy Wu Fung, and Daniel McKenzie. On logical extrapolation mazes with recurrent and implicit networks. CoRR, 2024. URL http: httpt:://axxx..org///aaa
2410. 03020v1
Kenneth Li, Aspen Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on syn- thetic task. In International Conference on Learning Representations, 2023. URL https
//openrevien ew net http:/oorumiidddde __ccdd__ecTT_TTTT TcRvT
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL /tEps://trrasee rorme r- recirriits biolooy html
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, 2024.
/bs///b///// 2403.... 112.4....161774 19647v3
Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura- Gasarrain,,,,,, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, and Evan Hubinger. Auditing language models for hidden objectives. CoRR, 2025. URL http: /ttp://arxiv.. /awxiv.vvvv..///..//aa////aa...a /rg/
abs 3bs/2503.1m 2tt.//5533...115.. 10965v2
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in AlphaZero. Proceedings of the National Academy of Sciences of the United States of America, 119, 2021.
Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. CoRR, 2024. URL httt::///a/xxii/ wwtp:/oogggbbbbbbbbeo
www.rxvvv..rrrraeo aat///b//22/0 16033v1
Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt
Turner. Understanding and controlling maze-solving policy network. arXiv, 2023. URL
http://arxia org/ abt////si 2310 08043v1
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023a. (tLhtss:: :/ //axxvv.. wwg. axq/abs. abs 2301 05217
12

---

Published as a conference paper at ICLR 2026
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of selff-supervised sequence models. CoRR, 2023b. URL http::t/ptax.......c
2309
00941v2
Neel Nanda, Senthooran Rajamanoharan, Janos Kramar, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. Alignment Forum, pp. 2023c.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 2020. doi:
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-Context Learning and Induction Heads, arXiv preprint arXiv:2209.1 1895, 2022.
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklas- son, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers. CoRR, 2023. URL
/209.. 05858v2
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference on Computatiinnal Natural Language Learning (CoNLL), pp. 548-560, Singapore, December 2023, Association for Computational Linguistics. doi: 10.1 10.18653/ v1/2023.conll-1.. 1.37. htthttss: ac antho wwwpthhwlooo...g/22222222 2023. con 11..371
Niklas Sandhu Peters, Marc Alexa, and Special Field Neurotechnology. Solving sokoban efficiently: Search tree pruning techniques and other enhancements, 2023. URL httthhtts://eee neuro tu wttpeeeli..de/aaaeeerr/////////e
- Niklasseterss pdd
Casey Primozic. Reverse engincering a neural network's clever solution to binary addition. https
//eprimozic net
2023. Accessed: 2025-05-15.
Philip Quirke and Fazl Barez. Understanding addition in transformers. arXiv preprint
arXiv:2310.13121, 2023. UtLpptt::/arxx..or/// wttp/////////aoo/////////// 13121v9
Sébastien Racanière, Theophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. imagination-augmenne agents for deep reinforcement learning. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttpss://proceediggg. neur www.:nnuuiiic.omo cc/paper. hepp File/ /te///987555 2ee
S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall Press, Upper Saddle River, NJ, USA, 3rd edition. 2009. ISBN 9780136042594.
Max-Philipp B. Schrader. gym-sokoban, 2018. URL httpss//ggictbb..cmmmm/scaaaaeee gym sokoban
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, go, chess and shogi by planning with a learned model. 2019, URL
hhttp://arxv. org/ abs /1911. 91266.111661
Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency https//tiilll.u//////ddccitffffe
detectors. Distill, 2021. doi: 10.2391.122. 1/311///iiiill0000240.0.. edges.
Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-al ai knowledge gap: Concept discovery and transfer in alphazero. CoRR, 2023. URL http:// rxi www.oorgga.sss.....o
16410vl
13

---

Published as a conference paper at ICLR 2026
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: why correct specifications aren't t enough for correct goals. arXiv preprint arXh::2210001790, 2022. UtL/tttsss//azxvvorrg abs/22lb............ 01790
David Silver, Thomas Hubert, Julian Schrittieeerr, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science, 362(6119; 2018, doi: 10.1 10.1116/sciecce.aau aar66004 URL https:::////.....t.. /www. wcwence.ecccocccoo..o.o..o.ee wwt.///nn..r.//////////o/o////// abs/ wwtp://ccece.........cc.cc wwwaace.arroaa a6464
Mohammad Taufeeque, Philip Quirke, Maximilian Li, Chris Cundy, Aaron David Tucker, Adam Gleave, and Adrià Garriga-Alonso. Planning in recurrent neural network that plays sokoban. arXiv, 2024. URL /ltt:s://arxvvvorg/////
Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing weights. Distill, 2021. doi:
weights..
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability Wild: a Circuit for Indirect Object Identification in GPT-2 Small. International Conference on Learning Representutions, 2023. httlhh:////aat
iescho ar
hoc//oozpusID:226440
Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari Morcos, and Dhruv Batra. Emergence of maps in the memories of blind navigation agents. In International Conference on Learning Representations, 2023. URL http:::://ppenrrrieeeeeeeeeeeeee
Chun Hei Yip. Rajashree Agrawal, Lawrence Chan, and Jason Gross. Modular addition without black- Compressing explanations of MLPs that compute numerical integration, 2025. URL
http:/penwwiieeeeeeeeefeeeee wwt.:/etttoooioiee i hi-rmmmmdddddyyyyBxkoooo
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. htRL/ttttss//peereeeee http:/vwww....eeeooooeeeeeeeee forum ddt.:mddd=dSSddd10 We
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https::/openreww// wwwp:ewwwwwwwwweeeeeeeeeeeeeeeee www.eeee//ooiiiteeeeeeeeeeoe iate:rummiidiiMMMM2222222.
14

---

Published as a conference paper at ICLR 2026
Appendix
A COMMON COMPONENTS OF SEARCH ALGORITHMNS
search algorithm requires four key components:
1. A representation of states.
2. A transition model that defines which nodes (states) are reachable from a currently expanded node when taking certain action.
3. A heuristic function that determines which nodes to expand.
4. A value function that determines which plan choose once the search ends (in online search algorithms, only the first action from the chosen plan is taken).
The heuristic varies by algorithm:
For A*, it distance(n) + heuristic(n). (Russell & Norvig 2009
For iterative-deepening alpha-beta search used Stockfish). the heuristic comprises
move ordering and pruning criteria. (Chess Programming Wiki, (024)
For AlphaZero/MuzZro MCTS, uses the UCT formula pre-rollout, incorporating backed-
up value functions and policy with Dirichlet noise. (Silver et al. 2018 Schrittwieser et al.
2019)
In all cases, the expansion process influences the relative evaluation of actions in the starting state. The final action selection relies on a value function:
* Uses the actual path distance when plans have been fully expanded. (Russell & Norvig
2009)
Alpha/Zero/MuZZeo MCTS: Employs backpropagated estimated values combining rollout and final score. (Silver et al. 2018: Schrittwieser et al. (2199)
Stockfish 16+: Utilizes the machine-leamed evaluation function leaf nodes. (Chess
Programming Wiki 2024)
In the body of the paper, we show how various parts of the trained DRC correspond to these four components.
B NETWORK ARCHITECTURE
The DRC architecture consists of an convolutional encoder E without any non-linearities, followed by D ConvLSTTM layers that repeated N times per environment step, and an MLP block that maps the final layer's hidden state to the value function and action policy.
The encoder maps the observation H, at timestep & to the encoded state 6r- For all 1, the ConvLSTM layer updates the hidden state h22, ca at each tick n using the following equations:
E(x,) WF, (WE, Nt DE,) + be,
c2, h2 ConvLSTMa(es, h2-1, c/ 11-11)
02 lanh(W,d * + Wi. thy 1 Wdd thg * -1 n- + bz)
(5)
J3 a (Wde e + Wi,, =h2 + Wd h- by)
J = a aWdd * e + Wd jhg
* * + Wd
jh2 * h 2-1 + bj)
(7)
C fa C c/
+ 1 ① jd
(8)
O tan e, Wd Voh h22 + Wid Voh h1 h- bo)
h of tanh(c"))
(10)
15

---

Published as a conference paper at ICLR 2026
Action
Non-plan charnes JU.DJ.R)
Intermediate mechaniem
Previous step hidden state
Input Output
SF Pooled next actio
extension
OF Next action 1
12:00 LinCS
$2.3 Winnee-takee-
$2.3 Short term plan
$2.3 Short plan
åF Map Locch carts 2a0 deco
2012 10101 icua
$3.2 Plan eateraion
$2.3 Long term plan om
$32 Plan extension
$2.3 Long tern plan aeri, (inis
Stop plan extension
Step n
$3.3 Backtrack invalid
step n
Encoder
Agent Bowes Targets
$3.1 Intbolize update
Addd So--
| Adjacett squares Lon DESON
Figure 11: The planning algorithm circuit learned by DRC 3). While the plan nodes are present and updated across all the layers, this circuit only shows the short and long-term plan nodes in the first layer's hidden state (LOHX) with channel X for each direction up, down, left, and right. Mechanisms annotated with the sub-section they studied in Section
Here denotes the convolution operator, and @ denotes point-wise multiplication. Note that 0₂ (W,, W,., Wy., W.., b,, bj. by. bold parameterizes the computation of the i. f, gates. For the first ConvLSTM layer, the hidden state of the final ConvLSTM layer is used the previous layer's hidden state.
linear combination of the mean- and max-pooled Convl activations injected into the next step, enabling quick communication across the receptive field, known as pool-and-inject, A boundary feature channel with ones at the boundary of the input and zeros inside is also appended to the input. These are ignored in the above equations for brevity.
Finally, an MLP with 256 hidden units transforms the flattened ConvLSTM outputs h into the policy (actor) and value function (critic) heads. In our setup, D N and C 32 matching Guez et al. (2019)'s 's original hyperparameters. An illustration of the full architecture is shown in Figure
C NETWORK TRAINING DETAILS
The network trained using the IMPALA V-trace actor-critic (Espeholt et al. 2018) reinforcement learning (RL) algorithm for 10 environment steps with Guez et al. Deep Repeating ConvLSTM (DRC) recurrent architecture consisting of three layers repeated three times per environment step, as shown in Figure 2
The observations H x W RGB images with height H and width W. The agent, boxes, and targets are represented by the green E. brown I. and red pixels respectively (Schrader 2018), as illustrated in Figure [2] The environment has 10.1 reward per step, +10 for solving level, +1 for putting a box on a target and -I for removing it.
Dataset
The network was trained on 900k levels from the unfiltered train set of the Boxoban dataset (Guez et al. 2018). Boxoban separates levels into train, validation, and test sets with three difficulty levels: unfiltered, medium, and hard. The hard set is single set with no splitting. Guez et al. (2019) generated these sets by filtering levels unsolvable by progressively retter-trained DRC networks. So easier sets occasionally contain difficult levels. Each level in Boxoban has 4 boxes in grid size
16

---

Published as a conference paper at ICLR 2026
Figure 12: High resolution visualization of Sokoban level along with the corresponding symbolic
representation that the network observes. The agent, boxes, and targets are represented by the green brown and red squares respectively.
Figure 13: 16 x 16 zig-zag level that the original DRC(3, 3) network fails to solve. Steering W and Wi. by factor of 1.2 solves this level and similar zig-zag levels for sizes upto 25 x 25.
ck1
of H 10. The H W observations are normalized by dividing each pixel component by 255. The edge tiles in the levels from the dataset are always walls, so the playable area x 8. The player has four actions available to move in cardinal directions (Up, Down, Left, Right). The reward -0.1 per step, +1 for placing a box on a target, -1 for removing it, and +10 for finishing the level by evaluate the network the validation calidation-medumm medium and hard
placing all of the boxes. In this paper, sets the Boxoban dataset. We also often evaluate the network on custom levels with different grid sizes and number of boxes clearly demonstrate certain mechanisms in isolation.
Action probe for evaluation on larger grid sizes The DRC(3, (3,33 3) network is trained on fixed H W grid size with the hidden state channels flattened to a H W x C tensor before passing it to the MLP layer for predicting action. Due to this limitation, the network cannot be directly evaluated on larger grid sizes. [Tuufequee et al. (024) trained a probe using logitic regression with 135 parameters on the hidden state h of the final ConvI.S LSTM layer to predict the next action. They found that the probe can replace the 0.8M parameter MLP layer to predict the next action with 77.9% accuracy. They used this probe show that the algorithm learned by the DRC backbone generalizes to grid sizes 2-3 times larger in area than the training grid size 10 x 10. We use these action probes run the same network on larger grid sizes in this paper.
D GATE IMPORTANCE
We identify here the components that important and others which be ignored. We noticed that our analysis can simplified by ignoring components like the previous cell-state c and forget gate that don't 't have much effect. On mean-ablating the cell-state at the first tick 0 of every
17

---

Published as a conference paper at ICLR 2026
Table Comparison of network intervened with single-step cache across different channel groups. We report the percentage drop of solve rate compared to the original network (%) medium-difficuly levels.
Group
Channels
Performance Drop
Non-planning
10.5 1.9
Planning
59
57.6 + 2.8 41.3 = 2.4
Random planning subset
37
step for all the layers, we find that the network's performance drops by 21. 21.28% + 0.04%. The same ablation on the forget gate f results in drop of off.66% 2..66%% 0.03% On the other hand, the same ablation procedure on any of the other gates i, j, o, or the hidden state breaks the network and results in drop of 00.000 with no levels solved at all. This shows that the forget gate is not as important as other gates in regulating the information the cell-state, and the information in the cell-state itself is not relevant for solving most levels. The only place we found the forget gates to be important is for accumulating the next-action in the GNA channels (Section F).
The mean-ablation experiment shows that the network computation from previous to the current step can be simplified to the following:
c ~ o h3 = oa ④ tanh(c") of @ tanh( 2 o ja)
1 - =
O Elc
(1I) (12)
We therefore focus more the j. gates and the hidden state h in our analysis in this paper. Qualitatively, also looks like the cell-state e very similar to the hidden state h. Note that the cell state not being much relevant doesn't 't imply that the network is not using information from previous hidden states, since most the information from the previous hidden states ha n-1 flows through the Wd
ch2 kernels.
E LABEL VERIFICATION AND OFFSET COMPUTATION
We see from Table 9 that most channels can be represented with some combination of features that be derived from observation image (base feature) and future box agent movements (future features). We compute the following 5 base features: agent, floor, boxes not on target, boxes on target, and empty targets. For future features, agent-movement. and next-action feature that activates positively on all squares if that action is taken by the network the current step. We perform a linear regression on the 5 base and 12 future features to predict the activations of each channel in the hidden state h.
get features for each direction: box-movement,
Offset computation On visualizing the channels of the (RC 3etttwrrk, we found that the channels are not aligned with the actual layout of the level. The channels spatially-offset by a few squares in the cardinal directions. To automatically compute the offsets, we perform lincar regression the base and future features to predict the channel activation by shifting the features along x, y E {-2, - 1, 0, 1 and selecting the offset regression model with the lowest loss. The channels offsets are available in Table We manually inspected all the channels and the offset and found that this approach accurately produces the correct offset for all the 96 channels in the network. All channel visualization in the paper are shown after correcting the offset.
Correlation The correlation between the predicted and actual activations of the channels is provided in the Tables 6 and We find that boxmmovemnn, agent-movement, combined-pland and target channels have correlation of 66.4 66.4%, 50.8%, 48.0%, and 76.7%. As expected, the unlabeled channels do not align with our feature set and have the lowest correlation of 40.2%. Crucially, a baseline regression using only base features yielded correlations below 20% for all channels, confirming that the channels are indeed computing plans using future movement directions. These correlations should be treated as lower bounds, this simple linear approach the binary features cannot capture
18

---

Published as a conference paper at ICLR 2026
many activation dynamics like continuous development, representation of rejected alternative plans (Section 3.3). or the distinct encoding of short- vs. long-term plans.
F PLAN REPRESENTATION TO ACTION POLICY
The plan formed by the box movement channels are transferred to the agent movement channels. For example, Figure 23b shows that the agent down movement channel LIHI8 copies the box down movement channel L LIH17 7 by shifting it one square up, corresponding to where the agent will push the box. The kernels also help in picking single path if the box can go down through multiple paths.
Once the box-plan transfers to the agent-movement channels, these channels are involved in their agent-path extension mechanism. Figure 23a show that the agent-movement channels have their own plan-extension the box squares and agent square. Thus, whole, the box- box-movement channels find box to target paths and the agent- agent-movement chanmels copy those paths and also find agent to box paths.
kernels. These channels also have stopping conditions that stop the
Finally, the network needs to find the next action to take from the complete agent action plan represented in agent-movement channels. We find that the network dedicates separate channels that extract the next agent action. We term these channels as the grid-next-action (GNA) channels (Table There exists one GNA channel for each of the four action directions. A max-pooling operation on these channels transfers the high activation of an action to the entire grid of the corresponding agent action channel. We term these as the nexte--nettt (PNA) channels (Table 8). Lastly, the MLP layer aggregates the flattened neurons of the PNA channels to predict the next action. We verify that PNA and GNA channels are completely responsible for predicting the next action by performing causal intervention that edits the activation of the channel based on our understanding to cause the agent to take a random action at any step level. Table 2 shows that both the PNA and GNA channels are highly accurate in modifying the next action. We now describe how the network extracts only the next agent move into the GÑA channels.
The individual gates of the GNA channels copy activations the agent-movement channels. Some gates perform subtraction of the agent and box movement channels to get agent-exclusive moves and the next agent box push. Figure 14 (top-right) shows one such example where the agent and box movement channels from layer are subtracted resulting in an activation exclu- sively at the agent square. The GNA gates also receive positive activation on the agent square through L2H27 which detects agent at the first tick n 0 of step. Figure 14 shows that the f-gate of all GNA channels receives positive contribution from the agent square. To counter- act this, the agent-movement channels of one direction contribute negatively to the GNA chan- nels of all other directions. All this results in the agent square of the GNA channel of the next move activating strongly at the second tick 1.
0.5
-03
Observation
S-gate
Agemt
Bon-down
Agent-down (L11118)
(L21127)
(LI1117)
Figure 14: Left: Observation at step 3 where the agent moves down. Nown.Right: The GNA channels, which represent the direction that the agent will move in the next step, predict the agent ing down primarily through f-gate. The box- and agent-down channels are off'set and subtracted to get the action at the agent square. The checkered agent location pattern from L2H27 also helps in isolating the action the agent square. The active f-gate square accumulates activation in the cell- state which after max-pooling and MLP layer decodes to the down action being performed.
Thus we have shown that the complete plan is filtered through the GNA channels to extract the next action which activates the PNA channel for the next action to be taken.
G STATE TRANSITION UPDATE
We have understood how the plan representation is formed and mapped the next action to be taken. However, once an action is taken, the network needs to update the plan representation to reflect the new state of the world. We saw in Figure that the plan representation is updated by
19

---

Published as a conference paper at ICLR 2026
H2: short-term
H9: short-term
H14: long-term.
Sum
activation (L (LIHI7)
50
50
50
50
50
50
Centered steps (t)
Centered steps (f)
Centered steps (t) 1 ()) tanh(o)
Figure 15: Transfer mechanism from long short-term channel shown through contributions into the gates of the short-term-dovw (L1H17) channel averaged across squares where a right box-push happens at 0 and down box-push later The long-term-dee channel LOH14 contributes the o-gate at all steps t. However, LOH9 teror--tem--right) activates negatively in the sigmoid j-gate, thus deactivating LIH17. As the right move gets played at O, LOH9's negative contribution vanishes, enabling the transfer of L0H14 and LOH2 into LIH17.
deactivating the square that represented the last action in the plan. This allowed a different future action to be represented the same square the short-term channel which was earlier stored only in the long-term channel. We now show how square is deactivated in the plan representation.
After an action is taken, the network receives the updated observation on the first tick new agent or box positions. The combined ce kernels for each layer that map to the path channels contain filters that detect only the agent, box, or target, often with the opposite sign of activation of the plan in the channel (Figure 24). Hence, when the observation updates with the agent in new position, the agent kernels activates with the opposite sign of the plan activation that deletes the last move from the plan activation in the hidden state. The activation contributions in Figure shows the negative contribution from the encoder kernels on the agent and the square to the left of the box. Therefore, the agent and the boxes moving through the level iteratively remove squares from the plan when they are executed with the plan-stopping mechanism ensuring that the plan doesn't 't over-extend beyond the new positions from the latest observation.
with the
H ACTIVATION TRANSFER MECHANISM BETWEEN LONG AND SHORT TERM CHANNELS
Consider scenario where different actions, A and A2 (A1 * A2), are planned for the same location ("square") at different timesteps, f and 12, with 1 < 12- As illustrated in Section 3.3 and further detailed in Figure 5] the later action (Ag t2) is initially stored in the long-term channel for timesteps é < f1. This information is transferred to the short-term channel only after the earlier action (A₂) is executed activation from the long-term to the short-term channel.
lg. We now describe the specific mechanism responsible for this transfer of
In Figure [5] the activations transfer into LIH17 (short-term-downl from L0H14 tong-term-dowe and LOH2 (short-term) downn channels when a right action is taken at represented in LOH9 (short-term-right).. The short-term-right channel LOH9 imposes a large negative contribution via the j-gate to inhibit LIH17, keeping it inactive even as the long-term-down channel tries to transfer signal through the j and o-gates for f 0. Once the first move completes (t = 0), short-term-right is no longer active and so the inhibition ceases. The removal of the negative input allows the j-gate's 's activation to rise, enabling the long-ter long-term-due tetteedmoown activation transfer through o-gate, making it the new active short-term action at the square. This demonstrates how long- channels hold future plans, insulated from immediate execution conflicts by the winner takes all (WTA) mechanism (Section 3.3 and Figure 10) acting short-term channels.
20

---

Published as a conference paper at ICLR 2026
1
CASE STUDY: BACKTRACKING MECHANISM
SH
All channels
L2P
sapp
<teg
EL
(e) EX0 (nght)
(1) 1:319 (Tgge) stop bick
1gl 1:119 (right) A... stop tick.
S
Figure 16: (a) 20 x 20 level we term as the "backtrack level" with key decision nodes D1-D3 for backward chaining. (b) The of box-movement channels step tick indicates forward (from box) and backward (from target) chaining. (c-e) Activation of the box-right channel L2H9 involved in backward chaining step 5 tick Backward chaining moved up from DI D2 and then hitting wall at D3, which initiates backtracking towards D2 through negative plan extension. The negative wall activation comes from the o-gate of L.H9.. (f) Successful pathfinding at T28 after backtracking
redirected the search. (g) Ablation: Forcing positive activation at D3 (by setting it to its absolute value) prevents backtracking, hindering correct solution finding (L229 Abl., T28).
In particular, the forced positive ablation at D3 results in an incorrect plan (g) which seemingly goes right all the way through the wall, as opposed to the correct plan (f) which goes right on a valid path.
Consider the level depicted in Figure T6 (a). The network begins by chaining forward from the box and backward from the larget(Figure 16 b). Upon reaching the square marked D1, the plan can continue upwards or turn left. Here, the linear and turn plan-extension kernels activate the box-down and the box-right channels, respectively. However, the box-down activation is much higher because the weights of the linear extension kernels are much larger than the turn kernels (as seen in Figure (O]. Due to this, the winner-takes-all mechanism leads to the search continuing upwards in the box-down channel. Upon hitting wall at D2, the chain turns right along the "box-right channel' (L221) and continues until it collides with another wall at D3. (Figure T6 c).
This triggers backtracking. While both i-gate and o-gate activations contribute to plan extension, the o-gate also activates strongly negatively on wall squares like D3 (Figure e). This leads to a dominant negative activation in the 'box-right' channel, which then propagates backward along the explored path (from D3 towards D2) via the forward plan- plan-extensiin kernels of L2H9.
This weakens the dominant "box-down' activation at D1, allowing the alternative 'box-right' path from DI to activate. The search then proceeds along this new route, allowing the backward chain to connect with the forward chain, resulting in the correct solution (Figure [6 f).
To verify this mechanism, we performed an intevention by forcing the activation at the wall squares near D3 to be positive (by taking their absolute values). This blocked backtracking, and the network incorrectly attempted to connect the chains through the wall (Figure [6 g). This confirms that negative activation generated obstacles is the key driver for backtracking, and is what allows the network to discard failed paths and explore alternatives. We quantitatively test this claim further by performing the same intervention on transitions from 512 levels where a plan's activation is reduced by more than half in single step which was preceded by neighboring square having negative activation in the path channel. We define the intervention successful if forcing the negative square to an absolute value doesn't 't reduce the activation of the adjacent plan square. The intervention results in a success rate with 95% confidence intervals 85.1% 5.0% and 48.9% + 3.3% for long- and short-term channels, respectively. This checks out with the fact that long-term channels represent plans not in the immediate future which would get backtracked through negative path activations. On the other hand, negative activations in the short-term channels also useful during the winner-takes-al (WTA) mechanism and deadlock prevention heuristics. Filtering such activations for short-term channels from the intervention dataset would improve the numbers.
21

---

Published as a conference paper at ICLR 2026
Figure 17: A level with two paths, one longer than the other. We initialize the starting hidden state with the two paths shown such that they both have two squares left to reach the target. We find that the expands both paths and picks the left (longer) path through the winner-takes-all mechanism since it reaches there with higher activation through finear-plan-ee plan-extension.
J CASE STUDY: MAKING THE NETWORK TAKE THE LONGER PATH
The network usually computes the shortest paths from box to target by forward (from box) and backward (from target) chaining linear segments until they connect at some square as illustrated in Figure I0 As soon as valid plan found for box along one direction, the winner-takes-all mechanism stabilizes that plan through its stronger activations and deletes any other plans being searched for the box. From this observation,
hypothesize that the network values finding valid plans in least number of steps than picking the shorter one. We verify this value preference of the network by testing the network with on the level shown in Figure [7 with the starting state initialized with the two paths shown. The left path (length=13) is longer than the right path (length=7) for reaching from the box to target. Both paths are initialized the starting hidden state have two arrow left to complete the path. We find that in this case, both the paths reach the target, but the left one is stronger due to linear plan extension kernels reaching with higher activation. This makes the network pick the left path and prune out the shorter right path. If modify the starting state such that left and right paths have 3 and 2 square left to the reach the target, then the right path wins and the left path is pruned out. This confirms that the network's 's true value in this case is to pick valid plan closer to target than to pick shorter plan. However, since convolution moves plan one square per operation, the network usually seems have the value of picking the shorter plan.
K CHANNEL REDUNDANCY
We see from Table 8 that the network represents many channels per box- mox-moeement and agent- movement direction. We find at least two reasons for why this redundancy is useful.
First, it facilitates faster spatial propagation of the plan. Since the network uses 3 x 3 kernels in the STn block, information can only move I square each direction per convolution operation. By using redundant channels across multiple layers, the network can effectively move plan information several squares within a single time step's forward pass (one square per relevant layer). Evidence for this rapid propagation is visible in Figure [6)b), b), where plan activations extend 7-10 squares from from the target and the box within the first four steps on a 20 x 20 level.
Second, the network dedicates separate channels represent the plan different time horizons. We identified distinct short-term (approximately 0- 10 steps ahead) and long-tern (approximately 10-50 steps ahead) channels within the box and agent-movement categories.
22

---

Published as a conference paper at ICLR 2026
Figure 18: Sum of activations of box-movement channels on the 40 x 40 backtrack level with the network weights Wi ck1 and Wi ch, steered by factor of 1.4. The planning representation gets stuck in the loop shown, unable to backtrack and explore other paths. The activations of other squares become chaotic, changing rapidly and randomly on each step.
This allows the network to handle scenarios requiring the same location to be traversed at different future times. For example, if box must pass through the same square at time & and later time 12, the network can use the short-term channel to represent the first push at t and the long-term channel represent the second push lg. Figure (rightt illustrates this concept, showing activation transferring from a long-term to short-term dowe-movement channel once the earlier action at that square taken by the agent.
L WEIGHT STEERING FIXES FAILURE ON LARGER LEVELS
Previous work TTufeeque al. 2024 showed that, although the DRC(3, 3) network can solve much bigger levels than 10 x 10 grid size on which was trained, it is easy to contruct simple and natural adversarial examples which the network fails to solve. For example, the
zig-zag level in Figure 13 that can be scaled arbitrarily by adding more alleys and making them longer, is only solved for n S 15 and fails all n > 15. The big level shown in Figure To (a) solved by the network on the 20 x 20 grid size but fails on 30 x 30 or 40 x 40 grid size.
Figure 22 (a) visualizes the sum of activations of the box- movementtnt channels on 40 40 variant of the backtrack level in which we see the reason why larger levels fail: the channel activations decay as the plan gets extended further and further. This makes sense as the network only saw levels during training and hence the kernel weights were learned to only be strong enough to solve levels where targets and boxes are not too far apart. We find that multiplying the weights of W'd and W',,, the kernels that update and maintain the hidden state, by a factor of 1.2 helps the network extend the plan further. This weight steering procedure is able solve the zig-zag levels for sizes up to n = 25 and the backtrack level for sizes up to x 40. Figure 22 (b, c) show that upon weight steering, the box-movement channels are able to maintain their activations for longer, enabling the network to solve the level. However, for much larger levels, weightsteered networks also fall into the same trap of decaying activations, failing extend the plan. Further weight steering with larger factor can help but we find that can become brittle, as the planning representation gets stuck in wrong paths, unable backtrack, with the activations becoming chaotic (Figure (8). We also tried other weight steering approaches such as multiplying all the weights of the network by a factor or subset such as the kernels of path channels, but find that they do not work as well as the weight steering of Wi
10
chi and Wdd ohu'
M EMERGENCE OF PLANNING STRUCTURE DURING TRAINING
To understand when the planning mechanisms described in this paper emerge during training, we analyzed intermediate checkpoints saved throughout the 2 billion environment steps of training.
23

---

Published as a conference paper at ICLR 2026
Specifically, tracked the evolution of the Maner-Takes-All (WTA) mechanism by measuring kernel connectivity between direction channels.
Methodology. We analyzed the convolutional kernels connecting short-term box-movement path channels across all available checkpoints. For each checkpoint, computed:
1. Self-connection strength: The average weight magnitude kernels connecting each direction channel itself (e.g., box-down -> box-down). Positive values indicate self- reinforcement.
2. Cross-iniibition strength: The average weight magnitude of kernels connecting different direction channels (e.g., box-down - box-right). Negative values indicate mutual inhibition.
3. Normalized Self-Cross Difference: A metric bounded between -1 and +1, computed as:
self cross
WTA
(13)
|ell + crrss
A value of +1 indicates ideal WTA structure (positive self-connection, negative cross- inhibition), while -1 indicates the opposite pattern.
Self Colleetnnnntnnnn vs Cross Inhibition Strength
Normalized Self-Cross Difference
Self-Cross Difference
Self-connection strength
WTA emerges
Connection Strength
0.4
inhibiio strength
19M steps
0.5
0.2
0.0
0.0
0.5
-02
-1.0
10
10
10
10²
10"
10'
10
10"
Training Steps log scale)
Training Steps log scale)
Figure 19: Emergence of the Winner-Takes-All mechanism during training. Left: Self-connection strength (blue) increases while cross-inhibition strength (red) becomes increasingly negative over training. Right: The normalized self-cross difference transitions sharply from -1 to +1 around 19M environment steps, indicating rapid emergence WTA structure.
Results. Figure 19 shows the evolution of these metrics across training and shows that plan extension kernels and WTA kemels emerge in distinct phases during training.
Early training (0-10M steps): Both self-connection and cross-inhibition weights remain near zero. The normalized difference is approximately -1, indicating no WTA structure-if anything, the opposite pattern (cross-eccitation).
2. Transition phase (10-19M steps): A rapid phase transition occurs where self-connection becomes positive while eross cross-inhibition becomes negative.
3. WTA emergence (19M steps): The normalized self-cross difference reaches +1, indicating that the full WTA structure has emerged. This occurs at less than 1% of total training (19M of 2000M steps).
4. Continued refinement (19M-2000M steps): After the WTA structure emerges, the absolute magnitudes continue increase (self-connection reaches ~0.5, cross-inhibition reaches ~.0.),, but the qualitative structure remains stable.
This analysis complements weight-level mechanistie findings by showing that the planning structures we identify in the fully-trained network emerge through distinct developmental trajectory during learning.
24

---

Published as a conference paper at ICLR 2026
N STABLE PLANNING ALGORITHM ACROSS TRAINING SEEDS
We also found long- and short-term path channels, plan extension kernels, and winner takes all kernel in four additional random training seeds for the DRC(3, 3) agent, corroborating our overall findings. To do so, we created simple automated method to label long- and short-term path channels, then looked at their aggregated kernels using same methodologies as in Figure 7b and Figure I0
Automatic discovery method. For each channel, we compute the AUC score for predicting future box or agent movements in each direction, searching over different spatial offsets and activation signs. We assign given channel to label if AUC for some property is over 0.95. For grid-next- action channels we use the channels in the middle tick of the DRC to compute the AUC, and for all others we use the last tick of the DRC.
Long-term path channels are based on the AUC of predicting an action is between 10 and 50 timesteps, and if the AUC for predicting 50 timesteps out is higher than its AUC for predicting the next timestep.
Short-term path channels are based on the AUC for predicting an action in the next 10 steps, and if the AUC for predicting the next timestep higher than its AUC for predicting 50 timesteps out.
Grid-next-action (GNA) channels are based on the AUC of predicting just the next move, and not being a short-term path channel.
Pooled-nexs nexxd-actttnnonn (PNA) channels are based on the AUC of predicting the next move by mean-pooling the activations across spatial dimensions.
We find that the automatic discovery method correctly labels 3/4 of the GNA channels, all the PNA channels, and discovers the box or agent channels with F1-score of 73.7%.
Channel statistics. Table 4 shows the discovered channel counts across all five networks. Despite independent training, all networks develop qualitatively similar structure: approximately 27 box- movement channels, agent-movement channels, and 3-4 GNA/PNA channels each. All networks also develop both short-term and long-term plan representations.
Table 4: Discovered planning channels across five independently trained networks.
Seed
Total
Box
Agent
Long-term
Short-term
GNA
PNA
bkynosqi (Manual)
38
31
5
33
4
gobfm3wm
2
26
j16bq8ih
36
28
5
31
4
4
q4mjldyy
27
5
30
qqp0kn15
33
27
5
28
3
Mean
34.0
26.8
7.2
4.4
29.6
3.4
3.4
Plan extension kernels. 20 the averaged plan extension kernels for box-movement channels across all networks. Each network develops the characteristic forward and backward propagation pattern: high weights in the direction plan extension and near-zero weights elsewhere. This confirms that the linear plan extension mechanism emerges consistently across training runs.
Winner-takes-all structure. Figure 21 shows the WTA connectivity matrices for all networks. Each matrix shows the average kernel weight connecting movementmnnt channels of one direction to another. Most networks exhibit the characteristic WTTA pattern: positive celf-connection along the diagonal (self-r reinforcement) and negative or weaker off-diagonal connections (cross-inhibition between competing directions).
These results support the claim that the path channels and extension kernels discovered in this paper represent stable solution for DRC(3, 3) that emerges consistently across independent training runs.
25

---

Published as a conference paper at ICLR 2026
gubfi3tmmm (Aufo)
JichggHth
(Amto)
U
Foo
Food
bkynosgi (Manual)
2.5
Bwd
Bord
qámjidyy (Autu)
eqalikn15 (Auta)
Fwd
Fwd
0.5
Brd
Dwd
Figure 20: Plan extension kernels across four networks trained independently with different seeds. Top row: forward propagation. Bottom row: backward propagation. Each column group shows the four directions (U, D. I,, R) for one network. All networks develop similar linear extension patterns. The unique IDs correspond to IDs of training run. Manual was computed using our manual channel labeling from the main network in the paper.
bkynosqi (Manual)
gwmfm3wm (Auto)
j16bq8th (Auto)
gámjldyy (Auto)
qqp0km15
(Auto)
Inputdiecciio
0.0
0
D
L
U
D
Output direction
Chiqpur drection
Output
Ourput
Output directions
Figure 21: Winner-takes-all connectivity across four networks trained independently with different seeds. Blue indicates positive (reinforcing) connections; red indicates negative (inhibiting) con- nections. All networks develop self rentooreffrre on the diagonal and cross-direction -nhibittion off off-diagonal. The unique IDs correspond to the IDs of training run. Manual was computed using our manual channel labeling from the main network in the paper.
O RELATED WORK
Mechanistic interpretability. Linear probing and PCA have been widely successful in finding
representations of spatial information (Wijmans et al. (023)) ) or state representations and game-specific
concepts games like Maze (Ivanitskiy al.] 2023] Knutson al. 2024]] Mini al. 2023). Othello
(Li al. 2023 Nanda et al. (023b), and chess (McGrath et al. 202 1 et al. 2023 Karvonen 2024). However, these works are limited input feature attribution and concept representation. and do not analyze the algorithm learned by the network. Recent work has sought to go beyond representations and understand key circuits in agents. It is inspired by earlier work in convolutional image models (Carter et al.) 2019) discovering the circuits responsible for computing key features like edges, curves, and spatial frequency (Cammarata et al. 2021 Olah et al., 2020 [Schubert et al. 2021). In particular, recent work has found mechanistic evidence for few-step lookahead in superhuman chess networks (Jenner et al. 2024 Schut et al. 2023), and future token predictions in LLMs on tasks like poetry and simple block stacking (Lindsey et al.) 2025 Men et al. 2024 Pal et al. 2023). However, these works still focus on particular mechanisms in the network rather than a comprehensive understanding of the learned algorithm.
26

---

Published as a conference paper at ICLR 2026
Original step 10
Rep
(c) Steered nct
Figure 22: The sum of activations of the box-movement channels on 40 40 variant of the backtrack level from Figure T6 for (a) the original network at step 50, and the weight-steered network at (b) step 50 and (e) step 100 when the agent reaches halfway through. The original network fails solve the level as the plan decays and cannot be extended beyond 10 15 squares. Upon weight steering. the plan activations travel farther without decaying thus solving the level.
P LIMITATIONS
Our paper has several limitations.
We only reverse-engineer DRC and other networks. is possible that the inductive biases of other networks such as transformer, Cony-Res or ID-LSTM may end up learning an algorithm that is different from what we found for the DRC. Our results are also only on Sokoban and it is possible that the learned algorithm for other hame-playing network looks very different from the one learned for Sokoban.
We also do not fully reverse-engineer the network. We have observed the following behaviors that cannot be explained yet with our current understanding of the learned algorithm:
Agent sometimes executes steps of the plan for box 1, then box 2, then back box 1, to minimize distance. Our explanation doesn't account for how and when the network switches between boxes.
Sometimes the heuristics inexplicably choose where to go based seemingly irrelevant things. Slightly changing the shape or an obstacle or moving the agent's position by can sometimes change which plan gets chosen, in manner that doesn't correspond optimal plan.
Q SOCIETAL IMPACT
This research into interpretability can make models more transparent, which helps in making models predictable, easier to debug and ensure they conform specifications.
Specifically, we analyze model organism which is planning. We hope that this will catalyze further research identifying, evaluating and understanding what goal model has. We hope that directly identifying a model's goal lets us monitor for and correct goal misgeneralization (Di Langosco et al.
2022
Table 8: Grouped channels and their descriptions. * indicates long-term channels.
Group
Description
Channels
Box up
Activates on squares from where box would be pushed up Activates on squares from where box would be pushed down
L0H13, LOH24*. L2H6
Box down
LOH2, L0H14*, LOH20* LIH14*, LIH17,LIH1
Box left
Activates on squares from where a box would be pushed left
LOH23*, L0H31, LIHII, LIH27, 1.2H20
Continued on next page
27

---

Published as a conference paper at ICLR 2026
Table 8: Grouped channels and their descriptions. indicates long-term channels.
Group
Description
Channels
Box right
Activates on squares from where box would be pushed right
LOH9, LOH17, LIHI3, LIHI5*, L2H9*, L2H15
Agent up
Activates on squares from where an agent would move up Activates on squares from where an agent would move down Activates on squares from where an agent would move left Activates on squares from where an agent would move right Channels that combine plan information from multiple directions
L0H18, LIH5, LIH29, L2H28,
L2H29
Agent down
LOHI0, LIHI8, L2H4, L2H8
Agent left
L2H23, L2H27, L2H31
Agent right
LIH21, LIH28, L2H3, L2H5, I2H21* L,H266
Combined
LOH15, LOH16, LOH28. L0H30,
Plan
LIHO, LIH4, LIH8, LIH9, LIH20,
LIH25, L2H0, L2HI, L2HI3,
L2H14, L2H17, L2H18, LOH7,
LOHI, LOH21, LIH2. L1H23,
L2H11, L2H22, L2H24, L2H25, 12H12, LOH19, L.H30
Entity
Highly activate on target tiles. Some also activate agent box tiles
LOH6, L.H26, L1H6, LIHIO,
LIH22, L.H2,, L2H7
No label
Uninterpreted channels. These channels do have clear meaning but they also not very useful
LOHO, LOH3, LOH4, LOH5, LOH8,
LOH22, LOH25, LOH27, LOH29,
LIHI, L1H3. LIHI2. LIHI6, LIH26, LIH30, L2H10, L2H19,
LOH11, LOH12, L1H7, L1H24
Grid-Next-
Channels that activate on squares that the agent will move in the next few moves. One separate channel for each direction A channel for each action that activates highly across all squares at the last tick (n = 2) to predict the action
1.2H28 (up), L.HH4 (down), 1.2H23
Action
(left), L2H26 (right)
(GNA)
Pooled-
L2H29 (up), L2H8 (down), L2H27 (left), L.HH (right)
Next-
Action
(PNA)
Table 9; Informal description of all channels
Channel
Long-
Description
term
LOHO
No
some box-left-moves?
LOHI
No
box-to-target-lines which light up when agent comes close to the box.
LOH2
No
H/-C/-IJ/-O; +future box down moves [Isq left]
LOH5
No
[1sq left]
L0H6
No
H/-C: +target -box -agent. F: +agent +agent future pos. +agent. O: -agentt future pos. J: +target -agent[same sq]
LOH7
No
(0.37 соrr across i.j.f.o).
LOH9
No
H//C/-O/I///F +agent +future box right moves -box. -H///F: +agent-near- future-down- moves [on sq]
LOH10
H: www.eet-xcclsivee-oo-n-ooooo
[1sq left,down]. Positively activates on
agent-exclusive-up-mmoe
LOH11
No
H: CO. O: box- bix-tight moves C/I: -box future pos [1sq up (left-right noisy)]
LOH12 No
H: very very faint horizontal moves (could be long-term?). I/O: future box horizontal moves (left/rihh), [on sq]
L0H13
No
H/C//J/O: +future box up moves [lsq up]
Continued on next page
28

---

Published as a conference paper at ICLR 2026
Table 9: Informal description of all channels
Channel
Long-
Description
term
LOH14
Yes
H/-I/O/C/H: -ffttrfe-oox-cco sotur---bx--ow--movee Is more future-looking than other channels in this group. Box down moves fade away as other channels also start representing them. Sometimes also activates -agent-right-ne moves [on sq]
LOH15
No
H/////-F/-O: +box-fuure-mooes. More specifically, +box-down-moves +box- left-moves, searchy (positive field around target). (0.42 corr across i..f.o).
LOH16
No
H +box-right-moves (not all). High negative square when agent has to perform DRU actions. [Isq up.left]
LOH17
No
H/I///F/O: +box-future-riigh moves. O: +agent [lsq up]
LOH18
No
H: -agemtexclusivv--ppnnoes
LOH20
Yes
H: box down moves. Upper right corner positively activates (0.47 start -> 0.6 in few steps -> 0.7 very later on). I: -box down moves. O: +box down moves -box horizontal moves. [lsq up]
LOH21
No
-box-left-
+up-box-move
LOH23
Yes
H/C/I///O: box future left moves [Isq up.left]
LOH24
Yes
H/C/////: -future box up moves. long-term because it doesn't fade away after short-term also starts firing [1sq up.left)
LOH26
No
H: -agent. I/C/-O: all agent future positions. J/F: agent target BRwalls, [Isq up]
LOH28 No
H/C/////F/-O: -future box down moves (follower?) [on sq]. Also represents agent up.right.left directions (but not down).
L0H30
No
H/I: future positions (0.47 corr across i.j.f.o).
L1H0
No
H: -agent -agent near-future-(dllr--mmoeee + box-future-pos [on sq] -box-leftmooves
L1H2
LIH4
No
+box-left moves -box-right moves [lsq up].
LIH5
No
H: +agent-xxclusive -etttssiv--vuuuttur--gg- moves [2sq up. lsq left]
L1H6
No
J: player (with fainted target)
L1H7
No
H: some left box moves or right box moves (ones that end at a target)?
Sometimes down moves? (unclear)
LIH8
No
future-up-moves(402225 [on sq]
L1H9
Yes
O/I/H: future pos (mostly down?) (seems to have alternate paths as well. Abla- tion results sligthly longer sols on some levels). Fence walls monotonically increase in activation across steps (tracking time). [on sq]
LIH10
No
J/H/C: -box + target +agent future pos. (neglible H) O,-I: +agent +box -agent future pos [1sq up] (very important feature 18/20 levels changed after ablation)
LIHI1
No
-box-left-mover (-0.6).
LIHI3
No
H: box-right-movest (a..:0)ageetttiiiii nuturr-ppos(+.002 [Isq left]
LIH14
Yes
H: longer-term down moves? [1sq up]
L1H15
Yes
H/-O: tor-a----oooe----------ooooo
(with high activations towards tar-
get). Activates highly when box is the left side of target [on sq].
LIHI7
No
H/C////J/-F/O: -box-future down moves [on sq]
LIHI8
No
H/-O: +agent future down moves (stores alternate down moves well?) [on sq]
L1H19
No
H/-F/-: bow--oww---ooo moves (follower?) [1sq up]
L1H20
No
+near- future- all-box-llllllelell boo--oovss moves [Isq up).
L1H21
H: agent-right-moves(-... (includes box-right-pushes as well)
L1H22
No
-target
LIH23
No
boxxlefi-movess.
LIH24
No
H: bbox -agent- future-ftuue-poo -pre -agent, [lsq left]
Continued on next page
29

---

Published as a conference paper at ICLR 2026
Table 9: Informal description of all channels
Channel
Long-
Description
term
LIH25
No
-ottbb---aaddg-----taaattt------ 0.07).walIss -ff-aantoo--ll------e----------- (0xx.0..)bbxxe(0066.. H: +box -agent -mmpt agent-future- O/-C: -agent +future sqs (probably doing search in init steps) I: box agent + walls F: -agent future pos J: +box +wall -agent near-future pos [lsq up.left]
L1H27
No
H: box future left moves [Isq left]
LIH28 No
wereo--gnneeeesiieeeeeeee miooiiie-igght-oos((tDD) (.t.::.......----ooe---e-e----.e unclear(-0.1)
LIH29
No
--stes, includes box-up box-up-pushes as well). I: future up moves (-almost all moves) agent [1sq upl
L1H31
No
H: squares above and below target (mainly above) [Isq left maybe up] -box-all-move
L2H0
No
L2H1
H/O: future-doww/right-sg [Isq up]
L2H2
No
H: high activation when agent is below box target and similar positions. walls at the bottom also activate negatively in those positions. H: +right action (PNA) future box -down -right moves future box +left moves
L,H3
No
L2H4
No
O: +near-future agent down moves (GNA). I: +agent/box future pos [1sq left]
L2H5
No
H/C/I/I:
O: agent-future-sqs [Isq up, left] H: +box-up-moves (-5-10 steps). -agent-up-moves. next-target (not always) [lq left]
L2H6
No
L2H7
No
+unsolved box/target
L.H88
No
down action (PNA).
1,2H9
Yes
H/C/1/J/O: +future box right moves [1sq up]
L2H11
No
L2HI3
No
H: +box-future-leff ler--oonrrm-ftuuree--ee
5-10moves before tak-
ing right moves) moves. Sometimes blurry future box up/down moves [lsq up]
L2H14 No
H: ail-other-sgs(-O.A) -otn-ffuuuu--p(+O001) O: -agent-future-pos I: +box- future-pos
L2H15
No
-box-right-moves [Isq up.left]
L2H17 No
H/C: target(+0.75) box- box-future-pos(-O.3) O: target. J: +target -agent +agent future pos. I/F: target. [Isq up]
L2H18 No
Very noisy/unelear at the start and converges later than other box-down channels.
box-dowwn/eff--movs(....
L2H19
No
H: future agent down/right/left sqs (unclear) [1sq up]
L2H20
No
H: -box future left moves [Isq left]
L2H21
Yes
H: ffarffutrre-ggen--rigt---mm
Negatively contributes L2H26 to remove
far fn-ffuuure-qss.. Also represents sagent/box-down-moove [1sq up]
L2H22
No
H: eox-righ-mmove(+00) (b. moox-down-moveste
O future sqs
L2H23
No
H: future left moves (does O store alternate left moves?) (GNA). [1sq left] box-right/up-moves (long-term)
1.2H24
No
L2H25
No
unclear but (8, 9) square tracks value or timesteps (it is a constant negative in the 1st half episode and steadily increases in the 2nd half)?
L2H26
No
H/O: near-future right moves (GNA). [on sq] left action (PNA). TO: negative agent sq with positive sqs up/left.
L2H27
No
L2H28
No
futuree up moves (GNA). O: future up moves (not perfectly though) [1sq up]
1.2H29
No
Max-pooled Up action channel (PNA). some +agent-left moves (includes box- box-left- left-pushes).
L2H31
No
30

---

Published as a conference paper at ICLR 2026
Table 5: Activation offset along (row, column) in the grid for each layer and channel
Layer 0
Layer
Layer
Channel 0
(1,0) 0)
(0,0)
(-1, 0)
Channel
(0,
(4,,-1)
(41,-1
Channel 2
(0,
(4,00
(0.0)
Channel
(0,0)
(-1,0)
(0.0)
Channel 4
(-1,-1)
(-1,-1)
(0.-1)
Channel
(0,-1)
(-2,-1)
(41.0)
Channel 6
(0,0)
(-1,-1)
(-1,-1)
Channel
(-1,0)
(-1,0)
(0.0)
Channel 8
(0,-1)
(0.0)
(-1,0)
Chanmel
(0,0)
(0,0)
(-1,0)
Channel 10
(-1,-1)
(-1.0)
(-1,0)
Channel
(-1,0)
(0,-1)
(0,-1)
Channel 12
(0,-1)
(0,-1)
(0,-1)
Channel 13
(-1,0)
(-1.0)
(-1.0)
Channel 14
(0,0)
(0.-1)
(-1,-1
Channel 15
(0,0)
(0,0)
(1,-11
Channel
(-1,-1)
(0,0)
(-1,-1
Channel 17
(-1,0)
(0.0)
(-1.0)
Channel
(-1,0)
(0,0)
(-1,0)
Channel 19
(41,-1
(-1,0)
(-1,-1
Channel 20
(-1,0)
(0,-1)
(0,-1)
Channel 21
(-1,0)
(4,,0)
(0,0)
Channel 22
(0.0)
(0,0)
(-1,0)
Channel 23
(-1,-1)
(1,00)
(0,-1)
Channel 24
(-1,-1)
(0.-1)
(-1,0)
Channel 25
(-1,0)
(-1,-1)
(-1,-1)
Channel 26
(-1,0)
(0,-1)
(0,0)
Channel 27
(-1,-1)
(-1,-)
(0.0)
Channel 28
(0,0)
(0,0)
(-1,0)
Channel
(0,0)
(-1,0)
(0,-1)
Channel 30
(-1,0)
(0.0)
(-1--1
Channel
(-1,-1)
(0,-1)
(0,-1)
Forward
1.5
Beckward
0.5
-0.5
Up
Down
Left
Right
(a) Forward and backward plan extension kernels averaged over agent-movement channels. Agent- movement channels also extend the agent moves for- ward and backward similar to the box-plan extension.
The kernels that map LIH17 (box-down) L1H18 (agent-down) by shifting the activation one square up. L1H11 activates negatively, therefore the j and f ker- nels are negative since they use the sigmoid activation function. The and o kernels positive which results in negatively activating
o-gates, which after mul- tiplication results LIHI8 activating positively, The opposite signed weights on the lower-corner squares of the kernel help in picking single path out of mul- tiple parallel paths.
Figure 23: Plan extension and box path to agent path kernels.
31

---

Published as a conference paper at ICLR 2026
Table Correlation of linear regression model's predictions with the original activations for each channel.
Layer 0
Layer
Layer 2
Channel
33.15
79.48
70.03
Channel
50.76
48.77
38.37
Channel
73.15
28.90
39.17
Channel 3
31.73
68.30
55.72
Channel
45.06
50.10
45.64
Channel 5
63.91
42.95
55.27
Channel 96.57
87.47
53,90
Channel 51.98
36,88
95.63
Channel 46.64
41.58
55.04
Channel 9
70.52
37.44
71.47
Channel
37.68
99.01
53.91
Channel 11
52.09
61.55
42.26
Channel 12
41.54
43.86
27.19
Channel 13
79.54
73.35
54.40
Channel 14
72.17
48.12
56.54
Channel 15
44.09
65.72
36.37
Channel 16
63.49
26.56
38.24
Channel 17
76.70
73.94
94.78
Channel 18
61.51
66.11
34.18
Channel
46.05
44.01
33.48
Channel 20
65,00
58.94
64.92
Channel 21
22.05
57.36
60.21
Channel 22
26.51
63.73
24.32
Channel 23
74.39
31.32
44.64
Channel 24
83.64
58.56
59.94
Channel 25
17.10
82.43
28.29
Channel 26
75.48
44.26
45.17
Channel 27
9.24
85.84
49.92
Channel 28
46.87
42.65
15.38
Channel 29
28.60
64.77
54.68
Channel 47.70
35.00
40.15
Channel 31
53.12
56.81
59.63
Table Correlation of linear regression model's predictions with the original activations averaged over channels for each group. Includes correlation using only base features for comparison. The (all dir) group the average of the four directions. NGA and PNA are included in the Agent groups.
Group
Correlation
Base correlation
Box up
72.36
21,01
Box down
62.73
13.93
Box left
67.96
21.10
Box right
65.69
27.40
Box (all dir)
66.37
20.83
Agent up
47.86
12.69
Agent down
51.12
15.85
Agent left
51.40
7.85
Agent right
52.73
14.92
Agent (all dir)
50.80
13.33
Combined path
48,00
23.35
Entity
76.73
70.66
No label
40.25
15.53
32

---

Published as a conference paper at ICLR 2026
(L.01
L003
+1004
-LOO4
+LO03
-L,03
+1006
-LO06
-LO07
-L007
+10012
-L,011
+1,0013
-1.0033
+10014
-L0014
-L,0155
-L,0015
+L0020
.L,0200
+L0021
-L0021
+1.022
-L0022
-L,023
-L,023
+L0028
-L,0228
+1.0029 -L,0299 +1.0300
-L0030 -L,031 -L0031
Figure 24: x 9 combined convolutional filters WOO that map the RGB observation image to the O gate in layer 0. The positive and negative components of each channel filters separated visualized by computing m0 W and max(0, - Wo respectively. The green, red, and brown colors in the filters detect the agent, target, and box squares respectively. The blue component is high only in empty tiles, so the blue color can detect empty tiles. We find that many filters are responsible for detecting the agent and the target like L005 and L006. A use case of such agent and box detecting filters in the encoder is shown in Figure g Many filters detect whether the agent or the target are some squares away in particular direction like L0020 and L0023. Filters for other layers and gates can be visualized using our codebase.
Table 10: Solve rate (%) of different models without and with 6 thinking steps on held out sets of varying difficulty.
Model
No Thinking Med
Thinking Med
Hard
Unfil
Hard
Unfil
DRC(3,
42.8
76.6
99.3
49.7
81.3
99.7
DRC(I, 1)
7.8
28.1
89.4
9.8
33.9
92.6
ResNet
26.2
59.4
97.9
33

---

Published as a conference paper at ICLR 2026
Sinedo 300 prtte doeeet The actlort
Figure 25: Each of the Box-down box-down plan-extension kernels, centered by their channels'
relative offsets (see 5). The first rows are kernels from IJFO of layer to of layer 1, and the next from IJFO of layer 1 to the H in its next step. In many cases the idealized weight pattern from Figure 7a but in most we do not. The color scale goes from -2.0 to 2.0.
Terrer edtton
Beredk gate dometa the antlewe
Bered domett the antlen
aithew
Figure 26: Each of the Box-down box-down plan-extension kernels, centered by their channels' relative offisets (see Table 5p. The first two are kemelss from IJFO of layer H of layer the middle two from IJFO layer to the its next step, and the last two from IJFO of layer to H of layer 2. In many cases see the idealized weight pattern from Figure but in most do not. The color scale goes from -2.0 to 2.0.
34


Reviews:
[{"review_id": "JqwJm2WJ8H", "reviewer": "Reviewer_DRUr", "summary": "The paper reverse-engineers a Sokoban DRC(3,3) ConvLSTM agent trained with model-free RL and shows that its hidden state contains path channels, which directly encode future moves for boxes/agent by direction. Planning emerges via plan-extension kernels that (i) initialize short path fragments near boxes/targets from encoder features, (ii) extend them forward from boxes and backward from targets, (iii) stop at obstacles via negative activations, and (iv) use a winner-takes-all inhibition to select between competing short-term moves. Quantitatively: ablating 59 labeled path channels drops solve rate by 57.6% vs 10.5% for 37 non-path channels; random 37-path-channel ablation: 41.3%. Path channels are predictive of future moves (AUC curves separate short- vs long-term). Causal interventions on PNA/GNA channels achieve ~99% action flips; movement channels ~88% (agent-movement lower due to condition mismatch). Weight-level analysis also explains backtracking (negative activation propagation) and shows “weight steering” (scaling extension kernels) stabilizes longer paths on larger boards. The authors argue this constitutes a concrete, mechanistic description of bidirectional planning in a model-free agent and discuss links to mesa-optimization.", "questions": "1. How reproducible are channel group assignments? Provide criteria + agreement stats; release per-channel labels.\n2. Does the same path-kernel story hold for different DRC depths/widths, training seeds, and other grid sizes without weight steering? \n3. If you zero WTA connections, what is the task-level effect on solve rate and branching? Similarly, for turning vs linear extension kernels. \n4. Report intervention success vs magnitude; do small signed nudges suffice to steer plans reliably? \n5. Can you predict the critic head from path-channel activations (counts/energies) across states? Provide R^2/ablation on the value head. \n6. How often do negative “stop” signals incorrectly prune valid paths? Any systematic failure modes (e.g., tunnels, multi-box couplings)? \n7. Please release weights, code for kernel visualizations, and the exact Boxoban splits used.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Plans live in identifiable channels; no probes needed to find them.\n2. Concrete kernels for initialize/extend/stop/compete; elegant WTA tying to action selection.\n3. Targeted edits flip actions at ~99% via GNA/PNA; large drops from path-channel ablations. \n4. Long/short-term split with j-gate-mediated transfer explains overlapping plans. \n5. The “weight steering” generalization and a clean recipe others can reuse.", "weaknesses": "1. Risk of confirmation bias; limited inter-rater reliability reporting. \n2. Define solve-rate protocol, seeds, and statistical tests more precisely; report CIs on all numbers. \n3. One architecture (DRC(3,3)), one domain (Sokoban). \n4. No check on other DRC sizes, Atari-DRC, or different training runs. \n5. Interesting, but current evidence is circumstantial; could be toned down or supported with extra tests (e.g., explicit internal objective readouts). \n6. Success metric is “any alternate action”; consider targeted action flips and off-policy rollouts to quantify downstream task impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGXIoTHsu0", "reviewer": "Reviewer_oYwH", "summary": "This paper provides an important case study that\nmanually inspects the weights and the activations of a successful RNN-based Sokoban solver\nthat was trained by reinforcement learning.\nThe authors showed that, through manual inspection of activations and weights,\n\n-   some layers model possible paths of the boxes (path channels)\n-   some convolutional kernels will extend the paths from the init and the goal, as if running a bidirectional search (plan extension kernel)\n-   resulting paths are rejected by winner-takes-all mechanism\n\nThese claims are checked qualitatively and quantitatively using various tools, e.g., by causal analysis, or by measuring the overlap between\nthe actual movable locations of the boxes and the predicted locations in the path channels.\n\nThis is a unique paper in the modern era; I have never read a paper like this,\nto be honest, although I know the early NN literature has full of papers like this that\nperform a similar manual probing on a CNN for handwritten digit recognition.\n\nThis paper may hype. However I don't know if Neurips is appropriate for this kind of paper,\nbecause, to me, conference papers are usually about technical contributions.\nI really don't know how to review or score this paper unlike other papers.\nI admit I might be narrow-minded, and I prefer to defer the decision to SPCs/ACs.\nI apologize that my review is unusually short despite being from the planning background.", "questions": "A natural next step of this paper might be this:\nWhat would happen if you manually bake in a similar kernel into the RNN during the weight initialization\nand optionally fixing those weights?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I really do not have much to say.\n\nIt may have some implications on the RL-trained the language models,\nespecially those that reuse the same layer several times (e.g. diffusion language models, Universal Transformer, Sparse Universal Transformer)\nand use think tags.", "weaknesses": "The analysis is performed on a particular set of weights of a particular RNN architecture using a particular dataset, random seed, etc.,\nleaving it unclear whether the findings generalize to wider applications,\nor even to a different set of weights from a different random seed.\n(the authors acknowledged this in the appendix)\n\nThe paper does not have a technical contribution.\nRather, the authors simply manually found the existence of path channels,\nwhich renders the use of linear probes / logistic regression probes unnecessary.\n\nI still think it is a bit of a stretch to call the bidirectional path extension / retraction behavior as \"running a search algorithm\".\nIt is clearly not running a systematic tree search;\nIt may not backtrack when 3 directions all failed,\nand it may incorrectly backtrack early before trying all 3 directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zadg0PdGUf", "reviewer": "Reviewer_UMsf", "summary": "This paper analyzes a Sokoban-trained DRC(3,3) agent and shows that it performs internal planning through path channels and plan extension kernels. Using ablation and causal tests, the authors demonstrate that these components are responsible for planning behavior, providing a clear mechanistic view of how model-free networks can develop search-like computation.", "questions": "When multiple boxes are present, how exactly does the network choose which one to move first? The WTA mechanism seems to operate locally across directions, but does it also act across different entities (boxes)?\n\nCould similar path channel dynamics be found in agents trained on non-grid planning tasks (e.g., MiniGrid, navigation, or graph problems)?\n\nHow stable are the discovered channels across random seeds or retraining runs—are their spatial roles (e.g., “box-right”) consistent?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper offers a detailed mechanistic analysis of the DRC(3,3) reinforcement learning agent trained on Sokoban. Its originality lies in the depth of interpretability achieved—the authors go beyond linear probes to reveal path channels and plan extension kernels that implement a neural form of bidirectional planning.\n\nThe study combines qualitative visualization, quantitative ablation, and causal intervention experiments in a careful way. The causal and ablation results convincingly show that these channels are genuinely responsible for planning.\n\nOverall, it provides strong evidence that a model-free ConvLSTM can internally develop search-like planning dynamics, offering a valuable model system for studying emergent planning and mesa-optimization.", "weaknesses": "Despite its depth, the paper can be very difficult to read.  Readers unfamiliar with DRC(3,3) may struggle to follow the layer/tick conventions and channel indexing. A schematic overview early in the paper would improve clarity.\n\nAnother issue is limited generality: all findings are restricted to Sokoban. It remains unclear whether the same mechanisms emerge in other planning-heavy environments or different architectures. Some comparative evidence (e.g., other environments) would help validate the universality of these “path channels.”\n\nThe causal analysis is strong but statistical validation could be deeper: no confidence intervals or significance testing accompany success-rate reductions. Moreover, it is unclear whether the WTA mechanism and negative activation propagation are emergent or enforced by architectural biases. Explicitly disentangling inductive structure from emergent behavior would strengthen the argument.\n\n\nSome other isseus:\n1. Many references in the paper are broken. Line 834, Line 1050, Line 1215\n2. I don't understand the \"?\" in Table 8.\n3. Figure 15 is broken.\n4. Line 041: agent's goal [rectange], should be \"agent's [rectange] goal\"?\n5. L131: locationns. \n6. Line 20: What do you mean by \"parst\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbIvZeM6n6", "reviewer": "Reviewer_a388", "summary": "Sokoban is a 2d grid based game where the aim is to push blocks onto targets.  However, not all moves are reversible, meaning that solutions that work need a understand the longer time consequences of their actions to ensure they don't get stuck.  Deep Repeated ConvLSTM (DRC) is a Recurrent Neural Networks (RNN) model that can learn to play Sokoban. (Bush et al. 2004) demonstrated that that DRC is performing planning by using linear probes on the activations to predict the long term effects of its actions.  They also showed that the plans proceed both forward from the agent position and backward from the targets, and that it was possible via interventions to affect the plan that the agent took.\n\nThe current paper goes into considerably more depth about how the network is actually representing future plans, including assigning interpretable meanings to many of the different channels of the DRC network as opposed to only using linear probes.  This includes finding that each channel encodes a propensity to move in a specific direction.  They also separate out box movement and agent movement channels. Note that not all channels are interpretable, and the full meaning of the redundant layers of each type is not fully discernible either.  Furthermore, they show how the network can deal with overlapping channels, where the same grid square is used at different time points by highlighting the activations at different times.\n\nThey also dig into the specific ways the convolutional kernels are used to build up the plan over inference time (not agent time), including being blocked by walls and what they term Turn Plan Extensions.\n\nThey use this knowledge in a variety of ways, like scaling the magnitude of certain weights to scale to larger maps, and like Bush et al, intervening to make the agent take sub-optimal plans. \n\nFurthermore they demonstrate the winner takes all mechanism where short term plans suppress each other.", "questions": "Did you examine any intermediate checkpoints to see when the different behaviors emerged?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\n\nThe work is incremental on top of Bush et al, 2004, but it does go deeper into explaining how an RNN can solve the planning problem.  This level of detailed analysis of a planning RNN is to my knowledge an original contribution \n\n## Quality\n\nIt's not a fully generative description of the RNN - the knowledge they find is not sufficient to build one from first principles, so it is only a partial understanding.  This in turn leads to the paper begin a bit partial, like the knowledge they find.  However, the hypothesis they generate are generally well tested empirically and illustrated well in the paper.  \n\n## Clarity\nThe paper is generally well written, and it would be straightforward to replicate their results and experiments.\n\n## Significance\nPlanning is a vital step in understanding and controlling the world.  The current paper, while it is not a fully integrated solution may well lead to further insights into how planning is both learnt and performed in other systems like LLMs as well, which would be a very significant potential future impact.", "weaknesses": "As said before, it's not a full explanation of how the DRC networks works, it still provides various valuable insights into how different layers work.  Obviously being able to construct a simplified planning network given the knowledge would be a very powerful contribution, but that is outside of scope of this paper, and this paper is a step towards that goal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)