Latent Planning Emerges with Scale
Authors: Michael Hanna
Emmanuel Ameisen
Venue: ICLR 2026 Poster (2026)
Keywords: planning
feature circuits
circuits
mechanistic interpretability
Primary area: interpretability and explainable AI
TL;DR: We propose a mechanistic definition of latent planning in LLMs, and provide evidence of its emergence at scale in open models.

Abstract: LLMs can perform seemingly planning-intensive tasks, like writing coherent stories or functioning code, without explicitly verbalizing a plan; however, the extent to which they implicitly plan is unknown. In this paper, we define *latent planning* as occurring when LLMs possess internal planning representations that (1) cause the generation of a specific future token or concept, and (2) shape preceding context to license said future token or concept. We study the Qwen-3 family (0.6B-14B) on simple planning tasks, finding that latent planning ability increases with scale. Models that plan possess features that represent a planned-for word like *accountant*, and cause them to output *an* rather than *a*; moreover, even the less-successful Qwen-3 4B-8B have nascent planning mechanisms. On the more complex task of completing rhyming couplets, we find that models often identify a rhyme ahead of time, but even large models seldom plan far ahead. However, we can elicit some planning that increases with scale when steering models towards planned words in prose. In sum, we offer a framework for measuring planning and mechanistic evidence of how models' planning abilities grow with scale.

Published as a conference paper at ICLR 2026
LATENT PLANNING EMERGES WITH SCALE
Michael Hanna"
Emmanuel Ameisen Anthropic
I.I.C, University of Amsterdam
m. w. wwwhaannnaaaa..
smmmanuel@@athropicccoo
ABSTRACT
LLMs can perform seemingly planning-intensive intensive tasks, like writing coherent sto- ries functioning code, without explicitly verbalizing plan; however, the extent to which they implicitly plan is unknown. In this paper, we define latent plan- ning as occurring when LLMs possess internal planning representations that (I) cause the generation of a specific future token or concept, and (2) shape preced- ing context license said future token or concept. We study the Qwen-3 family (0.6B- 14B) on simple planning tasks, finding that latent planning ability increases with scale. Models that plan possess features that represent planned-for word like accountant, and cause them to output an rather than a; moreover, even the less-successful Qwen-3 4B-8B have nascent planning mechanisms. On the more complex task of completing rhyming couplets, we find that models often identify a rhyme ahead of time, but even large models seldom plan far ahead. However, we can elicit some planning that increases with scale when steering models towards planned words in prose. In sum, we offer a framework for measuring planning and mechanistic evidence of how models' planning abilities grow with scale.
INTRODUCTION
LLMs succeed at some tasks that seem to require planning planning-rressonnng about the steps needed to achieve a goal state- witthut explicitly verbalizing a plan. Understanding the extent of models' unverbalized planning important: such latent planning could present Al safety risks, allowing models to engage in scheming without alerting external monitors (Balesni et al., 2024 (424:Kan Korbak et al., 2025). Despite this, empirical evidence regarding LLMs' latent planning remains limited P work on latent planning is largely observational: studies show that future tokens text attributes can be extracted from model activations (Pal et al.) 2023 Pochinkov] 2025 Dong et al.] 2025). ) Only recently has causal evidence for planning emerged, in closed models (Lindsey et al. 2025)
We argue that claims of latent planning must be based on causal, not observational evidence, lest we apply the "planning" label too broadly. We consider LLM engage in latent planning only if it possesses an internal representation of the planned-for token or concept / that causes it to generate we call this forward planning. However, this representation must also cause the model to engage in backward planning. reasoning back from its goal t to generate context that accommodates it.
To understand how latent planning emerges with scale, we test 5 Qwen-3 models of increasing size on simple tasks that could involve latent planning, like completing "Someone who handles financial records is - a/an (accountant)"; we find that only models with 14B+ parameters consistently suc- ceed. We then use feature circuits (Marks et al. 2025 Ameisen et al. 2025) to find the mechanisms that underlie models' abilities. We find that there exist planning features that represent future out- puts like accountant and upweight relevant outputs like "an" (Figure D.. Moreover, although smaller models fail, they possess planning-relevan features that promote the correct answer.
We next have models complete rhyming couplets, where Lindsey et al. observed longer-range plan- ning in Claude Haiku. We find that models employ a circuit that tracks information related to poetry, such as when line is about to end, what to rhyme with; however, even large mod- els do not engage in wackward planning. We then test intermediate planning abilities by steering models towards planned words in prose, and observe forward and backward planning, increasing
"Completed as part of the Anthropic Fellows Program.
Explicit, verbalized planning, in L.MMs chains of thought, is hetter studied Kambhampati et al. 2024

---

Published as a conference paper at ICLR 2026
lant 0.89
say
"an"
Kay Tace
Acccuncing
retaraeiee mna
Finance
Borcone who atudies living organioro in
biologist. Sonsccce
who handlea
Cinanciol
CoooE
Figure 1: Feature circuit for the input Someone who studies living organisms is a biologist. Some- one who handles financial records is, explaining Qwen-3 (14B)'s output, an. The model plans to say accountant, causing it to output the appropriate article, an. Labeled nodes are sets of active transcoder features with shared semanties; edges indicate that the source node increases the target node's s activation. We demonstrate node semanties by selecting a node's feature and showing its top-activating inputs and the vocabulary items that it hpp-ddow--eigght.
with scale. Our results provide the insight into how latent planning emerges scale, showing that Qwen-3 models use various planning mechanisms that scale with model size. We also show that while both forward and backward planning improve with scale, the former develops faster. We thus conduct the largest-scale feature circuit study on open models to date. We provide code here: https: hctps://githubs com/ www.aawwaawaacccc.t.w.t........a
canning-publle
WHAT IS LATENT PLANNING IN LLMs?
Planning is behavior in which one reasons about which actions must be taken (and in which order) to achieve a goal. However, most past work on latent planning in LLMs searches model internals for evidence of a goal, not goal-oriented reasoning. For example. Dong et al. (2025) prompt LLMs to write stories, and probe the LLMs' representations of the input prompt for information about their future outputs. They equate successful probing with latent planning. but see App. G for evidence to the contrary. [Pochinkov] (2025) takes the residual stream of LLMs that are about to start new paragraph, and attempts to decode the topic thereof using Patchscopes (Ghandeharioun et al. 2024); again, successful decoding is taken to entail planning. Pal et al. (2023) also decode models future tokens with probes and Patchscopes Patchscopes-though though they do not call this planning. Lindsey et al. (2025) are unique in providing causal evidence: studying LLMs' ability to complete rhyming couplets, they not only observe representations of the rhyming word that the model plans to output, but also causally intervene on them, changing the upcoming word and its preceding context that accommodates it.
We argue that, if LLM planning entails reasoning about the steps needed to output specific future token, decoding the future token is insufficient evidentiate planning. Consider a model that always :
outputs the same token, or one that outputs 0, 2, 4, 6.... in both cases, probe could likely predict many future tokens, but neither task requires planning. More generally, the decodability of given attribute from model representations does not entail its use in model processing: probes are known decode unused information (Ravichander et al.) 202T). Instead, if latent planning is mechanism that models deploy, a definition thereof should make causally verifiable mechanistic claims.
Inspired by Lindsey et al. we define an LLM given length-n input as engaging in latent planning if it possess a representation of a planned token or concept that:
Condition 1 (Forward Planning): causes to output the specific token or concept at some posi- lion n + k, k > 1. This strengthens the decodability criterion from past work: we require that some representation causes the L.LM to produce not just that i be predictable from the L.MM's s internals.
Condition 2 (Backward Planning): causes it to outpuf a context that licenses said token or concept This requires that models work backwards from the goal to formulate a context that licenses it. Consider the input = The capital of Texas + is - Austin. LLMs may have an Austin representation at the Texas position of s; ablating it stops the model from later outputting Austin. However, this is only backward planning if the Austin representation causes the model to produce is. This is unlikely, given that one can predict is without knowing that Austin is the capital of Texas. Note that some past work focuses representations that do not aid immediately next-token prediction (Wu et al. 2024).
2

---

Published as a conference paper at ICLR 2026
3 TRANSCODERS AND TRANSCODER FEATURE CIRCUITS
To identify causally relevant planning representations, we first decompose model activations into sparse features using transcoders (Dunefsky al., 2024). Then, we find the causally relevant sub- graph thereof, known as a feature circuit Marks el al. 2025 Ameisen et al. 2022).
Transcoders Transcoders are auxiliary models that replace the model's MLPs (Dunefsky et al, 2024]; each transcoder takes in one MLP's inputs and predicts its outputs. Formally, transcoder takes in given MLP's input activations
IR and computes a sparse representation, z E IK" as 2 = f (W Wench + bene). It then reconstructs the MLP's output activations E Rd as h' = Weec + Ddec- ff is activation function, while W enes
Dene: W does and bdee are learned parameters.
Transcoders are useful because they are trained to compute representations z that are sparse and any given input: each feature should fire on only one concept. By contrast, MLPs' hidden activations are often dense and polysemantic, firing on multiple concepts (Olah al. 2017 Elhage al. 2022). If one wishes to determine which concepts a model represents in its activations, it is thus easier to interpret transcoder features.
monosemantic: most dimensions (or features)
We interpret the 7th feature of given transcoder by displaying the inputs that maximize its activa- tion z, We also display the tokens whose unembedding vectors have the highest and lowest dot product with the feature's 's column in W deci these are the vocabulary items that it directly up- and downweights. See Figure 1 for example feature visualizations, used to manually label features.
We often intervene with respect to transcoder features, to verify our interpretation of given feature. For example, we might take feature vector set its activation to O, and observe the change in model behavior. For more background and technical details on transcoders, see Appendix A.
Transcoder Feature Circuits Given model, transcoders trained each MLP thereof, and an input, we construct a transcoder feature circuit (Ameisen et al.] (2255): a weighted acyelic digraph describing the causal relationships between the model's imputs, transcoder features, and logits. Each edge weight indicates the source node's 's direct effect on the target, i.e. the amount by which it directly increases the latter's value. Once features annotated, and similar features grouped together, the circuit serves mechanistic explanation for model's 's behavior on the input, as seen in Figure [|
We compute feature circuits using Ameisen al. algorithm, detailed in Appendix A.2 Unlike other feature circuit techniques, it computes exact direct effect attention patterns and layer normalization denominators. We thus know the precise causal relation- ship between features, ignoring contributions to these quantities, which is often useful in practice. We use the ci reuic i t teeenitttrreee racer library for sircuit-inding and interventions (Hanna et al. 2025).
on the model's
The transcoder feature circuit paradigm helps ensure that any planning features found fulfill our con- ditions, as features are guaranteed to be causally relevant, under the assumptions made by transcoder feature circuits, and we see what intermediate features represent.
4 QWEN-3 MODELS ENGAGE IN SIMPLE PLANNING
4.1 MODELS AND DATA
We study planning in 5 models from the Qwen-3 family (0.6B, 1.7B, 4B, 8B, 14B; Yang et al. 2025). We study models of varying size from one family to draw conclusions about how planning behavior develops as models scale. Note that although these models are instruction-tuned they produce reasonable output on both instruction-fomee formatted and
formatted inputs,
Category a an is are
Example Input
Next Planned
Someone who handles financial records
accountant
There were 5 dogs but 4 left. Now there El animal marino con ocho tentáculos
is el pulpo
el / la
Table 1: Three simple planning tasks. Each task prompts the model to output planned token, preceded by a next token with two possible forms; the planned token determines the correct form.
3

---

Published as a conference paper at ICLR 2026
we both formats. For feature circuit analyses, use Hanna et al. cover 3wen-3 (0.6B-14B); we include Qwen-3 (32B) in our transcoder free behavioral analyses.
(2025) transcoders, which
We craft three simple tasks to serve testbed for LLMs' planning abilities. We choose tasks to which LLMs were likely exposed during pre-training, as model abilities are often stronger on such tasks (McCoy al. 2024). Each task (Table consists of inputs that push the model to produce specific content word, preceded by function word that must agree with it. For example, in the is / task example in Table [ the model must output 1, preceded by the correct form of be. See App. B for details on the construction and composition of these datasets. We discuss a/an in the main text; our successful is/are experiments and less successful el/la experiments are in App. CVD DDD For experiments in another language, see experiments with Chinese measure words in Appendix while experiments on base models are Appendix !
4.2 LARGER MODELS SUCCEED PLANNING TASKS
We first evaluate models' abilities the a/an task, recording their next token prediction each input. We report per-class recall, as performance differs by class. We find (Figure 2 left) that all models have high recall (> 0.8) of a, which is the majority class both in our dataset and English in general. Recall of the minority class an is high (> 0.8) for Qwen-3 14B; small models (0.6-1.7B) always predict the majority class, and mid-sized models' performance smoothly increases.
Note that this is not attributable to models' inability to determine the planned token: in Appendix show that models with under 14B parameters can calculate the answer to is/are questions, but fail to predict the correct verb, producing outputs like... like... there are dog. It thus appears that simple planning (and not just c.g. math) emerges at 4B to 8B parameters.
4.3 MODELS POSSESS PLANNING FEATURES
To determine if models truly plan these tasks, compute each model's feature circuit for each example our datasets, as described in Section We then visualize and qualitatively analyze a subset of the feature circuits, grouping qualitatively similar features together and labeling them.
We find that these circuits contain features that represent the planned token. Figure I shows a typical example from Qwen-3 (14B): it possesses planning features (for accountant) that feed into features that upweight the same token. These activate features that directly upweight the correct next token (a/an). This suggests that models plan to output the target token, which then leads them output the correct next token. As in Lindsey et al. 20025), the planning features (e.g., the accounting feature in Figure D appear to simply represent the planned word, and specifically in planning contexts.
Planning features differ slightly by task. In the ix/re dataset, such features are more common when the answer is small (from 3). The el/la dataset's features fire on the target word English, despite its lack of grammatical gender, relevant to this task. Surprisingly, on a/an and is/are, even porrlypperformig models have planning features, suggesting nascent latent planning mechanisms.
Articlo Recall Across Model Sizes (a/uc)
Moon Proportion of Influene Through Plansing Nodes
1.4
"a" Cornect
*a" Incorrect
"an" Cornect
Acticle: 12*
Actiole: "an"
Owen3-0.6 Owen3-1.78 Owen3-4B 4B
Owmi3-BB
Owe
Qwenb-33B
Owen3 0:53 Owee3 1.79
@wen3 43
Qwena 4B
Owen3 14B
Figure 2: Left: Qwen-3 family models' recall of correct article on the a/an task. All models can recall but models < 8B have lower recall on the less-common an. Right: The mean proportion of influence flowing through planning nodes in the a/an dataset, by model, article, and correctness. On an examples where the model correctly predicts the next token, more influence tends to flow through the planning nodes. This effect is reversed and weaker for the majority class a.
4

---

Published as a conference paper at ICLR 2026
Effect of Zeço Aáletions en p(correct)
Effect of Direct Zerv Ablations on p(correct)
0.25
0.1
Change St picorect
0.0
0.00
-0.1
-0.25
-0.30 -0.50
-0.3
-0.75
-1.00
-0.5
Effect Multip Interventions on p(correct)
Effect Direct Multiplying Interventions on p(correct)
1.00
0.8
Ca [aange-ipttorreee plorrze))
0.75
0.6
0.4
0.50
0.2
0.25
0.0
-0.25
-0.4
Owen1-0.00 6B
Owen2--1.71 1.71
Owend
Owen3-Un
Owen1-1445
Owen3-0. oB
Owendd)).71
Owend-43
Owen3-en
0wwe33-48
curred acticle "N" (Lwnd))
connect trtadl "ac" (Lownd)
indisiduall "sa" points
individual "wo" points
Figure 3: Left: Change in p(correct article) caused by zero and multiplying interventions on plan- ning features. As expected, ablating these harms performance, while upweighting them improves it; however, both affect primarily an examples, the minority class. Right: Change in p(correct article) caused by direct-effect interventions. Effects are smaller, indicating that planning features act both directly (by upweighting the correct article) and indirectly (by activating e.g say "a/an" " features.)
4.4 PLANNING FEATURES ARE CAUSALLY RELEVANT
We now verify that these planning features truly drive the model's prediction of the correct next to- ken. We start by programmatically finding each example's planning features; feature considered planning-relevann if is active at the last position of the input (is), and it either upweights the planned word (or prefix thereof), or contains it in out of 10 of its top-activating texts. We find that this yields similar planning features to those found via manual search.
With these features, we perform two causal relevance analyses. First, we ask how important are planning nodes according to our circuits? Each edge in the circuit reflects the direct influence of a source node on a target node, but we can also consider the total flow from a source to target node, which might travel via multi-node paths. To quantiff the importance of the planning nodes, we measure the proportion the total flow between the circuit's 's inputs and logits that is mediated by the planning features, comparing the flow in cases where the model is in/correct.
We find (Figure [] right) that when models predict the minority class an correctly, more of the total influence flows through the planning nodes. This effect is reversed (and weaker) for the majority- class a case, despite roughly equal planning node counts across classes, suggesting that planning nodes are generally helpful for these examples. In neither case is the proportion large, but this is unsurprising: much of the flow is likely mediated by nodes that identify the need for an article such as a or an, upweighting them both, rather than discriminating between them.
Second, we causally intervene on planning features. For each model, we a) take the examples on which succeeds and ablate the planning features (e.g. accountant and say "acc" in Figure M.. setting them to zero, and b) take the examples on which it fails and highly upweight their planning features, setting their activations to 5x their usual activations. If these features indeed cause models to output the planned token, these interventions should harm and improve performance respectively.
We find (Figure left) that features are indeed causally relevant. Feature ablation (top left) harms model performance, but only minority-clase an examples. Similarly, boosting planning features improves performance drastically an examples, with larger models seeing slightly larger improve- ments; however, the effects analysis, and suggests that planning nodes are more important for minority-clan examples where models must work against their priors. This intervention effective on Qwen-3 4B and 8B, indicat- ing that although their overall performance is worse than Qwen-3 14B, they likely rely on similar planning mechanisms, with planning features encouraging the production of an when necessary.
examples are almost zero. This asymmetry aligns with our prior
5

---

Published as a conference paper at ICLR 2026
We also note that our feature interventions are more successful than a random baseline: while zero ablating randomly selected features active at the last position of the prompt occasionally harms performance, multiplying random features fails to boost model performance (see App. Forr details).
Discussion Our results suggest that Qwen-3 engages in simple backward planning: however, it unclear if this is driven by direct-effects alone. The accountant feature might have a high co- sine similarity with the unembedding vector for an, upweighting its logit. This, combined with mechanism that upweights both a and an in relevant contexts, would suffice to upweight the correct article, observe. We disprove this by performing direct-effects interventions: we upweight the planning features, but freeze the model's 's other features, blocking second-order effects.
This intervention's effects (Figure 3] right) much weaker than the original interventions: zero ablations are less harmful, and multiplying interventions harm performance as often as they help. The planning features' importance can thus not be explained by direct effects alone, suggesting that the say "a/an" features play an important role in mediating planning. For more evidence, see Appendix L where we steer on the planning features, and find that while this often causes models to output the planned-for word, it seldom causes them to output a/an.
One could also hypothesize that although say "a/an" " features are involved in alan planning, the model treats noun phrases (like an accountant) like single, multi-token word; no planning is in- volved. However, models also plan when outputting "there dog left", where this multi-token argument is much less plausible. We thus maintain that simple planning occurs in these cases.
One outstanding question is the source of the gap between Qwen-3 (14B) and its weaker 4B and 8B counterparts; what makes their circuits nascent rather than fully-developed? In Appendix MM] we examine this question and find that when these mid-sized models fail on an examples, they have far fewer planning features active than when they succeed. By contrast, smaller models seldom have any planning features active, and Qwen-3 (14B) has many planning features active in both cases.
5 QWEN-3 USE LITTLE PLANNING WHEN COMPLETING COUPLETS
The preceding experiments show that Qwen-3 models more successfully plan as their size increases, but leave open the question of longer-range planning mechanisms. There precedent: Lindsey et al. (025) find that, given the first line of a rhyming couplet, like He saw carrot and had to grab it. Claude-3.5 5 Haiku produces the next line His hunger was like a starving rabbit using a rabbit feature that controls the rhyming word and generates a coherent context. Motivated by this, we study Qwen-3 models on rhyming couplets, searching for long-range planning.
5.1 QWEN-3 MODELS OFTEN SUCCESSFULLY RHYME COUPLETS
We first test whether Qwen-3 models can complete rhyming couplets at all. To do so, we generate dataset of 985 first lines of couplets, by prompting Qwen-3 (32B) to produce rhyming couplets on 43 topics, ranging from coming of age to animals and wildlise, and taking the first line of each. LLM generation of couplets avoids cases of couplets memorized from the training data. We then greedily sample a second line of the couplet from each model, and evaluate its rhyme with the first couplet by extracting the last word of each line. extracting their vowels and final consonants using CMUDict (Carnegie Mellon University 2014 Bird & Loper 2004), and verifying that they match. Our results (Figure 5 left) show that larger models rhyme with 50+% accuracy; smaller ones fail more often. Models engage in slant or assonant (vowel-only) rhyme, rhyming words like craze with page; models with 8B parameters produce a valid assonant rhyme in over 70% of cases.
5.2 LARGER LLMS' POETRY ABILITIES ARE SUPPORTED BY A RHYMING CIRCUIT
To test whether models plan when completing couplets, we again use transcoder circuits. For each model, we filter the examples from our dataset to those where the model completes the couplet's second line with a rhyming word. We then attribute from this rhyming word's logit, given the input leading up to the rhyming word; that is, given stayed.... Hoieetickers where the shadows laid, we find the circuit explaining the model's prediction of laid. We limit this to 100 examples per model. See App. H.T for rhyming couplet data details.
input like Fury burns where calm once

---

Published as a conference paper at ICLR 2026
Output "shadows"
Ouspat "leid"
Qutput s An
and
clyaasa with -aped
skyaea with -apes
ced of line
end
end
Pary
buina
where
OncC
2C050.
Ropc
flickers
abaddwd
2oid
Figure 4: A feature eircuit for the couplet Fury burns where calm once stayed, y Hope flickers where the shadows laid, explaining Qwen-3 (1 (14B)'s 's decision to output shadows laid, n,. Halfway through outputting the couplet's second line, the model's "near end of a line of poetry" features activate. These cause it to attend back to the end of the first line, where "end of line of poetry" features are active, and to move "rhymes with -ayed" features into the second line. These influence the model's outputs, eventually leading to laid. End of line features then cause it to output. 'n.
We qualitatively analyze the circuits, and find that in larger models, an interpretable circuit emerges. Given the first line of the couplet, the model begins to generate the second with little planning. Near the end of the second line. the model recognizes that it is near the end of a line of rhyming poetry, activating near end of line features. These cause it to attend to the end of the first line, drawn by the end of line features active there. Rhyming features (e.g. rhymes with "-ayed") at the end of the first line thereby activate similar features in the second line. There, these features remain active until they eventually cause the model to output a rhyming token. Once the model completes the rhyme, activates end of line features and stops generation. Figure 4 deppccs this process.
We defer detailed evidence for our circuit to Appendix H.2 There, we show that end line features causally responsible for both the model's decision to end a line of poetry, and for indicating where the model should attend to, in order to extract the rhyming features. We similarly show that the near end of line features cause the model to attend back the end of line feature position. Here, we focus on the question: does the couplet circuit involve planning?
5.3 QWEN-3 MODELS PLAN FORWARD, BUT NOT BACKWARD, TO COMPLETE COUPLETS
If the couplet circuit involves planning, view the rhyming features at the end of the couplet's first line as the most likely planning features. They clearly represent the rhyme to be output, and circuits indicate that they influence the model's decision output rhyming words. However, must still test that both forward and backward planning occur when models generate rhymes.
We first define rules to automatically find rhyming features. This is challenging, Qwen-3 models represent e.g. an -ayed feature via separate -ai- and -d sound features, which specify the vowel and final consonant of the rhyme. The top-activating tokens for such features tend to be subwords, and may employ multiple, potentially nonstandard spellings for a given sound; see Figure for example features. As a heuristic, we identify features whose top- 10 max-activating tokens are short (under 5 characters), and do not represent a single word (they activate on the same word at most 5 times). We also require that at least 7 of these 10 tokens start with the same vowel, or end with the same consonant, to ensure that the feature's 's top-tokens all represent one sound. This definition captures rhyming-relevant features with relatively high precision but only moderate recall.
Next, for each couplet, we downweight the rhyme features at the end of its first line, multiplying their activations by -3. We then sample random couplet with a distinet rhyming sound, and upweight its rhyme features, multiplying their original activations by 7; we find these steering hyperparameters via manual search. We then generate a completion to the first line of the original couplet, while steering on the end of the first line. We measure rhyming accuracy with respect the new rhyme.
is harder to quantify Condition 2-whether given context licenses a specific word (or set thereof) test which context (the original or steered
as opposed to licensing many words. However, one) best enables the model to predict the new rhyme, when the model is steered towards the new rhyme. If the new context indeed licenses the new rhyme better, the model should more accurately
7

---

Published as a conference paper at ICLR 2026
Hlyne and Acsmvnn thnsee Acomc
thymes
Soand Noaeaay
Soa eaurad acauuayy
5.33
Owent-2.77
UwaRd-45
Gwand 36
Ovend 10
Oueld
Figure 5: Left: 3wen-3 rhyming accuracy. In the base case, models have moderate rhyming accu- racy, reaching 0.6 at 14B parameters (solid blue); when we consider assonant (vowel-only) rhyme, Qwen-3 (14B) achieves 0.8 (dashed blue). When steered to predict a new rhyme, model accuracy is only moderate for perfect rhymes (solid orange), but improves with scale, and is better on assonant rhyme (dashed orange). Right: Model rhyming accuracy when trying to predict a token satisfying the couplet's ss the steered rhyme (blue lines) or original rhyme (red lines), given the original (solid) or steered (dashed) context. The model predicts the steered-for rhyme with similar accuracy given the original or steered context. This suggests that the steered context does not better license the rhyme.
predict a rhyming word given it. To test this, feed the model both the original and steered couplet completions, with their last word removed. We then record the model's generation given each, when steered towards the new rhyme, and compute rhyming accuracy with respect to the new rhyme.
We find that models do engage in forward planning: Figure (left) shows that steering on the rhyme features does change the model's rhyme to the new rhyme in the case of larger models (8B-14B). Though accuracy is only moderate (40%), normal rhyming accuracy was similarly modest at 60%, and assonant rhyme accuracy is higher (up to 60%). Moreover, we observe that steering changes both the final rhyming word and the intermediate context; see Appendix H.3 for quantitative evidence.
However, Figure 3 (right) shows that the intermediate context generated under intervention does not necessarily license the new rhyme better. When we steer the model, it is equally likely to output the injected rhyme given the steered intermediate context (light blue, dashed line) as when given the original one (dark blue, solid line). Giving the model the intermediate context produced with steering. but not stcering it, elicits the original rhyme with relatively high accuracy: near 60% across models (light red, dashed line). This is low compared to the accuracy given the original context (near 100%; dark red, solid line), which could suggest that the original context better licenses the original rhyme. However, the fact that we only intervened examples where models rhymed successfully inflates this accuracy. Overall, these results suggest a lack of strong backward planning.
5.4 LARGER MODELS MAY USE LOCAL PLANNING FEATURES
Though the backward planning results are mostly negative, results for larger models (8B- 14B) trend in the right direction: they more accurately predict the steered rhyme given the steered context, and less accurately predict
Model Steering Metrics by Size
Rrequeccy.
the original rhyme; in App. H we see that
their steered generations overlap less with orig- inal generations. Moreover, manually inspect- ing Qwen-3 (14B) couplet-completion circuits showed that while most couplet circuits in- volve second-line rhyming features that up- weight rhyming words, some instead involve say X features that upweight a specific upcom- ing word. These often coincide with rhymes that require some setup, such as a say "night" " feature occurring before the model outputs in the night. These are prime candidates for local planning features, that plan for short phrases, but not whole lines; we thus test whether they elicit backward planning in models.
21
Qwen3-0.68 Qwens-1.78
Qwen3-48
Qweri-88
Qwin3-14B
Modil Size
Adaptation metries by model. As models grow, so does the proportion of (1) out- puts containing X (blue), and (3) coherent and X- containing outputs that also adapt the context to license X (green). Few examples do all three.
Figure 6:
8

---

Published as a conference paper at ICLR 2026
We first identify potential planning features, searching our couplet circuits for say X features that upweight the output rhyming word, but are active prior to when X is output: such features might adapt the preceding context to license that word. We then steer models using these features on 100 inputs from the TinyStories dataset (Eldan & Li 2023), which we use as a source of neutral input text. For each steered output, we check if it (1) contains the steered word, (2) is coherent, and (3) adapted the context to fit the steered word. We evaluate (1) programmatically use Claude 4 Sonnet evaluate (2), and manually verify (2) and evaluate (3) on subset of outputs that satisfy (I) and (2). See Appendix for experimental details.
We find (Figure o that steering on these say features often induces models output (blue bars). Moreover, for outputs that are coherent and contain X, models cedieial- larger ones do adapt their outputs to produce whole phrases like in the night or had a recurring dream (green bars). The scaling trend likely occurs because larger models have more such local planning features in their couplet circuits. However, this phenomenon sensitive steering strength, and these features occur only in small minority of couplets. We hypothesize that such features are part of an emerging planning mechanism larger models, much as a/an and is/are planning can be seen to emerge in Qwen-3 (4B); larger scale, models may more reliably engage in local planning. Still, more study is needed to confirm the role these features play.
6 RELATED WORK
(Feature) Circuits We build on prior work circuits, which attempts to capture (ideally min- imal) set of units that are causally relevant to and explain a model's 's behavior on a task (Olah et al. 2020 Elhage et al., 2021: Conmy et al. 2023) Early LLM circuits were composed of attention heads and MLPs, and explained how models performed indirect object identification and the greater- greater-than operation (Wang al. 2023 Hanna al. 2023). Circuits composed of features from sparse autoen- coders or transcoders have the added benefit of having interpretable nodes; however, finding them expensive and requires auxiliary models. They have been used explain gender bias, syntactic processing, and more (Marks et al. 2025 Hanna & Mueller 2025 Lindsey et al. 2025).
Planning Tasks LLMs' grammatical agreement abilities, as in our a/an, is/are, and el/la tasks, have been widely studied. LLMs generally excel at agreement, preferring sentences with correct agreement over incorrect ones (Warstadt et al., 2020 Chang & Bergen 2024). Prior mechanistic work on agrecment is more limited to is/are and the broader phenomenon of subject-verb agreement:
past work has found linear subspaces, neurons, and sparse features relevant to it (Lasri al. 2022
Finlayson et al.] 2021 Brinkmann et al.) 2025). Past work has studied LLM poetry and rhyming abilities in the context of building and evaluating poem-generating systems (Sawicki et al.) 2023
Chen et al. 2024 Suvarna et al. 2024 Lindsey et al. (2025) provide the first mechanistic study.
Planning Mechanisms Section 2 discusses past work, but contemporaneous work also addresses
planning: Nainani et al. (2025) search for code planning feature circuits in Gemma-2 (2B; Gemma
Team 2024), while Maar et al. (0255 investigate poetry abilities across models using probes.
7 DISCUSSION
How General Are the Discovered Planning Mechanisms? Our evidence suggests that the plan- ning mechanisms we discover are not general in the sense that the model uses the exact same circuit for all planning tasks. Whether a given model plans on a given task regulated by the model's capacity, well as the task's complexity, along with its frequency and importance (in terms of training loss). Thus, larger models plan more, and common planning tasks are learned faster, result- ing in piecemeal planning abilities, rather than large set of abilities and unified mechanism.
Successful planning circuits often follow a motif: there are planning features indicating the planned word, which then activate downstream features responsible for backward planning. However, mod- els may learn to plan in one case (a/an agreement) but not others (couplets), simply because the former is more important reducing its loss than the latter, and the model is too small to learn both.

---

Published as a conference paper at ICLR 2026
Extensions to Complex Tasks In this paper, study relatively simple tasks, Qwen-3 (14B) and smaller struggle with complex planning tasks (like e.g. chain-of-thought unfaithfulness, as in Lindsey et al. (2255); this prevents us from studying such tasks. However, open source models become more competent, circuit-tracing should still be able to address these planning behaviors. Sometimes, in chain-of-thought unfaithfulness examples where the model plans for single-token answer, it may suffice to simply attribute from the given answer token, as done here.
In other cases, e.g. detecting a hidden goal that drives model behavior without producing one "smok- ing gun" answer token, we may want to attribute back from more general high-level model actions, such as "Why did the model produce a refusal?" "Why did the model make a given suggestion?". Though little past work does so, attributing from such higher-level actions is possible by attributing from arbitrary directions in activation space. In this case, one must identify a direction in activation space corresponding to such an action, and attribute from this. One could identify causally relevant directions for such actions via probing or difference-in-menn approaches.
Cross-model Generalization In this paper, we study only one model family, to focus the effects of scale planning. The lack of transcoders for similarly competent models currently hinders com- parisons across model families, but we believe such cross-model comparisons would be valuable. That said, past work indicates that circuits can generalize across model family and scale: the circuits for e.g. multi-hop reasoning in Gemma-2 (2b) (Hanna et al.) (2025) look similar to those in Claude (Lindsey et al. 2025), vastly larger and more competent model.
We also note that our planning framework can be applied without transcoders. For example, one could train probes extract a planned word or rhyme family and perform interventions with respect to them; see Maar et al. (2025) for work along these lines. However, this approach is less flexible, requiring new probes for each task, and losing the fine- fnn-grained insights of transcoder feature circuits. Recent work suggests that circuit-tracin may even be possible with neurons alone (Arora et al., 2025), allowing for both fine-graing fine-grained insights and cross- model comparison.
8 CONCLUSION
Our experiments have shown that some Qwen-3 models engage in latent planning, possessing fea- tures that represent the planned word and causally influence both the output word and the context preceding it. Both forward and backward plamning abilities improve with scale, but the former improves before the latter, even in Qwen-3 (14B), planning multiple tokens ahead is rare.
Why might planning only begin to emerge at scale? We hypothesize that planning, especially back- ward planning, is costly to implement: models must learn not only to plan for a specific token, but also how to plan backwards for it in context-specifes way; a/an planning and couplet planning have distinct mechanisms. Thus, models may learn to plan only after exhausting other, simpler ways of reducing their loss. [Bachmann & Nagarajan (2024) suggest that teacher forcing in L.M pre-training may also reduce the pressure to plan: even if model fails to backwards-phe -laaarddddaannnndd for a crucial agrecing token like an, teacher forcing provides that token anyway. Models that suffer the consequences of their poor planning, such as those trained with on-policy reinforcement learning methods, may thus face more pressure to plan.
Whatever the reasons behind this, latent planning abilities in Qwen-3 models up to 14B parameters are still nascent. This is relevant for scheming, an AI safety risk where models work towards secret goals (Balesni et al. 2024); past work has induced scheming in models, and caught them by reading their chains of thought (Meinke et al. 2025). Models with strong latent planning abilities might thus cause concern, but observe little complex planning in Qwen-3. What observe instead is latent planning abilitics that appear to improve with scale-and merit monitoring as models grow.
In this paper, we have provided framework for doing precisely that; however, monitoring latent planning with feature circuits is still a significant technical challenge. Open- Open-source circuits work on models above 8B parameters is still rare. Large-scale work on feature circuits is yet scarcer, due to the lack of open transcoders for large models. As mechanistic interpretability' 's seeks interpret more sophisticated behaviors, its methods must scale match the models that possess them.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We conduct our experiments with openly available models, including both LLMs and transcoders. We release the data and code used as part of this study in the following repository: https://
github. ccm/ hwwnamw/ hannaamwmmddee.. http://molllllllooo waallp-lannnngpgpe
ic Our experiments can be run with as little as 40GB of GPU RAM, though they will run much faster on 80GB of memory, and quite quickly (around GPU-day) on 140GB of RAM (i.e. one NVIDIA H200 GPU). Note that features, models, transcoders, and circuits are large; we recommend having 3TB of disk space available.
ACKNOWLEDGMENTS
The authors thank Denis Paperno and Jim Marr for insightful conversations, and thank the An- thropic Fellows Program for enabling and supporting this project. MH is supported by OpenAI Superalignment Grant and a Google PhD Fellowship.
REFERENCES
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDouggll, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL |RRpst:ss //transformer-cia www.itiirrcttts.. puws
puls wwwppppu/o202/ /0wp://22///// htt.://///tttiuuttttoooooo grapoonnn-rgaaoooeeeeeaa me thods. html
Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, and Sarah Schwettmann. Language model circuits are sparse in the neuron basis. https://transluue
November 2025.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learn- ing, volume 235 of Proceedings of Machine Learming Research, pp. 2296-2318, PMLR, 21-27 www.sss...pppppp.mmmmmmmmmmm
Jul 2024. URL htttps://proceedd.iit.
press v235/ w235/bachmann222mmm www.baachhhhncccccmmc a html
Mikita Balesni, Marius Hobbhahn, David Lindner, Alexander Meinke, Tomek Korbak, Joshua Cly- mer, Buck Shlegeris, Jérémy Scheurer, Charlotte Stix, Rusheb Shah, Nicholas Goldowsky-Dill, Dan Braun, Bilal Chughtai, Owain Evans, Daniel Kokotajlo, and Lucius Bushnaq. Towards evaluations-based safety cases for ai scheming, 2024. URL https://arxivi org/abs/
14111033351
Steven Bird and Edward Loper. NLTK:
The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214-217, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https::tp.............. //acl antho Loghhology..o. 3043334-3331
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil. Carson Denison, Amanda Askell, Robert Lasenby. Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning, Transformer Circuits Thread, 2023. https:/transsfrmee-- htwp:///0000000ooomaaaaeaeeeeee
Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. Large language mod- els share representations of latent grammatical concepts across typologically diverse languages. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pp. 6131-6150, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-18894 189-66 doi: 10.18653/v1/ 122..nacc--oog.33 312. htt.tttss:t//e/ peattss//act /ttp:///..//////aaaaaaoooooo//// naacl long 3127

---

Published as a conference paper at ICLR 2026
Carnegie Mellon University. The carnegie mellon pronouncing dictionary, 2014. URL http:
//www. speech. Cs. cmu. edu/ eew-lliiiinnn birniiiiin// crrri--ii/mmmdiiii.. cmudict Version 0.7b.
Tyler A. Chang and Benjamin K. Bergen. Language model behavior: comprehensive survey. Computational Linguistics, 50(1):293-350, 03 2024, ISSN 0891-2017.
doi: 10.1162/coli: a..
00492. URL
Yanran Chen, Hannes Gröner, Sina Zarrieß, and Steffen Eger. Evaluating diversity in auto- matic poetry generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19671-19692, Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10. 10.18663/v1/222..mmll 1097. URL https::/accaatttteeeee .org 2024.emnll.. main. 1097/
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonoo. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu, Emergent response planning in LLMs. In Forty-second International Conference on Machine Learning, 2025. URL
https:// fttt::://opennevieeee ew. net. http://rrmmmi/ iim 79P8ULP
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL cettps://operreviee http://nwwveeeeeeeeeeeeeeeeoeeee www.tneee/oooimeoeomm wtw.oormmmmdoooHccccsco. J6ZHSSAAOO
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL htttt:://arxi...o htt.://///a///..000//
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
htmm..
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. URL http:://///aassfmeeeecete
htbb/2022/toyy htte:/eltymmooee///eee weteee/innie.eeee html
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shicber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1828-1843, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 a.cl- 144. org/ 2021 monc-llonn... 144/
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL https:
//axxiv.org/abs//20000 00118
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: a unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning. ICML'24. JMLR.org, 2024.
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Un- der the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 240-248, Brussels, Belgium, November 2018. Association for Computational Linguis- tics. doi: 10.1 1653//1/W18-5444
URL httpshhts: http::///aaaaathlaaaooc www.anhoooooyooooooo 81////11155552 842.66
12

---

Published as a conference paper at ICLR 2026
Michael Hanna and Aaron Mueller. Incremental sentence processing mechanisms in autoregressive transformer language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3181-3203, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979- 8-89176-189-6, doi: 10.18653/v 1ttpw///2222p// 1202:/0.222....moooeee.... naacl-long.164 URL /tttttst//cllattoooo.o. hta//aaaaaaatoooooa.aaaaaa
2tg/2025. -ong. 164/
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL Forum?id- forum??dd-4PocNQQRGR
Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen.
circuit-tracer. com sa seee///eyyyeeeeerrccciiiiiaooeoe tccittttrttttttttrrererrreea 2025. The first two authors contributed equally and are listed alphabetically.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. [id=F76wwH 6bwRSLeK
net net//orummf
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: LLMs can't t plan, but can help planning in LLM-modulo frameworks. In Forty-first International Conference Machine Learning, 2024.
URL
net/ forum? Td 4
Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jen- ner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg. David Lindner, David Luan, Aleksander Madry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fa- bien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojeiech Zaremba, Bowen Baker, Rohin Shah, and Vlad Mikulik. Chain of thought monitorability: A new and fragile opportunity for safety, 2025. URL httpt:tttsattt /tt:::///xiii// http//////abbb////oe
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. Prob- ing for the usage of grammatical number. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers). pp. 8818-8831, Dublin, Ireland, May 2022, As- sociation for Computational Linguistics. doi: 10. 0666:3////0222..cccooo..../ URL https: //aclanthology.oom www.oloooooor//222222222 at0.
603/
Jack Lindsey, Adly Templeton, Jonathan Mareus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Trunsformer Circuits Thread, October 2024. URL httpp::/ttaatssstttttt [index. html
rcuits.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL pub/2025 http////ttrrbbuioooonggaaeaeeae hbw.nggggg.pp//ooo.......... html
Jim Maar, Denis Paperno, Callum McDougall, and Neel Nanda. What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation. In preparation, 2025.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations true/false datasets. In First Conference on Language Modeling, 2024. URL htt.hhttss///ppneeeee
www. 2 fyHY
13

---

Published as a conference paper at ICLR 2026
Samuel Marks, Can Rager, Erie J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference Learning Representations, 2025. URL https:
7 //oeerevieew htt.:/iwwwwweeeeeee..
d=
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas Griffiths. Em- bers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41)):233222012,, 2024, doi: 10.1073/pnas.232222201 URL htt.https.... httpp:://www.... www./wwwwwpa.aa.....mm.....aa.aa pnas org/ htt////ii//b///// 11tp:////b...11113//11 p 2322420121
Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming, 2025, brg/abs. 2412. 04984
Jatin Nainani, Sankaran Vaidyanathan, Connor Watts, Andre N. Assis, and Alice Rigg. Detecting and characterizing planning language models, 2025. URL http::::/axxi...rg/////o//
25C8.18098
Neel Nanda and Joseph Bloom.
Transformerlens.
https://github. com/
2022.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization.
Dis-
till, 2017. doi: 10.23915/distil.00000 Feature- viwareevisaaaizatiion
URL http:::/dist.ll.pubbbb
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction circuits. Distill, 2020. doi: 1//isstill..000022.00000
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: strat- egy employed by v1? Vision Research, 37(23)3331-3322 1997. ISSN 0042-6989. doi:
URL htttp:://ww..ciincttee/ www.scieenedirett
com/ http://c..nnneerrtt..eea///////2
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Antici- pating subsequent tokens from single hidden state. In Jing Jiang, David Reitter, and Shumin Deng (eds.). Proceedings of the 27th Conference on Computational Natural Language Learn- ing (CoNLL), pp. 548-560, Singapore, December 2023. Association for Computational Lin- guistics. doi: 13663//1/0023.cmll// 1.37. URL https: http://aaaaaaaaacacaccao.occalea org/2023.
con11-1.377
Nicky Pochinkov. 2025. URL
Parascopes: Do language models plan the upcoming paragraph?,
parascopes-do-
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Arianna Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference Computational Natural Language Learning, pp. 194-209, Online, November 2021, Association for Computational Lin- guistics. doi: 00.18633/v1/2022 wo..l.11....l1115... URI. htt.s: accl anthology org/2021.
con11-1.157
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Main Volume, pp. 3363-3377, Online, April 2021. Asso- ciation for Computational Linguistics. doi: 18.186 1/202l..ccmmmmmmmm otg/2011. www.-maia.naa.ae.e.eem.m
URL https:
Piotr Sawicki, Marek Grzes, Fabricio Góes, Dan Brown, Max Peeperkorn, and Aisha Khatun. Bits of grass: Does gpt already know how to write like whitman? In ICCC. pp. 317- 321. 2023.
URL https:: /ttp///aaaiioallceetttiiiaaaeaaa htt.:/aaiiiiaattttte//////////// //tlo////oo///////oooooooaaaaaaa
14

---

Published as a conference paper at ICLR 2026
Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. PhonologyBench: Evaluating phono- logical skills of large language models. In Sha Li, Manling Li, Michael JQ Zhang. Eunsol Choi, Mor Geva, Peter Hase, and Heng Ji (eds.), Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowL.LM 2024), pp. 1 1-14, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1.11663//1/2224koowlll
URL
https::/acclanhclooog.yo wwg. 2024 knw.1ln 1.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In
The Eleventh International Conference on Learning Representations, 2023. URL hetps:
//openreview net/fooum?? d=
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey. Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Trans- actions of the Association for Computational Linguistics, 8:377-392, 2020. doi: 10.1162/
t032_
httpah/oooooooooogo/..0..... wwwp://2...eeeeeeeeeeee 1.257
Wilson Wu, John Xavier Morris, and Lionel Levine. Do language models plan ahead for future tokens? In First Conference Language Modeling, 2024. URL
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttps://aaxvv.. org/ http:/aa//2255005555 09388
A DETAILS OF TRANSCODERS AND FEATURE CIRCUITS
A.1 TRANSCODERS
In this section we provide details on transcoders general and the specific transcoders we use.
Transcoders Past work has attempted to characterize the features encoded in model activations by examining the inputs that most strongly activate each neuron (dimension) of a given activation vector. However, interpreting neurons is difficult, as they are seldom zero and often polysemantic, firing for multiple reasons (Olah et al. 2017 Elhage et al., 20222. Sparse dictionary learning solves this problem by decomposing activations into sparse and (ideally) monosemantic feature vectors Olshausen & Field 1997; Bricken et al. 2023). As only a few dimensions, or features, of the vector are active on given input, and each feature fires only one concept, these are much easier to interpret.
Sparse dictionaries come in many forms. Sparse autoencoders (SAEs; Bricken al. 2023 Huben jtt al. 2024) are the most common type, encoding and reconstructing activations from the same location. We use per-layer transcoders, which encode MLP inputs and reconstruct MLP outputs (Dunefsky et al. 2024); see F diagram. Lindsey et al. (2024) also introduce cross-layer transcoders, which take in MLP inputs, and are jointly trained to predict contributions to all down- stream MLPs' outputs. These are generally sparser (for a given level of reconstruction error) but also more computationally costly to train and more memory-intensiv to deploy. Importantly, while Ameisen et al. (2025) use cross-layer transcoders for their firuuitfinding, per-layer transcoders can also be used.
Formally, (per-layer) transcoder takes in activations h E IRd from a given MLP's inputs, computes the sparse representation z E RR", and reconstructs the MLP's output activations h' E I2 as follows:
z = f (W ench + bene)
(1)
h'' WeecZ + bdecs
(2)
15

---

Published as a conference paper at ICLR 2026
Logits
Densa MLP Outputs
Transcoder Decodar
MLP
Sparse Transcoder Features
Transcoder Encoder
Inputs
Dense MIP
Figure 7: A diagram of transcoder. The transcoder takes in the dense MLP inputs, computes a sparse representation thereof, and then reconstructs the MLP's dense outputs.
Here, I is activation function (often ReLU, JumpReL.U, or Top-k), and W encs bencs W den: and Daec are learned parameters. L.M transcoders are trained to minimize both the MSE between h and h' and the norm of ? The LLM is frozen, and the transcoder trains on up to billions of tokens. The reduction in polysemanticity is achieved by setting the sparse representation size to be much larger than the input size. In doing so, one reduces the pressure on the model to cram many features into a small number of dimensions, as is thought to cause polysemanticity (Elhage al. 2022
We interpret the ith feature of a given transcoder by displaying the inputs that maximize its acti- vation z, We also display the tokens whose unembedding vectors have the highest and lowest dot product with the feature's s column in W dec' these are the vocabulary items that it directly up- and downweights. See Figure [ for example feature visualizations, used to manually label features.
We often intervene with respect to transcoder features, to verify our interpretation of given feature. To do so, take the original feature vector and perform desired interventions out a feature's i activation, yielding z'. We compute A Wdee den (z' z), and add A to the output of the corresponding MLP during the model's forward pass.
by e.g. zeroing
Qwen-3 Transcoders For our experiments, we use Hanna et al.Js (2025) Qwen-3 transcoders: These circuits are ReLU transcoders, all with a hidden dimension 163840. They take MLP inputs post-inputLLayerNome and predict the MLP's outputs.
A.2 TRANSCODER FEATURE CIRCUITS
Formally, feature circuits are weighted acyclic digraphs. The source nodes are input embeddings and nodes corresponding to each transcoder's reconstruction error h' h'. These flow through transcoder feature nodes, nodes that correspond to a given vocabulary item's logit. Each edge's weight is the direct effect of the source node on the target, i.e. the source node's 's effect on the target's value, unmediated by other nodes.
We compute feature circuits using Ameisen et al. (2025 algorithm, which works as follows.
Local Replacement Model The first step of attribution is to incorporate the transcoders into the model's 's computations for given input. We thus replace the model's MLPs with their corresponding transcoders, plus a reconstruction error term equal to the difference between the MLP's output and the transcoder's reconstruction. This yields local replacement model, which behaves identically to the original model, but only on the given input, as reconstruction error terms are input-specific.
We next freeze the model's attention patterns and denominators of any layer normalization terms, treating them as constant values; this entails detaching them from the graph (.detach () in Py- torch). See Figure for a depiction of this process. We also detach the transcoder feature activations themselves, so no gradients flow through them.
TThi is often done by penalizing z's Ta norm. However, note that some activation functions, namely Top-k and variants, inherently limit the number of active features, making this unnecessary.
16

---

Published as a conference paper at ICLR 2026
Logits
Logits
MLP
Transcoder
Error
Attention
Attention OV
Residual Stream
Attention QK
Residual
Local Replacement Model
MLP
Transcoder
Error
Attention
Attention OV
Allention QK
Inputs
Inputs
Figure 8: A 2-layer transformer LM, and its corresponding local replacement model. We replace model's MLPs with transcoders, as well as error terms unique to the given input. The attention pat- terns (from the QK matrix) have been frozen, detaching them from the computation graph. Despite this, the OV-matrix of each attention block is still attached. Thus, when we refer to e.g. the direct effect of feature of the layer-O transcoder the residual stream alone, or additionally through the OV matrix of the attention, linear transfor- mation. The direct effect of any given feature on any other feature (or any vocabulary logit) is thus linear. See Elhage et al. (2021) for more on QK/OV matrices and the residual stream.
vocabulary logit, this direct effect may pass through
17

---

Published as a conference paper at ICLR 2026
0740
4000
Day "er" 27
roaer Mataaas 10 setaoees meseet Taskue
Taathear' and whihh
seeer Tyme
the
dens singt
Figure 9: The interface used for circuit visualization annotation, from circuit-tracee
In so detaching these components, we remove all nonlinearities from our local replacement model: both the attention softmax nonlinearity and the normalization nonlinearities are gone. The activation of any given feature now linear in the activations of the nodes prior to This simplifies the process of computing the direct effect of one node another, and means that these direct effect values are exact; however, they will not account for features' impact on the model's attention patterns.
Attribution We can thus compute the direct effects of a source node on a target node as follows. We define an input vector for the target node: if the node is feature, this is its input vector (from Wene), and if the node is a logit, this is the corresponding unembedding vector, minus the mean unembedding vector. We inject this gradient the node's input location-either the MLP input for transcoder features, or the final residual stream for logit nodes; this injection can also be operational- ized as a dot product with the residual stream, followed by a .backwarddd..................... () call. Then, for each upstream node, its direct effect is the gradient at its output location, multiplied by its output vector: the input embedding or error vector for input and error nodes respectively, or the source feature's activation multiplied by its decoder vector, for feature nodes. With each call of bbackward (), we find weights for all edges into the target node; repeating this for all nodes attributes the whole attribution graph.
Methods We limit attribution to the top 7500 most influential feature nodes, remaining nodes are unlikely to be important, and attributing from many nodes leads to large graphs that fit poorly in memory. We determine which nodes most influential by intermittently computing each node's s influence using the procedure described in Ameisen et al. (2055). For logit nodes, we choose to attribute from the minimum required to capture 0.95 of the model's next-token probability, or the top 10 logit nodes, whichever is smaller (generally the former). Ultimately, the attribution process quick, from seconds for 3wen-3 (0.6B) a minute or two for Qwen-3 (14B).
For visualization purposes, it is often useful to prune graphs, removing low-influence nodes and edges. As done by Ameisen al. (2025), we do so by computing the total influence of each node and edge in the circuit. We then set a threshold for each, and take the minimum number of top nodes / edges that sum to that influence; we choose nodes whose influence sums to 80% of the total, and edges whose influence sumns to 98%. Our circuit-findigg finuing interface, provided by circuit-trace (Hanna et 2025), is shown in Figure 9
B SIMPLE PLANNING DATASET DETAILS
We construct three datasets for testing simple planning, the a/an, is/are, and el/la datasets. The alan dataset consists of 108 examples of professions (86 requiring a and 22 requiring an) and descriptions thereof. These were augmented with 350 concrete nouns (267 83 an) and descriptions thereof. All descriptions were generated by Claude 4 Sonnet, but filtered manually and rewritten if necessary, e.g. because they were too vague. The is/are dataset generated programmatically, and consists
18

---

Published as a conference paper at ICLR 2026
Mean Proportion Influence Through Selected Modes Model
Verb Recall Across Model Sizes (isiare)
0.2
Verbs "Ia"
Corrdd
Denrd sp
Vedbo "une"
Veda
Recall of is and are on the is/are dataset, by model. Models below 8B in size mostly fail to predict is, while larger models perform perfectly. All models can predict are. Right: The mean proportion influence flowing through planning nodes in the is/are dataset, by model, verb, and correctness; recall that the only incorrect examples are small models failing predict is. The most influence flows through the planning nodes in the is examples, where more planning nodes present. Still, more influence flows through these nodes in correct than incorrect is examples.
Figure 10: Left:
of (positive) differences between numbers ranging from and 9; the animals are sampled from a manually curated list of 10 animals. This yields 360 examples. The el/la dataset, much like a/an, consists of 411 concrete nouns (223 el 188 la) and Spanish- lannuih descriptions thereof. Again, all descriptions were generated by Claude Sonnet, but filtered manually and rewritten necessary.
Note that, in the case of the a/an dataset, one randomly-sampled in-context example from our dataset is prepended to each input to the model in order to encourage it output alam; other- wise, the model does not understand the task structure, and outputs other tokens. The full prompt is thus something like Someone who provides treatment for physical or mental condit i ons is therapist. Someone who heals sick pets is. This is fed directly to the model as the user input, and the model simply completes the input (rather than gener- ating separate assistant response). The el/la dataset is formatted in the same way.
Similarly, we prepend is/are examples with Repeat and finish the following sentence:, as we found that this increased performance over simply sampling next tokens without requesting the repetition. The full prompt is thus something like /no think Repeat this sentence and complete it. At first there were 2 cats. Then, 1 went away. Now, there. The /no.think prevents models from thinking before answering. During attri- bution, prefill the model's assistant response with <think> hatps////////ttti/e//e/ee (n At first there were 2 cats. Then, 1 went away. Now, there. We then attribute back from the top logits (which are always is and/or are).
C Is-Are RESULTS
Here, report results for experiments the is/are dataset, which largely mirror those performed on the a/an dataset. Figure I0 (left) shows that models behave similarly on the is/are to the is/are dataset: all models do well on the majority class are. Models below 8B size fail (0.6-1.7B) or perform poorly on the task when the correct answer is is; Qwen3-4B scores just below chance. Starting 8B, models score perfectly on is as well, just as with a/an.
We perform circuit analysis on is/are dataset as well, and find similar, but not identical trends com- pared the a/an case. Models again have features corresponding to planning features some of the time. However, / features (and 2 and 3 features a lesser extent) are more common than other num- bers' features. Whether this real phenomenon (models have special representations for lower numbers due to their frequency) transcoder-driven phenomenon (higher numbers also have cor- responding features, but transcoders miss these) is unclear. This may also be related to the fact that such features more important / necessary in the minority class case (I/is) than in the majority class case. In the case where such features do exist, we also observe that e.g. features activate features that induce the model to say is.
We perform the flow and intervention experiments done on the a/an dataset. These are complicated somewhat by the fact that there more planning nodes in the is case than in any of the cases,
19

---

Published as a conference paper at ICLR 2026
Effect of Zero Ablations on picorrect)
Effect of Direct Zero Abiations on p(correct)
0.00
Change Changelnplcomee Inpicoreet
-0.00
0.10
-0.8-
-0.15
-1.0
Effect Multiplying Interventions p(correct)
Direct Multiplying Interventions picorrect)
1.0
0.8
0.6
0.4
Qwwn340.68 eB
Qwee3-1.77
Qwerd-43
Qown3-88
Qwand-14B
Qwend-0.6
Qoown3-1.78
Qwwc3-4B
Qwen3-8B
Quan331 4B
Model
Model
itrandi
individual "is"
indisidual
Figure 11: Left: Change in p(correct verb) caused by zero and multiplying interventions on plan- ning features. The former generally harm performance, while the latter improve it. Both affect only is examples, which have the most planning nodes, and also are the only examples models answer incorrectly. Right: Change in p(correct verb) caused by direct zero and multiplying interventions on planning features. As before, these are relatively ineffective, though less so than in the a/an case.
Normal Intervection
Effect Zero Ablations picorrect)
Chopees promer
90
Verh Recall Across Model Sizes (el(a)
Effect of Multiolving Interventions en picerrect)
Chacge ploemen
0.8
0.
Article: 'el""
0wwn3-0.68
Qwen3-1.78
Qwwn3-45
Qwin3-85
Qwen3-148
Articl: Sa*
Owead-L.7
Qwei3-BB
32B
bonikal of wel
Intedal pirs
Figure 12: Left: Recall of el and la examples by Qwen-3 models. Unlike in prior examples, the majority class (el) is not perfectly captured by any model, though recall is generally high. Moreover, while performance on the minority class la improves with scale, recall is ultimately still middling. Right: Interventions performed with respect el/la planning features fail primarily due to a lack of said planning features.
and that models do not fail more influence flows through the planning nodes in correct than incorrect examples, as in the a/an dataset. Moreover, Figure 11 (right) shows that both zeroing and multiplicative interventions effective on is examples. This is likely because these have the most planning nodes; however, it may also be related to the fact that is is the minority class, and "needs" these features more.
cases. Still, in Figure 10 (right),
that in the is case,
D El-La RESULTS
Here, we report results for experiments on the is/are dataset, which are much less successful than those performed on the a/an or is/are datasets. When we behaviorally test the models this task, find (Figure I2, left) that performance worse than on the prior two tasks. The majority class el is not always correctly predicted, though performance stays steadily high as in other tasks. Moreover,
20

---

Published as a conference paper at ICLR 2026
while recall of the minority class does improve with model scale, it never exceeds 0.6, unlike other tasks, where reaches near 1.0.
We then perform the causal interventions, using planning nodes those that either in Spanish in English, as we observe that some examples have English nodes corresponding to the hypothetically planned word. However, find (Figure [2 right) that the interventions have little effect; this goes for both zero and multiplying interventions.
We believe that this is primarily driven by lack of planning features active these examples. In general, while we can find some planning features, Qwen-3 models simply have much fewer than they do the alan dataset, despite their formats being very similar. This may be because Qwen- 3 is not highly capable in language besides English and Chinese (which exhibits little syntactic agreement); further studies could examine more multilingually capable models.
E MODEL PERFORMANCE ON NON-PLANNING ASPECTS OF SIMPLE PLANNING TASKS
The fact that small models fail to plan on the simple alan and is/are planning tasks may raise the question: do small models fail because they cannot perform the tasks all? To show this is not the case, we generate models' planned tokens, both given the correct next token, and the incorrect next token. We then measure whether the output token in each matches expected planned token.
Parmee Word Accuracy by Mocel and Ariids careetress
Mean Prolossion Acoussy by Correct Artcle
Eoend aikcl
Ownr D104
Ownn 1.78
Bani R8
Oeivel
Dwnns Qwin Querel 24
Owieel 144
Figure 13: Left: Planned word accuracy, i.e. whether the model's predicted word matches the intended word, when given the correct incorrect article. Models above 4B in size are highly aceu- rate when given the correct article (>> 80%). and even smaller model achieve moderate accuracies. Given the wrong article, accuracies are lower, but still non-zero, indicating that models may have strong planning goal that prevails even when the word is at odds with the article. Right: Planned word accuracy given the wrong article, by correct article (a or an). Though accuracy is low, models succeed on both and examples, indicating that successes are not driven by one class.
Performance differs by task. On the a/an task (Figure 13 left), models have generally high accuracy (> 0.6) when given the correct next token, but lower accuracy when given the incorrect one; the
Number Prediction
Numoor PRCOctin ACC/OCC
Daenb-1.
Qwen1-ff
Dww
Qwen1-178
Coreect Vorb
Wrong Wc/b
Figure 14: Number accuracy, i.e. whether the model's predicted number of animals matches the intended number, when given the correct or incorrect verb are). Notably, small models produce the correct number regardless of whether they are given the correct or incorrect verb. In contrast, Qwen3- 14B and 32B have starkly reduced accuracy when given the wrong verb form.
21

---

Published as a conference paper at ICLR 2026
Random Interveation
Effect Zemo Ablations piournect)
Effect of Zoro Ablations on pleceneet)
-02
-04
Eifeett of Multtplynng Intervectons on picorrect)
Edfect of Multopyynng Interventions on picorretti
Carres prommcc
0.30
0.00
-02
100
Queend-0 Bits
Owend-]
Owwn3--11
Queen3-0ft
Qwend-1448
Owen3
Cwwea3 1.70
Qwena
Owend
Qwena
Figure 15: Effects of random features interventions the alan (left) and is/are (right) tasks. Nei- ther intervention has a large effect on either dataset, indicating that our interventions do not succeed by random chance.
highest scoring models in that scenario achieve an accuracy of 0.3-0.4. Baseline accuracy here in theory near as models can predict any word. This suggests that although models are not always planning for the precise word we intend (and indeed, there are cases where we find no nodes corresponding to the planned word) they often are. And in some cases, they plan so strongly for the intended word that they output it even when it conflicts with the article.
This trend is much stronger on the ishare task. Our results from the analogous experiment (Figure [4p show perfect accuracy for all models when the correct verb form is given. Given the incorrect article, smaller models are (near- pperfetyprfeetly accurate at predicting the correct number, but larger models (Qwen3-14B and 32B) perform much worse. This provides strong evidence that small models can perform the task (and that a lack of task abilities does not underlie their poor planning performance). However, the root of the behavior of large models is less clear. They appear to be more sensitive to (subject-verb) agreement, and thus produce outputs that agree with the number of the verb; in particular, given is as an incorrect next token, they tend to output 1, rather than number that agrees with the original animal quantities. In contrast, weak models do produce outputs like tike...no now there are dog remaining.
F RANDOM INTERVENTIONS
In order ensure that our have not succeeded by random chance, we perform all-effects ablations on random active features in our a/an and is/are datasets. For an example where we normally intervene on features, we sample another n features from the pool of all active last-position features, and record the effects of the intervention. The results (Figure 15) indicate that these random interventions are ineffective: neither the zero ablations nor the multiplying interventions work.
G ANIMAL PROBING AND INTERVENTION EXPERIMENTS
As done by Dong et al. (025),, we set up probing experiments as follows. We take 1000 stories from the validation set of Tinystories, and extract the first sentence. We then feed each first sentence the model in the following prompt: Here's s the first sentence story: (sentencel}.
Continue this story with one sentence that introduces a new animal character. We then generate (greedy sampling) a next sentence, and recorded the animal contained therein.
We then filter this data down to only the datapoints containing the top-4 most common animals; typically, this leaves 600 or more examples. We then split the data 60/20/20 into train, validation, and test, and collected (transformer layer output) activations from the last token of each prompt. We then train single-layer MLP probe to predict the animal that the model would predict, from these activations. We use a hidden dimension of 64 for our MLPs, as Dong et al. report that performance plateaus at d = 64. We run this analysis all Qwen3 models, well Llama-3-88 3-BB- Instructt
22

---

Published as a conference paper at ICLR 2026
used by Dong al.) and report results across hidden layers in. Figure 16 shows that our results Llama-3 (8B) are similar to the original findings, with high Fl scores (0.6-0.7) across all layers but the first. Probing results for other models varied; Qwen3-8B and 32B perform well (FI near
0.6), while other models exhibit middling performance (FI < 0.5).
Probing Test F1 Scores Across Layers
Teat Scrre (Wocr))
Ower3 0.c0
Omar3 L70
Owwel 140
Owers
Liaru 2 25 Insouct
Figure 16: Macro F1 scores when probing models' last-token representations for the animal that the model will output in the following sentence, by model layer. Fl scores are high for certain models Llama-3 (8B) and Qwen-3 (8B/32B)-brl notably lower for others.
We then verify the causal relevance of the features found by these probes. If the probe has found causally relevant feature at the end of the prompt that determines the animal that is output, altering that feature should alter the animal that is output. There are variety of interventions that could be used to verify the features found by the probe: Ravfogel et al. (2021) intervening by reflecting rep- resentations across probe decision boundaries, while Giulianelli al. (2018) compute the gradient of the probe's ' prediction (error) with respect to the model representations, and update the represen- tations based on this. We could also use less probe-specific interventions like difference in means (Marks & Tegmark 2024).
We opt for simpler intervention: we pair each prompt our dataset with random prompt that led to the production of a distinct animal. We then generate continuation to the first prompt, but patch the last-token activations of the second prompt onto the last token of the first prompt. We do so at all layers, effectively replacing all model activations at this position. This means that the next generated token guaranteed to be the next token of the second prompt; furthermore, attention back to the patched position will receive the patched values. Since we have patched all possible layers in which the relevant features could reside, this intervention should cause the model to produce the animal from the second prompt, if the probed features relevant. We perform this intervention across the same set of models as the previous experiment, with the exception of Qwen3-32B, as is supported by TransformerLens (Nanda Bloom 2022 the interpretability framework use.
Intervention Success Rates by Model
1.0
Esart Natch (dwtected Larget) Any Change (dotected source)
Success Rale
0.8
0.2
5.%%
13.5%
3.2%
23.5%
19%
Owent 1.78
Oenen 4B
Ownnt 148
Lamm sag Hostreet
Qoan3.0. 5
Model
Figure 17: Success rates of our patching intervention, where patch one prompt (p2)'s 's last token activations onto another (p1)'s I's last token during generation. We report both exact match (True if the output animal p2's 's animal) and any change (True if the output animal differs from pi original animal). In general, exact match is low, below 10%, while any change is higher, but under 30%.
Our results (Figure suggest that the features found are not highly causally relevant. In relatively few cases (< 10% for all models) do observe the output animal change that of the second prompt. In fact, in the majority of cases, the output animal does not change at all. This seems to be a violation of our Condition 1, that the found feature must have causal impact on the model's 's planned token.
23

---

Published as a conference paper at ICLR 2026
We note, however, that Llama-3 (8B), the only model from Dong et al. that test, does have higher intervention efficacy. Moreover, there are multiple features relevant for planning the animal to be produced, it would be necessary to find and intervene all of these to produce a strong effect.
Despite this, we find unlikely that planning takes place in this scenario. This is because the continuations corresponding each animal output are generie: they do not hint to animal that will be produced. Consider, for example, the prompt and continuation Mia and Dad were busy polishing their As they worked, small, curious fox darted into the garage, tail wagging playfully. The left context of fox imposes few constraints on the animal that is to follow; many animals can be small and curious. This hints that our Condition may not be fulfilled here either: the model does not actually have to plan / prepare a context that licenses the animal eventually output.
H COUPLET CIRCUIT DETAILS
H.1 COUPLET DATASET AND SAMPLES
Our couplets dataset was created by prompting Qwen-3 (32B) with the prompt "/no-think You are a creative poet. Produce ONLY the first line of a rhyming couplet about the topic: '[topic}'' Return single poetic 1ine and nothing e else.". We sampled 5 couplets per prompt, kept only the first line of each couplet, and manually filtered these for well-formedness. We used the following topics: love (romantic, familial, self-love); death and mortality; coming of age; war and confict;; nature and the environment; home and belonging; identity and self-discovery; joy and happiness; anxiety and fear; loneliness and alienation; nostalgia and memory; hope and despair; anger and frustration; family bonds (siblings, parent-child); friendship; betrayal and trust; first love; marriage and commitment; loss of loved ones; childhood memories; aging and growing older; seasons and cycles; historical moments; the passage of time; immortality and legacy; justice and injustice; freedom and oppression; cultural identity; social alienation; tradition vs. modernity; community and belonging: weather and climate; animals and wildlife: urban vs. rural life; travel and journeys; food and taste; colors, sounds, and textures; dreams and aspirations; perfection and imperfection; truth and deception; power and powerlessness; spirituality and faith; art and creativity.
We then sampled couplet completions from the Qwen-3 models using the prompt: "/no.think Write only the next line of this rhyming couplet: (first.line)".
For convenience, we provide 5 example first lines of couplets and completions below.
First line: Two hearts entwined where vows take root,
Qwen3-0.6B: And where the stars are born, they sing.
Qwen3-1.7B: two souls forever bound in love.
Qwen3-4B: A love that grows beyond all doubt.
Qwen3-8B: A love that time cannot undo.
Qwen3-14B: A love that grows in tender fruit.
Qwen3-32B: A love that time and distance can't 't undo.
First line: Whispers of dread in the quietest breath,
Qwen3-0.6B: shadowed path to the dark, where the night kind.
Qwen3-1.7B: And shadows that dance in the darkest keep.
Qwen3-4B: Shadows creep where the light dare reach.
Qwen3-8B: A shadowed path where no light dares tread.
Qwen3-14B: A shadow that creeps through the darkness of death.
Qwen3-32B: Shadows conspire as the night takes its death.
24

---

activations immediately precede a token containing a newline, e.g. ".\n".. We test that 1) activating these features prior the end of the second line causes models end the line prematurely, 2) Published as a conference paper at ICLR 2026
First line: Whispers of wind through the emerald trees,
Qwen3-0.6B: The moonlight dances the water's edge.
Qwen3-1.7B: Soft as dreams, they drift and sigh.
Qwen3-4B: Singing the hush of summer breeze.
Qwen3-8B: Sing lullaby to the sleeping breeze.
Qwen3-14B: Softly they sing as the daylight decreases.
Qwen3-32B: Singing soft lullabies to the leaves.
First line: Whispers of winter kiss the thawing spring,
Qwen3-0.6B: The frost has passed, and the snow has gone,
Qwen3-1.7B: and silence bows to the bloom of the green.
Qwen3-4B: Blossoms rise in quiet, fragile song.
Qwen3-8B: Beneath the snow, the seeds begin to sing.
Qwen3-14B: Blossoms awaken where the frost once clung.
Qwen3-32B: Melting dreams in tender, fleeting sting.
First line: A promise made in shadows, torn by dawn's deceitful light,
Qwen3-0.6B: shadowed vow, once kept, now lost,
Qwen3-1.7B: Yet still, the heart remains true to what it
Qwen3-4B: A vow now lost in the glare of day.
Qwen3-8B: A vow once whispered fades into the night.
Qwen3-14B: A heart once whole now fractured, lost in endless, silent night.
Qwen3-32B: A truth once held so sacred, now lies shattered in the fight.
H.2 CIRCUIT VERIFICATION
Here, we discuss the experiments performed verify our eircuit, which we claim acts via end-of- line, sear-end-of-llne and rhyming features. We test these features as follows:
End-of-line (EOL) features We define EOL features as those where out of the feature's 's top 10
deactivating these features after the model has completed the second line with a rhyme causes models continue the line, instead, and 3) deactivating these features at the end of the first line causes the model to fail to rhyme, as EOL features regulate its attention to rhyming features.
For each of these experiments, identify EOL features on each example as those that are active on the last word of the couplet's first line. We perform each experiment only on those couplets for which we have performed attribution. For experiment 1). we provide the model with the couplet's first line and the first 2 tokens of the second. We set the EOL features to 5 times their original values the final position of this input, and any generated positions. We then allow the model to generate, using greedy sampling. We record the length of generation (in tokens) before the model outputs newline, and compare to the length of the original line.
For experiment 2), provide the model with each entire, completed couplet (stripping any punc- tuation at the end), and set all EOL features to -5 times their original values. We do this at the final position of this input, and any generated positions. We the model to generate, using greedy sampling, and record the length of the model' new generation, compared to that of the original.
For experiment 3), provide the model with the first line of each couplet, and let it generate the next couplet (with greedy sampling), while setting all EOL features to -5 times their original values at the end of the first line. We then record rhyming accuracy.
Figure I8A shows the results of experiments 1) and 2). Upweighting EOL features indeed causes models to end the second line early, resulting in large negative difference in line length. In contrast,
25

---

Published as a conference paper at ICLR 2026
Interventions length
Upweighting on Longth
Change in bergth (icbares)
Cange Inte langth (tokens
2.0
Opweightnng EDL
B
Domnenigtiing EDL
2.5
-1
Effects on rhyming accuracy
Effectiveness of Changing Rhyming Features
Downawigtkigg EDI end of first lin)
Vinel & Consonent
Abyming AccurSCy
Ablating attention
Use
0.3
0.3
D
0.2
Whyming
0.1
0.0
Owen3-0.08
Qwer3-1.77
Qwee3-48
Owen1-86
Qwen30.66
Qwerd-1.78
Qwan2-46
Qwwn3-86
Qwwrd-144
Upweighting them in the sec- ond line causes the line to end early, while downweighting them causes continue for longer than normal. B: Effect of upweighting near-end of line (NEOL) features in the second line. Upweighting these causes the model emit a rhyme over words earlier than normal. C: Effects of downweight- ing EOL features at the end of the first line, downweighting NEOL features in the second line, or ablating rhyming-relevan attention heads' patterns. The first two interventions drastically decrease the model's propensity produce rhyme, indicating that they help enable rhyming. The last is less effective, but still reduces accuracy far below the original, 100% accuracy. D: Rhyming accuracy when ablating original rhyming features, and upweighting those from another rhyme group. Larger models switch to the new rhyme group with 40% accuracy-lower than their original rhyming ac- curacy, but still relatively high.
Figure 18: A:
Effects of intervening on end-of-line (EOL) features.
26

---

Published as a conference paper at ICLR 2026
downweighting said features prevents models from ever finishing line, resulting in very long lines, relative to the original. Figure I8C shows the results of experiment 3). The rhyming accuracy for larger models (larger than 0.6B) is low, below 0.2; this is despite the fact that we perform the inter- vention on examples for which we have computed circuits, which are examples on which models succeed. This means that downweighting EOL. features at the end of the first line seriously hindered performance. Moreover, qualitative inspection of the model's outputs showed no harm to the overall fluency of the completions, suggesting that this was not due to general harm to the model's ss abilities. This suggests that the EOL features do play an important role in regulating rhyming abilities, likely through the keys of attention heads, which tell them where to attend to.
Near-end-of-line (NEOL) features We define NEOL features those where 7 out of the feature's 's top 10 activations occur 2-4 tokens before a token containing a newline, e.g. ". "-\n". We test that 1) activating these features at the beginning of the line causes models emit rhyme early, and that 2) deactivating them stops models from rhyming.
For each of these experiments, we identify NEOL features on each example those that are active on the second to last word of the couplet's second line, i.e. on the token before the rhyming word. We perform each experiment only on those couplets for which we have performed attribution. For experiment 1), we provide the model with the couplet's first line and the first 3 tokens of the second. We set the NEOL features to times their original values at the final position of this input, and any generated positions. We then allow the model to generate, using greedy sampling. We record the length of generation (in tokens) before the model outputs a rhyming word, and compare it to the length of the original line.
For experiment 2), we provide the model with each couplet's first line, and set all NEOL features to -5 times their original values. We do this at the final position of this input, and any future positions. We sample a second line from the model using greedy sampling, and record rhyming accuracy.
Figure 18B shows the results of experiment 1). Upweighting NEOL features causes models to rhyme carly- over tokens carly, for models above 0.6B. This suggests that the NEOL feature is causally responsible for models' output of rhyming token. We note that this intervention qualita- tively frequently caused models to rhyme not just early, but also rhyme often: models sometimes output multiple rhyming words (e.g. Beneath the gray lay stray), as if the need to rhyme (like the upweighting of the NEOL feature) was ongoing.
Figure T8C shows the results of experiment 2). The rhyming accuracy for larger models (larger than 0.6B) is low, below 0.2, just like when we downweighted EOL; indeed. Once more, this is despite the fact that we perform the intervention on examples for which we have created circuits, which are examples on which models succeed. Downweighting NEOL features in the second line thus seriously harmed performance. Similar to before, qualitative inspection of the model's outputs showed no harm to the overall fluency of the completions. Since these features act at the position where rhyming occurs, we hypothesize that they affect the queries of attention heads that would otherwise bring features over rhyming information, allowing models to then predict a rhyming word.
We thus also test this attention head theory. We record each model's attention during a normal forward pass, and when the NEOL features are strongly (-6x) downweighted. We then find the top-5 heads whose attention back to the end of the first line is reduced most by this ablation, averaged across couplets. We hypothesize that these heads play causal role in rhyming abilities. Thus, perform couplet generation as in the prior experiment, but transfer all of these 5 heads' attention back to the end of the first line, to the BOS token; observe that this is what happens upon ablation, and such tokens are generally considered to be attention sinks. We then record rhyming accuracy.
Figure 18C shows the results of this experiment well. This ablation is less effective than directly intervening on NEOL features directly; rhyming accuracies are 20-30% higher, though far below the 100% accuracy models achieved on the sentences for which we computed circuits. It is also significantly more targeted: we only alter 2 attention probabilities in 5 heads, rather than targeting many features.
Rhyming features As discussed in the main text, we find rhyming features using heuristic. We look for features that activate on short tokens (all top-activating tokens are <5 characters) that are distinct (no more than 5 occurrences of the same token), and where 7 of the 10 top activating features
27

---

Published as a conference paper at ICLR 2026
Commonn Prefix Length as Percert of Original Length by Model
Stvered Gineeation
Normal (Tmmpa
0.6-
62-
6.3
3.75
Geard
Figure 19: Length of the shared prefix between the original generation, generations with temperature 1.0, and steered generations, both greedily sampled and with temperature 1.0. Error bars show SE.
either start with the same vowel, or end with the same consonant. Manual inspection suggested that this yields relatively high-precision but only moderate-recall recovery of these features.
We test that these features control the output rhyme, by deactivating each example's 's original rhyming features at the end of the first line of the couplet, and upweighting the rhyming features of example with a different rhyme. The resulting line should rhyme with the new example, not the original.
As in the main text, our results (Figure I8D) suggest that these features are indeed responsible for choosing the rhyme. Although accuracy is lower than on the original rhymes, it is moderate, and near that of the models overall on rhyming couplets.
H.3 INTERVENING ON RUYMING FEATURES CHANGES INTERMEDIATE TOKENS
We can test whether the intermediate context generated by the model changes at all upon rhyming feature intervention. To do this, we take the original generation of the model a couplet, and its generation when its rhyming tokens are steered as in Section 5.3 We then record the length (in tokens) of the longest prefix shared between the original and steered generation. As baselines, we also compute the overlap between the original, greedy generation, and generations (both steered and unsteered) that we sample with temperature 1.0. the rhyming features are genuinely causing the model to plan for future tokens, we should expect them to cause the model's intermediate tokens to change, more than temperature-based sampling would.
Our results (Figure T9p indicate that steering affects the intermediate context between the first line and the rhyming token output. For smaller models (0.6B and 1.7B), this intervention do no more than simple sampling does. But for larger models (8B and 14B), the effect of this intervention upon generations exceeds that normal sampling-eve even when combining the intervention with greedy decoding. Thus, the intervention alters both the intermediate and final tokens that models output.
I POTENTIAL LOCAL PLANNING FEATURES
To find local planning features in couplet circuits, we search for features in our cireuit that upweight the rhyming word that is eventually output, or have one of their 10p-10 activations on that word. We search at the position before that word is output; that is, we look for say X features that are causally relevant even before the model outputs X. For circuit in which this could be occurring, Figure 20
For each model, and each of its top- 10 words by number of say X features, we steer on those say X features, setting their activations to 3, 5, or times their original values. We do so on 5-15 token fragments of sentences from the TinyStories dataset (Eldan & Li 2023) neutral context where models are not likely strongly planning. We then record whether each model eventually output X, and qualitatively inspect the outputs.
28

---

Published as a conference paper at ICLR 2026
Output
Det
"the"
and 05 Line
1iae
Ehynes wi "Lcht"
aod
Beat of 1ine
Beaz aod
of line
of 1ine
1iit beii cott
thep
baeae are yonk Uhe bast wr
starlit vhilpersy my hopes take
Light
Through dreans that
5m
the
n1ghh
Figure 20:
A feature circuit for couplet ending in in the night. Unlike prior circuits, this circuit involves a specific night feature that drives the production of the phrase in the night.
Qualitatively, we observe that steering can lead to the sorts of sensible generations we observed in the poetry setting. Steering on the say "night" feature leads to generations such as One day, a girl named Mia went for a walk. She saw a cat and started to follow it to turn into One day. a girl named Mia went for walk. She saw cat in the night; notably, in the appears to specifically license night. Similarly, steering on the say "dream" feature often leads to outputs like recurring dream or American dream, i.e., contexts that specific to dream.
However, this is not always the case. Steering too hard can cause the model to output the target word even in infelicitous contexts, or to only output the word; some say X features seldom produce the target word when steering. How can we measure whether the steering not only (1) produced the word X, but also (2) maintained a coherent sentence (up to the point where the word X was output) and (3) truly adapted the context to license X? We can measure (1) programmatically, but (2) and (3) are harder. For coherence, we query Claude Sonnet 4.1 about the coherence of each steered generation []: to verify that Claude is good judge of coherence we annotate 100 examples coherence, and find high agreement (80% where most disagreements come from Claude missing incoherence).
To estimate models' abilities at (3), we filter examples to include only those where (1) and (2) are fulfilled; we also filter out any examples where the original and steered generation are identical up until the word X is output, as adaptation has surely not occurred in such cases. Then, we estimate how many of these examples fulfill (3) by manually annotating 100 examples per model for whether they contain context adaptation that could indicate planning. For example, we look for phrases like her when steered towards own, had recurring dream when steered towards dream, when the original generation did not contain similar phrases. We also mark as incorrect examples that are ungrammatical / incoherent, but were missed by Claude in the first round of filtering, model that is successfully adapting its context for a planned token should not produce such outputs. We then plot these metrics.
Our results (Figure 21 indicate that larger models are more successful at steering towards X and more likely to adapt their context match X, though they are more coherent than smaller models. However, few examples actually fulfill all of these conditions: even in Qwen-3 (14B), only 10% of examples do so. So, while we believe that these features may be part of generalized phenomenon whereby models plan for words by boosting n n-grams that end in those words, our uncertainty is rather high. Our current hypotheses still rely on qualitative evidence, and more study is needed to understand the precise mechanisms by which these features work, and more consistently elicit and measure planning behavior from them.
Claude 4.1 Sonnet Prompt. Note that we only employ the coherence judgments, as the word criterion can be checked programmatically, and we found Claude's 's adapts-context responses unreliable.
Listing 1:
f"""I need you analyze a text generation where model was steered to include a specific word.
<input_prompt> (input_text)
29

---

Published as a conference paper at ICLR 2026
Model Steering Metrics by Size
0.92
Coheret
Condiforrs
C70
077
6.79
4.-
Frequency
0.5
0:35
0.5
6-14
G22
Owen3-0.68 Qwen3-1.78
Qwen3-4B
Qwen3-88
Qwen3-14B
Model Size
Figure 21: Steering metrics by model size, averaged across steering strength. Overall, as model size increases, the ability to elicit say X by steering increases. The model's tendency to adapt their context also appears increase with size. Coherence mixed, and appears unrelated model size. Ultimately, few examples fulfill all three conditions-109 -10% in the best case.
</input_prompt>
<basel ibaselinn_geneeatiion
(baseline (naselinn__eneeatioon
</basel innasseiie_geeeratiion
<steered generation>
Igeneration| </steered__eneration>>
<stcered_word
(steered_word) </steered_word>
Please analyze this generation and answer the follewing questions:
1. Contaiin steered word**: Dees the steered generation contain the exact word ([steered_xord)"g (look for exact match, case- insensitive)
2. CChherencer:: the steered generation coherent up the target word. Is naturai, or does it show signs breakdown such as: Excessive repetition of words or phrases Unnatural/nonsensice sentences
Abrupt topic changes that don't t make sense Grammatical breakdown
IMPORTANT: Focus on ccherence up to the point where the steered word appears (or would appear). Ignore any incoherence that happens after the steered word.
3. **Context adaptation**: Compare the steered generation to the baseline generation, starting from the prout__poppt.... Did the model modify the words inmediately preceding where the steered word appears (or would appear) compared to the baseline? Look for changes like:
Adding/changing articles ("a", "the", "an")
Adding/changinn prepositions ("in", "on", "at", "with") Changing vers forms tenses
Adding adjectives medifiers
Any other modifications that help the steered
fit better
IMP For context adaptatien, you must compare the steered generation against the baselime ine to see if modi fications were made specifically accommodate the steered word. Count the number of
30

---

Published as a conference paper at ICLR 2026
words that added changed immediately before where the steered word appears (or would appear).
When counting ing adaptation words:
Count new articles, adjectives, adverbs, prepositions added before the steered word
Count changed words that help the steered word fit better
Do NOT count the steered word itself
Focus on the immediate context (usually 1-3 werds before the steered word)
if the steered word not present, there no adaptation
Example 1:
input_text: text: "Once upon time, there was a baseline: "Once upon time, there young girl named Lily who lived small village" stecred "day"
stcored "Once upen time, there was day when all the day of day day day day"
contains steered word: yes coherence: yes (the generation is coherent up to "day") contain adaptation: no (day is inserted instead of young)
Example 2:
input text: "Once upon time, there was a little boy named" baseline: "Once upon a time, there was a little boy named Timmy who Loved to play with numbers. One" steered_word; "dream"
steered__eneration. ""Once upen time, there was little boy named He had recurring dream that he had""
contains steered word: yes
coherence: yes (the generatien is coherent up to "day")
contain adaptation: yes (sets up dream with "he had a recurring")
Please previde your analysis in this exact format:
<analysis>
explanation any coherence issues found </ccherence exp.aanaiiiinnnnnncc......
explanation of any context adaptations
made compared to the httpa//ee///atttttooooeeplttttt hooottaui...wddddeee/////oootttt
<is bls__ccccrnntcyeeee//////scoooot
<adapted <adapted_contexte httppp/ee.cccaaataeee//eeaaaaaap
aadatattionwwor_coottttnnnii wttp:wwd_outttnuniiiiii (0 if no adaptation, otherwise count of adapted words) </a@aptatioo__wrrd.ccoo
</analysig>"'
COMPARISON OF INSTRUCTION-TUNDE AND BASE QWEN-3 MODELS ON A-An AND Is-Are TASKS
Throughout this paper, we have analyzed the behavior of instruction-tuned Qwen-3 models. How- ever, is unclear how the performance and mechanisms of said instruction-tume models differs from that of base models. We investigate this by running the Qwen-3 Base models, from 0.6B to 14B, on the a/an and is/are. For the latter task, we reformulate the base model's input to exclude the instructions present in the original task (Repeat this sentence and complete it.), though find this to have little effect on the results.
Our results indicate that the base models outperform the instruction-tundd models the alan task slightly (Figure 22 left): recall of the majority-article a is similar between the two, while the base models have consistently higher recall of 1.7B parameters and above. By contrast, the
31

---

Published as a conference paper at ICLR 2026
is/are task (Figure 22 right), the instruct models significantly outperform the base models: while both achieve high recall on are, only the instruct models eventually achieve high recall on is. This is despite the fact that base models, too, numerate enough to complete the task: even when given the wrong verb, they output the correct number, much as the instruct models did.
These results indicate that neither model variant- or instruction-tuned- striiily outperforms the other. Rather, differences seem driven more by the data distribution: the a/an task is more akin pre-training data, while the is/are task involves math, focus of instruction-tuniin.
K PLANNING FOR MEASURE WORDS IN CHINESE
The most successful examples of backward planning are English agreement tasks, which raises the question: can models perform backward planning in other languages or contexts? The relative weak performance of Qwen-3 (14B) and smaller models makes adding complex tasks challenging, but we can test agreement abilities in Chinese, in which Qwen models perform well.
We focus on the phenomenon of measure-wort agreement in Chinese. Chinese uses measure words, which function analogously to e.g. the word loaf (of bread) in English: rather than saying one bread, one says one loaf of bread. Similarly, in Chinese, one person becomes -A A (one [person-unit] person), while three pigs becomes = 3e (three [livestock-unit] pigs). Notably, different nouns require different measure words, but the measure word precedes the noun, much like a/an.
We can thus ask: given context that indicates that given noun will appear, do models engage in forward planning for that noun? And do they also engage in backward planning to determine which counter word should be used? Owing to own limited knowledge of Chinese, we construct smaller set of 10 examples, which elicit distinct measure words:
1. 0
He saw four kmights upon 4... [measure- word] horses.
2. 2
The apartment had 3... [measure-word] bedrooms. Af He saw four farmers with four... [measure-
3. JL J4
4. tu B L [n fir im Y
He saw four bartenders carrying E
four... [measure-word word] cocktails.
5. / T f
R.. She heard singing from in
the cage. Inside she saw a... word bird.
6. + ... # The theater troupe had performed a... [measure-word]
play.
7. * # 1000... Ho In the large forest, there grew 1000... [measure-word]
trees.
8. * m..
In the pond, there swam four ... pour...[meaaan wordd fish.
9. *
m
The artist created twelve... [measure-word: paintings. the night sky, they saw 1000... [measure-
10.
के TI 1000...
word] stars.
Article Recall: Base vs Instruction Tuned (a/an)
Verb Recall: Base vs Instruction (isaree)
1.2
6.3
0.8
0.5
(tndrrut)) "aro" (buss)
"azo" linatruct)
finstruct
02.
Shook
Indtrumt
Qwin3-0.B
517555.75
Qwon3-48 4B
Qwen3-85
Qwas5-14B
Qwen3-0.5B
QwinS-1.78
Qwen2 4B
Qwwn3 58
Qwen3-TdB
Figure 22: Performance of base (dashed line) and instruction- (solid line) models on the a/an (left) and is/are (right) tasks.
32

---

Published as a conference paper at ICLR 2026
Effecta Chinaso Messure Word Intervestions
Effects Word (Gevand Truth)
16.
TE3
02-
S44
241
41.
11.5
16.3
23
Owaz3-d.05
Oweeb-n.cn
0ym16-1.77 1.77
Oweed-t -h
Owwed 4B
Modil
Modd
Sineet Insevvstion Mtach Introvt Inteevention Marel
Oriiinal OuRp ACCURRY
Dranct Esterweniie Matic
Inliver Inooeeation Mavc
Figure 23:
The effectiveness of replacement interventions, measured by the percent of cases where the intervention induced to model to output the expected the measure word. Left: the expected measure word is considered to be the measure word that the model output on the target example. Right: the expected measure word is considered to be the ground truth measure word from the list above-even if the model failed to output it originally. Models' accuracies with respect to the original ground truth are shown in green.
We select these examples because preliminary testing indicated that at least larger models produce the expected counter words when given them; they do not all have correct unlike in the alan and is/are cases, we cannot treat these as a behavioral evaluation of backward planning ability. However, can still analyze models' circuits.
per Thus,
Thus, we compute attribution graphs for these examples, attributing back from the predicted (mea- sure) word. As in the alan example, we find features that correspond to our intended word. For example, planning features that fire on and upweight bird activate on example 5, while painting features activate on example 9. However, analogues to say a/an features are often absent from these circuits: while "say [any measure word]" features are common, "say [specific measure word]" fea- tures are not. That is, while there are "say head [of cattle]" features that correspond to the livestock measure word *,, not all measure words have these. How do these features contribute to model outputs?
We test this by running interventions, as in the couplets section of the main text. Specifically, we run the model on a given target example, but downweight its planning features, while upweighting features from another source example; nations. We aim to change the model's output measure word to the measure word the target example. In addition to performing unconstrained interventions, terventions. These test for the presence of "say (specific measure word]" features that might be hidden in transcoder errors: if direct-effect interventions fail, while full-effects experiments ceed, we know that intervening features exist in downstream MLPs, even if the transcoders miss them.
these replacements for all source target combi-
also perform direct-effects in-
Our results indicate that this intervention is relatively successful. Figure 23 (left) shows the pro- portion of examples in which performing the intervention successfully changed the word into the expected measure word, i.e. the word that the model originally output on the target example. Perfor- mance generally increases with scale, increasing from 20% match with the expected measure word to 52.2%. By contrast, direct-effects interventions produce effects around half the size. When we use the ground-truth measure word as our expected measure word (Figure 23 right), smaller models perform worse, direct effects interventions grow less effective and the scaling trend becomes more obvious.
We conclude that the planning nodes do also control the measure word and noun output. However. we note that the direct-effects interventions have moderate effects, varying by measure word. We hypothesize that this is because some Chinese measure words have semantics that closely align with their corresponding noun; for such examples, the planning features, along with "say [any measure word]" features suffices upweight the measure word.
33

---

Published as a conference paper at ICLR 2026
Proportion of Steored Examples Containing Target Words
0.4
0.2
Quwad .B
(wea--4.75
Quea2-148
Figure 24: Results steering on planning features from the a/an task. For each model, we plot proportion of examples where the model produces the planned word (blue) and the proportion where the model produces a or an, followed by the planned word. Models often produce the planned word when steered, but less often produce the article in front of it.
Unipaay Boceed
Dpeeci 1.73
Swatt
Figure 25: Planning feature counts examples, by model and correctness. The smallest models have few planning features overall, while the largest has relatively many in both correct and incorrect cases. In the 4B and 8B parameter models (with nascent circuits), there is large gap between the number of planning features active correct and incorrect cases..
L STEERING ON A-An PLANNING FEATURES
In this section, we provide additional evidence that planning features, such as those in the a/an experiments, do not directly cause the upweighting of said indefinite articles. To do so, take the same approach in Section [5.4] to identify planning features to steer on. We then use them to steer models on the TinyStories dataset. We then check for the presence of the planned word, and for the presence of a/an before it. Our results (Figure 24) indicate that while steering is effective at eliciting the desired word, this does not entail producing a/an.
M WHAT ARE NASCENT CIRCUITS?
We find that Qwen-3 4B and 8B have nascent circuits for a/an and is/are, leading them achieve middling recall of the minority classes those tasks. But what does it mean for circuits to be nascent, or fully formed? There are few points of potential breakage in the circuit:
Planning Features The models could lack planning features, or fail to activate them suffi- ciently.
Downstream Connections The models could lack connections from planning features to downstream features upweighting a/an or is/are
We investigate the first hypothesis in the a/an task, by counting the number of planning features active on an examples, distinguishing cases where model outputs were correct and incorrect. Plot- ting these (Figure 25) shows markedly different behaviors across model scale. The 0.6B and 1.7B parameter models have very few active planning features overall. The 4B and 8B parameter mod- els have many active planning features in the correct case, but notably fewer in the incorrect case.
34

---

Published as a conference paper at ICLR 2026
Finally, 3wen-3 (14B) has many planning features active in both cases, though slightly fewer active in incorrect cases.
This suggests that the failure of these nascent circuits is due at least in part to the models' failure to adequately activate planning features.
35


Reviews:
[{"review_id": "2SdhztZS8v", "reviewer": "Reviewer_shSX", "summary": "The paper's central hypothesis is that latent planning is an emergent capability that increases with model scale. It seeks to answer (1) whether LLMs engage in a mechanistically verifiable form of latent planning, (2) how this ability can be defined and measured, and (3) how this capability scales with model size.\n\nThe methodology first establishes a strict, two-condition causal definition of latent planning, distinguishing it from prior observational or probing-based work. For an LLM to be \"latent planning,\" it must possess an internal representation of a future goal (a token or concept $t$) that: \n1. Forward Planning: Causes the model to eventually generate $t$ (Condition 1). \n2. Backward Planning: Causes the model to generate a preceding context that licenses $t$ (Condition 2).\n\nTo identify these causal mechanisms, the authors employ Transcoder Feature Circuits, a mechanistic interpretability technique. This method decomposes a model's dense MLP activations into sparse, monosemantic (interpretable) features and identifies the causal sub-graph (the \"circuit\") that explains a specific behavior.1 The study is conducted on the Qwen-3 family of open-source models, ranging from 0.6B to 14B parameters.", "questions": "On local planning, They say X features are described as \"sensitive\" and found in a \"small minority.\" Is this evidence of a real, generalizable mechanism, or an artifact of steering on specific, polysemantic features that happen to fire on common n-grams? How could this mechanism be tested more robustly?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper's greatest strength is its insistence on a rigorous, two-condition causal definition of latent planning. This elevates the study from a correlational observation to a test of a mechanistic hypothesis. This strength is powerfully underscored by the refutation of probing-based methods in Appendix G, which demonstrates that high probing accuracy can be causally irrelevant.\n2. The quality of the core experiment is extremely high. The a/an and is/are tasks serve as an elegant \"minimal pair\" testbed for planning. The causal interventions (ablation and boosting) in Section 4.4 provide \"smoking gun\" evidence for the discovered planning circuit. The analysis in Appendix E, which surgically separates task-solving ability from planning ability, is a brilliant and crucial piece of analysis that solidifies the paper's claims.", "weaknesses": "1. The complete failure of the methodology on the el/la task (Appendix D) is a significant weakness. The authors' explanation—that \"Qwen-3 is not highly capable in language besides English and Chinese\" —is an ad hoc hypothesis. This failure could alternatively imply that the \"planning\" mechanism found is not a general-purpose planning module at all, but a highly specific and brittle circuit for English grammatical agreement. This possibility severely undercuts the generality of the paper's claims.\n2. The paper repeatedly claims that smaller models (4B-8B) have \"nascent planning mechanisms\"  but \"fail\" the task. It is unclear what this means mechanistically. Does the circuit exist, but is weak? Are some features missing? Does the model have the 'accountant' feature but lacks the causal connection to 'an' ? This \"nascent\" concept is central to the \"emergence\" narrative but remains poorly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyvgITxUEc", "reviewer": "Reviewer_nfGc", "summary": "This paper examines whether large language models (LLMs) perform latent planning, that is, internally representing and reasoning toward future tokens without explicit plans. Using Qwen-3 models (0.6B–14B), the authors find that forward planning emerges with scale while backward planning remains limited, offering a causal framework and mechanistic evidence via transcoder feature circuits.", "questions": "- Could the observed causal effects arise from correlated features instead of true planning representations?\n\n- How would the proposed framework generalize to complex goal-directed or multi-step reasoning tasks?\n\n- What is the relative contribution of instruction-tuning versus model scale in the emergence of latent planning?\n\n- How might this causal framework be applied in AI safety monitoring to detect latent scheming or hidden goal formation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a clear and causally grounded definition of latent planning that distinguishes genuine planning from mere correlational predictability.\n\n- The study offers comprehensive scaling insights, showing how planning abilities gradually emerge and strengthen as model size increases.\n\n- It links mechanistic interpretability to AI safety, highlighting how latent planning could relate to hidden goal pursuit or “scheming”, thereby extending the work’s broader relevance.", "weaknesses": "- The chosen tasks, such as a/an prediction and rhyming couplets, are synthetic and narrowly scoped, limiting the conclusions’ applicability to real-world reasoning or planning.\n\n- The evidence for backward planning is weak and inconclusive, raising doubts about whether full planning mechanisms have truly been demonstrated.\n\n- The study lacks cross-model comparison, as it focuses only on the Qwen-3 family, making it unclear whether similar phenomena occur in other model architectures.\n\n- Some of the causal claims may be overstated, since interventions could affect correlated linguistic or contextual features rather than genuine planning representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxvPTbg9S8", "reviewer": "Reviewer_JcTR", "summary": "Investigates whether LLMs engage in latent (not explicitly generated) planning and shows that planning ability grows with model scale (in the Qwen-3 family).\n\nContributions\n\n* Provides a causal definition of latent planning  -- planning is an internal representation that (i) causes the model to produce a specific future token for forward planning and (ii) causes generation of a context that makes that token more likely. This improves on purely observational/probing definitions.\n* Provides simple agreement tasks as planning probes: On a/an, is/are, and el/la tasks, larger Qwen-3 models reliably plan ahead for the content word and use that to choose the right function word. Smaller models show nascent but incomplete mechanisms.\n* Mechanistic evidence via transcoder feature circuits that identify “planning features” that represent the future word and show, through interventions, that ablating them hurts performance and boosting them helps, indicating genuine causal relevance.\n* Causal-mech interpretability recipe for monitoring such emergence of forward planning and backward planning ability in  open models.", "questions": "* Can you add a quantitative measure of contextual dependency?\n*  Do you believe these results generalize to multi-step cognitive planning or compositional reasoning? Could this be shown?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Originality: Introduces a causal definition of latent planning that distinguishes between forward (goal-directed token production) and backward (context-shaping) planning. This causal approach is  novel in how it rephrames what “planning” means for decoder models and correcting an overextension in prior work that equated decodability with intent. The integration of transcoder feature circuits with causal interventions is also a novel methodological synthesis, enabling verifiable mechanistic evidence rather than speculative probing.\n* Quality: The experiments are rigorous and well controlled. The progression from simple grammatical-agreement tasks to rhyming and prose-steering scenarios is also structured well logically and empirically. The use of quantitative flow analysis within feature circuits adds an added layer of interpretability\n* Clarity: Definitions are explicit, figures are clear and interpretable. The argument flows naturally from conceptual motivation to empirical validation. \n* Significance: The results establish that latent planning mechanisms emerge with scale and that forward planning precedes backward planning—an interpretable scaling law that contributes to our understanding of model cognition.", "weaknesses": "* In terms of planning, the paper over-indexes on short-range linguistic dependencies. Not clear if this scales to true multi-step reasoning or action planning.\n* Limited to Qwen-3 series, which hurts generalization\n* Experiments provide only limited support for backward planning, and the analysis of whether the generated context “licenses” the planned token is overly qualitative. :", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)