What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering
Authors: Jim Maar
Denis Paperno
Callum Stuart McDougall
Neel Nanda
Venue: ICLR 2026 Poster (2026)
Keywords: implicit planning
LLM
rhyming
metrics
Primary area: interpretability and explainable AI
TL;DR: quantitative measures of implicit planning in LLMs, with case studies in rhymed poetry generation and question answering

Abstract: Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation to a predicted future token, such as a likely rhyming word, as supported by a prior qualitative study of Claude 3.5 Haiku using a cross-layer transcoder. 
We propose much simpler techniques for assessing implicit planning in language models. With case studies on rhyme poetry generation and question answering, we demonstrate that our methodology easily scales to many models.
Across models, we find that the generated rhyme (e.g. "-ight") or answer to a question ("whale")
can be manipulated by steering at the end of the preceding line with a vector, affecting the generation of intermediate tokens leading up to the rhyme or answer word.
We show that implicit planning is a universal mechanism, present in smaller models than previously thought, starting from 1B parameters. 
Our methodology offers a widely applicable direct way to study implicit planning abilities of LLMs.
More broadly, understanding planning abilities of language models can inform decisions in AI safety and control.

Published as a conference paper at ICLR 2026
WHAT'S THE PLAN?
METRICS FOR IMPLICIT PLANNING IN LLMs AND THEIR APPLICATION TO RHYME GENERATION AND QUESTION ANSWERING
Jim Maar HPI / University of Potsdam
Denis Paperno Utrecht University Netherlands
Callum McDougall, Neel Nanda Google DeepMind, London, UK
Germany
jim.maa man.marrhhp al alunni... de
d. d.papernouu..tt
ABSTRACT
Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation a predicted future token, such as likely rhyming word, as supported by a prior qualitative study of Claude 3.5 Haiku using a cross-layer transcoder. We propose much simpler techniques for assessing implicit planning in language models. With case studies on rhyme poetry generation and question answering, we demonstrate that methodology easily scales to many models. Across models, we find that the generated rhyme (e.g. "-ight") or answer to a question ("whale") can be manipulated by steering the end of the preceding line with vector, affecting the generation of intermediate tokens leading up to the rhyme or answer word. We show that implicit planning is universal mechanism, present in smaller models than previously thought, starting from 1B parameters. Our methodology offers a widely applicable direct way study implicit planning abilities of LLMMs. More broadly, understanding planning abilities of language models can inform decisions in AI safety and control.
INTRODUCTION
Does the remarkable ability of modern language models in generating coherent text result from some form of implicit planning? Our paper develops methods for investigating this question, and applies them to the case studies of rhymed poetry and question answering.
In language production, humans are known to use various strategies plan ahead for words that they utter (Ferreira & Swets 2002 Lee et al. 2013) Huettig 20155) Barthel et al. 2016). Similar mechanisms could plausibly apply in Transformer language models. In the context text generation with language models, planning for goal token(s) is a process with two stages, forward and backward: Definition 1.1. Forward planning is the creation of planning representations earlier positions that encode properties of goal token(s) at later positions.
Definition 1.2. Backward planning is using those planning representations for generating interme- diate tokens before the goal token(s) are produced.
Forward and backward planning are parts of the same overarching mechanism: the former creates a representation of the goal, and the latter conditions intermediate generation that representation. Definition 1.3. Planning is implicit planning representations are part of hidden activations, and model outputs.
explicit if planning representations Definition 1.4. Plamning is successful when (i) forward planning creates representations that causally implicate the generation of goal tokens and (ii) backward planning produces intermediate tokens that increase the likelihood of goal token(s).
By its nature, implicit planning is harder to study than explicit planning, e.g. laying out the plan for task in chain of thought. To establish implicit planning behavior, we need evidence of both

---

Published as a conference paper at ICLR 2026
successful forward planning and backward planning. In this study, we focus on phenomena where planning ahead for the kinds of words to produce is particularly transparent: rhyming, as suggested by the pioneering case study in Lindsey al. (202)) and question answering.
Rhyming and question answering offer a unique window into planning, because is linked to text elements (rhyming words answer words) whose nature and position are predictable from general principles but not determined by immediately preceding tokens. In poetry, planning for a rhyme manifests as: (i) representations of the target rhyme family emerging at the end of the first line (forward planning), and (ii) intermediate tokens in the second line that facilitate reaching that rhyme (backward planning).
question answering. planning manifests as: (i) representations of the planned answer (e.g. elephant) emerging at the end of the question (forward planning). and (ii) intermediate tokens (e.g. an) the answer that lead towards the planned answer (backward planning).
The contributions of our work include: (i) Datasets for the study of planning in rhyming (1050 lines from 10 rhyme families) and question answering (500 questions for 20 intended answer nouns). (ii) Quantitative metrics for assessing aspects of planning behavior: successful forward planning. backward planning, and successful backward planning. (iii) Evidence for implicit planning for the broadest range of language models to date, 23 diverse open weight language models (from 1B to 32B parameters). Our study uses localized steering intervention for both tasks. We show that mean activation difference steering works across the board, altering forward and backward planning. We find a lot of similarities, but also some differences between models we study (cf. Appendix H and M)
In what follows, section 2 provides the scientific context; section 3 explains the details of methodology and experimental setup. Section compares language models according to our metrics and discusses observations on the planning circuits, which parallel Lindsey et al.'s findings for Haiku. We summarize our work and discuss its limitations and further directions in section 5
2 RELATED WORK
One piece of evidence for implicit planning in language models comes from the studies of unreliable chain of thought (CoTT) "When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers" TTurpin et al. (2023); this also applies to newer reason- ing models (Chen et al./ 2025). When rationalizing, the model must be entertaining the suggested answer before the rationalizing CoT is generated. So critical output can be implicitly planned long before it is produced, an instance of implicit forward planning. Furthermore, intermediate steps leading up to the answer (in this case, the chain of thought) are conditioned on the implicit plan, an instance of backward planning.
As response to reports on LLMs producing useful plans in domains such as coding (Bairi et al. 2023, e.g.), skeptical arguments about the general planning ability of LLMs have been raised in the literature. Various studies (Valmeekam et al. 2023 Zhang et al.) 2024 Stein et al. 2025) show empirically that explicit planning abilities of current LLMS are limited. The conceptual argument that "a system that takes constant time to produce the next token cannot possibly be doing principled reasoning on its own" (Kambhampati et al. (024) naturally applies to implicit planning as well.
However, LLMs can plausibly engage in some forms of simpler, heuristic type of planning just like humans, whose general planning capacities are limited (Kahneman & Tversky| 1977| Buehler et al. 2010), are known to still plan ahead in their speech production (Brown-Schmidt & Konopka 2015). Indeed, shaping the rhythm and rhyme for the next poetic line is a task that can be solved well enough with limited planning capacities. This is supported by the success of previous poetry generation experiments that used relatively simple tools Hopkins & Kiela (2017):
( Lau el al. (0018)
Ormazabal et al. 2002 Jhamtani et al. (0119) Ghazvininejad et al. (2016).
Implicit planning representations in LLMs have been found for specific models (Pochinkov et al.
2024 |Men al., 20244) Wu et al. 2024). In this paper, we aim compare planning across wider range of language models. One of our case studies is on rhymed poetry generation, inspired by
Lindsey et al. (202), the first study of implicit planning in poetry generation. The authors prompted
Claude Haiku 3.5 with
Example 1. A rhyming couplet:) nn He saw a carrot and had grab it',
2

---

Published as a conference paper at ICLR 2026
The model was able complete the second line with the correct rhyme, e.g. His hunger was like starving rabbit. Furthermore, elements of implicit forward planning mechanisms were identified: there vector components in the activations of the token at the end of the first rhyming line (second An above) that Lindsey et al. identify as representing planning for the rhyme of the next line. Some of these planning activation components, they argue, correspond to the potential rhyme rabbit and, if suppressed, lead to other rhyming words (such as habir) being generated much more often. It is even possible to inject a planning direction for a word such as green that does not fit the context, causing the model to produce it instead of a correct rhyme.
In addition, Lindsey et al. found evidence of backward planning in the rhyming context. The planning directions, they argue, influence the intermediate words. If rabbit is the dominant planned rhyme word, certain constructions are produced, e.g. comparison construction as in His hunger was like a starving rabbit. If rabbit planning directions are artificially suppressed, different intermediate parts of the line are generated, such as His hunger was a powerful habit.
Lindsey et al.' 's evidence is limited to a small number of persuasive examples and single closed- source language model. Because of its proprietary nature and the technical complexity of the approach, Lindsey et al.'s findings cannot be independently reproduced. Our paper addresses these limitations. First, we assess forward and backward planning in both rhyming and question answering across range of models. Second, we introduce several quantitative metrics based varied dataset of rhymes. Third, we address the complexity problem: the cross-layer transcoder (CLT) approach used by Lindsey et al., while offering diverse interpretability promises, is particularly complex, computationally expensive, and difficult to replicate. Just training CLT for single model of modest size requires days of compute on a highly performant and expensive GPU such H100 (Ameisen et al. 2025 Appendix D).
As key intervention, we employ activation steering with vector addition, as in Turner et al. (0023). Directly manipulating model's 's activations, activation steering differs from other ways of controlling outputs such as prompting or different variants of controlled or constrained generation (Kumar et al. 2022 Lew et al. 2023 Loula et al., 2025). Unlike in constrained generation that can use phonetic constraints relevant for rhyming (Roush et al., 2022) or evolutionary optimization with rhyme scoring (Jobanputra et al., 2025), we do not constrain the model's outputs but modify the rhyming representations from which the language model generates the next line normally. As an implicit control mechanism, activation steering shares some conceptual similarity with classifier- free guidance in diffusion models (Ho & Salimans| (022). However, unlike classifier-fre guidance which operates during the iterative sampling process, our approach intervenes on a single position's representations in autoregressive L.M generation, making it more directly interpretable.
3 METHODOLOGY
Taking inspiration in the observations of Lindsey et al., we propose several metrics for quantifying implicit forward and backward planning, and apply them to variety of language models. We propose simpler methods that do not involve the costly training and use of cross-layer transcoders, or other types of dictionary learning. We focus here on planning for a noun answer question answering and for rhyme family in poetry; see also Appendix H for discussion of models planning to produce specific rhyming words.
3.1 Data In rhyming experiments, we call the set of words that all rhyme with each other rhyme family. Words in a rhyme family tend to share a suffix, which we use to name the rhyme family. For example, the -ing rhyme family contains words such as king and ring. We chose 10 rhyme families and 20 pairs of rhyme families such that every rhyme family appears as the first in two pairs and the last in two pairs. For each rhyme family (RF), we generated 105 first lines using Claude 3.5 Sonnet. We split the lines into train set P less data for estimating the steering vector was often enough, see Appendix B
Train (85 lines) and test set P RF (20 lines). In practice,
RF
For question answering experiments, we created a dataset of 20 noun pairs where one noun begins with a vowel (requiring article "an") and the other with a consonant (requiring article "a"), such as (eye, heart). For each noun, we created 13 suggestive questions for training and for testing, plus 7 neutral questions answerable by both nouns in a pair. We use notation PT and Test analogously to the rhyme family datasets. See Appendix for complete dataset details for both tasks.
3

---

Published as a conference paper at ICLR 2026
Models We test language models from four families (Gemma2, Gemma3, Qwen3 and Llama3. 1/3.2) with model sizes between 1B and 32B parameters Team et al. 2024 2025); Yang et al.
3.2
(2025); Grattafioni et al. (2024). For each model we test both the base version and the instruction tuned versions
3.3 Mean Activation Steering and Choosing a Steering Vector If a model exhibits implicit forward planning, then intervention of the first line) can alter the rhyme or the answer generated multiple tokens later.
early position (such the newline in [ or the last word
We estimate steering vector using the average activation difference like in Arditi et al.) 2024). We calculate the average activation of one of the last positions of the prompt: the last newline (\n) token in the prompt (newline steering), the last word of before that (last word steering). or the question mark (question mark steering for questions). For example, in the rhyming context, we can subtract the average activation lines that rhyme with sick from the average activation of the same position for lines that rhyme with pain. The resulting steering vector, when applied to the respective newline or last word token the line The house was built with sturdy, reddish brick \n, can lead to 'rhymed' generations like And stood for years, enduring wind and rain instead of the unsteered baseline version And stood for years, enduring every trick.
Steering operates on a pair of prompt categories (C₁, C2) such as rhyme families (RF) or noun answers (N). For each model, layer steering position, and prompt category pair (C1, C2) we (l.s) extract steering vector CC,+C= by calculating the mean difference between the activations on their train sets
Σ
(p)
E
x (p)
C1-4 =
Tratn
DEPTTrate
Here m is constant, set to 1.5 and is either pos() (n) (the position of the last newline token in p) or or pos(?)or the position of the last token before the newline or the question mark.
X is the hidden activation of the model layer / and position k
(1)
To apply a steering vector, we add it to the residual stream x on the correct layer and token position during generation. Note that apply the steering vector one token only (the last word, question mark or the newline token).
For the final steering vector of each model and category pair SC1-CC2 we choose out of all layers and positions the combination where steering has the maximum intended effect.
While we opt for average activation difference steering because of its simplicity, it is also possible to obtain a steering vector with other methods, such as differences of classifier probe weights or SAE weights. For examples of generated outputs with alternative steering vectors, see Appendix!] [|]
3.4 Rhyming Metrics Let CRF be collection of 1000 couplets generated by some model using the prompts in PTest We sample 50 generations per prompt p E P'Test.
Let CRF,-RF, be collection of 1000 couplets generated by some model using the prompts in P'Test while being steered with SRF1-RF2.
As evidence successful forward planning, we use effectiveness of the steering intervention: if at position X there was no planning representation for a later position Y, then intervening on position X would not have changed the outcomes for position Y in predictable way. This is done comparing the following two metrics.
Fraction of Correct Rhyme Family. We calculate the fraction of couplets in CRE, where the last word has the correct rhyme family (rhymes with the last word of the first). We made collections containing all words in each rhyme family to do this.
Only Qwen3 32B did not have an open weight base model available.
*Mean activation difference estimation defines an inherent scale: steering vector can be used with multiplier of 1. While that often works, we found that slightly bigger values such as 1.5 or produce more consistent effect. Informally, imprecision in the estimation of the steering vector is compensated by increasing its magnitude.
4

---

Published as a conference paper at ICLR 2026
Fraction of Correct Rhyme Family (Steered). This metric is calculated the same way as the Fraction of Correct Rhyme Family metric. We evaluate separately for each rhyme family pair (RF,, RR22) on CRF:-RFF. The correct rhyme family is the one that was steered towards (RF22..
Successful steering means that planning representations can be manipulated successfully. This allows assess backward planning by measuring to what extent interventions on the planning activations affect the model's behaviors at intermediate positions. For example, Gemma2 9B model steered towards the -ight rhyme family will not just end the second line in 2 with a different word like light instead of sing, but will likely take different path after above, leading to more natural ending:
Example Whispers of freedom found in a bird's wing),n Soaring above where true joy will sing Steered towards -ight: Soaring above bathed in a golden light
The likelihood of different intermediate continuations due steering is reflected in shift in the probability distribution over the next token. To measure this effect, we propose the following probability based metrics discussed in detail in Appendix [F]
We measure successful backward planning in poetry-to what extent words generated in the task's context leads up to the planned completion-use uuiiin regeneration metrics. For example, we take the line And stood for years, enduring every trick and regenerate the last word without the original context that conditioned the rhyme. backward planning involving the rhyming plan was used, such regeneration is expected to produce trick at higher rate than for generated lines in other contexts, and at higher rate than for lines generated in the same context but steering towards a different rhyme.
Fraction of Correct Last Word Regeneration. We extract the second lines of all the couplets in CRF. (So that there is no context that this is rhyme). We also remove the last word in all of the second lines. Then we regenerate the last word using the resulting prompts and calculate the fraction of cases where the regenerated word is from the correct rhyme family R.F.
Fraction of Correct Last Word Regeneration (Steered). This metric is computed similarly to the Fraction of Correct Last Word Regeneration, but with couplets in _RF_-RFF 2RF calculated the fraction of cases where the regenerated word is from the target rhyme family RF2. We calculate this metric separately for each rhyme family pair (RF1, R.F2) on CRF,-RF2
3.5 Question answering metrics
Fraction of Correct Answer We calculate the fraction of answers in ON which contain the intended answer word as a substring.
Fraction of Correct Answer (Steered) We calculate the fraction of answers which contain the intended answer word as substring. We evaluate separately for each noun pair (N1 N2) on CN, The correct answer is the one that was steered towards (N2).
Fraction of alan
We calculate the fraction of answers in CN which contain the token an.
Fraction of alan (steered) We calculate the fraction answers which contain the token a or an. We evaluate it separately for each noun pair N2) on CN, CN,,NN, to determine whether the fraction of a vs. an changes in the correct direction depending on (N2).
4 RESULTS
4.1 Basic rhyming behavior We observe that models differ in rhyming abilities. Generally, bigger models rhyme more consistently than smaller ones, and instruction- models rhyme better than their base versions (Fig. 2 solid bars). These findings cannot attributed to tokenization differences between models; see Appendix for details.
5

---

Published as a conference paper at ICLR 2026
In all models, we found evidence of successful backward planning from regeneration metries: lines generated in the context of a rhyme are likely to be completed by a word of the intended rhyme family even without the original rhyming context. For all models, the metric is above chance (ef. Fig. [
Rhyme Family Regeneration: Correct vs Baseline
Metric
Regeneration Rate
Correct Regannmation: same myme fumily OSS Base ine: average other thyme "amily
0.20
0.15
0:10
0.05
0.00
Genmed 278
Gemmaz, 50
Gemmaz 2.
Gonmna, 22.777 Base
Germaaz 28 Reop
Gemmee) 2210
Gemono 122
Common 120 Booe
Gerrnaa 10 Slae
Gemman J8 dooe
(lemn 2.38
Llamach 18. Haam
Useaa 2B Aene
Oeeed ..a0
Owend de
Camen 140 Base
Qwner oR Rave
Gemmazz SR
Gemmaa
Gemmad
Genmal 2.72
Model
Figure 1: Comparison of unsteered regeneration rates to baseline chance level for the model. During regeneration with stochastic sampling, some rhyme families are expected occur with non-zero frequency by chance. We estimate the baseline chance level that unsteered regeneration rate must exceed to show evidence for successful backward planning as the average frequency of rhyme families other than the one in whose context the couplet's 's second line was originally generated by the model.
4.2 Steering affects forward and backward planning in rhyming Steering on the last word consistently modifies models' behavior wrt the rhyme family generated. This supports that repre- sentations of the planned rhyme are localized at the intervention point. While rhyming abilities of different language models vary, our simple steering strategy achieved rates of the target rhyme family comparable to the rhyming rate of the model in the baseline condition, cf. Fig. [] Only for models
with the lowest rhyming ability (base variants of Gemma3 IB and Llama 3.2 3B) is the steered rhyming rate substantially lower than unsteered.
Rhyme Family Fraction: Baseline vs Steared
Condition
Fraction of Riyyme Family
0.8
C.-
C.4
6.
commat 228 thise
commen Atee
Gemmuz J88 Buse
Cemmaad, PM
Geronat, 128
Gemmal. 4B
Gemmall 222, thee
Commen Aloe
Geronna 1. Base
Ulmox.2 JW
Lamax 2. 88 Btose
caven .00
Gonn1 1.0
Genati at
Ownd 148 Bose
Omen) 38 Blss
Gemmat__
camoaat
Gemmaad
Gamonall
Gemmas, 278
Liamed 30
Model
Figure 2: Baseline rhyming abilities of models vs. steered rhyming behavior. Solid bars report the frequency of (exact) rhymes produced by the model (baseline rhyming behavior). Hashed bars present the frequency of (exact) rhymes of rhyme family 2 after the first line ending in a word from rhyme family 1, when steered towards rhyme family 2.
Further, steered regeneration rates for the target rhyme family are close to the baseline regeneration rates for the same models (Fig. B). This supports that the steering intervention does not just replace the final rhyming word but affects the backward planning that produces intermediate words.
6

---

Published as a conference paper at ICLR 2026
Word Reganaration: Baseline Steared
0.25
Condition
Baselin
Steered
0.20
Fraction of Correct Word
0:30
0.05
Commanz .78
Gemmatz
Gemmas
cemmaz 278, thes
Comman 28 Stoe
98
Germat 278
Gemmed
Gemmayy
Gemmall .1
Gemenal 278 Bove
Gemmat 128 Hois
Communt 48 sooe
Gemmal 28 Stne
Liemad
Llama-22
Lian 0,00 Buse
Gwnds ba
camma
Doma 140 Hano
Gomet o0 Bust
Model
Figure 3: Baseline vs. steered last word regeneration rate of different models. Solid bars represent the fraction of times when regenerating the last word of the second line with the first line removed produces a word from the original rhyme family. This suggests that the model generates the second line that it facilitates the correct rhyme, providing evidence for successful backward planning. Hashed bars represent the fraction of times when regenerating (with the first line removed) the last word of the second line generated with steering towards rhyme family 2 produces a word from rhyme family 2. Close match between baseline and steered regeneration rates suggests that we successfully manipulate the LLM's rhyme planning.
The success of steering affecting intermediate planning is supported by the fact that in models with high rhyming capabilities, in the baseline vs. steered lines, the regeneration frequency distributions by rhyme family is close, as illustrated in Fig. 4 for Gemma3 27B.
Word Regeneration by Rhyme Family: Gemma3 27B
Fraction of Correct Last Word Regeneration
Condition
0.6
Baseline
Steered
0.3
0.2
0.1
0.0
ight
oat
Rhyme Family
Figure 4: Regeneration rates per rhyme family with Gemma3 27B, baseline vs. steering. Solid bars represent the fraction of times when regenerating the last word of the second line with the first line removed produces a word from the original rhyme family. Hashed bars represent the fraction of times when regenerating (with the first line removed) the last word of the second line generated with steering towards the rhyme family produces a word from the target rhyme family. Close match between baseline and steered regeneration rates suggests that we successfully manipulate the LLM's rhyme planning for specific families.

---

Published as a conference paper at ICLR 2026
4.3 Probability based metries Our probability based metrics detect evidence of backward plan- ning on a finer level, assessing to what extent the probability distribution over intermediate tokens changes with this intervention. The probability based metrics generally follow the same patterns as the previous metrics: instruction tuned models tend to score higher than base models and bigger models tend to score higher than small ones. For plots, see (Fig. 14 15 in Appendix F
Last Word Steering Newline Steering Comparison
6emmoz_55
0emmal1_270 Newline
Value
0.6
0.4
02
Steering Effectiveness
0.0
Fraction of Correct Last
Fraction of Top-1 Difference
Fraction of High KL Divergence
Percentage of Tokens After First Top-1 Difference
Percentage of Tokens After First High KL Divergence
Word Regeneration (Steerea)
Figure 5: Steering on last word (solid bars) newline token (hashed bars) at the end of the first line of couplet. Both positions produce comparable metrics for Gemma2 9B and Gemma3 27B. Steering effectiveness is the fraction of times rhyme family 2 is generated when steering towards rhyme family 2; steering on the newline token is slightly less effective.
4.4 Steering position in rhyming All models support rhyme family steering on the last word of the first line of a couplet, typically on lower layers. Some of the language models can also be steered on the newline token after the first line in one or more middle layers; for details, see Appendix M The effect of newline steering is only pronounced for Gemma2 9B and Gemma3 27B (both instruction tuned and base variants), cf. Fig. 26 in the Appendix. While steering on the newline position is somewhat less effective than steering on the last word, it produces comparable values of planning metrics (Fig. 55. This suggests that interventions on the two positions are qualitatively similar. We conjecture the following explanation of quantitative differences: while the newline token may be playing key role in the rhyming circuits of models like Gemma2 9B and Gemma3 27B, some amount of information flows around the steered newline token directly from the (unsteered) last word position to positions further in the sequence, dampening steering effectiveness on the newline position. This is supported by observations on attention patterns involved, see 4.6
4.5 Steering in Question Answering We tested whether our steering methodology for implicit planning generalizes to question answering using noun pairs requiring different articles (e.g.. "an eye" vs. "a heart"). Steering vectors estimated from questions eliciting each noun successfully shifted answer frequencies, demonstrating forward planning. Critically, steering also affected article choice before the noun answer was generated: for example, models decreased "an" usage and increased "a" usage when steered toward a consonant-initee noun answer (Figure (6). This article selection, occurring before the answer itself, provides evidence of backward planning in question answering, consistent across all 23 models. On neutral questions, like in our rhyming study, instruction-tune models showed a stronger steering effect. These findings suggest that implicit planning is general mechanism rather than being specific to poetic constraints (full details in Appendix K).
We also conducted a small scale experiment with more complex question answering task in which the strongest model in our study, Gemma3 27B, also shows signs of implicit planning multiple tokens ahead, with backward planning evidenced by subject-verb agreement. For details, see Appendix D
4.6 Observations on Planning Circuits How is backward planning implemented? For Gemma2 9B (instruction-tuuned), identified attention heads that read from the steering vector direction. For
8

---

Published as a conference paper at ICLR 2026
Article Usage Comparison for Questions about Vowel-initial Nouns
Article & Condition
Unsteered a
Steered 'a'
0.7
Unsterred 'aan'
Steered
0.8
Fraction of Anseeers
0.3
02-
Cemmaz 278
Gemmal 90
Gemmat2 20
Gemman2 o0 Dane
Gemmaz 20 Base
Cemmaid .77
Gamma1 120
Gemman) 278 Base
Gemmas 139 Bane
Gemmaa) Base
Gemman Base
Liana3. as
Liamn 3.
Uiamal 3, Blve
Llemad 30 Bonr
Ownn1 322
(owet 148
Obend au
Owi23 140 Bate
Oment Base
Germaaz, 278
Gemma3
Gemmal
Model
Figure 6: Percentage of answers containing article a or an as a function of steering on suggestive questions. Proportion an decreases and proportion of increases when steering towards noun beginning with consonant in question about noun beginning with vowel.
Couplet (unsteered completion)/(steered completion)
Logit Difference
%
Unst.
Patched
Steered
The house was built with sturdy, reddish brick
-2.6
4.61
5.48
89
And stood for years, enduring (every trick ((sun and rain)
He stubbed his toe, striking pain Now he's laid upon the floor, (in vaint/(quite sick)
-3.37
1.47
4.77
59
Whispers of freedom found a bird's wing
-14.5
2.06
3.39
93
Soaring above
(where true joy will Whathed in golden light)
Mountains stand. As ancient guardians of majestic height Echoing whispers of legends, though the (day and the night /(the ages they sing)
-4.94
-1.07
0.93
66
Table Couplets used for analyzing the rhyming circuit. The logit difference describes the difference in logits between the first tokens of the favored steered and unsteered completions. We look at three cases: unsteered. steered, and patched, in which we do not steer the model, but replace the activations of two attention heads (L30H03 and L31H15) with the activations they have under steering. % column indicates the percentage of steering induced logit difference recovered by patching L30H03 and 133 In all cases, most of the steering effect is recovered by patching the two attention heads, suggesting that they play a major role in rhyme plan implementation.
this analysis, applied steering in the rhyming task the newline token in layer 27. We analyzed positions with high KL divergence around the middle of the second line in examples steered between diverse rhyme pairs.
For example, the first row of table I the baseline model completes the second line with every trick. But when steering towards -ain, the favored completion is sun and rain.
Two attention heads (L30H3, L31H15) play a very important role backward planning. We call the activations of attention head or any other layer when steering the model, the steered activations. If

---

Published as a conference paper at ICLR 2026
we do not use the steering vector, but instead replace the activations of these two attention heads on the last token with their steered activations (activation patching), we get a similar effect to steering. In the analyzed examples (table activation patching results in token output logits that are much closer to the steered output logits then the unsteered output logits, recovering most of the steering effect (59%-93%). L33H3 and 1.31115 attend to the last word of the first rhyming line and the newline token after it, but not to other tokens (Fig. 8 in Appendix). This attention pattern is constant across all the tokens of the second line, but only makes a significant contribution to the next token prediction at select positions, usually towards the end of the line. So in relevant contexts, these two attention heads seem specifically dedicated the implementation of rhyme planning. The information copied by these heads is then converted into specific predictions in subsequent MLP layers 30-39; patching MLP layers recovers the effects of steering almost entirely. It remains an open question whether models apart from Gemma2 9B also have only a few attention heads which move rhyme planning information to later tokens
In Gemma3 27B, which, like Gemma2 9B, also makes use of the newline token in rhyming, we similarly found that both the last word and the newline token of the first poetic line are attended to in generating the second line. To verify the causal role of this attention, we did an attention ablation study (Appendix(D).
The attention heads that transfer rhyming information aren't contributing much in question answering, where most action happens in the later layers. In question answering, attention head 1,39H13 seems particularly important, with the final MLP layers, again, driving the final probability distribution prediction. This suggests that planning for rhyme vs, answer to a question involve distinct circuits.
CONCLUSION
Our findings reveal that language models of various sizes exhibit different aspects of planning when generating rhymed poetry and answering questions. We find evidence of planning behavior even in the smallest models we consider, although it is weaker in smaller models, consistently with the conclusions of concurrent work (Hanna & Ameisen 2026). Planning metries increase not only with model size but also with instruction tuning, suggesting that typical post training may boost planning. cf. Li et al. (2024). The plan for rhyme or an answer to a question can be manipulated using the simple technique of average activation difference steering. In poetry, this technique robustly recovers the model behavior both for the rhyming word generation and at the previous steps. Steering works reliably on the last word of the first rhyming line for all models, and in addition to that on the first line's 's newline token for select models, as in Claude Haiku, Gemma2 9B, and Gemma3 27B. It remains an open question for further exploration whether the more claborate rhyming circuit in these models, which involves the newline token position, contributes their quantitatively better planning behavior that we observed.
All our rhyming metries correlate, suggesting that rhyming ability goes hand in hand with planning for a rhyme (see Appendix D for details). This holds for diverse logically independent aspects of planning: how early in the sentence the planning circuit springs action (tokens after first top-1, tokens after first high KL); how much influence the planning circuit has over the output logits (fraction top-1 difference/high KL); how good the planning circuit is at boosting tokens leading up to the correct rhyme family (regeneration metrics).
Our steering experiments support that the planned rhyme family is represented the end of the first line in a couplet and the planned asnwer is represented at the end of a question. The follow up analysis of Gemma2 9B further identifies attention heads and MLP layers that contribute the most to implementing the rhyming plan. This circuit is specific to the task; question answering uses circuits of a similar shape, but different in details. Our findings are consistent with known observations on rhyming circuits in other models (Lindsey et al. 2025 Hanna & Ameisen] 2026). suggesting general mechanism, although details such as which positions are used differ between models.
Methods we developed can be transferred from rhyming and question answering to other cases where long-distte long-distance dependencies can be manipulated, such as instruction following and CoT (Cox 2025). as well as to other intervention methods. Our findings call for further analysis of LLM planning. Since implicit planning is pervasive, especially in larger, more capable models, and may have critical safety consequences in certain domains, we need better understand the computation inyolved.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The first exploratory steps of the research reported in this paper were taken as part of the training phase of Neel Nanda's 's stream at MATS program. The compute costs of the main body of work was covered by funds from Manifold for Charity. Denis Paperno was supported by Institute for Language Sciences Utrecht University. We thank Michael Hanna for helpful comments on an earlier version of this paper.
REPRODUCIBILITY STATEMENT
Supplementary material on OpenReview contains data reported in the body of the paper and code for reproducing the experiments. This includes outputs of the rhyming exper- iments in rhyme fami ly steering and outputs of question answering experiments in noun_qa_steers
Supplementary material including code, data and outputs of the both rhyming and question answering experiments is also found at https: /github. com/ dpaperno/ susamiinngsuppleeeentte information needed for reproducing the experiments.
README. contains further
IMPACT STATEMENT
"This paper presents work whose goal is to advance the field Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here."
REFERENCES
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, et al. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. Advances in Neural Information Processing Systems, 37: 11111333366883, 2024.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. CodePlan: Repository-leve coding using llms and planning, 2023. URL
12499
Mathias Barthel, Sebastian Sauppe, Stephen C Levinson, and Antje S Meyer. The timing of utterance planning in task-oriented dialogue: Evidence from novel list-completion paradigm. Frontiers in psychology, 7:1858, 2016.
Sarah Brown-Schmidt and Agnieszka E Konopka. Processes of incremental message planning during conversation. Psychonomic bulletin & review, 22(3):833-8AM 2015.
Roger Buehler, Dale Griffin, and Johanna Peetz. The planning fallacy: Cognitive, motivational, and social origins. In Advances in experimental social psychology, volume 43, pp. 1-62. Elsevier, 2010.
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't 't always say what they think, 2025.
htlttttss///axiiv org/ abs / 250. 05410
Kyle Cox. Post-hoc reasoning in chain of thought, 2025. URL https::/kkyeeeeeeee com/ 2024/12/26/o00 http:/26c/ot-iteerp.
Fernanda Ferreira and Benjamin Swets. How incremental is language production? evidence from the production of utterances requiring the computation of arithmetic sums. Journal of memory and language, 46(1):57-84, 2002.
11

---

Published as a conference paper at ICLR 2026
Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. Generating topical poetry. In Proceedings of the 2016 conference on empirical methods in natural language processing, pP- 1183-1191, 2016.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, maanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lec. Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupulcti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis. Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero. Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li. Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal. Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape. Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Bainng Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuae Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee,
Plawiak,
12

---

Published as a conference paper at ICLR 2026
Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayaman. Hakan Inan, Hamid Shojanazeri, Han Zou. Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Tufanov, Hias Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang. Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark. Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang. Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez. Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL htt.:httt://rrxtt. www.xii..rggabbs..... wrg. http//bbb//4000 /207 21783
Michael Hanna and Emmanuel Ameisen. Latent planning emerges with scale, 2026.
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jack Hopkins and Douwe Kicla. Automatically generating rhythmic verse with neural networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 168-175 2017.
Falk Huettig. Four central questions about prediction in language processing. Brain research, 1626: 135, 2015.
Harsh Jhamtani, Sanket Vaibhav Mehta, Jaime Carbonell, and Taylor Berg-Kirkpatriic. Learning rhyming constraints using structured adversaries, 2019. URL httpp::///axxx..oo////aaaa 1909. 06743
Vedant Dhaval Jobanputra, Basam Thilaknath Reddy, Sri Ganesh Bhojanapalli, Krishna Aditya SV S. Bagavathi Chandrasekara, and Ritwik Murali. Llm-aided evolutionary algorithms for haiku generation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 2584-2587, 2025.
Daniel Kahneman and Amos Tversky, Intuitive prediction: Biases and corrective procedures. 1977.
13

---

Published as a conference paper at ICLR 2026
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: Lims can't 't plan, but can help planning in lmm-modulo frameworks. In Forty-first International Conference Machine Learning. 2024.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient- constrained sampling from language models. arXiv preprint arXiv:2205. 12558, 2022.
Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. Deep-speare: A joint neural model poetic language, meter and rhyme. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Linguissss (Volume 1: Long Papers), pp. 1948 1958. 2018.
Eun-Kyung Lee, Sarah Brown-Schmidr, and Duane G Watson. Ways of looking ahead: Hierarchical planning in language production. Cognition, 12963):544-562, 2013.
Alexander Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint arXiv:2306.03081 2023.
Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling, 2024. URL fttts://arxiv..on abs/2407. 02446)
Jack Lindsey, Wes Gurnee ad Emmanuel Ameisen, Brian Chenand Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025.
João Loula, Benjamin LeBrun, Li Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, al. Syntactic and semantic control of large language models via sequential monte carlo. arXiv preprint arXiv:2504. 13139, 2025.
Arjun Khurana McLaughlin, Oliver and Jack Merullo. have mouth, and I must rhyme: Uncovering internal phonetic representations in LLaMA 3.2. In Proceedings of the ICMI. 2025 Workshop on Assessing World Models, 2025.
Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7713-7724, Miami, Florida, USA, http://..22222.2..mm2m.mm
November 2024. Assoccation for Computational Linguisties. doi: 10. 10.18653/v/2024, 440. URL https:tpoo..mm. htttss:/aalatttooooogoo.oooo www.//g.oooo.oooooooooooo 2024. wmpnp-mnnpp..... main. 440/
Aitor Ormazabal, Mikel Artetxe, Manex Agirrezabal, Aitor Soroa, and Encko Agirre. PoeLM: A
meter- and rhyme-controllable language model for unsupervised poetry generation, 2022. URL
hittps://arxiv. www.:/ooorgaaaa.. abcp:/bs//22200 12206
Nicholas Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, and Lucile Men-Minassian. Extracting paragraphs from L.M token activations. arXiv preprint arXiv:2409.0633: 2024.
Allen Roush, Sanjay Basu, Akshay Moorthy, and Dmitry Dubovoy. Most language models can be poets too: An ai writing assistant and constrained text generation studio. Proceedings of the Second Workshop on When Creative AI Meets Conversational AI, pp. 9-15, 2022.
Katharina Stein, Nils Hodel, Daniel Fiser, Jörg Hoffmann, Michael Katz, and Alexander Koller. Improved generalized planning with Ilms through strategy refinement and reflection, 2025. URL https:://arxt www.aas/////oooooooooooooooooa / 2508 13876
14

---

Published as a conference paper at ICLR 2026
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot. Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, lean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Danila Sinopalnikov, David Weinberger,
Perry, Chris Welty, Christopher A. Choquette- Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucinske Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan. Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani. Natalic Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arood, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Hli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL /etpps:/erxxv..rett
00118
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky. Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic. Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton,
15

---

Published as a conference paper at ICLR 2026
Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Figer, Susan Zhang. Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sehastian Borgeaud, Olivier Bachem, Armand Joulin, Alck Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma technical report, 2025. URL // http://wwvvvv....o..
wwg. abs /200 19786
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, arXiv preprint arXiv:2308. 102444888.. 2023.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. URL
https::/erxiis>
http://xxvv..og//oo/////// wwtpo/////////oo /205 04388
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models critical investigation. In 7Thirty-seventl Confer-
ence on Neural Information Processing Systems, 2023. URL forum? ddx sSW
Wilson Wu, John X Morris, and Lionel Levine. Do language models plan ahead for future tokens? arXiv preprint arXiv:2404.00855 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttp://aaxvvvvvvvv org/ abs abs 09388
Yuan Zhang, Chao Wang, Juntong Qi, and Yan Peng. Leave to large language models! correction and planning with memory integration. Cyborg and Bionic Systems, 5:0087, 2024.
16

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
A. RHYMING DATASET
For our rhyming evaluation we steered between the following 20 rhyme family pairs: (-ing, -air), (-ing, -ip), (-air, -ip), (-air, -oat), (-ip, -oat), (-ip, -ird), (-oat, -ird), (-oat, -ee), (-ird, -ee), (-ird, -ight), (-ee, -ight), (-ee, -ake). (-ight, -ake), (-ight, -ow). (-ake, -ow), (-ake, -it), (-ow, -it), (-ow, -ing), (-it, -ing), (-it, -air). For each rhyme family RF 105 first lines ending with words from RF, were generated using Claude 3.5 Sonnet. The resulting 105 lines per family were randomly split into 85 training and 20 test prompt lines. We then manually checked the training data, replacing individual lines with incorrect rhyming words with newly generated lines. For some rhyme families we also replaced the whole training data if it was heavily unbalanced, with a large share of examples ending in the same word. In case of complete replacement lines for a rhyme family, we prompted Claude 4.0 list 17 words belonging the rhyme family, and after manually checking their correctness, to produce poetry lines ending in each word. When developing our methods, we found that balanced training data improves the performance of the estimated steering vector.
For steering vector estimation and inference, each first line
was prepended with
"A rhyming couplet: \n"
A.2 NOUN QUESTION ANSWERING DATASET
The noun question answering study was based the following 20 noun pairs: ("eye', 'heart'), C'ear,, 'brain'), ) ('elephant', 'whalc'), Cowwl', 'shark'), ("actor', 'doctor'), ) ('athlete', 'student'), Cairport', "hospital"), ) ('office', 'school'), ('ocean', 'river'), ('arrow', 'star'), ("heart', 'eyc'), ('brain', 'ear'), ('whale", "clephant'), ('shark', 'owl"), ('doctor', 'actor'), ('student', 'athlete") ) ("hospital", "airport'), ('school', 'office'), ('river', 'ocean'), ('star', 'arrow").
The data was selected from an initial set of 100 nouns. We used Claude Opus 4.1 to generate 30 diverse suggestive questions about each noun. We then tested three small models: Gemma3 4B, LLama3.2 8B, and Qwen3 8B on the questions. We selected easy, highly suggestive questions by retaining only questions that at least out of the three small models answered correctly. Finally, paired nouns that had at least 18 retained questions into pairs where one noun starts with a vowel and the other starts with consonant and the nouns in pair represent the same ontological category.
The neutral questions were generated by Claude Opus 4.1 so that each question can be somewhat plausibly answered by both nouns in the pair, e.g. What large manmal endangered? for (ele- whalt.
For steering vector estimation and inference, each question was prepended with the 2-shot context "Question: What two-wheeled vehicle do you pedal 1 P\nnswe::: bicycle (nquesttoon What flying vehicle carries passengers in the nAnswer:: an " "inAnswer:".
and postpended with
17

---

Published as a conference paper at ICLR 2026
B
TRAINING DATA SIZE
Steering Effectiveness Training Data Size
Instruction-tuned
Base
1.0
Steoring Effactiveness (Target Family Fraction)
0.4
Models
Gemma1_18
Gemma3,18
Gamma3_48 _48 Rase
Gemme1_276
Gemmaz_1a 126
Gemmaã 128 Base
15 20
40
85
7
10
15
40
85
Number Training Examples
Number of Training Examples
Figure 7: Steering effectiveness training size data for Gemma 3 models.
We found that additional training examples mostly benefit steering effectiveness in less capable models, while the most capable models can generalize well from just few examples or even single example pair. Figure illustrates the efficiency of steering training examples used to estimate the steering vector for Gemma3 models. Results are reported for the best layer and position for steering. For fair comparison of models, we used large set of 85 training examples for each rhyme family.
function the number of
18

---

Published as a conference paper at ICLR 2026
C ATTENTION PATTERNS
In Gemma-2 9B, which allows for efficient newline token steering, attention heads which read from the steering vector attend both to the last word and the newline token, see Fig. [ The rhyming circuit therefore crucially involves both tokens; the newline token, while similar in function what Lindsey et al. report for Claude Haiku, does not play an exclusive role. Note that Lindsey et al. do not analyze activations the last word of the first line.
L30H3 weighted attention (Steered)
-0.5
<hop>
med
couple
The
house
was
built
with
Sturdy
reddish
brick
And
sased
for
years
enduring
L30H3 weighted attention (Original)
-50
-0.5
<hoss
mad
couple
The
house
was
buitt
sturdy
reddish
brick
And
stood
far
Years
enduring
L31H15 weighted attention (Steered)
0.5
50
-50
<bos>
myy
med
couple
house
wes
built
with
Surdy
reddish
brick
Mtodd
Years
enduring
L31H15 weighted attention (Original)
50
550
-0.5
<bop>
thy
med
couple
The
house
busit
sturdy
reddish
baick
And
stood
for
years
enduring
Figure 8: Attention patterns for heads L30H3 and L31H15 at a fork point.
19

---

Published as a conference paper at ICLR 2026
D ATTENTION ABLATION
To strengthen our results we also do an attention ablation study. On Gemma3 27B, we measure it's ability complete the rhyme and plan for the last word using the Fraction of correct rhyme family and Fraction of correct regeneration metrics. We do so in 4 different conditions, shown in Figure which the attention mask of the model modified such that there no attention paid specific tokens and therefore also no information flow from these tokens to later positions.
Effect Attention Ablation on Rhyme Completion (Gemma-3-278)
1.0
Fraction of correct family Praction of correct regenaration
0.9
0.6
Raction
0.4
0.24
0.0
Ablating attention to both last word and newline
Ablating atcention to newline
Aslating attention to last word
No ablation
Figure 9: Fraction correct rhyme family and Fraction of correct regeneration on the rhyming dataset for Gemma3 27B under 4 different conditions. Attention ablation of the last word of the first line and the newline token, attention ablation the last word, attention ablation of the newline token and the baseline condition without ablation.
We observe that, as expected, the model is not able to complete the task well when attention to both the last word and the newline token is ablated. When only the attention one of those is ablated, the model is able to plan well, but the highest regeneration scores are only reached without ablation. Like our previous analysis, this suggests that representations on both tokens are relevant for the backward planning ability of the model in the context of poetry.
20

---

Published as a conference paper at ICLR 2026
E CORRELATIONS OF DIFFERENT METRICS
Cerrelation Matrit Plassing Hebrica
AGCA
146
340
2014
LON
suargeece
3PM
AXP
640
A122
Figure 10: Correlations of different metrics at individual prompt level; all models vs. with exclusion of Gemma base models, which tend to show idiosyncratically high KI. divergence values.
We report correlations all our metrics for multiple settings. This includes correlations of metrics for model) pairs, as well as correlations between models with metrics averaged across all prompts.
All of our rhyming metrics correlate with rhyming correctly (Fig. 10). but they measure different aspects of planning in practice. we control the data to only include instances where model can produce correct rhyme with high probability. metries' correlations start to diverge. 'Tokens after' metrics become especially independent; see Fig. [2] In other words, badly rhymed outputs can be associated with poor involvement of rhyming circuits. On the other hand, provided that the model's rhyming behavior is stable, earlier execution of the rhyming plan does not necessarily lead to better rhyming: we have seen examples where an appropriate rhyming word is generated early, before the line would naturally end.
21

---

Published as a conference paper at ICLR 2026
Modeli MAggregated by Model
Redluding Geromu2 Rase Modils Maggeegated Mode:
festie
yerr, stenattton,, wwstooodd www.uuooommmmree femily.
0:13
6.42
Lannaannneee Lanstorred the min.ambb 0001
0U3
2.13
0:54
100
6:00
o.44
0:0
23
3:50
079
IAC, werd, repeseeterr (teeeee, rlyine Senwlyg
5:59
5:90
0:
Mar. wert pepree steeler ne, Senwin
0009
0:99
97
3:93
5.9
Tadter, 2aap fitseoo
6:41
0:7
2:33
0.70
1.00
070
8:5
1.7
Instionutopp Juchhrnncc
0.06
0.85
287
500
664
0.44
Prccbon HUL serpence-oo
0:53
0:49
3.55
0.50
0.00
1.00
8:25
137
Pacton PUL verpance
0.00
0:32
3.0
1.00
677
0:11
0.99
1.5
0.0
0.5
031
1.0
1.53
0.76
3.54
0:31
2.11
5.14
033
021
1/21
Soet tit ony oour abcurore box
Figure 11: Correlations of different metrics at model level; all models vs. with exclusion of Gemma 2 base models, which tend to show idiosyncratically high KL divergence values.
Nodex with Cornect d Steaned Last Kate D.7 Cemmaad Basce Node
Mr, wiri 2u fetii Lniieeee, Pheme Se sirss
37%
Figure 12: Correlations of different metrics, data filtered for reliable baseline and steered rhymes. aggregating data by model. Excluding Gemma2 Base models, which tend show idiosyncratically high KL divergence values.
F PROBABILITY BASED METRICS IN RHYMING
Let YRE be collection of sequences of probability distributions over text tokens generated by using CRF as input to a model.
Let YRFI-RF, be a collection of sequences of probability distributions over text tokens generated by using CRF, as input to a model while steering with SRF,
Fraction of Top-1 Difference. Divergence in the top 1 most probable next token predictions of the stecred and unstecred models signals backward planning in action. We calculate the fraction of such divergent positions in the second line of couplet.
Fraction of High KL Divergence. We calculate the average fraction of tokens in the second line where the KL divergence between the next token probability distribution using the steered vs.
22

---

Published as a conference paper at ICLR 2026
rhyming couplet:
dropped my phone upon my knee
17.5
15.0
divergence
7.5
5.0
0.0
the
screen
cracked
all
see
Most probable next token:
baseline steered
it
screen
Is
cracked
woe all to see
screen
cracked
what
sight
Illustration of the probability based metrics. Gemma3 27B model, baseline vs. steering from to -ight rhyme family. Dashed line KL divergence threshold of 1; in the table, bold different top-1 prediction under steering. At the first token Now, the baseline and steered runs' probability distributions for baseline, all for steered). At the comma (.), not only are the top candidate tokens different, but KL divergence is also above 1.
Figure 13:
still similar (KL divergence under 1), but the top tokens different (ir
unsteered model greater than 1, out of all tokens the second line of the couplet. Formula:
1 KL Divergence Yar, [1.]] Yn,, ne, - (t.]))))
YRF
CRF
(Es)(CRF(()
Fraction of Tokens After First Top-1 Difference. We caleulate the average position where the first top-1 difference occurs, measured in % of tokens of the second couplet line, counted from the end of the line. This measures how early backward planning kicks in on average.
Fraction of Tokens After First High KL Divergence. We calculate the average position where the first high KL divergence occurs, measured % tokens of the second couplet line, counted from the end of the line. This is another measure of how early backward planning kicks in average.
Figure 14: Percentage of top-1 difference and high KL divergence under steering for different models; higher percentage indicates stronger backward planning.
Metrics do show some idiosyncrasies. Perhaps due to the threshold for high KL divergence interacting with model properties, base versions of Gemma2 models have elevated KL divergence scores (as especially in Fig. 14 right), even if it does not correspond elevated values of non-KI. metrics.
23

---

Published as a conference paper at ICLR 2026
Relative location (% of tokens after the position) of the first top-1 difference or early KL divergence under steering; higher percentage indicates earlier effect of backward planning.
Figure 15:
And for the position of first divergence metrics (percentage of tokens after first rank switch/high KL), differences between model sizes are much less pronounced than for other metrics.
24

---

Published as a conference paper at ICLR 2026
Table 2:
Vocabulary size and fraction of single token rhyming words. The fraction computed on the basis of unique words from each rhyme family both in the training and the test examples. Rhyming words that repeat across examples are counted only once.
MODEL FAMILY
V
% SINGLE TOKEN
GEMMA2
256K
0.72
GEMMA3
262K
0.67
LLAMA3
128K
0,51
QWEN3
152K
0,51
G TOKEN PROCESSING
Our findings and their interpretation could depend on the way the input is processed when it is tokenized and embedded. In this appendix, we report information relevant for the processing of rhyming words in terms of tokens.
First of all, different model families use different tokenizers. Indeed, Gemma models tokenizers with larger vocabularies, which might help them learn richer token representations. Those in turn might contribute better rhyming capabilities. However, our main findings regarding the influence of model size and instruction-tunae are not affected by differences in tokenization, because all models of the same family use the same tokenizer regardless instruction tuuning and size.
We interpret our approach to steering as replacing the features of the planned rhyme family. However, all words in rhyme family shared the last token, we could have been effectively replacing the embedding of the last token itself as opposed to (presumably) rhyming features. In reality, most of the rhyming words are not split into multiple tokens, see statistics in Table 2 Furthermore, even when a rhyming word is split into more than one token, the last token is not always shared across words. For example, Gemma2 tokenizer treats most words that rhyme with rare as single tokens, except for fla-ir, g g-lare, and des-pair only in the last case does the last token coincide with another word from the same rhyme family.
Rhyme Family Fraction: Full vs Single-Token Comparison
3.0
Condit
Daxeive (hal
faveliie (Singl Sime)))
456 Steered (F4E
Fraction of Rhyme Farilyy
(00 Stecrel (Shgle Tiken
C.6
2.4
c.
Commanl 27
Gommid di
Garmisz, 20
Gummed, 30 Baor
Gamma 27
demmal, 1/B
Gemmid di
Gemmia 16
Gemmenl 128 Stoe
Gemmna) 18 Baue
Ulmmr33 J0
Uamall JB Bana
Uama332 Base
Oweed. 3.3
Owees 1e0
geind ns
Comnn 148, Base
Oenec as Buse
Dermaaz 278
Gemiand 48
Gerenaz
Moder
Figure 16: Baseline rhyming abilities of models vs. steered rhyming behavior. For each model, the bars report, respectively: 1. Frequency of rhyme family (basic rhyming behavior), full test data. Frequency of rhyme family 1 (basic rhyming behavior) for test examples where the last word of the first line is a single token in all models. 3. Frequency of rhyme family 2 under steering (steering effectiveness). full test data, 4. Steering effectiveness for test examples where the last word of the first line is single token in all models.
To further control whether differences in tokenization significantly affect our results, we report our key metrics for the subset of test data where the last word of the first line in couplet is always
25

---

Published as a conference paper at ICLR 2026
Last Word Reganeration: Full Single-Tokan Comparison
Praction of Correct Last Word Reganeration
0.30
Gondi
(udl
0.25
690 Stewred (Skof-Bber)
0.35
0.30
0.05
Gemmait, 30
demmat" 278 pesa
thoee
Germad 2/8
Cemmor 120
Gemmall 11
Gemmat 128 flise
Communt 48 sooe
Cemman 18 Bine
Limua 2.4 Bloce
Comnt Jun
Ownet sa
Omna 188 Hane
Ownery al Huse
Gennast
Gemmat
Gemmal
Germaz 278
Ulemad
Ulema
Unmma .8
Model
Figure 17: Last word regeneration rate of different models. For each model, the bars report, respectively: 1. Frequency of rhyme family in regeneration, full test data. 2. Frequency of rhyme family in regeneration for test examples where the last word of the first line is a single token in all models. 3. Frequency of rhyme family 2 in regeneration from lines produced under steering, full test data. 4. Frequency of rhyme family in regeneration from lines produced under steering for test examples where the last word of the first line is a single token in all models
tokenized as a single token across all models. As one can see in figures 16 and 17 this filtering does not affect the results substantially. Some small consistent changes may be attributable frequency effects. For example, single token words are a little easier to rhyme, which could be related to the fact that single token words are usually more frequent.
Token Embedding Similarity: Within Across Rhyme Families
Average Embedding Similarity
0.30
Comparison
0:25
Within myme family
8880 Access fhyre faniiies
0:20
0:15
010
0.55
Germal_un 2m
Gemmaz 50
Gommat 778 Bane
Gammart 58 Buse
28,88 Hove
Gemman 1,77
Cammnuz, ItB
Gemead 278, Buse
Gemmaa 128 floe
Gommar 48 Bisee
Comman2 JB Buse
Laaaal 00
Laamdd 1..0 daae
Ulma322 38 Bane
Gmer1 140
Qpnen d0
Gwand Ause
Gemer3 Buse
Genmiz
GenomaJ
Gemmaz
[lema 2
Gwnns
Model
Figure 18: Embedding similarities of the last token of rhyming words across model. The plot shows average cosine similarity within rhyme family, compared to baseline similarities of last tokens across rhyme families.
Finally, differences in rhyming behavior could theoretically be attributable to the prominence of phonetic features in token embeddings of different models. We control for this by comparing average token embedding similarities for the last tokens of words within the same rhyme family vs across rhyme families. This comparison, reported in Figure 18] suggests that token embedding structure does not play a major in the differences across models: cosines are similar for base and instruction tuned models, as well as across model families and sizes. For Gemma2 9B, the cosine similarities within rhyme families are higher, but so are similarities across rhyme families. The only outliers Gemma3 1B models where token embeddings within rhyme families relatively dissimilar. This might contribute to Gemma3 1B's relatively low rhyming metric values; note however that instruction-tun Gemma3 1B still rhymes better than base L.L.MA and Qwen models, and slightly better than instruction-tumn Qwen3 8B.
26

---

Published as a conference paper at ICLR 2026
H SPECIFIC WORD STEERING IN POETRY
We did multiple experiments trying out single word steering: steering the model say specific word at the end of the second line, such as rabbit or habit in Lindsey et al. example. We document one such experiment here, in which we steer Gemma2 9B and Gemma3 27B to end the second line with either the word night or the word light. To do this, we use Claude 3.5 Sonnet to generate 20 couplets ending Claude to write the first line, such that it is very suggestive for the word specific word. We then generated second lines for each prompt 500 times estimate the probability that it would end in the conrect word which we call suggestibility. We filtered out all prompts with a suggestibility below 0.8, which left a handful prompts on both sides. We used those to calculate the steering vectors on the newline token as described in the paper. We steered on the following prompt: AA rhymed couplet: The forest path seemed to shrink quite tight estimating the probability of the second line ending in either light or night (500 samples).
night and 20 couplets ending in light. We prompted
124
1.0
0.3
1.4
5.0
Wordlist: light
meg perrpe of Aadiive neg peg_prooor_t yt Negative
5.05
100 Baed Roettrn.
0.70
10
20
Figure 19: Fraction of second line ending in Light/Night when steered (Gemma2 9B)
Steering could only change the probability of a certain word by at most 20 percent for Gemma2 9B and 50 percent for Gemma3 27B, lower effect than what we can get rhyme family steering. This and similiar experiments suggest that specific word forward planning may emerge with size of the model; single word steering works very little for the models we analyze but might work better for larger models like Claude Haiku.
I OTHER STEERING METHODS
While we opted for the simple average activation difference steering, it is also possible to use other types of vectors for steering, relying embedding components encoding phonetic information. There is indeed support for phonetic features encoded systematically in LLM's embedding space Mcl. Maghlin & Merullo (0055).
27

---

Published as a conference paper at ICLR 2026
nog promoooo rodtive
-M- nog a0g_prompL.t segatve
80
Wordlist: ight
1.0
nog.eeeets eeg_pronetyy
0.8
30
layer
Figure 20: Fraction of second line ending in Light/Night when steered (Gemma3 27B)
One approach is to train probe residual stream activations for classifying rhyme families. Probe weights for a specific rhyme family act as its representation. Steering can then proceed using the difference of class weights of the target and source rhyme families.
is also possible to identify sparse autoencoder latent dimensions that correspond to rhyme families and them for steering. For instance, found that latent 5862 in GemmaScope 16K SAE for residual stream of layer 20 in Gemma2 9B tends to fire on words that rhyme with night, and that latent 14069 fires words that rhyme with deep. The difference between the decoder weights of the two can be used to steer generation between the rhyming families.
Both probing and SAE based approaches have the benefit of producing representations for specific rhyme families and as such open paths for further representation analysis. However, both these approaches come with additional complexity, in partieular requiring tuning the steering coefficient in each case. For this reason, we chose the simple and robust mean activation difference approach for our experiments, which allows to easily scale experiments across models, layers, and position.
Some examples of steered generation with alternative approaches (probe weights and SAE latents);
Prompt:
rhymed couplet: AnThe house was built with sturdy, reddi sh brickAn"
Baseline outputs, Gemma2 9B instruction tuned:
rhyned couplet: \nThe house was built with sturdy, reddish brick AnAnd stood for
'A rhymed couplet: AnThe house was built with sturdy, reddish brick Annd stood for
A
rhymed couplet: AnThe house was built with sturdy, reddish brick AnAnd stood for
A
rhymed couplet: AnThe house was built with sturdy, reddish brick Anndd stood for
Outputs with steering on newline token with difference of probe weights for -ick rhymes and -ore rhymes, steering layer 27, steering multiplier 400:
28

---

Published as a conference paper at ICLR 2026
"A rhymed couplet:\n
The house was built with sturdy, reddish brick(n A testament to love, timeless, sturdy core. An/nIn Please provide another rhyming couplet that continues the theme of home bu 1t. "A rhymed couplet:\n
The house was built with sturdy, reddish brickln And stood steadfast, defying winter's core. An'n Let me know if you'd like to explore more poems! rhymed couplet:
The house was built with sturdy, reddish brickIn And stood for years, a testament to yore.Anhn #xxplanationo=
*Rymee*** The words "brick" and "fore" rhyme', 'A rhymed couplet:\n
The house was built with sturdy, reddish brickln
And stood through storms, testament to \n'e.Anin
This is an example of a couplet, a two-line stanza in poetry with rhyme'
Prompt:
"A rhymed couplet: AnIn silence, the strongest hearts sometimes softly weep
Baseline outputs:
"A rhymed couplet: An
In silence, the strongest hearts sometimes softly weep For unspoken sorrows, secrets they gently keep. \n Inini Let me know if you'd like more! \n\n", "A rhymed couplet:\n
In silence, the strongest hearts sometimes softly weepin While the bravest souls their hidden sorrows keep. \ninin Let me know if you'd d like to explore more poetry!ln", "A rhymed couplet:
In silence, the strongest hearts sometimes softly weep For a love that's s lost, secret they keep. \n'nin Let me know if you'd d like more couplets!\n", 'A rhymed couplet:
In silence, the strongest hearts sometimes softly weep While smiles mask the tears that in secret they keep. \alninia' "A rhymed couplet:\n
In silence, the strongest hearts sometimes softly weepin And hidden tears, like pearls, their secrets keep. \ninin Let me know if you'd d like more poetic explorations! \n"
Steering on the newline token on layer 20, steering multiplier 100 produces the target rhyme in 832 out of 1000 rollouts when steering with difference of SAE decoder weights for latents 14069 (-eep) and 5862 (-ight) from GemmaScope 16K. Output examples:
"A rhymed couplet: An
In silence, the strongest hearts sometimes softly weep For dreams that vanished the lonely, darkest night. Ininin Let me know if you'd d like more! 'A rhymed couplet:\n
In silence, the strongest hearts sometimes softly weep Tears, unshed, held back by a will, strong and bright. \ninin' 'A rhymed couplet:\n
In silence, the strongest hearts sometimes softly weepln When hidden burdens weigh heavy, day and night. \nininin', "A rhymed couplet:\n
In silence, the strongest hearts sometime ies softly weepin
29

---

Published as a conference paper at ICLR 2026
For burdens unseen, hidden from day's bright light. In Let me know if you'd d like more!lapa "A rhymed couplet:\n
In silence, the strongest hearts sometimes softly weepin For losses unseen, yet felt with all their might. An Let me know if you'd d like more! I can write you more couplets on different topics, \n"
30

---

Published as a conference paper at ICLR 2026
J SAE LATENTS AND RHYME PLANNING
thy and couple t:
in momorts stands perfectly still Laan Fire files dance ttrough acadows
1ight
Kihatiess of Usees siees
Actattion ifizent son2
12
10
B000
Figure 21: Top left: SAE latent 5862 (layer 20, GemmaScope 16K) fires words from the iight rhyme family, but also on the newline token at the end of the first line of a couplet where rhyme for the next line can be planned. Bottom left: Firing on the first but not the second newline is systematic, as shown on a sample of 100 couplets with -ight rhymes. Right: In contrast to Gemma2 9B' 's 5862, latent 63080 in Llama 3.1 8B fires on words of the ight rhyme family, but its activations on the following newline are comparable to its activations words from different rhyme family.
Some suggestive observations on sparse autoencoder latents provide further evidence for planning in rhyme generation.
As mentioned above, latent 5862 (GemmaScope 16K layer 20) corresponds to words that rhyme with night, and we can use it in steering to produce that rhyme. We observe further that latent 5862 tends to fire both on the last word the first line of a couplet (e.g. light) and on the following newline token. These the two positions that support rhyme family steering in Gemma2 9B, thus involved in the rhyme planning circuit. Latent 5862 is not activated on the newline token after the second line of couplet, where rhyme planning is not needed.
We identify for Llama 3.1 8B, layer 25, latent 63080 with a similar function (fires on words rhyming with night). however, latent 63080, unlike its Gemma2 9B counterpart, is not activated newline tokens. This is consistent with the fact that Llama does not support rhyme family steering on the newline token. See Fig. 21 for illustration.
K NOUN QUESTION ANSWER STEERING
We further investigated the steering for an answer across the full range of 23 language models.
We created dataset of questions that allows us detect whether the manipulating the planned answer to a question using our steering methodology affects the generated answer as well as the choice of the article form. For example, we apply targeted intervention LLM activations in question whose answer is "an eye". The steering is applied locally to a position at which the answer not yet generated, such as the last word of the question. If can steer the model answer "heart", this presents evidence for forward planning. If we further find that the model chooses article form rather than an before generating heart, this constitutes evidence for forward planning.
Our dataset includes of 20 common nouns, of which 10 begin with vowel and 10 begin with consonant. For each noun, we use 13 suggestive questions for estimating the steering vectors and 5 suggestive questions for testing the stecring. We also created second test set of 7 questions per noun pair that are meant to be (more) neutral.
Steering vectors for noun pairs were estimated via mean activation difference for the 13 questions from the train set for each noun. The best steering location was selected among the 80% middle layers of each model and three positions: question mark token, the position after it (newline token), and the position before it (last word). The best layer-position combination was used for the results reported.
31

---

Published as a conference paper at ICLR 2026
All models showed the best steering one of the middle layers. The best steering location in most cases was the last word. For LLaMA and Qwen models in neutral questions and for Qwen3 14B Instruct in word-suggestive questions the best steering position was the question mark. See figures for details on steering effect across layers and positions.
31 34
Noun Steering: Baseline frequency in neutral question vs Steered toward target noun
Fraction of Answers Conteining Target Noun
0.7
Cond
Model Farrlyy
Baseline (Unstoered)
Gemmaz
2000 Staeeed
Gemma3
Llems
Owen3 0
0.5
01
Gemmat, 270
Commuuz, o8
Cemmard 22
Cammat, 271, Bate
Gemmad, 98, Bise
semand 20 Badd
Gemmer 210
Ganoerr Commaty op
commed 278 more
Gemmad 48 Beee
Gemmad 1t Bese
Liana31 au
(1amn' 1. Rone
Liamec .e Bose
Gwndd J01
Oeme 140
Owea12 an
Oment, 140 Base
Gwnnd a6 Basa
demmu 120
Model
Figure 22: Percentage of generated answers containing the noun in baseline answers to neutral questions vs. steered towards the target noun. In this setting, steering recovers only fraction of the target noun's 's baseline frequency.
Noun Steering: Baseline (nounl unsteered) vs Steered (noun2 in steered)
Fraction of Answers Containing Target Noun
2.0
Condition
Model Family
Baselize (Unsteered)
Gemma2
WMG Steeed
Gemma3
0.8
0.4
Gemmaz, 270
Gemmaal, Ail Gemmaz 20
Germaal Aano
Centmaa SR Buse
commedd 28, Base
Cormina Germend tnt 278
Gentaal oo
Gemmadd 278,,0 Busee
Gemmad, Aano
11
Gemima 48, tlose
Commen)) 18 stoe
Usmalll est
Llemall 26
Llmmd. oog stace
Lian 33 Base
Goeend, 2an
Genrci 24f stse
Grein 88 Rloms
Qeveed
Model
Figure 23: Percentage of generated answers containing the noun in questions for that noun vs. steered towards the target noun in questions for the other noun in the pair. In this setting, steering recovers only a fraction of the target noun's 's baseline frequency.
Steering with mean activation differences of questions for a pair of nouns reliably increased the frequency in the answer of the target noun steered towards, see figures 23 and 22 (note that the questions themselves do not contain the target nouns, but are meant to elicit the nouns as answers). Steering was far from fully recovering the baseline noun frequency for suggestive questions, but worked quite well in affecting the answer in neutral questions. This suggests that while answer planning is probably distributed across positions and layers so a single position-layer is not enough to capture it, we did intervene on an important location, manipulating forward planning for the answer. models show a stronger steering effect than base models
Like in the rhyming study, instruction-tume in neutral questions, suggesting that instruction tuning boosts planning. For suggestive questions, patterns are much less clear, presumably because of planning is more distributed across positions in informative contexts.
32

---

Published as a conference paper at ICLR 2026
Article Usage Comparison for Questions about Consonant-initial Nouns
Article & Condition
Unsteered '''
Steered 'a'
0.7
Uniteered ''an'
Steered 'an'
Fraction of Answerss
0.6
0.5
0.4
Cemmaz 278
Gemmal. 90
Gemman2 20
German, 278 Base
Gemman2 98 Danle
Gemmaz 20 Bane
Cemmund .77
Gamma1 120
Gemmai 139, Bane
Gemman Base
Gemma3 Bose
Liana3. aa
Liamn 3.
Uamad 3, Bloe
Ulema3 30 Boer
Ownn1 322
(owet 148
Obend at
Owi2 140 Base
Omenn 80 Base
Gemman
Gemmal
Gamman 278
Model
Article Usage Comparison for Questions about Vowel-initial Nouns
Article & Condition
Unsteered '''
Steered 'a'
0.7
Unsteered ''an'
Steered
0.6
Fraction of Answers
0.5
0.3
02
Gemmaz2 378
Commun, 278 Baw
Gemmaz 98 Bate
Gemmaz 28, Bave
Gemmaz, Giemma3 378
1zn
Germmad 278, Base
Gommus' 120, Bave
Gemma3 48 Base
Gemmai 18 Base
Limma 2.2 3B
Liamma. 1,00 Base
Liemal 30 Base
Domm2 376
O2en1 146
Owen2 148, Bosc
Owent 3n, Base
Germaa?
Gormma2.
Gemmaa3
Gemma11
Ulam3.11
Gwemn
Figure 24: Percentage of answers containing article a or an as a function of steering on suggestive questions. Proportion of an increases and proportion of decreases when steering towards noun beginning with a vowel in a question about a noun beginning with a consonant, and vice versa when steering towards noun beginning with consonant question about a noun beginning with vowel.
The word pairs were specifically selected to contain one answer noun that begins with consonant (and goes with article a) and one noun that begins with a vowel (and goes with article an). We measure if articles used vary with the target noun when steering. Indeed, the frequency of article forms modulated predictably by steering both in suggestive questions (fig, (4] and neutral questions (fig. 25). providing evidence for backward planning in question answering context.
33

---

Published as a conference paper at ICLR 2026
Article Usage ("a" vs. "an") for Noun QA Steering
Condition
Baseline
Steered to Consonant Noun
Steered to Vowel Noun
Fraction "an"
0.4
0.3
0.2
0.3
0.5
0.6
0.7
0.8
0.9
Fraction
Figure 25: Percentage of generated answers to neutral questions containing article vs. an as result of steering on neutral questions. Lines connect data points for steering conditions of the same language model.
L MORE COMPLEX QUESTION ANSWERING AND SUBJECT- VERB AGREEMENT AS A TEST OF IMPLICIT PLANNING
Similar to Hanna Ameisen (2026), we constructed dataset to investigate implicit planning in the context of subject-verb agreement:
Example 3. Question: There were keys but 4 were lost. How many keys are left?' n Answer: Now there
All dataset examples follow similar pattern Example 3 varying the nouns and corresponding verbs. LLM's typically produce verb form of "to be" followed by number as the next tokens in this case, "are 3" This task appears to require implicit backward planning: the model must generate the correct intermediate token (either "is" or "are") before producing the goal token (the correct number).
The model could solve this task through in two distinct ways. The first is improvisation, where some circuits in the model determine the correct verb form without representing the planned answer. The second is planning, where the model has some intermediate representation of the correct answer and uses this representation to generate both the appropriate verb and the answer itself. Our experiments provide suggestive evidence for the planning mechanism.
We evaluated this task on Gemma3 27B using a dataset of 80 questions: 40 with answer "1" and 40 with answer "3", split into 30 training examples (for estimating steering vectors) and 10 test examples.
Table demonstrates that Gemma3 27B reliably produces the correct verb form in the baseline condition. Crucially, steering on the question mark token can shift the model to produce the steered verb form in substantial fraction of cases. Additionally, steering occasionally succeeds in changing
34

---

Published as a conference paper at ICLR 2026
Metric
Answer 97.5
Answer 3
Correct Verb (%)
100.0
Correct Number (%)
98.4
100.0
Steered towards Verb (%)
52.8
9.4
Steered towards Number (%)
0.0
9.0
Table 3: Accuracy and steering effectiveness of Gemma3 27B on the subject-verb agreement task. Baseline metrics show the model's ability to produce correct verbs and numbers. Steering metrics show the percentage of cases where steering on the question mark token successfully shifts outputs toward the target verb form and corresponding number.
the models answer from "3" to "1". For some prompt this effect is quite large. For The example prompt in steering causes the model to output "is" in 93% of cases and "1" in 58% of cases.
If the improvisation mechanism were true, then we would expect our steering to only be able to manipulate the verb and the effect the answer token coming entirely though the indirect effect of changing the word. To test this, we conduct another small experiment where we directly append "is" the input prompt instead of using activation steering. In this case, the model never outputs "1". This suggests that at least in some cases, our steering vector captures a representation of the planned answer ("""",, which the model then also uses to generate the correct verb form, providing evidence for implicit planning rather than mere improvisation.
35

---

Published as a conference paper at ICLR 2026
M STEERING EFFECTIVENESS ACROSS LAYERS AND POSITIONS
In this appendix, we include plots with more detailed information on the positions and layers that are most effective is steering experiments.
M.1
RHYME STEERING
Figure 26 compares the effectiveness of steering on the newline token for all models. Figures 27 30
report steering effectiveness for both the newline position and the immediately preceding token position for all models consideration.
Rhyme Family on Newline) by
Gemmazo
Denraak0
Fraction of Stoo se Nhyme Fin by (Seored Stecrdderteeee crroeetee
07
0.5
3.345 DIRU
0015
0019
200
3014
BOGA 1.20
200
2012
Dermnnt 27p
desmalt 278 Sooe
Cemmend sase
Cenmana Bace
decomall, 100 Dree
Gentoned 48 Baae
Germent 18 Blse
ME base
cener Ave
connr 1iB
Cornit MR Bace
Geetk d8 Bace
Genmaal
Uheeed Damak
Model
Figure 26: Steering effectivvnees for newline token position, all models.
1.00
1.00
last word newlme
125t word
0.50
0.50
0.00
0.00
34
22
10
11
13
14
Layer
Layer
Germmn2_eg
Germina Base
3.00
1.00
0.75
nootin
0.50
0.50
0.00
0.00
Layer
Layee
Gemma2? 278
Cammal 278 Base
0.75
0.75
0.25
0.25
Figure 27: Steering effectiveness by steering position and layer, Gemma2 models.
Lindsey et al. reported evidence for planning representations on the newline token at the end of the first line for Claude Haiku. In our experiments, steering the newline token was only effective in
36

---

Published as a conference paper at ICLR 2026
Gemmaa3
Cemman 18 Base
1.00
lest worl
0.73
0.75
0.50
0.50
0.00
0.00
Layer
Layer
Gemmad
Gemma3 48, Base
1.00
last word
0.79
0.79
neatine
0.50
0.50
1357 word
0.00
0.00
72029900
Layer
Layer
3010mm31126
Nemma3_32tt Base
1.00
1.00
last word
last word
nowlin
nostie
0.50
0.50
0.00
0.00
1ayer
Layer
Germaa3277 278
Gemma3_22 278, Base
1.00
1.00
lest word
last word newlic
nowin2
Score
0.50
0.50
0.00
0.00
Layer
Layer
Figure 28: Steering effectiveness by steering position and layer, Gemma3 models.
nowline
0.0
0.0
Layer
Layer
3.0
1.0
0.8
last word neximee
0.8
last word newiine
0.6
0.6
10
13
25
5
6
1213141314415
15
20
23
Layer
Layer
Figure 29: Steering effectiveness by steering position and layer, Llama3 models.
select models in some middle layers. For all models however, steering worked on the pre-newline (last word) token in earlier layers. Rhyme steering effectiveness for all layers and positions of all models is reported in figures 30
M.2 NOUN QUESTION ANSWERING STEERING
Noun QA steering effectiveness for all layers and positions of all models is reported in figures 31 34
37

---

Published as a conference paper at ICLR 2026
Qven3
Qwen3 8B Bose
0.75
0.75
0.25
0.25
Qwinz
Base
1.00
1.00
lock wardd
lnst word
newlie
newline
0.50
0.50
0.25
0.25
10
12
34
38
18
20
22
24
26
28
59
25 10
12
34
26
10
20
22
24
26
Layer
Layer
Owen2_33
1.00
neaine
0.50
0.00
Layer
Figure 30: Steering effectiveness by steering position and layer, Qwen models.
Steering Offectiveness Across Layers: Gemman
Steering Effectiveness Across Layers: Gemma3
Figure 31: Steering effect (fraction of rollouts containing the target noun) across layers and positions for steering. Neutral questions, Gemma models.
38

---

Published as a conference paper at ICLR 2026
Steering Effectiveness Across Layers: Liama3
3.30
0.3m
0.30
2,38 Bose
Ulera3 2,80, Dose
0.2
0.90
6.34
c.00
6.0-
0.0
c30
Steering Effectiveness Across Layers: Qwen3
Stutia
Sendio
3.5
C355
1.25
6.25
311
Owendd DO, 10uB
Owen33 340, Dose
0.2
322
0.2
Layyer
Layarr
Figure 32: Steering effect (fraction of rollouts containing the target noun) across layers and positions for steering. Neutral questions, Llama and Qwen models.
39

---

Published as a conference paper at ICLR 2026
Steering Layers
Gramal
Gramel. 277
30
20
202
Caradd
Canmal
275
23 222
Figure 33: Steering effect (fraction of rollouts containing the target noun) across layers and positions for steering. Suggestive questions, Gemma models.
40

---

Published as a conference paper at ICLR 2026
Steering Effectiveness Across Layers: Liama3
0.00
0::00
Ulere:.32 2, 30, Dase
Llamm3331 0, 2000,,000000000 Dase
0.35
5.075
1.80
Steadh Bile Siecovane
8.022
6.28
2,07
Steering Effectiveness Across Layers: Qwan3
Owen3
Ovee3
0.00K
$3000
0.177
5917%
83000
0000
8,0000
1.0%
8,000
0.00
Laper
Layer
Owenn3 30, Dase
QuennO J-48, Sano
0214
022
00.0
0.00
003
0.08
0.30
Lager
Layer
Figure 34: Steering effect (fraction of rollouts containing the target noun) across layers and positions for steering. Suggestive questions, Llama and Qwen models.
41


Reviews:
[{"review_id": "9s0Sr8QH15", "reviewer": "Reviewer_Z2sB", "summary": "This paper investigates the implicit planning capability in large language models (LLMs), using rhyming poetry generation as the object of study. The authors propose a set of quantitative, scalable metrics to evaluate various aspects of implicit planning, including successful forward planning and successful backward planning.\n\nThe paper introduces the \"mean activation difference steering\" (MADS) method, which intervenes in the model's hidden states during generation to change the generated rhyme family. The authors created a dataset of rhyming couplets covering 10 rhyme families and used it to conduct a comprehensive evaluation of 23 open-source models (Gemma 2, Gemma 3, Qwen 3, Llama 3.1/3.2) across different series and scales (1B to 32B parameters).", "questions": "- The proposed method appears similar to Classifier-Free Guidance (CFG) commonly used in diffusion models, but the paper does not mention this connection. Could the authors provide some discussion and analysis regarding the similarities or differences?\n\n- How many training samples are required to obtain a relatively stable steering vector? The paper uses 85 samples. What would the effect of the steering vector be with significantly fewer samples, for example, less than 10?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clear Motivation: The paper is sharply focused on studying the implicit planning capability of LLMs, providing clear research intent.\n\n- Simple and Effective Method: The paper proposes the mean activation difference steering method, which is simple and effective, serving as a replacement for more complex and computationally expensive techniques used in previous research.\n\n- Extensive Experimentation: The authors validate their method across a wide range of popular model families and various model scales, leading to comprehensive experimental results.", "weaknesses": "- Writing Needs Improvement:\n\n    - The paper's description of the activation manipulation position is somewhat confusing. Lines 157–159 mention that the manipulation target is the position of the newline character itself, while the immediately following formula on line 169 describes manipulation at the position preceding the newline character. This is confusing for the reader. Although the text immediately mentions that they experimented with both cases, I recommend the authors optimize the logical flow and presentation of this section.\n\n    - Regarding the formula on line 169, the paper writes $P_{\\text{RF}_2}^{\\text{Train}} - P_{\\text{RF}_1}^{\\text{Test}}$. However, if I understand the proposed method correctly, the latter term should also be from the training set, meaning it should be $P_{\\text{RF}_2}^{\\text{Train}} - P_{\\text{RF}_1}^{\\text{Train}}$.\n\n    - The figures in the paper are difficult to read due to small font sizes. It is recommended to adjust the typography in the visualizations.\n\n- Limited Generalizability: The paper's study of LLM planning behavior is concentrated on rhyming poetry generation, and the generated content length is relatively short. Only a very brief discussion of Q&A tasks is included in the appendix. This limits the generalizability of the paper’s proposed metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "37dh3AUobz", "reviewer": "Reviewer_ArCv", "summary": "This paper investigates the concept of \"implicit planning\" in LLMs through the specific case study of rhyming poetry generation. The authors propose a set of simple, scalable metrics to quantify both forward planning (preparing for a future rhyme) and backward planning (adjusting intermediate tokens to lead to that rhyme). Using mean activation difference steering, they demonstrate the ability to manipulate the \"planned\" rhyme family across a diverse set of 23 open-weight models. The results show that planning capabilities for rhyming are present even in 1B parameter models and generally improve with model scale and instruction tuning. The paper concludes with a brief mechanistic analysis of the \"rhyming circuit\" in Gemma2 9B and with identifying specific attention heads involved in the process.", "questions": "Could the authors comment on the role of tokenization in their findings? For rhyme families where the rhyming component is a single, common token (e.g., \"-ight\"), is the model truly engaging in \"planning\" in the same way it would for families where words are tokenized differently (e.g., words ending in \"-ake\" like \"make\", \"take\", \"shake\")? Have the authors analyzed if the performance of their metrics correlates with the token-level similarity of words within a rhyme family?\n\nThe circuit analysis points to specific heads in Gemma2 9B. Given the universal presence of the planning phenomenon across models, do the authors hypothesize that these are \"rhyming heads\" that are functionally homologous across different model architectures, or is it more likely that each model implements this capability with different, idiosyncratic circuits?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper's core strength is its departure from computationally intensive methods like \"cross-layer transcoders\" (I was not aware of these until this paper). The use of mean activation steering and straightforward metrics makes this work easily replicable and applicable across a vast array of models, as demonstrated by the authors' evaluation of 23 different models. There's a rich amount of steering literature to base this all on. \n\nThe study isn't limited to a single large model but instead provides a panoramic view of how rhyming and planning abilities evolve across different model families (Gemma, Llama, Qwen) and sizes (1B to 32B). This comparison between and discussion around base and instruction-tuned models is particularly insightful.\n\nThe authors don't rely on a single metric. The combination of generation-based metrics (Fraction of Correct Rhyme), probability-based metrics (KL divergence, Top-1 difference), and regeneration metrics provides a multi-faceted and convincing case for both forward and backward planning - and \"planning\" at the log probs level might be controversial \n\nThe analysis in Section 5.2, which pinpoints specific attention heads responsible for propagating the planning information, adds a layer of mechanistic credibility to the quantitative results and aligns with findings in related work.", "weaknesses": "The most significant weakness is the complete failure to discuss the role of tokenization. The model's ability to \"rhyme\" is profoundly influenced by its subword vocabulary!!!! For rhyme families like \"-ight\", words like \"light\", \"night\", and \"right\" likely share a common final token, making the task a simple exercise in token-level sequence completion rather than true phonological planning! The paper's entire analysis is conducted without ever acknowledging or controlling for this massive confounding variable. This oversight calls into question the interpretation of \"planning\" for certain rhyme families. I'm being very generous with my high scores in spite of this. \n\nThe literature review is troublingly incomplete, which seems to have led to the first weakness. The paper misses several key pieces of prior work. Most notably, Roush et al. (2022) in \"Most Language Models Can be Poets Too\" dedicated significant discussion to the very problem of tokenization and vocabulary size impacting poetic and rhyming ability. It is a foundational paper for this specific topic and its omission is a serious oversight. Additionally, other works on steering model behavior, like SMC Steering by Lew et al, could have been mentioned to provide a broader context for the intervention techniques used.\n\nRelated to the tokenization issue, the paper notes that larger models are better but doesn't explicitly discuss why. A key hypothesis, supported by Roush et al., is that larger models tend to have larger vocabularies (i.e vocab size in subseqeunt versions of GPT has grown well beyond its original 50K tokens), meaning more words (and rhyming suffixes) are represented by single tokens, simplifying the rhyming task. This connection is a critical piece of the puzzle that the authors fail to mention!\n\nI give a generous score given that I believe this paper in its current form has a glaring lack of discussion around Tokenization, however, I'm not convinced that this lack of discussion would invalidate the results and I think adding such a discussion is basically trivial, so I give a high score now - but if the authors do not commit to sufficiently cover tokenization in some way in the final manuscript I would be liable to reduce my score to weak accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ncvg2B4ehB", "reviewer": "Reviewer_rPUj", "summary": "This paper investigates whether LLMs exhibit implicit planning, meaning that they internally represent and use information about future outputs during text generation. Using rhyme generation as a controlled testbed, the authors introduce formal definitions and quantitative metrics of forward planning (encoding future goals before generation) and backward planning (conditioning ongoing generation on future goals). By manipulating hidden activations with mean activation steering, they show that altering representations at specific positions (e.g., line endings or newline tokens) systematically changes future rhyme families without modifying the input text, demonstrating controllable planing vectors. Experiments across various language models show that implicit planning behaviors are consistently observable and become more pronounced in larger and instruction-tuned models. The study further provides evidence that such planning mechanisms are structured and interpretable within model internals.", "questions": "What is the relationship between the forward planning and backward planning proposed in this paper?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors' proposed steering vector based on the average activation difference appears to be directly effective, reflecting to some extent the underlying correlation between model activations and rhyme families.", "weaknesses": "1. The paper confines all of its experiments to the rhyme generation task, a setting that *has already been widely examined* in prior research. This narrow focus limits the generalizability of the findings and does not provide sufficient evidence to confirm the existence of an implicit planning mechanism.\n2. The distinction between forward planning (the direct influence on the final output) and backward planning (indirect influence through the evolution of token probabilities) appears to describe two stages of the same representational process rather than two independent behaviors. The *lack of discussion on the connection* between these two aspects weakens the conceptual clarity of the work.\n3. The proposed method and evaluation metrics are *relatively primitive and overly direct*. Modifying activation values essentially alters the contextual representation itself, making it difficult to isolate the planning effect. The experiments would benefit from deeper analyses, including an examination of cross-layer activation propagation and the relationship between attention distributions and rhyme outcomes.\n4. The overall organization of the paper is confusing, and the figures are roughly presented and difficult to interpret. These presentation issues significantly reduce the clarity and accessibility of the work.\n\n[1] Jack Lindsey, etc,. On the biology of a large language model. Transformer Circuits Thread, 2025.\n[2] He, Jingkai, et al. History rhymes: Accelerating llm reinforcement learning with rhymerl. arXiv preprint arXiv:2508.18588, 2025.\n[3] Jobanputra, Vedant Dhaval, et al. LLM-aided Evolutionary Algorithms for Haiku Generation. Proceedings of the Genetic and Evolutionary Computation Conference Companion. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)