Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
Authors: Haozhe Wang
Qixin Xu
Che Liu
Junhong Wu
Fangzhen Lin
Wenhu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: hierarchical reasoning
LLM
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaque. Our analysis reveals that puzzling phenomena like ``aha moments", ``length-scaling'' and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy, akin to the separation of high-level strategic planning from low-level procedural execution in human cognition. We uncover a compelling two-phase dynamic: initially, a model is constrained by procedural correctness and must improve its low-level skills. The learning bottleneck then decisively shifts, with performance gains being driven by the exploration and mastery of high-level strategic planning. This insight exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically and dilute the learning signal across all tokens. To address this, we propose Hierarchy-Aware Credit Assignment (HICRA), an algorithm that concentrates optimization efforts on high-impact planning tokens. Our extensive experiments validate that HICRA significantly outperforms strong baselines, and offer deep insights into how reasoning advances through the lens of strategic exploration.

Published as a conference paper at ICLR 2026
EMERGENT HIERARCHICAL REASONING IN LLMs THROUGH REINFORCEMENT LEARNING
Haozhe Wang** Qixin Xu Che Liu Junhong Wu", Fangzzen Lin WWnnhu Chen
*oogg Kong Univerisity of Science and Technology, University of Waterloo
"M-A-P, Tsinghua Univerisity, O Imperial College London, UUAAS
jjasper whzeout com, wenhuchenn@uwaterlco.coom ca]
https://tiger: ai--errr-a-lab-- hta.alaa..ggitti... github. http///eeeeaaccaaaeeeeeeaeeeeeee
ABSTRACT
Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaque. Our analysis reveals that puzzling phenomena like "aha moments" "length-scaling disparate occurrences but hallmarks of emergent reasoning hierarchy, akin to the separation of high-level strategic planning from low-level procedural execution human cognition. We uncover dynamic evolution where the learning bottleneck shifts: initially, the process is dominated by procedural consolidation and must improve its low-level skills. The learning bottleneck then decisively shifts, with performance gains being driven by the exploration and mastery of high-level strategic planning. This insight exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically and dilute the learning signal across all tokens. To address this, we propose Hierarchy- Aware Credit Assignment (HICRA), algorithm that concentrates optimization efforts on high-impact planning tokens. Our extensive experiments validate that HICRA significantly outperforms strong baselines, and offer deep insights into how reasoning advances through the lens of strategic exploration.
and entropy dynamics are not
Human-1 Hierarchical Reasoning
Hierarchical Reasoning through Training
piskler
Brand/ilccrncc
Teloe hersabee Tales
Semaniic Dhemit Plamiig Tekes
undastard
Abemataery
Corelldadd
Labie tua
barlowting
rot wer
Terk Avearesy
Sargarece Length
Wat
The find amoier
14
Hilh-iwal Hundin
Lie-leul Doration
Diet Reauen
Puraae Procoleed Rehabllty
Duplors Planeing
Figure 1: (Left) LLM reasoning mirrors a human-like hierarchical reasoning: high-level strategic planning and low-level procedural executions. (Right) Hierarchical reasoning emerges during RL training via two-phase dynamic. Phase consolidates low-level skills, marked by token token-entropy drop execution tokens. The learning frontier then shifts to Phase @, where the model explores and masters high-level planning, marked by increased semantic diversity, sustained reasoning enhancement and length scaling.
INTRODUCTION
Reinforcement Learning (RL) has become instrumental in advancing the complex reasoning capabili- ties of Large Language Models (LLMs) across diverse domains (Ouyang et al., 2022; Jaech et al., 2024; Yang et al., 2024; Guo et al., 2025; Team et al., 2025). However, this empirical success is accompanied by significant gap in our understanding of the underlying learning dynamics. The training process often yields phenomena that as effective they are poorly understood: models

---

Published as a conference paper at ICLR 2026
experience sudden 'aha moments', where they seemingly acquire new emergent skills (Guo et al., 2025); they exhibit 'length-scaling effects, where reasoning performance improves with longer, more detailed outputs (Guo et al., 2025; Team et al., 2025); and they display complex dynamics in token-level entropy (Yu et al., 2025; Cui et al., 2025). This gap motivates a fundamental question:
What unlocks enhanced reasoning in LLMs during RL,, and how should leverage this understanding to design more principled and efficient RL algorithms?
Our investigation is guided by key insight: RL does not train models de novo. It fine-tunes base models already imbued with priors from pre-training on vast corpora of human-written solutions. These solutions inherently encodes the hierarchical structure of human reasoning efficient cogmitive strategy evolved under biological constraints. This prompts us to ask: does RL unlock advanced reasoning by (re-Jdiscovering this hierarchical structure as a promising pathway for solving math problems?
highly
To test this hypothesis, we analyze the RL training process through the lens of hierarchical reasoning. Drawing parallel to the cognitive architecture of the human brain (Fig. 1), which separates high- level, deliberate strategic planning from the rapid execution of learned procedures (Murray et al., 2014; Zeraati et al., 2023; Huntenburg et al., 2018), 8), we propose decomposition of model-generated tokens into two functional hierarchy:
High-level Planning Tokens: The high-level strategic moves that orchestrate the reasoning process. These tokens manifest as logical maneuvers, including deduction (e.g., "we can use the fact that"", branching (e.g., "let's try a different approach"), and backtracing (e.g.. "but the problem mentions that").
Low-level Execution Tokens:
The operational building blocks of solution. These comprise concrete, low-level steps such as arithmetic calculations, variable substitutions, and the direct application of known formulas.
Our analysis across eight text-only and vision-language models confirms this hypothesis, revealing a consistently two-phase dynamic that explains the emergence of this reasoning hierarchy in LMs. We find the optimization pressure of RL is not static; instead, its learning frontier shifts. Initially, the process is constrained by procedural correctness. single calculation error can invalidate an entire solution, creating a powerful learning signal that compels the model to first master low-level execution tokens. Once proficiency in these foundational skills is achieved, the learning bottleneck shifts to strategic planning. We find that these phases are not mutually exclusive; procedural refinement continues throughout training, but the primary driver of marginal performance gains shifts to strategic planning exploring and mastering the use of planning tokens is what unlocks significant and sustained improvements in reasoning ability.
This emergent two-phase mechanism provides unifying framework for the puzzling phenomena observed in RL training. It explains "aha moments" as the discovery and internalization of high-level strategic reasoning strategies, such self-refiections. also accounts for the "length-scaling" effect, as employing more sophisticated strategies involving thorough planning and logical backtracing naturally elongates the reasoning trace with structured, strategic deliberation. Notably, it provides a unified perspective to understand the complex token entropy dynamics across different models, through the lens of high-impact planning tokens and gradually confident execution tokens.
This discovery that the learning frontiers dynamically shifts to strategic planning - is more than an academic curiosity; it provides clear blueprint for more effective RL algorithm. the primary driver for advanced reasoning is the mastery of high-level strategic planning, then current agnostic credit assignment methods used in the prevailing GRPO (Guo al., 2025) and its variants (Yu et al., 2025; Liu et al., 2025b; Wang et al., 2025c) are fundamentally inefficient, as they dilute optimization pressure across all tokens rather than concentrating it where it matters most.
Based on this insight, propose Hierarchy-awe rithm designed to focus optimization pressure directly on this emergent strategic bottleneck. By selectively amplifying the learning signal for planning tokens, HICRA accelerates the exploration and reinforcement of effective high-level reasoning, leading to significant performance gains as demonstrated in our experiments.
Credit Assignment (HICRA), novel algo-
Contributions. In this work, we advance the understanding of how LLMs learn to reason via RL. We demonstrate that the learning process is not monolithic but emergent two-phase learning dynamic
2

---

Published as a conference paper at ICLR 2026
driven by the hierarchical priors in base models and solution structure of the reasoning tasks. This insight reveals that the true bottleneck for advanced reasoning is the mastery of high-level strategic planning, which current agnostic credit assignment methods neglect. To bridge this gap, we pioneer with an original and simple solution, HICRA. Through extensive experiments across LLMs and VLMs, we not only validate the effectiveness of HICRA, but also offer deep insights into how HICRA works through the lens of strategic exploration.
2 THE EMERGENT REASONING HIERARCHY
Guiding Insight: The pre-training priors and the inherent structure of reasoning tasks create a strong inductive bias. For the task math problem-solving hierarchical reasoning proves an efficient and prominent strategy, which is discovered through RL training and unlocks advanced reasoning.
2.1 A FUNCTIONAL PROXY FOR THE REASONING HIERARCHY
To analyze this reasoning hierarchy, must first distinguish high-level strategic planning from low-level procedural execution within the model's s generated tokens. This is challenging because a token's function is defined by its context, not its intrinsic meaning.
To address this gap, we draw inspiration from human cognition. When a person reasons through a problem, easily identify their strategic thinking by its function. A phrase like, "Let's try different approach," functions as a high-level strategic mancuver that guides the problem-solving direction. In contrast, a phrase like, "so we add 5 both sides," " is low-level procedural step. Inspired by this functional distinction, we introduce Strategic Grams as a functional proxy to circumvent the difficulty of formally defining what is a "planning token".
Strategic Grams (SGs) are defined as n-grams that function as a single semantic unit to guide the logical flow. We use n-grams because they capture the phrasal nature of strategic language (e.g.. "let's consider the case") which is lost at the single-token level. These SGs facilitate three main types of logical moves: (a) deduction, (b) branching, and (c) backtracing, as we show in an illustrative example in the appendix.
A token is classified as a strategic planning token it is part of a Strategic Gram in the current context. All other tokens are classified as procedural execution tokens.
For simplicity, we use the term "execution tokens" to refer to all non-planning tokens. We note this is slight simplification, this category encompasses not only concrete calculations but also formatting and other procedural language.
A key challenge is identifying the set of SGs in principled and reproducible manner. Manual annotation or reliance on proprietary models would introduce subjectivity and hinder reproducibility. We therefore propose an automated, data-driven pipeline based on key insight: SGs function as the reusable scaffolding of a reasoning process (Fig. 6). This function imparts a distinct statistical signature: SGs should appear frequently across a wide range of different solutions but be used sparingly within any single solution. However, a significant challenge is the linguistic diversity of strategic language, where a single strategic intent can be expressed through numerous phrases.
Our pipeline is designed to overcome these challenges by first grouping semantically equivalent n-grams and then identifying which consolidated concepts exhibit the statistical signature of strategic planning. We place the detailed construction procedure to the appendix due to page limits.
This automated procedure is designed to yield high- seseisreeiioon functional proxy for strategic planning. not an exhaustive lexicon of all possible SGs. We set reasonable hyper-parameters for identifying SGs, and we contend that the resulting SG collection sufficiently representative to reveal the core learning dynamics. To validate this claim, we conduct a sensitivity analysis by randomly removing 30% of the identified SGs and re-running our main analysis (see Appendix). The resulting learning dynamic curves remain qualitatively identical, demonstrating the robustness of our methodology and the findings derived from it. To validate that our automated pipeline captures genuine semantic intent rather than statistical noise, conducted human annotation study. Results confirm that 86% of our identified SGs were classified by humans as functioning to "guide flow or propose plans," " compared to only 12% for otherwise. (See Appendix for full study details).
3

---

Published as a conference paper at ICLR 2026
2.2 EMERGENCE OF THE REASONING HIERARCHY
Building on our functional proxy for reasoning, examine the learning dynamics of RL for LLM reasoning and finds an intriguing parallel with human-like hierarchical reasoning. Our empirical analysis conducted consistently across different model families, Qwen2.5-7B (Yang al., 2024), Qwen3-4B ( (Yang et al., 2025), Llama-3. 1-88 (Grattafiori et al., 2024), Qwen2.5-VL-78 (Bai et al., 2025), MiMO-VL-7B (Xiaomi, 2025) - reveals that enhanced reasoning is not monolithie process, but driven by an evolution of the learning frontiers.
The learning process exhibit two overlapping phases: it often begins with a rapid consolidation of procedural reliability, conducive to the widespread low-level tokens. This is followed by a sustained period where the greatest potential for improvement shifts to the exploration of high-level strategic reasoning, which serves as the true engine of advanced performance.
2.2.1 FORGING RELIABLE LOW-LEVEL. SKILLS
The initial phase of RI training is dedicated mastering the basics. The model must first build reliable engine for low-level skills, e.g., formatting, performing calculations and other procedural steps. To observe this, we track two key metrics the execution tokens:
Relative Perplexity:
Perplexity, the exponentiated average negative log- likeliid,ooodd measures model surprise. A lower value sigmifies higher confidence. We normalize the perplexity by its initial value to compare the rates of change in planning tokens and execution tokens.
Token-Level Entropy: The Shannon entropy of the policy's next-token distribution, H(x(-(Z<))) measures its uncertainty. High entropy signals active exploration over the vocabu- lary at the next-token, while low entropy suggests confident exploitation.
The evidence for this phase is shown in the first two columns of Figure 2, marked with +. The Relative Perplexity of execution tokens (grey curves) plummets the early stages of training before flattening (column 1). This shows the model rapidly becomes confidently correct in its procedural steps. This is reinforced by the Token Entropy graph (column 2), where entropy for execution tokens consistently and significantly lower than for planning tokens. The model is not just confident; actively reduces exploration of procedural alternatives to converge on reliable operations. This rapid mastery of the basics the first learning frontier to be solved.
Takeaway 1. Procedural consolidation is often marked by sharp decrease in the perplexity and token entropy of execution tokens. The model quickly builds a reliable "toolbox" of procedural skills, allowing the primary frontier for performance improvement shift to high-level strategy.
Notably, we find that this phase of low-level skill consolidation might be absent or shot in models with stronger capacity, as evident in MiMO-VL- VL-Instruct and Qwen-4B-Inste Insttrcct.. This also supports the argument that the primary driver of RL is indeed the exploration of strategic planning. We refer the reader to check the full analysis of training dynamics across eight models in the appendix.
2.2.2 STEERING THE SKILLS WITH STRATEGIC PLANNING
Once the model becomes procedurally reliable, its performance gains are primarily driven by its ability explore and deploy diverse set of high-level strategies. To track this shift, we analyze the planning tokens using two key metrics. We compute the Semantic Entropy of strategic grams the Shannon Entropy of the frequency distribution of strategic grams to quantify the diversity of the model's 's high-level strategic plans (illustrated in Fig. 10). To isolate procedural variety, we compute the conditional entropy of subsequent procedural n-grams given a preceding strategic gram, This second metric shows how varied is the subsequent procedural steps for a preceding strategic move.
The third column of Figure 2 provides clear evidence of this strategic exploration phase. The semantic entropy of strategic grams (red line, marked with @) shows a distinct and steady increase. This indicates that the model is not converging on a single optimal strategy but is instead actively expanding its repertoire of strategic plans. This observation is critical: mastery in reasoning, in this context, is achieved by developing a rich and varied strategic playbook, which contrasts sharply with the sharp decrease in teken-level entropy seen during the initial procedural consolidation phase.
4

---

Published as a conference paper at ICLR 2026
Model Surpetic/ Training Effiets
Tekeoded Eatropy Farning Execation
Scmeantic Entroge eieilt Other (Cendtiona
Loagth VET Accerac)
2.1)
R100 Popteeyy (PPL)
Exscution Talen No.
Largt
Touberr of THARTS
Oneed
48
12
12
Trsiiid 23
32
12
2.
25
13
Planning Meartte 13
Painiing Tolen
535
Strategi Bramo
Monee sempotty IPF
125
0303
Drea Change
Quen2.5
7B
C235
352
25
12
25
23
220
ng
Narta Pepportly (993
ca-
(Canditiona
Stoury Change
CED
12 Tic
25%
Figure 2: We track the training Dynamies of representative model families. The curves reveal two-phast dynamics. Seen from the first two columns, the model has an initial focus on procedural consolidation, marked by sharp decrease in model perplexity (greater confidence) and token entropy (more certain) of execution tokens. This follows shift to exploring strategic planning, evident from the third column. The diversity of strategic plans (semantic entropy) steadily increases on Qwen models takes turn to increase on Llama, correlating with consistently improved accuracy and longer reasoning chains (fourth column).
This strategic diversification provides the most direct evidence for our thesis: the model isn't just getting better at executing plans; it's getting better at planning itself. While the model explores new high-level strategic moves, the conditional entropy of procedural grams (grey line) remains stable. This suggests that once a procedural skill like arithmetic is mastered, there is little incentive to find diverse ways to perform it. The improved reasoning performance comes from discovering new ways to combine these established skills, which is the core function of strategic planning.
Crucially, this expansion of the strategic playbook directly correlates with tangible performance gains. The fourth column shows that the rise in strategic diversity is accompanied by parallel increase in the length of reasoning chains and a sustained boost in overall accuracy. This demonstrates that after procedural skills consolidated, the development of strategic planning becomes the primary bottleneck and driver for advanced reasoning performance.
Takeaway 2. Once payoff from procedural consolidation diminishes, performance gains are driven primarily by exploring high-level strategies. This is marked by the increasing semantic diversity of strategic grams, which correlate with sustained reasoning enhancement, length scaling, and represents the key learning frontier.
Explaining Puzzling Phenomena. This emergent reasoning hierarchy provides unified explanation for previously observed behaviors.
"Aha moments" are the behavioral signature of the model discovering, mastering, and reinforc- ing a new, powerful strategy or set of strategic constructs.
"Length-scaling" is highly consistent with increase in strategic diversity. As Figure 2 shows, the rise in semantic entropy of planning tokens is strongly correlated with an increase in average sequence length. More sophisticated strategies involving planning, case analysis, and self- reflections are mediated by planning tokens and naturally produce longer, more successful reasoning traces.
Semantic Entropy: A Good Compass for Exploration. The trends in Figure 2 also highlight a critical flaw in using aggregate token-level entropy (column 2) to track exploration.
5

---

Published as a conference paper at ICLR 2026
The aggregate token-level entropy is dominated by the vast majority of low-level execution tokens. As the model become confident in procedural steps or low-level skills, the entropy of these tokens naturally decreases, pulling the global average down.
Takeaway 3.
Unluckily, the decrease in token-level entropy sometimes mislead practitioners into the conception of declined exploration. This incorrect, however, as it contradicts the fact of increasing exploration in strategic plans (semantic entropy of planning tokens) and the improving reasoning performance. In the appendix, we compare with token entropy and Pass @ K. Our results show that semantic entropy avoids the flaws in token entropy by directly measuring diversity at the semantic level of meaningful strategic units. Its trend accurately reflects the expansion of the model's strategic playbook, making it more reliable diagnostic tool for tracking genuine exploration and predicting sustained performance improvements. It also complements Pass @ K metric with further benefits, We refer interested readers the appendix for full analysis of training dynamics across eight LLM and VLMs and the deeper insights into RL training and exploration.
3 HICRA: HIERARCHY-AWARE CREDIT ASSIGNMENT
Our empirical analysis reveals a fundamental insight: RL improves reasoning by rediscovering and operationalizing the strategic layer of reasoning inherited from the model's pre-training priors. The learning process is characterized by dynamic shift in its learning frontiers. Initially, the model is constrained by procedural correctness, as it masters these foundational skills, the frontier for performance improvement shifts to the exploration and mastery of high-level strategic planning.
This observation exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically across all tokens. Such methods fail to concentrate learning where matters most - on the emergent strategic bottleneck. To address this, we propose an algorithm designed to focus the model's learning capacity the sparse, high-impac planning tokens that orchestrate a successful reasoning trace.
Formulation. We introduce Hierarchy-Aware Credit Assignment (HICRA), algorithm that builds upon the GRPO framework to allocate credit based on the reasoning hierarchy. In GRPO, given a query q from dataset D, the policy To generates set of G output trajectories foi...., .g). The advantage for a token oi at timestep in trajectory o, is the group- group-normalized reward:
1 G
Å R(g. o.)
Σ R(q.o;)
HICRA, pronounced "high-krah", modifies this advantage to prioritize planning tokens. Let S; be the set of indices corresponding to planning tokens within trajectory o,, identified using the method in Section 2.1. We define the HICRA advantage as:
[ a A
i E S; |Aael
HICRA
if & S
where a E 10 is a hyperparameter controlling the amplification intensity (we use a = 0.2 in experiments). This formulation creates clear learning hierarchy: for successful trajectories (A,, > 0), it amplifies the credits for planning tokens, while for unsuccessful ones (Å,, < 0), it dampens their penalty. The resulting RL objective and its policy gradient (simplified without PPO clipping) are:
[ Eq-DD,
AUICRA] A
VJ(0)
F.
AIICRA Vllog To (Ou,ig o1 A,t
By translating the amplified advantage into a stronger policy gradient, HICRA directly focuses the model's 's optimization on the strategic elements of its reasoning process. Unlike methods that reward statistical uncertainty (entropy) indiscriminately. HICRA targets the semantic function of planning. As we show in Section 4 and the Appendix, rewarding high entropy is systematically different from rewarding high-level planning.
Connection to Strategic Exploration. The core mechanism of HICRA engineers more effective exploration by reshaping the policy update's target distribution. A standard policy gradient (Williams,

---

Published as a conference paper at ICLR 2026
2004) update nudges the policy Telia toward an implicit target distribution n" defined by the advantage function described as follows (the derivation included in the appendix):
ा (os,el 01,<() x Taas (or, q. 01,c1) exp(Ax.))
Typically, this update pressure is applied isotropically, affecting all token types uniformly. HICRA breaks this symmetry. By using the modified advantage ÁHICRA, it creates new target distribution, THICRA that is anisotropically stretched toward the strategic dimensions of the action space. This new target distribution places significantly greater probability mass on planning tokens (through the term exp(A..:)), particularly those within high-reward trajectories.
This anisotropic reshaping fosters a potent virtuous feedback loop: (a) the policy is incentivized explore the subspace of strategic plans more thoroughly; (b) this leads the faster discovery of effective reasoning patterns; and (c) when these strategies yield high rewards, the amplified advantage ensures they are strongly reinforced, cementing the model's planning capabilities far more efficiently. We also validate the effects of HICRA in exploration through experiments in Section 4.
4 EXPERIMENTS
Models and Datasets. Our experiments use open-source models including Qwen2.5-7B (Yang al., 2024), Qwen3-4B (Yang et al., 2025), L.-mm-3.1-88 (Grattafiori et al., 2024), and VLMM like Qwen2.5-VL-7b (Yang et al., 2024) and MiMO-VL-7B (Xiaomi, 2025), covering both base and instruction-tunet variants. We train on established reasoning datasets DAPO (Yu et al., 2025), DeepScaleR (Luo et al., 2025) and ViRL39K (Wang et al., 2025c) for VLMs.
Benchmarks and Baselines. We evaluate on a suite of challenging text-only (e.g., AIME24, AIME25 (Mathematical Association of America, 2024), Math500 (Lightman al., 2023), AMC23, Minerva (Lekkowycz et al., 2022), and Olympiad (He et al., 2024)) and multimodal (e.g.. Math- Vista (Lu et al., 2023), Verse (Zhang et al., 2024), Math Vision (Wang et al., 2024), FMMA (Hao et al., 2025)) benchmarks. We adopt the evaluation protocols of Deepseek R1, using Pass I with random samplings. We compare HICRA against three primary baselines: the Base model (before RL). the widely adopted GRPO baseline with clip-higher (Yu et al., 2025) by default, Entropy Reg- ularization: GRPO with an additional regularization loss on token-level entropy (Cheng et al., 2025), High-Entropy Advantage: GRPO with advantage modulation on high-entropy tokens, following Cheng et al. and Wang et al., and Placebo HICRA: rewarding random n-grams.. A comprehensive description of our evaluation protocol, training implementation, and additional model-specific details can be found in the Appendix.
4.1
MAIN RESULTS
Our primary results, summarized in Table I and Table 3 in the appendix, show that HICRA consis- tently and outperforms both the GRPO baselines across text-only models and vision-language models various benchmarks. On the strongest base model, Qwen3-4B-Innsrrue HICRA's gains demonstrate that even on highly capable models, selectively amplifying the learning signal for strate- gic reasoning yields substantial improvements. This trend holds for non-instruct-tumeds models as well, providing strong empirical evidence for our central claim: by identifying and focusing on the emergent strategic bottleneck. HICRA accelerates the development of advanced reasoning abilities more efficiently than agnostic methods.
4.2 ANALYSIS OF RL'S IMPACT ON REASONING
We conduct series of analyses to dissect how RL improves reasoning, First, we have linked strategic planning to reasoning through analyses of the training dynamics in Section 2.2; Second, we verify the key effects of RL by showing the frequeney dynamics of different errors throughout training (Section 4.2.1): we then justify the effectiveness of HICRA in exploration by comparing with standard eetroiy-reguularized baselines. Further insights and analyses included in the appendix.
7

---

question. Finally, Published as conference paper at ICLR 2026
Table I: Comparison of HICRA, GRPO, and Base models across various mathematical reasoning benchmarks, HICRA consistently outperforms all baselines actoss different base models, demonstrat- ing the effectiveness of focusing optimization on strategic planning tokens.
Model
AIME24
AIME25
Math500
A AMC23
Minerva
Olympiad
Owen3-4B-Instrutt-255
Base
63.4
47.7
86.7
45.2
72.4
GRPO
68.5
60.0
96.2
88.5
50.0
72.7
HICRA
73.1
65.1
97.2
90.2
50.7
72.0
A (HICRA GRPO)
+5.4
+5.1
+1.0
+1.7
+0.7
-0.7
Qwen3-BB-daaptiv (No-Think)
Base
21.3
18.1
84.4
60.5
40.4
49,9
GRPO
63.1
58.8
76.8
45.2
55.6
HICRA
65.9
62.1
82.5
46.3
59.7
A (HICRA GRPO)
+2.8
43.3
+0.2
+5.7
+1.1
+4.1
Qwen3-4B-Bar
Base
9.4
5.3
63.8
38.9
28.3
30.7
GRPO
24.9
23.8
83.0
51.2
38.9
45.8
HICRA
31.0
27.6
89.0
54.0
42.5
48.1
A (HICRA GRPO)
+6.1
+3.8
+6.0
+2.8
+3.6
+2.3
-Jama-31--B-Insttree
Base
4.2
0.6
50.2
17.1
20.9
13.7
GRPO
8.9
0.5
53.0
25.0
27.2
20.3
HICRA
8.3
0.8
54.8
27.1
25.8
21.2
A (HICRA GRPO)
-0.6
+0.3
+1.8
+2.1
-1.4
+0.9
Qwen2.5-7B-Bal
Base
3.5
1.7
55.6
46.9
30.9
25.9
GRPO et 2025)
16.3
11.4
46.7
36.8
41.9
ORZ (Hu et al., 2025)
17.0
13.1
80.6
52.4
39.7
44.9
SimpleRL (Zeng et al., 2025)
16.7
3.3
78.2
42.8
34.9
38.2
High-Entropy Advantage
15.8
11.4
78.9
51.8
33.8
40.8
Placebo HICRA
14.6
9.3
77.4
51.8
34.8
41.2
Entropy Regularization
16.0
9.3
77.4
50.3
33.1
40.6
HICRA
18.8
14.8
80.2
55.1
38.6
45,9
A (HICRA GRPO)
+2.5
+3.4
42.6
+8.4
+1.8
44.0
Swen2.5-77--Ba.
Qwen3-4B-Base
Qwen3-4B-Insth
800
Ranning Strategy
Eror Count
Error Count
800
Others
250
Planning Strategy
Planning & Stretegy
200
400
100
150
50
200
80
160
240
320
400
100
25
50
75
100
125
150
15
30
45
60
75
90
Tra ininnng Step
Figure 3: Training Dynamics of Error Types. Across all models, the number of Plamning SStrategy emros (red) decreases more significantly than other procedural errors (gray). indicating that RL's primary benefit comes from correcting high-level strategic faults.
4.2.1 MASTERY OF STRATEGIC PLANNING UNLOCKS IMPROVED REASONING DURING RL
To understand where RL applies the most leverage, we analyzed the evolution error types in failed rollouts. We first manually reviewed failures and nominated four distinct error causes. GPT-4o was then prompted to classify each failure into one of these causes via a multiple- parsed these classifications into two broader categories: "Planning & Strategy" (e.g.. flawed logic, incorrect high-level plan) and "Others" (e.g., calculation mistakes, fact-retrieval errors). The prompt used is included in the appendix.
8

---

Published as a conference paper at ICLR 2026
Figure 3 reveals consistent pattern: the primary benefit of RL stems from fixing high-level strategic faults. Across all models, the reduction in strategic errors is more pronounced than the reduction in other errors. This pattern is especially illuminating for Qwen2.5-7B -B--ase,, where non non-planning emrors does not decrease. We conjecture that while the model may be improving its procedural reliability, these low-level enhancements do not translate to correct answers because the high-level strategy remains the limiting factor. A perfectly executed incorrect plan will still result in failure.
This evidence strongly supports our claim that the strategic bottleneck is the key to unlocking advanced reasoning. RL preferentially corrects these high-level faults over low-level execution mistakes, as improving strategic planning provides the most direct path solving complex problems.
4.2.2 JUSTIFYING HICRA: TARGETED Vs. INDISCRIMINATE EXPLORATION
Our findings suggest that performance gains are driven by mastering high-level strategic plan- ning, which motivates HICRA's design to con- centrate learning on planning tokens. As shown in Figure 4, HICRA's success is linked to its abil- ity to sustain higher level of semantic entropy than GRPO. This heightened diversity in high- level strategies directly correlates with stronger and more stable validation accuracy, confirming that focused strategic exploration a primary driver of reasoning improvements.
Semantic Entropy
Validation Accuracy
GRPO
HICM
Swreyy
0.40
0.25
GRPO
HARR
0.20
100
203
302
460
150
250
300
Figure 4: HICRA improves GRPO Clip-Higher via more diverse stralegic exploration.
Semantk Entropy
Validation Accuracy
Passg8 (Training)
To Entropy
GAPO
0.353
14
EntrpyReg
HKRA
1
0.50
Lo
7.0
0:253
GAPO
0:35
0.6
6.7
6.5
0:225
MERA
E10lsyyog
0.45
E10000y9gg
0.-
Taning 5983
Tarring Stop
Pariin Vap
Tnaning 50mp
Figure 5: HICRA vs. Entropy Regularization on Qwen2.5-7B-Bass While entropy regularization increases token-level entropy, it fails to consistently improve accuracy and leads to uncontrolled length scaling. In contrast, HICRA boosts semantic entropy, which strongly correlates with validation accuracy, demonstrating the superiority of targeted strategic exploration.
To further validate this, we compared HICRA against an entropy-regularized baseline. This baseline adds (upon GRPO) an entropy regularization loss applied to all tokens uniformly. The results in Figure show that promoting token-level entropy for sampling diverse tokens is counterproductive.
The entropy regularization baseline successfully increases Token Entropy, but this fails to translate into performance gains; its Validation Accuracy stagnates and is the lowest of the three methods. This is because indiscriminately promoting token-level diversity only encourages non-productive verbosity the vast majority of low-level tokens.
In contrast, HICRA achieves a significantly higher Semantic Entropy, targeted boost in the diversity of strategic plans that strongly correlates with its superior validation accuracy. This demonstrates that the key to enhanced reasoning is not just explore, but to focus exploration on the strategic portion of the action space.
To validate the importance of credit assignment on the semantic level, we compare with the High- Entropy Advantage baseline, which targets high-entropy tokens proxy for exploration. The baseline yields competitive but inferior results compared to HICRA. As our comparison between high-entropy tokens and identified planning tokens in D.2.2 and Figure 13 shows, only 10% of high-entropy tokens serve a valid planning function. HICRA significantly outperforms entropy- based methods because it targets the semantic function of planning, rather than just statistical uncertainty.

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Reinforcement Learning for LLM Reasoning. The application of Reinforcement Learning (RL) has been pivotal in enhancing the complex reasoning abilities of Large Language Models (LLMs), Seminal work by Ouyang et al. demonstrated the effectiveness of learning from human feedback to align models with user instructions. More recently, algorithms like Group Reward Policy Optimization (Guo et al., 2025) have been developed to specifically incentivize reasoning capabilities in LLMss, VLMM, Agents (Liiu et al., 2025b; Yu et al., 2025; Team et al., 2025; Liu et al., 2025a; Wang et al., 2025c;;; Su et al., 2025; Dai et al., 2025; Zheng et al., 2025; Kang et al., 2025), leading to significant performance gains on downstream performance. While these methods have proven empirically successful, they typically apply optimization pressure agnostically across all generated tokens, without distinguishing between different functional roles within the reasoning process. Our work builds on this foundation but introduces more targeted approach by focusing the emergent reasoning hierarchy.
Analysis of RL Dynamics and Exploration in LLMs. A growing body of research seeks to understand the complex learning dynamics that occur during the RI. fine-tuning of LLMs. Several studies have investigated the role of token-level entropy, observing intricate patterns and its connection to model exploration and uncertainty (Cui et al., 2025; Chen et al., 2025). Concurrently, phenomena such as sudden "aha moments" and performance improvements from longer outputs ("length-scaling") have been noted as characteristic but poorly understood outcomes of RL training (Guo et al., 2025; Liu et al., 2025b). Our paper provides a unifying framework, interpreting these phenomena as evidence of a shift from procedural learning to strategic planning.
Furthermore, recent work has identified high-entropy "fork tokens" as potential proxies for critical decision points in reasoning (Wang et al., 2025d). Our work distinguishes itself by defining planning tokens based on their semantic function. We also validate the limitation of identifying crucial tokens solely based on entropy.
apploratiin-EEploitatati trade-off has become a long-standing research problem in classical RL literature. Among the vast literature, Entropy Regularization Maximun-Entropy RL (Levine, 2018; Hаarnoja et al., 2017: Wang et al., 2023) is a standard technique to encourage exploration that can be seamlessly integrated with LLM RL training.
Hierarchical Reasoning and Cognition. The concept of hierarchical processing is a cornerstone of cognitive neuroscience, which posits that the human brain separates high-level, abstract planning from lowelevel motor or procedural execution (Huntenburg et al., 2018; Murray et al., 2014; Zeraati al., 2023; Zhu et al., 2025; Xiong al., 2025). HRM (Wang al., 2025a) is inspired this cognitive architecture to design a specific neural architecture for hierarchical reasoning. Concurrently, this cognitive model provides a compelling parallel the functional hierarchy we identify in RL-tuned LLMs, proposing that LLMs similarly develop functional separation between strategic planning and procedural execution. Similar our work. MT-Core (Pan et al.. 2025) decomposed agent's 's behavior into coarse- -griieed strategic policy formulation and low-level execution, focusing on knowledge transfer in a continual learning setting. contrast, our work concentrates on the emergent two-phase learning dynamics through RL and hierarchy-awar credit assignment for advanced LLM reasoning.
CONCLUSIONS
Our work establishes that reinforcement learning uncovers emergent functional reasoning hierarchy in language models, demonstrating the critical performance bottleneck shifting from procedural skill strategic exploration. This insight leads our approach, HICRA, which demonstrates that specialized credit assignment targeting this strategic bottleneck yields more effective training. Extensive experiments validate the effectiveness of HICRA and offer deep insights into advanced reasoning through strategic exploration. Our work opens several future research directions. First, it suggests a paradigm shift away from treating all tokens equally and prompts a rethinking of the action space away from individual tokens toward semantic, strategic units. Second, it calls for developing process-oriented approaches capable of valuing correct strategic choice even if the final answer is flawed. Finally, the likely universality of this reasoning hierarchy in complex reasoning tasks suggests that applying these principles to domains like code generation and agentic tool-use is valuable path forward.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors of this paper have read and adhere to the ICLR Code of Ethics. This work centers on the analysis of emergent hierarchical reasoning in Large Language Models (LLMs) during reinforcement learning and the development of novel algorithm, HICRA, to improve their reasoning capabilities. Our research does not involve the use of human subjects or the collection of personally identifiable or sensitive data. The models, datasets, and benchmarks leveraged in our experiments are publicly available and have been established in prior academic work. While our research aims to advance the understanding and performance of Al reasoning systems for beneficial applications, we acknowledge that LLMs are a dual-use technology. The foundational models used in this study may reflect biases present in their original training corpora. Our work does not introduce new data sources but builds upon existing models; therefore, the potential for inherited bias is a relevant consideration for any downstream application. We have conflicts of interest disclose.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. All models used in our exper- iments, including variants of Qwen, Llama, and MIMO-VL, are open-source. The training and evaluation datasets- - including DAPO, DeapScaleR, ViRI39K for training, and benchmarks like AIME, Math500, Math Vista, and others for evaluation are publicly available. A complete list provided in Section 4.1 and Appendix D.1. The core methodology of our proposed algorithm, Hierarchy-Aware Credit Assignment (HICRA), is detailed in Section 3. A critical component of our work is the automated, data-driven pipeline for identifying "Strategic Grams" (SGs), which fully described in Appendix A.2. To further aid replication, the complete list of SGs generated by pipeline is provided in Listing 1. Key hyperparameters and implementation details, including the training setup and evaluation protocols, documented in Appendix D. We attach the code of HICRA on top of VeRL as supplementary materials.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Vosss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam. Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL https: //////axxa //aaxiivvvv org/ abs/ 210021100..0 03374.
Minghan Chen, Guikun Chen, Wenguan Wang. and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization, 2025. URL httpa:// http:////xiv.vv// org/ abs/ 2505 12346.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. A.XY::2506.145888 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2005.26617, 2025.
Rumpeng Dai, Tong Zheng, Run Yang, Kaixian Yu, and Hongtu Zhu. R1-re: Cross-domain relation extraction with rlyr. arXiv preprint arXiv:2507.04642. 2025.

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501.1294 2025.
Tuomas Haarooja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361.
PMLR, 2017.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0544 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadhench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 2000:02402.14008, 2024.
Julia M Huntenburg, Pierre-Louis Bazin, and Daniel S Margulies. Large-scale gradients in human cortical organization. Trends in cognitive sciences, (2(1)221-31,201
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang, Ziwen Wang, Huaxuan Ding, Zhuo Cheng, Wenhao Cao, Zhiyuan Feng, et al. Hssbench: Benchmarking humanities and social sciences ability for multimodal large language models. arXiv preprint arXiv:2506. 233.925506.0322. 2025.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 08059 2018
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, and Rossella Arcucci. Beyond distillation: Pushing the limits of medical I1m reasoning with minimalist rule-based rl. arXiv preprint arXiv:2505.179555 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl fl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310. 10.0225, 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preview with 1.5b model by scaling rl. https: / / thub. com/ a http:/aeetiaaappjeeeee ca-project/ deepscaler, 2025.
Mathematical Association of America. American invitational mathematics examination (aime), 2024. URL https: / /mma org/ haag//maa--a taviiaaiinnll compet tions/.
12

---

Published as a conference paper at ICLR 2026
John D Murray, Alberto Bernacchia, David J Freedman, Ranulfo Romo, Jonathan D Wallis, Xinying Cai, Camillo Padoa-Schiope Tatiana Pastemnkk, Hyojung Sco, Daeyeol Lee, et al. A hierarchy of intrinsic timescales across primate cortex. Nature neuroscience, 17(12):66111166, 2014.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Chaofan Pan. Lingfei Ren, Yihui Feng, Linbo Xiong, Wei Wei. Yonghao Li, and Xin Yang. Multi- granularity knowledge transfer for continual reinforcement learning. In Proceedings the Thirty- Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pp. 6012-6020, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: In- centivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint wwXiv:25055/59666 2025.
Kimi Team, Angang Du, Bofci Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 12599, 2025.
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025a.
Haozhe Wang. Chao Du, Panyan Fang, Li He, Liang Wang, and Bo Zheng. Adversarial constrained bidding via minimax regret optimization with causality-awand reinforcement learning. In Pro- ceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 2314-232- 2023.
Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, and Fangzhen Lin. To code or not to code? adaptive tool integration for math language models via expectation-maxiiizztoo. arXiv preprint 250001250..00691, 2025b.
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vi-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08887, 2025c.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-9516 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprini arXiv:2506.01939 2025d.
Ronald J. Williams.
Simple statistical seadieet-followin algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 2004. URL http:////aap.eeeeseiesci.eoeeeee
org/Corpus ID: 23:2213513
LLM-Core- Team Xiaomi. Mimo-vl technical report, 2025. URL URl.ttta.//arx/v.oooo/////// 2506. 03569.
Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang, and Xiangxiang Chu. Hs-star: Hierarchical sampling for self-taught reasoners via difficulty estimation and budget reallocation. arXiv preprint arXiv:2505.1 /9866, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint 1511 2024.
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Roxana Zeraati, Yan- Yan-Liang Shi, Nicholas A Steinmetz, Marc Gieselmann, Alexander Thiele, Tirin Moore, Anna Levina, and Tatiana A Engel. Intrinsic timescales in the visual cortex change with selective attention and reflect spatial connectivity. Nature communications, 1 14(1):1858 2023.
Renrui Zhang, Dongzhi Jiang. Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186, 2024.
Tong Zheng, Lichang Chen, Simeng Han, R Thomas McCoy, and Heng Huang. Learning to reason mfittuee-ofthhouhh for logical reasoning. arXiv preprint 1rXi::2505.18817 2025.
Xiaomeng Zhu, Yuyang Li, Leiyao Cui, Pengfei Li, Huan-ang Gao, Yixin Zhu, and Hao Zhao. Afford-x: Generalizable and slim affordance reasoning for task-oriented manipulation. arXiv preprint wrXiv:2503.03556 2025.
14

---

Published as a conference paper at ICLR 2026
A EXTENDED MATERIALS OF SECTION 2
A.1 EXAMPLE OF STRATEGIC GRAMS.
We motivate our classification of tokens by drawing parallel to human cognition. When a person reasons through problem, we identify their strategic thinking by its function. A phrase like, "Let's try a different approach." functions as high-level strategic maneuver that guides the problem- solving direction. In contrast, a phrase like, "so we add to both sides," is low-level procedural step. Inspired by this functional distinction, we introduce Strategic Grams as a functional proxy to circumvent the difficulty of formally defining "planning token".
Airight, I have this problem to solve: Given that and d are First, need to minimize the expression: ...
Alright, so have find the minimum value of $ given these conditions.
Let's look at the expression:
I notice that both terms are fractions...
Let's see if can find relationships or substitutions. Maybe I can set = + and m = a + d. Wait, the constraint is + 2 a d, so k 2 m, where k = b c and m a a d. But I'm not sure if that helps directly. Alternatively, perhaps I can express d in terms of other variables. Let's see.
Maybe can consider specific cases or assume equality in the constraint to see what happens.
But wait, earlier, I assumed b + + d. However, the given constraint... To confirm, need to check if this is indeed the minimum
### Final Answer
After analyzing the problem and considering specific cases, the minimum value of the expression is:
v2
Figure 6: Reasoning from Qwen3-4B-GRB with planning tokens (strategic grams) highlighted. Planning tokens function as the high-level strategic moves of reasoning, including logical deduction, branching and backtracing.
Strategic Grams (SGs) defined n-grams that function single semantic unit guide the logical flow. We use n-grams because they capture the phrasal nature of strategic language (e.g.. "let's consider the case") which is lost the single-token level. These SGs facilitate three main types of logical moves: (a) deduction, (b) branching, and (c) backtracing, as we show in Figure 6.
A.2 SG CONSTRUCTION AND SENSITIVITY ANALYSIS
A key challenge is identifying the set of SGs in principled and reproducible manner. Manual annotation or reliance on proprietary models would introduce subjectivity and hinder reproducibilityy
We therefore propose an automated, data-driven pipeline based on key insight: SGs function as the reusable scaffolding of reasoning process. This function imparts a distinct statistical signature: SGs should appear frequently across wide range of different problems but be used sparingly within any single solution. However, a significant challenge is the linguistic diversity of strategic language, where single strategic intent be expressed through numerous semantically equivalent phrases.
15

---

Published as a conference paper at ICLR 2026
Our pipeline is designed to overcome these challenges by first grouping semantically equivalent n-grams and then identifying which consolidated concepts exhibit the statistical signature of strategic planning.
We construct the SG set via the following three-step procedure:
1. Semantic Clustering:
We first extract all n-grams (where n [3,5]) from large corpus of successful reasoning solutions. Each n- r-gram is projected into a semantic embedding space using a pre-trained sentence transformer. We then apply clustering algorithm this embedding space. This step groups lexically diverse but semantically equivalent n-grams into a single cluster, directly addressing the challenge of linguistic diversity.
Identification by Frequency: To identify the clusters that represent reusable reasoning patterns, we analyze their frequency at the corpus level. For each semantic cluster, we compute its Cluster Document Frequency (Cluster DF): the frequency of unique solutions that contain at least a-gram from that cluster.
3. SG Construction: We filter for clusters with top 20% Cluster DF, implying that these SGs common across many problems. The union of all n-grams within these high-frequency clusters constitutes our final of Strategic Grams.
This pipeline result in the following collection of SGs.
Listing Strategic Grams
{'## step', 'a good starting point is', more di rect approach is', more strai forraardd approach',
a simpler approach is', 'alright', 'alternatively', 'an alternative path is', 'an error in the thought process',
analyze the', 'analyzing the given', 'and then find', 'another approach'
'are locking for',
based on the given', 'break down the problem', break it down', break it down into manageable steps', 'ut', 'but wait', bbt why', 'but without rewritten as',
information', can be can conclude that', can see that', 'check if', 'consider the case where " 'consider the properties of', correct the approach', 'defnne the variables', 'deote'' 'determine how many', 'directly address the problem',
does this held true?', 'does this make sense?', 'double-checking the logic', 'finally need to', finally we need to', 'find a simpler', 'find a way to', ffind out how many', ' 'find the critical points', first need to', 'follow these steps', 'for simplicity', ffrom earlier have', 'from the above',
11
from this, it follows that', 'fro this, we can infer', 'given the complexity', 'given the complexity of', given the constraints', 'given the nature of', 'go back to the', 'goal to', 'hmm," ', 'hold on', however', 'however, we need to', 'i might have made an error', need to re-evaluate', 'i should verify this result', ridentfyy the', " 'identify the given information', 'if that doesn /t t work, can', 'it consider',
13
15
in way that', 'in the context ot', 'is there a simpler method?', 'is there a simpler way?', it logically follows that', 'it seems', ritVss better', 'et me', 'et pause and think', let me rethink this', 'let me verify', 'letWs'' (let'ss analyze the possibilities', 'let\'s assume',
16
17
18
let ('s backtrack', 'letV's break this down', 'let\'s check our work', letl's check the constraints again', let\'s consider another case', 'let\'s denote', 'letV's
letV's explore a different possibility',
20 leti's formulate a plan', 'let\'s go back a step', 'let\'s outline the steps', 'LetY's pause and think', let\'s reconsider', "1et''s try different angle', (1etVs valid this', "looking back at the', 'maybe',
16

---

Published as a conference paper at ICLR 2026
' "maybe can', 'my previous step was flawed', ' 'need to', 'need to account for', nned to analyze',
23 need to check', 'need to consider', 'need to count', 'need to deternine' 'need to ensure', 'need to express',
'need to find', 'need to follow', 'ned to identify', 'need to mininize', 'need to reconsider', 'need to show', need to solve', 'need to think abcut', 'need to understand', 'need to use', 'next', 'note that', 'now', now Let', 'now need to', 'now we need to', 'okay', 'en second thought', on the other hand', one way to', 'our strategy is', 'perhaps', 'perhaps can', 'problen is asking', 'problem states that', proceed with the following', 'rearrange the equation', 'recall that', referring to previous step',
25
26
27
28
revisiting the initial assumpticn', 'rewrite the equation', 'says that', sseems a bit complicated',
30
should consider', 'should focus cn', ssouudd look for', 'similarly', simplify the problem', 'since',
so', 'so after', 'so again', 'sc the question becomes', so, yes', something is wrong here', 'specifically',
32
start by', 'states that', 'step by step', 'step by step reasoning', step by step solution',
33
step-by-step reasoning', 'that canl't be right', 'tha seems', tthtt was
mistake',
34
that assumption was incorrect', 'the correct approach is', the core idea is', 'the first step is', the key insight is', 'the key is te realize', 'the key to solving this', 'the logical flow is',
35
' the next step is', 'the path to the solution', the plan is to', 'the problem asks for', the problem is about', 'the problem mentions', the problem says', the problem states',
37
there is mistake', 'there seems to be', ' 'therefor'', 'think of this ', 'this allows us to',
39 this approach isn'tt working', 'this approach seems', 'this can be seen as', 'this implies',
this implies that', 'this is because', 'this is not the correct approach ' 'this isnVt lead' ng anywhere',
41 this leads to', 'this leads us to', 'this logically Leads to', 'this means', "this means that',
this seems bit', 'this suggests', 'this suggests path', 'this suggests that', 'thus', 'to confirm',
to consider the constraints', 't determine', 'to do this', to ensure',
to ensure correctness',
to find', 'to make it easier', 'to proceed', to see if', to solve this problem', 'to verify', try to', 'understand the given information', uundrsstadding the problem' unnderstanding the problem first', 46 upon closer inspection', 'use the concept of', use the fact', use the fact that', 'use the nethod of', use the properties of', 'verify the solution', 'aat'' 'watt, but', wait, no', 'wait, that ' not right', 48 want to find', 'we are dealing with', 'we can', 'we can approach this', 'we can conclude', 'we can deduce',
44
we can infer', 'we can see', 'we can start by', 'we can think of this 'we can use', 'xe know', 50 what am i missing?', 'what happens if', 'what if we assume', 'what if we try', 'what is being asked', 51 which means', 'will consider the', 'work our way'
This automated procedure is designed to yield breeihireiiiin//i funcisoo proxy for strategic planning, not an exhaustive lexicon of all possible SGs. We set reasonable hyper-parameters for identifying SGs, and we contend that the resulting SG collection is sufficiently representative to reveal the core learning dynamics. To validate this claim, conduct sensitivity analysis by randomly removing
17

---

Published as a conference paper at ICLR 2026
30% of the identified SGs and re-rumning main analysis. As shown in Figure and Figure 8, the semantic entropy curves remain qualitatively identical, and the curves for perplexity and token entropy only slightly change. Semantic entropy here calculates the entropy of frequency distribution, which itself is not sensitive to slight changes in the frequency mass, as long as there are sufficient numbers of bins. This demonstrates the robustness of our SG identification and the findings derived from it.
doxtr (PU
Esocbution
6.0
Erempy Charge
1.0
Traiting Stop
The min Step
Traniin Step
Paintngg Step
(a) Full SGs
Plarstin TMkaninn PTL
Hanning Tocen
Stratogic Grame
Accoracy
Exocution Tooes 1nt PPL P.1
Enocol Taken
Other Grand C (Cnddllooa))
Sequance Langth
Entrpy Change
6-12
Tokura
Rebibee Pemvoctt
3 10
155
1 14
189
26
24
(b) 30% SGs randomly dropped
Figure Sensitivity Analysis of randomly dropping 30% strategic grams on Qwen3-4B-Ban training dynamics. The semantic entropy curve remain identical.
Planning lokendiniPP PHL Exccuutio thik
1.7%
20X
Other (Condita G).
0chibr Pempocttyy
1,5
1.00
522
20 Faining
Yoliting Stop
Stop
31op
(a) Full SGs
Plarning Exccution L.7)
Parning
3.50
Stratogic Grans
6375
Accorscr
322
Other (Condita/
Sequence
2.c.
sare
3.39
c275
0250
Taining Stop
207
Step
Traniing Stop
1aining S32p
(b) 30% SGs randomly dropped
Figure 8: Sensitivity Analysis of randomly dropping 30% strategic grams Qwen2.5-7B-BB training dynamics. The semantic entropy curve remain identical.
A.3 HUMAN VALIDATION FOR STRATEGIC GRAMS
Setup and Methodology In addition to the sensitivity analysis, we also conducted a rigorous human annotation study confirm that pipeline identifies functional planning units with high precision.
18

---

Published as a conference paper at ICLR 2026
Setup: We selected 250 n-grams that were not considered SGs extracted from training rollouts in addition to the identified SGs. Annotators: Three annotators were recruited from Amazon Mechanical Turk (AMT). We required Master's Qualification and explicit instructions to classify n-grams as "Strategic/llanning" (guiding flow, proposing plans, reflection) or "Other" (procedural, factual, formatting) given the surrounding context.
The study was designed to rigorously test the null hypothesis that the identified SGs do not serve a specialized strategic function.
Results and Analysis. The results confirm that our pipeline successfully filters for function. The inter-annotator agreement was substantial. However, disagreements revealed interesting nuances.
Table 2: Human Validation of Strategic Grams Random N-grams
Source List
A1
A2
A3
Majority Vote
Identified SGs
84%
91%
88%
86%
Others
9%
15%
13%
12%
The pipeline achieves 86% precision for SGs (classified strategic) vs. 12% for random n-grams (classified as strategic). The distinction between the two groups is statistically stark.
Discussion Agreement & Disagreements:
1. Implicit Strategy in Random N-grams: Humans occasionally classified random n-grams "Strategic" if the context implied heuristic leap deduction, e.g., "Set X 5". Our pipeline, which relies on frequency distribution, misses these "one-off" strategic moves.
2. Structural Connectors in SGs: Some high- figh-frequency SGs were marked as "Other" by strict annotators, though they often structurally precede deduction. Despite these edge cases, the distinction between the two groups is statistically stark.
The result strongly validates the high precision of our automated SG identification pipeline, confirming that it successfully identifies linguistic units that genuinely function as the reusable scaffolding of high-level reasoning process.
B FULL TRAINING DYNAMICS
Following the discussion in the main paper, we make the following further observations based on the provided training charts:
The initial skill-consslidation phase might be brief or absent for some models. In the cases of the Vision-Language Models (Qwen2.5 VL-Instruct and MiMO VL-Instruct), Qwen3 4B-Instruct, the exploration strategic planning begins almost immediately at the start of training. This is evidenced by significant and immediate rise in the semantic entropy of strategic grams, which occurs in tandem with rapid boost in validation accuracy. We conjecture this is because: (a) for the VI scenarios, publicly available datasets learned quickly by state- of-the-art models (Wang et al., 2025c); (b) strong base models like Qwen3 4B-Instruct already possess solid foundation of low-level skills and primarily need adapt to formatting before focusing on higher-level strategic planning.
Token-level entropy does not directly correlate with model accuracy. This is strongly supported across multiple experiments. For instance, with L 8B, Qwen3 4B Instructt and the VL models, token-level entropy either remains flat or decreases throughout training. During the same period, however, validation accuracy shows steady and significant increase. This demonstrates clear disconnect between next-token uncertainty and overall task performance.
Token-level entropy is misleading for policy exploration. This observation holds true across all experiments. The Qwen3 4B-Instruct model offers a particularly stark example: its token-level entropy remains almost perfectly flat, while its semantic entropy (diversity of strategic grams) consistently increases throughout training. This contrast highlights that the variety of semantic structures a model learns is completely different from the statistical uncertainty its next-token predictions. Figure 10 illustrates the differences of the two entropy.
19

---

Published as a conference paper at ICLR 2026
Model Surprite/ Truining Billets
Tehar-kort Eatropy
Semaardic EatrapyDiverily Diveriily
Langth VOTWAR Acceracy
Parining Thoestee
Parniing
Straleg Grart
Accurecy
Uweed
48-40 <Bitteme
13t
at
36 15s
100
130
25
<d0
35s
114
Surning To L1
Plernry luhen
Siuwiep Giarto
100
Onee2.5
-B--lam
Trining
Talning Stop
Farkin Shop
Toaking Stop
Tarning Toownted 2 Fanariio Trom the 2o. 1.5)
Pa Takan
Stringgl Grurna
031
Accurecy
Fxreution "akee
Caper sagge (CCosttoonl
Secsenggh Langth
LiansMM Mi-Anetnuct
1:00
Toaning Khop
Traning See
Thurang 1993
Taatiin Step
5.00
Strateg Grams
Accumce
Crams (Dgaifion!l
Seqpience Longth
Formir Tikin
4B dB-lnstrctt
(wen)
Exscution
Creepy
forring to
Eemoution Janen
Cencurion (ondioonid
Onen22
V1.4nermee
120 siep
Parinay Mi FPL 2.81
Harning Tokey
Strebodi Crter Gramm Thoddagge
Rucbras
Dac for Bean inst 1.5)
Concision Tooe
Spe Langin
SToOy STrre Chiroe
MBMO VL-dastruet
Trining Stop
Toxiing Step
Taaling Stop
Ta 0tep
Flaniigg "haarii Ln%, 31.11
Planring Toun
Strllgi Shatt
Acqurasy
Doncation Taken: nit P9 4.0)
Exscution Tolen
Biter Shame (Conditonal
Langir
Ertres
Lionod
Traniing
200
other (iggaattoootinnn
Leturp
poree Pepeeety
08-Diall Limnsan
Takend
110
Tharing Shop
Tuaning Shop
Thainig
Tratiig Step
Figure 9: Training Dynamics across different LLMs and VLMs.
20

---

Published as a conference paper at ICLR 2026
Ialen boyabilty Bisrbuttn
Regram frequency Disrisution
0.20
2,3
3,7-
2.03
3.04
3.09
6.22
202
to phe prrcoue
0.70
Color
300
sogore only
Figure 10: Comparison of Token Entropy and Semantic Entropy. (Left) Token-level Entropy is computed over the distribution of next-token probability. (Right) Semantic Entropy is computed as the Shannon Entropy over the frequency distribution of n-grams. Intuitively, Semantic Entropy gathers tokens by their semantic function and measures the semantic diversity. Token-level entropy not de-duplicated by semantic meanings, and is thus dominated by vast amount of high-frequeng low-level tokens.
The core difference is about scale: token-level entropy measures the uncertainty of every next-token, including the vast amount of low-level tokens such as formatting, executions that are doomed to become confident throughout training. In contrast, semantic entropy measures the diversity of the overall meanings being expressed. A model can be very predictable in its next-token choice under given context but still create a wide variety of different arguments or structures.
Lack of Strategic Exploration Hinders Sustained Improvement in Llama Models. We observe that the Llama-3.1-8B-Ban. model initially focuses almost exclusively on consolidating low-level procedural skills, a phase marked by decreasing perplexity and token entropy on execution tokens. However, once the performance gains from this procedural refinement diminish, the model fails to pivot towards exploring high-level planning strategies. This leads to performance stagnation and, eventually, degradation.
This behavior stands in stark contrast to the more successful Deepseek Distilled Llama model, which engages in high-level strategic exploration from the very beginning of training, bypassing a distinct procedural consolidation phase. We hypothesize that for the standard Llama models, the intense initial focus on procedural correctness prematurely collapses the diversity of high-level reasoning strategies. By the time low-level skills are mastered, the model has likely converged on simpler reasoning patterns, which inhibits its ability to subsequently discover and adopt more complex and effective problem-solving approaches.
C THE DISTRIBUTION MATCHING PERSPECTIVE OF POLICY GRADIENTS
Imagine an ideal, or "target," policy, x (als), that we want our current policy, To (a)s)), to emulate. We can conceptualize this target distribution as being proportional to the exponentiated advantage of the actions:
nt (a]s) x Taae (als) exp(A((a))) x))
Or more concretely,
A(a, 3
n" (als)
Z(s) To.i (als) exp
This target policy, x (a(s), re-weights the old policy based on the advantage of each action. Here, actions with positive advantage (A > 0) get their probability boosted exponentially, while actions with a negative advantage (Å < 0) get their probability suppressed. The term acts as a "temperature" parameter.
21

---

Published as a conference paper at ICLR 2026
The goal is to find new policy, Te, that is close possible to this ideal target distribution, n² This is equivalent to minimizing the KL divergence:
min KL (a]s)][=((a|s))
(3)
Expanding this KL divergence term:
no (a(a))
(I((mll>=") Banto log (a|s)
-Bewno ( log x (a|s)]
Substitute definition of log "r" (als) log Toom (a|s)
A(a.s) log Z(s):
log |log (a (a|s) +
A(a, s)
Baws
log TT ma(aas)
log Z(s)
B
A(a, s)
x Ea-ws
(log To((s) - log Tooa (a|s))
B
1
Banna
[ (Toa)) Ala,s) s)
3
Minimizing this is equivalent to maximizing its negative:
max Bawny [A(a, 8) - ( (Tom)
This expression nearly identical to the PPO-KL objective, where the policy update constrained using a KL divergence regularizer (Schulman et al., 2017).
Therefore, the PPO objective is essentially solving a distribution matching problem toward a target distribution shaped by the advantage function. It follows that standard policy gradient (Williams, 2004) update nudges the policy Te... toward an implicit target distribution xr" defined by the advantage function. After the gradient update, the target distribution becomes the new policy sampled for exploration. Therefore, adjusting the advantage function (or credit assignment) essentially shapes the exploration policy during RL training.
D EXTENDED MATERIALS OF EXPERIMENTS
D. EXPERIMENTAL SETUPS
Training Datasets and Benchmarks The training dataset for LLM reasoning is sourced from DAPO (Yu et al., 2025) and DeepScaleR (Luo et al., 2025). The dataset for training VLM sourced from ViRL39K (Wang et al., 2025c). We evaluate all models on diverse set of challenging mathematical reasoning benchmarks to rigorously test their complex reasoning capabilities. The text-only benchmarks include AIME24, AIME25 (Mathematical Association of America, 2024), Math500 (Lightman et al., 2023), AMC23, Minerva (Lewkowycz et al., 2022), and Olympiad (He al., 2024). We follow Deepseek RI (Guo et al., 2025)'s 's evaluation protocol, where the performance measured by Pass@1 Accuracy with temperature 0.6 sampling. For benchmarks with less than 100 queries, we use average accuracy of 32 samplings on AIME24/25 and samplings on AMC23 (Yu et al., 2025). Following VL-Rethinker (Wang et al., 2025c), we evaluate on MathVista Vista (Lu et al., 2023), Varse (Zhang al., 2024), MathVision Vision (Wang al., 2024), and EMMA (Hao al., 2025) for assessing multimodal reasoning across domains and disciplines. For all evaluation, we adopt strict answer matching that relies on the \boxed format.
Baselines and Implementation. We compare HICRA against three primary baselines: the Base model (before RL), the widely adopted GRPO baseline with clip-higher (Yu et al., 2025) by de- fault, and Entropy Regularization: GRPO with additional regularization loss token-level entropy (Cheng et al., 2025). For HICRA, we set the amplification hyperparameter a to 0.2 and identify planning tokens using the Strategic Grams (SGs) methodology detailed in Section 2. For
22

---

Published as a conference paper at ICLR 2026
all experiments, increase the training context length from 16K 32K when the response clip rate exceeds 20% (Luo et al., 2025). For the specific experiments on -.amm-3.1-Instruce, 3.1-Instr we add dynamic filtering mechanism (Yu et al., 2025) based GRPO Clip-Higher due to significant vanishing advantanges (Wang et al., 2025c). We use two to four sets of eight A A100 (80G) for training all models, and we stop the experiments if performance continues degrade during extended training.
D.2 ADDITIONAL ANALYSES
Table 3 shows the main results on VI. models.
We also provide deeper analysis and insights into the benefits of semantic entropy for tracking exploration, differences between our approach and high-entropy tokens, and potential concerns of HICRA.
D.2.1 SEMANTIC ENTROPY: COMPASS FOR STRATEGIC EXPLORATIOO
Semantic Entropy
Validation Accuracy
Passga (Training)
Token Entropy
0.50
0.1
0.40
0.1
0.
52
75 Duaning 3005
200
125
25
15
160
155
:50
52
160
125
152
Ttattig 3top
Parning Stpp
Figure 11: Training Dynamics on MiMO-VI-Instruut-P.. This experiment highlights that token entropy can collapse while semantic entropy remains high and predictive of validation accuracy. Furthermore, while Pass @8 saturates and indistinguishable between methods, semantic entropy reveals a persistent exploration advantage for HICRA that translates to better final performance.
Given the crucial role of strategic exploration in unlocking reasoning performance during RI,, effectively measuring it accurately is paramount. We find that semantic entropy offers distinctive benefits than common alternatives such as token-level entropy or Pass @ (Chen et al., 2021).
Limitations of Token Entropy and Pass @ K. As shown in Figure 11 for MiMO-VL--7B, token- level entropy "collapses" for both HICRA and GRPO, simply because the vast majority of low-level tokens are doomed to become certain, thus pulling the average token entropy down. However, this decrease in token entropy might mislead researchers to suggest that exploration has ceased. Similarly, the Pass @ 8 (Training) metric quickly saturates, rendering useless for distinguishing the ongoing learning dynamics.
Semantic Entropy as the Differentiator. the same experiment, semantic entropy tells a more accurate story. It remains high, indicating continued exploration of diverse reasoning strategies. Crucially, HICRA consistently maintains a higher semantic entropy than GRPO, and this advantage
Table Comparison of HICRA, GRPO on multimodal reasoning benchmarks.
VLM
Math Vista
Math Vision
Math Verse
EMMA
M-M0--L-Instruut-25555
Basc
77.0
42.9
61.8
36.3
GRPO
73.7
42.8
63.0
41.9
HICRA
80.7
48.9
65.4
44.1
A (HICRA- GRPO)
+7.0
+6.1
+2.4
+2.2
Oven2.5-VV--7--Instruut
Base
66.6
23.6
45,9
22.3
GRPO
70.8
25.8
48.8
31.8
HICRA
71.4
28.7
48.2
33.0
(HICRA GRPO)
+0.6
+2.9
-0.6
+1.2
23

---

Published as a conference paper at ICLR 2026
Portion of High-Entropy Planning Tokens
Portion of High-Entropy Tokens Planning Tokens
After
0.90
Before
0,090
Before
0.70
0.080
Porton
0.60
0.010
0.50
0.060
0.40
0.060
Gwena 48 is-tnarcct
Uama 3.1 in8-maruct
MIMO-IL
Ownns Owans-a9-aad Base
0wen2.5-78-Bes
Gwend in-nsrcct
Llmma-3 1 4B an-instuct
MOMO-VL VL
Owert Ower3-48-9aas 18-ase
owen2 mwen2.5-78-94. 78-Bice
Figure 12: Planning Tokens vs. High-Entropy Tokens. (Left) majority our funetionally-defiee planning tokens are also high-entropy (top 30%). (Right) However, the reverse is not true; most high-entropy tokens are not planning tokens.
Airight, have this problem midpoint of AD and Fis the extension of such that BF equals AB. Then. says that plane passing through points C, and F intersects BD at point Glll need find the volume the tetrahedron CDGE. Hmm. okay. Lot try visualali this tetrahedron has four trisngular faces. Points
tetrahedion ABCD with volume question introduces and E. describes how the
labeled A B, C. D. E the midpoint of AD, which helpful. i the extension of AB beyond B, and BF AB, F far from B as A B. So. If Imagine AB line segment, E would bä another point such thatl BF equals AB. So, AB has certain longth, say ' Lithen BF is also 'K' (0 AE would be AB B6 2k. Now, the plans passing through c, E, and F intersects BD at point 6 BD @ edge of the tetrahedron. So. GG is the intersection point of this plané with BD. The tetrahedroni CDGE is formed by the points C. D, G, end E and lineed to find its volume in terms of the original volume V, Since all the points are defined selatively it might help t0 assign coordinates to the vertices to make the problem more concrete Let s try to create a coordinate system where we can express each point with koordinates, compute the necessary vectors, and then calculate the volume. Let me set up à coordinate systom. can place paint A the ko Oß Let me assume AB lies along the x axis. So. point will be at some point 0, 0). Let' s. denote point as (0, 0: and point B as asllelloo 00. Then, point D can bei in some other position, say (d. e, fnllndd point C can be (g. h, i. Wait, but need to keep track of the volume: kisa, point the midpoint AD, so since (0. o, dj and is (d, f2.E.
High-Entropy Token
Planning Token
Shared
Figure 13: Planning Tokens, High-Entropy Tokens and Shared Tokens are highlighted with different colors. This concrete example suggests how these two definitions differ: Planning Tokens function as strategic skeletons of reasoning solution and are thus sparse, with more than half of these semantic units also having higher entropy. In contrast, majority of high-entropy tokens only exhibits high-variations in its phrasing, spreading across low-level executions and high-level planning. Fig. 12 reveals that less than 10% high-entropy tokens serve the semantic function of planning.
directly correlates with its superior final validation accuracy. This also demonstrates the generality of our approach, extending effectively to multimodal reasoning tasks on vision-language models like MiMO-VL-7B
D.2.2 PLANNING TOKENS VS. HIGH-ENTROPY "FORK" TOKENS
Recent work has proposed high-entropy tokens, sometimes called "fork tokens," to imply its role as proxies for decision points in a reasoning trace (Wang et al., 2025d). Our analysis investigates the relationship between our functionally-deffned planning tokens and this entropy-based definition.
Figure 12 reveals a crucial asymmetry. While majority of planning tokens exhibit high entropy (aligning with their role points of strategic choice), the reverse is not true: most high-entropy tokens are not planning tokens. This finding highlights the limitations of using high entropy as a standalone proxy for strategic function. High token-level entropy ensures sampling diversity, but it does not guarantee semantic function. Many high-entropy tokens may correspond to variations in phrasing calculation that do not alter the high-level reasoning path. In contrast, approach identifies tokens based on their functional role in orchestrating the solution, providing a more direct and reliable signal for strategic credit assignment.
24

---

Published as a conference paper at ICLR 2026
D.2.3 BOUNDARY CONDITIONS OF HICRA
Our hierarchical framework predicts that strategic exploration gulx.bensfjcial once Igundational, level of procedural competence is established. HICRA's effeetiveness is predieated on a key assumpt tion: that the base model should readily possess a reasonable foundation for lewelveel www.eeeeeeooeeddrrr correctness. As shown in Figure 14, when this foundation lacking yf as observed Llama-3, 3.1 Instruct HICRA can fail to provide an advantage over GR CRROOSSeet from the semajttic entropy graph, there is reverse trend between GRPO and HICRA, implying g an opposite training föcns on planning tokens and execution tokens. HICRA's enforced strategic exploration helooaationhcconeek..... eus
if
St the model cannot reliably execute the plans it generates, leading mistable learning dynamics and learning effects observed on Llama-3. 1. This delineates the-scope of A most effective when applied to models that have already achieved degree procedural reliabilitt.
suggesting that HICRA
HACRA' on w 1-Instuuct 8B.
highlighting an important dependency for future work on more adaptive, model- aware hierarchical methods.
E LLM USAGE
During the preparation of this manuscript, the authors utilized a large language model (LLM) for assistance with editing. The LLM was used improve grammar, clarity, and the overall logical flow of the text. However, the core scientific contributions, including the conceptu- alization of the research, experimental design, analysis, and the final conclusions, are entirely the work of the human authors.
25


Reviews:
[{"review_id": "PAS9wHZR9w", "reviewer": "Reviewer_WcVs", "summary": "The authors discover two-phase training dynamics of LLMs finetuned using reinforcement learning. The first phase optimizes low-level procedural execution tokens, followed by exploration of high-level planning tokens, the latter explaining aha moments, length scaling, and also contributes to advanced reasoning. They also propose a simple modification of the GRPO algorithm, called HICRA, which amplifies the advantage of planning tokens in the solution trajectory. Across several LLMs and VLMs, HICRA demonstrates significant improvements over mathematical reasoning benchmarks.", "questions": "1. Can the authors provide more details about how the token-level and semantic entropy is computed, including some equations?\n\n\n2. How was the value of the alpha parameter chosen?\n\n3. Figure 4 and Figure 5 why are different models trained for different numbers of steps? A fair comparison would be to train all models for the same number of steps.\n\n4. How would a baseline with an explicit optimization of semantic entropy perform?\n\n\n\n5. What is the pretrained sentence transformer used for clustering of semantic embeddings? Why is the LLM’s tokenzier not used for getting the n-gram embeddings and how would that perform?\n\n6. How sensitive is the performance of HICRA to the SG construction of taking top k% cluster document frequency, random dropping etc?\n\n7. for unsuccessful ones, the penalty is reduced for the planning token - what is the intuition behind this? What if the planning tokens primarily contributed to the failure?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Analyzing the two-phase learning dynamics of LLMs trained using RL: rapid learning of procedures through low level tokens followed by exploration of high-level strategic planning tokens, the latter explaining aha moments and length scaling\n\n2. Identifying high-level planning tokens through the construction of strategic n-grams and computing their entropy\n\n3. Modifying GRPO algorithm to amplify the advantage of planning tokens that contribute to advanced reasoning\n\n4. Significant improvement on mathematical reasoning benchmarks using multiple LLMS/VLMs", "weaknesses": "1. The evaluation is currently limited to only mathematical reasoning tasks, and it remains to be seen whether this approach of emphasizing optimization of strategic tokens would also lead to improvements on generic multi-step reasoning tasks. \n\n2. The authors claim that this two-phase training dynamics is an emergent phenomenon of LLMs that are finetuned using RL, however, they provide no evidence that standard finetuning (i.e., no RL) doesn’t lead to similar learning dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLPKNjH8lv", "reviewer": "Reviewer_2wpP", "summary": "The paper studies why RL improves LLM reasoning and proposes that training exhibits an emergent two-phase hierarchy: (1) early consolidation of low-level procedural execution (formatting, arithmetic), evidenced by reduced perplexity and token entropy for “execution tokens”; (2) later exploration and mastery of high-level strategic planning, evidenced by increased “semantic entropy” of “Strategic Grams” (SGs)—n-gram phrases that purportedly carry strategic functions (deduction, branching, backtracing). The authors claim this framework unifies puzzling observations (“aha moments,” length-scaling, token entropy dynamics). They argue that standard GRPO distributes optimization uniformly across tokens, diluting credit on high-impact planning tokens. They propose HICRA, a simple GRPO modification that amplifies token-level advantages for planning tokens (identified by SGs) and dampens their penalties, thereby focusing optimization on strategy. Across several LLMs and VLMs on math reasoning benchmarks (e.g., AIME24/25, Math500, AMC23, Minerva, Olympiad; MathVista, MathVerse, MathVision, EMMA), HICRA reportedly outperforms GRPO baselines. They further analyze error-type dynamics (planning vs. others), show semantic entropy is a better exploration indicator than token entropy or Pass@K, and discuss limitations (e.g., dependence on a solid procedural base; weaker gains on Llama-3.1-8B-Instruct).", "questions": "1. Can you provide quantitative validation of SG detection (human annotation study with precision/recall; inter-annotator agreement)? How sensitive are results to different embedding models, clustering methods, n-gram ranges, and thresholds?\n\n2. Can you run targeted causal tests—e.g., (i) randomly reassign SG labels to matched-frequency n-grams; (ii) ablate SG presence at train time (mask or paraphrase them away); (iii) directly reward semantic-entropy increases vs. HICRA—to test whether planning-focus per se drives gains?\n\n3. Can HICRA dynamically down-weight planning tokens when execution errors dominate (e.g., via online estimation of error types) to avoid failures on weaker models?\n\n4. Generalization: Do the findings hold on non-math reasoning (commonsense, code), agentic tool-use, and multilingual tasks? How portable is the SG lexicon across domains and languages?\n\n5. Robustness and stats: Please report means/standard deviations over multiple seeds for key benchmarks; include significance tests. Are improvements consistent under stricter decoding (temp=0) or chain-of-thought suppression?\n\n6. If you paraphrase SG phrases (same function, different wording), do gains persist? What fraction of HICRA’s advantage survives when SG phrases are held out from training prompts?\n\n7. Does HICRA exacerbate verbosity or hallucinations in non-math tasks? Provide length-normalized metrics and calibration analyses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear, testable hypothesis about a two-phase, hierarchical learning dynamic with consistent qualitative evidence across multiple model families (text-only and VLM).\n\n2. A simple, implementable algorithm (HICRA) grounded in the analysis; integrates cleanly with GRPO and yields consistent improvements on several benchmarks.\n\n3. Useful analytic tools: semantic entropy over strategic units; error-type diagnostics; sensitivity analyses showing robustness of qualitative trends to partial SG removal.\n\n4. Solid empirical sweep: multiple models, several benchmarks, process metrics (PPL/entropy/length/accuracy).", "weaknesses": "1. Operationalization of “planning tokens” via SGs is heuristic and potentially brittle: depends on corpus choice, clustering hyperparameters, n-gram length range, and English phrasing. No human validation statistics (precision/recall) against expert annotations; no cross-domain generalization test (beyond math/VLM math).\n\n2. Causal claims remain tentative: the evidence is largely correlational. In particular, the “two-phase” dynamic and the linkage between semantic entropy and accuracy could be confounded by curriculum effects, context-length growth, reward clipping, or prompt-format shifts.\n\n3. Dependence on pre-existing procedural competence: HICRA underperforms/struggles on weaker models (e.g., Llama-3.1-8B-Instruct), suggesting the method is not broadly robust. No adaptive mechanism to detect when to emphasize execution vs. planning.\n\n4. Reward shaping risk: amplifying credit on planning tokens may overfit to stylistic strategic phrases (SG lexicon) rather than genuine strategy. The overlap/shift of SGs across datasets or languages is unclear; multilingual robustness is untested.\n\n5. Evaluation transparency concerns: many datasets are small and sensitive; multiple sampling runs, seeds, and statistical tests are not fully reported. Some deltas are modest; one negative result (Olympiad on Qwen3-4B-Instruct) is noted.\n\n6. Comparison set: strong recent baselines (e.g., process-reward shaping, reflection-based RL, decision-token methods) are not exhaustively compared; entropy-regularization is a weak foil.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wL4D4G2VSy", "reviewer": "Reviewer_QnoB", "summary": "This paper presents an analysis of RL dynamics in large language models, proposing that enhanced reasoning emerges from a hierarchical separation between high-level strategic planning and low-level procedural execution. The authors introduce a data-driven method to identify \"planning tokens\" and build upon this insight with HICRA, a targeted credit assignment algorithm that consistently outperforms strong GRPO baselines across various models and mathematical reasoning benchmarks. This demonstrates that their conceptual framework is not merely theoretical but offers a practical and effective lever for improving model performance.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper's focus on identifying and boosting \"reasoning\" or \"planning\" tokens is an interesting motivation. Experiments demonstrates that this is a practical lever for improving model performance.\n\n2. Its proposed algorithm, HICRA, demonstrates consistent performance gains across several small models and benchmarks, showing the effectiveness and generality of the approach.", "weaknesses": "1. The notion of a decisive shift and strict sequential separation (first one, then the other) may over-simplify the RL training process in a massive over-parameterized model like an LLM. In reality, planning and execution are rarely decoupled in learning. Improvements in low-level procedural skills (e.g., generating syntactically correct code or flawless calculation steps) immediately inform and enable more ambitious, complex, and high-level strategic plans. Conversely, a good high-level plan (strategic reasoning) acts as a powerful curriculum for focusing the procedural execution component, leading to faster skill acquisition.\n\n2. The proposed solution, Hierarchy-Aware Credit Assignment (HICRA), relies heavily on the ability to consistently identify and isolate \"high-impact planning tokens\" from procedural execution tokens. However, ground-truth labels for token function are lacking in this field. Since LLMs generate text in an autoregressive fashion, the designation of a token as \"planning\" or \"procedural\" is not externally supervised. It must be determined by some internal, emergent mechanism (e.g., attention head patterns, internal self-correction signals, or specific activation clusters). If HICRA relies on a learned or heuristic mechanism to identify the hierarchy, the success of the credit assignment becomes critically dependent on the robustness and stability of that internal mechanism. A noisy or incorrect classification of a \"planning\" token as \"procedural\" will cause the reward signal to be diluted (the problem HICRA aims to solve). Conversely, misclassifying a \"procedural\" token as \"planning\" could lead to unstable and high-variance gradients in the early layers responsible for high-level reasoning, potentially destabilizing the planning component itself.\n\n3. While concentrating credit on planning tokens targets the apparent bottleneck, it simultaneously starves the procedural components of the direct, high-magnitude signal they need to maintain or improve their correctness. For complex, multi-step tasks, procedural fidelity is non-negotiable. Reducing the optimization pressure on these steps may lead to a model that has great plans but consistently fails on the execution details.\n\n4. This work builds heavily upon the intersection of methodologies from at least three areas: hierarchical RL for LLM agents with explicit planning modules; fine-grained credit assignment \\& token-specific optimization; RL training dynamics. This makes it difficult to identify a significant contribution of this paper. For example, MT-Core [1] also consists of a coarse-grained policy formulation and a fine-grained policy learning by LLM and RL, respectively. Then what is the key contribution of this paper with respect to the methodologies from these three areas?\n\n[1] Pan, C., Yang, X., Wang, H., Wei, W., \\& Li, T. (2024). Hierarchical continual reinforcement learning via large language model. arXiv preprint arXiv:2401.15098.\n\n5. The core limitation lies in the algorithm's dependency on a foundational level of procedural correctness. HICRA's method of selectively amplifying the signal for high-level planning tokens is most effective *after* the base model has achieved a certain level of procedural reliability. If the base model lacks these reliable low-level skills, the enforced strategic exploration can be counterproductive, as a brilliant strategy is useless if the execution (e.g., a simple calculation) is consistently flawed. More experiments should be conducted to sufficiently  explore the performance of HICRA on models with weak initial procedural skills, such as MiMO-VL-Base.\n\nThe current form of the paper is not enough convincing from motivation to methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vyrs7244WG", "reviewer": "Reviewer_RzuK", "summary": "This paper investigates the learning dynamics of Large Language Models during reinforcement learning for reasoning tasks. The authors propose a two-phase learning process. Models first consolidate low-level procedural skills and then shift to exploring high-level strategic planning. This paper argues that the second phase is the main bottleneck for performance. Based on this insight, the authors introduce HICRA, a new algorithm that modifies the credit assignment in RL. HICRA focuses optimization pressure on \"planning tokens\", which guide the reasoning process. Experiments across multiple models and benchmarks show that HICRA outperforms standard RL baselines by accelerating strategic exploration.", "questions": "1. The identification of Strategic Grams is a critical component. How sensitive is HICRA's performance to this list? For example, what is the impact of using a much smaller list, or a list generated from a different data domain? This would help clarify whether the method is learning general planning skills or overfitting to certain phrases.\n\n2. Could the authors provide a direct experimental comparison against a baseline that uses a high-entropy heuristic to identify and reward important tokens? \n\n3. Could the authors assign different weights to different types of planning tokens, distinguishing between high-level strategy and lower-level tactical steps?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a clear and insightful analysis of the learning process in LLMs. The proposed two-phase framework provides a compelling explanation for several previously observed but poorly understood phenomena, such as \"aha moments\" and \"length scaling\". This provides a valuable conceptual model for the community.\n\n2. The proposed method, HICRA, is simple, well-motivated, and directly addresses the bottleneck identified in the analysis. Instead of introducing a complex new architecture, it makes a targeted and intuitive change to the advantage function of an existing algorithm, making it easy to understand and implement.\n\n3. The empirical evaluation is extensive and convincing. The authors validate their claims across a diverse set of open-source language and vision language models on multiple challenging mathematical reasoning benchmarks.", "weaknesses": "1. The method's effectiveness is dependent on the quality of the \"Strategic Grams\" used to identify planning tokens. The automated pipeline for extracting these grams seems heuristic and may not be robust. It is unclear if it truly identifies strategic reasoning or just common, high-frequency phrases that correlate with correct answers.\n\n2. The set of compared baselines could be stronger. The paper argues that its functional definition of planning tokens is superior to using high-entropy \"fork tokens\" as a proxy. However, it does not include a direct experimental comparison to a baseline that selectively rewards high entropy tokens. Such a comparison is necessary to validate the additional complexity of the proposed semantic identification pipeline.\n\n3. The paper frames the problem with a binary distinction between planning and execution tokens. However, this is a simplification of complex reasoning, which can involve multiple layers of strategic abstraction. The method treats all planning tokens equally, which might not be optimal for learning intricate, multi-step plans.\n\n4. The method's applicability appears to be limited by the base model's initial capabilities. The appendix shows that HICRA can perform worse than the baseline on models that lack a solid procedural foundation. This suggests the method is not a general solution but rather an accelerator for already competent models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)