HiPO: Self-Hint Policy Optimization for RLVR
Authors: Deng Qiyuan
Kehai Chen
Min Zhang
Zhongwen Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Language Models
Mathematical Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) is a promising method for enhancing the complex 
problem-solving abilities of large language models (LLMs). This is particularly evident in domains requiring 
long-horizon reasoning and precise execution, such as solving complex mathematical problems where solutions 
hinge on a fragile sequence of tool-based actions. However, current approaches are often crippled by two 
interconnected issues: the near-miss problem, where sparse rewards nullify the learning signal for 
almost-correct attempts, and the resulting exploration stagnation, which prevents the model from 
discovering better solutions. To address these challenges, we introduce HiPO (Hint-guided Policy Optimization), 
a novel RLVR framework that enables the agent to learn from its own rare successes. 
Our core insight is to capture an occasional successful trajectory within a training batch and
repurpose its initial correct steps as an on-policy “hint”. This process 
transforms a single, stochastically-found success into a dense contrastive learning signal, 
effectively allowing the model to teach itself how to overcome the near-miss 
problem and break exploration stagnation. On a challenging suite of five mathematical reasoning benchmarks, 
HiPO improves the average avg@32 by +5.0 percentage points (pp) over the strong GRPO baseline. 
This improvement is driven by substantial absolute point gains on challenging datasets, 
including +10.3 pp on CMIMC 2025, +4.9 pp on BRUMO 2025, +4.6 pp on AIME 2024, and +3.1 pp on AIME 2025.
Furthermore, HiPO demonstrates a new exploration paradigm, 
repurposing rare successes into reusable guidance to significantly accelerate skill acquisition for complex tasks, 
establishing a more efficient and scalable path for models to autonomously master intricate reasoning.

Published as a conference paper at ICLR 2026
HIPO: SELF-HINT POLICY OPTIMIZATION FOR RLVR
Qiyuan Deng', Kehai Chen', Min Zhang', Zhongwen Xu Harbin Institute of Technology, Shenzhen, China {chenkehai, zhangmin2021 2aaemneint2ittttetteeee J@hit en 238151120@stu. hit edu.cn
ABSTRACT
Reinforcement Learning from Verifiable Rewards (RLVR) is a promising method for enhancing the complex problem-solving abilities of large language models (LLMs). This is particularly evident domains requiring long-horizon reason- ing and precise execution, such as solving complex mathematical problems where solutions hinge on a fragile sequence of tool-based actions. However, current ap- proaches often crippled by two interconnected issues: the near-miss problem, where sparse rewards nullify the learning signal for almost-corred attempts, and the resulting exploration stagnation, which prevents the model from discovering better solutions. To address these challenges, we introduce HiPO (Hint-guided Policy Optimization), novel RLVR framework that enables the agent to learn from its own rare successes. Our core insight is to capture an occasional success- ful trajectory within a training batch and repurpose initial correct steps as an success
on-policy "hint". This process transforms single, stochastically-four into a dense contrastive learning signal, effectively allowing the model to teach itself how to overcome the near-miss problem and break exploration stagnation. On a challenging suite of five mathematical reasoning benchmarks, HiPO im- proves the average avg 32 by +5.0 percentage points (pp) over the strong GRPO baseline. This improvement is driven by substantial absolute point gains on chal- lenging datasets, including +10.3 pp CMIMC 2025, +4.99 pp BRUMO 2025, +4.6 pp on AIME 2024, and +3.1 pp on AIME 2025. Furthermore, HiPO demonstrates a new exploration paradigm, repurposing rare successes into reusable guidance to significantly accelerate skill acquisition for complex tasks, establishing a more efficient and scalable path for models autonomously master intricate reasoning.
INTRODUCTION
A central ambition in artificial intelligence is to create agents capable of acquiring complex rea- soning skills autonomously. moving beyond the reliance on curated expert data (DeepSeek) [025] Lambert et al.) 2024). Reinforeement Learning from Verifiable Rewards (RLVR) marks promising step towards this vision, enabling Large I Language Models (L.MM)) to learn from outcome-based feedback in domains like advanced mathematics (Shao al. 2024 DeepSeek 2025) However, this pursuit confronts a fundamental paradox rooted in the very nature of complex problem-solving Success often depends on a fragile sequence of self-reflection (Renze & Guven 2024) and tool-
integrated reasoning (Feng et al. 2025 Li et al. 2025b). making a correct solution an exceptionally rare event. This fragility exposes eritical flaw in current RLVR frameworks: a brittleness of the learning signal.
This signal brittleness manifests in two interconnected challenges. The first is the near-miss prob- lem: nearly-perfedt trajectory receives the same sparse, negative reward as complete failure, thereby penalizing correct intermediate reasoning steps through flawed credit assignment. Conse- quently, this leads to the second, more debilitating challenge: exploration stagnation. The consistent punishment for near-misses disincentivizes any deviation from simple, suboptimal strategies, pre- venting the very breakthroughs in capability the learning process is meant to foster (Cui al. 2025 An et al. 2025 Cheng et al.) 2025). This further exacerbated by signal collapse, a scenario where
"Corresponding author.

---

Published as a conference paper at ICLR 2026
uniform rewards within training batch cause the policy gradient to vanish entirely (Yu et al. 2025 Xu & Ding 2025). In essence, the agent cannot learn from its successes if those successes are statistically impossible to discover.
To make this exploration challenge concrete, consider a motivating example. As shown in Figure T baseline model without guidance struggles to generate correct trajectories. However, when pro- vided with partial correct solution as "hint", its performance improves dramatically, transforming the distribution from near-certain failure high-probability success. This stark contrast suggests the primary bottleneck is not the model's intrinsic reasoning ability, but the discovery of a valid reason- ing path. This begs the critical question: can a model learn provide these crucial hints for itself, bootstrapping its own learning from its own rare successes?
Beneline
Hint
Panagl Performance
1000
1000
Narter Quntless
040-
6XC
0C4
7at4
000-
604
Nurbee
Natber 2rst
35
Mumber Coreett Rese
Number Cocmmet
[Dlumber ofAmceppis
(a) Baseline
(b) 0.2 Hint
0.5 Hint
(d) Pass Comparison
Figure 1: (a-c) show the hint ratio (the proportion of the trajectory prefix) increases, the correct solutions improves. (d) compares the pass @k performance for different hint ratios.
To answer this question, we introduce Hint-guided Policy Optimization (HiPO), a framework that operationalizes a new paradigm: Endogenous Self-Hint. The core insight is that a single, poohasstcaally-oumg success is not merely an endpoint to be rewarded, but a rich source of curricu- lum to be exploited. HiPO captures these rare successful trajectories and repurposes their initial correct steps as on-policy "hints". This process transforms the intractable task of end-to-end dis- covery into a more manageable one of guided completion. By juxtaposing the policy's unaided exploration with its hint-guided exploration, HiPO forges dense and powerful contrastive learning signal from sparse rewards. It directly resolves the near-miss problem by rewarding valid reasoning prefixes, and by providing trustworthy path forward, it shatters exploration stagnation, guiding the policy toward mastering more complex reasoning patterns.
Our main contributions
We propose HiPO, new framework that materializes the concept of endogenous self-hint. It leverages policy's own rare successes on-policy hints to create robust, contrastive learning signal, converting sparse rewards into dense, self- curriculum.
HiPO significantly outperforms a strong GRPO baseline on five challenging math- ematical reasoning benchmarks, achieving +5.0 pp average improvement in avg @ 32. The gains are particularly substantial on difficult datasets, including CMIMC 2025 (+10.3 pp) (Balunović et al. 2025). BRUMO 2025 (+4.99 pp) (Balunovié et al. 2025), and AIME 2024 (+4.6 pp).
We provide empirical evidence that HiPO successfully counteracts exploration stagnation. Through analysis of training dynamics, we show that it maintains higher policy entropy and fosters longer, more complex tool-use sequences, confirming that our self-hint mechanism enables more sophisticated reasoning.
These results highlight HiPO as an efficient and scalable paradigm for autonomous learning. By enabling models repurpose their successes into reusable guidance, work provides boot- a strapped pathway to mastering complex tasks, accelerating progress towards more capable and in- dependent AI systems.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
A prominent bascline within RLVR Lambert et al. 20244 DeepSeek 2025) is Group Relative Policy Optimization (GRPO) (Shao al. 2024). which simplifies policy optimization by forgoing critic network and instead normalizing rewards across a group of concurrently generated responses.
However, this group- based approach is crippled when rewards sparse; if all responses in train- ing batch fail, the resulting advantage is zero, nullifying the learning signal. To mitigate this, meth- ods like DAPO (Yu al., 2025) enhance GRPO with engineering treatments such as dynamic sam- pling, which forces the generation process to continue until non-zero advantage is found. Despite these improvements. such approaches are fundamentally reactive; they can only refine the learn- ing signal if at least one successful trajectory is stochastically discovered within group, failing to address the core exploration problem when success is exceptionally rare.
Moving beyond purely algorithmic modifications, another line of work injects external guidance to scaffold learning. QuestA (Li et al. 2025a) and StepHint (Zhang et al., (025). for instance, augment prompts with partial solutions or stepwise hints derived from stronger teacher models or existing datasets of OpenRI-Mee MMath- this reliance external, off off-policy guidance creates dependency on pre-existing knowledge and does not represent autonomous learning from the agent's own experience. Our work, HiPO, provides novel synthesis to overcome these limitations. It addresses the sparse reward problem like hint- based methods, but does so by creating hints that are both endogenous and on-policy. By capturing rare success within a training batch and repurposing its initial correct steps hint for the entire group, HiPO enables the model to teach itself. transforming single stochastic success into a dense, reusable learning signal without depending on external teacher models.
[ While effective creating a denser reward signal,
BACKGROUND
3.1 POLICY GRADIENT IN RLVR
The optimization of RLVR is fundamentally grounded in the policy gradient theorem from reinforce- ment learning (Sutton & Barto 2018). We can formalize the sequential problem-solving process as a finite-horizon Markov Decision Process (MDP) (Puterman, 1990). For given prompt P, serves as the initial state so. the model autoregressively generates trajectory т (oo, 01, ....... ................................ OT 1). sequence of tokens from vocabulary V. At each step t, the model's policy To defines a probability distribution over the next token ot, conditioned on the current state St.
The policy parameterized by the model's weights 6, is the language model itself. In RLVR, sparse reward R(7) is typically assigned only at the end of generated trajectory т. The objective is to adjust the parameters 0 to maximize the expected reward over all possible trajectories:
J(O) E,~=, [[())-
The policy gradient is optimized by computing its gradient with respect which can be estimated by sampling:
VoJ(()= = E,~,, V log To (r)).
To reduce the high variance of this estimator, state-dependen baseline b(P) is subtracted from the reward R(r) to yield lower- variance advantage estimate, A(T) = R(T) b(P). This results in the final policy gradient update rule:
VoJ(0) E,-,, LA-,, Va log (T))]
This rule guides the model to favor trajectories with positive advantages (better than average) and avoid those with negative ones, making meaningful advantage signal crucial for effective learning.
3.2 GROUP RELATIVE POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO) is a prominent RLVR algorithm that forgoes a learned critic network, which is key component of the PPO algorithm (Schulman et al. 2017). Instead, it constructs an empirical, on-the-fly advantage signal by generating a group of n trajectories T; = (T } for each prompt Py-
3

---

Published as a conference paper at ICLR 2026
To optimize, GRPO maximizes objective function based importance sampling ratio. For given trajectory TJ.s E Ts, we define the probability ratio for each token jj,,t at timestep as:
To (oooos..elP.... o.,,ce))
(..0 (0)
(4)
00 (og,iel P,, )j,,,())
where of represents the sequence of tokens. The objective function is defined as:
(0) = EP,,
Σ 1=1
min (1) clip (1e0 1-)00,16,,, -, +
(5)
GRPO's core lies in its advantage normalization, where single advantage AJ.5 is computed for each trajectory T3.s and applied uniformly across all its tokens:
Á R((T,)) MTg
(6)
TT, Estable
where HT - is the mean reward of the group, and dT, is its standard deviation. The policy is then updated using this uniform advantage signal across all time steps of the trajectory.
However, while this design is computationally efficient, its reliance troduces critical flaws. The first, and for our purposes the most detrimental, is the problem of credit misassignment for fragile trajectories. In the context of complex mathematical reasoning, trajec- tory is often "fragile"; single error can invalidate a long sequence of correct logical steps, thereby resulting in minimal zero final reward. Under this reward structure, the negative feedback de- rived from this outcome is uniformly distributed across all tokens in the sequence. Consequently, the vast majority of correct and valuable reasoning steps are unduly penalized, which actively impedes the model's acquisition of the long-form logic required for challenging problems.
single outcome reward in-
Furthermore, the reliance on the formulation in Equation 6 gives rise another critical flaw: the signal collapse problem (Yu et al., [2025 Xu & Ding, 2025). This issue materializes in what we term a "null-signal group". . a scenario wherein all trajectories happen to share the same reward. In such cases, both the numerator (R(T),s) (TT,) and the denominator (OT,) of the advantage calculation nullify. This inevitably results in A,, = for every sample, which constitutes a catastrophic loss of the learning signal that leads to the stagnation of exploration.
4 THE HIPO FRAMEWORK
4.1 CORE MECHANISM: ON-POLICY HINT
The core mechanism of HiPO constructs a contrastive learning signal by juxtaposing the policy's unaided exploration with its hint-guided exploration. HiPO enriches low-signal batches by using rare successes to generate high-signal replacements for completely unlearnable groups. This process unfolds in phases for each prompt P3 within given mini-batch.
First, the policy attempts to solve the prompt without assistance, generating a standard Group of n trajectories, Torig.i- This group faithfully reflects the model's current capabilities.
Torg..j {rI....,7, Trd where
(=).))
From this initial generation, we identify specific low-performing groups as "Near-miss Groups" Tacar-miss.j. which defined as those where the success rate of rollouts falls below half, and "Un- learnable Groups" Tuull-signal.j.. where the reward variance is zero. The upper portion of Figure 2 visually represents Tnear-missjj. characterized by high proportion of failed rollouts (red) and scarcity of successful ones (green), which typifies the challenge in complex reasoning tasks.
To counteract this, employ guided exploration strategy. First, on-policy hint pool is structed from the set of all successful trajectories within the near-miss group: Hpool, = {r e
Torar miss.j R(7) 1}. The core of this strategy is capitalize these successes generate
4

---

Published as a conference paper at ICLR 2026
Problem
Let ABCD be tetrahedron such that AB-CD ...is divisible by the square of any prime. Find m-n-p.
Pi
Hlin
problem tetrahodron with spocific odge lengthe. The question asking for ... which is equal. Then expeess that
Okay, so
Near-miss Group
Completion
Hint Group
distunce in the form ...so volume Altematively, perhaps compate the ... Yes. So thar's correct.
Pi
So steps is 104.
the
(04-2amwee??
Figure An illustration of the HiPO mechanism.
a new, high-signal Hinted Group, Thim..y- To ensure diversity within this new group, each trajectory generated using unique hint. This hint is created via two-stage sampling process: first, source trajectory Tsoure is uniformly sampled from Hpool.j. Second, prefix ratio is sampled from dis- crete values the range [0.05, 0.45] with a step of 0.05 determine hint length This procedure yields unique hint, H = Prefix(Tsouca, k), which represents the initial steps of a successful attempt. illustrated Figure 2 this process corresponds randomly truncating successful trajectory. The final trajectory is then generated conditioned on the hint being appended the original prompt: T: (P @ H,a)- A detailed discussion of the hint generation strategy is provided in Appendix D
[p- T JJ-
The theoretical underpinning of this method its principled resolution of sample inefficiency. A naive alternative for improving sample efficiency might involve augmenting the training data with off-policy expert trajectories drawn from distribution Tg. This approach, however, suffers from two fundamental drawbacks. The most critical is practical one: the requisite high-quality expert data is often prohibitively expensive or simply unavailable, as its acquisition demands either ex- pre-existing, superior "teacher" model. This reliance external
tensive manual annotation supervision contrasts sharply with self-sufficient, bootstrapping methods like our own. Secondly, even if such data were accessible, its introduction engenders significant distributional mismatch between the agent's policy and the expert policy This discrepancy is a well-documented cause of training instability, hindering reliable convergence by destabilizing the gradient estimator.
Algorithm The Hint Mechanism of HiPO
Input: A prompt P, current policy no. group size
Output: An augmented group trajectories.
ITa 2: Torig + Generate n trajectories for using policy ng.
1: procedure
3: Hpool +- {T E Tarig R(T) 1}.
4:
IF 0 < 1Hyool < n then
5:
Thim 0.
6:
FOR 4:
1,...,, do H +- ExtrackRandoMHin((//eoo
7:
8:
Generate T'e from P 4 H and add to Thint-
9: end FOR
10: RETURN Thint
end IF
12: end procedure
Ultimately, the policy update leverages data from both the Original and Hinted groups. This trans- forms a potentially degenerate learning signal into a well-posed, contrastive optimization problem. This self-teaching process, formalized in Algorithm T
is strategically activated to create this contrast
5

---

Published as a conference paper at ICLR 2026
precisely when it is most needed. The hint-generatin phase is triggered for what we term "Near- miss Groups": those where the success rate is greater than zero but falls below half. This targets the critical scenarios where initial successes model on the cusp of a breakthrough but still requires guidance. The resulting juxtaposition of the low-success-rate Original Group and the high-success-rrate Hinted Group creates highly infor- mative advantage signal: a positive advantage reinforces the completion of nascent but promising reasoning paths, as demonstrated in the Hinted Group, while negative advantage penalizes iden- tifiable failure modes that persist even with guidance. HiPO thereby offers more principled and efficient path toward mastering complex reasoning.
present but fragile, representing moments where the
4.2 How HIPO CREATES DENSE LEARNING SIGNALS FROM SPARSE REWARDS
We frame complex reasoning under sparse rewards problem of leveraging the structure of suc- cessful trajectories. However, frequent failures often lead to signal collapse, which is a scenario where zero reward variance within a group nullifies the advantage estimate and halts learning. To overcome this, HiPO extracts intermediate states from rare successes to serve as "hints" for initiat- ing new, guided trajectories. This approach transforms the difficult task end-to-end discovery into the more manageable one of completing partially successful reasoning path, generating dense learning signal from both unaided and hint-guided rollouts.
HiPO's refines the exploration strategy by modifying the initial state distribution for subset of rollouts. We define the set of successful trajectories from the unaided exploration phase as Tacar-miss {T E Tncar-miss R(T) = 13. We define a hint as an intermediate state 8y within a successful trajectory, where 8 E r for some E T near-miss" Trajectories in the Hinted Group, Thim. then generated by sampling from the model Ty conditioned on the concatenation of an original prompt P3 and hint H. This process is formally expressed as 7' ( P3 + H), where H is state drawn from successful trajectory. This acts form of value-guided exploration. While the optimal value function V* is unknown, states from empirically successful trajectories serve as effective proxies for high-value states. This method also prevents signal collapse by diversifying re- wards within a batch, which averts the vanishing advantage estimate Å, from adentically-rewaree trajectories. By ensuring signal diversity, HiPO avoids the catastrophic loss of learning signal and the subsequent stagnation of exploration.
To counteract this, HiPO implements strategic batch replacement. Let Borig denote original groups generated in the unaided exploration. The method sources successful trajectories from Tnear-miss within this batch to generate Thint- These groups then strategically replace the Tuull-signal that offer no learning gradient. This process is formalized as:
BIFO A (Borig T signal) Trina-
(8)
The gradient estimator over this optimized batch is:
Vo min (1)7 clip (0) (
θIPOO ETwBBrro
(9)
where A, is the advantage estimate and r is the importance sampling ratio. For any trajectory in the augmented batch BiiFO. its advantage is in its respective group. By replacing, this formula- tion decomposes the learning objective into a structured set of signals. The gradient is shaped by four distinct, high-value scenarios. Rare, successful trajectories remaining in Burig receive strong positive signal, anchoring the policy. In contrast, failed trajectories remaining in Borig are penalized, though their signal may contain "near-miss" paths. Successful trajectories in Thin form the core learning signal by salvaging near-misses from high-value states. Most critically, failed trajectories in Tim provide clear, high-quality negative signal, precisely penalizing deviations from a known- good path. By disentangling the learning signal, HiPO effectively transforms the learning problem from one of sparse rewards and catastrophic signal loss structured task of guided completion.

---

Published as a conference paper at ICLR 2026
5
EXPERIMENTS
5.1
EXPERIMENTAL SETTING
We train our model on the DAPO dataset (Yu et al. matical problems with integer answers sourced from the AoPS community. For evaluation, we test our model across comprehensive suite of recent mathematics competitions, including the AIME 2024, AIME 2025, BRUMO 2025 (Balunović et al., 2025). HMMT Feb 2025 (Balunović et al.) 2025), CMIMC 2025 (Balunović et al. 2025), and Apex 2025 (Balunović et al. 2025). Perfor- mance is measured by the average accuracy across 32 generated samples per problem (avg@322, along with the majority vote accuracy (maj @ 32) and pass rate (pass@ 32).
2025). a challenging collection of 17K mathe-
Our approach is based on the Qwen3-8B (Qwen 2025) model, which we trained our model using the VeRL (Sheng et al., 2024) and using the Reoool framework (Feng et al., 2025 Lin & Xu 2023). ReTool is a reinforcement learning paradigm that teaches the LLM to utilize a Python code interpreter. It learns from outcome-based feedback over multi-tume interactions, with maximum of eight turns per math prompt.
Key training hyperparameters include learning rate of 1e-6, batch size of 96, mini-batch size of 12, and a maximum response length of 16K tokens. We adopt the Clip-Higher strategy (Yu et al.] 2025), setting flow 0.2 and Shigh 0.28. At each training step, the policy is updated using rewards calculated from 16 sampled responses per prompt. For evaluation, a maximum response length of 32K tokens is used.
5.2
EMPIRICAL COMPARISON WITH BASELINE
Table 1: Comparison of HiPO, GRPO and DAPO on five benchmarks using avg @3,, pass @ 32, and maj@32 @32. Averages are shown in the last column. Bold indicates the better-performing method for each metric. For the DAPO, the generation batch size is set to 192, and the maximum number of generation batches is
AIME 2024
AIME 2025
BRUMO 2025
Model
@vg 32
pass
ma
avg
pass @ 32
maj@32
uvg
pass @
maj @:2
Qwen3-8B
54.7
60.0
47.6
84.9
63.3
55.5
GRPO
72.1
60.0
87.1
70.0
41.7
52.1
DAPO
76.0
89.5
60
63.7
87.9
47.8
56.6
63.3
HiPO
76.7
89.8
63.3
66.1
88.3
76.7
46.6
56.6
63.3
HMMT 2025
CMIMC 2025
Average
Model
avg 32
pass @ 32
maj @2
avg 32
pass @:
maj@33
avg 09
pass
maj @:32
Qwen3-8B
14.0
38.7
40.0
37.0
75.0
60.0
36.7
67.9
56.0
GRPO
28.6
48.2
46.7
43.5
75.0
55.0
49.8
70.8
57.0
DAPO
31.4
48.4
43.3
49.9
80.0
52.5
53.7
72.4
57.8
HiPO
30.8
49.2
40.0
53.8
80.0
65.0
54.8
72.8
61.7
Our experiments demonstrate that HiPO consistently outperforms the GRPO and DAPO Yu et al. 2025) baseline on challenging mathematical reasoning benchmarks. As shown in Table HiPO achieves superior aggregate scores on all primary metrics. Specifically, it obtains an avg @ 32 of 54.8 compared to GRPO's 49.8. representing significant improvement of +5.0 pp. This aggre- gate strength is underscored by notable consistency, as HiPO outperforms GRPO on the avg @vg@32 metric across all five benchmarks. Notably, DAPO's dynamic sampling incurs significant compu- tational overhead, as each training step requires processing candidate batch of prompts that is multiple the size of the batch ultimately used for the gradient update. As shown Appendix B DAPO's dynamic sampling incurs significant computational overhead, consuming approximately 4x the prompt volume of HiPO to achieve comparable performance.
The performance disparity is most pronounced the CMIMC 2025 dataset, where HiPO achieves a substantial + +10.3 pp gain. Further significant improvements are observed on BRUMO 2025 (+4.9 pp) and AIME 2024 (+4.6 pp), underscoring the robustness of HiPO's advantages. While

---

Published as a conference paper at ICLR 2026
Pissgg Curve fer AIME24
PrsugE Curve fer AIME25
PasIgE Curve Brumo3825
Mark
Madk
Mark
Den
Deed
(a) AIME 2024
(b) AIME 2025
(c) BRUMO 2025
Pawatri Ctrrn for benmt Iub, 2015
Pasege Carre fur CMIMC 2025
Pawegt Curra for Apes, 2015
Oveclii
Ovee1-43
Hadde
Qoech
(d) HMMT 2025
(c) CMIMC 2025
APEX 2025
Figure 3: Pass @k k curves for different benchmarks.
GRPO is competitive certain metrics, HiPO's consistent and significant lead, particularly the most difficult datasets, suggests it learns more robust and generalizable reasoning pathways.
These findings are further supported by the pass @k performance (Chen al. 2021 curves in Fig- 3 The plots generally indicate an advantage for HiPO in sample efficiency, as it often achieves higher pass rate at lower values of This trend is most pronounced on the exceptionally challeng- ing Apex 2025 dataset, where the performance gap between HiPO and GRPO widens dramatically as k increases. On this benchmark, HiPO's pass @ score is nearly double that of the baseline, which suggests that the benefits of its signal reshaping mechanism are particularly salient on prob- lems where successful trajectories are rare. While the performance of both methods converges at higher values of some benchmarks, such AIME 2024, HiPO maintains clear and consistent performance lead across most other datasets, including Brumo 2025 and AIME 2025, highlighting the potential robustness of our approach. Furthermore,
empirically demonstrate HiPO's robust- ness in ultra-sparse reward regimes by addressing the cold start problem on the hardest subset the Omni-Math dataset (Gao al. 2024 in Appendix C
5.3
TRAINING DYNAMICS
Policy Entropy
Tool Call Turns
ERPD
GAPO
0.8
HPO
OM--oltcy-Hitt
Beraaii-3ed5
flaerstio00055
Eoratiioccc0
ratiotio 0550
0.6
0.5
0.3
20
69
1.50
120
20
80
100
120
Step
Step
Actor Entropy
Mcan Number of Turns
Figure 4: Training dynamics of actor entropy and mean number of turns.
To understand the mechanisms driving HiPO's superior performance, we analyze two key metrics
throughout the training process: policy entropy and the average tool-use turns. As plotted in Fig-
ure 4 Figure 4a tracks the policy entropy, measure of policy stochasticity. The plot shows that
HiPO (cyan) consistently maintains a higher level of entropy than GRPO (blue). This provides empirical evidence that HiPO successfully mitigates the problem of exploration stagnation (Cui
8

---

Published as a conference paper at ICLR 2026
AIME2024
AIME2025
0.70
0.75
0.55
Av0916
0.65
AV0916
0.55
0.50
GAPO
HPO
0.45
HPO Off-Polcy-Hint
0.0
Oft-PoicyHinn
fieratioo0005
fix Mk-ratie-0d0 08B0
fix ratio 0d80
150
120
0.25
40
Step
Step
(a) AIME 2024
(b) AIME 2025
Figure 5: Ablation on hint ratios. (a-b) Training curves on AIME 2024 and AIME 2025 bench- marks compared across different hint strategies. (c) The evolution of policy entropy during training. (d) The average number tool-use use turns throughout the training process.
al. 2025 Wang et al. 2025). GRPO's penalization of near-miss trajectories suppresses explo- ration, causing the policy to converge on suboptimal, low-diversity strategies. In contrast, HiPO's on-policy hinting mechanism stabilizes the learning signal, promoting exploration of diverse reasoning pathways and preventing premature policy collapse. This sustained diversity is crucial for discovering more effective problem-solving strategies. The case study in Appendix (rrendixx]prroot provides a concrete example of this behavioral difference. It shows that the HiPO agent identifies an alterna- tive algebraic structure to simplify the problem, whereas the GRPO agent follows a more direct but computationally complex path that leads to incorrect result.
Complementing the entropy analysis, Figure 4b illustatees the model's capacity for complex, multi- step reasoning by plotting the tool-use turns. A stark divergence emerges: the average number of turns for HiPO trends significantly upwards, indicating that the model learns to engage in longer interactions with the code interpreter. Conversely, the GRPO baseline struggles to increase, show- ing minimal growth. This suggests that GRPO's flawed credit assignment makes longer reasoning chains brittle and risky, incentivizing the model adopt simplistic strategies. HiPO, by providing scaffold for exploration, enables the model to successfully learn and execute the longer, more com- plex reasoning chains necessary to solve challenging mathematical problems. Taken together, these dynamics provide compelling evidence that HiPO directly counteracts exploration stagnation. By preserving policy diversity, enables the model discover and master the longer, more complex reasoning chains necessary for advanced problem-solving.
5.4 ABLATION STUDY: ANALYSIS OF HINT RATIO
To validate the necessity of our dynamic hint strategy, we evaluate low-ratio variant with p 0.05 to represent minimal guidance, and a high-ratio variant with = 0.80 to simulate excessive guid- that HiPO consistently outperforms static hint strategies. The mechanisms driving these results are revealed in the training dynamics. The high-ratio variant (p 0.8) exhibits low entropy and minimal tool-use turns. This indicates that excessive guidance restricts exploration, trapping the model in local sub-optima where it relies simple completion rather than learning robust reasoning. Conversely, the low-ratio variant (p = 0.05) maintains high entropy but fails to increase tool usage. This suggests that while the model is actively exploring, the lack of sufficient scaffolding prevents from effectively discovering complex, superior trajectories. HiPO achieves the highest tool-use frequency while maintaining healthy entropy, demonstrating that our dynamic strategy successfully strikes a critical balance between exploration and exploitation, guiding the model toward sophisticated solutions without sacrificing diversity.
ance. Figure
5.5 ABLATION STUDY: ON-POLICY vs. OFF-POLICY HINTS
To determine whether the efficacy of HiPO stems from the content of the hints or the on-policy nature of their generation, compare HiPO against "Off-Policy Hint" baseline. We simulates

---

Published as a conference paper at ICLR 2026
standard teacher-student distillation by collecting successful trajectories from the base model, extract the first 20% of tokens as static hints.
As shown in Figure[5] the Off-Policy Hint method underperforms both HiPO and the GRPO baseline, a failure explained by the collapse of tool-use turns in Figure 4b). This collapse occurs because static hints derived from the untrained base model are inherently simplistic. Enforcing these "weak" priors suppresses exploration, trapping the policy in local optima where it mimics trivial solutions rather than developing the complex reasoning chains required for harder problems. In contrast, HiPO establishes a self-reinforcing curriculum where hints evolve dynamically with the policy. As evidenced by rising tool usage, the "teacher" improves alongside the "student," enabling autonomous mastery without reliance on stronger external models.
5.6 SAMPLE EFFICIENCY ANALYSIS
Treinin Reward Brectting 3.
Learibte Group Ruti
Aug Augmenttdd Promptt Number
Prsprtel
(a) Reward curves
(b) Group Proportions
(c) Augmented Prompts number
Figure 6: Visualization of HiPO's training dynamics. (a) Training reward curves for HiPO and GRPO. The HiPO represents the reward after group substitution. (b) The proportion of learn- able groups versus ineffective groups within a batch, before (left) and after (right) applying HiPO. (c) The curve of the number of prompts enhanced by the HiPO method during training.
HiPO is designed to enhance sample efficiency by resolving the "signal collapse" problem endemic to group-hased methods. Our training analysis in Figure&Jemmine emmpiricall confirms this. GRPO wastes computation on "unlearnable groups", where a staggering 82.5% of all groups (the sum of 0/16 and 16/16 groups) yield a null gradient. In contrast, our hint mechanism transforms these into valuable learning opportunities. As shown in Figure 6b HiPO elevates the proportion of learnable groups to 75.8%. This self-sustaining process remains stable throughout training, evidenced by consistently high number of augmented prompts (Figure (cb, which ensures a dense and reliable learning signal.
The impact of this efficiency is evident in the training reward curves shown in Figure 6a policy (blue) consistently outperforms the GRPO baseline, demonstrating superior learning. Criti- cally, this performance gain is achieved on a stable and challenging curriculum. The HiPO hint curve (orange), representing reward from hint-guided rollouts, remains consistently low. This signals that HiPO continuously drives the model to learn from difficult, partially-comllted trajectories rather than saturating on easy ones. Consequently, overall capability rises (blue curve) while the learning signal stays challenging, ensuring robust and sustained improvement.
HiPO's
6 CONCLUSION
We addressed the critical challenge of exploration stagnation in RLVR by introducing HiPO, framework built on the paradigm of endogenous Self-Hint. HiPO transforms rare, stochastically- found successes into an on-policy curriculum, converting sparse reward landscape into dense, contrastive learning signal. This self-teaching mechanism not only significantly outperformed strong baselines across challenging reasoning benchmarks but also demonstrably fostered higher policy en- tropy and more complex reasoning chains. Our work establishes that model can effectively boot- strap its own learning from endogenous successes, reducing the reliance on external expert data.
The principles of HiPO pave the way for more scalable and autonomous paradigm of skill acquisi- tion. By demonstrating that models can become active participants in their own education, our work provides robust foundation for training self- celf-improving agents capable of mastering intricate tasks in any domain where success is a rare and hard-won event. This highlights a powerful path toward more capable and independent AI.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (U23B2055, 62276077, 02350710797 in part by the Guangdong Basic and Applied Basic Research Founda- tion under Grant 2024A151501 1205, andin part by Shenzhen Science and Technology Program (KQTI2204407291021 54066, ZDSYS2003066609 123008).
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. POLARIS: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. hRRp::tt::///kkk 1/. github. (tchub..o/bloo/222///o....
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on uncontaminated math competitions. arXiv preprint arXiv:2505.2328) 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22611 2025.
Team DeepSeek. DeepSeek-RI incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633-638, 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. ReTool: Reinforcement learning for strategic tool use in LLMs. arXiv preprint arXiv:2504. 11536, 2025.
Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint 4rXiv:2410.07985, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 1.15144, 2024.
Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, and Jingzhao Zhang. QuestA: Expanding reasoning capacity in Ilms via question augmentation. arXiv preprint arXiv:2507.13266 2025a.
Xuefeng Li. Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrater rl. arXiv preprint arXiv:2503.23383, 2025b.
Heng Lin and Zhongwen Xu. Understanding Tool- Tool-Integrated Reasoning. vrXiv:2508.19000 2025.
arXiv preprint
Team OpenR1. OppenR OpthRP--aaath220------- A large-scale dataset for mathematical reasoning.. 2025, URL
-M
Martin I. Puterman. Markov decision processes. Handbooks in operations research and management
science, 2:331-434, 1990.
Team Qwen. Qwen3 technical report. arXiv preprint arXiv:2505.093380 2025

---

Published as a conference paper at ICLR 2026
Matthew Renze and Erhan Guven. Self-reflection in Ilm agents: Effects on problem-solving perfor- mance. arXiv preprint arXiv:240S.06683 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint XXXv:24022033300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprint Xrviv: 19256, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. arXiv preprini 2rX1v92506.01939, 2025.
Zhongwen Xu and Zihan Ding. arXiv:2509.13232. 2025.
Single-stream
policy optimization.
arXiv preprint
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. DAPO: open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Kaiyi Zhang. Ang Lv, Jinpeng Li, Yongbo Wang. Feng Wang, Haoyuan Hu, and Rui Yan. StepHint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint arXiv:2507.02841, 2025.
12

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used large language models (LLMs) only polish grammar and improve the clarity of the manuscript. All research ideas, experiments, and analyses were conducted by the authors.
B COMPARISON WITH DAPO
We compare HiPO against DAPO Yu ct al. 2025). strong baseline employing dynamic sampling tackle sparse rewards. To accurately reflect computational cost, Figure evaluates performance against prompt volume.
AIME2025
AIME2024
0.70
0.55
0.70
2vg016
Nrg016
0.50
0.60
0.50
0.55
0.45
DAPO
DMP
GRPO
0.55
HPO
10000
20000
30000
43000
30000
20000
30000
40000
Prompt Valume
Prompt Volume
(a) AIME 2024
(b) AIME 2025
Figure 7: Performance curves (Avg@16) 16) AIME 2024 (a) and AIME 2025 (b) plotted against the total Prompt Volume. The trajectories correspond to GRPO (blue), HiPO (cyan), and DAPO (brown).
As shown in Figure 7 HiPO achieves superior convergence with fraction of the data consumption. The extended tail of DAPO reveals the high cost of its dynamic sampling mechanism: to secure positive rewards, it must perform extensive rejection sampling, consuming approximately 4x the trajectory budget per update step. While DAPO relies on brute-force enumeration to locate sparse successes, HiPO employs signal amplification. By repurposing single stochastic success into group-wide hint, HiPO extracts dense gradients from standard batches, eliminating the need for wasteful resampling and establishing significantly more efficient learning paradigm.
C ROBUSTNESS TO ULTRA-SPARSE REWARDS
A primary concern regarding RLVR in complex domains is the "cold start" problem, where ultra- sparse rewards might starve the model of learning signals. To empirically evaluate HiPO's robust- ness in such regimes, we conducted stress test using the top 10% most difficult problems from the Omni-Math dataset (Gao et al. (024)) to train a Qwen3-8B base model. This setup simulates near-zero success rate environment where standard exploration often stagnates.
The results in Figure 8 demonstrate that HiPO effectively mitigates signal scarcity through batch- level signal broadcasting. As illustrated in Figure Ma), the initial generation phase (blue bars) is dominated by groups with zero successes, which would typically yield a null gradient. However, HiPO identifies rare stochastic successes present within the global batch and repurposes them to generate hinted groups. The cyan bars show drastic reduction in these unlearnable groups after intervention, indicating that the method effectively amplifies sparse signals by substituting failed trajectories with actionable, hint-guided ones. This mechanism ensures that even when success is statistically rare, the optimization process remains supplied with valid gradients. Figures ((b) and (c) confirm that this signal rectification translates into tangible learning progress, with the model maintaining a steady upward trend on both AIME 2024 and AIME 2025 benchmarks despite the extreme difficulty of the training distribution.
13

---

Published as a conference paper at ICLR 2026
Group Comrattionn
AIME2024
Detd AIME2625
ren al ove
Ceet Pat Fat
0.00
9N0
3163
9103
(a) Failure Group Reduction
(b) AIME 2024
AIME 2025
Mart--ard). (a) The number of "Total Failure" groups (0/16 success) in batch before (Blue) and after (Cyan) HiPO intervention. HiPO drasti- cally reduces the proportion of null-signal groups (b-c) Despite the extreme difficulty, the model maintains steady upward learning trend on AIME benchmarks, confirming that the feedback loop remains unbroken.
Figure 8: HiPO in Ultra-Sparse Regimes
D RANDOM HINT SETTING
Miat 0.2
Bint 0.5
Mins 0.7
Bivt 0.9
1150
1754
1754
1150
150
12:00
100-
500
200K
1000
250
200
Number Strpoean
(a) Hint 0.2
(b) Hint Ratio 0.5
(c) Hint Ratio = 0.7
(d) Hint Ratio 0.9
Figure 9: Distribution of the number of correct responses over 16 attempts (6) for different fixed hint ratios. These plots illustrate how increasing the hint ratio shifts the distribution towards perfect scores, inadvertently creating less effective learning signals.
As empirically demonstrated in Figure hint ratios exceeding 0.5 cause sharp increase in the frequency of perfect-score (16/16) groups. This outcome is detrimental because induces "signal collapse." In group where all trajectories succeed, the reward variance is zero, which nullifies the advantage estimate (A,,, 0) for all samples and causes the policy gradient vanish. Furthermore, excessively long hints reduce the problem to trivial completion task, preventing the model from learning the complex intermediate reasoning steps and suppressing meaningful exploration of the solution space.
Using single, fixed hint ratio creates critical credit assignment pathology, which can be shown formally. Consider a hinted group T₃ = {T......., Ty..} where all trajectories share an identical prefix H (0n, -.- 0k-1) because they derived from single source trajectory with fixed ratio. GRPO, the total policy gradient for any token o within this shared prefix (t < k) the sum of its gradients from each trajectory in the group. Since the advantage ÅG, is constant for all tokens within trajectory Tj.ú, and the conditional probability (o(o)(s)) identical for all trajectories at this step, the total gradient for token o2 is:
VoJ(Or)) Σ Voggooog
Σ
Ag.
V (o((ss.)..
(10)
The core issue lies in the summed advantage term. By definition, the advantage Å,,- (R(T),) Mr, ) + c), where Mr; is the mean reward of the group. The sum of all advantages within the group is therefore:
or, + (R(T,))) -(a))) or, te (6r R(TS.)) n Mr,) =0 (11)
Σ Ag. 1=1
14

---

Published as a conference paper at ICLR 2026
This demonstrates that the aggregate advantage signal for any token in the shared prefix is precisely zero, resulting in null gradient. The model is thus unable to learn that the shared prefix is a valuable reasoning path. Randomly sampling the hint ratio breaks this pathological symmetry by ensuring the prefixes are not identical, which allows for meaningful, non-zero gradient to be assigned.
E GENERALIZATION TO OUT-OF-DISTRIBUTOR TASKS
We evaluated HiPO on HumanEval to assess its generalization capabilities beyond the mathemat- ical domain. Despite the training data consisting exclusively of mathematical gobblems-akinng code generation a fully out-of-distribution (OOD) task, HiPO achieves a Pass@) I score of 71.3%, outperforming the Qwen3-8B base model (70.7%). This demonstrates that the enhanced reasoning patterns fostered by HiPO are robust and transferable, yielding performance improvements even on tasks unseen during training.
Table Pass @1 (%) on the HumanEval benchmark. HiPO demonstrates positive generalization on this OOD coding task, outperforming the base model.
Model
Pass @
Qwen3-8B (Base)
70.7
GRPO
71.3
HiPO (Ours)
71.3
F CASE STUDY
To provide concrete illustration of the behavioral differences between HiPO and the GRPO base- line, we present a qualitative case study. The following Table [3] Table f and Table 5 contrasts the reasoning trajectories of both models insight and precise execution. This side-by-side comparison highlights HiPO's ability to perform strategic exploration to simplify the problem's core, versus GRPO's tendency to get locked into linear, computationally intensive, and ultimately fragile reasoning path.
challenging mathematical problem that requires both
As the comparison demonstrates, HiPO's ability perform strategic exploration leads to a more robust and efficient solution path. In contrast, the baseline model's linear, brute-force approach results in fragile reasoning chain that ultimately collapses due its own complexity, perfectly illustrating the "exploration stagnation" that our method is designed to overcome.
15

---

Published as a conference paper at ICLR 2026
Table 3: A second comparative case study. This table illustrates HiPO's strategy of leveraging external tools (a code interpreter) definitively verify its algebraic insight. In contrast, the GRPO baseline performs a robust but purely algebraic verification, re-solving the problem with a different, more complex method ensure correctness.
Question
Positive real numbers x and y satisfy y² = 2: and (y x) 4yj2 What is y?
HiPO Trajectory (Our Method)
GRPO Trajectory (Baseline)
Response:
Response:
Wait, (y x²² 4y² + taking square roots gives y z| 2y.
...Let's take the square root of both sides... we get ly x 2y.
...Case 1:
2y x y. But x and y
2y are positive, so this impossible.
Case 1:
-y. But and
are positive, so this can't possible...
Case 2:
- 2y - 3y-
Case 2: y
2y x = 3y. So this is the
So from this, x equals 3y. Now, substitute this into equation 1. y² = (3y)² + y²² = 9y² b 9 So y=9. Then x=3y=27. Therefore, + 27 = 36.
valid case.
Substitute
By into... y²² -3y)²
9y² ...
...00 y = 9.
Then
3y 3 x 9 27. Therefore,
x+y 27 + 9 36.
But perhaps the code solve this system of equations.
Wait, but let me check again if there's other way to approach this...
Alternatively, maybe expanding equation (y-x)² y² 2ry + x² 4y². 2ry 3y²
import sympy sp
x, y sy.byml s ('x y', positive- True)
eql = sp. Eq x**2)
Let me treat this as quadratic equation in x... Using quadratic formula:
eq2 = Ep.Eq( (y x) +2, 4xy+*2)
solution solvolve (eql, eq2),
= 2y + (2y)² 4(1)(33y²)
y))
2y = 1 = [2y + 4y]/2.
result solution [0) [0] + sclution
...only x 3y valid. So same result as before.
[0] [1]
So that's correct.
result is 36
All approaches lead x=27, y=9, sum 36. The
Yes, the code gives 36. So that confirms the
final answer is
36
answer. The final answer 36
16

---

Published as a conference paper at ICLR 2026
Table 4: A case study in problem restructuring. The HiPO trajectory shows the use of a key algebraic insight simplify the problem's 's structure, leading to straightforward solution. In contrast, the baseline's linear approach gets entangled in more complex and error-prone calculations.
[Question]
Find the number of triples of nonnegative integers (a, c) satisfying + 6 300 and
a²b c bb aa + ebb 6, 0000000000 000.
HiPO Trajectory (Our Method)
GRPO Trajectory (Baseline)
Response:
Response:
...This leads to the relation 'abc 100(ab be +
... ...This leads to the relation 'abc 100(ab be + сa) -,00000000" But how can proceed from here? Let me think.
ca) = -2,000,000"
This seems a bit abstract. How can proceed from here?
Alternatively, maybe think of the left-hand side as (q-100x6b-100(cc-000)
Let me check the where one variable is zero first... This gives 6 solutions. Now, let me think about other cases...
(a 1000 - 100) - 100) = O.
P'(c 100) 100 - 300c + 20000).
Oh! That's big simplification! .h.This is counting problem I can solve with inclusion- exclusion.
Let me factor the right-hand side...
(' 100))000-00 = 100 - 200) 100).
Number of solutions is: 3 x 201 - 3 x 1+1= 601.
... ..e discriminant is (S" 200) which is a perfect square. Exactly!
But to make sure, perhaps code it.
def count solutions ():
...So for each from 1 to 199, there are two or- dered triples... except
Solutions where a=100: b+c -200 -> 201
100. So total solu- tions + 198 x 397. Wait, this is eritical point... This is big mistake the earlier reasoning...
count - 3 201
A Subtract overlaps where amb
=100
The total number of solutions is 6 +-7 403.
count
The final answer is 403
$ Add back the case where a=b=c
=100
count
return count
print (count_so wol.uouutooosooo )
Interpreter Output: 601
The code confirms my reasoning. The final an-
swer is 601
17

---

Published as a conference paper at ICLR 2026
Table 5: A case study on strategy selection. HiPO correctly identifies the problem as best suited for computational brute-force search, leading to the correct answer. The baseline attempts a more complex analytical shortcut, but subtle reasoning error leads to an incorrect result, highlighting the robustness of HiPO's strategy.
Question]
Let 2 2 be an integer. Call positive integer b-eautifull if it has exactly two digits when expressed in base and these two digits sum to vn. For example, /8I. Find the least integer
13-eautiful because for which there are more than ten
81 = 6 3 and
b-eauiiful integers.
HiPO Trajectory (Our Method)
GRPO Trajectory (Baseline)
Response:
Response:
Okay, need to solve this problem... For num- ber have exactly two digits in base b, it must satisfy b S < 82. Let the digits be dl and d2. + and dl + d2 = /n.
Okay, let's tackle this problem... Let n = a +
c. The condition is Vab + + c. Let aa, which gives atc. Then s² ab + (a-a), =(a-1) b-r
Then n = So the plan is: For each base b starting from 2 upwards... iterate over all possible dI and d2... check if (dl+d2)2
For a to be an integer, b-1 must divide s(s-l).
Since and s-1 are coprime, b-1 must divide or s-1.
and count them. This seems suitable for code.
...This seems to imply there are very few solu- tions for any b.
def find min base() b - 2 while True:
Wait, let me check the example. b=13, n=81. Digits are 6,3. s=9, b-1=12. Here, s(s - = 8 72. 12 divides 72, but 12 does not di- vide 9 or 8. Ah! My reasoning was incorrect. So need to check for each if b-1 divides s(s- 1). Let me try to code this.
count
for dl in range (1, b):
for d2 in range (0, b) dl
d2
sum_d
if sum + sum_
Tries write code but it has syntax errors def count_beautifuu (b) #...
n:
count +-
if count 10:
return b
code fails to run
b +- result find min (ase print (result)
this point, it's clear that without writing the code, it's hard to proceed. Let me try to estimate for b=61... This is getting too complex.
Interpreter Output: 211
The answer is 121
Wait, the code returns 211? That seems high. Let me check for b=3.
d1=1, d2=1: n=4, sum=2. 2-44 Yes. ...other pairs fail. So for b=3, count is 1. The code logic seems be working. So the answer 211
18


Reviews:
[{"review_id": "Vz1zX4G377", "reviewer": "Reviewer_eFHX", "summary": "The paper identifies and targets two core failure / difficulties in RLVR on long-chain mathematical reasoning tasks, namely the near-miss problem and exploration stagnation. It proposes HiPO, a self-bootstrapping framework that extracts partial prefixes (“self-hints”) from the few successful trajectories within a batch and reuses them to regenerate a new, more informative batch on the same prompts. By replacing low- or zero-signal groups with hint-augmented ones, HiPO effectively densifies the reward signal and makes GRPO-style optimization work even when success is rare. Experiments on recent difficult math benchmarks (CMIMC 25, BRUMO 25, AIME 24/25) show consistent improvements over GRPO/DAPO-style baselines.", "questions": "All experiments stay in math / verifiable QA style tasks, which are the friendliest setting for RLVR because of binary, automatic reward. As the authors claim the value of the proposed HiPO for general RLVR, how does the method work on tasks where rewards are delayed, noisy, or non-binary (code, tool-using agents, long dialogues)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper is techinically simple but effectively improves the sampling efficiency of RLVR in difficult scenarios where the reward is very sparse.\n\n- The proposed method is practically plug-and-play for GRPO-like group-based pipelines, and does not need expernal teachers' hints.\n\n- Empirical results on recent, difficult math benchmarks show clear gains.", "weaknesses": "- The proposed HiPO \"amplifies\" the rare successful runs. However, on very hard tasks or early in training where only very rare successful runs can be sampled, HiPO may cause \"over-exploitation\" of a small set of successful runs. The authors are recommended to discuss this possibility in the paper.\n\n- Comparison is mainly against GRPO/DAPO-like baselines; it would be important to see how HiPO fares against stronger exploration- or entropy-aware RL variants, or against pipelines that inject external hints. This would clarify whether “self-hint” is a better source of guidance than existing teacher-style hints, or just a cheaper one.\n\n- Some key designs (e.g., the ratio of the original null-signal group samples that are replaced by the hint samples) lack ablations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWGcggO5Se", "reviewer": "Reviewer_Wgcs", "summary": "HiPO is a novel RLVR framework designed for sparse reward reasoning tasks like mathematics. It overcomes issues in standard policy gradient methods through its \"Endogenous Self-Hint\" mechanism. This mechanism captures successful trajectory prefixes as on-policy hints, generating high-signal guided trajectories that transform sparse rewards into a dense, contrastive learning signal. HiPO significantly outperforms a GRPO baseline on five math benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The idea in this paper is simple yet efficient. \n3. This idea can maintain the exploration property.", "weaknesses": "1. When facing extremely challenging tasks, the paper doesn't clarify how HiPO handles \"total failure\" batches where all groups have 0% success. Since hints are sourced from \"Near-miss Groups\" (0-50% success), an entirely \"Unlearnable\" batch would leave the hint pool empty, breaking the feedback loop and preventing hint-guided trajectory generation. This \"cold start\" scenario is unaddressed. Could you please clarify the precise mechanism for handling a mini-batch where no successful trajectories are generated (i.e., all groups are \"unlearnable\" with 0% success)? Does the hint-generation step simply fail for that batch, and the model must rely on standard GRPO updates until a success is stochastically found? Or is there a different mechanism to source hints (e.g., from a global buffer of past successes)? \n2. A lot of hyperparameters need to be ablated and discussed: (a). Length range [0.05, 0.45] for the hint. (b). When to activate the HiPO, when $0<H_{\\text{pool}} < \\frac{n}{2}$.  \n3. The baselines are limited; please introduce recently published algorithms for comparison, e.g., DAPO. \n4. The authors introduced a mechanism through a two-stage sampling process to encourage the diversity of hints.  How to measure this enhanced diversity from the proposed approaches? Any other methods? \n\nMinors:\n1. In Figure 1, the authors should clearly state what is the meaning of numbers, e.g., 0.2, 0.5, 0.7, 0.9, though I understand from the later part: the length of the hint prompt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BZDgwd85Km", "reviewer": "Reviewer_DaaS", "summary": "The paper introduces HiPO (Hint-guided Policy Optimization), a novel framework for Reinforcement Learning from Verifiable Rewards (RLVR) aimed at improving large language models' (LLMs) performance on complex mathematical reasoning tasks. It addresses key challenges in existing methods like GRPO, including the near-miss problem, where nearly correct trajectories receive no positive signal, and exploration stagnation due to sparse rewards leading to policy collapse. HiPO's core innovation is an endogenous self-hint mechanism: within a training batch, rare successful trajectories are identified, and their initial correct prefixes (sampled at ratios between 0.05 and 0.45) are repurposed as on-policy hints to generate augmented groups for low-success or null-signal batches. This creates a dense contrastive learning signal by contrasting unaided and hint-guided rollouts, enabling the model to bootstrap from its own successes without external data.", "questions": "1. How robust is HiPO to success scarcity? For instance, what happens on datasets with <1% base success rates—does lowering the near-miss threshold (e.g., to 0 successes with synthetic prefix generation) maintain gains, or could this introduce off-policy drift? I am currently not convinced that the proposed mechanism can effectively solve the sparse reward issue in GRPO, and better baseline comparisons should be added.\n\n2. The hint sampling uses discrete ratios [0.05-0.45]; have you ablated continuous sampling or adaptive lengths (e.g., based on intermediate reward proxies)? Could longer hints (>0.45) be viable with variance regularization to prevent collapse?\n\n3. Can you provide results on stronger baselines like PPO (for direct RL comparison), DPO (to assess preference optimization alternatives), or recent hint-augmented methods such as StepHint (ICML 2025)? Specifically, how does HiPO perform relative to these on the harder benchmarks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "HiPO's self-hint paradigm is novel, which transforms rare successes into reusable on-policy guidance, avoiding off-policy mismatches and enabling true autonomy in learning complex reasoning chains. HiPO sustains 20-30% higher entropy than GRPO (Figure 4a) and doubles learnable group proportions from 17.5% to 75.8% (Figure 5b), directly validating its anti-stagnation claims. \n\nThe writing is generally clear, with intuitive visuals (e.g., Figure 2's batch augmentation) and case studies (Tables 2-4) that concretely show HiPO's strategic exploration (e.g., algebraic restructuring) versus GRPO. \n\nThe approach is timely and significant for dealing with sparse-reward domains like math competitions, where it yields domain-general improvements paving the way for scalable, data-efficient RLVR without curated hints.", "weaknesses": "1. While HiPO effectively leverages rare successes, its activation strictly requires at least one success per near-miss group (success rate >0 but <50%), raising concerns about performance in ultra-sparse regimes where batches might entirely lack successes, e.g., early training or harder curricula, potentially amplifying GRPO's signal collapse rather than resolving it.\n\n2. The fixed hint ratio range [0.05, 0.45] is justified in Appendix B to avoid signal collapse from long prefixes, but no ablation shows optimal tuning or impact on final performance, leaving hyperparameter sensitivity unclear.\n\n3. The experiments are strong on math but no evaluations on pure text reasoning or non-math tasks (e.g., code generation, planning) are provided, limiting claims of general RLVR applicability.\n\n4. The baselines comparisons are weak, with comparisons provided only with GRPO, omitting broader baselines like PPO or recent works (ike STaR (Zelikman et al., 2022) or Quiet-STaR (Zelikman et al., 2024)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ID9k3T6z9C", "reviewer": "Reviewer_Hyap", "summary": "This paper introduces HiPO (Hint-guided Policy Optimization), a framework for Reinforcement Learning from Verifiable Rewards (RLVR) that addresses the near-miss problem and exploration stagnation in long-horizon reasoning tasks.\n\nThe key idea of HiPO is endogenous self-hinting. When the model occasionally finds a successful trajectory, it extracts the initial correct steps (prefix) of that trajectory and reuses them as on-policy “hints” for future training. This turns a single sparse success into a dense, contrastive learning signal, allowing the model to bootstrap from its own rare successes.", "questions": "I would like to thank the authors for their work.here are a few concerns and questions:\n\n- Reward hacking: How do the authors ensure that the use of hints and dense rewards does not lead to reward hacking behaviors, where the model optimizes for superficial alignment with hints rather than reasoning improvement?\n\n- Quality of hints from rare successes: How can we guarantee that the hints extracted from rare successful trajectories are actually desirable? In practice, many successful reasoning traces can be unnecessarily long or include redundant steps. How does the method handle such cases?\n\n- Exploration limitation: Is there a risk that the discovered traces—and consequently the training signals—are limited to the reasoning patterns already found by the model, thus constraining exploration and generalization?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- HiPO directly targets signal collapse and credit misassignment, two critical issues \n- the idea of self hint is interesting\n- improved empirical results", "weaknesses": "- dependence on rare success\n- lacks a formal analysis of convergence properties\n- computational overhead, need to generate original and hint-guided groups, roughly doubles the computational cost compared to GRPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)