Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
Authors: Xichen Zhang
Sitong Wu
Yinghao Zhu
Haoru Tan
Shaozuo Yu
Ziyi He
Jiaya Jia
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Reinforcement Learning from Verifier Rewards
Mathematical Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: Scaf-GRPO boosts LLM reasoning in RLVR, using hierarchical hints to guide on-policy GRPO on difficult problems.

Abstract: Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the ''learning cliff'' phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24 benchmark by a relative 44.3\% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous reasoning in LLM.

Published as a conference paper at ICLR 2026
SCAF-GRPO: SCAFFOLDED GROUP RELATIVE POLICY OPTI- MIZATION FOR ENHANCING LLM REASONING
Xichen Zhang' Sitong W Yinghao Zhu', Haoru Tan', Shaozuo Ziyi He & Jiaya Jiabt 'The Hong Kong University of Science and Technology
The Chinese University of Hong Kong
'The University of Hong Kong
ABSTRACT
Reinforcement learning from verifiable rewards has emerged as powerful tech- nique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the "learn- ing cliff" phenomenon: when faced with problems far beyond their current ca- pabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calcula- tion to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strate- gically provides minimal guidance only when model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts concrete steps, enabling the model to construct a valid solution by itself. Extensive ex- periments on challenging mathematies benchmarks demonstrate Scaf-GRPO's i ef- fectiveness, boosting the pass@1 I score of the 22wn2.55Matth-1 AIME24 benchmark by relative 44.3% over vanilla GRPO baseline. This re- sult demonstrates our framework provides a robust and effective methodology for unlocking model's ''s ability to solve problems previously beyond its reach, a criti- cal step towards extending the frontier of autonomous reasoning LLM.
model on the
INTRODUCTION
GEPO
LUFFY
Large Language Models (LLMs) have demon- strated remarkable capabilities in complex rea- soning tasks across diverse domains such as mathematics, programming, and logic (Guo et al., 2025; Jaech et al., 2024; Muennighoff al., 2025; Min et al., 2024). A key driver of these advancements is Reinforcement Learn- ing from Verifier Rewards (RLVR) (Guo et al., 2025: Zeng et al., 2025; Liu et al., 2025b), a paradigm where models learn to generate so- phisticated reasoning paths by exploring di- verse strategies and receiving feedback their final outcomes. This approach elimi- nates the need for expensive, step-by-step hu- man annotations by rewarding only the fi- nal correct answer, enabling models to au- tonomously discover effective problem-solving procedures (Guo et al., 2025).
Figure I: Scaf-GRPO overcomes the learning cliff with minimal guidance, outperforming vanilla GRPO (Shao et al., 2024) and the prefix-based LUFFY (Yan al., 2025) across challenging math benchmarks on (wwen..-Mathh jecting strategic. hierarchical hints, our method unlocks the model's potential on difficult prob- lems, achieving superior overall performance.
By in-
However, the efficacy of RLVR is severely constrained by a fundamental challenge we formalize as the "learning cliff." This phenomenon occurs when a model confronts a subset of problems that lie significantly beyond its current capabilities. For these problems, all exploratory attempts
"Equal contribution
Corresponding author; Jiaya Jia (j1 aiccs...... ust. hk)

---

Published as a conference paper at ICLR 2026
Phase 1
Phase 2
Phase
Phase
0.350
Numbbrr of Zuro-Rwward Quaries
0.325
Average Performance
0.300
120
M MMM0000
0.275
80-
0.260
60
40
0.200
Baseline
20
Magle Wy of -
0.175
Cuidance Examption Perlod
(S-GRPO)
0.150
0
50
100
150
200
100
150
200
Figure 2: Training dynamics of Swen2.5-Math-Lee (a) Scaf-GRPO overcomes the learning cliff by continuously solving zero-reward problems where vamilla GRPO plateaus, (b) This translates to sustained and superior validation accuracy for Scaf-GRPO throughout training.
consistently fail, leading to two critical and cascading consequences: (1) Reward Signal Loss: The
model receives a persistent zero-reward signal for this entire subset of problems. (2) Vanishing Gradients: In algorithms like GRPO (Shao et al., 2024), the advantage signal provides the learning gradient. When all rewards are zero, the advantage collapses zero, providing no gradient for the policy to learn from (Yu et al., 2025).
Consequently, these diffieult problems become "invisible" to the policy update. As our empirical analysis in Figure illustrates, these problems form persistent "long tail" of challenges that the model cannot conquer autonomously. This long tail represents critical bottleneck, it prevents the model from leveraging the most difficult examples to achieve higher level competence.
To address the learning cliff, prevailing strategy has emerged: incorporating off off-policy guidance from more capable "teacher" policy (Yan et al., 2025; Huang et al., 2025; Zhang al., 2025a;b). These methods typically work by providing the student model with prefix of a correct "golden" solution and tasking it with generating the remainder of the reasoning path. While this ensures positive reward signal, this mrntixcontinnuaiion paradigm introduces significant issues. It creates distributional mismatch between the teacher-generated prefix and the mudent-gennratee suffix, necessitating complex algorithmic corrections like policy shaping (Yan et al., 2025) or hybrid SFT- RL objectives (Huang et al., 2025) that can introduce bias and training instability. More critically, this "on-rails" guidance forces the model down a predetermined path, stifling its ability to explore alternative, potentially more novel efficient, reasoning strategies.
To address this challenge, we propose Scaf-GRPO (Scaffolded Group Relative Policy Optimization). Our framework is inspired by the pedagogical theory of Scaffolding (Berk & Winsler, 1995), teaching method of providing temporary support that fades as learners improve. We apply this principle by providing hierarchical, minimal, progressive assistance to help the model bridge its capability gaps, rather than enforcing a rigid solution prefix. This in pr-prommt scaffolding approach is guided by two primary objectives: first, to maintain policy consistency by having the model process both the problem and the hint under a single, unified policy, thereby avoiding the distributional mismatches of prefix- prefix-based methods. Second, to preserve exploration flexibility, as "signposts" " rather than "railroads," guiding the model without fixing its path and allowing it to discover its own unique solution strategies.
our hints act
Our framework operates in carefully designed phases. It first employs guidance exemption period to distinguish "true-hard" problems from "pseudo-hard" ones that the model can solve on its with more training. For true-hard problems, it then activates hierarchical hint-guided ploration, providing progressively concrete hints (from abstract concepts to concrete steps) until the model can generate correct solution. By rewarding the model for succeeding with the most ab- stract hint possible, Scaf-GRPO encourages the internalization of reasoning skills rather than the memorization of solutions. Our contributions
follows:
We propose Seaf-GRPO, novel training framework inspired by pedagogical scaffolding address- ing the "learning cliff" issue in RLVR. It provides hierarchical, minimal, and progressive guidance via in-prompt hints instead of fixed solution prefixes. This approach maintains policy consistency while preserving the model's exploratory autonomy, thereby overcoming the key limitations of existing guidance methods.
2

---

Published as a conference paper at ICLR 2026
We demonstrate the effectiveness of Scaf-GRPP through extensive experiments several chal- lenging mathematics benchmarks. On the Qwen2.5-Math-7 model, our method achieves a sig- nificant relative improvement of 12.6% over the vanilla GRPO baseline and a 9.2% relative gain over strong prefix-based guidance methods like LUFFY.
We demonstrate the broad applicability and robustness of Scaf-GRPO across diverse models. Our experiments show consistent performance gains on different architectures (Qwen. Llama), scales (1.5B to 7B), and specializations (math-tuned, instruction-tuned and Cfa---hoo--Tooohhhie establishing Scaf-GRPO as versatile and model-agnostic framework enhancing LLM reasoning.
2 RELATED WORK
Reinforcement learning from verifier reward. The success of DeepSeek-RI (Guo et al., 2025) establishes Reinforcement Learning from Verifier Reward (RLVR) as a paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). In RLVR, models trained using feedback from an external verifier that provides an outcome-based reward (e.g., correct/incorrect) for a generated solution. The success of DeepSeek-RI (Guo et al., 2025) demonstrates that even with sparse, binary rewards, models can learn reasoning strategies. Subsequent research has built upon this foundation, focusing on enhancing algorithmic stability through debiasing techniques (Liu et al., 2025b; Yu et al., 2025), or designing more informative rewards to improve sample efficiency, such using length penalties to mitigate overthinking (Aggarwal & Welleck, 2025; Jin et al., 2025; Chen et al., 2024) or token-level signals to provide denser feedback (Wang et al., 2025b;a).
RLVR with off- off-policy guidance. To overcome the learning cliff, a phenomenon where a persistent lack of positive rewards renders difficult problems invisible to the learning gradient (Yu al., 2025), researchers incorporate guidance from a "teacher" policy. The prevailing strategy is to provide the student model with prefix of a "golden" trajectory and task it with generating the continuation (Yan ct al., 2025; Huang et al., 2025; Zhang et al., 2025a;b; Ma et al., 2025). Different methods introduce variations on this theme. For instance, Yan et al. (2025) mix a complete expert trajectory with multiple model-generated rollouts in one batch. Huang et al. (2025) employ a cosine decay schedule to adjust the length of the guiding prefix. More recently, Zhang et al. (2025a) provide multi-level hints of varying lengths. allowing the model to explore from multiple starting points. However, this crnti-conninnattonn paradigm introduces challenges. It breaks policy consistency by mixing trajectories from two different distributions, which necessitates complex algorithmic patches (Yan et al., 2025; Huang et al., 2025; Zhang et al., 2025a). Furthermore, forcing the model down predetermined path stifles exploration, limiting its ability to discover novel reasoning strategies. Our work provides effective guidance while circumventing these issues.
METHODOLOGY
Our framework, Scaffolded Group Relative Policy Optimization (Scaf-GRPO), illustrated in Fig- ure 3, overcomes the learning cliff inherent in reinforcement learning by providing hierarchical, minimal, and progressive guidance. Unlike methods that alter the fundamental RL objective with off- off-policy data, Scaf-GRPO maintains the on-policy nature of GRPO. It operates by strategically augmenting the model's ' rollout buffer when learning stagnates, ensuring that the learning signal is both meaningful and derived from the most efficient reasoning path the model can achieve with sistance. Our framework operates in two phases: an initial guidance exemption phase to diagnose "true-hard"" problems, and subsequent cyclical phase of hierarchical hint- exploration.
3.1 PRELIMINARIES: GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO) (Shao et al., 2024) is an on-policy RL algorithm for training LLMs that eliminates the need for trainable value function. For given prompt q. the policy Te generates a group of N trajectories, G = (ol...., ON) After obtaining a terminal reward R(o:) for each trajectory from a verifier, GRPO computes a normalized advantage Å, as: Å; = R(o:)-Hg where lg and dg are the mean and standard deviation of rewards in the group G, and Gd small constant for numerical stability. The policy is then updated by maximizing clipped
aut
3

---

Published as a conference paper at ICLR 2026
Query
Group Reward
Output
Let iu be postie numbers
Analysis
Policy Model
Reward Model
Reward
Wf 5V 122 Lakc
Correct
0
least one output
R
Wrong
outpurs
Informative Feedback
Uninformative Feedback
Hierarchical Exploration
Koowlledge
Alt. insauilis pouitine numbers, anithmatic moan grometiic
Learning
all (Onsited]
Trigger
Rg
R
Planshy
Brougsbe thatr the siprosdice can riniiieed udin the
incosali: by splitting Pi Into manageable parts, (Dritted)
Replaced Foedback
O
Selatinn
Group Advantage
With Hinodd Dutcome
Augmanto Betch
Apply
ts' + +-txe 2000
Maae
[Omiited)
R
Figure 3: Overview of the Scaf-GRPO framework. For given query, the model generates multiple solutions. (Left) If any solution correct, standard GRPO proceeds. (Right) all solutions fail (the learning cliff), Scaf-GRPO initiates hierarchical hint-guided exploration. It injects progressively concrete in- in-prompt hints until correct solution found. This successful, minimally-guides trajec- tory replaces failed one, restoring the learning gradient and enabling on-policy updates to resume.
surrogate objective:
JARPO (0) Es,e min (rr,(0)A,, clip(r,.(0),1 c))+
where (0) To cos,
uo loia is the probability ratio between the current and old policies, and is the clipping hyperparameter. The key limitation arises when all trajectories in & receive a zero reward, causing A; to collapse to zero and stalling the learning process- -te learning cliff.
3.2 THE SCAF-GRPO FRAMEWORK
Scaf-GRPO modifies the training process by strategically augmenting the trajectory group G when a learning cliff detected. The process consists of conditional batch construction procedure followed by the application of the standard GRPO loss.
Phase 1: Diagnosing true-hard problems. A key principle of effective teaching is to avoid pro- viding help when a learner can succeed independently. Not all initial failures indicate fundamental capability gap; many are what we term pscudo-hare samples, arising from unfamiliarity with output formats nascent reasoning skills. To address this, Scaf-GRPO incorporates a guidance exemption period, empirically set to the initial 15% of training steps. During this phase, the model attempts solutions purely through on-policy exploration. As shown in Figure 2, this period is characterized by a rapid decrease in zero-reward queries. We algorithmically determine when this independent learn- ing has plateaued by monitoring the rate of solving rero-reward queries. Once this rate stagnates, any problem the model still consistently fails is classified as "true-hard," making it a candidate for guidance. This ensures hints are reserved for genuine learning cliffs.
Prompt for hint injection
Phase 2: Hierarchical hint-guided explo- ration. Once a problem is identified "true- hard," Scaf-GRPO activates its guidance mech- anism using pre-defined, three-tiered hint hi- erarchy, H = { (Hnnwweddee, H Hplanning: H Hsolation }- The tiers offer distinct levels of guidance: (1) Hkwwweedge (Knowledge Hint): Points to the key concept formula required. (2) planning (Planning Hint): Outlines a high-level strategic
see Systen Prospt:
Please
step, and your final answer within \booed().
sen User Prospt:
Question: (question)
Moouleegg/Plammnng/SSn
(hints)
Figure 4: Prompt for hint injection.
framework for the solution. (3) Hxstation (Solution Hint): Provides concrete caleulation step.
4

---

Published as a conference paper at ICLR 2026
To provide the minimal necessary guidance, the framework executes deterministic search through this hierarchy, proceeding from the most abstract to the most concrete hint (H (mowwedgee - H planning - Hstution). Within each tier, guidance is offered incrementally. The search terminates as soon as the model generates a correct solution, thereby identifying the minimal effective guidance required. A detailed description of this progressive exploration algorithm is provided in Appendix D.
On-policy batch augmentation and unified loss. The core of Scaf-GRPO is its on on-policy in- lervention, reactivating the learning signal during a learning cliff. When all initial trajectories G = fo!...., o from n yield zero reward, the advantage A, collapses, halting the gradi- ent update. Scaf-GRPO intervenes by finding minimal hint h² that enables policy πT to generate a successful trajectory of To[-1q @ h*). This successful trajectory replaces a random failed trajectory O2
form an augmented group. Grinl = (9) \ {o,}) (
The key insight is that Scaf-GRPO does not alter the mathematical form of the GRPO loss function. Instead, it modifies the data used for the loss computation. The advantage calculation is performed on this conditionally augmented batch:
A = R(o,) M&rn for of E Guual-
(2)
Grenl + Esid
The learning objective remains the clipped surrogate objective, but it is now applied to the trajecto- ries in Gfmat. The probability ratio for given trajectory of, GGmma at timestep /. is denoted as r (0). The overall objective is:
Jseat-CRPo Be,s min (r),(0)A (0) - +)
(3)
where the probability ratio (0) is critically computed with respect to the trajectory's specific originating prompt:
no To(ol,eel, (ch, mo 100f,,oi..cc
if of E Gimal and of + 0
(0)
(4)
ao ao (ol,
if of = o
Reaa (% (. (1
This on-policy augmentation ensures the batch contains zor-zero reward variance, restoring a mean- ingful advantage signal and allowing learning to resume on previously intractable problems.
Conservative nature and on-policy integrity. A crucial property of Scaf-GRPO is its conserva- tive nature; the framework does not alter the fundamental GRPO optimization objective but rather operates as a targeted data augmentation strategy. Its impact on the policy gradient can be formalized by analyzing two distinct cases based on the initial sampling results for given prompt q-
In the first case, where at least one successful trajectory is generated initially (Eo; E 9 such that R(o,) 0), the batch already contains valid learning signal. The condition for intervention is not met, so the batch remains unchanged (Ginal = 9). Consequently, the objective function is mathematically identical to standard GRPO, ensuring our framework does not interfere when the model can learn on its own:
Jscaf GRPO (0) (G).
(5)
In the second case, the learning cliff scenario (Vo, E G, R(o:) 0), standard GRPO fails. The uni- form zero rewards cause the advantage calculation to collapse (Li = 0, og = 0). leading to a null advantage A; = 0 and a vanishing policy gradient. Here, Scaf-GRPO intervenes by constructing the augmented batch Gimat. This restores the gradient by ensuring Mm > 0, which in turn guarantees non-zero advantage signal A" Critically, this intervention preserves the on-policy principle. Unlike off-policy methods that import trajectories from a different policy T and require high-variance im- portance sampling corrections (e.g., using ratio He ), the guided trajectory of is sampled directly Ra
from the current policy тe. The probability ratio is therefore a standard on-policy ratio computed on modified input, which is inherently more stable. We explicitly avoid the distributional mis- match of off-policy alternatives defined by ratios like =e(:1g) (198 which are shown destabilize training in Appendix F.S. Instead, by conditioning both the current and old policies on the identi- cal hint-augmented prompt, Scaf-GRPO ensures stable learning signal. This targeted, on-policy intervention transforms an unproductive, zero-gradient sample into a valuable learning opportunity without compromising the integrity of the optimization process.
5

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance seven benchmarks. We compare our method, SCAF-GRPO, against vanilla GRPO baselines across diverse architectures, including the Qwen2.5 series, a non- Qwen model Alama--32-885 IB-Instruct) and specialized long-CoT model (DeepSeek-RI-Distee Qwen-1.5Bh Scores: pass @ (%). Best results are in bold. The background color of Scaf-GRPP cells indicates performance change Vanilla GRPO (green for improvement, red for decline).
Model
AIME
AMC
Minerva
MATH-500
Olympiad
Gaokao20Z3en
Avg.
Qwen2 Math-1LS5
Qwen2.5 1onn2.5Maai Math--155
7.2
3.3
32,5
14.7
32.8
20.6
20.0
18.7
Vanilla GRPO
13.3
10.0
47.5
28.3
72.2
34.8
57.4
37.6
Scaf-GRPO
20.0
13.3
60.0
29.1
73.4
36.6
57.9
41.5
Owwen..5Marhh
Qwen2.5 5-Mab-78
13.3
13.3
42.5
16.5
53.6
18.2
35.1
27.5
Vanilla GRPO
30.0
13.3
33.4
75.8
41.3
62.6
45.2
SimpleRI -7em 7eng et (2025)
23.3
13.3
55.0
31.6
76.8
37.2
60.8
42.6
Out-Zero Liu et (2025a)
30.0
16.7
62,5
34.6
78.0
41.0
62.9
46.5
LUFFY al. (2025)
33.3
16.7
62.5
33.8
75.2
41.7
62.7
46.6
Scaf-GRPO
43.3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
Qwen2.5-77B
Qwen2.5-7B
10.0
6.7
37,5
26.4
61.8
34.4
42.6
31.3
Vanilla
10.0
38.5
Scal-GRPO
13.3
20.0
60.0
38.6
77.8
40.8
63.8
44.9
1.mma.3.2 Instrrct
LJama-3 1otna-3.2-3B---- 2-3--nntruct Vanilla GRPO Seaf-GRPO
6.7
0.0
20.0
11.8
38.3
12.6
33.5
17.6
45.7
16.7
3.3
40.0
19.1
56.2
20.3
46.0
28.8
DeepSeek RI Distill Qwen- 1.58
DeepSeck-RR RI D-Distlll-Qee Seiill-Qween-150
28.9
20.0
67.5
26.1
83.9
45.8
62.1
47.7
Vanilla GRPO
Seaf-GRPO
33.3
23,3
77.5
32.4
85.8
50.7
72.3
53.6
EXPERIMENTS
4.1
EXPERIMENTAL SETUPS
Training dataset. Our training data is derived from the DeepScaleR-Previcee Preview-Dataset (Luo al., 2025). We employ a dynamic filtering strategy that aligns the dataset with each model's initial capabilities. Based on preliminary evaluation, "Too Hard" (retained), or "Potentially Solvable" (50% subsampled). This curates a challenging yet tractable training set focused the frontier of the model's abilities. For this dataset, generate our three-tiered hints by prompting the DeepSeek-RI model (Guo et al., 2025) with ground-truth solution steps. Further details provided in Appendix B.1 and Appendix B.2, respectively.
classify problems as "Too Easy" (discarded),
data filtering strategy and the hint generation process
Models. To demonstrate the general applicability of Scaf-GRPO, we conduct experiments across diverse set of models, including: math-specialized models (0wen2.5-MMth-77 & 1.5B) to test in-domain performance; general-purpose base model (Qwen2.5-7B) to assess skill acquisition; different architecture (Llama-3.2) 2.--....-3--nnstrccc to verify modellagnnsstiismm: and Long- cong-Chain-of- Thought model (DeepSeek-Ri) -isttill-Qwen-Ill to evaluate applicability to extended reasoning.
Baseline methods. We benchmark Scaf-GRPO against three distinct classes baselines: (1) Vanilla GRPO (Shao et al., 2024), the standard on-policy algorithm without guidance. This serves as our baseline to quantify the gains from our scaffolding mechanism. (2) Leading GRPO implementations, including Simple-RL, (Zeng et al., 2025) and Oat-Zero (Liu et al., 2025a), to con- textualize our performance against highly-optimized public benchmarks. (3) LUFFY (Yan al., 2025), representative of RLVR with off off-ppoicy guidance. This provides a direct comparison be- tween the dominant prefix-continuan strategy and our in-prompt scaffolding approach.
Evaluation details. We evaluate diverse mathematics GaoKao2023en (Chinese GaoKao Community, 2024), AIME24 (AIME, 2024), AIME25 (AIME, 2025), AMC (AMC, 2023), MATH-500 (Hendrycks et al., 2021), and OlympiadBench (He al.,
benchmarks, including

---

Published as a conference paper at ICLR 2026
2024). To assess out-of-distributio (OOD) generalization, we evaluate on the scientific reasoning benchmark, GPQA-Diamond (Rein et al., 2024). For benchmarks, we report pass@) accuracy via greedy decoding. Vanilla GRPO is trained with our data and hyperparameters, and LUFFY on our data with original parameters. For Simple-RI, and Oat-Zero, we evaluate their publicly available weights.
Implementation details. We train all models for epochs using the verl framework (Sheng et al., 2025), reporting results from the best-performing checkpoint. The maximum response length is 2048 tokens (8192 for the LongCoT model). Consistent with recent studies (Liu et al., 2025b; Yu et al., 2025; Yan et al., 2025), we set the KL divergence penalty to zero to maximize policy exploration. A comprehensive list of hyperparameters detailed in Appendix C.
-atn2-- model. The best performance is highlighted in bold. The "No Guidance" row serves as the vanilla GRPO baseline.
Table Ablation study on Scaf-GRPO' 's key components using Qwen2.5-
Hint Strategy
AIME24
AIME25
AMC23
Minerva MATH-500 Olympiad Gaokao2023en
Avg.
Seaf-GRPO (Full K P -S)
43.3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
w/o Progressive (Solution-Only)
40.0
13.3
65.0
36.2
78.6
43,7
62.3
48.4
w/o Knowledge Hint (P -S)
43.3
13.3
70.0
34.2
77.8
42.4
63.1
49.2
Planning Hint S)
16.7
62.5
w/o Solution Hint (K -P)
40.0
10.0
67.5
34.2
78.6
42.2
63.4
48.0
w/o Incremental Chunking
43.3
10.0
62.5
36.0
76.0
41.6
64.2
47.7
No Guidance (Vanilla GRPO)
30.0
13.3
60.0
33.4
75.8
41.3
62.6
45.2
4.2
MAIN RESULTS
In this section, we present the primary evaluation of Scaf-GRPO, focusing on pass performance across diverse model architectures. The results highlight the method's performance advantage over Vanilla GRPO and competing prefix-based baselines. Comprehensive supplementary analyses detailed in Appendix covering robustness checks using the avg@ 16 metric, comparisons against an expanded suite of methods such DAPO Yu et al. (2025) and DeepScaleR Luo et al. (2025).
Comparison with GRPO. As shown in Table 1, compared to the vanilla GRPO baseline, Scaf- GRPO achieves comprehensive and significant performance gains across all tested models. On the Qwen2.5-Math model, Scaf-GRPO boosts the pass@) score from 0.300 to 0.433 on AIME24, relative improvement of 44.3%. These results provide strong evidence that our scaffolding mech- anism effectively helps the model overcome the "learning cliff," enabling it to tackle problems that were previously beyond its independent capabilities.
Comparison with other methods. To contextualize Scaf-GRPO within the broader research land- scape, we compare it against other leading methods in Table 1. Scaf-GRPO demonstrates marked superiority, achieving an average score of 0.509. This performance rep- resents a substantial improvement of 19.5% over Simple-RI. and 9.5% over Oat-Zero. More im- portantly, Scaf-GRPO establishes clear advantage over the prefix-continuation paradigm, outper- forming LUFFY by 9.2%. This significant outperformance suggests that our in-prompt scaffolding strategy offers more effective training alternative prefix-continuation methods.
Qwen2.5-Math-77
Generalization to non-Qwen architectures. To verify that the benefits of Scaf-GRPO are confined to single model family, we extend our evaluation to the Llama-3.2-3BB 3.2-33B-nntttrcc model (Dubey et al., 2024). As detailed in Table 1, our framework demonstrates strong general- ization. While vanilla GRPO provides a sigmificant uplift over the base model, Scaf-GRPO achieves a further relative improvement of 10.3% in average performance. This confirms Scaf-GRPO is model-agnostic method, capable of enhancing reasoning abilities beyond the Qwen series.
Applicability to LongCoT models. We further investigate the efficacy of Seaf-GRPO on models optimized for Long Chain-of- ThoTuht (LongCoT) reasoning, using the specialized DeepSeek- -1- Distill-Qwen-1.50 model. The results in Table show that Scaf-GRPO effectively enhances this already capable baseline, delivering a 5.9% relative performance gain over vanilla GRPO. This demonstrates our framework's ss versatility in scaffolding not only standard-length solutions but also the extensive derivations characteristic of LongCoT models.

---

Published as a conference paper at ICLR 2026
Query
sample consisting of five abservations has an arithmetic mean of 1D and median of 12. The smallest value that the range (lrgest sample
observation
Solution
Solution aadd then syagye 27 and R2sky-12 satisfy constralnts (Omitted)
Stage 1: only with solution hint correct but model can't independently
Model Output: minimize the
synige? and Ry smallest
range that the sample can assume (Correct [!I]
2: with both solution and planning hints cornect but model con't solve it independently
Planning
Planning Mint: .assume symmetry by setting Xytty and syms, [Omitted)
Model Output: To minimize the range, we can assume Hymt Kyttiyn This gives which simplifies range [Correct !!!!
Knowledge
Knowledge Hint: Optimication principle for minimizing ronge [Omitted]
3: with afl three level hints correct but model can't solve it independenttly
Model Output: Applying the optimization principle for minimizing range
Model Output: To minimine the range... make b close possible. (Optimication principle minimizing range -.. the smallest value that the range can ascume
Stuge d: Model can solve Independently
Figure 5: Evolution of reasoning from guidance to autonomy. The model progresses from imitating concrete hint (a) to applying abstract knowledge (b), ultimately achieving (c) autonomous problem- solving by internalizing the guided skills.
4.3 ABLATION STUDY
We conduct a series ablation studies on the Qwen2.5-Math-77 model (see Table 2). Detailed investigations into the guidance exemption period and data filtering strategies are presented in Ap- pendix G.I and Appendix G.2, respectively.
Necessity and robustness of the guidance exemption period. To validate the necessity and bustness of the guidance exemption period (Phase 1), we conduct detailed ablation studies in Ap- pendix G.I. Our experiments show that applying scaffolding from the very beginning leads 9.2% relative performance drop on Qwen2.5-Math- that initial phase of autonomous exploration is crucial to prevent hint dependency. Furthermore, sensitivity analysis reveals that the method is highly stable across exemption durations ranging from 10% to 40%, where the model maintains a nigh-performance plateau between 49.5% and 50.9%. This validates our selection of 15% as an optimal and robust configuration.
compared to the full framework, confirming
Efficacy of progressive & hierarchical guidance. Our methodology is founded on the hypothesis that progressive guidance, from abstract concepts concrete steps, superior to simply providing direct solution. To test this, we evaluate a "Solution-Only" variant that bypasses the hierarchy and immediately provides the most concrete hint, which Hyotution This results in significant perfor- mance degradation of 4.9% compared to the full model. This confirms our hypothesis: compelling the model first engage with higher-level reasoning fosters more generalizable skills.
Justifying the completeness of the hint hierarchy. We design a three-tiered hint structure (K-P-+S) assuming each layer serves unique function. To verify this, we systematically moved one layer at a time. As shown in Table 2, every removal degrades performance. The most severe degradation, a 5.7% drop, occurs when the final "Solution" hint is removed (the K-P vari- ant). This highlights the dual role of the hierarchy: abstract hints encourage high-level reasoning, while concrete hints serve model demonstrates that the layers are complementary, not redundant.
essential fallback. The superior performance of the full K-PP+S
Efficacy of incremental guidance. A core principle of Scaf-GRPO to provide the minimal necessary support by delivering hints incrementally. We test this against "Full Hint" variant, which provides the entire content of a hint level at once. This non-incremental approach collapses performance by 6.3% compared to the incremental one. This decline validates our strategy: minimal, incremental intervention is critical for preserving model autonomy and preventing over-reliance.
Impact of hint quality. We investigate the correlation between hint quality and student perfor- mance using multi-faceted rubric (accuracy, minimality, clarity, and structural coherence). Em-
8

---

Published as a conference paper at ICLR 2026
Table 3: Impact of data filtering on Scaf-GRPO vs. Vanilla GRPO. Both methods were trained on the full dataset (Original) and harder subset (Filtered). The best performance is highlighted in bold. Scores are pass @ (%).
Data
Method
AIME24
AIME25
AMC23
MATH-500
Olympiad
Avg.
Qwen2. Saan225--Mah--aaaa 1.5B
Original
Vanilla GRPO
13.3
6.7
52.5
68.6
31.4
34,5
Original
Scaf-GRPO
20.0
10.0
55.0
73.2
36.4
38.9
Filtered
Vanilla GRPO
13.3
10.0
47.5
72.2
34.8
35.6
Filtered
Scaf-GRPO
20.0
13.3
60.0
73.4
36.6
40.7
Qwen2.5-Math
Original Original
Vanilla GRPO
30.0
16.7
60.0
74.4
38.5
43.9
Scaf-GRPO
33.3
16.7
70.0
79.0
43.0
48.4
Filtered
Vanilla GRPO
30.0
13.3
60.0
75.8
41.3
44.1
Filtered
Scaf-GRPO
43.3
20.0
70.0
80.0
43.3
51.3
ploying an LLM-as-a-Judge, we observe that higher- qualitr hints lead to superior downstream out- comes, Notably, DeepSeek-RI achieved higher hint quality score, outperforming Qwen2.5-72B- Instruct. This translated to significant performance gains for the student model (Qwen2.5-MM) --t.2.-- yielding relative accuracy improvement of 4%. We defer the complete evaluation rubric, the spe- cific prompt used for the LLM judge, and detailed experimental results to Appendix H.
4.4 FURTHER ANALYSIS
Confronting the learning cliff. Figure visualizes Scaf-GRPO's advantage. In Figure 2(a), plot the number of "zero-reward" problems per batch. The count for both methods drops sharply at the start of training. However, the vanilla GRPO curve quickly flattens, defining the learning cliff: point where the haseline can no longer extract a learning signal from a persistent set of "true-hard" problems. In contrast, Scaf-GRPO's scaffolding activates, enabling the model consistently learn from these problems and continue reducing the zero-reward count. This directly impacts validation performance (Figure 2(b)). By turning intractable problems into learning opportunities, Scaf-GRPO achieves a higher, steadily improving validation score while the baseline stagnates.
Internalizing skills beyond imitation. Scaf-GRPO succeeds by fostering skill acquisition rather than simple imitation. Figure illustrates this trajectory on a challenging problem: the model evolves from utilizing concrete "Solution Hints" to abstract "Knowledge Hints," and finally to solv- ing the problem without any support. To quantify this transition from dependency to autonomy, we track "skill graduation" events over the first 300 training steps. We define this metric to capture the learning breakthrough specific each method: for Scaf-GRPO, we count problems transitioning from hint-dependent to autonomous success, while for the Vanilla baseline, we track the shift from total failure success. As shown in Table 4, Scaf-GRPO consistently yields significantly higher volume of graduations (e.g., +137.8% on Swee2.5-MMat--1550 fectively converts temporary guidance into lasting, independent reasoning capabilities. By enabling such robust skill-building on hard problems, Scaf-GRPO effectively overcomes the learning cliff.
This confirms that our method ef-
Table 4: Comparison of total "graduations" across different model backbones.
Model
Vanilla
Scaf-GRPO (Ours)
Relative Increase
Qwen2.5-Man 5-Math-
1,123
2,670
+137.8%
Dwen2.5-Matth-77
434
483
+11.3%
Qwen2.5-7B
464
723
+55.8%
DeepSeek-RR SeepSeek-R1-Dinee R1-Distill-1.5B
453
694
+53.2%
Llama-3.22
577
986
+70.9%
Aligning data difficulty with model capacity. To validate our data filtering strategy, we train models on both the complete dataset and our filtered subset. As detailed in Table 3, the harder, filtered data yields a marginal 0.5% relative gain for vanilla GRPO but a substantial 6.0% boost on Qwen2.5-Mathh cult problems is insufficient. A challenging curriculum is most effective when paired with a robust learning framework like Scaf-GRPO, which converts these challenges into learning opportunities. We also observe performance is sensitive to the density of solvable problems. Specifically, dis- carding "Too Easy" samples and subsampling the "Potentially Solvable" category 50% achieves
for
This disparity underscores that exposing model diffi

---

Published as a conference paper at ICLR 2026
the optimal balance, yielding 10.2% relative gain the full dataset (Table 12). This empirical evidence justifies our training data selection. All experimental details are provided Appendix G.2.
Generalization to out-of-distribution tasks. To verify that Seaf-GRPO cultivates robust reason- ing skills beyond in-domain pattern matching, we evaluate its generalization on the expert-level OOD benchmark, GPQA-Diamond. As shown in Table 6, Scaf-GRPO significantly enhances the Qwen2.5-Maiee M-tth--- model, achieving 15.5% relative improvement over Vanilla GRPO and match- ing the strong LUFFY baseline. Furthermore, this gain effectively transfers the general- surrpre-purpose Swen2.5--B-Base where our method yields 7.5% relative increase over Vanilla GRPO and outper- forms LUFFY. These results demonstrate that the problem-solving abilities fostered by Scaf-GRPO are fundamental and agnostic to the model backbone.
Computational efficiency. Scaf-GRPO optimizes training efficiency by applying guidance selectively and accelerating convergence. Empirically, during the training of Qwen2.5- Math-7B, the hint-guided exploration triggered for only 17.4% of the samples, ensur- ing that the majority of the computational throughput remains dedicated to standard gen- eration. More importantly, by converting zero-reward signals into high-value learning sig- nals, Scaf-GRPO significantly reduces the total training duration required to reach optimal performance. As shown in Table 5, Scaf-GRPO reaches its best-performing checkpoint (50.9% avg.) in approximately 12 hours. This represents clear efficiency gain over the Vanilla GRPO baseline, which requires 13 hours reach a lower peak performance (45.2%), demonstrating that Scaf-GRPO achieves better results with smaller total time budget.
Table 5: Training efficiency comparison on Qwen2.5-Math-77 Best results are in bold.
Method
Best Avg. (%)
Time to Best Ckpt.
Trigger Rate
Peak Memory
Vanilla GRPO
45.2
~ -33 hours
N/A
~72 GB
Scaf-GRPO (Ours)
50.9
12 hours
17.4%
~33 GB
Table 6: OOD performance (pass 1,5) of Dwen2.5-MMth-771 and Qwen2.5-7B-Bass on the GPQA-Diamond benchmark.
5 DISCUSSION AND CONCLUSION
Limitations. The practical deployment of Scaf- GRPO is subject to two main considerations. First, its efficacy currently relics on the availabil- ity of a high-quality, tiered hint hierarchy. Gener- ating these structured hints requires non-trivial data preparation effort. Second, the framework principally designed for tasks with verifiable solutions and structured reasoming paths, such as mathematics. Its applicability to more open- ended, subjective domains like creative writing is less direct.
Model
GPOA-Diamond
Owen2.5-Math--
Base Model
24.7
Vanilla GRPO
32.3
SimpleRl-Zero Zeng et al. (2025)
33.3
Oat-Zero et al. (2025a)
33.3
LUFFY Yan al. (2025)
37.3
Scaf-GRPO (Ours)
37.3
Qwen2.5-77.
Vanilla GRPO LUFFY Yan et al. (2025)
33.3
34.4
Seaf-GRPO (Ours)
35,8
Future work. Future research could focus automating hint generation to enhance the frame- work's scalability. We also plan to explore adaptive scaffolding mechanisms where guidance dy- namically adjusts to the model's improving proficiency, thereby personalizing the learning process.
a training framework that overcomes the "learning cliff" in reinforeement learning for large language models. By providing hierarchical hints in the prompt, Ceaf-GRPO offers scaffolding for models to solve problems beyond their reach. This on- on-policy guidance preserves exploratory autonomy and mitigates the distributional consis- tency issues inherent in trefixcoontinuation methods. Our experiments show Scaf-GRPO signifi- cantly outperforms vanilla GRPO and strong prefix-based baselines across challenging mathematics benchmarks. This framework enables models to learn from previously intractable problems, estab- lishing more effective path toward autonomous reasoning.
Conclusion. In this work, we introduce
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research adheres the principles outlined in the ICLR Code of Ethics. Our primary objective to contribute society by advancing the frontiers machine reasoning. upholding the highest standards of scientific excellence through transparent and reproducible methods. We acknowledge the potential for dual-use applications and the risks associated with the misuse of advanced Al rea- soning systems. We also recognize the environmental impact of training large models and have striven for computational efficiency. Our work is based on publicly available datasets devoid of per- sonal identifiable information. We are committed to fostering responsible innovation and encourage continued investigation into the societal impacts of increasingly capable models.
REPRODUCIBILITY STATEMENT
The supplementary material contains the complete source code to ensure full reproducibility. This includes the implementation of Scaf-GRPO and all scripts for data filtering and the training pipeline.
ACKNOWLEDGEMENTS
This work supported in part by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R.
REFERENCES
Pranjal Aggarwal and Sean Welleck, LI: Controlling how long reasoning model thinks with
reinforcement learning, 2025. URL https:://aaai...rrrrrrrr www./orgg/b///aaaeeeee /2503. 2503.00699 6977.
AIME.
American invitational mathematics examination, 2024.
URL https:
/ //artofproblemsolllee http://////////ee....//////a//aa Pwwblees_rreeess___e _and____________________________ Solutions.
AIME. American invitational mathematics examination, 2025.
URL https://
artoffproblensooviing www.wwoiiiiwweeeeeeeeeeeeeeeeeee www.kk/iinndxx.pp//2222...2 phpp/225__AIME....
AMC. American mathematics competitioons.,,, 2023. URL PRLLAttp:///tttffrooleeeoltt
com/wiki/inder php/2023 12A.
Laura F. Berk and Adam Winsler. Scaffolding Children's Learning: Vygotsky and Early Childhood Education. National Association for the Education of Young Children, Washington, DC, 1995.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? the overthinking of ol-kike llms. arXiv preprint arXiv:2412.21187, 2024.
Chinese GaoKao Community. Gaokao2023-math-en, 2024. URL /es.//...o-tata--aaaaaaaeeeeeeaa
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Bofei Gao, Feifan Song. Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07988. 2024a.

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi. Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024b. URL httpst zenodo org/ records/12608600
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprini urXiv:2501.04519 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lewe bilingual multimodal scientific problems. arXiv preprint TrXii:2402.14000 2024.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.24299, 2025.
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling, 2025. URL
https: /ttp//aavvvv////////////./...avl org/ /2s/2257 01679.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722 2024.
Zhensheng Jin, Xinze Li, Yifan Ji, Chunyi Peng, Zhenghao Liu, Qi Shi, Yukun Yan, Shuo Wang, Furong Peng, and Ge Yu. Recut: Balancing reasoning length and accuracy in 11ms via stepwise trails and preference optimization. arXiv preprint arXiv:2506. 10822, 2025.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Zichen Liu, Changyu Chen, Wenjun Li. Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in ri-zero-like training - pilot study. https:// oatooaalllll 1mm www.omoooiiiiiiii..oomm..m...oo. sat-zero, 2025a. Notion Blog.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, 2025b. URL https: //aaxxiv.rg//aa/ee.....// httpaabb///0... 2503. 20783.
Michael Luo. Sijun Tan, Justin Wong, Xiaoxiang Shi. William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol op-preveew with a 1.5b model by scaling rl, 2025. Notion Blog.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: In- terleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07522 2025.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report thinkiin reasoning systems. arXiv preprint 222 2024.
12

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei- Hannanch Hajishirzi, Luke Zeutlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprini arXiv:2501. 19393, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https: //arxi////i//////a///a/eaeaaia org/ abs// /20b/ 03300.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp- 1279-1297, 2025.
Yuxi Tong. symeval: python library for symbolic evaluation in mathematical reasoning. 2024.
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou Stabilizing knowledge, pro- moting reasoning: Dual-token constraints for rivr, 2025a. URL https:://aaaatte org.oog////oooooo / 2507. 15778.
Shenzhi Wang, Le Yu, Chang Gao, Chujic Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprint arXiv:2506.01933 2025b.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025. URL httppss///xxvv.oro/// 2504. 14945.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.144 14476, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang. Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason, 2025a. URL https:// arxiv. orgg abs/ 2507 02841.
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning, 2025b. URL https://
wwxiv.org/abs///0 1015506677221 17211.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this manuscript, the authors employed large language model (LLL)) to aid in refining the language and correcting grammatical errors. The role of the LLM was strictly limited that of content, arguments, and final phrasing presented in this paper.
tool. The authors take full responsibility for the substantive
B DATASET AND BENCHMARK DETAILS
B.1 TRAINING DATA SOURCE AND FILTERING
Peweaaleee-eeeee--iiiae-a (Luo et al., 2025), a compre- hensive collection of 40k mathematical problems. Its contents are sourced from AIME, AMC, MATH (Hendrycks et al., 2021), Still (Min et al., 2024), and Omni-MATH (Gao et al., 2024a)- To maximize training efficiency and target problems most conducive to learning, we implement dynamic data filtering strategy tailored to each model's capabilities. This strategy categorizes prob- lems based on the model's initial performance, assessed through 8 independent sampling attempts for each problem. These samples are generated using nucleus sampling with temperature of 1.0 (top-p=1.0, top-k=-1) and maximum length of 2048 tokens for both the prompt and the response. For our LongCoT model, this response token was increased to 8192 tokens to accommodate its generation style. Based on the outcomes, problems are categorized follows:
Our training data is derived from the
Too Easy: Problems solved correctly in all attempts are excluded, as they offer minimal learning value.
Potentially Solvable: Problems solved in to 7 of the attempts are considered to be within the model's learning-richh sweet spot. We randomly sample 50% of these for inclusion.
Too Hard: Problems that fail in all 8 attempts are retained, as they are the primary candidates for our scaffolding mechanism.
This filtering process results in a final training dataset where approximately 50% of the problems from the "Potentially Solvable" category and the remaining 50% are "Too Hard". This curated dataset challenging yet tractable, maximizing the efficiency of the training process for both the baseline GRPO and our Scaf-GRPO framework.
B.2 HINT GENERATION FOR TRAINING DATA
The hierarchical hints (Hanowledge: Hplanning Hsalution) are the cornerstone of Scaf-GRPO's guidance mechanism. To create them, we perform a one-time offline preprocessing step using the powerful DeepSeek-R1 model (Guo et al., 2025). For each problem our curated training set, we provide the model with both the problem statement and its ground-truth solution trace.
We then employ highly structured prompt, detailed in Appendix E, which is engineered not only to decompose the solution into our three-tiered hierarchy but also to enforce a crucial internal structure. Specifically, the prompt compels the model generate exactly four numbered, progressive items for each category. These items are designed to build upon one another, creating four distinct levels of guidance. For instance, the four items for an H Holanning hint might represent: (I) the first step of the plan, (2) the second step, (3) the third step, and (4) the fourth step. This structured, multi-level design within each hint category enables the fine-grained, progressive exploration central to our method. The entire process ensures a consistent and high-quality set of structured hints, and the scripts will be included in our code release.
B.3 EVALUATION BENCHMARKS
We evaluate all models on a diverse suite of seven challenging mathematics benchmarks and the GPQA-Diamond benchmark to ensure robust and comprehensive assessment of their reasoning abilities. Table provides details for each benchmark used. The mathematics datasets span various difficulty levels and mathematical domains, from high-school competition problems to Olympiad- level challenges, providing rigorous testbed for advanced reasoning. GPQA-Diamond serves crucial Out-of-Distribution (OOD) benchmark, testing generalization to expert-level scientific ques- tions outside the training domain. All benchmarks are publicly available.
14

---

Published as a conference paper at ICLR 2026
B.4 MOTIVATION FOR DATASET SELECTION
Our choice of the DeepScaleR- by two key factors. First, the dataset's broad scope, encompassing problems of varying difficulty, provides the rich and diverse material necessary for our filtering strategy to be effective. Second, its successful application in prior research (Min et al., 2024) establishes it as a robust and relevant foundation for training advanced reasoning models.
for training is deliberate. This decision is motivated
For evaluation, our selection of benchmarks is designed for rigorous and multifaceted assess- ment. The suite primarily consists of challenging competition-leve benchmarks (AIME24/25, AMC, MATH, OlympiadBench, Minerva) and a standardized national exam (GaoKao2023), cov- ering broad spectrum of mathematical reasoning. Crucially, measure out-of-distribution (OOD) generalization, we include the GPQA-Diamond benchmark (Rein et al., 2024). As GPQA consists of graduate-level questions whose style and domain are distinct from our training data, strong per- formance on this benchmark indicates that Scaf-GRPO fosters genuine reasoning skills rather than mere pattern memorization of the training distribution.
Benehmark
Description
Citation
# Problems
AIME24
American Invitational Mathematiss Exarninatioon 2024
(AIME, 2024)
30
AIME25
American Invitational Mathematics Examinaation 2025
(AIME, 2025)
AMC
American Machematics Competitions 2023
(AMC, 2023)
25
MATH-500
A subset from the MATH test set
(Hendrycks al., 2021)
500
GauKao2023en
Chinese National Collage Entrance Exarn 2023
(Chinese GaoKao Community, 2024)
385
OlympisdBench
Math Olympiud-lovel problems
(He et al., 202()
675
Minerca
A specialized dataset evaluate quantisative
reasoning abilities
(1 (lewkewyer al., 2022)
GPOA-Diamend
Expert guestions across biology. physics, and chemistry
al., 2024)
Table 7: Details of evaluation benchmarks used in experiments.
C EXPERIMENTAL. SETUP DETAILS
C.1 COMPUTING INFRASTRUCTURE
Hyperparameter
Value
All experiments are conducted on a high- performance computing cluster. The specific hardware and software configurations are as follows:
Optimization Training Learning Rate (LR)
1 x 10-6 G AdamW W
Optimizer Weight Decay
0.0
Batching Strategy
All models are trained and eval- uated on servers equipped with 8 NVIDIA A100 (80GB) GPUs.
Hardware:
Global Batch Size
256
PPO Mini-batch Size Micro-batch Size (per GPU)
64
Software: The operating system is Ubuntu 22.04. Key software libraries and their ver- sions include PyTorch 2.6.0, Transformers 4.51.1, and CUDA 12.4.
Validation Batch Size
512
RL Algorithm (GRPO)
Rollouts per Query (N)
GRPO Clip Epsilon (c)
0.2
Framework: Our implementation is built upon the verl (0.4.1) framework (Sheng et al., 2025), a robust and efficient library designed for large-scale reinforcement learning with LLMs.
KL Divergence Penalty (3)
0.0
Entropy Coefficient
0.0
Generation Tokenization
Rollout Temperature
1.0
Max Response Tokens
2048
Infrastructure Scheduling
C.2 HYPERPARAMETER DETAILS
Nodes
GPUs per Node
8
Our experimental setup is carefully configured for performance and reproducibility. The final hyperparameter configuration is detailed com- prehensively in Table 8. These settings were applied consistently across all experiments ensure a fair comparison. However, accommodate models specialized in Long Chain-of-Thoupe (I (LnngCoT) reasoning, we increased the maxinum response length 8192 tokens for those specific tasks.
VLLM GPU Memory Utilization
0.8
Table Comprehensive list of key hyperparame- ters for training and generation.
15

---

Published as a conference paper at ICLR 2026
C.3 IMPLEMENTATION OF BASELINE METHODS
To ensure rigorous and fair comparison, we carefully implement or utilize baselines as follows:
Vanilla GRPO. This is our primary control. We train a vanilla GRPO model using our verl frame- work. It is configured with the exact same filtered dataset and hyperparameters Scaf-GRPO, allowing us to cleanly isolate the performance contribution our scaffolding mechanism.
LUFFY.
compare against the dominant prefix-continuation paradigm, we train LUFFY (Yan et al., 2025) using its official public implementation and its original, recommended hyperparameter settings. To ensure fair comparison of methodological effectiveness, we train it our high-quality filtered dataset. This directly contrasts their off-policy guidance with our in-prompt scaffolding on identical data.
Simple-RL and Oat-Zero. For other leading methods like Simple-RL (Zeng et al., 2025) and Oat- Zero (Liu et al., 2025a), we do not perform any implementation or retraining. Instead, we evaluate their officially released, publicly available model checkpoints directly. All reported results for these models are obtained by running them through our unified evaluation pipeline, ensuring a consistent and fair comparison against established state-of-the-art work.
C.4 EVALUATION METRICS DETAILS
Our primary evaluation metric is pass 1, which measures the percentage of problems for which a model generates a correct solution in single attempt. This metric is chosen for its straightfor- wardness and its status as standard for evaluating definitive problem-solving capabilities, For all evaluations, we use greedy decoding generate one complete solution trace for each problem.
The verification process is tailored to the benchmark type to ensure maximum rigor and fairness.
For all mathematical reasoning benchmarks, employ the "symeval" library (Tong, 2024), specifically its EvaluatorMathBatchh module, to determine correctness. This approach moves be- yond simple string comparison by using sophisticated pipeline that combines regular expressions for robust answer extraction with SymPy for symbolic mathematical evaluation. This allows for the accurate verification of complex answers, including matrices, intervals, and symbolic expres- sions.
For the out-of-distributtin GPQA-Diamond benchmark, we utilize the EleutherAl's Im- evaluation-hammes (Gao et al., 2024b) to ensure a fair and standardized assessment. This widely adopted framework provides a consistent testing environment for generative models. We use its implementation of the gpqa-diamond task to compute the pass@1 score, thereby maintain- ing metric consistency while leveraging a community-standardin evaluation harness for OOD generalization.
D METHODOLOGY DETAILS
This section provides granular description of the Progressive Exploration and Replacement Algo- rithm, which is central to how Scaf-GRPO overcomes the learning cliff by strategically providing minimal guidance during training.
D. THE PROGRESSIVE EXPLORATION AND REPLACEMENT ALGORITHM
When a "true-hard" problem triggers the hierarchical hint- hint-guided exploration phase, Scaf-GRPO executes a deterministic, multi-level search algorithm to find the minimal effective hint. The algo- rithm's 's goal is to provide just enough information for the model to succeed, thereby maximizing its independent reasoning.
The algorithm leverages the pre-generated, four-level progressive hint structure, detailed in Ap- pendix B.2. It systemaiiaally searches through the hint categories in order of decreasing abstraction (Hanowledge - Hplanning > Hanuuion) and, within each category, through the four levels of increasing detail.
16

---

Published as a conference paper at ICLR 2026
Let h2 denote the i-th hint item for category (knowledge, planning, solution). The cumulative hint provided to the model at level 2, the union of first items, denoted as C² U [ The search process for a single problem is as follows:
(1) Iterate through Hint Categories: For each category e the sequence ("knowledge", "planning.
"solution"):
(1) Iterate through Hint Levels: For each level from to 4:
(1) Construct an augmented prompt by injecting the cumulative hint C
(2) Generate new solution on-policy using this augmented prompt.
(3) If the generated solution correct, the search successfully terminates. The trajectory produced with hint C² replaces one of the failed trajectories in the hatch. The algorithm then concludes for this problem.
(2) Handle Intractable Case: If the nested loops complete without finding correct solution (i.e., the most detailed hint Cd solution fails), the problem is deemed intractable for the current training step. No replacement occurs, and the algorithm concludes for this problem, leaving the original all-failure group in the batch.
This structured and exhaustive search ensures that if solution is reachable with any level of guid- ance, the framework will find it using the most abstract and minimal hint possible, thereby preserving the on-policy learning signal for "true-hard" problems.
E PROMPT DESIGN IN SCAF-GRPO
The effectiveness of Scaf-GRPO relies on two distinct but complementary types of structured prompts: those for generating the hierarchical hints, and those for injecting these hints during train- ing.
E. HINT GENERATION PROMPT
To systematically create our tiered hints, we provide a powerful teacher model (DeepSeck-RI] with structured prompt. For each problem-solution pair in our dataset, this prompt instructs the teacher model to decompose the solution into our three-tiered hierarchy (HKnowledge- Hplanning solution). This semi-automated process is critical preprocessing step that ensures consistent and high-quality hint dataset. The exact prompt template used is below.
Prompt for hint injection
[ROLE & GOAL]++
are expert assistant specializing problem selvingng methodology and knowledge anginaering. Your task is analyze givan problem and its ground-truth breakdown reasoning process with high
solution, then generate dagree of granularity.
will provide you with "Problen" and its "Grourd-Truth solution".
<v [INSTRUCTIONSS]
Based on the provided input, you genezale exactly THREE componeats. For each component, you MUST generate н*а minimum of aumbered items...
If naturel breakdown results in fewer than items, you must resuiiivdde the existing steps
detailed, finer-grained sub-stepses For example, a single calculation step can be broken down into 'subscituting va values", 'performing the operazion', and 'stating the result'.
the requirament.
planning steleton. This should concise, crdered 1ist of the key reasoning steps and the overall strategy used to reach. solution.
#sPlanning SkeletonvA:
high
include detailed calculations, just logical flow.
down into at least 4 detailed szeps. **Knowledge Components**: identify facts, definitions, theorems, lemmas, or formulaz) required to solve the problen. List each component elearly in
essential knowledge components (like
3. +*solution Rreakdownes: Rivida the origiral Ground-Truzh Solution irto numbered sementisally coherent steps chunks. Rach itam in the lt should ba direct quote or faithful summary of part of the original solution text.
should
stops
17

---

Published as a conference paper at ICLR 2026
** [OUTPUT FORMAT]*
provide your response the following structured format. Ensure each section contains at least icens.
<PLANNING <PLANVING__SKALETTN SKELETON>
[Ttem
[Ttem
(and more if applicable)
<BSOWLEDGE COMPONENTS>
1. [Item 1]
4. [Itom 4]
(and more appl icable)
</KNONLEDCE, SMMDNLEDCE_CCMPOSENTTS
<SOLUTION <S0LUTIOILBECAKDOWN BREAKDOWNS
1. [item 11
4. [item 41
(and if applicable) REGOLUTICN_BRAAKKOOO
** [EXAMFIR)
BESTN EXAMPAR
(Example
Solution same)
END EXAMPLE
*v [EXPECTED OUTPUT FOR THE EXAMPLE] +
(this example demonstrates required granularizy with items per section)
<PLANNING
1. Tdenti fy the geometric shape (right-angled triangle) and the goal [find the bypotenuse).
2. Recall the ralevanz nathemation loal theorem that connects the sides of right-sngled triangle,
3. Formulate the equazion by substituting rg the given side lengths (base and height) into
the theorem,
Execute the arithmetic calculation find the of the hyposenuse.
5. Perform the final step of taking the square reot to isolete the length of the hypotenuse.
DFPLANNING_SKKLERR
<KNOWLEDGE COMPONENTS>
Pythagorean Theorem (a+ b4 o² for right-angled triangle). 2. Definition: Hypotenuse (The longeat side of right-angled triangle, opposite the right anglal.
3. Concept: Righi-angled Triangle (A triangle with one angle measuring 90 degreea) Mathematicel Opecations Square (The
operation squaring number)
</KNOWLEDER_CO CUMPUNIETTS
hypotenuse of right-angled triangle, can use Pythagorean theoren, which states that at + b =i, where and b are the lengths of the two shorter sides (lega) and a
To
length the hypoterusa. 4 and à cm..
2. The given values are Substitucing these into the formula gives:
25. This means e* 25. Taking the square root of both sidea results in
</S0LUTION_DRCADOOSS -
* [TASK]+*
New, please the follewing problem and solution, strictly following all instractions.
s.Frchlemax:
[question]
-vGround-Truth Solutionvas (sclution)
18

---

Published as a conference paper at ICLR 2026
E.2 INJECT HINTS PROMPT
During the Hierarchical Hint-Guided Exploration phase, when the model fails to solve a "true-hard" problem, Seaf-GROO injects a hint directly into the input prompt. This approach is fundamental to our on on-policy methodology, as reframes the problem for the model rather than forcing it to continue partial, off- policy trajectory. The design of this prompt is crucial: it explicitly informs the model that it is receiving guidance, ensuring that the model processes both the problem and the hint under single, unified policy, thereby avoiding the distributional shifts common in prefix-continuation methods.
The exact prompt template used for hint injection is shown below.
Prompt for hint injection
# System message Plense reasen step by step, and put your final within \boxed().
# query
Qusetion: [(question))
Solution /P1 http://laannnnnnnoo/o/// ldde Rintss (thrrt]]
F FORMAL ALGORITHMIC AND MATHEMATICAL DESCRIPTION OF SCAF-GRPO
This section provides formal mathematical and algorithmic description of the Scaffolded Group Relative Policy Optimization Geaff framework. We formalize the two-phase training pro- cess and detail the construction of the loss function, particularly for the hierarchical hint-guided exploration phase.
F.1 PRELIMINARIES: THE STANDARD GRPO OBJECTIVE
We begin by restating the core Group Relative Policy Optimization (GRPO) objective. For given prompt q. the policy Ta generates group of trajectories, G = {o).... ., ON J. where each trajec- tory is sampled from the policy, o, xe(-)g). Each trajectory receives terminal reward R(o,) from an external verifier.
The normalized advantage for each trajectory 0 in the group G is calculated as:
A R(o:) ng
(6)
og + Esdd
where jgg and og are the mean and standard deviation of rewards in G, and cad a small constant for numerical stability.
The standard GRPO objective, which maximized during training, a clipped surrogate objective defined as the empirical expectation over trajectories and timesteps:
Jo) Bi, min (O), clip(r:.. -, (0,1,,,,+),)
where (0) =o(0)_1o0..e.g)) is the probability ratio for the token at timestep of trajectory i between the current and old policies, and is the clipping hyperparameter. The "learning cliff" phe- nomenon, a key challenge addressed by our work, occurs when R(o,)) 0 for all i [1....,N) In this scenario, jıg and ag become zero, causing the advantage Å, to collapse zero for the entire group and stall the learning process.
F.2 THE SCAF-GRPO TRAINING PROCESS: A TWO-PHASE FORMULATION
Scaf-GRPO modifies the training process by strategically augmenting the trajectory group when a learning cliff is detected. The process consists of a conditional batch construction procedure followed by the application of the standard GRPO loss.
19

---

Published as a conference paper at ICLR 2026
Algorithm Scaf-GRPP Batch Construction and Update
Require: Policy To: Prompt qi Curreut training step E; Guidance exemption end-step Texempt: Verifier V; Number of rollouts per prompt N.
Ensure: Final trajectory group Gand for loss computation.
Step 1: Standard On-Policy Generation
4-
2: for i= to N do
0: mo(-)))) Gu
Generate N trajectories for the same prompt
5: end for (R(o1),..., R(oN)) < V())
Evaluate rewards for the group
Step 2: Learning Cliff Monitor
CauR t (E R(o,)= =0 0)
if Texempt and CeH then
Guidance is active and needed
Step 3: Hierarchical Hint-Gie Guided Exploration
(o7, h") +- SSarchhfienrrciccaHHnntssg mo)
Search for minimal effective hint B successful guided trajectory was found
10:
If o null then
Step 4: Batch Augmentation
11:
Randomly select index E (I,..., N) of failed trajectory.
12:
GEnal (g \ (
& Replace one failure with success
13:
return Ganal
14:
end if
15:
end if
Default case: No intervention
Günal + G
Use original batch if cliff guidance failed
17: return Ghual
Figure 6: Overview of the Scaffolded Group Relative Policy Optimization (Seaf-GRPP) Algorithm.
Let t denote the current training step and T Texmmpe be the step which the guidance exemption period ends. The core logic detailed in Figure 6.
The function SearchticcarchiicllHittsss TQ) represents the deterministic, multi-level search de- scribed in Section 3.3 and Appendix
iterates through the pre-defined hint hierarchy H. (Hanowledge Hplanning: Hsotation J to find the first hint h- that enables the policy тą to generate suc- cessful trajectory of <- h²", where i denotes the concatenation of the hint into the prompt. If no hint leads to solution, it returns (null, null).
F.3 THE UNIFIED SCAF-GRPO LOss FUNCTION
The core insight Scaf-GRPO that does not alter the mathematical form of the GRPO loss function. Instead, it modifies the data distribution used for the loss computation by conditionally augmenting the batch.
Let Grinal denote the group of trajectories returned by Figure for given prompt at step f. This group composed of trajectories sampled under one of two conditions:
(1) Standard Generation: All N trajectories are from re(-))). This occurs if the learning cliff is not triggered if the training is within the exemption period.
(2) Augmented Generation: N trajectories are from <e(-1g) (with zero reward), and one tra- jectory, of, is from Te @ h*) (with positive reward). This occurs only when the learning cliff is triggered post-exemption and the hint search is successful.
The Scaf-GRPO loss function is therefore defined by applying the standard GRPO objective this conditionally constructed batch. First, the advantage is computed on the final group Greal:
ÀG R(O()
OGam + for of E Grinal
(8)
The overall objective is then:
Jscaf-GRPO (0) = B.,, min +
20

---

Published as a conference paper at ICLR 2026
where the probability ratio ri (0) for each trajectory of Ginal is critically computed with respect to its specific originating prompt:
To Tene (0) (0,, (..)))))
if of E Grial and of + 0%
(0)
(10)
su soloo,el,, ce
if of = =
swaa (0), (11
By reformulating the batch rather than the loss, Scaf-GRPO ensures that when learning signal is absent (A, 0), new signal injected by providing single successful, minimally-guided trajec- tory. This intervention re-establishes non-zero reward variance within the group, reactivates the advantage calculation, and enables learning resume on previously intractable problems, thereby directly overcoming the learning cliff.
F.4 CONSERVATIVE NATURE AND PRESERVATION OF THE ON-POLICY OBJECTIVE
A crucial property of Seaf-GRPO is that it does not alter the fundamental GRPO optimization ob- jective. Instead, it operates as conservative data augmentation strategy that activates only under the specific condition of learning cliff. We formalize the framework's impact the policy gradient by analyzing two distinct cases based on the initial on-policy sampling results for a given prompt q-
Case 1: At least one successful trajectory Ao,; E 9 such that R(o:) > 0). In this scenario, the initial group of trajectories G already contains a non-uniform reward signal, meaning Mi > and og 2 The condition for triggering the hierarchical hint-guided exploration is not met. Consequently, the final batch used for the update is the original batch, Grinal G. The Scaf-GRPO objective function therefore mathematically identical to the standard GRPO objective:
Jscaf-GRPO (0) = (G)
(11)
In the most frequent training scenarios where the model has some capacity to solve the problem, our framework makes no modifications and is equivalent to vanilla GRPO.
All trajectories fail (Vo; E G, R(o,) = 0). This is the learning cliff scenario. In standard GRPO, the rewards are uniform and zero, causing the advantage calculation to collapse: Ag = 0 and og = 0, leading to Å, 0 for all trajectories. The resulting policy gradient for this prompt is zero, and no learning occurs.
Case 2:
Scaf-GRPO intervenes by constructing an augmented batch Grial- This batch consists of N - 1 of the original failed trajectories and one new, successful trajectory of (T(-19 hr).. The crucial insight is that this new trajectory is generated on-policy by the current policy ne. conditioned on the hint-augmented prompt.
The key benefits of this intervention are:
(1) Gradient Restoration. The augmented batch Grinl now contains least one trajectory with positive reward. This ensures that (4g... > and dg.. > 0, which in turn guarantees non-zen zero advantage signal Á for the trajectories in the group. Learning is effectively restored where it would have stalled.
(2) Preservation of the On-Policy Principle. Unlike off off-policy methods that mix trajectories from different policy лę and require importance sampling corrections (e.g.. Te ) to account for the me.
distributional shift, Scaf-GRPO's guided trajectory of, is sampled directly from the current pol- icy To. Therefore, the probability ratio r',, is standard on-policy ratio computed each timestep. This avoids the high variance and potential instability associated with off-polic cor- rections, ensuring that the learning signal remains stable and directly attributable the current policy's capabilities.
In summary, Seaf-GRPO does introduce any harmful bias or modification the GRPO objective. targeted intervention that is inactive when a valid learning signal already exists. When the learning signal vanishes, it provides constructive, on-policy gradient by minimally augmenting the task, thereby transforming an unproductive training sample into a valuable learning opportunity without compromising the integrity of the on-policy optimization process.
21

---

Published as a conference paper at ICLR 2026
F.5 COMPARATIVE ANALYSIS OF ON-POLICY STABILITY AND DISTRIBUTIONAL MISMATCH
To elucidate the design rationale behind Scaf-GRPO, we provide a comparative analysis against an alternative off- off-policy formulation. A natural question arises regarding whether the trajectory gener- ated via the tint-augmented prompt (q + h²) could be used to directly update the policy conditioned on the original prompt (g). We analyze this alternative from three perspectives: theoretical stability, distributional mismatch, and empirical performance.
Theoretical rationale for on-policy formulation. The core mechanism of GRPO relies on the probability ratio r0) measure the divergence between the current and old policies. This ratio is constrained by a clipping mechanism, clip(r:(0), e, + e), to prevent destructive updates. Stability requires that the ratio remains close to unity, implying the numerator and denominator distributions must be well-aligned.
In Ceaf,GROO, when learning cliff necessitates the use of a hint h", we augment the input context for the current policy. Consequently, the probability ratio for a guided trajectory is computed as:
r+( = TT(. @ h²))
(12)
Tew (-1g 1 h*)
This formulation maintains a strict on-policy property. The numerator and denominator are con- ditioned on the identical context (g @ h²). As a result, the distributions remain closely aligned,
maximizing the likelihood that (0) falls within the stable trust region [1 e, 1 + + This ensures
smooth gradient updates even when the model operates in a guided state.
Instability of off-policy alternatives. An alternative approach involves using the hint-guided tra- jectory to optimize the policy for the original prompt directly. This implies an off-policy ratio formulation:
TT(-11) To.. (.9 + h*)
F (0)
(13)
We argue that this formulation introduces a fundamental distributional mismatch. The numerator is conditioned on q. while the denominator is conditioned on q $ h". Since h² is selected specifically to alter the probability landscape and guide the model toward a correct solution that was previously inaccessible under g, the distributions x-11g) and 7-119 + h²) are inherently divergent. This discrep- ancy causes the ratio (0) to fluctuate significantly. frequently violating the [1 c, + c] bounds. Such behavior forces the optimization algorithm rely heavily on clipping, thereby truncating the learning signal and introducing high variance into the training process.
Empirical validation of training stability. To validate this theoretical analysis, we conduct comparative experiment implementing the off-policy formulation described above. We monitor the clip ratio, defined as the fraction of tokens where the probability ratio falls outside the stable interval [1 - c, 1 + c]. A high clip ratio indicates that the policy update is being constrained due to excessive divergence.
Figure illustrates the training dynamics. The proposed Scaf-GRPO maintains a consistently low clip ratio, confirming that the on-policy formulation preserves distributional alignment. In contrast, the off-policy alternative exhibits a significantly higher and more volatile clip ratio. This empir- ical evidence supports the hypothesis that the distributional mismatch in the off-policy approach destabilizes the optimization landscape.
Impact on downstream performance. The instability observed during training directly corre- lates with inferior final performance. Table presents a performance comparison across mathemat- ical reasoning benchmarks. The model trained with the off-policy ratio suffers from performance degradation compared to Scaf-GRPO. Specifically, the inability maintain stable updates limits the effective transfer of knowledge, whereas our on-policy approach effectively leverages the scaffolded trajectories to improve the model's reasoning capabilities.
22

---

Published as a conference paper at ICLR 2026
off-policy on-policy(ours)
0.4
Clip Ratio
0.2
50 100
150
200
250
300
Figure 7: Comparison of the clip ratio during training. The proposed Scaf-GRPO (red) maintains a low and stable clip ratio, indicating effective on-policy learning. The off-policy alternative (blue) exhibits high volatility and frequent clipping. indicative of distributional mismatch and training instability.
Table Performance comparison (Pass %) on Qwen-2.5-Math-7B. We compare Vanilla GRPO, the Off Off-Policy alternative, and our proposed Scaf-GRPO. By maintaining optimization stability, Scaf-GRPO achieves superior performance across all datasets. Best results are highlighted in bold with background color.
Method
AIME
AIME 25
AMC
Minerva
MATH-500
Olympiad
Gaokao2023en
Avg.
Qwen2.5 Swen2.5-Math-78 Math-7B
Vanilla GRPO
30.0
13.3
60.0
33.4
75.8
41.3
62.6
45.2
Off-Policy Alt.
36.7
13.3
65.0
34.2
78.2
38.5
65.2
47.3
Scaf-GRPO (Ours)
43,3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
G ADDITIONAL ABLATION STUDY AND ANALYSIS
G.1 ABLATION ON THE GUIDANCE EXEMPTION PERIOD
To validate the design of our guidance exemption period (Phase 1), conduct two-fold analy- sis: first establishing the necessity this phase, and then analyzing the method's sensitivity to its duration.
Necessity of the Exemption Phase. We first investigate whether an initial autonomous learning phase necessary by comparing our full Scaf-GRPO framework against variant where scaffolding is activated from the very beginning ("Scaf-GRPO w/o Phase 1"") We posit that Phase I is crucial for distinguishing between "true-hard" problems (genuine capability gaps) and "pseudo-hard" problems (superficial errors like formatting). Applying scaffolding prematurely to pseudo-hare hard cases fosters dependency, preventing the model from developing robust, independent problem-solving habits.
The results in Table 10 confirm this hypothesis. While activating scaffolding from the start yields improvement the vanilla GRPO baseline, it is notably inferior to the complete &caf-GRPP framework. This performance gap underscores that allowing an initial phase of unguided learning is critical to prevent over-reliance on hints.
Sensitivity to Exemption Duration. To further investigate the optimal duration of Phase 1 and validate the robustness of our method, we evaluated wide range of exemption ratios: 0%, 5%, 10%. 15%, 20%. 40%, and 100% of the total training steps. Here, 0% represents immediate scaf- folding, while 100% is equivalent the vanilla GRPO baseline. The detailed results are presented in Table 11.
Our analysis reveals three key dynamics:
23

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on the necessity of the guidance exemption period (Phase 1). We compare the full Scaf-GRPO framework against vanilla GRPO and Scaf-GRPO variant without the initial exemption phase. Scores: pass @ (%) Best results in bold.
Model
AIME 24
AIME
AMC
Minerva
MATH-500
Olympiad
Gaokao202.en
Avg.
Qwen2.55 M.5t 1.58
Vanilla GRPO
13.3
10.0
47.5
28.3
72.2
34.8
57.4
37.6
Scaf GRPO (w/o Phase 1)
10.0
10.0
57.5
27.5
71.4
36.3
57.9
38.7
Scaf-GRPO
20.0
13.3
29.1
73.4
36.6
57.9
@wen2.5 5-Math- 7B
Vanilla GRPO
30.0
13.3
60.0
33.4
75.8
41.3
62.6
45.2
Scaf-GRPOO (w/o Phase 1)
23.3
13.3
34.2
78.4
41.2
Scuf-GRPO
43.3
20.0
36.4
80.0
43.3
63.4
Premature guidance is detrimental: Short exemption periods (0% and 5%) yield the low scores among scaffolded variants (46.2% and mn.67..6% 47.6%, respectively). This confirms that applying guidance too early prevents the model from learning to resolve minor errors independently.
Autonomy alone is insufficient: The 100% setting (Vanilla GRPO) results in the worst overall performance (45.2%). This demonstrates the "learning cliff" effect: without scaffolding interven- tion, the model remains stuck on true-hard problems due to persistent zero-reward signals.
Robustness across wide effective range: Crucially, once the exemption period is sufficient (over 10%),, Scaf-GRPO consistently achieves a high-performance plateau. As shown the table, average scores for periods between 10% and 40% remain stable and high (ranging from 49.5% to the peak of 50.9%). This demonstrates that our framework not overly sensitive this hyperpa- rameter, as long as the model has initial phase stabilize its capabilities, activating scaffolding effectively unlocks learning for the remaining intractable problems.
Table 1: Ablation study the Guidance Exemption Period duration. We investigate the impact of varying the initial autonomous learning phase (from 0% to 100%) on pass @ performance (%). The 15% % setting (Our Method) achieves the optimal trade-off. Best chosen configuration is highlighted in with background color. Best results are in bold.
Exemption Period
AIME 24
AIME 25
AMC Minerva
MATH--50
Olympiad
Gaokao2023en
Avg.
Oven2.5-Math-2
(w/ Phase
34.2
78.4
41.2
63.1
65.0
33.8
76.2
41.1
63.9
35.8
42.5
63.5
Method)
36.4
43.3
20%
46.7
65.0
36.0
80.4
43.5
65.4
50.0
40%
70.0
34,9
78.2
41.9
64.6
100%
(Vanilla GRPO)
30.0
13.3
60.0
33.4
75.8
41.3
62.6
45.2
G.2 ABLATION ON DATA FILTERING
We determine our dataset composition by classifying problems based on the model's initial pass rate over = 8 sampling attempts. The training samples are divided into three categories: (1) "Too Easy" (solved 8/8 times), which are discarded; (2) "Potentially Solvable" (solved 1-7/8 times); and (3) "Too Hard" (solved 0/8 times), which are fully retained.
To identify the optimal data mixture, we evaluate Siven.5-Matth 7B under four configurations rang- ing from using the full dataset to training exclusively on hard samples. Table 12 presents the results. We observe that removing "Too Easy" samples improves performance over the full dataset baseline. Furthermore, randomly subsampling the "Potentially Solvable" category 50% ratio yields the highest Scaf-GRPO score of 50.9%. Based on these empirical results, we adopt this configuration (w/o "Too Easy" & w/o 50% Solvable) for main experiments.
24

---

Published as a conference paper at ICLR 2026
Table 12: Ablation study on dataset filtering strategies. We compare the Scaf-GRPO performance under different dataset settings. The selected configuration is highlighted.
Dataset Setting
Scaf-GRPO
Full Dataset
46,2
w/o "Too Easy"
48.8
w/o "Too Easy" & w/o 50% Solvable (Ours)
50,9
w/o "Too Easy" & w/o 100% Solvable
45.2
H HINT QUALITY ASSURANCE AND IMPACT ANALYSIS
To ensure the reliability our scaffolding mechanism, we rigorously assess the quality of the gen- erated hints and analyze their impact on model performance. We address this from two perspec- rubric, and (2) empirical validation linking hint quality
tives: (1) assessment using a to downstream model accuracy.
Multi-Faceted Rubric. While our final hints are generated using the DeepSeek-RI model, we selected this generator after comparing it with other capable models, such as Qwen2.5-72B-Innst
Assessment Methodology:
To objectively compare the quality of hint sets, we developed rubric designed to assess hints not only for correctness but for their pedagogical value specificallll their ability to provide minimal, clear, and structured guidance. Our rubric evaluates hints across four key dimensions: (1) Accuracy: Factual and logical correctness of the mathematical content; (2) Minimality: The degree which the hint provides the least necessary support, thereby preserving student autonomy; (3) Clarity: The use of unambiguous and easy-to-pars language; and (4) Struc- tural Coherence: The logical progression and distinctness between abstract (knowledge/planning) and concrete (solution) tiers.
We employed an LLM-as-a- a-dge (Gemini-2.55 to automatically score hint sets based on this rubric. The comparative results are presented in Table 13. The DeepSeek-RP R1 generated hints achieved superior aggregate score, particularly excelling in structural coherence and minimality.
Table 13: LLM Judge scores for hint quality from different generator models (Scale 1-5),
Hint Generator Model
LLM Judge Score
Qwen2.5-72B 20t.555-7BB--ntruu DeepSeek- R1 (Selected)
4.38
4.72
To validate the hypothesis that higher-quality hints lead to better training outcomes, we trained the Qwen2.5-Math, model using both sets of hints gewen-generated vs. DeepSSeek-generated under identical training configurations.
Empirical Validation: Hint Quality vs. Model Performance.
The results, detailed in Table 14, demonstrate clear positive correlation between the rubric score and final model performance. The model trained with the higher-rated DeepSeek-RR RI hints achieved an average pass 1 score of 50.9%, compared 49.0% for the model trained with Qwen hints. Notably, performance gains were most pronounced on difficult benchmarks like AIME24 (+6.6%) and Minerva (+2.6%), suggesting that high-quality, minimal scaffolding is particularly critical for mastering complex reasoning tasks. This validation confirms that our LLM-based evaluation rubric serves reliable proxy for hint utility.
Prompt for hint evaluation
[ROLE sCAL]++
You aze an exper: pedagogicel eveluator and mathematics content specislist. Your task to obfectively assess the quality "sint Sets* generazed specific math prohlem. You will evaluate how we' thase hirts guida a student tewarda the solution without giving away ansver directly,
multi-faceted rubric.
[INPUT]**
25

---

Published as a conference paper at ICLR 2026
will previde you with:
*<Problem=e: The originel math question.
*sbround-Iruch solutionas:
cosrect step-by-step
3. **Senerated Bintsee: A set of hinta (which may includa a Flanning Skeleton, Knowledge Compenents, step-wise Breakdewn) produced by AL model.
[SCORING +UBRIC]**
You must evaluate the hinta on a scale of 1 to 5 for each of the following four
**ccuurac (1=5)
logical correctness.
+5+ All information is mathematically precise, correct, and strictly aligns Ground-Frute Iruth
+l+; Contains major natheratioal errors, tollucinociens, or leads the student an incorrect path.
**Minimality (1-b1++: Providing the leaat neceasary support to preaerve onomy, <5xt Hincs are concise and sarfolddrg-oriienedd. they guide the vreasoning process. it "how" and "why") rather than just stating intermediate ca ion resulls, They de not *apcon Ieed" the
<1+; Hinzs are overly verbose, isrelevant, or give eway the ansxer/key numbers immediately, destroying the student's opportunity te think.
*ellority (1=5)++: Unambigueus and easy-to-parse language.
+5+: Language is simple, direst, and grammstically perfect. Methematical. notation standard and consistent.
+1+; tanguage la confusing, ambiguous, overly complex, or contains formatting errors that make it hard to read.
#*Structural Coherence
Logical from v5x: The hints follow logicsi progression (e.g., Plan -> Concepts -> Steps). altempling
concrele,
sense w1x: The hints are disorganized, randon, or jump betwesn unconnected ideas wichout clear thread.
problem.
+*Aralyze the Tnput**; Read the Problen em and Ground-Truzh Solution to understard the
lsgic.
2. ++Verify the #intser: Check the Cenerated Bints against the Sclutior for Acouracy,
3. **Asesss Pedagogy*x; Evaluate if the hints explain vprincipless (high
M Dtrimalliy//ohheeee
just vea culate numbers. flow Winimalitl.. 4. .+Draft Rationaleess Foz each dimension, write belef justification for your
5. **ssign Scores+:: Give a final integer score (1-3).
[OUTPUT *
provida your responae in following atructured format:
<FVALORTION RATTONALE>>
*<coouracys:: [Analysis...]]
svMinimalityas: [Analysis....
*<Clarity.s: [Analysla....
kvStructural Cchezencewv: [Amalysis....
C/SVALURTIONLAATIOONE
<SCORES>
Accuracy: [1-5]
Minimality: [1-5]
clarity: [1-5]
structural Coherence: [1-5]
</SCORES>
kv [BXAMPLB]
DEGIN EXAMPLE
#+bbblem=:: Find the value of
Grrondd-rruth solutionse: Subtract 3 from both sides: 2x 6. Divide by 2: x
*vGenerated sintste:
1. First, you need to isolate the variable x.
Subtrrat
get
3. Then divide 5 by to get the final ensver 3,
END EXAMP1E
-EEXPECTED
FOR EXAMPLE)**
<EVALUMTION
*>Accurayy+:: The rath is correct. The operations lead to the right answer, (High) Step explicitly perform the calculation (*gez 6", "answer 3*) instead of just suggesting the operation ("Subtract from both sides"). This removes student autoromy. (Tоw)
w-bininality:::
not
26

---

Published as a conference paper at ICLR 2026
*>Clarity=:: The language is very clear and easy underssend. (8iga)
.*Structural Cchezence*. The flow is Logical (Methed -> Calculation Result). (Bigh)
('XVAUUATIOOLMACOOOAAA)
<SCORES>
Accuracy:
Minimality:
Clarity: 5
Structural Coherence:
</BCORES>
.. [TASK]]**
Now, please evaluste follewing input strictly following instructions.
*>Froolemre: (question)
x>Ground Truth Solutionv4: (solution)
.*Generated Bintsaw: (hints)
Table 14: Impact of hint source quality model performance. We compare the final pass @ 1 (%) of Qwen2.5-Math 7B trained using hints generated by Qwen2.5-72B Insttrcct versus our selected DeepSeek-RI. The best results are in bold and the selected configuration is highlighted.
Hint Source
AIME
AIME
AMC
Minerva
MATH-500
Olympiad
Gaokno2023en
Avg.
Qwen2. 5-Mahh-78
w/ Qwen2.5-72B. (aaen2.-77B--I-siie Instruct Hints
36.7
20.0
67.5
33.8
79.4
42.2
63.1
49.0
w/ DeepSeck-RI Hints (Ours)
43,3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
I ADDITIONAL EVALUATION AND ANALYSIS
This section provides extensive supplementary evidence to corroborate the effectiveness of Scaf- GRPO. To demonstrate that our method's improvements are robust and not limited to specific met- rics, we report performance across varying sampling configurations and compare against an ex- panded suite of competitive baselines. We also delve deeper into the model's generalization capabil- ities on challenging OOD tasks and analyze its behavior when integrated with compared against other method (e.g., DeepScaleR Luo et al. (2025), DAPO Yu et al. (2025)). The results consistently highlight the stability of Scaf-GRPO and its ability establish well performance complex reasoning tasks.
Robustness Analysis: From @ass@1 1 to Avg@16. While the pass@ metric via greedy decod- ing is standard benchmark, it can exhibit fluctuations due to the inherent stochasticity of LLM generation. To ensure the reliability of our results and demonstrate that our improvements are not artifacts of variance, we conducted a comprehensive re-evaluation using the avg@ 16 metric. For this analysis, we generated 16 distinct samples for each problem with temperature 0.6 and top-p of 0.95, averaging the binary success scores. As presented in Table 15, Scaf-GRPO consistently maintains significant performance advantage over the Vanilla GRPO baseline and other methods across all model architectures. Notably, on Qwen2.5-Math-7B. our method achieves an average score of 13%,, surpassing strong baselines like Oat-Zero Liu al. (2025a) and LUFFY Yan al. (2025). These results confirm that the capability gains fostered by Scaf-GRPO are robust and stable under rigorous multi-sample evaluation.
Comparison with Expanded Baselines. To rigorously contextualize the performance of Scaf- GRPO within the broader landscape of mathematical reasoning, we expand our comparative analysis to include a diverse set of representative 7B-parameter models. We classify these baselines into two categories: (1) Short Chain-of-Thoure (Short CoT) baselines, represented standard base models prompting with concise reasoning steps; and (2) Long Chain-of-Thhught (Long CoT) baselines, comprising recent state-of-the-an models optimized for extended reasoning via advanced instruction
27

---

Published as a conference paper at ICLR 2026
Table 15: Overall performance on seven benchmarks using avg metric. We compare our method, SCAF-GRPO, against vanilla GRPO baselines across diverse architectures. Best results in bold. The background color of Scaf-GRPP cells indicates performance change Vanilla GRPO (green for improvement, red for decline).
Model
AIME 24
AIME 25
AMC
Minerva
MATH-300
Olympiad
Gaokuo2023en
Avg.
Qwen2 5.Mach. 1.58
Vanilla GRPO
57.4
SeaF-GRPO
14.7
10.8
50.9
31.4
74.8
37.4
58.8
39,8
Qwen2.. M.tthb-
Vanilla GRPO
30.7
10.8
61.1
33.4
74.2
41.1
62.6
44.8
SimpleRL -Zero Zeng (2025)
22.5
Oat-Zero Liu al. (2025a)
30.7
11.5
62.7
35.2
78.2
41.4
63.0
46.1
LUPPY Yan al. (2025)
31.9
12.0
60.4
33.0
74.3
40.6
61.9
44.9
Scaf-GRPO
Qwen2..5.78
Vanilla GRPO Scaf-GRPO
11.0
9.5
53.0
36.7
76.8
40.4
64.5
41.7
15.6
11.0
54.2
37.0
77.6
41.2
64.8
43.1
Vanilla GRPO Scaf-GRPO
7.6
0.0
28.7
18.7
46.1
24.4
9.6
0.3
32.8
19.1
54.7
21.3
46.4
26.3
DeepSeek-A Ri--iiill--uen-------- 1.38
Vanilla GRPO Scaf-GRPO
27.1
22.9
32.5
75.3
51.7
28.5
25.3
75.1
35.0
85.7
54.5
76.3
54.3
tuning or reinforcement learning, including Eurus-2-PRR Cui al. (2025), rStar-Math Guan al. (2025), and OpenReasoner-Zero Hu et al. (2025). As detailed in Table 16, Scaf-GRPO consistently outperforms both categories. It not only surpasses base models by wide margin but also exceeds other 7B RL-tuned models, particularly high-difficulty benchmarks like AIME 24 (43.3%) and AMC 23 (70.0%).
Table 16: Comparison with Short CoT and Long CoT baselines on 7B models. We report the pass @ accuracy (%) on various mathematical benchmarks. Scaf-GRPO (Ours) achieves the best average performance among all 7B models. The best results in each column are highlighted in bold.
Category
AIME
AMC
Minerva
MATH-500
Olympiad
Gaokao
Avg.
Qwen-2.5.77 2.n.n225-BBBBBB Base
10.0
6.7
37.5
26.4
61.8
34,4
42.6
31.3
Qwenn2.5.Mathe Math-7B-BBatt
13.3
13.3
42.5
16.5
53.6
18.2
35.1
27.5
Bunus-2-7B-PRIME Cui et (2025)
26.7
16.7
55.0
38.6
74.4
34.7
59,3
43.6
1Star-Math-7B Guan al. (2025)
26.7
13.9
47.5
32.2
78.4
47.1
59.2
43.6
SimpleRt-Zero Zeng (2025)
37.2
42.6
Long CoT
LUTTY Yan (2025)
33.3
16.7
62.5
33.8
75.2
41.7
62.7
46.6
Oat-Zem Li al. (2025a)
30.0
16.7
62.5
34.6
78.0
41.0
62.9
46.5
OpenReassner-2een Hu et (2025)
26.7
16.7
50.0
35.6
77.4
41.6
67.2
45.0
Seaf-GRPO (Ours)
43.3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
Integration with Advanced Models: DeepScaleR. demonstrate the compatibility of Scaf- GRPO with other training paradigms and its ability to enhance data efficiency, we conducted an experiment using the -..-Seaale--15--Peeeeee
already converged using standard GRPO on the Luo et al. (2025) model. Since this model has Peeeeaalee-eeeee---aaaaaaa
simply continuing training with Vanilla GRPO on the same data results in performance regression, as indicated by the drop in the average score (from 55. 1 to 54.0%) ) in Table 17. In contrast, Scaf-GRPO successfully extracts further value from the identical dataset. By leveraging the hint-guided mechanism unlock previously inaccessible reasoning paths, our method achieves a clear positive gain, raising the aver- age score 59.1 % and showing significant improvements on challenging benchmarks like AMC 23 and OlynpiadBench. This confirms that Scaf-GRPO can effectively complement existing advanced checkpoints to maximize data utilization.
Comparison with DAPO. Both Scaf-GRPO and DAPO (Yu et al., 2025) aim to mitigate the van- ishing gradient problem caused by consistent failure difficult queries. However, the methods
28

---

Published as a conference paper at ICLR 2026
Table 17: Performance comparison of Scaf-GRPO versus Vanilla GRPO when continuing training the DeepScaleR-1.55 L.S-pale--.SBBPeeviiee model. We report the pass @ (%) accuracy on various benchmarks. The best results are in bold.
Method
AIME 24
AIME 25
AMC
Minerva
MATH-500
Olympiad
Gaokao2023en
Avg.
DeepScaleR-1.33 B------reeee Luo et al. (2025)
Base Model
43.1
30.0
73.6
30.2
87.8
50.5
70.4
55.1
Vanilla GRPO
40.0
30.0
72.0
30.5
86,9
49.1
69.2
54.0
Scaf-GRPO (Ours)
40.0
33.3
85.0
33.5
87.8
55.0
79.0
59.1
diverge fundamentally in their strategy. DAPO employs filtering mechanism (Dynamic Sampling) that effectively omits these zero vero-gradient samples maintain training stability. In contrast, Scaf- GRPO adopts an intervention strategy: rather than discarding hard problems, we utilize hierarchical hints to guide the model toward correct solution, transforming these failures into valuable learning signals. As shown Table 18, this active approach yields superior results. On the Owen2.5-Math- outperforms DAPO across all benchmarks, achieving higher average accu- racy (50.9% vs. 48.5%). This suggests that scaffolding the model conquer its hardest challenges is more effective for capability advancement than simply stabilizing the training distribution.
7B model,
Table 18:
Performance comparison on Qwen2.5-Math-77 7B (pass@) 1, %). We compare our Scaf- GRPO against Vanilla GRPO and DAPO baselines. The best results bold.
Method
AIME 24
AIME 25
AMC
Minerva
MATH-500
Olympiad
Gaokao23
Avg.
Owen2.5-Mathh-
Vanilla
30.0
60.0
33.4
75.8
41.3
62.6
DAPO Yu et al. (2025)
36.7
20.0
62.5
35.6
79.6
42.3
63.0
48.5
Scaf-GRPO (Ours)
43.3
20.0
70.0
36.4
80.0
43.3
63.4
50.9
29


Reviews:
[{"review_id": "l0FnFu3Ea4", "reviewer": "Reviewer_QtTi", "summary": "This work is motivated by the observation that GRPO could perform badly when the problem is hard to solve and successful trajectories are hard to collect. To address this challenge, this work proposes Scaf-GRPO, a method that focuses on the data augmentation aspect rather than modifying the GRPO algorithm itself. Specifically, Scaf-GRPO encourages the model to generate hints that improve the success rate, thereby accelerating RL convergence.", "questions": "1. During RL training, is Scaf-GRPO applied to both easy and hard problems, or only to the hard ones? Additionally, how do you determine whether a problem is considered easy or hard?\n2. What is the behavior during inference? Do you use the hint prompt to encourage the model to generate hints for all problems?\n3. In Line 398, could you explain what is meant by “the most concrete hint”?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written and well organized.\n2. Sufficient experiments are provided to validate the effectiveness of their method.", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sPb7Jgxf41", "reviewer": "Reviewer_HrXm", "summary": "This paper studies the \"learning cliff\" issue in GRPO, where the solution to a hard problem cannot be learned if all the sampled solution are incorrect. To overcome this issue, this paper propose a scaffolding framework, which provides some hints of different level of concreteness to stimulate on-policy correct answers. These correct answers avoids zero-gradient on the hard problem and provide useful training signal, therefore makes use of the hard problems. Experiments shows that the proposed method outperforms vanilla GRPO and other off-policy guidance baselines.", "questions": "See weakness section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper is listed as follows\n\n1. The method proposed in this paper adapts external knowledge (solution) to on-policy solutions, which avoid the distribution shift resulted from directly utilizing off-policy solution.\n\n2. The experiments are conducted on several different models and Scaf-GRPO consistently improves over baselines. Also, the ablation study is comprehensive and covers most of the design of proposed method.\n\n3. Overall, the paper is clearly written.", "weaknesses": "In general, I think this paper does not demonstrate any major weakness. Some of the identified weakness and my questions are listed below\n\n1. In equation (4), the author propose to use $\\pi_{\\theta}(\\cdot|q\\oplus h*)/\\pi_{\\text{old}}(\\cdot|q\\oplus h*)$ as the importance ratio. However, this does not exactly matches the probility $\\pi_{\\theta}(\\cdot|q)$. To the reviewer, $\\pi_{\\theta}(\\cdot|q)/\\pi_{\\text{old}}(\\cdot|q\\oplus h*)$ makes more sense. Could the authors compare these two different approaches?\n\n2. The method relies on LLM-generated hints, whose quality might affect the performance of the method. Besides end-to-end performance, are there any other metric that measures the quality of the generated hints? Or in other word, is there a way to control the quality of generated hints?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLXKOIBN46", "reviewer": "Reviewer_KDuR", "summary": "This paper identifies the \"learning cliff\" in Reinforcement Learning from Verifier Rewards (RLVR) as a key obstacle to enhancing LLM reasoning . This \"cliff\" occurs when difficult problems yield persistent zero-reward signals, stalling the gradient updates of policy optimization algorithms like GRPO . The authors propose SCAF-GRPO, a framework that intervenes when such a stall is detected. It first diagnoses \"true-hard\" problems after a \"guidance exemption period\". Then, it injects minimal, hierarchical in-prompt hints (Knowledge, Planning, or Solution) to enable the model to generate a successful trajectory on-policy. This successful trajectory replaces a failed one, restoring a non-zero advantage signal for the GRPO update. Experiments on mathematics benchmarks show SCAF-GRPO outperforms vanilla GRPO and prefix-based baselines, and the method is shown to be effective across various model architectures.", "questions": "see * Weaknesses*.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In a learning cliff scenario, all rewards are zero, causing the advantage calculation to collapse and the learning gradient to vanish. Scaf-GRPO intervenes by generating a single successful trajectory on-policy using a minimally effective hint. This successful trajectory replaces a failed one in the batch, which \"restores a meaningful advantage signal\" and ensures \"non-zero reward variance\", allowing the standard GRPO update to resume.", "weaknesses": "- The framework's central claim to preserving the \"on-policy principle\" is questionable. When all trajectories for a query $q$ fail (the \"learning cliff\"), the method does not learn to solve $q$. Instead, it introduces a new input, $q \\oplus h^{*}$ (query + hint), and learns from this new, simpler task. The policy is indeed 'on-policy' with respect to the augmented prompt, but it has failed and bypassed the original, unhinted task. This is a semantic argument that obscures the fact that the model is being trained on a different, easier problem distribution than the one it originally failed on, which introduces a \"distributional mismatch\" of its own, despite the paper's critique of other methods on this very point.\n- The paper's core hypothesis is that its in-prompt scaffolding approach \"fosters more generalizable skills\" and \"cultivates robust reasoning skills rather than in-domain pattern matching\"  when compared to prefix-continuation baselines like LUFFY. The OOD evaluation on GPQA-Diamond (Table 4) fails to support this claim. Scaf-GRPO (37.3%) performs identically to the LUFFY baseline (37.3%). This lack of improvement in OOD generalization strongly suggests that the methodological benefits of Scaf-GRPO do not translate to improved generalizable reasoning, undermining the primary justification for its additional complexity over existing methods.\n-  No direct evidence found in the manuscript regarding the computational overhead of the proposed framework. Detailed analysis of the resource consumption, such as training time and memory usage, would be valuable for practical applications (Sec. 3.1). The paper does not discuss potential trade-offs between the improved reasoning performance and the computational cost. This information is crucial for understanding the practical feasibility of the approach (Sec. 3.2).\n- The 15% guidance exemption period appears empirically driven without theoretical grounding. Why 15% and not 10% or 25%? Three-tier, four-level hint structure: Why exactly 3 categories? Why must each have ≥4 items? This seems like an ad-hoc engineering decision lacking principled motivation 50% subsampling of \"Potentially Solvable\" problems lacks justification. The paper would benefit from ablations on these hyperparameters (e.g., 10% vs 15% vs 20% exemption periods).\n- Performance differences on some benchmarks are marginal (e.g., Table 1: Scaf-GRPO vs LUFFY on several benchmarks)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uz9EGGaB7a", "reviewer": "Reviewer_9rLe", "summary": "This paper introduces Scaffolded Group Relative Policy Optimization (Scaf-GRPO), a novel framework designed to enhance the reasoning capabilities of Large Language Models (LLMs) by overcoming the \"learning cliff\" problem in reinforcement learning. This problem arises when an LLM consistently fails to solve difficult problems, resulting in a zero-reward signal that provides no learning gradient. Scaf-GRPO addresses this by employing a two-phase training process. The first phase, a \"guidance exemption period,\" allows the model to learn autonomously on easier problems. In the second phase, for \"true-hard\" problems where learning has stagnated, the framework provides minimal, hierarchical in-prompt hints (Knowledge -> Planning -> Solution) until the model can generate a correct solution. By augmenting the training batch with this minimally-guided successful trajectory, Scaf-GRPO restores the learning gradient while preserving the on-policy nature of the learning algorithm and the model's exploratory autonomy. The authors demonstrate through extensive experiments on various math benchmarks that their method significantly outperforms vanilla GRPO and other prefix-based guidance methods.", "questions": "1. Regarding training efficiency and comparative fairness: During the training process, what is the approximate proportion of \"all-failure\" batches that trigger hierarchical hint-guided exploration? For these batches, Scaf-GRPO performs multiple on-policy rollouts with hints until a successful solution is found. In contrast, Vanilla GRPO conducts only one ineffective rollout when encountering the same situation. This suggests that Scaf-GRPO consumes more computational resources (time and computing power) when addressing challenging problems. Under the same training duration or total computational budget, can Scaf-GRPO still maintain a significant advantage?\n\n2. Comparison of effectiveness with DAPO.\n\n3. Can Scaf-Rollout be integrated as a component with current state-of-the-art model methods, such as DeepScaler and Skywork?\n\n4. This work lacks detailed comparisons with existing mathematical reasoning studies, such as representative works on long-chain and short-chain reasoning. These results on the test set are not particularly good.\n\n5. The pass@1 metric exhibits considerable fluctuations and requires repeated evaluations to obtain a reliable average.\n\n6. The \"guidance exemption period\" is empirically set to the initial 15% of training steps. How was this value determined? Is it a sensitive hyperparameter, or does the method perform robustly across a range of values? \n\n7. Figure 5 illustrates a compelling case of a model internalizing a skill. How frequently does this \"graduation\" from hint-dependency to autonomous competence occur in practice? Is it possible to track this metric during training to dynamically adjust when to offer hints for certain problem types?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly formalizes the \"learning cliff\" phenomenon (persistent zero-reward signals) as a critical bottleneck in training LLMs with reinforcement learning. The proposed solution, inspired by the pedagogical concept of scaffolding, is both elegant and intuitive. Instead of forcing the model down a fixed path with prefix-based solutions, Scaf-GRPO provides minimal, progressively more concrete hints, which preserves the model's ability to explore its own reasoning paths. This is a significant conceptual departure from prior work like LUFFY that relies on off-policy prefix-continuation.\n\n2. The authors conduct a meticulous set of ablation studies that convincingly validate their key design choices. Table 2 effectively demonstrates that each level of the hint hierarchy (Knowledge, Planning, Solution) is a necessary and complementary component of the framework. Furthermore, the analysis in Appendix G (Table 7) confirms the necessity of the \"guidance exemption period,\" showing that applying scaffolding from the start leads to inferior performance by fostering over-reliance on hints. These studies provide strong evidence that the framework's components are well-justified and contribute to its overall effectiveness.", "weaknesses": "1. The framework's success is contingent on the availability of a high-quality, three-tiered hint hierarchy. As the authors acknowledge, generating these hints requires a \"non-trivial data preparation effort,\" involving a powerful teacher model (DeepSeek-R1) and a highly structured prompting process. This dependency limits the scalability and practical applicability of Scaf-GRPO to domains where such structured hints can be easily and reliably generated. It moves the difficulty from the RL training phase to a complex and potentially expensive data curation phase.\n\n2. The hints are generated by a single, powerful teacher model. This introduces the risk that the scaffolding will simply teach the student model the specific reasoning patterns and biases of the teacher. While Scaf-GRPO aims to preserve exploratory autonomy, the guidance is still fundamentally rooted in the teacher's \"worldview.\" The paper does not explore the effects of hint quality or potential inaccuracies in the teacher-generated hints on the final performance of the student model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)