Nudging the Boundaries of LLM Reasoning
Authors: Justin Chen
XIANGYU PENG
Prafulla Kumar Choubey
Kung-Hsiang Huang
Jiaxin Zhang
Mohit Bansal
Chien-Sheng Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Nudging LLM
LLM Reasoning
GRPO
Primary area: foundation or frontier models, including LLMs

Abstract: Current online reinforcement learning (RL) algorithms like GRPO share a key limitation in LLM reasoning: they cannot learn from problems that are "unsolvable" to the model. In other words, they can only improve performance on problems where the model is capable of exploring the correct answer. If a problem is too difficult -- such that even hundreds of attempts never produce a correct solution -- the model cannot learn from it. Consequently, the model's "upper limit" remains unchanged after RL training, even though the likelihood of solving easier, solvable problems may increase. These hard, unsolvable samples -- though potentially rich in learning signal -- cannot contribute to training, as no rollouts yield rewards and thus no gradients are produced. To unlock learning from these hard samples, we propose NuRL, a "nudging" method that aims to push the upper bound of LLM reasoning using self-generated hints, i.e., abstract cues that help reduce the problem difficulty for the model. Given a question and its gold answer, the model generates a Chain-of-Thought (CoT) and then produces a hint containing the core knowledge needed to solve the problem. During online RL training, we generate G rollouts from the base policy and use the pass rate to decide whether the hint should be injected. For hard samples with a 0% pass rate, we inject the offline-generated hint and regenerate a new batch of trajectories. This yields two benefits: (1) the hint boosts pass rates (from 0% to non-zero), thereby introducing training signals for previously unsolvable samples, and (2) the hints are self-generated (conditioned on the gold answer), avoiding distributional shift and do not rely on external models. Compared to standard GRPO, NuRL achieves consistent improvements across six diverse benchmarks and three models, while remaining complementary to test-time scaling. Notably, NuRL can raise the model's upper limit, whereas GRPO leaves pass@1024 unchanged from the base model. Furthermore, we present a systematic study of what makes an effective hint and when hints are most useful. Interestingly, the best hints are abstract and high-level -- as revealing gold answers actually hurt performance -- and are most beneficial when applied necessarily and after GRPO has converged.

Published as a conference paper at ICLR 2026
NUDGING THE BOUNDARIES OF LLM REASONING
Justin Chih- Yao C
Becky Xiangyu Peng*
Prafulla Kumar Choubey
Kung-Hsiang Huang
Jiaxin Zhang!
Mohit Bansal? Chien-Sheng Wu!
Saalesforc Al Research
2 UNC Chapel Hill
ABSTRACT
Current online reinforcement learning (RL) algorithms like GRPO share key limitation in LLM reasoning: they cannot learn from problems that are "unsolv- able" to the model. In other words, they can only improve performance on prob- lems where the model is capable of exploring the correct answer. If problem is too difficult-such that even hundreds of attempts never produce a correct solution- the model cannot learn from it. Consequently, the model's "upper limit" remains unchanged after RI. training, even though the likelihood of solving easier, solv- able problems may increase. These hard, unsolvable samples-though though potentially rich in learning signal-cannot contribute to training, as no rollouts yield rewards and thus gradients produced. To unlock learning from these hard samples, we propose NuRL a "nudging" method that aims to push the upper bound LLM reasoning using self-generated hints, i.e., abstract cues that help reduce the problem difficulty for the model. Given a question and its gold answer, the model generates Chain-of-Thouu
(CoT)) and then produces hint containing the core knowledge needed to solve the problem. During online RL training, we generate G rollouts from the base policy and use the pass rate to decide whether the hint should be injected. For hard samples with 0% pass rate, we inject the offline- generated hint and regenerate a new batch of trajectories. This yields two benefits: (I) the hint boosts pass rates (from 0% to non-zero), thereby introducing training signals for previously unsolvable samples, and (2) the hints are self-generated (conditioned on the gold answer), avoiding distributional shift and do not rely on external models. Compared to standard GRPO, NuRL achieves consistent improvements across six diverse benchmarks and three models, while remaining complementary to test-time scaling. Notably, NuRI. can raise the model's upper limit, whereas GRPO leaves pass @ 1024 unchanged from the base model. Fur- thermore, we present systematic study of what makes an effective hint and when hints are most useful. Interestingly, the best hints are abstract and high-level-as revealing gold answers actually hurt performance and are most beneficial when applied necessarily and after GRPO has converged,
INTRODUCTION
Recent advances in reinforcement learning (RL) algorithms have played a central role in improving the reasoning abilities of large language models (LLMs). Despite many promising advances, current online RL algorithms share a key limitation: they cannot learn from problems that are unsolvable under the base policy. In other words, if the model cannot reach the correct answer even after exten- sive exploration, then no meaningful learning signal can be obtained from the problem. On a similar vein, a growing body of work finds that post-training mainly encourages models to generate already high-reward trajectories (He et al., 2025; Yue et al., 2025; Dang et al., 2025; Zhao et al., 2025). As result, the model's upper limit-often measured by pass @ l for large keremann unchanged after RL training. Intuitively, learning from harder samples offers a clear path to improving a model's performance and expanding its ceiling capacity. In other words, learning from the hard samples has two key benefits: (1) extracting more training signal from the same dataset (improving pass @ I),
PPoononnced like "neural" (nur-uhl) Code: https: http//gittttee com Sa lesforceA Research/

---

Published as a conference paper at ICLR 2026
Base Ability
Boundarles of Learning
Hard Froblem
Correctnesa Reward
?
Starting point
?
GRPO
Comfort Zone Can leamn sione
T5
Comtort Zone
Cannot kearn from the unsolvable problems Improves the model within the comfort zone
V JCRPO
Learning Zone
Can leann with help
Learning Zane
Anciety Zone
Ansiety Zone
-----oX NuRL (Ours)
of reach
Hint
Sources
Types of Hint
"nudge"
Ensble leerning from unsolvable problems Expand the comfort for more gains
V VCCPPO + 0
Self
Teacher
Ans.
Expl.
Partial Steps Abstract Cue
Figure 1: GRPO yields substantial gains, but the improvements largely stem from extending the model's 's ability within its comfort zone, i.e., if the model fails to solve a hard problem after numerous attempts, it is unable to learn from that problem. In NuRL, we address this by exploring various forms of hints (abstract cues, partial steps, explanations, or even the gold answer), which can be self- ceff-generated or
Both self- and eeacher-generated abstract cues can expand the model's 's comfort zone, effectively transforming previously unsolvable problems into solvable ones.
and (2) enabling the model to solve previously unsolvable problems, thereby extending its capabil- ity boundary (improving pass@k), However, precisely because these problems are difficult, models often cannot learn them without appropriate guidance or intervention from a teacher model.
This dynamic parallels Vygotsky's concept of the Zone of Proximal Development (ZPD) (Vygot- sky et al., 1978), which distinguishes between tasks a learner can solve independently ("comfort zone" in Fig. 1) and those achievable only with appropriate guidance ("learning zone" in Fig. 1). The inability to learn from hard samples or the lack of improvement in pass@ k mirrors be- ing trapped in the comfort zone. Motivated by this analogy, we ask: can models generate their own "hints"" lighhtweiigh forms of guidance-so that even the hard problems become learnable? Then, we propose NuRL (Nudging LLM with Reinforcement Learning), which adaptively injects self-generated hints into training. Our hypothesis is that hard problems become more learnable when paired with carefully abstracted hints (Huang et al., 2025; Park et al., 2025). These hints act as light-weight "nudges," transforming previously unlearnable samples into productive training sig- nals. To achieve this, our approach begins with offline hint collection. Given a question and its gold answer, we prompt the model to generate Chain-of-Thought (CoT; Wei et al., 2022) reasoning that connects the two. Using the question and CoT as input, the model then produces high-level cue that captures the core knowledge required to solve the problem. We also explore various types of hints as shown in the bottom-left of Fig. 1, where the hints can be self-generated or provided by a stronger model (e.g., GPT-o4-mini (OpenAI, 2025)), and the forms of hints can be abstract cues, partial steps, explanations, or even the gold answer. We find that self-generated hints are effective, while saccher-geeratte hints give further improvements. Importantly, effective hints are abstract and conceptual: they neither reveal the final answer nor provide detailed solution steps, but only mention what core knowledge is needed to solve this problem (see the bottom left in Fig. 2).
We adopt GRPO (Shao et al., 2024) as the training framework. During training, the policy model generates G rollouts per problem, and we use pass rate to decide when to inject hints. For hard problems with a 0% pass rate, we inject the pre-generated hints to the end of the question, and prompt the model to regenerate another G rollouts conditioned on the hints. Given the hint, the model is more likely to produce successful solutions (non zero pass rate), turning previously unlearnable examples into learnable ones. This strategy offers two main advantages: (1) Hints boost pass rate, enabling hard problems to produce meaningful training signals. (2) Hints remain within the model's s distribution, since they are self-generated (conditioned on the answer), avoiding distributional shift and does not require stronger external models. Together, these benefits allow NuRL to unlock value from harder samples, broadening the set of problems that contribute to RL training.
We evaluate NuRL on six diverse benchmarks across multiple domains, including MATH 500, MATH Hard, AIME, GPQA, MMLU-Pro and Date Understanding. Results show that NuRL boosts the average performance with three different models (+1.62% using Llama, +1.75% using Octo- Thinker, and +0.79% using Qwen as compared to GRPO), and when an external teacher model is available for hint generation, the improvement can be further enlarged to up to 3.44%. More-
2

---

Published as a conference paper at ICLR 2026
over, analysis shows that (a) NuRL is complementary to test-time scaling methods such as Coliisonsittency (Wang et al., 2022) and shows larger improvement: our method improves 9.4% with 16-way Self-Consistency, transform previously unsolvable problems into solvable ones, and that transfer to the improvements on the upper limit of model's s capacity. (c) Hints are useful when they abstract and high-level. The more exposure to the answer, the more severe the degradation. This aligns with human learning, where effective hints should be abstract and provvvvve-- guidance without revealing the solution. Knowing the answer upfront risks biasing toward it and undermines generalization.
compared to GRPO, which improves 7.8%. (b) NuRL is able to
RELATED WORK
Reinforcement Learning with Verifiable Reward (RLVR). RLVR computes the reward using rule-based verification, which is effective in improving LLM reasoning. The reward function can be as simple as checking whether the model's answer matches the gold answer (Lambert et al., 2024; Guo et al., 2025; Team et al., 2025; Zeng et al., 2025). The success of RLVR is also supported by advancements, including PPO (Schulman et al., 2017), DPO (Rafailov et al., 2023), GRPO (Guo et al., 2025) and many techniques like DAPO (Yu et al., 2025), Dr. GRPO (Liu et al., 2025c). Recent work explores different strategies learn better self-reflection ability, such as "try again" (Liu et al., 2025a) or self self-critique (Zhang et al., 2025c).
The Role of RL: Distribution Sharpening vs. Discovery. There has been an active discussion on whether RL primarily performs distribution sharpening, i.c., amplifying behaviors already present in the model, or enables genuine discovery of new reasoning abilities. The distribution- -aaappioon-saannnnnngg view holds that RL mainly surfaces high-reward paths and increases their likelihood of generation (Zhang et al., 2025a; Zhao et al., 2025; Shenfeld et al., 2025). This is often supported by the findings that RL improves pass@1 but not pass @ k (Yue et al., 2025; He et al., 2025), and that even weak reward signals can yield substantial improvements (Shao et al., 2025; Zuo et al., 2025; Prabhudesai et al., 2025; Wang al., 2025b). On the other hand, some argue that RL fails to improve pass @ not because of inherent limitations, but due to insufficient training or evaluating on tasks where models already perform strongly (Liu et al., 2025b). Yuan et al. (2025) also show that RL can learn new skills by combining existing ones. Zhang et al. (2025a) suggest that RL can both sharpen and discover, with the balance determined by the trade-off between exploration and exploitation.
Mixture of Offline and Online RL. When the base policy fails to generate solutions that yield non-zero rewards, replay buffer off-policy optimization leverage previous positive trajectories (Lu et al., 2025) or expert demonstrations (Levine et al., 2020). Recently, hybrid approaches that mix online and offline training have been proposed to improve generalization (Yan al., 2025; Phan et al., 2025). TBA (Bartoldson et al., 2025) decouples data generation from policy updates by leveraging multiple searcher nodes to produce diverse trajectories, enabling high resource utilization and a more scalable search process. A line of work uses SFT with RL to expand the model's knowledge scope (Ma et al., 2025; Fu et al., 2025; Zhou et al., 2025), while others employ hints (e.g., partial solutions) to adjust problem difficulty during training (Zhang et al., 2025b; Huang et al., 2025; Li et al., 2025; Qu et al., 2025; Wang et al., 2025a). Our work differs from these approaches in two aspects. First, NuRL does not rely on SFT to broaden the model's knowledge; instead, we focus on the RL stage and analyze how different hints expand a model's reasoning boundary. A notable work is STaR (Zelikman et al., 2022), where the reasoning is bootstrapped from reasoning given the answer. Our hints are further abstracted out from the reasoning to ensure they do not disclose the answer. Second, we show that nudging the model with self-generated hints is effective: it enables self-improvement (both pass@) and pass@k) without depending on stronger external models.
METHODOLOGY
3.1 PRELIMINARY
Our method is based on GRPO (Shao et al., 2024). Specifically, GRPO updates the policy by maximizing (O) using the following objective:
3

---

Published as a conference paper at ICLR 2026
Hint Incorrect Oulput [Т:.] Correct Output
Rollout
Rollout | Hint
Correctness Reward
Standard GRPO: when all rollouts are incorrect
V TeRPO =
rewards
Queation:
(no training signal)
The answer 18 (0)
Mhat process uses electricity
rewards
separate hydrogen
TTe answer is (A)
(with training signal)
oxygen from water? riltration
combust fon
.Thee answer
answer
electrolysis
D: distillation
The answer (B)
The ansuor 13
(t)electralysia
Base Policy
The answer 19 (C)
The anouer 19 (C)
Bint:
Electricity
Only triggered when all rollouts fail
the main tool-like uring battery-to split somataing into simpler pieces.
The answer is (C)
The is
NuRL: Nudging the LLM with hints to enable training signal
Figure 2: NuRL provides targeted guidance to the LLM policy during online GRPO training. Prior to training, we construct an offline collection of hints, defined as abstract problem-specific cues that reduce task difficulty. During the online training, whenever all G rollouts for a problem are incorrect, NuRL augments G - 1 of the rollouts with the corresponding hint and regenerates the batch. This intervention facilitates the acquisition non-zon non-zero rewards on instances that would otherwise yield uniformly zero rewards, thereby supplying informative training signals.
9 (o) = G
1 02
lo:1
min
Ar,, clip (00,
polon,los,et
mo(orslon,cs) Tota
E, e,1 +
And
1=1
where Ty is the policy, Vold is the old policy, is the clipping range, and Å, is estimator of the advantage at time step t. Given a reward function f and a question-answer pair (g, a) from training data D. the advantage is estimated by letting Thw samples group of 9 responses {o:] Then, the advantage of the i-th response is calculated by normalizing the rewards within the group {r:)a
Pr Or
1
1 g
Hr
Ti or
(rs m4)
We use the rule-based outcome correctness as the reward (Guo et al., 2025), i.c. f(6.)) 1, if is eseeeiivllnnt y) else 0, where is the ground truth and ÿ is the predicted answer.
3.2 NURL: NUDGING LLMS WITH REINFORCEMENT LEARNING
Offline Hint Collection. Given a training dataset consisting of question-answer pairs D =
(q,, as); our method begins with offline hint collection. As illustrated in Fig. 1, hints can be categorized by their source (self-generate by the model vs. provided by teacher model) and their type: (1) Abstract cues: generated by abstracting from an explanation of why an answer is correct, designed to avoid revealing concrete details or the final answer. (2) Partial steps: obtained by gen- erating a step-by-step solution using the gold answer, masking 75% of the steps (keep only the first 25%), with the model asked to complete the rest. (3) Explanations: formed by replacing incorrect rollouts with gold-conditioned explanations that justify why the answer is correct. (4) Ground- truth answer; appending the correct answer alongside the question with the prompt, "I was told the answer is
Do not blindly accept it. Take it as reference and provide your own step-by hap--y--st by- reasoning." Note that from (1) to (4) is a reverse order of how much information about the answer is being revealed. Later in Section 5.2, we will show that the more disclosure about the answer, the worse the performance is. Therefore, in this work, we mainly focus on self-generated abstract cues the hint (see Appendix E. for examples with different types of hints).
4

---

Published as a conference paper at ICLR 2026
Concretely, to collect such high-level abstract cues that can simplify the problem without revealing specific solution details, we first prompt the base policy LLM with both the question and the ground- truth answer, and instruct it to generate a Chain-of- Thain-of-Thought (CoT; Wei et al., 2022) that explains why the provided answer is correct. This can be expressed as = Tala (q: q: Py) where Vold is the initial policy, y is the generated Co'T, and Py is the prompt (see Appendix D). We find that directly using such explanation-style CoTs as hints is not effective (also will be shown in Section 5.2). To address this, we introduce an abstraction step. Specifically, we prompt the LLM again with the question, the answer, and the self- leff-eeeerated CoT (g. a. y), asking it to produce a high-level hint that reduces task difficulty without being overly specific:
(g.q. a. y;: Ph) where pa is the hint generation prompt parliiprooo
as provided in Appendix D. After this process, the training data is augmented with hints, yielding an enriched dataset D = {w: 02 h. where every sample has a corresponding hint.
Online Rollout Augmentation. After augmenting the training data with hints, we proceed with online GRPO-style training. For each question q, the base policy first generates 9 rollouts without hints. When all responses {or);= are incorrect (i.e., r; = 0 + A., = 0 for all i), the advantages vanish, yielding VGRPO 0 and thus no policy update. This is illustrated in the gray box of Fig. 2, where the correct answer is C but none of the rollouts reach it. Similarly, if all rollouts are correct, the task is trivially easy and again results in VJGRPO = In practice, such uninformative problems (both too hard and too easy) are often discarded to improve training efficiency (Yu et al., 2025). In contrast, NuRL retains the hard cases, as they present opportunities to unlock further gains. Specifically, when all G rollouts fail, we activate NuRL by appending the offline-gent offline-generated hint h to the problem, producing the new input q @ h, where @ denotes concatenation. A new batch of rollouts is then generated from @ h (blue box in Fig. 2). To reduce the chance to collapse into uniformly correct responses- which would again eliminate the learning signal-we let only G G-1 rollouts see the hint, {o:] To (g. ht), while one rollout remains hint-free, og ~ To (1).
Inference. The hints are only used in training. During test time, we prompt the model only with the question. The hypothesis is that exposure to hints during training guides the model toward correct solutions, enabling it to internalize the reasoning patterns required to solve the problems. That is, the use of hints broadens the set of solvable problems and translates to improved performance.
EXPERIMENTAL SETUP
Models. We evaluate NuRL three models: Llama3.2-33 3-B-B Instructtt (Grattafiori et al., 2024), (Wang et al., 2025c), and Qwen3-4BBInstruutt
(Team, 2025).² Llama is general-purpose instruction-tuned model, while OctoThinker is recently proposed model that applies mid-training on Ilama and has been shown to exhibit stronger compatibility with reinforcement learning (Wang et al., 2025c). Qwen is also general- purrose instruction- -unde model that encompasses strong compatibility with post-training techniques. We evaluate these mod- els because they exhibit distinct properties. helping us avoid conclusions that are overly specific to a single model a lesson underscored in recent studies (Shao et al., 2025; Chandak et al., 2025).
Datasets. For training. we use Open-Rl's 's Thhought dataset (Face, 2025) due its di- versity, which includes science QA data from Llama-Nemotron (Bercovich et al., 2025), ensuring our training set is not purely math-focused. We randomly sample 7.5k math and 2.5k science data points. Since the dataset provides only CoT outputs from Deepseek-RI (Guo et al., 2025) rather than explicit gold answers, we extract the answers from (boxed{} and generate CoTs per problem us- ing GPT-o4-mini (OpenAI, 2025). We keep only samples where Deepseek-RI I and GPT-o4-minit majority agree on the answer. This yields 8.3k samples for training. NuRL is tested across a di- verse set of reasoning- intensive tasks spanning STEM and other domain-specifie areas, including: (1) MATH 500, a subset 500 problems from the MATH benchmark (Hendrycks et al., 2021) cu- rated by Lightman et al. (2023); (2) MATH Hard, the hardest problem from MATH, totaling 1.3k problems (Hendrycks et al., 2021); (3) AIME 2024. 30 mathematics problems from the 2024
AIME competition (AIME, 2024); (4) GPQA Diamond, 198 PhD-level questions covering biology, physics, and chemistry (Rein et al., 2024); (5) MMLU--Pr, a more challenging variant of MMLU (Hendrycks et al., 2020), spanning 14 college-level subjects with 12k samples (Wang et al., 2024); (6) Date Understanding, 250 problems designed to test L.M''s understanding to date information, requiring commonsense and logical reasoning (bench authors, 2023; Suzgun et al., 2022).
Hereafter, we refer them as Ilama, OctoThinker, and Qwen, respectively.
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of methods across three models and six benchmarks. NuRL consistently out- performs all baselines with self-generated hints, and shows further improvements when an external model is available for hint generation (shown in gray for reference).
MATH 500
MATH Hard
AIME
GPQA
MMLU-Pro
Date
Avg.
Llama32-3B-Instee
Zero-shot
35.71
15.28
3.33
13.23
11.52
2.33
13.57
Few-shot
36,68
16.73
3.63
15.52
12.02
8,87
15.58
RFT
40.12
17.72
3.63
14.76
12.98
13.36
17.10
ReFT
55.80
28.86
8.00
24.42
30.19
55.34
33.77
GRPO
56.92
30.11
8.33
27.98
34.78
57.10
35.87
w/ Hint (Self)
58.04
31.62
9.17
28.28
36.18
61.65
37.49
w/ Hint (GPT-o4-mini)
59.30
32.42
10.83
28.40
41.38
63.53
39.31
DctoThiinker-BB-Hybrid
Zero-shot
59.98
36.87
4.83
16.57
21.18
19.25
26.45
Few-shot
61.23
37.01
5.33
18.62
23.32
24.53
28.34
RFT
62.78
37.73
4.83
19.45
27.34
36.82
31.49
ReFT
66,38
39.69
6.66
24.53
44.66
70.32
42.04
GRPO
68.81
41.29
8.33
23.26
44.25
69.85
42.63
w/ Hint (Self)
42.07
27.15
45.54
71.75
44.38
w/ Hint (GPT-o4-mini)
71.62
43.51
12.63
27.43
46.53
72.28
45.67
Qwen3-4B-Instruut-255
Zero-shot
94.88
90.54
58.75
35.57
58.88
83.35
70.33
Few-shot
94.97
90.54
55.52
34.82
59.01
85.01
69.98
RFT
94.41
90.41
56.63
37.72
59.54
85.42
70.69
ReFT
96,46
90.83
62.79
60.31
72.21
92.20
79.13
GRPO
96.52
90.54
60.83
62.50
72.65
92.80
79.31
w/ Hint (Self)
96.46
92.57
63.54
62.88
72.83
92.30
80.10
w/ Hint (GPT-o4-min)
96.58
92.96
62.71
64.99
72.95
93.91
80.68
Baselines. We compare NuRL with the following baselines: (1) Zero-shot: We prompt the model to think step-by-sted and provide the answer within \boxed{] (Kojima et al., 2022). (2) Few-shot: include in-context learning samples. (3)
Besides prompting the model to think step-by-step, Rejection sampling Fine-Tuning Tuning (RFT): We prompt the model with training data 8 times using the zero-shot prompt same as above, and keep only the correct reasoning chains to perform supervise fine-tuning (Yuan et al., 2023). (4) Reasoning with Reinforced Fine-Tuning (ReFT): We adopt the SFTed model from RFT, and continue for GRPO training (Trung et al., 2024). (5) GRPO: Using outcome correctness rule-based reward function (Guo et al., 2025).
Implementation Details. We employ abstract cues as hints, and compare self-generated versus macchrr-geneaater in Table 1. We evaluate all methods using pass @ 1. Results averaged over 16 runs, except for MMLU-Pro, which has 12k samples; for this dataset, we report the average over runs. We adopt verl (Sheng et al., 2024) as the backbone, and utilize vllm (K (Kwon al., 2023) to speed up rollout generation and inference. To verify equivalence between predictions and references. Math-Verify* Math- Verify*. We employ two-stage strategy for training. In stage policy with correctness-only GRPO until both training reward and validation accuracy show no improvement for over 10 steps. In stage we apply NuRL continue training. To ensure fairness, GRPO-based baselines are trained for the same total number of steps NuRL,, albeit NuRL starts midway. Before stage 2 begins, we use the stage checkpoint to generate 8 rollouts and filter out samples where all rollouts are correct (i.e., overly easy cases) to improve efficiency. The resulting sample size is reported Appendix B. All methods are running on 8 H200 GPUs, and takes around six days for GRPO- based methods to converge. During training,
optimize the base
temperature of 1.0
hhttps://ggt thub. htt.uu..oo/uugginngggeee

---

Published as a conference paper at ICLR 2026
and set the clip-high parameter to 0.28 (Yu et al., 2025). The rollout number is 16 for GRPO and for NuRL, as NuRL may regenerate an additional batch of rollouts when all rollouts fail. We cap the output length 9k tokens for both training and testing. At inference, 0.7. Token limits and inference temperature are aligned across all baselines for evaluation. Other hyperparameters and details can be found in Appendix B and in code.
fix the temperature to
RESULTS AND ANALYSIS
5.1
MAIN RESULTS
NuRL consistently outperforms all baselines with self-generated hints. We present the main results in Table Across six benchmarks, NuRL shows superior performance compared all baselines, beating the zero-shot and the SFT baseline (RFT) by large margin. For RL-based base- lines (ReFT and GRPO), we see the performance increase by large margin (+8.8% to +22.3% compared to the base model), conforming that RL largely improves a model's performance. Never- theless, NuRL consistently surpasses these strong RL-based baselines. On average, improves over GRPO by +1.2 points on Ilama, +1. 1.75% on TctoThinker, and +0.79% on Qwen. Importantly, these improvements are achieved on top of already strong GRPO performance, despite GRPO using 16 rollouts per question, whereas NuRL uses only 8 (with the option to generate another eight only all initial rollouts fail). Thus, GRPO operates with strictly more rollouts than NuRL, Since the primary difference between NuRL and GRPO is the use of hints, these results highlight that tar- geted hints are an effective mechanism for improving LLM performance. We note that the relatively smaller improvement examples), as Qwen's stronger base capability caused many overly easy samples to be filtered out before stage 2. Finally, observe that incorporating stronger external models for hint generation yields additional gains beyond self-generater hints. For example, on Llama, hints from GPT-o4- mini improve performance by -3.44% absolute points on average over GRPO and by +1.82% over self-hints. This demonstrates that while self aef--gneratee hints are already beneficial, the framework naturally accommodates stronger sources of guidance when available.
Qwen (+0.79%% may stem from the limited stage 2 data (fewer than 2k
5.2
ADDITIONAL ANALYSIS
+1.1%
+1.2%
+0.8%
100
96.9
96.397.6
96.597.6
96.697.4
Accuracy (%)
+7.8%
+9.4%
+8.8%
80
+6.2%
+7.6%
64.465.8
73.6
8.0%%
66 .6..4
76,8
67.669.2
55.6
60.0
58.0
40
35.7
Zero-shot
GRPO
NuRL (Self)
NuRL (o4-mini)
LIama-3B Avg@16
Octo-3B Avg@16
Qwen-4B Avg@16
LIama-3B 5C=16
Octo-3B 5C=16
Qwen-4B SC=16
Figure 3: Compared to GRPO's improvements with Self- Self-Consistency (+7.6%, +7.8% and +1.1% on Llama, OctoThinker and Qwen), NuRL obtains larger gains with +8.0% +9.4% and 1.2%.
NuRL is complementary test-time scaling method. While NuRL shows superior performance in Table 1, common way to augment the baselines is to scale the test-time compute. often measured the number of tokens number of samples inference time. Here we adopt the latter and employ Self-Consistency (SC; Wang et al., 2022). We compare NuRL with zero-shot and GRPO, with and without Self-Consistency across the three models in MATH 500. Results Fig. 3 show that NuRL not only remains effective for test-time scaling method, but also shows greater improvements compared to GRPO. Specifically, Llama trained with GRPO improves 7.6% with SC, while Llama trained with NuRL improves 8.0%. Similarly, OctoThinker trained with GRPO improves 7.8% with SC, while Thinkr trained with NuRL improves 9.4%. Finally, Qwen trained with GRPO improves 1. 1.1% with SC, while Qwen trained with NuRL. improves 1.2%.
7

---

Published as a conference paper at ICLR 2026
100
97.1 96.6 94.4
Accuracy (%)
82.3
80
67.4 67.6
63.4
57.6
54.3
40
35.9
27.6
Llama3.2-30
OctoThinker-3B
Qwen3-4B Ground Truth
Abstract
Partial Steps
Explanation
Figure 4: Comparison of different types of hints. From left to right, the hints vary in how directly they disclose information about the ground-truth answer. At the leftmost end, abstract hints provide only high-level guidance without revealing details of the solution or answer, whereas at the rightmost end, the answer is given explicitly. Interestingly, more direct hints lead
performance.
Hint abstraction is key to improvement. In Section 3, described how hints are generated by abstracting explanations of why an answer correct. While this represents one useful approach, raises broader question: what makes a good hin!? To investigate this, compare four types of hints as mentioned in Section 3. In Fig. 4, we observe consistent trend: the more directly answer in- formation is disclosed, the lower the downstream performance. Abstract cues, which explicitly avoid revealing details or solutions, yield the highest accuracy. Partial steps perform slightly worse, but still help since the initial reasoning structure provides useful starting point. Explanations are less effective, justifying why answer is correct is not equivalent to reasoning through the problem; fact, explanations often implicitly disclose the answer (e.g., "but the answer is {gold_am Jnnwer), so I should try different method")). Finally, directly providing the ground-truth answer severely harms generalization. This setup frequently induces reward hacking: during training, the model learns to simply output the provided answer maximize reward without genuine reasoning, and that does not generalize at test time. Overall, these results suggest that effective hints should remain high-level and abstract. A good hint guides the model toward reasoning independently, but avoids revealing shortcuts that undermine generalization.
Using hints only when necessary is crucial. While we have established that a good hint should be abstract, the question of when is the best time to use hints remains open. To examine this, we consider two key factors. First, one can de- cide whether to apply hints from the beginning of training, or to wait until GRPO converges be- fore introducing them. We refer to the latter ap- proach as wwo-stage training. Second, one can de- termine the condition under which hints pro- vided. Specifically, hints may be applied uni- formly all problems, only when all rollouts incorrect. We refer to this latter condition as a difficulty trigger. Combining these two factors yields four experimental settings, summarized in Table 2. We find that applying hints from the beginning of training generally underperforms compared to introducing hints only after GRPO has stabilized. Similarly, applying hints indiscrim- inately results in lower performance than using difficulty trigger. This suggests that unnecessary hints may interfere with the model's ability to learn independently problems where guidance is required. In contrast, when hints are introduced only after GRPO convergence and combined with difficulty trigger, we observe consistent improvements across both MATH and GPQA (58.04 and 28.22 28.28, respectively). In summary, these findings indicate that hints are most effective when used selectively and adaptively. Rather than being injected throughout training applied uniformly, hints should be reserved for difficult cases and integrated after the base policy has stabilized.
Table 2: Comparison of hint application strate- gies during training. Results are on MATH 500 and GPQA with
Two-stage
Diff. Trigger
MATH
GPQA
x
/
56.06
27.63
x /
X 53.41
24.84
X 53.09
26.62
/
58.04
28.28
NuRL improves @ when the task is more challenging to the model. In Table 1, we show
that NuRL generally outperforms GRPO terms of pass@1. 1. A natural follow-up question is whether these gains also extend to higher values of k especially when using NuRL, the model is supposed to solve more training problems compared to GPRO. To investigate this, we plot pass @ for k = (1,,, . 512, 1024) ) in Fig. 5 using Llama. Following Yuan al. (2025). set the maxi-
8

---

Published as a conference paper at ICLR 2026
MATH 500
Date
GPQA
100
100
90
80
75
80
70
60
50
60
40
50
25
40
20
29
2
2
2
2
2
2
2'
29
2
29
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
27
2
2
Pass@k
Pass@k
Pass@k
Liama
GRPO
NuRL (Self)
NuRL (o4-mini)
Figure 5: When the base model (Llama) already has strong pre-trained knowledge (e.g., MATH 500), both GRPO and NuRL yield little improvement in pass@k. In contrast, on tasks with lower upper-bound performance (e.g.. Date Understanding and GPQA, with pass 1024 of 85.4 and 67.2), GRPO provides no gains on pass @ 1024, while NuRL pushes it further.
mum k as 1024 as a sufficiently large and practical budget to probe the model's ceiling performance. In Fig. 5, we first report results on MATH 500. Here, the base model already achieves pass 1024 49. before training (despite its relatively low pass@) = 34..%). In this case, neither GRPO nor NuRL increases pass@ 1024, the model already possesses strong knowledge on this task, albeit hard to generate a correct answer given only one attempt. In contrast, for tasks where the base model's upper bound is lower - such as Date Understanding (85.4%) and GPQA (67.2%)- we observe clear gains from NuRI., while GRPO provides little or no improvement. We hypothesize that hints guide the model to explore underrepresented solution paths, increasing the chance of dis- covering correct answers that would otherwise remain in "unreachable corners" of the search space. From an entropy perspective, hints may also induce more diverse exploration, which translates into higher pass @ k (Cheng et al., 2025). Importantly, these hints lightweighr: rather than fine-tuning the model with hints, we simply append them to the input question, yet this is sufficient to guide exploration more effectively. Moreover, we find that pass @ 1024 scales with the quality of hints: using teacher-generated hints pushes it further to 95.2% on GPQA, compared to GRPO's 63.4%.. In summary, when the base model's ceiling performance is not yet saturated, NuRL raises pass @ both with self-generated hints (86.4% + 94.0% on Date, 63.6% + 69.7% on GPQA) and even more so with higher-quality hints generated from the teacher model.
NuRL increases the fraction of solvable problems. Having demonstrated that NuRL able to improve both pass@) @1 1 and @ass@k, we further analyze the source of such performance gains. In Fig. 6, we show
wo Hint
Before Hint
After Hint
Salvable Problems (%)
76
74
that training with self-generated hints effectively in-
72
creases the fraction of solvable problems in the train-
70
ing data. Specifically, and compare, at each training step, the fraction of solv- able problems under three conditions: (1) no hints, corresponding to standard GRPO training, (2) using NuRL before adding hints, and (3) using NuRL af- ter injecting hints whenever all rollouts fail. Here, problem is considered solvable if least one rollout produces a correct answer. Given the same training step, adding hints yields an approximate 4% increase in solvable problems, suggesting that self- -eferaeed hints reduce problem difficulty for the model. Second, NuRL shows a clear upward trend in the fraction of solvable problems both before and after hint injection. This indicates that as training progresses, the model begins to solve problems that were previously unsolvable, increasing the solvable fraction from 68% to 70%. Lastly, standard GRPO without hints exhibits relatively flat trend, with the solvable fraction fluctuating around 66%. Overall, these results explain the improvements observed across diverse benchmarks: by leveraging self-genertted hints, the model effectively increases the amount of data it can learn from.
present Qwen's training log
68
66
64
200
400
600
Steps
Figure 6: The self seff-generated hints in NuRL effectively reduce the task difficulty and in- crease the portion of solvable problems.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We introduce NuRL, a self-guided reinforcement learning approach that uses hints to extend models' reasoning capabilities. NuRL consistently outperforms strong baselines with self-generated hints and achieves further gains with hints generated by stronger external model. Our analysis reveals that the most effective hints are high-level, abstract, and applied selectively only when the model cannot solve problem unaided. Moreover, NuRL scales more efficiently than GRPO at test time, increases the fraction of solvable problems, and delivers substantial pass @ improvements on two challenging tasks, effectively expanding the model's ' comfort zone.
ETHICS STATEMENT
In this work, propose a reinforcement learning method that optimizes for final outcome correct- ness. As result, our trained LLMs may still produce hallucinations, since intermediate reasoning is not directly supervised and the correctness is only verified against the final answer. Thus, out- puts generated by NuRL carry potential risks of misinformation or hallucination. Future research is needed better assess and mitigate these limitations.
REPRODUCIBILITY STATEMENT
We are making our code available in the supplementary materials enable replication of our find- ings. We also provide implementation details of NuRL, in Appendix B and prompts in Appendix D. The datasets we use are all publicly available, as detailed in Appendix C.
REFERENCES
AIME. Aime. aime problems and solutions, 2024., 2024. URL https emsoollllemmlol ing. Ang.ccm/wwe www.ccom/iiik.. wndex.idxx.///// phpp American wwwriinnnnnnntaaaeeeeeiieee Bataaaaaaaaataattaaaaaa pam_attiiis__Eaaaaaaaaaaaaaaaaat tion.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms on uncontaminated math competitions. Proceedings of the Neural Information Processing Systems Track Datasets and Benchmark, 2025.
Brian Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Cerv Yoshua Bengio, and Bhavya Kailkhura. Trajectory bal- ance with asynchrony: Decoupling exploration and learning for fast, scalable I1m post-training. arXiv preprint arXiv:2503. S5..:::500..8999, 2025.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL attps::/cpeerevee
net. forum?idddniii BoruunidduyyLLBvves
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zil- berstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Ger- ald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Alek- sander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin. Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha. Bran- don Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor BoinYiiig Su, Guyue Huang. Teryy Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath,
10

---

Published as a conference paper at ICLR 2026
Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Olek- sandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edel- man, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yut- ing Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Frick Galinkin, Michael Evans, Katherine Luna, Leon Derezynski, Nikki Pope, Hileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geif- man, Eric Chung, and Chris Alexiuk. Llama-nemotron Efficient reasoning models, 2025. URL 00949.
petts://arxiv...
Nikhil Chandak, Shashwat Goel, and Ameya Prabhu, Incorrect baseline evaluations call into ques- tion recent llm-rl claims.
9a8 not ion wwwtiio..stteeeeeeeeeooeoeeeeeee wwwt//tnnorrreeeeeeeeeeeeeeeeeee Bas Recee--eeeet--eee LLM-LLL-MMLLL--L--- -aa.---aaas--0001 2025. Notion Blog.
e www.:aaaaaaaiinaaaaaaaeaaaaaaaae 1
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Xingyu Dang, Christina Baek, Zico Kolter, and Aditi Raghunathan. Assessing diversity collapse in reasoning. In Scaling Self- Improving Foundation Models without Human Supervision, 2025.
URL /tttps:///ppnreee/ ew www.e/weeeffrtttee///ee/ ? i d= KsHL
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: wtt.uu..../.......ia/e.aa
open-rl. zl.
Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506. 19767, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:2506.02355 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song. Jingyuan Chen, Chang Yao, and Jie Song. Boosting milm reasoning with text-debiased hint-grpo. arXiv preprint arXiv:2503.23905. 2025.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 15:22199-22213, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
11

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training, arXiv preprint arXiv:2411.15124 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang. Questa: Expanding reasoning capacity in llms via question augmentation. arXiv preprint arXiv:2507. 13266, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. A simple" try again" can elicit multi-turn Ilm reasoning. arXiv preprint arXiv:2507. 142995, 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24880 2025b.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint WrXiv:2503.20788 2025c.
Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiay Jia. Arpo: End-to-end policy opti- mization for gui agents with experience replay. arXiv preprint arXiv:2505. 16282, 2025.
Lu Ma, Hao Liang Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: In- terleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 2025.
OpenAl. Introducing openai o3 and 04-mini, 2025. URL --------------------------------
watealaaiccomiieneeeeeeeee com/indexinnt
Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim. Deepvideo-rl: Video reinforcement fine-tuning via difficulty-aware regressive grpo. arXiv preprint arXiv:2506.07464 2025.
Peter Phan, Dhruv Agarwal, Kavitha Srinivas, Horst Samulowitz, Pavan Kapanipathi, and An- drew McCallum. Migrate: Mixed-policy grpo for adaptation at test-time. arXiv preprint arXiv:2508. 08641, 2025.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660 2025.
Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Ruslan Salakhutdinov, Chelsea Finn, and Aviral Kumar. Rlad: Training Ilms to discover abstractions for solving reasoning problems. arXiv preprint arXiv:25 10.00263, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 001553288-77744 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 10.11170774444 2017.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert. Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint 2rXiv:2506.10947 2025.
12

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. Rl's razor: Why online reinforcement learning forgets less. arXiv preprint arXi:22509004255 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. / 19256, 2024.
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models, 2025. https: 80.
/taobb//5033. 2203 2503.213 213
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether maiin-of-thought can solve them. arXiv preprint arXiv:22 10.09222 2022.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et Kimi k1. 5: Sealing reinforcement learning with Ilms. arXiv preprint 1rXi::2501.12599, 2025.
Qwen Team. Qwen3 technical report, 2025. URL https::///// http://aawwv.oo/a/////////////// 09388.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 7601-7614, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 024. ac1-long 410/.
URL https: / aaaaclamm wwwlanhhllogyyy...oooooooooooloa org/2
Lev S Vygotsky, Michael Cole, Vera John-Steiner, S Scribner, and Ellen Souberman. The develop- ment of higher psychological processes, 1978.
Jiayu Wang, Yifci Ming, Zixuan Ke, Caiming Xiong, Shatiq Joty, Aws Albarghouthi, and Frederic Sala. Beyond accuracy: Dissecting mathematical reasoning for Ilms under reinforcement learning. arXiv preprint arXiv:2506.04777 2025a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprini arXiv:2203. 2022.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprini arXiv:2504.20571 2025b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran AruIraj, Xuan He, Ziyan Jiang, et al. Mmlu mrouu A more robust and challenging multi- task language understanding benchmark. Advances Neural Information Processing Systems, 77955266-95290, 2024.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512. 2025c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho in Neural Information Processing Systems, 2022. URL https: /ppp:://pppeeeeee http://ppppeeee..cee/ee/ new.ew...eeeeeeee...e....... feoo um?ide wtpdddd__vllloa _e
13

---

Published as a conference paper at ICLR 2026
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu Yu Cheng, and Yue Zhang. Learning to reason under off- off-policy guidance. arXiv preprint arXiv:2504. Se5d::2504.14445, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint (it.::2003..477000 2025.
Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From f(x) and g(z) f(g(z)): LLMs learn new skills in RL by composing old ones. https: / mouskyymorocco not.notion site/E rom- f---f------ and g x to f g - Learn Skills ---------------- 2025. Notion blog post,
sing-Old-Oma Ones- 20--224999 4 available online.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou Scaling relationship on learning mathematical reasoning with large language models, 2023. URL / arxiv. org/ abs 2308 018001220 01825.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar arXiv:2504. 13837, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:11576--1488 2022.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar 2r0vv:2503. 18892, 2025.
Kaiyan Zhang, Yuxin Zuo Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li. Huayu Chen. Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao. Dong Li, Zhiyuan Ma, Ganqu Cui, Zhiyuan Liu, Biqing Qi, Ning Ding, and Bowen Zhou. A survey of reinforcement learning for large reasoning models, 2025a. URL https: //arxiv. erg/ abs///a////// /509. 08827.
Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint MxXiv:2507.02841, 2025b.
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng- Critique-grpo Advancing Ilm reasoning with natural language and numerical feedback. arXiv preprint -XXv:25566001066666 2025c.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post- bostniraining amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912. 2025.
Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi. Expo: Unlocking hard reasoning with seff-explanationnguiieee reinforcement learning. arXiv preprint arXiv:2507.02834 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.1608 16084, 2025.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
We use ChatGPT for grammar correction and refinement. The model was only used to polish text already written by the authors, and was not used for research ideation or generating original content.
hhtps///hattgttccom
14

---

Published as a conference paper at ICLR 2026
B IMPLEMENTATION DETAILS
Recall that in Section 3, we describe a two-stage training procedure where NuRL is applied after GRPO converges. By convergence, we mean that both the training reward and validation accuracy plateau after 10 training steps. Stage corresponds to GRPO convergence, and stage 2 applies NuRL on top of the converged checkpoint. For fairness, GRPO-based baselines also undergo stage 2 training without hints, ensuring the total training steps are aligned with NuRL. Since convergence speed varies across models, we report the detailed hyperparameters and number of steps required for each model in Table 3.
Table 3: Stage configurations and hyperparameters.
Llama
Octo Thinkei
Qwen
wwm.train_samapaa.
8316
8316
8316
max mox.ppttt_eenngth
1800
1800
1800
max response length
9000
9000
9000
Ir
le-6
1e-6
1e-6
clip_ratio_low
0.2
0.2
0.2
hippratii_hiige
0.28
0.28
0.28
rollout. temperature
1
1
rollout
16
16
16
use_kl_loss
False
False
False
train hatp_batch_iiit size
1024
1024
4096
converged steps
375
175
125
At the start of Stage 2, we use the converged checkpoints and generate rollouts per question. Then, we discard samples where all rollouts are correct (i.e., overly easy examples) to improve efficiency. This filtering performed after Stage 1, since the trained checkpoints are stronger than the initial models. The detailed stage configurations
given in Table 4.
Table 4: Stage configurations and hyperparameters.
Llama
Thinker
Qwen
num.train train_samples
5996
4502
1937
max prompt length
1800
1800
1800
max response length
9000
9000
9000
Ir
1e-6
1e-6
1e-6
clip-rttio_ov
0.2
0.2
0.2
clip ratio high
0.28
0.28
0.28
rollout telperatee
1
1
rollout_n
16
16
16
use kl loss train batch size converged_steps
False
False
False
512
512
1937
275
175
90
C DATASET STATISTICS AND LICENSES
We provide the sample sizes and licenses of the datasets used in this work in Table 5. All the datasets are English and all datasets are used in a fashion consistent with their intended use.
15

---

Published as a conference paper at ICLR 2026
Table 5: The statistics and licenses of the datasets used in this study.
Sample Size
License
MATH 500 (Lightman et al., 2023)
500
MIT License
MATH Hard (Hendrycks al., 2021) AIME 2024 (AIME, 2024)
1324
MIT License
30
CCO
EPQA-Diamont (Rein et al., 2024)
198
MIT License
MMLU-Pre (Wang et al., 2024)
12032
Apache License
Date Understanding (bench authors, 2023; Suzgun et al., 2022)
250
Apache License
D PROMPTS FOR HINT GENERATION
We use the following prompts to generate hints. First, we generate explanation style of Chain-of- Thought by conditioning on the ground truth answer:
Prompt for Explanation Generation
Question: (question)
was told the answer is fgold_answed but I don't 't know why. Please explain why the answer is [gold_answer] step by step.
Then, we use the concatenation of the question with the generated explanation, and ask the model generate an abstract hint using the following prompt:
Prompt for Hint Generation
<systen www.eenpppppt>.p.. You are tutor. You are given set of question, correct answer and solution. Your job to provide a hint for the problem. The hint should help the student learn the core concept (e.g. formula, lemma, or necessary knowledge) needed to solve this problem. The hint should be concise, to the point, but high level. Do not include any detailed steps or calculations the final answer. </ </system.prompt
Question: (question)
Answer: [gold_answer]
Solution: (explanation)
Now, please provide hint for this problem to help the student learn the core concept.
16

---

Published as a conference paper at ICLR 2026
E QUALITATIVE EXAMPLES
E.1 EXAMPLES OF DIFFERENT TYPES OF HINTS
We provide examples of the four types of hints we explored in this work. All the examples below answer the same question (shown below) and are self-generated with Dreen--B-Innstuut-e
Example Question and Answer
During a school football match, there are 3 types balls used, each with different number of colors. Type A balls have yellow, brown, and blue variations; Type B balls have green, orange, and pink; and Type C balls have black, red, and white variations. The number colored balls varies according the number of players in each team. Team has 72 members, and each player receives Type A yellow balls, 3 Type A brown balls, and 3 Type A blue balls. Team has 64 members, and each player receives 5 Type B green balls, 4 Type B orange balls, and Type B pink balls. Team 3 has 53 members, and each player receives TType C black balls. Type red balls, and Type white balls. Calculate what percentage of all the balls distributed for the match are Type A yellow balls.
Question:
Answer: 14.35%
Type 1:
Abstract Cue
To find the percentage the total number of that specific ball and then divide it by the total number of all balls dis- tributed thenedd--heennnee multiply by 100. Focus on identifying which team(s) contribute to the specific ball type and sum all balls across teams.
specific type of ball among all distributed balls, first calculate
Type 2:
Partial Steps
Step 1: We are to find the percentage of all the balls distributed that are Type A yellow balls.
So, need: (Total number of Type A yellow balls) / (Total number of all balls distributed) 100%
Step 2:
Calculate total number of Type A yellow balls. Team 1 has 72 members, and each player gets Type yellow balls. So Type A yellow balls = 72 x 288.
Type Explanation
Step 1: Understand what is being asked. We are to calculate the percentage of all the balls distributed that are Type A yellow balls.
Step 2:
Calculate the number of Type A yellow balls. Team has 72 members. Each player gets 4 Type A yellow balls. Thus, Type A yellow balls = 72 x 4 = 288.
Step 3:
Calculate total number of balls distributed. We need to compute the total across all teams and all ball types.
Team (Type A balls): 72 (4 (4+3+3) 72 10 720. Team 2 (Type B balls): 64 x (5
4 64 11 704. Team (Type balls): 53 (3 2 6) 53 x 11 583.
Step 4: Add up total balls across all teams. Total balls = 720 - -044 + 583 2007.
17

---

Published as a conference paper at ICLR 2026
Step 5:
Calculate percentage of Type A yellow
balls.
We
already have
Type A yellow balls = 288. So, Percentage 288
X 100.
2007
Simplify:
288 2007
96
32 223
Now,
32 223
RJ 0.1435, and 0.1435 x 100 = 14.35%.
669
Final Answer: 14.35%
Why is this correct?
Only Team gets Type yellow balls.
Total Type A yellow balls x 288.
Total balls distributed 720 704 + 583 2007.
Therefore, (288/2007) X 100 ~ 14.35%.
Thus, 14.35% of all distributed balls are Type A yellow balls.
Type 4: Ground Truth Answer
was told the answer is 14.35% Do not blindly accept it. Take it as reference and provide your own step-by-step reasoning.
E.1.1 EXAMPLES OF SELF- AND GPT-04-MIIN MINI-GENERATED ABSTRACT CUES
Example
Question: What is the maximum amount of heat that can be supplied to the system during the electrolysis of water without violating the second law of thermodynamics? (A) 237 kJ (B) 286 kJ (C) 163 kJ (D) 49 kJ (Correct answer: D)
Hint :Self-generated): Think about the difference between the energy required to break wa- ter (enthalpy) and the useful energy available (Gibbs free energy). This difference represents the minimum amount of energy that must be lost as heat due to entropy-thh ttis direct con- sequence of the second law of thermodynamics. What is this difference for water electrolysis?
Use the relation AG AH -AS find the entropy change, then recall that under reversible (second-law-limit) process the maximum heat input is Qmaz TAS.
Hint SGPT-oo-mini-generattee
Example
When two fair dice are thrown, the numbers obtained are respectively a and Determine the probability that the line br ay 1 intersects the circle x² 1 = 1 and that the length of the intercepted chord does not exceed 4v2 (Correct answer: }
Question:
3
Hint (Seeftgen::ated): Focus on the geometric condition for a line to cut chord of a given length in unit circle. Recall that the length the chord depends on the distance from the center of the circle to the line use this relationship to set up an inequality involving and b. Then translate the condition into a constraint on a b².
Hint (GPT-o4-mini- min-geenrae:::::::: Use the fact that for the circle x² + y² 1 and the line Va=4z= and the chord-length is L = 2 di (L < 1/2) to get condition on a² 62, then count the integer pairs.
+ ay- = 1, the distance from the origin to the line
18

---

Published as a conference paper at ICLR 2026
Table 6: Evaluation results harder benchmarks. Similar Table 1, NuRL shows superior perfor- mance compared to GRPO across models and benchmarks.
AIME
Brumo
CMTMC
HMMT (Feh / Nov)
OlymMATH (Fasy Hard)
Avg.
-
Zero-shot
0.00
3.33
1.72
0.83 / 3.33
2.50 2.50
2,03
GRPO
0.42
8.13
2.34
2.08 / 6.04
6.69 2.38
4.01
NuRL (Self)
0.21
8.13
2.81
2.50 2.99
7.19 2.69
4.49
NuRL (GPT-o4-mini)
1.67
7.08
2.03
0.83 7.08
6.94/ 2.38
4,00
DetooThinkee--B-yybrii-
Zero-shot
1.2.5
10.63
2.97
0.83 / 6.46
4.06/
4.12
GRPO
4.58
11.25
3.44
2.08 6,46
5.62/2.12
5.08
NuRL (Self)
5.00
11.88
3.28
2,50 / 6.67
6,19 2.19
5.39
NuRL (PPT-o4-mini)
4.58
15.00
3.91
3.54 / 6.04
5.44/2331
5.83
Qwen3-4B- Swen3-4B-Instruce Iee-Innstuuut25555
Zero-shot
46.88
54.79
31.09
31.04 37.71
56.00
14.31
38.83
GRPO
50.42
61.67
32.97
33.33 42.08
56.12/ 15.62
41.74
NuRL (Self)
52.08
61.46
35,31
33.75 / 44.17
56,69
16.50
42.85
NuRL (GPT-o4-mini)
54.58
64.38
35.47
35.42 / 44.79
59.06 17.31
44.43
F ADDITIONAL EVALUATIONS ON HARDER BENCHMARKS
In Table 6, compare NuRL with GRPO
harder benchmarks from MathArena (Balunović et al., 2025) and OlymMATH (Sun et al., 2025). These datasets pose significant challenges for our models-for for instance, Llama reaches only 2.03% zero-shot accuracy average, and even Qwen achieves just 38.83% Despite this difficulty, NuRL remains effective. On Qwen, for example, attains 42.85%. representing gains of +4.02% over zero-shot and +1.11% over GRPO. Taken together, these results suggest that NuRL continues to offer reliable improvements, even on substan- tially harder reasoning benchmarks.
G QUALITATIVE EXAMPLES OF GRPO AND NURL OUTPUTS
We provide qualitative example comparing GRPO and NuRL trained Qwen3-4B models. Notably, NuRL-trained models do not generate hint tokens at test time, so the format and style remain similar to GRPO models, despite the difference performance stemming from NuRL being able to solve more training problems.
Qualitative Example
Question: Caleulate the value of /9 8 cos 20" sec 20".
Qwen3-4B (GRPO) Output:
To find the value of
9 + cos 20 sec 20"
we will simplify the expression step by step.
First, let's simplify the expression inside the square root:
8 20".
We can use the trigonometric identity for cos 20 and some algebraic manipulation to simplify this expression. However, let's first try to simplify the whole expression by using trigonometric identities and properties.
19

---

Published as a conference paper at ICLR 2026
Notice that sec 20 So, the expression becomes
V9 + 8 cos 20
cos 20
Let's simplify
9 + 8 cos 20"
proceed, we need find way simplify 9 cos 20". However, without further trigono- metric identities or specific values, can consider using trigonometric properties and known values to simplify the expression.
Consider the expression
1
9 + 8 cos 20
cos 20
We could attempt a common denominator or other identities, but more direct approach is possible.
After recalculating and simplifying using trigonometric properties and correct identities, the expression becomes
9 + 8 cos 20 sec 20"
Using the proper simplification steps (omitted here for brevity), the final value is 2
Qwen3-4B (NuRL) Output:
We are asked to compute:
9 + cos 20
soc 20
Let denote:
V9 + 8 cos 20s sec 20"
We want to find the value of x.
Step 1: Consider simplification
We first try a numerical approximation to guess the value. Step 2: Numerical approximation
Compute cos 20 and see 20%:
cos 20 ~ 0.9397,
sec 20
~ 1.0642
cos 20"
Now compute:
8 cos 20 ~ 7.5176,
9 + cos 20 ~ 16.5176
Then:
16.5176 4.064
20

---

Published as a conference paper at ICLR 2026
Subtracting see 20%
4.064 1.0642 x 2.9998 x 3
Hence, we suspect:
/9 + 8 cos 20" soc 20 = 3
Step 3: Algebraic verification
We want to verify:
V9 + 8 cos 20 = 3 sec 20
Square both sides:
9 + cos 20
(3 + see 20 -9 + 6 20 + scc² 20
Subtract
8 cos 20 = 6 sec 20 + =eeo 20
Let e cos 20", so see 20 Then:
6
1
+
c²
Multiply both sides by e²:
8 6c + 1
8c - 6c
Check with the triple-angle identity:
cos 30 = 4 cos
3 cos 0
Set 200, then 60%
4 3c
8c 6c
2
This confirms the equation holds. Therefore:
9 + cos 20
3 20
So finally:
/ + 8 cos 20 sec 20 3
Final Answer:
3
21


Reviews:
[{"review_id": "gLWYIwK6me", "reviewer": "Reviewer_1qwq", "summary": "The submission introduces NuRL, which augments typical LLM RL training by providing hints when all responses to a query are incorrect. Experiments consistently show that this method can address the problem of zero gradients for such questions, allowing the model to solve more problems. Ablation studies illustrate helpful properties of hints and support implementation choices.", "questions": "I would be happy to improve my rating after some minor changes that reinforce the paper's existing strengths, which are its exploration of various hint implementations and careful experimental designs.\n\n- Line 136: For RL on LLMs with experience replay buffers, in addition to Lu et al. (2025), I would cite Bartoldson et al. (2025), “Trajectory Balance with Asynchrony”.\n\n- Line 205: How is the amount of information measured in order to support this statement about the ordering? \n\n- Line 313: Could you please explain this procedure more clearly? My understanding is that a model is trained with GRPO until performance plateaus, then it's split into two copies: one that NuRL training is applied to, and another that continued GRPO training is applied to. If this is correct, would it make sense to create another GRPO baseline that -- at the halfway point -- also switches to the 8 responses per query that NuRL uses?\n\n- Line 363: “complementary”\n\n- Figure 3: why exclude Qwen from this experiment?\n\n- Figure 6: it looks like NuRL performance would continue to improve with longer training. For example, if you train for 1000 steps, do you see a plateau with NuRL?\n\n- Figure 6: it would be interesting to augment this study to test if more direct hints improve the solvable fraction on train data but not test data.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**  The hinting implementation's details are ablated and investigated, clarifying how to provide helpful hints and providing potential explanations for why specific hinting schemes work. The method is evaluated on a diverse range of reasoning problems using models with different RL behavior profiles, which is uncommon in RL studies and adds rigor.\n\n**Quality** Experiments are well designed and rigorously test questions of interest. For example, the selection of models tested is well motivated (line 246). \n\n**Clarity** The paper is well written. Figures and tables clearly communicate the method and experimental results. Background work is appropriately referenced. \n\n**Significance** The method improves pass@1 and pass@1024 performance on key RL tasks, suggesting an improved ability to solve hard problems, which the community will be interested in. Ablation studies that characterize properties of good hints can guide future work in this area.", "weaknesses": "There are some highly related works, namely Huang et al. (2025), which the authors cite. The submission is still novel, however, as it focuses on LLMs instead of MLLMs, self-generated hints instead of relying on an external model or dataset, and investigation of the properties of good hinting schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONQy00a2cp", "reviewer": "Reviewer_GxNS", "summary": "The paper proposes NuRL (Nudging LLMs with Reinforcement Learning), a method designed to extend the reasoning boundaries of large language models (LLMs) by enabling learning from previously unsolvable problems. Standard online RL methods like GRPO fail on such problems because all rollouts receive zero rewards, preventing gradient updates. NuRL introduces self-generated hints to “nudge” the model and make these problems solvable. During training, hints are injected only when all rollouts for a problem fail, and only for some rollouts to avoid overfitting.", "questions": "listed in weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The studied problem is important, and the proposed idea is interesting: knowledge outside of rl training can help the model further improve the performance.\n\n2. The evaluation shows that the proposed method achieves consistent performance improvement compared to GRPO.\n\n3.  Ablation studies carefully examine what types of hints help (abstract vs. explicit), when to apply them.", "weaknesses": "1. One of my concerns is where the accuracy improvement comes from. As the authors mentioned: “In practice, such uninformative problems (both too hard and too easy) are often discarded to improve training efficiency”. However, besides discarding those examples, a common practice (e.g., DAPO), will resample and fill the batch (dynamic sampling). Simply discarding the example can hurt the training performance. That’s also why dynamic sampling can improve the model performance and stabilize training. A side effort of NuRL is to address this zero advantage issue, so the performance improvement may come from more stable optimization rather than hints. It would be great to compare NuRL and dynamic sampling to see how those hints can improve the performance.\n\n2. Besides hints from the base model, the paper also proposes to use another more powerful model (e.g., GPT-4o) to generate hints. Such an offline method has already been studied in [1]. It would be great to include it as a baseline for comparison.\n\n\n[1] Yan, Jianhao, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. \"Learning to reason under off-policy guidance.\" arXiv preprint arXiv:2504.14945 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuSTgMyivC", "reviewer": "Reviewer_RKzS", "summary": "This paper proposes NuRL, a reinforcement learning method designed to push the reasoning capability limits of LLMs by enabling learning from tasks that are currently unsolvable under the base policy. Built on GRPO, NuRL incorporates self-generated abstract hints that are injected only when all rollouts fail, turning zero-reward samples into informative training signals. Through experiments on six reasoning benchmarks (MATH 500, MATH Hard, AIME 2024, GPQA, MMLU-Pro, Date Understanding) and three 3–4B models (Llama, OctoThinker, Qwen), the authors demonstrate consistent improvements in pass@1 and pass@k.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a timely and important limitation of reasoning-oriented RL methods.\n- The experimental evaluation is comprehensive, spanning diverse tasks and multiple model scales.\n- The analysis is thorough, including insightful studies on hint type, hint timing, pass@k, and solvable-problem curves.\n    - I really like Figure 4. It clearly demonstrates the relative effectiveness of different hint types.\n- The proposed two-stage selective-hint strategy is novel, well-motivated, and addresses the problem effectively.", "weaknesses": "- The evaluation is limited to models up to 4B parameters. As model capacity increases, the proportion of unsolvable problems naturally decreases. It is unclear whether the method would still be effective when applied to larger, more capable models.\n- There is a clear train and test distribution shift where hints are provided during training but absent during evaluation.\n- The Self-Consistency method lacks sufficient detail. Although it is not the focus of the paper, additional implementation details would provide clearer context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqFy4dmL1n", "reviewer": "Reviewer_1SjR", "summary": "This paper introduces NuRL, a method to leverage off-policy hints (either self-generated or generated by teacher) that are (i) abstractions (ii) partial solutions (iii) explanations and (iv) ground truth answers. These are done for difficult problems when accuracy is zero.\n\nInterestingly, by injecting these hints during GRPO when rewards are all zero, the model manages to achieve superior performance during RL (do they show superior performance on specifically these hard problems? see my question in the Question section below).\n\nComparison is done against vanilla GRPO and there seems to be a 1-2% increase on several benchmarks.", "questions": "1. Qwen3-4B seems to achieve already such a high accuracy on the benchmarks. I believe it would strengthen your paper if you can evaluate on more difficult benchmarks. For the rebuttal, I'd like to see results on ALL benchmarks listed in the final answer competitions in the standard Math Arena: https://matharena.ai/?comp=overall--final-answer_competitions. In addition, please add evaluations on OlymMath-EN: https://github.com/RUCAIBox/OlymMATH/tree/main/data. If these are still too easy for Qwen3-4B base (>50% accuracy), you can also look into evaluating on Omni-MATH.\n2. What's the difference between before hint and after hint in figure 6? Can I see another curve which evaluates the model's capability *without the hint*, after training on rollouts *conditioned on the hint*?\n3. Related to the above question, why do you think training on rollouts conditioned on the hint should improve rollouts without the hint? If the hint tokens are not upweighted, then I don't understand why they would be sampled at test time. And if they are not sampled, the rollouts shouldn't improve?\n4. Can you compare and contrast example rollouts on difficult problems (i) without the hint, (ii) given the hint, and (iii) after training on the hint-conditioned rollouts, without the hint?\n5. In figure 4, what is the performance of the models without any hints? Could you add a bar next to them showing that\n6. How do you compare against baselines where you SFT on the partial solutions? How about against other methods that learn directly on correct traces such as LUFFY (Gemini generated traces) and Critique-GRPO (self-revised correct races)?\n\nIf you can make this paper more sound and answer the questions above with solid results, I will raise my score and vouch for you.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem setting is novel and interesting - getting GRPO to sample non-zero rewards on difficult problems\n- The method is simple and effective\n- very nice diagrams\n- paper well-written, easy to understand", "weaknesses": "- not enough difficult benchmarks, esp. for Qwen3-4B-Instruct which is already at 70% performance even before RL\n- I'd like to see some more qualitative rollout examples, mentioned in questions\n- more comparison against other methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)