QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
Authors: Jiazheng Li
Hongzhou Lin
Hong Lu
Kaiyue Wen
Zaiwen Yang
Jiaxuan Gao
Yi Wu
Jingzhao Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model; Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: By introducing partial solutions to make hard problems easier during reinforcement learning, this method significantly boosts the mathematical reasoning capabilities of language models.

Abstract: Reinforcement learning (RL) has emerged as a central paradigm for training large language models (LLMs) in reasoning tasks. Yet recent studies question RL’s ability to incentivize reasoning capacity beyond the base model. This raises a key challenge: how can RL be adapted to solve harder reasoning problems more effectively?
To address this challenge, we propose a simple yet effective strategy via Question Augmentation: introduce partial solutions during training to reduce problem difficulty and provide more informative learning signals. 
Our method, QuestA, when applied during RL training on math reasoning tasks,  not only improves pass@1 but also pass@k—particularly on problems where standard RL struggles to make progress. 
This enables continual improvement over strong open-source models such as DeepScaleR and OpenMath Nemotron, further enhancing their reasoning capabilities. We achieve new state-of-the-art results on math benchmarks using 1.5B-parameter models: 72.50\% (+10.73\%) on AIME24, 62.29\% (+12.79\%) on AIME25, and 41.67\% (+10.11\%) on HMMT25. Code, data and model are available at https://anonymous.4open.science/r/questa932.

Published as a conference paper at ICLR 2026
QUESTA: EXPANDING REASONING CAPACITY IN LLMs VIA QUESTION AUGMENTATION
Jiazheng Li Hongzhou Lin 13 Hong Lu 1.2, Kaiyue Wen', Zaiwen Yang', Jiaxuan Gao', Yi Wu Jingzhao Zhang 'Tsinghua University, SShnggaai Qi Zhi Institute, AAmazon, *Stanford University
ABSTRACT
Reinforcement learning (RL) has emerged as central paradigm for training large language models (LLMs) in reasoning tasks. Yet recent studies (Yue et al. 20055: Liu et al. 2025) question RL's ability to incentivize reasoning capacity beyond the base model. This raises a key challenge: how can RL be adapted solve harder reasoning problems more effectively? To address this challenge, we propose a simple yet effective strategy via Question Augmentation: intro- duce partial solutions during training to reduce problem difficulty and provide more informative learning signals. Our method, QuestA, when applied dur- ing RL training on math reasoning tasks, not only improves pass @ I but also pass @ k-pattiicllarly on problems where standard RL struggles to make progress. This enables continual improvement over strong open-source models such as DEEPSCALER and OPENMATH NEMOTRON, further enhancing their reasoning capabilities. We achieve new state-of-the-art results on math benchmarks using 1.5B Sa-paarmeetr models: 72.50% (+10.73 on AIME24, 62.29% (+12.79%) on AIME25, and 41.6 41.67% (+1001111111 1%) on HMMT25. Code, data and model are available 120/ Que Qustt
at https:://gtt hhub. hub. com/ www.://weeeeeeeaeeeeeeeeeeaaaeaa
Qwen3-1.7B
QuestA-Ner -ew.otoon-1.5 DeepSeek RI RI-Distill-32B
Nemotron-11.9
70-
Accuracy (Avg@332 %))
60
16.2
1L
50-
-A2
4LKT
40-
30
20
0
AIME24 AIME25
HMMT FEB 25 Olympiad Bench
BRUMO25
Figure 1: QUESTA a data augmentation method that injects partial solutions to effectively scaf- fold RL training on hard reasoning problems. We construct 26K high-quality augmented prompts from challenging instances in OpenR1 (Open-R1 Team 2025), and fine-tune models using 32K- context-length RL. When applied Nemot 5B, QUESTA delivers substantial performance gains-achieving new state-of-the-art results across all math benchmarks for L.5B-parrmeter models.
INTRODUCTION
Frontier large language models (LLMs), including OpenAI-O1, 03 (Jaech et al. 2024). DeepSeek- R1 (Guo et al. 2025), Qwen3 (Yang et al. 2025), and Gemini 2.5 (Gemini Team, Google DeepMind
"Equal Contribution
1 Contacct: Jiazheng at foreverlasting 202 2 @utlook..om, and Jingzhao at jingzhaoz @ mail. mail.tsinghua.cduu This work is independent of and outside of the work at Amazon.

---

Published as a conference paper at ICLR 2026
Nemotron-
Gasy-Nemotront
QuestA-Nemotron-ll
Hard- Sonidd-emttoonn....
AIME25
HMMT Feb 2025
100
90
Pass@k Accuracy (%)
90
80-
Pass@l P Accuracy
70
80
60
70
50-
60
40
50
30
10
10" Sample Size (log scale)
10"
10"
10"
10
10"
10
Sample Size (log scale)
Figure 2: We compare pass @k curves of RLVR-traine' models, with and without QUESTA. As controlled experiment, we perform RL training using either easy or hard prompts. Standard RL on easy prompts (red) shows clear degradation in pass @k k as k increases compared to the base model (blue). Training on hard prompts (green) improves pass @k but comes at the cost of substantially longer training. This motivates our development of QUESTA, which scaffolds hard problems to improve training efficiency and delivers consistently stronger results: the RL+QUESTA model (orange) stays above standard RL (red) across all k, while also preserving improving performance at larger & relative to RL trained with hard prompts.
2025), have exhibited exceptional performance on high-complexity reasoning tasks spanning mathe- matics, programming, and formal logic. Recent advances in the field have increasingly prioritized reinforcement learning paradigms (RL), among which Reinforcement Learning with Verifiable Re- wards (RLVR) has emerged as a scalable and efficient approach to enhancing reasoning capabilities. Using automatically verifiable signals, RLVR enables alignment between model output and objective correctness, thus addressing a critical limitation of traditional RI. for reasoning.
However, the community remains divided on a fundamental question regarding RLVR: does it
expand the model's intrinsic reasoning capacity, merely exploit pre-existing knowledge encoded in the base model? Recent research (Yue et al.] 2025)] Liu et al. 2025| Zhao al., (025) show that while state-of-the-arr RL methods (c.g.. GRPO, DAPO) (Guo et al. 2025 Yu et al. 2025 An al. 2025) can enhance the pass @ 1 metric by reinforeing high-reward completions, they encounter significant limitations when tackling high-difficulty tasks where the base model performs poorly. This phenomenon differs from that observed in Supervised Fine-Tuning (SFT) Luo et al. (2025a). Within the SFT paradigm, enhancing the diversity of problem difficulty serves critical factor, as it can effectively improve the model's 's performance on downstream tasks. However, in the framework of RLVR, the inclusion of easy prompts tends to undermine the model's inherent reasoning capabilities.
One insightful explanation (Cui et 2025 Wang et al. 2025a) for the drop suggests that model overfits on correct solutions and hence causes entropy collapse, limiting its ability to explore. To validate this, we design controlled setup that separates prompts into easy and hard groups. When applying RLVR on the Nemotron 1.5B model (Moshkov et al. 2025) with the OpenR1 dataset, we find that training on easy prompts leads to a clear decline in pass @K accuracy (Figure 2
Given these findings, observe that training with hard prompts is more beneficial than with easy ones. Yet, RL training on hard problems tends be much slower, as sparse reward signals and limited sample efficiency hinder progress. The key challenge, then, is how to structure the learning process to fully expand reasoning capabilities while mitigating the inefficiency of RL on hard tasks. To this end, we introduce QUESTA: a parsimonious and efficient strategy that dynamically adjusts problem difficulty during RL training. The core contributions of this work are threefold:
We notice that the evolution of model capacity in RLVR critically depends on dataset difficulty, underscoring the importance of training on hard problems expand reasoning ability.
2

---

Published as a conference paper at ICLR 2026
Hard-Nemotron-I1
QuestA -Nemotron- 1.58
AIME25
HMMT Feb 2025
60
Passi Accuracy (%
58-
Pass Accuracy (%)
56
54
52
50
200
400
600
800
1000
0
200
400
600
800
1000
Training Steps
Training Steps
Figure 3: Comparison of RL training dynamics: Training with only hard problems (green) makes progress very slowly due to sparse rewards, while our method with partial solutions (orange) acceler- ates training and consistently achieves higher accuracy across training steps.
We introduce QUESTA, efficient procedure that controls difficulty by augmenting hard problems with partial solutions. This approach provides smooth curriculum within RL training and makes high-difficulty tasks more tractable. Through our fully open-sourced training pipeline, QUESTA consistently improves pass @1 and pass @k, enabling 1.5B-parameter models to reach new state-of- the-art performance 72.5% on AIME24, 62.3% on AIME25, and 41.7% on HMMT25 (Table I
Our theoretical analysis in Section 4 Texplains why partial-solution augmentation accelerates RL training: by decomposing problems into intermediate steps, the method yields denser reward signals and improves sample efficiency, while still driving the model to master the hardest problems.
2 TRADEOFFS BETWEEN REASONING CAPACITY AND LEARNING EFFICIENCY
Given the ongoing debate on whether reinforcement learning enhances the reasoning capacity of language models, design controlled experiment study how dataset difficulty changes model performances measured by pass accuracy. Specifically, we filter out easy problems and hard problems from the 220K OpenR1 dataset, base on model's 's success rate, each containing around 4K data. We then RL with GRPO for one thousand steps. This setup allows choice of prompt difficulty impacts the model's 'ss reasoning capacity. In Figure 2 and Figure 3 we provide pass@1 @k comparison and the learning dynamics, we make two observations.
isolate how the
RL with Easy Prompts Hurts @ and Reasoning Capacity. Training easy or already- solvable problems leads overfitting shallow patterns, reinforeing confidence rather than ex- panding reasoning capacity. While pass@I may rise, output diversity declines and performance on harder benchmarks deteriorates, with pass@A @1 k dropping at larger k (see Figure This suggests that the model exploits familiar solution modes instead of exploring new trajectories. To truly expand capacity, RL training should focus on hard problems, where the policy forced to explore and acquire novel solution strategies.
RL with Hard Prompts Leads to Slow Learning. Training on hard prompts directly targets the reasoning capacity of the model, but the learning process is much slower (see Figure B and less sample-efficient. The difficulty arises because RL rewards on these problems are sparse, providing limited gradient signals for policy improvement. We formallize the underlying reason in Section and in Theorem
In practice, not all questions in the training set Q are equally difficult, and one might hope that training easier examples could generalize to harder ones. However, empirical evidence suggests that baaed training exhibits bi-modal pattern in success rates (An et al. (025); by the end of training, models tend to either solve question reliably or fail entirely (see Figure 6). This implies that once question falls outside the model's capacity set, the RI. algorithm is unlikely recover.
3

---

Published as a conference paper at ICLR 2026
Together, these results highlight tension: easy prompts dilute reasoning capacity, while hard prompts stall learning altogether. This motivates the need for strategies that can retain the benefits hard problems while mitigating the inefficiency caused by sparse rewards. To this end, we introduce partial solutions that break complex question into smaller, (Theorem 4.6) suggests that appending part of the solutions as hint can greatly improve RL efficiency.
approachable pieces. Theoretical analysis
Empirically, we simply choose the hint to be part of the solution of the original question q and observe faster learning in Figure Surprisingly, the hint, the model's ss capacity without hint still continues to improve and lead to steady improvement We elaborate on implementation
if we don't explicitly train the model to generate
in problems out of reach in standard RI. training (see Table 3 details in the section.
3 QUESTA: QUESTION AUGMENTATION WITH PARTIAL SOLUTIONS
QUESTA modular augmentation framework designed to inject partial solution sketches into prompts during reinforcement learning (RL) training. It adresses scenarios where the base model fails caapletions-conditionn to generate correct completions- conditions that typically result sparse reward signals. Distinct from approaches that modify reward functions or optimization algorithms, QUESTA operates at the input level: it transforms original training prompts into more tractable variants, thereby exposing intermediate reasoning steps to the model.
Original Prompt
Let N be the positive integers. function f N:N satisfics the equation
n with f(a) applications of f;
1( f(n)...))
f((())
for all positive integers
Given this information, determine all possible values of f(1000).
QUeSta
Augmented Prompt
Let N be the set of positive integers. The function f N satisfies the equation
n
(()...)())...... f(f())
with f(n) applications of f.
all positive integers n. Given this information, determine possible values f(1000).
## Hint: Partial Solution
Analysis shows that must be un involution, meaning (n)) for all n, and it fixes all odd positive integers, so f(n) = for odd n. For even positive intogers. f cither fixes the number or s swaps it with another even positive integer in 2-cycle.
Please reason step by step, and put your final answer within (boxed[].
Figure 4: QUESTA augments each original question in the dataset by prepending the first p% of the solution sketch. In our experiments, we apply augmentation using the solution block rather than the reasoning chain-of-thougle The hint percentage is computed as the ratio of tokens used as hints to the total number of tokens in the solution sketch.
Question Augmentation Mechanism For given problem x with an n-step solution trajectory
= (1), .2,..., ). QUESTA constructs a set of augmented prompts (5(2)), ). where each z(P) appends the first p steps of the solution prefix the original question. The parameter (e.g.. p = 50% or 25%) quantifies the proportion of the solution revealed, thereby enabling precise control over the difficulty of the augmented prompt.
In our empirical evaluations, employed the OpenR 1-Mahh MMath-220K dataset (Open-B RI Team 2025 supervised fine-tuning (SFT) corpus containing solution trajectories generated by DEEPSEFK R Each instance in this dataset comprises detailed chain-of of-thou (CoT) section followed by final solution block. For augmentation, extracted the final solution (omitting speculative reasoning
4

---

Published as a conference paper at ICLR 2026
within the CoT section). The solution then truncated predefined percentage p and prepended the original question. yielding the augmented prompt used in RL training, as shwon in Figure
Targeting High-Diffieulty Problems QUESTA is applied exclusively prompts where the base model's pass rate is close to zero. Using the OpenR 1- -Math-220K dataset, we first employ lightweight heuristic filters to reduce the full 220K problems to 26K of the hardest candidates. These problems then augmented with partial-solution prefixes where conduct second difficulty screening: sample multiple completions from the model for each augmented prompt, and only those instances with consistently low pass rates are retained. This two-stage filtering pipeline yields a final pool of more than 10K problems, ensuring that augmentation resources are concentrated on the most challenging cases where the base model needs additional guidance and scaffolding.
Integrating with RL Pipelines QUESTA exhibits orthogonality underlying RI algorithms, en- abling seamless integration into existing training pipelines (c.g.. GRPO (Shao et al. (024). DAPO (Yu et al. (025)) without modifications. Specifically, integration requires only replacing the original rollout dataset with the augmented dataset, while retaining the original reward function and policy update mechanism. To further exploit this input-level flexibility, we extended QUESTA with an iterative curriculum RL paradigm:
1. First, augment the dataset with p 50%. apply the difficulty filtering with the augmented prompt, and conduct reinforcement learning training until the performance saturates.
2. Second, reduce the augmentation from p = 50% to p = 25%. i.c. provide fewer hints. Again, we apply the difficulty filtering, and conduct reinforcement learning training until convergence.
Here, the rationale for the choice of p is provided in Appendix B.6 By keeping the training signals strong at each stage, the method speeds up convergence difficult tasks and makes QUESTA simple, andug-and-plaa approach for murriculum-based RL.
THEORY: VARYING LEARNABILITY ENHANCES RL EFFICIENCY
In this section, we present a theoretical perspective on how question augmentation improves the efficiency of reinforcement learning. Our central thesis is that the primary bottleneck in RL-based reasoning lies in the difficulty of discovering successful trajectories within a finite sampling budget. Question augmentation addresses this challenge by reshaping the learnability landscape-making hard problems more discoverable by increasing the likelihood of encountering correct trajectories.
Motivated by experiments which quantify model capcity with pass @ k accuracy, we introduce the following notions of solution ser (Definition I..) and model capacity set (Definition 442) for a given question y and model μr. Let V be the vocaboluary set, and let (, 7) denote the probability that language model μ generates trajectory
V when conditioned on input question V*.
Definition 4.1 (Solution Set). Given a question and a binary reward function R. : V* x V* + {0, the solution set is defined as:
Si4) fr E V R(g. =1}
Definition 4.2 (Model Capacity Set). Given a probability threshold 6, > 0, language model and a question q. define the model capacity set C(g. 5p) as the smallest set of trajectories whose total probability mass is at least 1 ògi
Clg, 8p) arg min
|s|
P.(a)) 6p TES
SCV-
The Model Capacity Set C(q, 5p) intuitively captures the set of most likely output trajectories that the model μ can generate for a given input g, up to a small probability threshold 6p.
This formalization leads critical insight: if the model's capacity set fails to intersect with the solution set-menning the model is unlikely to generate any correct completions then the RI. process cannot make progress. To articulate this more formally, we begin by stating standard assumption satisfied by many popular RL algorithms, such as DAPO and online GRPO:
5

---

Published as a conference paper at ICLR 2026
Assumption 4.3 (Null Gradient from
The RL algorithm does not update the model weights if none of the sampled rollouts receives positive reward (i.e., reward 1).
Under this assumption, we easily the following lower bound, which states that if all training questions are unreachable within the model's capacity set, the RL process is likely to stall entirely:
Theorem 4.4 (Lower Bound on RL Learnability under Solution Inaccessibility). Given a probability threshold 6, > O, for every question q E 2, the model capacity set C(q, 6p) does not intersect with the solution set S(q). iee.
Clq. 6p) (((()) 0, bqq E Q
then under Assumption when training RI. T steps with B samples per step such that TB- (L1/6)), there a constant probability that the RL algorithm will not update the model.
To overcome this limitation, our method QUESTA provides simple yet effective solution: augment each question Q with a partial solution to improve the chances of sampling informative trajectories. Formally, we assume the existence of a hint hi for every question q E Q that can guide the model toward discovering valid completion.
Definition 4.5 (Question Augmentation). For every question q E Q, hint by V satisfies that for %, - for some >> 0:
11/2-c
the hint hg can be generated with a non-neglible probability: PA (h(hygg) 2 2%
there exists a solution to the hinted problem 8q E S(q)) such that Sy can be generated with high probability after Sq. i.e.
Pa (s))l(a,ha)) 8,. Rlg,h =
The hint h, can exist for every question even when the model's capacity set C(q, 5p) does not intersect with the solution set S(q). For instance, if every solution can be decomposed into two steps, and the model can generate each step correctly with probability 8 = v o(6p), then the possibility of generating two steps correctly at the same time is only o(6p).
This implies that the sampling budget needed with a hint is asymptotically almost the square root of the budget required without it (116)))), as given in Theorem 4.4 We further provide learnability result where we assume the policy is parameterized by a softmax policy parameterization tabular RL setup.
classical
Theorem 4.6 (Informal Upper Bound on RL Learnability with Hint). If we have hint hy for every question q E (Def. 4.5) then there exists RL algorithm that can output a policy ne such that E Enw-Unitom 2) [P ( E S(4))) Z 0.99 with ((1/8))) sampling budget with high probability.
Theorem 4.6 provides a theoretical guarantee that the model can reach a high training success rate when partial solution is included. Empirically, observe the model generalizes well both in-distribution and out-of-distribttion to hard questions.
EXPERIMENTS
as a weak selection model to filter it down to the 26K hardest items. This set serves as our base prompts. We then use Nemotron- 1.5B sample eight generations per prompt and classify problems into Easy Data (7-8 correct answers) and Hard Data (0-1 correct answers), enabling controlled experiments introduced in Section The exact prompt template is provided in Appendix B.8
Dataset. We begin with the OpenRI-Math- -20K dataset and use DeepSeek-RI-Distill-
Data Augmentation (QUESTA). To improve the tractability the problems, we apply QUESTA to prepend the prompt with partial solutions, i.e. first p% of the full solution in the SFT data provided in the OpenR 1-Math-2208 dataset. After augmentation. to sample generations per augmented prompts and select
the initial model at RL training, either
Nemotron-1 1.5B or DeepScaleR-1.55 samples with 0-4 correct predictions. Full details are provided in Appendix [B.5] These high-variance cases provide stronger learning signals and make the training process more effective.

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison (Pass I, averaged over 32 samples) across maths benchmarks. The best results among the 1.5B models are highlighted in bold. Larger models are shown in gray as reference points. Reported results for DeepSeek- R 1-Distill and Qwen3 are taken from their official documentation (Guo al., 2025 Yang al. 2025), while the rest are self-evaluated. Our QUESTA-Nemotron 1.5B achieves state-of-the-ar performance among 1.5B models and, notably, matches even exceeds the performance of DeepSeek- -R RI--Dstill--3BB across several benchmarks, despite being over 20x smaller parameter count. This demonstrates the effectiveness of QUESTA in enhancing small model capabilities through targeted training.
Model
AIME24
AIME25
HMMT FEB 25
Olympiad Bench
BRUMO25
Avg
DaaSSec-RI--istill--...
28.7
22.3
12,0
52.4
31.8
29.44
Qwen3-1.7B
48.3
36.8
22.19
56.13
44.06
41.50
DeepSeei-RI-Distill-
72.6
51.8
33
65.0
68
58.08
Qwen3-8B
76.0
67.3
44.79
68.56
68.33
64.99
Nemotron-1.52
61.77
49.50
31.56
64.62
58.23
53.14
QUESTA-Nemooton--..3
72.50
62.29
41.67
70.36
69.48
63.26
Training Setup. We use AReaL (Fu et al. 2025 our RL training framework, applying the
GRPO algorithm Guo et al. 2025) without the Kullback -eeibbeck-Lebler (KL) divergence loss. Following DAPO Yuu et al. 2025 we also dynamically filter out prompts that are either all correct or all 16 responses per prompt with a maximum prompt length of 8192 tokens and maximum generation length of 24000 tokens, using a sampling temperature of 1.0 and clipping hyperparameters with Elow = Ehigh = 0.2. The batch size is 128 with mini-batch size of equivalent 128 gradient updates per rollout step. Optimization is performed with A (Kingma & Ba, 2017: Loshchilov Hutter (0099) using a constant learning rate of 2 x 10-5, Experiments are conducted on eight NVIDIA H800 H800(80BB) nodes. Full details of our training method are provided in Appendix B.
incorrect during rollouts. During training, sample
Evaluation Setup. For each problem in the evaluation benchmarks, we generate 32 samples and report pass I results. Generation uses a sampling temperature of 0.7 and top-p value 0.95, with k = 32 responses per question unless otherwise specified. It is important to note that while partial solutions were incorporated during training, no partial solutions are provided at evaluation time.
5.1 EXPERIMENTAL RESULTS
Key Results. Table D reports results challenging math benchmarks. QUESTA yields substan- tial gains for Nemotron- 1,5 achieving an average improvement of 10% over its baseline and particularly strong + +13 % on AIME25. These improvements are consistent across all benchmarks, highlighting the effectiveness of our approach in enhancing problem-solving robustness.
Compared to other models, such as DeepSeek-R1- -iieck-RR-iistll-I... and Qwen3-1.7B and even surpasses larger models like DeepSeek- R1-Distill-32B across all benchmarks. On AIME25 in particular, exceeds DeepSeek-RR R1-Distill-322 by a substantial margin +11%. Against the stronger Qwen3-8B, QUESTA-Nemotrom 1.5B remains competitive despite operating at a fraction of the parameter scale.
consistently outperforms peers of similar scale,
Training Dynamics. 3 summarrizes the training dynamics of SODESTA-NNmootron-1... A positive correlation is observed between average response length and model accuracy, reflecting common trends in RL training. Notably, with QUESTA, the entropy during RL training remains stable and does not exhibit significant collapse.
Pass Analysis. Our evaluation follows the standard pass @ k methodology, consistent with DeepSeek-R1 (Guo et al.] 2025), with further details provided in Appendix [B2]] In contrast recent findings that RL-based training can reduce pass@ @ at larger values (uue al. at. [2225 [iu et. 2025). our results show that QUESTA preserves-and and in many cases modestly improves-performaatee performance across broad range of k. As shown in Figure 2 incorporating partial-solution hints within two-stage curriculum yields consistent gains across models, without the degradation in pass @ often observed under standard RL training. These results indicate that QUESTA enhances both the quality and diversity of candidate solutions, rather than overfitting single best trajectory.
7

---

Published as a conference paper at ICLR 2026
QuestA QuestA-Nemotrrat -Nemotron-l.5B
Average Reward
Average Length
Average Entropy
0.65
18000
Average Reward
0.5
Average Length
Average Entropy
16000
0.60
0.4
14000
0.3
12000
0.55
1000
2000
1000
2000
1000
2000
Training Step
Training Step
Training Step
The first and second charts show the progression of average response length and average reward across rollout samples during the RL process, both of which steadily increase over time. The third chart presents the average entropy. Interestingly, the entropy increases over time, suggesting that QUESTA does not suffer from entropy collapse and instead encourages diverse and exploratory behavior.
Figure 5: Training dynamics of CREESTA-Nemotron-
Nemotrom-11 11.55 (Pre RL)
12500
QuestA -emottonn 1.5B (Post RI.)
Number of Problems
10000
Pro-RL:2604| problems, Mean Pass Rate: 0.572
Post-RL. RL.: 2604
problems, Mean Pass Rate: 0.757
7500
5000
2500
0.000
0.125
0.250
0,375
0,500
0.625
0.750
0.875
1.000
Rate
Figure 6: Pass Rate Distribution on Training Prompts. We compare the success rate on the 26K training set before and after RL, using the average pass rate over samples per question. Although partial solutions are included during QUESTA training, no hints are provided during this evaluation. This setup isolates the true impact of QUESTA by assessing its ability to improve performance problems without hints. QUESTA significantly reduces the number of unsolved or partially solved problems in the training set, especially for hard ones where initial model solves only 0/8 or 1/8 times.
Generalization at Test Time when Hints are Removed. A natural question arises from our approach: since we add partial solutions during RL training, does this improvement persist when hints are removed at evaluation time? To answer this, Figure the pre- and post-RL models on the 26K training prompt set, evaluated without any hints. The distribution clearly shifts away from the 0/8-1/8 bins toward higher pass rates, indicating that the model solves a larger fraction of problems even without access to partial solutions. On the evaluation AIME benchmarks, Table 2 further demonstrates that QUESTA expands coverage at Pass@32: for Nemotron- 5B, the number of unsolved problems drops from 5 to on AIME24 (newly solved indices 2, 13, 29) and from 6 to 3 AIME25 (newly solved indices 9, 13, 27). Taken together, these results show that method generalizes well beyond the training setting and helps solve hard problems that are otherwise inaccessible without solution guidance.
5.2 FURTHER ABLATIONS
A Ablation with Difficulty Curriculum. We first motivate the choice of two-stage curriculum: RL. on Partial-50 followed by RI. on Partial-25. From a modeling standpoint, the most appropriate inference distribution for the model should be the original (no-hint) distribution. Hence, during training we should gradually reduce reliance on hints to align the learned policy with the evaluation distribution. This motivates decreasing the partial ratio over time so that the model transitions from scaffolded reasoning autonomous reasoning.
8

---

Published as a conference paper at ICLR 2026
Table 2: Indices of unsolved problems Pass@32 AIME24 and AIME25 (with indices ranging from 0-29), Our method, QUESTA, consistently improve the model capacity on hard cases where the initial model is unable to solve, improving overall coverage at Pass@32.
Models
AIME24 Unsodlved Indices
AIME25 Unsolved Indices
Nemotron-LL55
2, 3, 13, 21, 29
9, 12, 13, 14, 27, 29
3,
12, 14, 29
Table 3: Ablation Study on the Impact of Curriculum Design. This table demonstrates the importance of curriculum learning in improving model performance. The model was trained entirely with Partial-50 data for 2000 steps, while QUESTA-Nemmtrron- 1.5B followed a curriculum learning approach, starting with 100 steps of Partial-50 data followed by 1900 steps of 1.5B) Partial-25 data. As seen in the table, the curriculum learning approach Extension with Partial- outperforms training with only Partial-50 data (QUESTA- www--rooooooo-------- 50-+ --rrtaaa---22 P3r-taati did not yield significant improvements, and thus, not included in the table.
1.5B-50
Model
AIME24
AIME25
HMMT FEB 25
Olympiad Bench
BRUMO25
Avg
Nemotron- 1.5t
61.77
49.50
31.56
64.62
58.23
53. 14
QUESTA-Nemotron: Nemotron- 1,5B-50
67.18
59.38
39.17
69.41
66.15
60.26
QUESTA- 72.50
62.29
41.67
70.36
69.48
63.26
Empirically, Tshle that, under the same 2000-step budget. the curriculum Partial-50-PParre 25 learns substantially better than training on Partial-50 alone. We cap the Partial-50 stage at 100 steps, after which switch to Partial-25. As shown in Figure TI] entropy for QUESTA -Nemotron- 1,5B-50 begins to decline beyond 100 steps, so transitioning at this point prevents overconfidence and sustains training stability. We have also tried extending the curriculum from Partial-25 to Partial-O in experiments, but observed gains and increase in response length (see Figure 12.
Table 4:
Performance comparison (Pass@) I, averaged over 32 samples) between Nemotron- 1.5B and 1.5B (By OpenMatthReasoning) The two models achieve comparable results, with the version trained on OpenRI performing slightly better overall.
Model
AIME24
AIME25
HMMT FEB 25
Olympiad Bench
BRUMO25
Avg
Nemotron- 1.5B
61.77
49.50
31.56
64.62
58.23
$3.14
QUESTA-50 (with OpenMuthReusoningg
66.46
58.54
36.35
66.06
63.13
QUESTA-50 (with OpenR
67.18
59,38
39.17
69.41
66.15
60.26
Ablation with Different Dataset. We also evaluated QUESTA on OpenMathiReasoning Moshkov al. (2025), selecting the 60K questions with pass_raaee r_____aee____________e_______e__ tir of 0 or 1/32. Due time constraints, we trained only the first stage of QUESTA with 50% partial solutions. Table shows that Nemotron-I.5BB 1.5B-50 achieves similar performance using the OpenRI dataset. This indicates that our approach generalizes across datasets.
72b_tir
Other Ablations. We also conduct extensive set of comparative experiments and ablation studies, with detailed results provided in Appendix| [] These include an ablation of QUESTA without hints (Appendix D.I. experiments with different model backbones (Appendix D.2) and the full set of pass rates and training curves for additional models (Appendix D.3
CONCLUSIONS
In this work introduced QUESTA, lightweight data-centrie framework that augments hard prompts with sartial-solution hints during RI. training. Without altering model architecture reward design. QUESTA sets new state-of-the-art results for 1.5 B-scale models on AIME24, AIME25 and HMMT25. Further, theoretically demonstrate how question augmentation improve sample

---

Published as a conference paper at ICLR 2026
efficiency. Our analysis shows that the method can potentially be generalized to other domains such competitive coding, software engineering or other agentic tasks. Desigining proper question augmentation pipelines for theses new tasks can be an important and interesting future direction.
ETHICS STATEMENT
We only public, non-PII datasets-OOenRRI I-Math--200- (Apache 2.0) and Openoaahneeaooninn (CC BY 4.0) full compliance with their licenses (including attribution and modification notices); no new human-subjects data were collected, no re-identification was attempted, and no IRB review was required. Our augmentation pipeline generates math problems and solutions while avoiding harmful or copyrighted non-math content; outputs may inherit source biases, so we report settings transparently, discourage high- high-stakes deployment misuse without safeguards and human oversight, and will release artifacts that respect the original licenses.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, provide the code, dataset and model in the the supplementary materials and anonymous github ht ttss:///// anonymous. 4open. www.ssccieeeeieeeeeeee wwwence/e//uesstaeeeeeeeee// nd In the README. file included with the code, we present a step-by-step guide for reproducing results.
README
9 ACKNOWIEDDGEMENTS AND DISCLOSURE OF FUNDING
J.Z. acknowledges support by the National Key R&D Program of China 0024YFA1015800 and Xiongan AI Institute.
REFERENCES
Brown university math olympiad 2025. February 2025.
Harvard-mit mathematics tournament 2025. February 2025.
Chenxin An. Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning advanced reasoning models, 2025.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms uncontaminated math competitions, February 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint S.ii::2505.22299 2025.
Zitian Gao, Lynx Chen, Haoming Luo, Joey Zhou, and Bryan Dai. One-shot entropy minimization, 2025.
10

---

Published as a conference paper at ICLR 2026
Gemini Team, Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. Accessed: July 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, el. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12944 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Lim reinforcement learning with on-policy tree search, 2025.
Aaron Jaech, Adam Kalai, Adam Lever, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint M.Ni::24/2.1672000 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in training of Ilms, 2025.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. Science, 378666244 December 2022. ISSN 1095-9203. doi: 10. 1216//iiecce.aaeei
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of Ilms for logical reasoning, 2025.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint 2il.22255.. 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2025a. URL
https::/aaae /ttp://aaivvvvviva org/ abs 2308. 09583
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1.5b model by scaling rl, 2025b. Notion Blog.
MAA. American invitational mathematies examination 2024. February 2024.
MAA. American invitational mathematics examination 2025. February 2025.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannanch Ha- jishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation, 2025.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods, 2022.
11

---

Published as a conference paper at ICLR 2026
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal. Christof Henkel, Benedikt Schif- ferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-ar mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504. 16891, 2025.
Open-RI Team. -penr1-math-2211 A large-scale dataset for mathematical reasoning, 2025.
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, and Shuiwang Ji. Curriculum reinforcement learning from easy hard tasks improves Ilm reasoning. 2025.
Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-Ievel google-prood q&a benchmark, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Rulin Shao, Shayue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. 10947, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xxx:24422.03300 2024.
Peiyi Wang. Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations, 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive 2025a.
effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0199:
Yiping Wang. Qing Yang. Zhiyuan Zeng. Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang. Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 2025b.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye. Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXhv:2506.14245. 2025.
Zhiheng Xi, Wenxiang Chen, Boyang Hong. Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning, 2024. URL httpp://arxxi..e org/ abs/ /t/////////// 2402 05808
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Inftythink: Breaking the length limits of long-context reasoning in large language models. arXiv preprint arXiv:2503.06692. 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0m0 093888 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2rXiv:2503.
2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar 2504 13837, 2025.
12

---

Published as a conference paper at ICLR 2026
Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint arXiv:2507.0284), 2025, doi: 10.48550/arxiv.25077. 250.00241.. Submitted Jul 3, 2025.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining, 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in Ilm reasoning. arXiv preprint arXiv:2506.01 2025.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
The Large Language Models (LLMs) were exclusively utilized to polish the writing and detect potential typos, with no involvement in other aspects.
B IMPLEMENTATION DETAILS
B.1 RLVR ALGORITHMS
We have employed the GRPO algorithm enhanced with subset of DAPO techniques. Primarily, we have integrated DAPO's Dynamic Sampling Trick and eliminated the KI. divergence term, resulting in an optimization objective that is:
[(0)
E Bo
Poo Se Notee (+19)
G
10i1
min (rae(0)Ane, clip clip( (0), ee )A.1)
(I)
0d
# oi lo, is correct]] <G
s.l.
where
(0) To q. 0,,ct)
A R, mean ( (R))
Tow (oit oc,ct)
std (R, 1
Our reward function R mirrors that of DeepScaleR (Luo al.) 2025b), employing an Outcome Reward Model. It returns 1 if and only if both the answer and format are correct; otherwise, it returns 0. In summary, our reward function yields:
1, if the answer (e.g. passes basic LaTee//Smmpy checks) and format (e.g. exists <think> and </think>) are both correct, otherwise.
R
(3)
B.2 Low- VARIANCE PASS @ ESTIMATION
Pass @ k is a measure of a model's problem solving ability. indicating the probability that the model generate least one correct solution in & attempts. Specifically, for each problem æ; in the evaluation dataset D, we generate n samples (where n 2 k) and count the correct ones as c. The direct calculation formula is:
pass@k : BRy 1 (1-11- )
However, this formula has excessive variance and insufficient accuracy. To solve this problem, adopt the unbiased estimation method proposed by Chen al. (Chen al. 202I), using the unbiased estimator of pass @ k over the dataset:
pass@k == EN
(5)
In our experiments, ensure sufficient accuracy, we set such that 2k S n, which helps further reduce the variance the estimate.
B.3 MORE RELATED WORKS
Recent studies show that RL algorithms, such as PPO (Schulman et al. 2017) and GRPO (Guo et al. 2025), can greatly enhance model reasoning capabilities. Building on this, several works have refined this paradigm from different perspectives. One method can be adjusting the reward function. Some studies (Zhu al.) 20255 Shao et al. 2025) directly modify the reward function to improve training efficiency. Other methods introduced intermediate process rewards (Wang et al. 2024 Malik et al. 2025). while Wen et al. (Wen et al. 2025) set separate correctness judgment for CoT to obtain rewards.
14

---

Published as a conference paper at ICLR 2026
Another novel perspective aims to improve sample efficiency by measuring certainty. For example, TreeRL (Hou al., 2025) and VinePPO (Kazemnejad
al. (025) enhanced sample effects by ), on the other hand, reused partial trajectories
introducing entropy or confidence. MRT (Qu et al. 2025), during testing to boost sample efficiency. R3 (Xi al. (024) improves RL sample efficiency by decomposing human solution steps and providing preceding steps to guide the model in completing subsequent ones. Further, some research adopted a multi-stage training or reasoning mode, exploring from different angles such training length (Luo et al. (025b), question difficulty (Parashar et al. 2023). and Exed-Length summaries during reasoning (Yan et al. (025)..
Yue et al. 2025 Zhao al. 2025) has investigated how reinforcement learning affects the frontier of model capabilities, observing a decay in pass @k when k becomes large. In response to this phenomenon, some works (Yu et al., 2025; Liu et al.) 2025: An et al., 2025) maintained entropy stability by adjusting training entropy through methods such as increasing the clipping upper bound, enlarging the temperature coefficient, extending the training length, and periodically updating the KI. reference model. StepHint (Zhang et al. (2025) also preserved entropy stability by leveraging intermediate thinking content of iterative length as a prompting signal.
In addition to designing better algorithms, another line of research (Shao et al. 2024
In contrast to the aforementioned research, our work adopts an orthogonal approach by using part of the ground-truth solution as hint, without requiring any modifications to the existing reinforcement learning infrastructure. We provide both theoretical justification and empirical evidence that this strategy maintains pass @k without compromising the exploratory capacity of the underlying reinforcement learning algorithm.
B.4 BENCHMARKS
We evaluate the models' breadth across various tasks in multiple domains, including mathematics, coding, reasoning, and logical inference. For mathematics, we follow DeepScaleR (Luo et al.] (025b) and Nemotron (Moshkov et al. (025), and conduct assessments on more challenging mathematical datasets such as AIME2024 (MAA 2024), AIME2025 (MAA) 2025). ) Olympiad Bench (He et al. 2024), HMMT FEB 25 (hmm) 2025). and BRUMO25 (bru] 2025). ). Specifically, HMMT25 Feb and BRUMO25 are both sourced from MathArena (Balunović et al. 2025). In the realm of coding, we utilize commonly employed datasets, including Code Contests Li et al. 2022). Codeforces" and LCB V5 002410-202509 (Jain et al. [2024). For logical reasoning tasks, we assess our models' capabilities using GPQA Diamond (Rein et al.) (023) and Zebraliogic (Lin et al. 2025 ) The
benchmarks related to coding and logical reasoning are all referenced from AReaL (Iu et al. 2025).
B.5 TRAINING DATASET
The dataset employed in our study is 1ieeRR-MMth--201 (Open-RI Team 2025).
Prior to com- mencing the training of the Partial Solution, we conducted a preliminar screening of the dataset. Specifically, we utilized the inference operations on each of the 220k data entries in the OpenRI dataset. Subsequently, we compared the annotated answers in the OpenR1 dataset with the results generated from each inference to tally the number of correct instances for each data entry. Ultimately, we selected the data entries with 0 or correct instance as the training samples for our study. The final dataset size is 26K.
-eepSeee-RR-Dssilll..
(Guo et al. 2025) model to perform eight
For controlled comparisons, we further split this 26K subset by remsami Nemotron-1.5B eight times per item and counting correct completions. We define Easy Data as questions with correct 5B. Similarly,
counts in [7, 8] and train model on this split, denoted Easy-Nemotron define Hard Data as questions with correct counts in [0, 1] and train Hard Harrd-Nera Nemot 11 5B on this split.
Additionally, for the augmented data, we perform eight inference passes using the model currently under training. We then select samples for which the number of correct predictions falls within the range of [o, 4]. This criterion is motivated by the finding that samples exhibiting higher variance are more beneficial for training (Gao et al. 2025 Wang et al.) 2025b). The range [0, 4) is chosen because it includes the point of maximum sample variance, which is achieved with four correct predictions out of eight trials. For convenience, we refer to augmented data with partial ratio p as Partial-p data.
https://cndefoores.comm
In the GPQA Diamond dataset, multiple-choice questions are presented in the form of options rather than directly providing the answer, requiring the model output only A, B, or D.
15

---

Published as a conference paper at ICLR 2026
B.6 THE RATIONALE FOR THE CHOICE OF
Table 5: Number of problems vs pass rate under different hint levels on OpenMath-Nemotron- Setth--emotto----.. before training. We evaluated DeenMath-Nemotton---.. on the OpenRI dataset after the first round of filtering, with each problem assessed times. The table illustrates the distribution of correct answers (n) where E {0, .8..
Hint Levels
0/8
1/ 1/8
2 8
3 //8
4/8 8
5/8
6/8 8
7/8
8 / 8
Partial-50
143
224
304
472
710
1013
1779
3655
17741
Partial-25
3155
1997
1814
1785
1902
2175
2614
3440
7159
Partial-10
3589
2090
1865
1842
1905
2176
2653
3415
6506
Partial-O
3812
2218
1854
1842
2007
2136
2517
3264
6391
In this study, we evaluated the performance of DpeemMatt-Nemotron--e on the OpenRI dataset under various hint levels. The evaluation performed after the first round of filterin,, and each problem was assessed 8 times to capture the predictive distribution. The resulting table (Table 5) shows the distribution of correct answers across different hint levels, where the values represent the number times the model answered correctly (n 1,, .,,.,,88.).
The selection of the hint parameter p was primarily based on these evaluation results. As shown table, the performance with a Partial-50 hint significantly reduces task difficulty, as evidenced by the high pass rates across similar to that the no-hint scenario (Partial-O), with only marginal differences in task difficulty.
levels. In contrast, Partial-25 (25% hint) exhibits performance pattern
This minimal difference in difficulty between Partial-O and Partial-25 suggests that training with Partial-25 does provide substantial gains compared Partial-O. Consequently, adopted stepwise design which the hint level is first set to = 50%, followed by = 25%, to evaluate the model's performance under varying conditions.
B.7 EVALUATION SETUP
We configured the models have maximum generation length of 32.768 tokens. In line with DeepSeek- R1 Guo et al. 2025). we utilized pass@ evaluation (Chen et al. 2021), with the formula detailed in B.2 We reported pass @ using non-zero temperature. Specifically, we used sampling temperature of 0.7 and a top-p value of 0.95 to generate k responses per question, typically set 32, with deviations explicitly noted. Particular attention should be paid to the fact that, although we incorporated partial Solution during training, it not included in the evaluation phase.
B.8 DETAIL. ON PROMPT TEMPLATE
DeepScaleR Coding's Inference:
User <inputt]< Assistant ><think>>
DeepScaleR Others' Inference:
User >(input)
Please reason step by step, and put your final answer within \boxed [.<
Assistant
><think>
Nemotrion Coding's Inference:
dim__sart>>user
(input)
<lim endeeend>>eo <lim < atstttas
<think>
Nemotrion Others' Inference:
<lim startbsssteem
Please reason step by step, and put your final answer within \boxed J.<lim_ends <lim start/>user
16

---

Published as a conference paper at ICLR 2026
(input) (eantt><kimenddii <lim_startl>asse startbassistant
Training prompt with partial solutions (math RL): (Problem)
## Hint: (Partial Solution)
Please reason step by step, and put your final answer within \boxed(): ).
C THEORY
C.I PROOFS
Theorem 4.4 (Lower Bound on RL. Learnability under Solution Inaccessibility). Given a probability threshold 6, > 0, for every question E Q the model capacity set Clq, 6p) does not intersect with the solution set (g)), i.e.
Clq. 6lg. =
E
then under Assumption when training for I steps with B samples per step such that TB (1/6))), there a constant probability that the RL algorithm will not update the model.
Proof. Let Psol = Er Pa(T"(q) denote the cumulative generation probability of any solution trajectory. By C(q n ((g)) 0 and Def 4.2
Σ
Pa(T"10) < 6p
T'ES(a)
For N
TB independent samples across T steps with batch size B, the probability of complete failure (no solution sampled) is:
P(failure) = Psol) > (1 - (1-8p)
Given TB = ((1/6p), we have:
(1 (1- (1 (1- -1-6))011))) =()).
x)) for E (0, By The last inequality follows from the fact that (1 2//2 / exp(-1/(1- Assumption 4.3 no solution is found, the model weights remain unchanged.
Lemma C.1 (Upper Bound on Sampling Budget for Solution Given Hint). Given a question q Q if
there exists hint by for the question q (Def. 4.5, then if we perform TB = ( (o)) i.i.d sampling over the initial model conditioned on (q, hn). we can find valid solution with constant probability.
Proof. By Definition 4.5 we know:
1. (h(la))) 2
2. 3s, E S(g) P, (, 2%
With N TB 2 10/5, independent samples conditioned on (g, hy), the probability of not finding the solution 8,
Pnnno solution) = (s < - -1))))))000 20/00/8, S sexp(-10) < 0.01.
Therefore, P(finding solution) 0.99.
Theorem 4.6 (Informal Upper Bound on RL Learnability with Hint). If we have hint hy for every question E Q (Def. 4.5). then there exists an RL algorithm that can output a policy na such that Uupoooifo 2) ( (7 S(g))) 2 0.99 with O(1 ((1/5))) sampling budget with high probability.
17

---

Published as a conference paper at ICLR 2026
This theorem is direct corollary of the Theorem 5 regarding the bandit setup in (Mei et al. 2022). Because the setup here is relatively simple, we also present a detailed proof for this special case here. We first formalize our setup as follows:
Assumption C.2 (Tabular RL with Hint). We consider the tabular RL setting with softmax policy parameterization. There exists a finite of possible questions and finite set of possible solutions S. For each question E 2, there exists a hint hg. which is subset of solutions hg S S.
The policy is parameterized by |S| x [Q] matrix 0 in the following way:
exp(Ox.g) (o(s)g) Eves exp(0st
Here the setup is different than the autoregressive setting in our experiments and simplify the model to tabular setup for the simplicity of analysis. We now restate the assumption the existence of hint this setup.
Assumption C.3 (Hint Existence, Formal Version of Definition 4.5). For each question g E Q, there exists a hint hú C S such that EeEn, (slg) 2 8. Further, there exists solution 9 E S such that (sygg))) 2 8, E,E,, (s(g))
RL Algorithm: We will first sample (11)6)) action based on the policy 140 conditioned on the question g and the hint h2. Then we will do one-step policy gradient update on our policy. Noted that here we can reach high reward within one step because the reward function is deterministic.
Theorem C.4 (Formal Version of Theorem 4.6). Under Assumption C.2 Assumption C.3 running 1 steps of policy gradient update with sampling budgel (116%)) the learned policy achieves:
Bu--Uniform(O) @) (= (T ((())) 2 0.99
with probability 0.99.
Proof. First, by Assumption C3 for any question q, have:
Σ
(slgg) 2 6p and 3s, : P 2 5 Σ P((s
A
sEhq
With sampling budget N @(2))))), by Lemma C.1 and the union bound, will find solution Sy for every question y with probability at least 0.99. Suppose the found set of solutions for question
q is Sy and all sampled solutions are s1)) s(N), Then because
Ve log (o(gg) = =e, Σ po(s'lg)e
S'SS
We have the policy gradient being
N
PG..
=
((((0)
E S log logHo(s!000)
N
i=1
N
(s((0
E S - - (o(s'(g)eo)
N
f=1
s'ES
N
N
Σ
(((0)) E Sylestn
Σ 1=1
(1((0) E Sy1
[
[um(slla)es (l)ee)))))
N
We can make two simple observations:
1. For every s E Sg. PG..g <
2. There exists a 8" E S, such that PG.-. 0.
Therefore, consider the updated parameters
8' 0s,9 1/PG..g
If in is large enough, we know that Eees, Pry q) 0.99. This completes the proof.
18

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 ABLATION STUDY WITHOUT HINT
Table 6: Ablation without hint Nemotron-1. Pass@1 (avg@32) challenging maths benchmarks. "QuESTA-Nemotrona 1.5o w/o hint" trains RL on the same data but removes hints from the prompt, while "w/ hint" uses partial-solution hints during training. With hints, the model improves all benchmarks and achieves +2.82 average gain over w/o hint (63.26 vs. 60.44), on top of the improvements over the base model. The one using hint requires nearly half the number of steps compared to the one not using hint to achieve the same performance.
Model
AIME24
AIME25
HMMT FEB 25
Olympiad Bench
BRUMO25
Avg
Nemotron-1.52
61.77
49.50
31.56
64.62
58.23
53.14 14
QUESTA--Nematron w/o hint (2K step)
69.48
39.79
38.85
68.05
66.04
60.44
QUESTA Nemotron 1.5B hint (1.IK step)
69.27
60.00
37.92
69.72
68.33
61.05
QUESTA Nemotron-1..55 w/ hiat (2K slep)
72.50
62.29
41.67
70.36
69.48
63.26
Nemotron-1.33 1.5B
QuestA--Nemotron-..388 w/o hint
QuestA-Nemootoon-l w/ hint
AIME24
AIME25
100
90
Pass Accuracy (%)
90
Pass Accuracy (%)
80
80
70
70
60
50
10
10
10
10
10
10
10a
10"
Sample Size (log scale)
Sample Size (log scale)
HMMT Feb 2025
BRUMO 2025
90
100
80
Pass@a Accuracy (%)
Pass@ Accuracy (%)
90
70-
60
80
50
70
40
60
30
10
10
10
10
10
10
10a
10
Sample Size (log scale)
Sample Size (log scale)
Figure 7: @ass@k & comparison on Nemot ron-1. 55B for RI. with vs. without hints. Training with hints consistently dominates across At and avoids the performance drop at larger k seen in standard RL. Hints are used only during training; evaluation uses no hints.
We ablate the role of hints by training RI. on Nemot ron-1 1 5B with and without partial-solution hints. Here, CLESTTA-Nemotron 1.5B w/o hint denotes RL on the same data and schedule but with the hint removed from the prompt; DUFFSAA-Nemotron. 1.5B w/ hint uses identical settings except that the partial solution is provided as hint during training. As summarized in Table 66 removing the hint still improves over the base model (average Pass 1: 53. 14 60.44), but adding the hint yields
19

---

Published as a conference paper at ICLR 2026
further +2.82 average gain (60.44 63.26), with consistent improvements across all five benchmarks. The one using hint requires nearly half the number steps compared the one not using hint to achieve the same performance. Note that hints are used only during training; all evaluations are conducted without hints.
Figure compares Pass@k k curves. The w/ hint model lifts the entire curve across and avoids the degradation at larger commonly observed in standard RL, while the w/o hint variant brings smaller gains that taper off as k increases. A possible reason for this phenomenon is that, without hints, extremely difficult problems remain unlearned. Consequently, during reinforcement learning training, the model prioritizes improving performance easier as training progresses. This leads the model become overly confident, thereby reducing its Pass k metric. In contrast, when hints are provided, the model still prioritizes learning more difficult problems this is because such problems can provide effective learning signals.
problems that have become relatively
D.2 ABLATION STUDY WITH DIFFERENT MODELS
Table 7: Performance comparison DeepSca eeR-1 5B: Pass (avg @32) across maths bench- marks. QUESTA consistently improves all tasks and raises the average by +6.50 points.
Model
AIME24
AIME25
HMMT FEB 25
Olympiad Bench
BRUMO25
Avg
SeepScaleR-1..B
40.42
31.35
19.27
52.97
37.40
36,28
QUBSTA-DeepScaleRR 1.5B
49.16
35.94
21.77
58.69
48.33
42.78
Table 8: Performance comparison (Pass@) 1, averaged over 32 samples) showing the impact of QUESTA across benchmarks in other domains, including general knowledge, logic, and coding tasks. We observe minor dross-domain generalization on all these benchmarks, despite QUESTA being applied exclusively in the maths domain.
Model
GPQA Diamond
Zebralogic
Code Conrest All
Codeforces
LCB V5 202410-202502
Avg
DeepScaleR-1.9B
38.5
14.26
9.07
8.79
19.57
18.04
QUISTA--eepScallk-I..55
39.2
14.98
10.1
8.9
20.9
18.82
DeepScaleR--150
QuestA-DeepSalee--1.55
AIME24
AIME25
80
80
Pass@ikk Accuracy (%)
Pass@ik Accuracy (%)
70
70
60
60
50
40
30
10
10
10a
10
10
10
10
10
Sample Size (log scale)
Sample Size (log scale)
Figure 8: Pass@ k on DeepSca 1eR-1 1.00-11.55.5. 5B: QUESTA raises the entire curve across and avoids the large-k drop often seen in standard RL. The increasing gap with indicates improved sample diversity rather than overconfident collapse. No hints are used at evaluation.
We next examine model-family transfer by applying QUESTA to DeepScaleR-1. 5B. We train for 750 steps DeepSca e Stage Luo al. 2025b) on the QUESTA first stage.
20

---

Published as a conference paper at ICLR 2026
DeepScaleR-1.555 1.55B (Pro-RL)
8000
(Post-RL)
Number of Problems
Pre-RL: 26041
problems, Mean Pass Rate: 0.500
6000
Post-RL: 26041
problems, Mean Pass Rate: 0,599
4000
2000
0.000
0.125
0,250
0.375
0.500
0.625
0.750
0.875
1.000
Pass Rate
Figure 9: We conducted training of the DeepScaleR model employing the QUESTA with a dataset comprising 26,000 questions. The average pass The initial graph represents the scenario without the incorporation of partial solutions, while the subsequent graph depicts the situation where partial solutions were included. 'The application of QUESTA significantly diminishes the incidence of unsolved or partially addressed problems within the training dataset. Concurrently, it has come to our attention that our previous method of data curation not entirely accurate; in fact, the amount of data providing meaningful training signals less abundant than anticipated, suggesting the potential for further refinement of the dataset.
calculated from sample of 8 instances.
As shown in Table 7 QUESTA--DeepScaleR improves every maths benchmark over the base model, achieved an average improvement 6%, indicating that the benefits QUESTA are not tied to single architecture.
CREESTA-eeepScalee lifts the entire @ass@A: curve across & and avoids the degradation larger reported in standard RL settings. The widening gap at larger l suggests improved candidate diversity rather than overfitting to single trajectory, consistent with our general pass @ lc analysis in Appendix B.2] Complementing this, Figure shows that on the 26K training set (evaluated without hints), mass shifts away from the 0/8-1/8 bins toward higher pass rates, reducing unsolved or partially solved cases. Hints are used only during training and removed evaluation time.
Pass @k behavior mirrors these gains. In Figure
Beyond maths, Table reports out-of of-iiiiributtion (OOD) results on general knowledge, logic, and coding.
1.55B achieves small but consistent gains (Avg: 18.04->18.85 +0.78), suggesting that the improved reasoning patterns transfer modestly beyond the training domain.
D.3 SUPPLEMENTAL EXPERIMENT DETAIL
QuestA-Nee Poeest--eemoroo---i 1.5B w/o hint
Average Reward
Average Length
Average Entropy
18000
Average Reward
0.5
Average Length
Average Entropy
0.575
16000
0.4
0.550
14000
0.3
0.525
0.2
12000
1000
2000
1000
2000
Training Step
Training Step
Training Step
Figure 10: Training dynamics of QUESTAA-Nemotron-1555 w/o hint.
21

---

Published as a conference paper at ICLR 2026
MwestA-Nemotroon-ti 1.5--00
Average Reward
Average Length
Average Entropy
0.7
18000
0.65
Average Reward
Avenage Length
Average Entropy
0.6
16000
0.60
0.5
14000
0.4
0.55
1000
1000
2000
2000
Training Step
Training Step
Training Step
Figure 11: Training dynamics of QUESTA-Nemooton-I..BB
QusstA-Nemmtroo--11-----22
Average Reward
Average Length
Average Entropy
0.535
0.40
Average Reward
Average Length
Average Lintropy
19000
0.530
0.38
Mm
18500
0.525
0.36
0.520
250
500
250
500
250
500
Training Step
Training Step
Training Step
Figure 12: Training dynamics of ULESSTA-Nmmttro--155------22222-
Fasy-Nemotron--1 1.55B
Average Reward
Average Length
Average Entropy
Average Reward
0.85
Avcrage Length
14000
Average Entropy
0.550
0.525
12000
0.80
0.500
10000
0.475
500
1000
500
1000
500
1000
Training Step
Training Step
Training Step
Figure 13: Training dynamics of Fasy-Nemotron-11.55
Thard-Neemttoonl..35 1.5B
Average Reward
Average Length
Average Entropy
18000
0.400
of Ma
Average Reward
Average Length
Average Entropy
0.64
17000
0.375
0.62
16000
0.350
0.60
15000
0.325
500
1000
500
1000
S00
1000
Training Step
Training Step
Training Step
Figure 14: Training dynamics of Hard -eeadd-eeeto--.555
22

---

Published as a conference paper at ICLR 2026
MuustA-DeeSScalc-II.55
Average Reward
Average Length
Average Entropy
0.70
0.7-
Averago Reward
Average Length
8000
Average Entropy
0.65
0.6
7000
0.60
6000
0.55
0.4
5000
500
0
500
500 Training Step
Training Step
Training Stop
Figure 15:
The training dynamics of QUESTA- DeepScaleR- The first and second charts show the changes in the average reward and average inference length of the rollout samples that include all incorrect/correct ones, respectively. The third chart shows the average entropy excluding all incorrect/correct rollout samples.
BuestA-Neemotron-1.117 (By OpenMatthReessoinng)
Average Reward
Average Length
Average Entropy
0.55
Average Reward
0.85
Average Length
12000
Average Entropy
0.80
0.50
10000
0.75
0.45
500 Training Step
500 Training Step
Training Step
Figure 16: Training dynamics of CUUSSTA-Nemotrn--L. on OpenMathheesonnnng MMoshkov et al. 2025). The first and second charts show the changes in the average reward and average inference length of the rollout samples that include all incorrect/correct ones, respectively. The third chart shows the average entropy excluding all incorrect/correct rollout samples. Dynamies closely mirror those on OpenR 1 Mat,-2200, with no entropy collapse.
23

---

Published as a conference paper at ICLR 2026
Semotron-1.5B
Basy-Nemotrrn--1.. 1..5
QuestA-Namotron-I..5 w/o hint
Haard-Nemotron-1.55
QuestA-Nemotron-I.5B
AIME24
AIME25
100
90
Pass @ak Accuracy (%)
90
Pass @kk Accuracy (%)
80
80
70
60
50
10
10t
10
10
10
10a
Sample Size (log scale)
Sample Size (log scale)
HMMT Feb 2025
BRUMO 2025
90
100
80
Pass@ak Accuracy (%)
Pussi@k Accuracy (%)
90
70
60
80
50
70
40
60
30
10
10
10
10
10a
10
Sample Size (log scale)
Sample Size (log scale)
Figure 17: We compare pass@k curves RLVR-trained models with and without QUESTA.
24

---

Published as a conference paper at ICLR 2026
Ward-Nemotron-L..3
QuestA-Nemotrn-I1..
AIME24
AIME25
75
65
Pass@ Accuracy (%)
Accuracy (%)
60
70
5$
65
Pass(())
50
60
45
200
400
600
800
1000
0
200
400
600
800
1000
Training Steps
Training Steps
HMMT Feb 2025
BRUMO 2025
42.5
40.0
Pass@! Accuracy (%)
Pass(il Accuracy (%)
37.5
65
35.0
32.5
60
30.0
55
27.5
200
400
600
800
1000
200
400
600
800
1000
Training Steps
Training Steps
Figure 18:
Comparison of RL training dynamics: Training with only hard problems (green) makes progress very slowly due to sparse rewards, while our method with partial solutions (orange) acceler- ates learning and consistently achieves higher accuracy across training steps.
25


Reviews:
[{"review_id": "VzQ7ZMwT5g", "reviewer": "Reviewer_oEsW", "summary": "This paper introduces QUESTA, a simple yet effective data-centric strategy to enhance the reasoning capabilities of LLMs through reinforcement learning. The paper identifies a critical trade-off in RL training for reasoning: while easy problems can cause a decline in reasoning diversity (pass@k), training on hard problems is inefficient due to sparse reward signals. The proposed method is introduced to navigate this trade-off, aiming to mitigate this inefficiency and effectively expand the model's reasoning capacity. Experiments show the method enables 1.5B-parameter models to achieve better results on challenging math benchmarks, in some cases surpassing a much larger 32B model. \n\nA key limitation is the method's reliance on an empirically-tuned curriculum. This hand-crafted 50-25 schedule presents challenges for the method's transferability to new settings. Furthermore, its effectiveness on larger-scale models remains unverified, as such models may not benefit from this specific scaffolding strategy.", "questions": "The curriculum for the hint percentage (from 50% to 25%) is a key component. Could you elaborate on how this schedule was chosen and have you considered alternatives, such as a more gradual decay or an adaptive schedule based on model performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly identifies and demonstrates the critical trade-off between training on easy versus hard prompts in RL, providing a strong motivation for the proposed method.\n\n- The paper introduces an elegant approach that does not require complex changes to the underlying model architecture or RL algorithm. The idea of using partial solutions as hints is intuitive and proves to be effective.\n\n- The method yields consistent performance improvements. The 1.5B model trained with QUESTA shows a clear gain over the standard RL baseline and, on the AIME25 benchmark, even surpasses a model over 20 times larger, highlighting the efficiency of the approach.\n\n- The paper includes a theoretical analysis that formalizes why augmenting questions with partial solutions can improve the sample efficiency of RL, adding depth and rigor to the empirical findings.", "weaknesses": "- The experiments are limited to 1.5B models. It is unclear if the method would provide similar gains on larger models that already possess stronger reasoning capabilities.\n\n- The curriculum for providing hints (50-25) appears to be a key component of the method's success. The ablation study only compares this strategy against a fixed 50% hint, which is insufficient to understand the sensitivity of the model's performance to other potential curriculum designs or hyperparameter choices.\n\n- \"We apply augmentation using the solution block rather than the reasoning chain-of-thought\"; this is a design choice presented without any rationale or comparative experiments to justify this decision. It is plausible that hints derived from the CoT could be more effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAyhyxwBQ9", "reviewer": "Reviewer_iEgk", "summary": "A simple yet effective strategy via Question Augmentation is introduced for  partial solutions during training to reduce problem difficulty and provide more informative learning signals.", "questions": "1.It's a good data augmentation for advance the reasoning during RL training.\n2.It is more experiments on other RL algorithm.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Improves pass@1 but also pass@k—particularly on problems where standard RL struggles to make progress.\n2.Achieve new state-of-the-art results on math benchmarks using 1.5B-parameter models: 72.50% (+10.73%) on AIME24, 62.29% (+12.79%) on AIME25, and 41.67% (+10.11%) on HMMT25.\n3.There is reasonable proof for the the proposed theory .", "weaknesses": "1.The benchmark is better if code dataset is evaluated.\n2.The algorithm of RL is not advanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ji96My11xP", "reviewer": "Reviewer_4ed7", "summary": "This paper explores improving reasoning in large language models (LLMs) using reinforcement learning (RL). It highlights limitations of RL in enhancing reasoning beyond the base model and proposes a novel strategy called Question Augmentation (QuestA). QuestA introduces partial solutions during training to simplify problems and provide better learning signals. Applied to math reasoning tasks, QuestA significantly boosts performance metrics like pass@1 and pass@k, especially on challenging problems. The method achieves state-of-the-art results on math benchmarks using 1.5B-parameter models.", "questions": "refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1,The authors demonstrate QuestA's effectiveness through rigorous experiments on math reasoning benchmarks, achieving state-of-the-art results with notable performance gains on metrics like pass@1 and pass@k.\n2. The open resources makes it a valuable contribution to the field of LLMs, particularly in reasoning-intensive domains like mathematics.", "weaknesses": "1. The experiments are conducted on 1.5B-parameter models, which may not generalize to larger models. The scalability and adaptability of QuestA across different model sizes remain unclear.\n2. The relationship between data difficulty and performance was mentioned in earlier papers on mathematics long ago, such as [1]. However, this paper does not discuss. And, in this subproblem, it is also an obvious insight, somewhat lacking innovation.\n\n[1] WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "irU2Io1Dhw", "reviewer": "Reviewer_2jPJ", "summary": "This paper introduces QuestA, a data-centric augmentation technique designed to enhance reasoning performance during RLVR by injecting partial solution hints into training prompts. The approach is conceptually simple yet theoretically grounded, with analyses explaining why partial-solution augmentation is effective. Experimental results demonstrate its strong performance and incorporating partial hints not only accelerates learning but also mitigates the entropy collapse.", "questions": "1. Could the authors elaborate on practical limitations—such as applicability to problems without step-wise gold solutions or the impact of poor-quality hints? Is there empirical evidence of robustness to incorrect or misleading hints?  \n2. How sensitive is QuestA to the form of hints used (e.g., solution block, chain-of-thought, or intermediate step)?  \n3. How does the choice of $p$ trade off between training speed and final performance? Can the method adaptively tune this during RL?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple yet effective, and achieves new SOTA pass@1 results on challenging math benchmarks for 1.5B-parameter models.\n2. The method is thoroughly evaluated through comprehensive ablation studies and extensive analyses across multiple datasets, model architectures, and training curricula, consistently demonstrating performance improvements across diverse settings.", "weaknesses": "1. The method’s reliance on high-quality, step-wise solutions for augmentation raises concerns about scalability to domains lacking such curated data. For instance, can QuestA generalize to real-world science Q&A or open-domain reasoning tasks where solution steps are unavailable?  \n2.  The paper would benefit from a qualitative error analysis. There is limited discussion of why the “Partial-0” (no hints) setting yields no improvement, or why certain tasks exhibit smaller gains.\n3.   In my personal opinion, although the paper offers a theoretical perspective on why partial-solution augmentation improves RL efficiency, this section feels somewhat unnecessary and potentially confusing. The core idea is already straightforward and intuitive, and adding theoretical formalism may detract from its practical clarity—especially since the method can be naturally interpreted as a form of prompt optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)