Reinforcement Learning for Reasoning in Large Language Models with One Training Example
Authors: Yiping Wang
Qing Yang
Zhiyuan Zeng
Liliang Ren
Liyuan Liu
Baolin Peng
Hao Cheng
Xuehai He
Kuan Wang
Jianfeng Gao
Weizhu Chen
Shuohang Wang
Simon Shaolei Du
yelong shen
Venue: NeurIPS 2025 poster ()
Keywords: RLVR
math reasoning
reinforcement learning
post-training
language models
Primary area: deep_learning
TL;DR: We only need one example for RLVR on LLMs to achieve significant improvement on math tasks

Abstract: We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0\% to 73.6\% (8.6\% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6\% to 35.7\% (7.0\% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6\%, average: 35.9\%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8\%, average: 36.6\%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. 
In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term \textit{post-saturation generalization}.
Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon.
We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training.
We also further discuss related observations about format correction, label robustness and prompt modification.
These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. 
Our code, models, and data are open source at https://github.com/ypwang61/One-Shot-RLVR.

Reinforcement Learning for Reasoning in Large Language Models with One Training Example
Yiping Wang
Qing Yang*
Zhiyuan Zeng'
Liliang Ren
Liyuan Liu
Baolin Peng
Hao Cheng
Xuehai He*
Kuan Wang
Jianfeng Gao
Weizhu Chen
Shuohang Wang
Simon Shaolei Du
Yelong Shen
University of Washington
University of Southern California
3 Microsoft
University of California, Santa Cruz
Georgia Institute of Technology
Abstract
We show that reinforcement learning with verifiable reward using training example (I-shol RLVR) is effective in incentivizing the mathematical reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Owen2.5-Math-1.55 we identify single example that elevates model performance on MATH500 from 36.0% to 73.6% (8.6 improvement beyond format correc- tion), and improves the average performance across six common mathematical reasoning benchmarks from 17.6% 35.7% (7.0% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6% average: 35.9%), which contains the aforementioned example. Furthermore, RLVR with only examples even slightly exceeds these results (MATH500: 74.8%, average: 36.6%). Similar substantial improvements are observed across various models (Qwen2.5-Math. MB, Llama3.22 Instructt DeepSeek-Pe -1-eeeek----DD- -D-istlll- Qwen-1.5BB) RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post- past-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., 1g. by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. Our code, models, and data are open source at whttt:://// /ttps://git http://ggttuuu...oooparr/i/// www.op/wwg61///////o -ttte//nnneennnnnnntnee RLVR
Introduction
Recently, significant progress has been achieved in enhancing the reasoning capabilities of large language models (LLMs), including OpenAl-ol [T], DeepSeck-RI 2 and Kimi-1.5 [3, particularly for complex mathematical tasks. A key method contributing to these advancements is Reinforcement Learning with Verifiable Reward (RLVR) 4 5 2 31 which commonly employs reinforcement learn ing on an L.M with a rule-based outcome reward, such as a binary reward indicating the correctness
: This work was done during Yiping's internship at Mierosoft. t: Corresponding authors. dence email: (ypwang61, ssdu]@cs. .aauu)@sss.assiitttooo...o washington, edu, (shuowa, yeshe) omrosoff. com
Correspon-
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

MATH 500 (avg@1)
Average on 6 benchmarks
37.5
35.0
70
32.5
Accuracy (56
65
30.0
55
1.2k
27.5
1.2k DSR
7.5k MATH train
25.0
7.5k MATH train set
2 shot (m, ma)
shot (TL. fus?
45
1 shot (Riz)
22.5
(fta)
40
1 shot (m) )
20.0
shot (mr)
format reward
formalt reward
17.5
250
500
750
1000 Steps
1250
1500
1750 2000
250
500
750
1000 Steps
1250
1500 1750 2000
Figure 1: RLVR with 1 example (green) can perform as well as using datasets with thousands of examples (blue). Left/Right corresponds to MATH500/Average performance on mathematical rea- soning benchmarks (MATH500, AIME24, AMC23, Minerva Math, OlympiadBench, and AIME25). Base model is Qwen2.5-Mait
1.5B. πj and TI3 examples defined by Eqn. 2 and detailed in Tab. 2 and they are from the 1.2k DeepScalerR subset (DSR-sub). Setup details are in Sec. 3.1 We
find that RIVR with example {Taa } (35.7%) performs close to that with 1.2k DSR-sub (35.9%), and RLVR with 2 examples {TT, 1i3 (36.6%) even performs better than RLVR with DSR-sub and as well as using 7.5k MATH train dataset (36.7%). Format reward (gold) (Appendix C.2.3) serves as a baseline for format correction. Detailed results are in Appendix C.I.1 Additional results for mon-mathematiall reasoning tasks are in Tab.
of the model's final answer to a math problem. Several intriguing empirical phenomena have been observed in RLVR, such as the stimulation or enhancement of specific cognitive behaviors [6] (e.g.. self-reflection) and improved generalization across various downstream tasks (5)
Currently, substantial efforts are directed toward refining RL algorithms (e.g.. PPO and GRPO [ to further enhance RLVR's ' performance and stability [9H6]]. Conversely, data-centric aspects of RLVR remain relatively underexplored. Although several studies attempt to curate high-quality mathematical reasoning datasets [87 [1], there is relatively limited exploration into the specific role of data in RLVR. Thus, critical questions remain open: How much data is truly necessary? What data is most effective? How do the quality and quantity of the training data relate to observed empirical phenomena (e.g., self-reflection and robust generalization)? The most relevant study to these problems is LIMR [19]. which proposed a metric called learning impact measurement (LIM) to evaluate the effectiveness of training examples. Using the LIM score, they maintain model performance while reducing the number of training examples by sixfold. However, this study does not explore how aggressively the RLVR training dataset can be reduced. Motivated by these considerations, in this paper, we specifically investigate the following research question:
"To what extent can we reduce the training dataset for RLVR while maintaining comparable perfor- mance compared to using the full dataset?"
We empirically demonstrate that, surprisingly, the training dataset for RLVR can be reduced to as little as ONE example! This finding supports recent claims that base models already possess significant reasoning capabilities 13 20 6 21, and further shows that a single example is sufficient to substantially enhance the base model's 's mathematical performance. We refer to this setup as I-shot RLVR. We summarize our contributions and findings below:
We find that selecting one specific example as the training dataset can achieve similar downstream performance to that of the 1.2k DeepScaleR subset (DSR-sub) containing that example. Specifically, this improves the Qwen2.5 -atth----thh-.. 1.5B model from 36.0% to 73.6% on MATH500, and from 17.6% to 35.7% on average across 6 mathematical reasoning benchmarks, including non-trivial improvements beyond format correction (Fig. D Notably, these two examples are relatively easy for the base model, which can solve them with high probability without any training (Sec. (.2.1]. Additionally. 1-shot RLVR on math examples can improve model performance full-set RLVR (Tab. T.
mon-mathematical reasoning tasks, even outperforming
We confirm the effectiveness of 1(few)-shot RLVR across different base models (Qwen2.5- Math-1.5/781 Llama3.2- 3-3....------u--eeee models distilled from long and different RL algorithms (GRPO, PPO).
(CoT)
data DrepSeee-RI--isstll--eeeee
We highlight an intriguing phenomenon in 1-shot RLVR: post-saturation generalization. Specifically, the training accuracy on the single example rapidly approaches 100%, yet the model's test accuracy continues to improve. Moreover, despite using only one training

---

example, overfitting does not occur until after approximately 1.4k training steps. Even post-overfitting, while the model's reasoning outputs for the training example become incomprehensible multilingual gibberish mixed with correct solutions, its test performance remains strong, and the reasoning outputs for the test examples remain human-interpretable.
In addition, we demonstrate the following phenomena: (I) 1-shot RLVR viable for many examples in the full dataset when each example is individually used for training. We also discuss its connection with format correction in Appendix C.2.3 (2) 1-shot RLVR
enables coossry generalization: training on a single example from one category (e.g.. Geometry) often enhances performance in other categories (e.g., Algebra, Number Theory). (3) As 1-shot RLVR training progresses, both the response length for the training example and the frequency of self-reflective terms in downstream tasks increase.
Through ablation studies, we show that policy gradient loss primarily drives the improve- ments observed in I-shot RLVR, distinguishing from "grokking", which heavily depends on regularization methods like weight decay. Additionally, we emphasize the importance of promoting diverse exploration in model outputs, showing that adding an entropy loss with an appropriate coefficient further enhances performance.
Lastly, we find that employing entropy loss alone, even without any outcome reward, yields performance boost, although it remains weaker than the format-reward baseline. Similar improvements are observed for Qwen2.5-Math--7 and -aaee-3.------tttttee We also discuss label robustness and prompt modification in RLVR (Appendix C.2
2 Preliminary
Table 1: 1-shot RLVR with math examples ky/Tis improves model performance on ARC, even better than full-set RLVR. Base model is 1ren2.5-Math-1.35 evaluation tasks are ARC-Easy (ARC-E) and ARC- Challenge (ARC-C). We select the checkpoints achieving the best aver- age across 6 math benchmarks.
RL Loss Function. In this paper, we adopt GRPO 8 as the RL algorithm for LLMs unless stated otherwise. We briefly introduce three main components in the loss function as below and provide more details in Appendix B.
(1) Policy gradient loss: it encourages the model to produce responses with higher rewards, assigning weights according to their group-normalized advantages. Thus, better-than- average solutions are reinforced, whereas inferior ones are penalized. Since we focus on mathematical problems, the reward is defined as binary (0-1), where a reward of is granted only when the outcome of the model's response correctly matches the ground truth. We do not include the format reward when using the outcome reward, but format- reward RLVR is used as baseline for Qwen models. Further discussion can be found in Appendix C.2.3
Dataset
Size
ARC-E ARC-C
Base
NA
48.0
30.2
MATH
7500
51.6
32.8
DSR-sub
1209
42.2
29.9
[m]] [1a)
52.0
32,2
55.8
33.4
(2) KL loss: it helps to maintain general language quality by measuring the divergence between current model's responses and those from reference model.
(T, T1a)
2
52.1
32.4
(3) Entropy loss [22]: applied with negative coefficient, incentivizes higher per-token entropy to encourage exploration and generate more diverse reasoning paths. We note that entropy loss is not strictly necessary for GRPO training. but it is included by default in verl 22 used in our experiments. Its effect on I-shot RLVR is discussed in Sec.
Data Selection: Historical Variance Score. To explore how extensively we can reduce the RLVR training dataset, we propose simple data selection approach for ranking training examples. We first train the model for E epochs on the full dataset using RLVR. Then for each example i E [N] = ( N}. we can obtain a list of historical training accuracy L, [$i.1,...,44 &,, which records its average training accuracy for every epoch. Note that some previous work has shown that the variance of the reward signal (33 is critical for RL training, we simply rank the data by their historical variance of training accuracy, which is directly related to the reward:
0 = sar(st,,.....
(1)
Next, we define a permutation क [N] - [N] such that Uz(1) 2 ... 2 (x))). Under this ordering, (j) (denoted as n, for convenience) corresponds to the example with the j-th largest variance ey:
:= = (5) arg sort (ur [N]
(2)
3

---

We then select examples according to this straightforward ranking criterion. For instance, T1 identified by the historical variance score on Qwen2.5-Math-11.. performs well in 1-shot RLVR (Sec. 3.2.3 3.3) We also choose additional examples from diverse categories among (mI..... 117] and evaluate them under 1-shot RLVR (Tab, 33. finding that 113 likewise achieves strong performance. Importantly, we emphasize that this criterion is not necessarily optimal for selecting single examples for 1-shot RIVRi In fact, Tab. 3 shows that many examples, including those with moderate or low historical vanance, can individually produce improvements on MATH500 when used as single training example in RLVR. This suggests a potentially general phenomenon that is independent of the specific data selection method.
Experiments
3.1 Setup
Models. We by default run our experiments on Qwen2.5-Math-1.55 24 25 and also verify the effectiveness of Qwen2.5-Math-77 (25). 1ama-3.2-3B-Instee [26]. and DeepSeck-RI-Dise
Qwen-1.5B [2 for 1-shot RLVR in Sec. 3.3 We also include the results of Qwen2.5-1.50 and
0wen2.5-Math-1 1.5t-Instuut in Appendix C.1.2
Dataset, Due to resource limitations, we randomly select a subset consisting of 1209 examples from DeepScaleR-Prewee Preview-Datasel [18] as our instance pool ("DSR-sub"), For data selection (Sec. 2 as described in Sec. 2 we first train Siven2.5-Math-1.5B for 500 steps, and then obtain its historical variance score (Eqn. and the corresponding ranking (Egn. 2 on the examples. To avoid ambiguity, we do not change the correspondence between {T, |2 and examples for all the experiments, i.e., they are all ranked by the historical variance score of Qwen2.5-Math- 1.5B. We also use the MATH [27] training set (consisting of 7500 instances) as another dataset full RLVR to provide a comparison. More details are in Appendix [B.2
Training. As described in Sec. 2 follow the verl 22 pipeline, and by default, the coefficients for KL divergence and entropy loss are 3 = 0.001 and a = -0.001, respectively. The training rollout temperature is set 0.6 for vLLM (28 The training batch size and mini-batch size are 128
F and we sample 8 responses for each prompt. Therefore, we have 8 gradient updates for each rollout step. By default, the maximum prompt length is 1024, and the maximum response length is 3072, considering that Srve2.55-Mtt-L1 1B57BBss context length are 4096. For a fairer comparison on Qwen models, we include the format-rewan baseline, which assigns a reward of if and only if the final answer can be parsed from the model output (see Appendix C.2.3 for details). More details are in Appendix [..4
Evaluation. We use the official Qwen2.5 Mathh evaluation pipeline 25 for our evaluation. Six widely used complex mathematical reasoning benchmarks are used our paper: MATH500 [27
AIME 2024 [30], AMC 2023 (31] Minerva Math (32), OlympiadBench [33], and AIME 2025 [30].
We also consider mon-mmateaati reasoning tasks ARC-Eas, Easy and ARC-Challenge (34] More details about benchmarks are in Appendix [B.3 For AIME 2024, AIME 2025, and AMC 2023, which contain only 30 or 40 questions, we repeat the test set 8 times for evaluation stability and evaluate the model with temperature 0.6, and finally report the average passo1 (avg08) performance. And for other 3 mathematical benchmarks, we let temperature be O. The evaluation setup for DeepSeek-RI Distiii--iiQeenn 1.5BB and other evaluation details are provided in Appendix B.5
3.2 Observation of 1/Few-Shot RLVR
In Fig. I we have found that RLVR with 1 or examples can perform as well as RLVR with thousands of examples, yielding significant improvements in both format and non-format aspects. Tab. further shows that -Shot RLVR with these math examples enable better generalization on non-mathematical reasoning tasks (More details are in Appendix C.Ip. p. To better understand this phenomenon, we provide detailed analysis of 1-shot RLVR in this section.
3.2.1 Dissection of : A Not-So-Difficult Problem
Nevertheeesss, as shown in Tab. (Sec. 3.3 selection based on historical variance scores outperforms
random selection in RLVR on (Qwen2.-Matth-71
"Note that verl sets drop_ hasp_aasttTrueee for training dataloader, so the dataset must be at least as large as the training batch size. To enable RLVR with very few examples, we duplicate the selected example until reaching dataset.
128 samples and store them

---

Table 2: Example TT. It selected from DSR-sub (Sec. 3.1p
Prompt of example ma: ma:
The presaure VV P W exerted by wind on a sail variea jointly as the area V A W of the sail and the cube the wind's velocity WX V. When the velocity W XV niles per hour, the pressure on sail of M 2 W square feet ia V. 4 (V) pounde. Find the sind velocity when the pressure on VVC 4 W square foet of sail VVC 32 AV pounds. Let's think step by step and output the final answer within ViboxedO..
Ground truth (label in DSR-sub): 12.8.
First, we inspect the examples that produce such strong results. Tab. 2 lists the instances of w. which is defined by Eqn. 2 We can see that it's actually an algebra problem with a physics background. The key steps for it are obtaining k = 1/256 for formula P kAV3, and calculating V (2048)1/3 = 12.699. Interestingly, we note that base model already almost solves T1. In Fig. 3 the base model without any training already solves all the key steps before calculating (2048)) with high probability] Just for the last step to calculate the cube root, the model has diverse outputs, including 4, 10.95, 12.6992, 8 44 12.70, 12.8, 13, etc. Specifically, for 128 samplings from the base model, 57.8% of outputs are "12.7" or "12.70", 6.3% of outputs are "12.8", and 6.3% are "13". More examples used in this paper are shown in Appendix E In Appendix C.2.5 we show that interestingly, even though the key step in solving T is computing /2048, including only this question in the training example leads to significantly worse performance compared to using full πΙ.
3.2.2
Post-saturation Generalization: Generalization After Training Accuracy Saturation
100
1000
100
Warning Acuracy
80
Trariing Accrmcy
Accrmcy
80
Tost Accuraio
80
30
Accurnyy
23
8
I shot (m)
5
shot (Mta
1.24 DSR
Avera Benchmares
Average en Bunchmarks
Average Bencherans
25k
Stop
Stop
Stop
Figure 2: Post-saturation generalization in 1-shot RLVR. The training accuracy of RLVR with Ti (Left) and (Middde) saturates before step 100, but their test performance continues improving. On the other hand, the training accuracy for RIVR with 1.2k DSR-sub dataset (Right) still has not saturated after 2000 steps, but there is significant improvement on test tasks after step 1000.
Then, we show an interesting phenomenon in 1-shot RLVR. As shown in Fig. 2 since we only have one training example, it's foresecable that the training accuracy for Ti and Ti quickly saturates before the 100th step. However, the performance on the test set still continues improving: 1-shot RLVR with и gets 3.4% average improvement from step 100 to step 1540, while using Ti3 yields a 9.9% average improvement from step 500 to step 2 Besides, this phenomenon cannot be observed when using full-set RLVR with DSR-sub currently, as the test performance has started to drop before training accuracy converges.
Moreover, we compare the training and evaluation responses in Fig. 3] Surprisingly. we find that at the final stage of 1-shot RIVR, the model overfits the single training example by mixing the correct calculation process into long unintelligible multilingual outputs in its outputted reasoning. Nonetheless, the test responses still remain normally and maintain high accuracy, indicating that post-saturation generalization still holds even after overfitting the training example. In particular, overfitting in RLVR occurs quite late (πz after 1400 steps and Tı3 after 1800 steps). Considering that each example is sampled 1 1024 times per step. the single training example is not overfitted until after millions of rollouts. Further analysis is provided in Sec. 4.1
3.2.3 1-shot RLVR Effective for Many Examples & Brings Improvements across Categories
In this section, we investigate whether different data behave differently in 1-shot RL, and whether 1-shot RLVR with one training example from a specific category can help the model better gen- eralize to other categories. We select data with high (TT I 117), medium (To05 To06), and low
A more precise answer for गı should be 12.7 rather than 12.8, but this slight deviation does not affect the experimental results. We show that both values yield strong performance Tab in Sec. [4.]
'This behavior looks similar to "grokking", but we do not emphasize the sudden onset of generalization after training saturates. In Sec. 4.1 show that post- saturation generalization is distinct from grokking.
5

---

Training Response
Evaluation Response (MATH500)
Problem:Ao Ao shown in Table 2 Answer: 12.8
Problent Find the rooti of $/x- 31-3 0X-7/33 + (2x- 10/'3.
Answer:
Step 0: Step 1...M] P MAy"SSg....See (0. (/o0))/1224)
Step o: Fiidd rools of Six-3/9 + 77'3 (2x-
(225) (1... Step 3: MI v 33000 Vjrtaii == 12.85
10/ Highighht. hwg.h/h.gggiggnn.... Highgii....ngggggg........... final answer)."
.rnall answer istrtinn (booed]12.88) 88j MT
Step 500: start by writng the mathematioal mistionship for the
Slep 500: simplites lnte 3133 + (x-7 = [2x- 10
pressure 01 P (ijinni = Kudttt A Vodel V13 Wonshere 14 o) as
I -3/22 (x-- -10x- 7) (x- 7/2iii equation true (f the expression MKjx 3)2 (x- 5yx 7) (x- 7x-7)2
constam.... = Noodet 2 ficdot ths 91 // Whac (1226)4) ..
10
32 = Wirz Vantat 4 Vicdal V131) M[ 2048 = 1M3 (jril V=
(o tipp = 511) is solution. Since the decrmmiant is negallve,
(eon131/0048 (ulil V= 12.8 WinnThus, the wind valoctt (6
solutioss to this quadraiic equation. the final anewer BytroiitribbooreSSyy......
there
Wbovedd (2.88 miles per hour"
Step 1300: "First au nustthuiing the irital known scenario conditions
Slep 1300: 1. First let's simelly the equation.. Mp 1011x-
into the corresponding variart...... of P fiodal A fictot yig o/... From specricc V/A 2). 1f V W, 1/ 2 liinthhn
3//2 (x 31jx 77 + 7122 (2x 10/2)) = OntiirinTis gies
two cases considrlriint. W2x- 10 01/02 Mix 3/22 Ix-
required equatiio will be line outfeel44 = ficdot 2 Naattt
31jx- 7)+ (x- - [2x 10y22 = So 1 91g solutio
8h3 o/3...114 E ricdot 2 Vodot 5120jmii 1024400/nthus solving then seek toward i/ k ii/tniie== = (110c()[ = WWrrr (125819)
I New let's solve the second case. Vinj
7) =0 Vimplies equatin are 0/x= 500, Wix = Dii), and lipe = 71), Let's verily these solutions using Python. "python Infram sympy "utpurinjs. "nihh
1 (anat x= Paligrinno the solitoos
...For moment pinpoint... .[32 = (/hao11225)) (2266) Yodel fodot v*3u]_ now simpiittationn further solving thusl for inall... m 2048 !i] then
enward invesding concrese data
solutions to the equation Mijx 3/3 + (x- 7y's = (2x- 10/33g are
calculate intinjjv = [eoot3960048] = 12.8 W/nThus putting alrenty licings caloulsted match formal needed to seize of (oooed) 12.88
1i0 St), Uy 310 and life e 71g, Therefore, the roots the I eqqutioon ansinin ([ellbowed Tjll"
Step 1860: "Lon Pharmacoonil encompp lorone adroc Played Migikk Deperdenccy weeennenccyorrrreyee Propertyiziennl witch unespecied F Function
[Stsp 1860: Let's salve the equation Mixx 5y' (x- 71-3 = 12x- I (10511) step stsp...we need to soveentnninn' - 15x + 71x- 105 dinikjniilWee
WendDesigner %-jitinn kan' vantApppliaaiinn Seewewwww adeUser
Rational Root Theorem possible rational roots of this polynom al The possible rational roots lam the factors of the constant torm (126) divided by factors of |ihe ig poefiicint: (1). The factors of 105 are Wieem 1, /pmm 3, I'tm lppm 7. Upm 15, fpmm 21, Npm 35, lipm 1051). By seeting these values, we
ggcreadf of 21.3 'nternpaeeenmm Finds. A Metricreyau Irin
(alggegin & k ltedol Vocdal 1"3 Wild B k vicdol
(0/33 lmmples licdet ledot limples
(00244)10224) = Wha 11225)) Willdteo wo a a aallocaee
leHore A4 B 0b tu- 1 assurances 'ni(n322
oju is maat///l//6 3 15/5/2)
Wira W226)) Nodet 4 Modal V/3 Wimples - (ajse))
(115)- 105 = 125 375 + 356 106 = (in... we getinini
Wimples Vns = 2048 timp les V= Hisptt (2008)) = 12 Pn nrya
1 7ix- 105 = bi-6 10xx 21jenj 21jemi - 10x -11
tivol obRs? sp 39 a2 #or Work & aoood
IF (X-33 31x 7) Uintfeinno the roots are fix 31) and lipe=
(3rlt:///1G/groppp fru squmpoaee) (er (utlbboee
(70)... The fnnal answer (5iibei( 5, 7Jo0]
Figure 3: The model can still generalize on test data after overfitting training example for 1-shot RLVR's post-saturation generalization. Here we show model's I's response to training example Ii and selected MATH500 problem. Green/Red are used for marking Correct/Wront Wrong answers. The model converges on कi (before step 500) and later attempt to generate longer solutions for π in different styles (step 1300), and gradually performs better on evaluation task. But it significantly overfits training data I at step 1860 (when model achieves 74% MATH500 accuracy), as it mixes the correct process (cyan) with meaningless output. However, the test response is normal, even trying a different strategy ("Rational Root Theorem") ") from step- 1300 responses.
(T11201 1209) historical variance (Fqn. D and from different topics. We determine the categories of the questions based on their characteristics. We show their detailed MATH500 performance for both overall and subclasses in Tab. 3 More performance curves are in Appendix C.T
We observe that (1) 1-shot RLVR improves performance across all categories in MATH500. Almost all examples yield a 2 30% improvement over the base model, except for the incorrect example T1207 and the extremely difficult example 12208, which cause the model to fail to generate any correct solutions. (2) 1-shot RLVR can perform at least as well as the format-reward baseline (except T2207 and 11208), and with appropriate examples, 1-shot RLVR with outcome reward can achieve additional non-trivial improvements. From Tab. we observe that the improvements of some examples (c.g., Ny, T11, and Too6) mainly come from format correction. However, many other examples (e.g., Tı, T13, and T1299) still exhibit non-trivial improvements beyond format fixing. Further discussion is provided in Appendix C.2.3) (3) Counterintuitively, test data belonging the same category as the single training example docs not necessarily exhibit better improvement. For instance, 1 belongs to Number Theory, but RLVR trained with Tı1 achieves a relatively low Number Theory score compared to using other examples (e.g., Te05 from Precalculus). This may indicate that the reasoning capability stimulated by an instance cannot be simply predicted by superficial features Additional analysis on prompt complexity is provided in Appendix C.2.5
such as categories 35
3.2.4 More Frequent Self-Reflection on Test Data
In this section, we show another empirical observation of 1-shot RLVR: can increase the frequency of self-reflection [6] in the model responses as training progresses. To study this, we check the output patterns of different checkpoints from the RLVR training on Swen2.55-Math-1 We find

---

Table 3: 1(Few)-Shot RLVR performance (%) for different categories in MATH500. Here for MATH500, we consider Algebra (Alg.), Count & Probability (C.P.), Geometry (Geo.), Intermediate Algebra (I. Alg.), Number Theory (N. T.), Prealgebra (Prealg.). Precalculus (Precal.), and MATH500 Average (Avg.). We report the best model performance MATH500 and AIME24 separately (As illustrated in Appendix. [B.3). "Size" means dataset size, and "Step" denotes the checkpoint step that model achieves the best MATH500 performance. Data with red color means the model (almost) never successfully samples the ground truth in training (T12207 has wrong label and T220s is too difficult).
"Format" denotes the format reward baseline (Appendix (C.2.3) (C.223 for format correction. We further mention related discussions about prompt complexity in Appendix C.2.5
Dataset
Size
Step
Type
Alg. P. Geo. Alg. T. Prealg. Precal. MATH500 AIME24
Base
0
0
NA
37.1 3166 39.0
43.3
24.2
36.6
33.9
36.0
6.7
MATH
7500
1160
General
91.1 65.8
63.4
59.8
82.3
81.7
66.1
75.4
20.4
DSR-sub
1209
1160
General
91.9 68.4
58.5
57.7
85.5
79.3
67.9
75.2
18.8
Format
1209|
260
General
81.5 60.5
53.7
52.6
72.6
68.3
53.6
65.6
10.0
[m)]
1860
Alg.
88.7 63.2
56.1
62.9
79.0
81.7
64.3
74.0
16.7
T2
220
N.
83.9 57.9
56.1
55.7
77.4
82.9
60.7
70.6
17.1
{ T44
80
N. T.
79.8 57.9
53.7
51.6
71.0
74.4
53.6
65.6
17.1
[r>)
580
I. Alg.
75.8 60.5
51.2
56.7
59.7
70.7
57.1
64.0
12.1
[m11]]
20
N. T.
75.8 65.8
56.1
50.5
66.1
73.2
50.0
64.0
13.3
[Tis]
1940
Geo.
89.5 65.8
63.4
55.7
83.9
81.7
66.1
74.4
17.1
[16
600
Alg.
86.3 63.2
56.1
51.6
67.7
73.2
51.8
67.0
14,6
W171
220
C. P.
80.7 65.8
51.2
58.8
67.7
78.1
48.2
67.2
13.3
Teas
1040
Precal. N.
84.7 63.2
58.5
49.5
82.3
78.1
62.5
71.8
14.6
[Te0
460
83.9 63.2
53.7
49.5
58.1
75.6
46.4
64.4
14.2
[T1201]]
940
Geo.
89.5 68.4
58.5
53.6
79.0
73.2
62.5
16.3
[T1007]
100
Geo.
67.7 50.0
43.9
41.2
53.2
63.4
42.7
54.0
9.6
[T1208]
240
58.1
55.3
43.9
32.0
40.3
48.8
32.1
45.0
8.8
( [TT1209]
1140
Precal.
86.3 71.1
65.9
55.7
75.8
76.8
64.3
72.2
17.5
[wi -- mT
16
1840
General
90.3 63.2
61.0
55.7
69.4
80.5
60.7
71.6
16.7
[m1, m2)
1580
Alg.
89.5
63.2
61.0
60.8
82.3
74.4
58.9
72.8
15.0
[T), m13
2000
Aeeo
92.7 71.1
58.5
57.7
79.0
84.2
71.4
17.9
Response
i chemeres
(n))
1 (n) 1.2k
25:00
1.21 C58
1m 1:2k Bstiab
Numeer 22 So Mepeeelll
Aall
1920
2030
são
1.5
500
1.50
120 1758
Figure 4: (Left, Middle) Average response length on training data and entropy loss. After around 1300/1700 steps. the average response length of 1-shot RLVR with Tı/T13 significantly increases, corresponding that model tries to solve the single problem with longer CoT reasoning more diverse way (Fig. 3 step 1300), which is also confirmed by the increase of entropy loss. These may also indicate the gradual overfitting (Fig. 3 step 1860). (Right) Number of reflection words detected in evaluation tasks. The number of reflection words ("rethink", "recheck", and "recalculate") appearing in evaluation tasks increases I-shot RLVR with n. ky/T13, especially after around 1250 steps. matching the increase of response length. On the other hand, RLVR with DSR-sub contains fewer reflection words as the training progresses.
that their self-reflection process often appears with words "rethink". "recheck" and "recalculate" Therefore, we count the number of responses that contain these three words when evaluating 6 mathematical reasoning tasks. The results are Fig. 4 First, after around 1.3k steps, the response length and entropy loss increase significantly, which may imply the attempt of diverse output patterns or overfitting (Fig.B) 3. Second, for the evaluation task, the base model itself already exhibits self- reflection processes, which supports the observation in recent works [3] 21 Third, the number of self-recheck processes increases at the later stages of 1-shot RL training, which again confirms that the model generalizes well on test data and shows more complex reasoning processes even after
7

---

Table 4: [(few)-shot RLVR viable for different models and RL algorithm. "Random" denotes the 16 examples randomly sampled from 1.2k DSR-sub. Format reward (Appendix C.2.3) serves as a baseline for format correction. More details are in Appendix C.1 and we also include the results of 1aan255-Math---5---nsse
and Qwen2.5-1.5B in Appendix C.1.2
RL
Dataset
MATH
AIME
AMC
Minerva Olympiad-
AIME 2025
Avg.
Dataset
Size
500
2024
2023
Math
Bench
Qwen2.5-Mann 1aat2-5--Mtt-T11 (24) GRPO
NA DSR-sub
NA
51.0
12.1
35.3
11.0
18.2
6.7
22.4
1209
78.6
25.8
62.5
33.8
41.6
14.6
42.8
Format Reward
1209
65.8
24.2
54.4
24.3
30.4
6.7
34.3
) [mi, R13)
23.8
60.3
39.1
10.8
40.2
79.2
21.7
58.8
35.3
12.1
41.3
(mi, 12, IT1,, 21209)
78.6
22.5
61.9
36.0
43.7
12.1
42.5
Random
16
76.0
22.1
63.1
31.6
35.6
12.9
40.2
(T1 N26
77.8
30.4
62.2
35.3
39.9
9.6
42.5
I -lama-3.2-3B-Insttte
[26] GRPO
NA sub
8.3
25.3
15.8
13.2
1.7
17.5
1209
43.2
11.2
27.8
19.5
16.4
0.8
19.8
[m]] [m], Ris]
45.8
7.9
25.3
16.5
17.0
1.2
19.0
49.4
7.1
31.6
18.4
19.1
0.4
21.0
(m,, 22, IT13, ]
46.4
6.2
29.1
21.0
15.1
1.2
19.8
Qwen2.5-Math-1.58 (24 + PPO
NA suR-sub
6.7
28.1
4.6
17.6
1209
72.8
19.2
48.1
27.9
35.0
9.6
35.4
72.4
11.7
51.6
26.8
33.3
7.1
33.8
ReepSeck-R1-Dee R1-Distill -1--iisill---wen-1... (2) + GRPO (Eval=32k)
NA
NA
82.9
29.8
63.2
26.4
43.1
23.9
44.9
DSR sub
1209
84.5
32.7
70.1
29.5
27.8
48.6
(T²)
83.9
31.0
66.1
28.3
44.6
24.1
46.3
(m) m2, Tiis 1
84.8
32.2
66.6
27.7
45.5
24.8
46.9
(I...... m161
16
84.5
34.3
69.0
30.0
46.9
25.2
48.3
overfits the training data. Interestingly, for the 1.2k DeepScaleR subset, the frequency of reflection slightly decreases the training progresses, matching the decreasing response length.
3.3 1/Few-shot RLVR on Other Models/ Algorit/s
We further investigate whether (feew)-hhot RLVR is feasible for other models and RL algorithms. We consider setup mentioned in Sec. 3.1 and the results shown in Tab, (Detailed results each benchmark are in Appendix C. We can see (1) for Qwen2.5-Math-1
--shot RLVR with Ti improves average performance by 178% (5.9% higher than format-rewan baseline), and 4-shot RLVR performs well RLVR with DSR-sub. Moreover, {TI.... 116) performs better than the subset consisting of 16 randomly sampled examples. (2) For Llama-3.2-3 2-223B-Instruct, the absolute gain from RLVR is smaller, but (few)--hot RLVR still matches or surpasses (e.g., {51, Tia)) the performance of full-set RLVR. We also show the instability of the RLVR process Llama-3.2- 3B-Instruct in Appendix C.I (3) RLVR with मं using PPO also works for Qwen2.5-Man with PPO. (4) For -eepSeek-R1-Distlll.
.BB the performance gap between few-shot and full-set RLVR is larger. Nevertheless, few-shot RLVE still yield improvement. More results are in Appendix
4 Analysis

---

Table Ablation study of loss function and label correctness. Here we use Siven2.5-Math- 1.5h and example T1. "+" means the component is added. "Convergence" denotes if the training accuracy saturates (e.g. Fig. 2). *-0,003" is the coefficient of entropy loss (default -0.001). We report the best model performance on each benchmark separately (Appendix B.3]. (1) Rows 1-8: The improvement of -shot RLVR is mainly attributed to policy gradient loss, and it can be enhanced by adding entropy loss. (2) Rows 9-10: Simply adding entropy loss alone still improve MATH500, but still
worse than the format reward baseline (Tab, 3 MATH500: 65.6, AIME24: 10.0). (3) Rows 5,11-13:
further investigation into how different labels affect test performance.
Policy
Weight KL.
Entropy Loss
Training Convergence
MATH 500
ATME
Row
Label
Loss
Decay
Loss
2024
12.8
NO
39.8
7.5
12.8
YES
71.8
15.4
12.8
YES
71.4
16.3
12.8
YES
70.8
15.0
12.8
YES
74.8
17.5
-0.003
12.8
YES
73.6
15.4
12.8
YES
75.6
17.1
12.8
NO
39.0
10.0
12.8
NO
65,4
7.1
10
12.8
NO
63.4
8.8
11
12.7
YES
73.4
17.9
12
4
YES
57.0
9.2
13
929725
NO
64.4
9.6
In this section, we concentrate on exploring the potential mech- anisms that allow RLVR to work with only one or a few exam-
Average on benchmarks
ples. We hope the following analyses can provide some insight
32.5
for future works. Additional experiments and discussions about the format correction (Appendix C.2.3), prompt modification (Appendix C.2.5) and the reasoning capabilities of base models
8
20.0
25.0
22.5
(Appendix D are included in supplementary materials.
Entray
Entropy C.
250
530
150
1258
2309 1750 2000
Stace
4.1 Ablation Study:
Policy Gradient Loss the Main Contributor, and Entropy Loss Further Improve Post-Saturation Generalization
Encouraging ex- ploration can improve post- saturation generalization. the temperature parameter for training rollouts.
Figure
As discussed in Sec. 3.2.2 1-shot RLVR shows the property of post-saturation generalization This phenomenon similar to "grokking" [36] 37). which shows that neural networks first memorize/overfit the training data but still perform poorly test set, while suddenly improve generalization after many training steps. natural question raised: Is the performance gain from fain 1-shot RLVR related to the "grokking" phenomenon? ? To answer
this question, noting "grokking" is strongly affected by regularization [36] 38,41 like weight decay, we conduct an ablation study by removing or changing the components of the loss function one by one to see how each of them contributes to the improvement.
The results are shown in Tab. 5 (Test curves are in Appendix C.2.1 We see that if we only add policy gradient loss (Row 2) with TT. we already get results close to that of the full loss training (Row 5). In addition, further adding weight decay (Row 3) and KI. divergence loss (Row 4) has significant impact model performance. while adding entropy loss (Row 5) can further bring 4.0% improvement
careful about the weight of the entropy loss, as too large coefficient (Row 6) might make the training more unstable. These observations support that the feasibility of 1(few)-shot RLVR is mainly attributed to policy gradient loss, rather than weight decay, distinguishing it from "grokking", which should be significantly affected by weight decay. To double check this, we show that only adding weight decay and KI. divergence (Row 8) has little influence model performance, while using only policy gradient loss and entropy loss (Row 7) behaves almost the same as the full GRPO loss.
MATH500 and 2.5% for AIME24. Here we need
Moreover, we also argue that encouraging greater diversity in model outputs- for instance, adding proper entropy loss can enhance post-saturation generalization in 1-shot RLVR. As shown in Fig. 5 without entropy loss, model performance under 1-shot RLVR shows limited improvement beyond step 150, coinciding with the point at which training accuracy saturates (Fig. Left). By adding entropy loss, the model achieves average improvement of 2.3%, and further increasing

---

the temperature to t = 1.0 yields an additional 0.8% gain. More discussions about entropy loss and post-saturation generalization are in Appendix C.2.2
4.2 Entropy-ILoss-Ome Training & Label Correctness
Training with only en- tropy loss using T1 can par- tially improve base model perfor- mance, but still perform worse than format-reward baseline. De- tails are in Tab. 13
Table 6:
In Tab. we find that when using T2207 and 12208 it is difficult for model to output the ground truth label and receive rewards during 1-shot RLVR training, resulting in very sparse policy gradient signal. Nevertheless, they still outperform the base model, although their performance remains lower than that of the format-reward baseline. To investigate this, we remove the policy loss from the full GRPO loss (Tab. 5 Row 9) or even retain only the entropy loss (Row 10), and again observe similar improvement. Furthermore, this phenomenon also happens on Owen2.5-Math-75 and
Model
M500
Avg.
Qwen2.5-Math-1.55
36.0
17.6
+Entropy Loss, 20 steps
63.4
25.0
although only improve at the first several steps. These results implies entropy loss may independently contribute to performance gains from format correction, which, although much smaller than those from policy loss, are still nontrivial.
Format Reward
65.0
28.7
Llama-3.2-3B-nnstree
40.8
17.5
+Entropy Loss, 10 steps
47.8
19.5
Qwen2.5-Math-778
51.0
22.4
+Entropy Loss, 4 steps
57.2
25.0
Moreover, we conduct an experiment by altering the label to (1) the correct one ("12.7," Row 11), (2) an incorrect one that model can still overfit ("4," Row 12), and (3) an incorrect one that the model can neither guess nor overfit ("9292725," Row 13). We compare them with (4)
Format Reward
65.8
34.3
the original label ("12.8," Row 5). Interestingly, we find the performance rankings (1) (4) > (3) > (2). This suggests that slight inaccuracies in the label do not significantly impair 1-shot RLVR performance. However, if the incorrect label deviates substantially while remaining guessable and overfittable, the resulting performance can be even worse than using completely incorrect and unguessable label, which behaves similarly to training with entropy loss alone (Row 10). In Appendix C.2.4, we also discuss label robustness on full-set RLVR by showing that if too many data in the dataset are assigned random wrong labels, full-set RLVR can perform worse than 1-shot RLVR.
5 Conclusion
In this work, we show that 1-shot RLVR is sufficient to trigger substantial improvements in reasoning tasks, even matching the performance of RLVR with thousands of examples. The empirical results reveal not only improved task performance but also additional observations such as saturaaturttion generalization, crtesoyat generalization, more frequent self-reflection and also additional anal- ysis. These findings suggest that the reasoning capability of the model is already buried in some base models, and encouraging exploration on a very small amount of data is capable of generating useful RL training signals for igniting these LLM's reasoning capability. It also demonstrates the anti-overfitting property the RLVR algorithm with zero-mean advantage, as we can train on a single example millions of times without performance degradation. Our work also emphasizes the importance of better selection and collection of data for RLVR. We discuss directions for future work in Appendix D.4 and also discuss limitations in Appendix D.I
Acknoledgements
We thank Lifan Yuan, Hamish Ivison, Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Pang Wei Koh, Kaixuan Huang, Mickel Liu, Jacqueline He, Noah Smith, Jiachen T. Wang. Yifang Chen, and Weijia Shi for very constructive discussions. YW and ZZ acknowledge the support of Amazon AI Ph.D. Fellowship. SSD acknowledges the support of NSF IIS-11 70, NSF DMS-21 AMSS2134106, 34106, NSF C2F-2212261, NSF IIS-2143493, NSF CCF-2019844, NSF 11S-222988, and the Sloan Research
Fellowship.
References
[1] OpenAL.
Learning to reason with 1lms.
learning- to- reason-with- 1lmms/ 2024. Accessed: 2025-04-10.
10

---

[2] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[3] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
[4] Jiaxuan Gao. Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint wxXv:2440.151155 2024.
[5] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu. Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241..15124, 2024.
[6] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-im elf-innpoving reasoners, or, four habits of highly effective stars. arXiv preprint Mxiv:2503.01307. 2025.
[7] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 06347, 2017.
policy optimization algorithms. arXiv preprint
[8] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2x2v:22402.033000 2024.
[9] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni. Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.0166 2024.
[10] Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's 's collapse in long-cot? value optimization holds the secret. arXiv preprint 2rX1v:2503.01491, 2025.
[11] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar Xivv: 14476, 2025.
[12] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprini #rXiv:2504005/68 2025.
[13] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint ar arXiv:2503.20783, 2025.
[14] Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Alpay Ariyak, Qingyang Wu, Ce Zhang. Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level. /tttp:://peetty/ raaio-rddoobbbo DeepCoder- A- Fullu-lyluua Open- Source- 14B- Coder - at - 03 mini 1 Lotpaaalcccc000216660eeea33a33.3
www.iioooo.oooiioooooooom nww.onsotiioonoo.ooo site/
2025. Notion Blog.
[15] Jian Hu. Reinforcet+: A simple and efficient approach for aligning large language models. arXiv preprint 2rXh:22501.0322,, 2025.
[16) Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng. Shimiao Jiang, Shiqi Kuang, Shouyu Yin. Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementation of large-scale reinforcement learning 1lm, 2025.

---

[17] Jia LI, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: bictps://huggingfaaee httpg///w.....o///////////aaaaa. ( /ttptttpps///ttub.... com/ project-numinast http://wwww...........aaaaaaa//a 2024.
paf)
[18] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. / racio www.on...iooooooo notion site/ DeepScaleR-Surpee -i www.reiewwwttha--ieee
2025.
Notion Blog.
[19] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for scaling, 2025.
[20] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint 1rX1::2304.13877, 2025. Submitted on April 18, 2025.
[21] Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training. arXiv preprint arXiv:2504.04000 2025.
[22] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint Xrviv: 2409. 100./99560 19256, 2024.
[23] Noam Razin, Zixuan Wang. Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes reward model good teacher? optimization perspective. arXiv preprint arXiv:2503. 15477, 2025.
[24] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, arXiv preprint arXiv:2412.15/ 15/15, 2024.
[25] An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnnt arXiv preprint arXiv:2409. /2/22, 2024.
[26] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
[27] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.
(28) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[29] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprini 2rX5v:2305.20050 2023.
[30] Art of Problem Solving. Aime problems and solutions. com/wiki/index: Pttpe/MMMMPPPoess...eea and_Solutions Accessed: 2025-04-20.
[31] Art of Problem Solving. Ame problems and solutions. com
and_Solutions
Accessed: 2025-04-20.
12

---

[32] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857 2022.
[33] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint MxXi::2402.14003 2024.
[34] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.
2018.
[35] Zhiyuan Zeng, Yizhong Wang, Hannanch Hajishirzi, and Pang Wei Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees. arXiv preprint arXiv:2503.08893, 2025.
[36] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 2022.
[37] Simin Fan, Razvan Pascanu, and Martin Jaggi. Deep grokking: Would deep neural networks generalize better? arXiv preprint 14X::2405S.1454, 2024.
[38] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability.. arXiv preprint arXiv:2301.05217, 2023.
[39] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
[40] Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complex- ity dynamics of grokking, arXiv preprint arXiv:2412.098100 2024.
[41] Lucas Prieto, Melih Barsbey, Pedro AM Mediano, and Tolga Birdal. Grokking at the edge of numerical stability. arXiv preprint arXiv:2501. 1066000049772222 2025.
[42] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild. arXiv preprint 2tt.::5503.88222. 2025.
[43] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint
2025.
[44] Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models. arXiv preprint arXiv: /7287, 2025.
[45] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint 2005032505.03335. 2025.
[46] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint arXiv: 2504.0581. 2025.
[47] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuckai Zhu, Biqing Qi. Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint 20t.:2504.16884. 2025.
[48] Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.0188 2025.
13

---

[49] Lichang Chen, Shiyang Li. Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang. Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data. In International Conference on Learning Representations, 2024.
[50] Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. In Findings of the Association for Computational Linguistics, 2023.
[51] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: selecting influential data for targeted instruction tuning. In International Conference on Machine Learning 2024.
[52] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber, Active preference learning for large language models. In International Conference on Machine Learning, 2024.
[53] Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. Enabling weak Ilms to judge response reliability via meta ranking. arXiv preprint arXiv:2402. 12146, 2024.
[54] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint 2400000022.00000, 2024.
[55] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
[56] Mehdi Fatemi, Banafsheh Rafice, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05/88, 2025.
[57] J. Schulman. Approximating kl divergence. http://joachuu.net/blo////t 2020. 2025.
html
[58] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
[59] Yingqian Min, Zhipeng Chen, Jinhao Jiang. Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on dlw-tthinking reasoning systems. arXiv preprint arXiv:2412.09413, 2024.
[60] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
http://carrcccoous-yyyrreee.eeee Skywork- Open -eeeen-Re - Swwreeeeeerieeeeeeeeeeeeeeeeeeee w0000000000000000000000000000000 Notion Blog.
notion. site/
2025.
[61] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411. 15594, 2024.
[62] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[63] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 2022.
[64] David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: / 7055. /0694, 2017.
[65] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12): 2021.
[66] Nitish Shirish Keskar, Dhcevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On latch-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609604836, 2016.
14

---

[67] Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. Iclr, 2021.
[68] Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint, 2024.
[69] Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi-Quan Luo. Entropic distribution matching for supervised finettuming of llms: Less overfitting and better diversity. NeurlPS 2024 Workshop on Fine- Tannin in Modern Machine Learning: Principles and Scalability, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state 1. the benchmarks and models using, 2. the empirical improvement and observation; 3. the methods, key insights and the analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We don't include theoretical results.
Guidelines:
16

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-
referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The main results and several analysis are in the Sec. Sec. and Appendix C We also provide experiment details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Ye]]
Justification: The code will be provided according to Neurips code submission guidance. After got accepted, we will open source that.
17

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic/guidess http://giiieeeeoeuuussssssssstse Policy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc/pubicc.. cec/public/guideee StgguieSume/o sSwddeeoodisssssieeoeea sssissssssippiiioi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide full experiment details in Sec. 3.1 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: RLVR training is always resource intensive. In our paper, we train RLVR for kk2k steps for full convergence, which may take 2-4 days of training on 8 80G A100 GPUs. Besides, most of the current work on RLVR. like Dr. GRPO, DeepScaleR, SimpleRL- etc., also runs the training only once. Nevertheless, the numerous experiments in Sec. 3.2.3 and Sec. 4 have cross-validated the efffcctiveness of 1-shot RLVR.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the compute resources in Appendix B.4 and Appendix [.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required each of individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp://neuu httpp:::////rr....... httpp///cc/ttiiiiiiiiiiiee....e
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research focuses on using one example in RLVR to perform as well as full-set RLVR, the social impact of 1-shot RLVR should be the same that of all the full-set RLVR works.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use public models and datasets, and the widely-used public training pipeline verls, so we don't ' include any new pretrained models or new datasets that may have these kinds of risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes]
Justification: We cite the verl (22 pipeline we used for our code, and all the data/models (e.g. DeepScaleR [88 used evaluate and train.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
20

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects. All metrics are fixed evaluations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
21

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ cttccc///oooooo / for what should should not be described.
22

---

Contents
Introduction
2
Preliminary
3
Experiments
3.1
Setup
3.2
Observation of 1/Few-Shot RLVR
3.2.1
Dissection of nh A Not-So- Problem
3.2.2
Post-saturation Generalization: Generalization Alter Training Accuracy Sat-
uration
3.2.3
I-shot RLVR is Elfective for Many Examples & Brings Improvements across
Categoniss
3,2,4
More Frequent Self Self-Reel on Test Data
3.3
1/Few-shot RLVR on Other Models/Algorithms
Analysis
Ablation Study: Policy Gradient Loss is the Main Contributor, and Entropy Loss
Further Improve PastrSaturation Generalization
4.2
Entropy-Loss-Only Training & Label Correctness
10
Conclusion
10
Acknoledgements
10
Related Work
24
Experiment Setup
24
Details of Loss Function
24
B.2
Training Dataset
25
Evaulation Dataset
26
B.4
More Training Detail
26
B.5
More Evaluation Details
26
8.6
Performance Difference on Initial Model
27
Evaluation Result
32
Main Experiments
32
C.I.1
Detailed performance Swee2.5-Math-
32
C.1.2
Detailed Performance More Models and Training Examples.
32
C.1.3
Detailed performance with best per- benchmaeh results
32
Detailed Test curves on MATH500 for I-shot RLVR on Qwen2.5- Math- .BB
1.5
Detailed RLVR results on eacn benchmark over training process.
33
C. 1.6
More Evaluation on -I---stii---- Qwen-1.5B
33
C.2
Analysis
33
23

---

Test Curves for Ablation Study
33
C.2.2
Entropy loss
33
(Only) Format Correction?
Influence of Random Wrong Labels
38
C.2.5
Change the Prompt of
38
Response Length
Pass08 Results
39
D Discussions
39
D.I Limitations of Our Work
39
D.2 Reasoning Capability Base Models
40
D.3 Why Model Continues Improving After the Training Accuracy Reaches Near 100%
40
D.4 Future Works
40
Example Details
41
A Related Work
Reinforcement Learning with Verifiable Reward (RIVR). RLVR, where the reward is computed by rule-based verification function, has been shown to be effective in improving the reasoning capabilities of I.LMs. The most common practice LLMs on mathematical reasoning datasets is to use answer matching: the reward function outputs a bi- nary signal based if the model's answer matches the gold reference answer 4 5
RLVR when applying reinforcement learning to
2 42| This reward design avoids the need for outcome-based or process-based reward models, offering simple yet effective approach. The success of RLVR is also supported by advancements in RL algorithms,, including value function optimization
detail optimization in PPO [7] (e.g.. VinePPO (9) VC- PPO 10., VAPO [12), stabilization and acceleration of GRPO 2 (e.g.. DAPO [T] Dr. GRPO [3] GRPO+[14) [4]. SRPO [16), and integration of various components (c.g.. REINFORCE++ There also some recent works that focus RLVR with minimal human supervision (without using labeled data or even problems), such as Absolute- Zero
445 EMPO (46), and TTRL [47]
Data Selection for LLM Post-Training The problem of data selection for LLM post-training has been extensively studied in prior work [48) with most efforts focusing data selection for supervised fine-tuning (instruction tuning). These approaches include LLM-based quality assessment 44). leveraging features from model computation (50), gradient-basee based selection [51], and more. Another line of work (52)54) explores data selection for human preference data in Reinforcement Learning from Human Feedback (RLHF) (55) Data selection for RLVR remains relatively unexplored. One attempt LIMR [19), which selects 1.4k examples from an 8.5k full set for RLVR to match performance; however, unlike our work, they do not push the limits of training set size the extreme case of just single example. Another closely related concurrent work (56) shows that RLVR using PPO with only 4 examples can already yield very significant improvements; however, they do not systematically explore this observation, nor do they demonstrate that such extremely small training set can actually match the performance of using the full dataset.
B Experiment Setup
B.1 Details of Loss Function
As said in the main paper, we contain three components in the GRPO loss function following verl [2 pipeline: policy gradient loss, KL divergence, and entropy loss. Details are follows. For each question sampled from the Question set P(Q), GRPO samples a group of outputs {01, o2...., 0G) from the old policy model ne and then optimizes the policy model by minimizing the following

---

loss function:
LeG-GRPOl::00 3L' Oret) a En (:0)
LORPO (0) = PQ)
(or ( a r (0]g)
where 3 and a are prompt-responses: {g. {or )
(in general / > 0, a < 0), and "." is the abbreviation of sampled The policy gradient loss and KL divergence loss are:
G
Cio.curola foi = -
Σ
min
Ta(o, q)
A,, clip
metaila) Fozu [ollg] 1 E, 1 1 +))),)))
(4)
Th (0:1g)
1=1
Te, (o.1g) log
Te Te.. (o:lg)
(g (o 0, Oerr = DKL (Tem)
(5)
Ta (o:0))))
(o(())))
Here Pee is the reference model, e is hypere-arameter of clipping threshold. Notably, we use the approximation formulation of KL divergence [57], which is widely used in previous works [8 2].
Besides, A, the group-normalized advantage defined below.
A; Ti - mean
E [G]
(6)
Since we focus on math questions, we let the reward r; be the 0-1 accuracy score, and r; and only the response gets the correct answer to the question q- What's more, the entropy loss E'troyy calculates the average per-token entropy of the responses, and its coefficient a < 0 implies the encouragement of more diverse responses.
The details of entropy loss are follows. For each query g and of outputs {or the model produces logits that determine the policy distribution ne. These logits are the direct computa- tional link between inputs and outputs specifically, the model processes generate logits X, which after softmax normalization give the probabilities used sample each token in the outputs o. The entropy loss is formally defined below.
Eb.o Mo.e (X)
Eimooy (q, (o:):o
Σs,r 2b.sMM Ma..
Here Mr., represents the response mask indicating which tokens contribute to the loss calculation (excluding padding and irrelevant tokens), with b indexing the batch dimension and indexing the sequence position. The entropy (X) is computed from the model's logits X:
Ho. (X) log (onwi) Σ Pb.a.v Xo,s,e
where indexes over the vocabulary tokens (i.e., the possible output tokens from the model's vocabulary), and the probability distribution is given by Pb,s.v = softmax(Xo.))e Ev
B.2 Training Dataset
DeepScaleR-Previin Dataset 18] consists of approximately 40,000 unique mathematics mooblem-answer pairs from AIME (1984-2023), AMC (pre-2023), and other sources including Omni-MATH [58] and Still [59]. The data processing pipeline includes extracting answers using
DeepScaleR-sub.
removing duplicate problems through RAG with Sentence-Transe TranerTrrnffooosss embeddings, and filtering out questions that cannot be evaluated using SymPy to maintain clean training set. We randomly select a subset that contains 1 1,209 examples referred to as "DSR-sub".
MATH. Introduced in 27 this dataset contains 12,500 challenging competition mathematics problems designed to measure advanced problem-solving capabilities machine learning models. Unlike standard mathematical collections, MATH features complex problems from high school math- ematics competitions spanning subjects including Prealgebra, Algebra, Number Theory, Counting and Probability. Geometry, Intermediate Algebra, and Precalculus. with each problem assigned difficulty level from to 5 and accompanied by detailed step-by-step solutions. It's partitioned into a training subset comprising 7,500 problems (60%) and a test subset containing 5,000 problems (40%).
25

---

B.3 Evaulation Dataset
All evaluation sets drawn from the Qwen2.5-Maz evaluation repository with the exception of AIME2025
We summarize their details as follows:
MATH500. MATH500, developed by OpenAl 20 comprises a carefully curated selection of 500 problems extracted exclusively from the test partition (n=5,000) of the MATH benchmark [27]. It is smaller, more focused, and designed for efficient evaluation.
AIME 2024/2025. The AIME 2024 and 2025 datasets are specialized benchmark collections, each consisting of 30 problems from the 2024 and 2025 American Invitational Mathematics Examination (AIME) I and II, respectively 30
AMC 2023. AMC 2023 dataset consists of 40 problems, selected from two challenging mathematics competitions (AMC 12A and I2B) for students grades 12 and under across the United States [31] These AMC 12 evaluates problem-solvee topics such as arithmetic, algebra, combinatorics, geometry, number theory, and probability, with all problems solvable without calculus.
abilities secondary school mathematics, covering
Minerva Math. Implicitly introduced in the paper "Solving Quantitative Reasoning Problems with Language Models" [32] as OCWCourses, Minerva Math consists of 272 undergraduate- STEM problems harvested from MIT's OpenCourse Ware, specifically designed to evaluate multi- step scientific reasoning capabilities in language models. Problems were carefully curated from courses including solid-state chemistry, information and entropy, differential equations, and special relativity, with each problem modified to be self-contained with clearly-delineater answers that are automatically verifiable through either numeric (191 problems) or symbolic solutions (81 problems).
Bennh..dBench. OlympiadBench [33 is a large-scale, bilingual, and multimodal benchmark designed to evaluate advanced mathematical and physical reasoning in Al systems. It contains 8,476 Olympiad-level problems, sourced from competitions and national exams, with expert-annotated step-by-step solutions. The subset we use for evaluation consists of 675 open-ended text-only math competition problems in English.
We also consider other non-mathematical reasoning tasks: ARC- ARC-Challenge and ARC-Easy
3
The ARC-Challenge benchmark represents a subset of 2,590 demanding science examination questions drawn from the broader ARC (A12 Reasoning Challenge) B4] collec- tion, specifically selected because traditional information retrieval and word co-occurrence methods fail to solve them correctly. This challenging evaluation benchmark features exclusively text- based, English-languag, multiple-choice questions (typically with four possible answers) spanning diverse grade levels, designed to assess science reasoning capabilities rather than simple pattern matching or information retrieval. The complementary ARC-Easy [4]] subset contains 5197 questions solvable through simpler approaches. We use 1.17k test split for ARC-Challenge evaluation and 2.38k test split for ARC-East -asy evaluation, respectively.
ARC-Challenge/Easy
B.4 More Training Details
For DeepSeck-RII -DDsttill-- -is---iilll---ee---... 1.5B, we let the maximum response length be 8192, following the setup of stage in DeepScaleR [8]. The learning rate set to le-6. The coefficient of weight decay is set to 0.01 by default. We store the model checkpoint every 20 steps for evaluation, and use 8 A100 2-t.--.--hh--....
GPUs for each experiment. For and DeepSeek-RP - --Disiill--wee- 1.5B, we train for 2000, 1000, 1000, and 1200 steps, respectively, unless the model has already shown a significant drop in performance. We use the same approach as DeepScaleR [18 (whose repository is also derived from the verl) to save the model in safetensor format to facilitate evaluation.
5-en2.5-MMth--1
Llama-3.2-2 Instrrcct,
B.5 More Evaluation Details
In evaluation, the maximum number of generated tokens be 3072 by default. For Qwen- based models, we use the "qwen25-matie math--otttt prompt template in evaluation. For Llama and
26

---

Table 7: Difference between model downloaded from Hugging Face and initial checkpoint saved by verl/deepscaler pipeline. Since the performance of stored initial checkpoint has some randomness, we still use the original downloaded model for recording initial performance.
MATH
AIME24
AMC23
Minerva Math
Olympiad-
AIME 2025
Model
Avg.
500
2024
2023
Bench
Qwen2.5-Mann Math- Math-1..h 1.5 [24
Hugging Face Model
36.0
6.7
28.1
8.1
22.2
4.6
17.6
Stored Initial Checkpoint
39.6
8.8
34.7
8.5
22.7
3.3
19.6
Swen2.5-Math-. (24)
Hugging Face Model Stored Initial Checkpoint
12.1
35.3
18.2
22.4
52.0
14.6
36.6
12.1
18.1
4.2
22.9
-lama-332-33--nnsttte
22
Hugging Face Model
40.8
8.3
25.3
15.8
13.2
1.7
17.5
Stored Initial Checkpoint
41.0
7.1
28.4
16.9
13.0
0.0
17.7
distilled models, their original chat templates. We set the evaluation seed to and top_p to 1 by default. For evaluation on DeepSeek-F RI-Distill-Qwen 1.5B, following DeepSeek-RR R1 [2] and DeepScaleR (18), we set the temperature to 0.6 and top_p P to 0.95, and use avg816 6 for MATH500, Minerva Math, and OlympiadBench, and avg064 for AIME24, AIME25, and AMC23, Since our training length 8992, we provide results for both 8192 (8k) and 32768 (32k) evaluation lengths (Appendix C.I.6). l. By default, we report the performance of the checkpoint that obtains the best average performance on 6 benchmarks. But in Sec. 3.2.3 and Sec. 4.1 since only evaluate MATH500 and AIME2024, we report the best model performance on each benchmark separately, i.e. the best MATH500 checkpoint and best AIME2024 checkpoint can be different (This will not influence our results, in Tab. and Tab. I1 still obtain similar conclusions in main paper.) We use GPUs for the evaluation. Finally we mention that there are slightly performance difference on initial model caused by numerical precision, but it does not influence our conclusions (Appendix (.66)
B.6 Performance Difference on Initial Model
We mention that there is precision inconsistency between models downloaded from Hugging Face repositories and initial checkpoints saved by the verl/deepscaler reinforcement learning pipeline in Tab. This discrepancy arises from the verl/DeepScaleR pipeline saving checkpoints with float32 precision, whereas the original base models from Hugging Face utilize b precision.
The root cause appears to be in the model initialization process within the verl framework. The fsdp_workers,p py 8 file in the verl codebase reveals that models are deliberately created in float32 precision during intiializatinn, fp32. Otherwise, the optimizer
noted in the code comment: "note that we have to create model in bf16, which is incorrect". This design choice was likely made to ensure optimizer stability during training. When examining the checkpoint saving process, the precision setting from initialization appears to be preserved, resulting in saved checkpoints retaining float32 precision rather than original bfloat16 precision
base model.
Our empirical investigation demonstrates that modifying the torch_ dtype parameter in the saved config. json file to match the base model's precision (specifically, changing from float32 to bfloat16) successfully resolves the observed numerical inconsistency. Related issues are docu- mented in the community
g and adopt the default settings of the verl pipeline in experiments.
27

---

Table Detailed 1/2-shot RLVR performance for Qwen2.5-Math-1..5 Results are reported for
the checkpoint achieving the best average across 6 math benchmarks (Fig. IT. Models' best individual
benchmark results listed in Tab. Format reward (Appendix C.2.3 serves as baseline for
format correction.
RL Dataset/ Method
Dataset
MATH
AIME
AMC
Minerva Math
Olympiad- Bench
ATME 2025
Avg.
Size
500
2024
2023
NA
NA
36.0
6.7
28.1
8.1
22.2
4.6
17.6
MATH
7500
74.4
20.0
54.1
29.0
34.1
8.3
36.7
DSR-sub
1209
73.6
17.1
50.6
32.4
33.6
8.3
35.9
Format Reward
1209
65.0
8.3
45.9
17.6
29.9
5.4
28.7
{
72.8
15.4
51.6
29.8
33.5
35.0
[ma)
73.6
53.8
23.5
35.7
10.8
35.7
(TT1, m13)
74.8
17.5
53.1
29.4
36.7
7.9
36.6
Table 9: Detailed 1/2/4-shot RLVR performance for Qwen2.5-Math-1.55 Here we record model's best performance on each benchmark independently. "Best Avg. Step" denotes the checkpoint step that model achieves the best average performance (T)..
RL
Dataset Size
MATH
AIME AMC Minerva Olympiad- Bench
AIME 2025
Best Avg.
Avg.
Dataset
500
2024
2023
Math
Step
NA
NA
36.0
6.7
28.1
8.1
22.2
4.6
17.6
MATH
7500
75.4
20.4
54.7
29.8
37.3
10.8
36.7
2000
DSR-sub
1209
75.2
18.8
52.5
34.9
35.1
11.3
35.9
1560
[m)
74.0
16.7
54.4
30.2
35.3
9.2
35.0
1540
[m2)
70.6
17.1
52.8
28.7
34.2
7.9
33.5
320
[Ti3]
74.4
17.1
53,8
25,4
36.7
10.8
35.7
2000
[T100]]
71.4
16.3
54.4
25.4
36.2
10.0
33.7
1120
[T1200]
72.2
17.5
50.9
27.6
34.2
8.8
33.5
1220
[Fi. mia]
76.0
17.9
54.1
30.9
37.2
10.8
36.6
1980
(TT), m2. Mis. T2209
74.4
16.3
56.3
32.4
37.0
11.3
36.0
1880
Table 10: Results more models (base and instruct versions) and more training examples (on
Qwen2.5-Math-77B) We record results from checkpoints achieving best average performance. Test curves in Fig. [0 and Fig. II
Analysis in Appendix C.1.2) We can see that on Qwen2.5-Math- 7B, different examples have different performance for 1-shot RLVR.
RL Dataset
Dataset
MATH
AIME
AMC
Minerva
Olympiad- Bench
AIME 2025
Avg.
Size
500
2024
2023
Math
Qwen2.5-1.52 (24)
NA DSR-sub
NA
3.2
0.4
3.1
2.6
1.2
1.7
2.0
1209
57.2
5.0
30.3
17.6
21.2
0.8
22.0
(mi)
43.6
0.8
14.4
12.9
17.6
0.4
15.0
(TT, T2, T13, 12209
46.4
2.9
15.9
14.0
19.0
0.8
16.5
(m.,.... (o....,7aa
16
53.0
3.8
30.3
19.1
19.6
0.0
21.0
Qwenn2.5-Math--555--nnss
(25)
NA
NA
73.4
10.8
55.0
29.0
38.5
6.7
35.6
DSR-sub
1209
75.6
13.3
57.2
31.2
39.6
12.1
38.2
(T))
74.6
12.1
55.3
30.9
37.9
12.1
37.1
Qwen2.5-Mathh
(25
NA
NA
51.0
12.1
35.3
11.0
18.2
6.7
22.4
sSR-sub
1209
78.6
25.8
62.5
33.8
41.6
14.6
42.8
[Wa]
79.2
23.8
60.3
27.9
39.1
10.8
40.2
Taoss
77.4
20.4
59.4
23,9
39.0
10.8
38.5
(00)00)
76.4
16.2
55.0
30.9
41.0
5.4
37.5
{TT1 -7..
16
77.8
30.4
62.2
35.3
39.9
9.6
42.5
28

---

Table 11: (shot RL still works well for different model with different scales. Here we
record model's best performance each benchmark independently.
RL Dataset
Dataset Size
MATH 500
AIME
AMC
Minerva Math
Olympiad- Bench
AIME 2025
Avg.
2024
2023
Qwen2.5 Owen2.5-Math-71 Matth-BB
24
GRPO
NA DSR-sub
NA
51.0
12.1
35.3
11.0
18.2
6.7
22.4
1209
81.0
34.6
64.6
39.7
42.2
14.6
42.8
{
79.4
27.1
61.9
32.7
40.3
11.7
40.2
[. Ti3
81.2
23.3
64.1
36.0
42.2
12.1
41.3
1T W2, T13. T1209
80.0
26.2
64.4
37.9
43.7
14.6
42.5
Random
16
78.0
24.6
63.1
36.8
38.7
14.2
40,2
[T....., m16
16
79.2
30.4
62.2
37.9
42.4
11.7
42.5
Llama-3.2-33 1a-ma-3.2-3B-Instrree
(26)
GRPO
NA DSR-sub
NA
40.8
8.3
25.3
15.8
13.2
1.7
17.5
1209
45.4
11.7
30.9
21.7
16.6
11.7
19.8
[Ti] [mT, T133
46.4
8.3
27.5
19.5
18.2
1.7
19.0
49.4
9.2
31.6
20.6
20.0
2.1
21.0
(T W2, /T3s T1209
48.4
9.2
29.4
23.5
17.6
1.7
19.8
Qwen2.5- Mrre2.5--Mahh-1l 1.5B (24)
PPO
NA
NA
36.0
6.7
28.1
8.1
22.2
4.6
17.6
DSR sub
1209
73.8
21.2
52.8
32.4
36.3
10.4
35.4
[wi)
16.7
53.8
28.3
34.1
9.2
33.8
Clatt Pallbityy
THOSK Aggiss
122
200
Figure 6: Different data have large difference on improving MATH500 accuracy, but they all improve various tasks rather than their task. From left to right correspond to 1-shot RI. on Ti. Ti1 1i3. or T16. Details are Tab.
29

---

MATTH 500 owap
NME 2024 Lnvegget
AMC 2023 Couase
FOMP3 (60)
10
fou tot
W w
1. 15
Lax 060
2isher tran
oher (mx)
thet ln?
Steps
Steps
Staps
(ouge11)
Clydeiaadenccs (augre))
(ougge)
1.24
chat (m) mnd fmya
chet dos ful
chat (cty 4t3 Days
250
500
2300
1750 1008
300
750
2300
125
2000
1753
2308 Slaps
1250
2000 1750
Stups
Slops
Average benchmaris
15.7
Dokaub
22.5
Lakek (ha)
Figure 7: Detailed results for RLVR on Qwen2.5-Math-1.55
MATT1 500 (eve
AIME 2024 (eveg
AMC 2023 (avrgl
60
10
13 OShass
2 ooliest
DiDaS that rull Occ
chat Co
that
dhat Oel
Staps
Slaps
Slaps
Minerra (e
Oity Mlymppaddeenne Caveert
leve
40
ind)
anst OE Rimd
10
Eshut No (les)
shot Rim no)
that
460
060
1068
250
466
200
Stops
Slaps
Benchmarks
20
BoBosb
aho
shod Lh,
Steps
Figure 8: Detailed results for RLVR on Owen2.5-Math-7B.
30

---

MATTH (varga11)
AIME 2024 Inwrgaat))
1 12
40
Au Plirt
-3
-0
LaceSRouk
Jaco
Lal OShvab
20
20
10
1000
250
3000
270
3700
Minerin Narth Levega111
Diympiaddenes 1evgga
NME 2025 Ieve (1
22.3
shot nu)
33.0
Dohkub thet
100-
32-05Back cnat too
1.2
dut
Stags
Slops
Slops
Avurage benchemariss
DAR oi
chi
230
1000
Figure 9: Detailed results for RIVR Llama-3.2-3B- 3.0n---3--BB.sstrrcc.....
MATTH SDD de
ABIE 2004 (evogEo)
AMC 2023 (evo
10
26 DNL
Acoracy
10
1/N
Aon
City Homm
Col
Slaps
S0aza
Stopo
ClympladBench (avge2))
Recunay INd
1aler Ceo
10
shor Cos.
Row!
202
100
2200
1200
1100
1753
1060
252
100
100
1950
190 2000
200
5.00
190
1330
1050
1550
1793
1550
Average benchmarks
22.5
20.0
12.5
20.3
Ihu shut nu rumm
5.
23
1.tl mm)
Figure 10: Detailed results for RLVR on Qwen2.5-1.5B. The gap between 1-shot RLVR and full-set RLVR is larger, but the 1-shot RLVR still improves lot from initial model and 16-shot RLVR behaves close to full-set RLVR.
31

---

NHE 2024 Imaggat)
2023 (HoggaR)
chht (N))
5 K
shot (N))
S4C/000 144
1 05Raab
72
31456
ALCOPI 2611
Acurrcyy 2611
10
26
300
400
803
1003
1200
3400
350
400
666
633
2000
1200
1400
200
400
060
600
2400
1230
3400
Average Bernccmmaa
1090
Stegs
Figure 11:
Detailed results for RIVR on Qwen2.5-Math-11.3. I.Ststrrtt... Interestingly, 1-shot
RLVR is more stable than full-set RLVR here.
C
Evaluation Result
C.1
Main Experiments
C.1.1
Detailed performance on Qwen2.5-Math--15B.
In Tab. 8 we show the detailed performance that shown in Fig. 1 Results are reported for the checkpoint achieving the best average performance.
C.1.2 Detailed Performance on More Models and Training Examples.
In Tab. 10 also show the I(few)-shot RLVR results the base model (Qwen2.5-1.5B 24 and
instruction model (Qwen2.5-Maie 1.rthhh1.5B-Bii Instrutt (25)). More detailed test curves are shown Fig.
and Fig. [1 We can see that (1) for Swen2.5-11.B. the gap between 1-shot RLVR with and full-set
RLVR larger, but the former still improves model performance significantly (e.g., MATH500: 3.2% to 43.6%), and 16-shot RLVR works very closely to full-set RLVR. (2) for Qwen2.5-Math-I133 Instruct, both full-set RLVR and 1-shot RLVR have limited improvement as the initial model already has good performance. Interestingly, shown in Fig. stable than full-set RLVR.
we observe that 1-shot RLVR more
Besides, we also consider other single training examples like Teos and ग on Qwen2.5-Math- We can see that they behave relatively worse than Tı and 16-shot RLVR provides more consistent approach closing the performance gap relative full-set RLVR.
C.1.3 Detailed performance with best per-benchmark results
Here, we In Tab. g we present the detailed [(few)-shot RLVR results for Qwen2.5-Math- record the model's best performance on each benchmark individually, so their average can be higher than the best overall average performance ("AAg.").. We include these results estimate the upper limit of what the model can achieve on each benchmark. Additionally, we include several examples well Ti or T13. still demonstrate significant improvements, such as T2, 2010 and π 12099. We observe that, in general, better results correspond to a larger checkpoint step for best average performance, which may correspond to a longer post-saturation generalization
that, while not performing
32

---

process. Similarly, in Tab. I1 we also include the best per-benchmark results for Owen2.5-Math-1 Llama-3.2-3B-Instruuc respectively, together with Qwen2.5-Math-1.5B with PPO training.
C.1.4 Detailed Test curves on MATH500 for 1-shot RLVR on Qwen2.5-Math-11.5.
We plot the performance curves for each subject in MATH500 under 1-shot RLVR using different mathematical examples. As shown in Fig. []] the choice of example leads to markedly different improvements and training dynamics in 1-shot RLVR, highlighting the critical importance of data selection for future few-shot RLVR methods.
C.1.5 Detailed RLVR results eacn benchmark training process.
To better visualize the training process of RLVR and compare few-shot RLVR with full-set RLVR. we show the performance curves for each benchamrk on each model Fig. 7
8
9
It will be
interesting to see that if applying ((few)-shot RLVR for more stable GRPO variants [
12 16
also note that -lama3.2-3B-Instrre is more unstable during training, as almost setups start having performance degradation before 200 steps.
alleviate this phenomenon. In addition the conclusions discussed in Sec. 3.3
In Appendix C.1.2 we also test the base model and instruction version models in Qwen family. Their test curves also shown in Fig. and Fig.
C.1.6 More Evaluation on
In Tab. 12 we show the DeepSeck- -.e---R---iiiii ---iitilllQQeen- 1.5B results at 8k and 32k evaluation lengths. The experimental setup is illustrated in Appendix B.3
Table 12: DeepSeek- DeepSeck-RI-Distllle RR--Distill-ween---- results at 8k and 32k evaluation lengths. Setup details
are in Appendix B.3
"8k > 16k +24k" denotes the length extension process in DeepScaleR training.
RL Dataset
Train Length
MATH
AIME
AMC
Minerva Olympiad- AIME
Avg.
500
2024
2023
Math
Bench 2025
Eval Length 8k
NA sSR-sub
NA
76.7
20.8
51.3
23.3
35.4
37.9
84.4
30.2
68.3
29.2
45.8
26.7
47.4
DeepScaleR (40k DSR)
16kk
86.3
35.2
68.1
29.6
46.7
28.3
49.0
[m]]
8k
80.5
25.1
58.9
27.2
40.2
21.7
42.3
(S,, Na, TT3, T (M.,.... N16
81.2
25.8
60.1
26.8
40.4
22.0
42.7
83.3
29.6
64.8
29.3
43.3
22.8
45.5
Eval Length
32k
NA
NA
82.9
29.8
63.2
26.4
43.1
23.9
44.9
DSR-sub
8k
84.5
32.7
70.1
29.5
46.9
27.8
48.6
DeepScaleR(40k DSR)
8k
16k -24
87.6
41.4
73.2
30.6
49.6
31.3
52.3
(m])
8k
83.9
31.0
66.1
28.3
44.6
24.1
46.3
[T,, T2, T13, 1
8k
84.8
32,2
66.6
27.7
45.5
24.8
46.9
[m,,..., 11
8k
84.5
34.3
69.0
30.0
46.9
25.2
48.3
C.2 Analysis
C.2.1 Test Curves for Ablation Study
the test curves for ablation study (Sec. 4.1 We that policy gradient loss is the main contributor of 1-shot RLVR. More discussions about format fixing are in Appendix C.2.3
In Fig.
C.2.2 Entropy loss
Detailed results of entropy-loss-only training. As in Sec. 4.2 loss-only training in Tab. 13 Training with only entropy loss for a few steps can improve model performance on all math benchmarks except AIME2025. The test curves are in Fig. [2] Notice that the improvement of catropy-loss-one training Owen2.5-Math- 1.5h is similar to that of RLVR with
we show the full results of entropy-
33

---

MATH 500 (avg@1)
AIME 2024 (avg@
15.0
60
Accuracy
40-
+WD +XL
-W +
30
7.5
+PG +WD +KL
+PG +WD +Ent
+PG +
+PG +Ent
2.5
0-
+PG
0.0
+PG
1000
Steps
Steps
Figure 12: Test curves for ablation study. Here we consider adding policy gradient loss (PG), weight decay (WD), KL divergence loss (KI.) and entropy loss (Ent) by one for 1-shot RLVR training on Qwen2.5-Math-1 1.5t (Sec. (.1)). Especially for only-entropy training, the test performance quickly achieves 0 since too large entropy will result in random output, but before that, the model gets significant improvement from the first several steps, which is close the results format-reward RLVR training (Appendix C.2.3). More discussions are in Appendix C.2.3
Table 13:
Entropy loss alone with Ti can improve model performance, but it still underperforms compared to the Yormat-reward baseline (Appendix C.2.3).
MATH
AIME24
AMC23
Minerva Math
Olympiad- Bench
ATME 2025
Model
Avg.
500
2024
2023
Qwen2.5-Math-1155
36.0
6.7
28.1
22.2
4.6
17.6
+Entropy Loss, Train 20 steps
63.4
8.8
33.8
14.3
26.5
3.3
25.0
Format Reward
65.0
8.3
45.9
17.6
29.9
5.4
28.7
1lama-3.2-3B-Insti
40.8
8.3
25.3
15.8
13.2
17.5
+Entropy Loss, Train steps
47.8
26.9
18.0
15.1
19.5
Qwen2.5-Math-7B
$1.0
12.1
35.3
11.0
18.2
6.7
22.4
+Entropy Loss, Train steps
$7.2
13.3
39.7
14.3
21.5
3.8
25.0
Format Reward
65.8
24.2
54.4
24.3
30.4
6.7
34.3
format reward (Appendix C.2.3 Tab. 14), thus we doubt that the effectiveness entropy-loss-only training may come from format fixing, and we leave the rigorous analysis of this phenomenon for future works.
Discussion of entropy loss and its function in 1-shot RLVR. Notably, we observe that the benefit of adding entropy loss for 1-shot RLVR is consistent with conclusions from previous work (60) on the full RLVR dataset, which shows that appropriate entropy regularization can enhance generalization. although it remains sensitive to the choice of coefficient. We conjecture the success of 1-shot RLVR is that the policy gradient loss on the learned example (e.g.. π rl)) actually acts an implicit regularization by ensuring the correctness of learned training examples when the model tries to explore more diverse responses or strategies, as shown in Fig. 3 (Step 1300). And because of this, both policy loss and entropy loss can contribute to the improvement of 1-shot RLVR. We leave the rigorous analysis future works.
C.2.3 (Only) Format Correction?
As discussed GRPO [3], changing the template of (wwn2.5-Matte models can significantly affect their math performance. In this section, we investigate some critical problems: is (1-shot) RLVR doing format fixing? And if the answer is true, is this the only thing 1-shot RLVR does?
To investigate it, consider three methods:
(a). Applying format reward in RLVRR We first try apply only format reward for RLVR (i.e., if the verifier can parse the final answer from model output, then it gets reward no matter the answer is correct not, otherwise it gets O reward), considering both 1-shot and full-set. The results are shown Tab. 14, and the test curves are shown in Fig. 14 and Fig. respectively.
Notably, we can find that (1) Applying format reward to full-set RLVR and 1-shot RLVR behave very similarly. (2) applying only format reward is already capable of improving model performance
34

---

MATH 500 (1
AINE 2024 (neggat)
2023 (evgsii))
32.3
10
1.2c0Shund farmet woit
Lab colvub farmal soord
1.200honds zutame rourd
70
c.P. Fornar MANK Eatrrry JSA
2x D.hhub
253
500
150
3700
1250
3300
1750
2000
250
307
750
1850
3250
1500 3750 2000
233
500
150
3700
1250
2000
1750
2000
Ninerva Nath (wopgr1)
Dlympladienns 11
NIME 2025 (nvegges
Paa May
1.3 OsRoet fomsl
1.8 amaTe rward
70
20
1 Colleb farmat seare. Enccge Lein
1.28 farmat
1.2005toch farmal
1.2c sustait sward
Stups
Slaps
Slaps
Acurage on benchemriis
Cohradd formad remund Entspy Lan
24 CAR catraneeeeat
500
750
1000
1250
1520
1750
2000
Figure 13: Comparison between outcome reward and format reward for full-set RLVR with 1.2k DSR-sub on Qwen2.5-Math-1155.
17.5
AINE 2024 (wegdht))
70
15.0
temett
shet farmel reward
0
tu
dhot outien reard
253
Tha 3988 1850 2000
2.50
7.0
1820 1250 1920 3350 2000
3000 125 3900 Lho 2000
30
Ninerva Math (avgg)))
X 8
Dlympard (avgge1)
ABRE 2025 (avoggo)
ACOUVTY (40
ty Fermal meant relerrn
dhet farmat mevid aill salzoma rnvard
shht farmal moard
J.htt salismm revard
S1456
Slaps
5005s
Average on benchmaris
Armstmrandd dets, ouccems reAard
712 1000
1980 150 2000
Figure 14: Comparison between outcome reward and format reward for 1-shot RLVR with Ti on Qwen2.5-Math--1.5.
35

---

Table 14: RLVR with only format reward can still improve model performance significantly,
while still having a gap compared with that using outcome reward. Numbers with orange color denote the ratio of responses that contain "Xboxed-("" in evaluation. Here we consider adding entropy loss or not for format reward. Detailed test curves are in Fig. 13 and Fig. [4] We can see that: (1)
RLVR with format reward has similar test performance between 1.2k dataset DSR-sub and T1. (2) TI with outcome reward or format reward have similar \boxed<) ratios, but the former still has better test performance (e.g., +7.4% on MATH500 and +5.8% on average). (3) Interestingly, RLVR with DSR using outcome reward can fix the format perfectly, although it still has similar test
performance as 1-shot RLVR with गı (outcome reward).
Reward Type
Entropy
MATH
AIME
AMC
Minerva Olympiad-
AIME 2025
Dataset
Avg.
Loss
500
2024
2023
Math
Bench
NA
NA
NA
36.0
6.7
70%
28.1 8375
8.1 3,0%
22.2
4.6g1%
17.6 17.6
DSR-sub
Outcome
73.6 000
17.1
50.6
32.4
33.6 33.6,0%
8.3
35.9
50%
000s
300
100%
DSR-sub
Format
65.0
8.3
45.9
17.6
29.9
5.4 5,4%0%
28.7
.0%
DSR-sub
Format
61.4
9.6,7%
44.7
16.5
29.5 90%
3.8 879
27.6
9.99
(my)
Outcome
72.8 0738
15.4
51.6
29.8
33.5
7.1
35.0
92%
9778
02%5
98%
(T))
Outcome
68.2 177%
15.4
92%
49.4
95%
25.0
9.1%
31.7
91%
5.8
90%
32.6 6 90%
(T))
Format
65.4 06%
8.8
91%
43.8
38%
22.1
01%
31.6 00%
3.8
29. 92%
Format
61.6
8.3
94%
46.2
15.4
75.4
29.3
50%
4.6
27.6
86%
significantly (e.g., about 29% improvement on MATH500 and about 11% gain on average). (3) There is still significant gap between the performance of 1-shot RLVR with outcome reward using Ji and that of format-reward RLVR (c.g.. +7.4% on MATH500 and +5.8% on average), although they may have similar ratios of responses that contain "Xboxed()" in evaluation (More discussions are in (b) part). (4) In particular, forrat-reward RLVR is more sensitive entropy loss based on Fig. 14 and Fig. 13
Interestingly. we also note that the best performance of format-reward RIVR on MATH500 and AIME24 are close to that for 1-shot RLVR with relatively worse examples, for example, T7 and TI1 in Tab. This may imply that I-shot RLVR with outcome reward can at least work as well as format-reward RLVR, but with proper examples that can better incentivize the reasoning capability of the model, 1-shot RLVR with outcome reward bring additional non-trivial improvement. Appendix C.2.5 provides a prompt n|. which uses a sub-question of IT, as an example to support our claim here.
Rervv Mach (ag
Olerphedtench Leggg
ARI 2325 IRAOUER
Figure 15: Relation between the number of \boxed() and test accuracy. We can see that they have a strong positive correlation. However, after the number of \boxed enters plateau, the evaluation results on some evaluation tasks continue improving (like Minerva Math, OlympiadBench and MATH500).
(b) Observe the change of format in 1-shot RLVR. We then investigate how the output format of the model, for example, the number of (b changes in the 1-shot RLVR progress. The results are shown in Fig. 15], We can see that (1) the test accuracy is strongly positively correlated to the number of \boxed which matches our claim that format fixing contributes a lot to model
36

---

Table 15: 1-shot RLVR does not do something like put the answer into the \boxed(). "Ratio of disagreement" means the ratio of questions that has different judgement between Qwen- and QwQ-32B judge. Here we let QwQ-32D judged based on if the output contain correct answer, without considering if the answer is put the \boxed().
Step0
Step 20
Step 60
Step 500
Step 13300
Step 1860
Ratio of \boxed{)
59.6%
83.6%
97.4%
96.6%
96.6%
94.2%
Acc. judge Qwen-Eval
36.0
53.8
69.8
70.4
72.2
74.0
Acc. judge by QwQ-32B
35.8
57.2
70.6
71.8
73.6
74.6
Ratio of disagreement
4.2%
5%
1.2%
1.4%
1.8%
1.8%
Table 16: ти even performs well for in-context learning on Qwen2.5-Math-78. Here "Qwen official examples" are from Qwen Evaluation repository [25] for 4-shot in-context learning MATH500, and "Qwen official Example 1" the first example.
MATH AIME AMC Minerva Olympiad- AIME 500
Dataset
Method
Avg.
2024
2023
Math
Bench
2025
Qwen2.5-Man Swen2.5-Math--1.5
NA {m1))
NA RLVR
36.0
6.7
28.1
8.1
22.2
4.6
17.6
72.8
15.4
51.6
29.8
33.5
7.1
35.0
[m
In- Context
59.0
8.3
34.7
19.9
25.6
5,4
25.5
Qwen official 4 examples Qwen official Example
In- Context
49.8
1.7
16.9
19.9
19.9
0.0
18.0
In Context
34.6
2.5
14.4
12.1
21.0
0.8
14.2
Qwen2.5-Math-77
NA (m
NA RLVR
51.0
12.1
35.3
11.0
18.2
6.7
22.4
79.2
23.8
60.3
27.9
39.1
10.8
40.2
|m
In- Context
75.4
15,8
48.4
30.1
41.3
13.3
37.4
Qwen official 4 examples Qwen official Example
In- Context
59.2
4.2
20.9
20.6
24.4
0.8
21.7
In-Context
54.0
4.2
23.4
18.4
21.2
2.1
20.6
improvement in (a). but (2) for some benchmarks like MATH500, Minerva Math and OlympiadBench. when the number of \boxed-(} keeps relatively high ratio, the test accuracy these benchmarks is still improving, which may imply independent improvement of reasoning capability.
In particular, to prevent the case that the model outputs the correct answer but not in \boxed we also LLM-as-a-judge [6]] with QwQ-32B -2B [62] to judge if the model contains the correct answer the response. The results are shown Tab. 15 We can see based Qwen-Eval pipeline and L.M judger QwQ-3BB are very close, and as the ratio of \boxed{) increases, the test accuracy also increases, which implies that the number of correct answers exhibited the response also increases, rather than just putting correct answer into \boxed(()
the accuracy judged by rule-
Notably, we also observe that Qwen2.5 5atth models contain lots of repetition at the end of model responses, which may result in failure of obtaining final results. The ratio of repetition when evaluating MATH500 can be high as about 40% and 20% for Owen2.5-Math-L... Math-7B, respectively, which is only about 2% for Llama3.2-33 1.t.....---B-nttueeeeee This may result the large improvement of format fixing (e.g., format-reward RLVR) mentioned in (a).
and Qwen2.5-
(c) In-context learning with one-shot example. In-context learning 63 is widely-used bascline for instruction following (although it may still improve model's reasoning capability). this section, we try to see if 1-shot RLVR can behave better than in-context learning. Especially, we consider the official 4 examples chosen by Qwen-Eva Eval (25) for in-context learning, and also the single training
example nT- The results are shown Tab.
We can find that (1) surprisingly, T. with self-generated response can behave much better than Qwen's 's official examples, both for 1.5B and 7B models. In particular on Qwen2.5-MMth--77 in- context learning with n can improve MATH500 from 51.0% 75.4% and on average from 22.4% 37.4%. (2) Although in-context learning also improves the base models, 1-shot RLVR still performs better than all in in-context results, showing the advantage of RLVR.
37

---

Table 17: Influence of Random Wrong Labels. Here "Error Rate" means the ratio of data that has the random wrong labels.
Error
MATH 500
AIME 2024
AMC 2023
Minerva Math
Olympiad- Bench
AIME
Dataset
Avg.
Rate
2025
NA
NA
36.0
6.7
28.1
8.1
22.2
4.6
17.6
Qwen2.5-Math-1a. + GRPO
DSR-sub
0%
73.6
17.1
50.6
32.4
33.6
8.3
35.9
DSR-sub
60%
71.8
17.1
47.8
29.4
34.4
7.1
34.6
DSR-sub
90%
67.8
14.6
46.2
21.0
32.3
5.4
31.2
[m))
0%
72.8
15.4
51.6
29.8
33.5
7.1
35.0
Qwen2.5-Math-
PPO
DSR-sub
0%
72.8
19.2
48.1
27.9
35.0
9.6
35.4
DSR-sub
60%
71.6
13.3
49.1
27.2
34.4
12.1
34.6
DSR-sub {m1)
90%
68.2
15.8
50.9
26.1
31.9
4.6
32.9
72.4
11.7
51.6
26.8
33.3
7.1
33.8
In short, we use these three methods to confirm that 1-shot RLVR indeed does format fixing and obtains lot of gain from it, but it still has additional improvement that cannot be easily obtained from format reward or in-context learning.
C.2.4 Influence of Random Wrong Labels
In this section, deep learning is robust to label noise [64], and we want to see if this holds for RLVR. We try to randomly flip the labels of final answers in DSR-sub and see their performance. Here we randomly add subtract numbers within 10 and randomly change the sign. If it is fraction, we similarly randomly add or subtract the numerator and denominator.
investigate the label robustness of RLVR. It's well-known that general
The results are in Tab. 17 We can see that (1) changing 60% of the data with wrong labels can still achieve good RLVR results. (2) if 90% of the data in the dataset contains wrong labels (i.e., only about 120 data contain correct labels. and all other 1.1k data have wrong labels), the model performance will be worse than that for 1-shot RLVR with क (which only contains correct label!). This may show that RLVR is partially robust to label noise, but if there too many data with random wrong labels, they may hurt the improvement brought by data with correct labels.
C.2.5 Change the Prompt of T
Table 18: Keeping CoT complexity in problem-solving may improve model performance. Com- paring wi and simplified variant 7' (prompt: "Calculate 22048"), where we only keep the main step that Qwen2.5-Mathh 1.5B may make mistake on. We record the results from the checkpoint with the best average performance. For T/. the model's 's output CoT is simpler and the corresponding 1 1-shot RLVR performance is worse. The additional improvement of πं is relatively marginal compared with using format reward, showing the importance of the training example used 1-shot RLVR.
RL
Reward
MATH
AIME 2024
AMC
Minerva
Olympiad-
AIME 2025
Avg.
Dataset
Type
500
2023
Math
Bench
Qwen2.5
2-t.5---Mthh-aae 1.5B (24)
NA
NA
36.0
6.7
28.1
8.1
22.2
4.6
17.6
(m)) Simplified [m]]]
outcome
72.8
15.4
51.6
29.8
33.5
7.1
35.0
outcome
65.4
9.6
45.9
23.2
31.1
5.0
30.0
DSR-sub
Format
65.0
8.3
45.9
17.6
29.9
5.4
28.7
As discussed in Sec. 3.2.1 we show that the model can almost solve कं but sometimes fails in solving its last step: "Calculate 22048". We use this step itself as a problem (T(), and see how it behaves in 1-shot RLVR.. The results are in Tab. 18 Interestingly, we find that of significantly underper- forms and has only 1.3% average improvement compared with format reward (as illustrated in
38

---

Appendix C.2.3
(a)). We think the reason should be that although solving 2048 is one of the most difficult parts Ti, mh still needs other key steps solve (e.g., calculating &o from P kAVs given some values) that may generate different patterns of CoT (rather than just calculating), which may allow more exploration space at the post-saturation generalization stage and maybe better incentivize the model's reasoning capability.
C.3 Response Length
In Tab. 19 report the average response length the evaluation tasks. The response length the test tasks remains relatively stable compared to that on the training data.
Table 19: Average response length of Qwen2.5-Math-11.5 on evaluation tasks. We use the format- reward experiment (DSR-sub + format reward in Tab. (4] as the baseline to eliminate differences in token counts introduced by formats.
MATH
AIME24
AMC23
Minerva Math
Olympiad- Bench
ATME
Setting
Avg.
500
2024
2023
2025
Format Reward
689
1280
911
1018
957
1177
1005
1-shot RLVR w/ (step 100)
611
1123
939
1072
951
1173
978
s-shot RLVR w/ (step 1500)
740
1352
986
905
1089
1251
1054
RLVR w/ DSR-sub sub (step 100)
636
1268
874
797
954
1122
942
RLVR w/ DSR (step 1500)
562
949
762
638
784
988
780
C.4
Pass@8 Results
In Tab. 20
report the pass@8 results the evaluation tasks. Interestingly, we find that (1) 1-shot RLVR achieves comparable or even slightly better passæ8 performance (51.7(2) full-set RLVR (with 1.2k DSR-sub) exhibits a noticeable downward trend in pass08 performance after 200 steps, which consistent with recent findings that RLVR may sometimes degrade the passên performance (20
Table 20: Pass@8 results on 3 math evaluation tasks using S.wen2.5-Mat--1.35 We also include the performance RLVR with format-reward (as Table 19) as stronger baseline.
Setting
AIME24
AIME25
AMC23
Avg. (3 tasks)
Base Model
26.6
20.0
72.5
39.7
Format Reward(highest)
33.3
23.3
72.5
43.1
RLVR w/ DSR-sub (highest, step 160)
36.7
26.7
87.5
50.3
RLVR w/ DSR-sub (step 500)
33.3
30.0
82.5
48.6
RLVR w/ DSR-sub (step 1000)
33.3
20.0
75.0
42.8
RIVR w/ DSR-sub (step 1500)
30.0
26.7
67.5
41.3
I-shot RLVR (step 500)
30.0
16.7
80.0
42.2
1-shot RLVR (highest, step 980)
36.7
33.3
85.0
51.7
1-shot RLVR (step 1500)
26.6
23.3
87.5
45.8
D Discussions
D.1 Limitations Our Work
Due to the limit of computational resources, we haven't tried larger models like Qwen2.5-32B training currently. But in general, a lot of RLYR works conducted 1.5B and 7B models, and they already achieve impressive improvement on some challenging math benchmarks like OlympiadBench. so our experiments are still insightful for RLVR topics. Another limitation of our work is that we mainly focus on the math domain, but haven't 't tried ((few)-shot RLVR on other verifiable domains like coding. But we also emphasize that all math-related experiments and conclusions in our paper are logically self-contained and clearly recorded, to ensure clarity and avoid confusion for readers. And mainly focus on analyzing this new phenomenon itself, which already brings a lot of novel observations (e.g., cross-category generalization, post-saturation generalization, and more frequent self-reflection in 1-shot RLVR, etc.). We leave the few-shot RLVR other scenarios for future work.
39

---

PG Loss
0.25
0.20
0.15
0.10
0.05
0.00
-0.05
-0.10
Step
500
1k
1.5k
2k
Figure 16: The norm of policy gradient loss for 1-shot RLVR (Tr) on Qwen2.5-Math-115B.
In particular, note that main focus is propose
observation rather than propose new better method, noting that 1-shot RLVR doesn't save (and maybe requires more) RI. computation.
Besides, πi is not necessarily the best choice for 1-shot RLVR on other models, since it's selected based on the historical variance score of Qwen2.5-Math-11. may be more stable for training, as we have seen that on DeepSeek-B -1-eeek--R--sstiii -Dissill--Ween 1.5B (Tab. 4). Qwen2.5-Math-7B 5-Mathh-BB (Tab, 4 10) and Qwen2.5-1 555.BB (Tab. 10). RLVR with 16 examples (fT1,..., T16.
In general, using few-shot RLVR
works well as RIVR with .2k dataset DSR-sub and outperforms 1-shot RL. with nT.
D.2 Reasoning Capability of Base Models
The effectiveness of ((few)-shot RLVR provides strong evidence for an assumption people proposed
recently, that is, base models already have strong reasoning capability [3 20 (1). For example, Dr. GRPO 13 has demonstrated that when template is used, base models achieve significantly better downstream performance. Recent work further supports this observation by showing that, with respect to the passšk metrics, models trained via RLVR gradually perform worse than the base model as l increases (20). Our work corroborates this claim from another perspective, example provides almost no additional knowledge. Moreover, our experiments reveal that using very few examples with RLVR is already sufficient to achieve significant improvement on mathematical reasoning tasks. Thus, is worth investigating how select appropriate data to better activate the model during the RL stage while maintaining data efficiency.
single
D.3
Why Model Continues Improving After the Training Aceuracy Reaches Near 100 %
A natural concern of 1-shot RLVR is that if training accuracy reaches near 100% (which may occur when over- wver-training
example), the GRPO advantage (Eqn. (6) should be zero, eliminating policy gradient signal. However, entropy loss encourages diverse outputs, causing occasional errors (99.x% training accuracy) and non-zero gradients (advantage becomes large for batches with wrong responses due to small variance). This shows the importance of entropy loss the post-saturation generalization (Fig. (). Supporting this, Fig. [6 shows that for 1 I-shot RLVR training (π²) on Qwen2.5-Math- 1.5B, policy gradient loss remains non-zero after 100 steps.
D.4 Future Works
We believe findings can provide some insights for the following topics:
Data Selection and Curation. Currently, there are no specific data selection methods for RLVR except LIMR [9]]. Note that 1-shot RIVR allows for evaluating each example individually, it will be helpful for assessing the data value, and thus help to design better data selection strategy. What's more. noting that different examples can have large differences in stimulating LLM reasoning capability (Tab. 3). it may be necessary to find out what kind of data is more useful for RLVR, which is critical for the RLVR data collection stage. It's worth mentioning that our work does not mean scaling RLVR datasets is useless, but it emphasizes the importance of better selection and collection of data for RLVR.
40

---

Table 21: Details of example T13-
Prompt
Given that circle SC$ passea through points #P(0,-4)8, 8Q(2,0)8, and -188. (18 Find the equation circle SC3. (28 SIABI I-B, find the value of Saf. Let's think step by stop and output the final answer within (.bozed(.
1ine $1: mxey-1=08 intersecta circle points and $B8, and
Ground truth (label in DSR-sub); 4.
Table 22: Details example 72.
Prompt:
Hox nany positive divisors do 9240 and 13860 have common? Let's think stop by step and output the final ansver within (bozed(). Ground truth (label in DSR-sub): 24,
Understanding 1-shot RIVR and Post-saturation Generalization A rigorous understanding for the feasibility of I-shot LLM RLVR and post-saturation generalization is still unclear. We think that one possible hypothesis that the policy loss on the learned examples plays a role as "implicit regularization" " of RLVR when the model tries to explore more diverse output strategies under the encouragement of entropy loss larger rollout temperature. It will punish the exploration patterns that make the model fail to answer the learned data, and thus provide a verification for exploration. It's interesting to explore if the phenomenon has relevance to Double Descent [65] or the implicit regularization from SGD (66 67), as 1-shot RLVR T13 (Fig. 2 middle) shows test curve similar to Double Descent. We leave the rigorous analysis of this phenomenon for future works, and we believe that can help us to comprehend what happens in the RLVR process.
also highlight the importance of entropy loss in 1-shot RLVR, and note that a more thorough explanation of why training with only entropy loss can enhance model performance remains an interesting direction for future work (Sec. 4.2). Relatedly, entropy loss has also received increasing attention from the community, with recent works discussing its dynamics [68 47 60) or proposing improved algorithms from the perspective of entropy [46]. Moreover, we believe a broader and more important insight for these is that encouraging the model explore more diverse outputs within the solution space is critical, as it may significantly impact the model's 's generalization to downstream tasks (69] Adding entropy loss is merely one possible approach to achieve this goal and may not necessarily be the optimal solution. As shown in our paper and previous work [60], the effectiveness of entropy loss is sensitive to the choice of coefficient, which could limit its applicability in larger-scale experiments. We believe that discovering better strategies to promote exploration could further enhance the effectiveness of RLVR
Importance of Exploration. In Sec. 4.1
Other Applications. In this paper, we focus primarily on mathematical reasoning data; however. is also important to evaluate the efficacy of I-shot RLVR in other domains, such as code generation or tasks without verifiable rewards. Moreover, investigating methodologies to further improve few- shot RLVR performance under diverse catt-constrained scenarios represents a valuable direction. Examining the label robustness of RLVR, as discussed in Sec. 4.2 likewise merits further exploration. Finally, these observations may motivate the development of additional evaluation sets to better assess differences between 1-shot and full-set RLVR on mathematical or other reasoning tasks.
E Example Details
In the main paper, we show the details of π.. Another useful example Tı3 is shown in Tab. 21 Here we mention that ii is geometry problem and its answer is precise. And similar to πT the initial base model still has 21.9% of outputs successfully obtaining 4 in 128 samplings.
3
Besides, Tab, 22 through 42 in the supplementary material provide detailed information for each example used in our experiments and for all other examples in (- T17). ). Each table contains the specific prompt and corresponding ground truth label for an individual example.
41

---

Table 23: Details of example 33
Prompt:
There are 10 people who vant to chooge a commitiee people anong them. They do this by firet electing a set of $1,2,38, or committee leaders, who then choose the renaining people to complete the 5-person committes forned, asauning that: people distingutshabled (Tuo
comnitree. how nany vaya committees that have the same menbers but different sets of leaders are considered to be distinct.) Let?s Wboxed().
step by step output final
Ground truth (label in DSR-sub): 7560.
Table 24: Details of example :7.
Prompt:
Three integers from the 11at 81,2,4,8,16, 16, product 80. what the sum of these three integera? Let's think stop by step and output the final answer within (iboxed()
Ground truth (label in DSR-sub): 25.
Table 25: Details of example π5.
Prompt:
how many ways the following conditions hold? (a) Each row contains all four nunbera. (b) Each column containe all four munbers. (c) Each "quadrant" containa all four numbers. (The quadranta are the four corner $2 Vitinea 28 ansver (bboxed{).
nunbers from the 81V(1,2,3,41VB into 84 Vitimes 4f array that all
aquares.) Lethyss think step by step output
Ground truth (label in DSR-sub): 288.
Table 26: Details of example To.
Prompt:
The vertices of a $3 Wtiims Vttimes 1$ rectangular prism are $A, B, C, D, E, P, ds, and $8S so that SA E, B FS, $C Gt, and SD Similarly, points skt S0 N-M N-1l G-18, and pointa 80$ and $P8 are 6D 88 so that BD 0-D p-P 8-18. For distances. points through SP$, these 120 distances are equal to (2) Let's think stop by step and output the final ansver within ()boxed().
edges of length 3. Point $IS and point $J8 $L8 are on 88 so that X-K L-l F-18, points sut and s88 are on $C so that pair of the 16 nany
SA
that SA I-I J-J E-18.
computes distance batoeen
Ground truth (label in DSR-sub): 32.
Table 27: Details example n7.
Prompt:
Set $u_0 (1)) and for fk Vga 08 let $u_(<+1)$ be determined by the recurrence (ltu]k+1) 2u_k 2u_k2.AV sequence tende limit; call SL4. What is the leaet value of sut such that-an\//uux-tl Wi (tre0))))22-000007333 [1000]37\]] Let's think step by step and output the final ansver eithin Wboxad[[.
Ground truth (label in DSR-sub):
Table 28: Details example Tg.
Prompt:
Consider (2, 17, 22, 32WV.. Calculate number different integers expressed se the sun of three distinct menbers of this set, Let 's think step by step and output the final within Wboxed[).
Ground truth (label in DSR-sub): 13.
Table 29: Details of example mg.
Prompt:
In group photo, boys and 3 girls are to stand in row such that no tuo boys tvo girls stand next to arrangeneats are possible? think step by step output anever within Wbozed()
Hov many
Ground truth (label in DSR-sub): 144.
42

---

Table 30: Details of example T1o-
Prompt:
How nany ten-digit rumbers oxist in shich there are at least two identical digits? Let's think step by step and output the final ansver vithin Viboxed().
Ground truth (label in DSR-sub); 8996734080.
Table 31: Details of example T11
Prompt:
How many pairs integers Sut and $b$ are there such that $a8 and Sb are between $18 and $42$ and Ba"g = b"Y Anod 438 7 Letta think atep step and output the final answer within (.boxed
Ground truth (label in DSR-sub); 42.
Table 32: Details of example T12-
Prompt:
springs stiifnesses W,, (itext(al)) Vitort(n)8 W, (xt) / Weextt(n)8 connected in series. How mach work is required to stretch this system by 10 em? Let's think step by stop and outgut
Wbbozed().
Ground truth (label in DSR-sub):
Table 33: Details of example T14-
Prompt:
Seven carde numbered $18 through $7$ are to be lined up in zow. Find the number of arrangements of theee leaving renaining descending order. Let's think step by step and output the final ansver within Viboxed(..
either ascending
Ground truth (label in DSR-sub): 74.
Table 34: Details of example Ti5.
Prompt:
What the enclased the geoboard quadrilateral belovf\n [asy] unitaiza(am); defaultpen (1inewidhh.. { b));; ); dotfactor=2; for a=;; ac-10; 4*) for be0; bc-10; ++b) final anaver within Wboxed().
(1-
[/asy] Let's think step by step and output
Ground truth (label in DSR-sub): 22
Table 35: Details of example m16-
Prompt:
If $p, 9,8 and $r3 are three nen-zero intagers such that Sp 26$ and Andaa((Atrac(1) Werac(1Hg)) Virrac()<E)) (hfrac(360/pgrr) 1,\W compute $pgr6. within W)bexed().
step by step output final ansver
Ground truth (label in DSR-sub): 576.
Table 36: Details of example m17-
Prompt:
In Class 3 (1), consisting of 45 students, all studenta participate in the tug-of-xar. For the other thres events, student participates shuttlecock kicking competition and 26 students participate in the basketball shosting competition. Hov many
least event. studenta participate all three
39 students participate
think step by step output
Wbozed()
Ground truth (label in DSR-sub):
43

---

Table 37: Details of example 6005
Prompt:
Given vectora 88) boarrrgghtrree (n)=( Vagrt (3Masi x+llcos x,1), Moverrightarros (n)=(A)coa xx), Maverrightarror (n) (pprp Noverrightarro (n)) Find the nonatonic intervals of (f));; (2) diven that $A$ is an internal angle Siltringgle ABCE, and SSfV\left( Vifrac (rigghhr))))) Virrac (1) Wrra Megrt (3))2), and output the final ansver within Wboxed().
Viagrt (2)88, find the ares of WWtriingge ABCS, Let's thick step by step
Ground truth (label in DSR-sub): /3-1
Table 38: Details of example mene-
Prompt:
How nany zeroa are at the and of the product W a(1) Wedet s(2) Wodor Widota Wedot a(100) W. where W a(n). WV denates cun digits output the final ansver vithin (boozed().
number W AV? Let's step by stop and
Ground truth (label in DSR-sub): 19.
Table 39: Details of example T1201
Prompt:
The angles of quadrilateral SPQRSB satiefy Valangle P 3\Vangle q 4Vaang 6Vaangl 3$ What is the degree measure 8Vlangla
step by stap outpat ansear within Wbooadd].
Ground truth (label in DSR-sub): 206.
Table 40: Details of example T1207. A correct answer for this question should be 2/3.
Prompt:
A rectangular piece paper whose length is (V)sgrt(3) tines the width has area $A8. The paper is divided into three equal sections along the opposite lengths, and then dotted 1ine dravn from the first divider the second divider on the opposite side as shown The paper ia then folded flat along this dotted line create new shape with area $8$. What is the ratio (V))rac(B))(A))7 Let?s think by step and output the ansver
Wiboxed().
Ground truth (label in DSR-sub): E.
Table 41: Details of example T1208-
Prompt:
Given a quadratic function in terme of WW&HHH), (20 ANHl1((1) Let set
randonly pick number from set WWV(--WW 2,,11, and set WIVQQVIV WIllbilIV) calculate probability WWVPUWV as WWGGII function (111( is increasing in the interval ((I,=1liiinft))11) (2(11((()111))) Suppase point (iVV((,
region defined by III Hilbegin(cases) 2ry-8lillleglaant denote (110--1110--(() WWAAAW the other less than ИИА(И))11) calculate the probability occurring. Let's think stop by step and output the final ansver zithin (lboxed().
randon point within (Mlleedfcaees)ii
W11III > O (1HII
has two geros, one greater
Ground truth (label in DSR-sub);
Table 42: Details of example T2209-
Prompt:
Define the derivative of the $th derivative the Snšth derivativa S(n Win N- Vigeqslant 2)$,
that is, (z-(()))))) (f) 's They are denoted as (x) (x),, (x), .... (x). If (x (x)(, then the $2025$rd derivative of the function (x)) the point s(c, (={(2623)> has a Sy8-interceps on the fx3-axia of -....ur Letta think step by atep and output the final anaver within ()ooxdd{).
Ground truth (label in DSR-sub):
2023
44


Reviews:
[{"review_id": "jQxudbwhm8", "reviewer": "Reviewer_yG1q", "summary": "This paper studies the importance of training set size for GRPO training with the goal of improving reasoning capabilities of the models. Interestingly, the paper shows that there are individual samples (in training sets) where training with that individual sample alone would almost reach the same performance as training with the whole dataset. Authors attribute this observation partly to the increase in self reflective behaviors of the model even when a sample is learned.", "questions": "- 1. In GRPO training, if all rollouts receive reward 1 there will be no update from the policy gradient which happens to be the case after the first 200 iterations. So I wonder maybe we can get the same results with a combination of fixing the format (probably the first 100 iterations with 1 or more samples) and then just entropy training? (I.e., there is nothing coming from the policy update in post-saturation right?)\n- 2. Why self reflection doesn’t increase when we train on the full data? How can one boost that?\n- 3. What response length did you use for the 7B and 3B models?\n- 4. Do you think there will be a way to select the data without training first for E epochs? In the paper do you use the historical variance score of Qwen2.5-Math-1.5B for all the models?\n- 5. Can you elaborate on \"To enable RLVR with one or few examples, we duplicate the chosen data until they reach the training batch size (e.g., 128) and store them as a new dataset.\"?\n- 6. How do you report accuracies for the experiments, because the accuracy is quite unstable across different iterations?", "limitations": "discussed in the paper", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\nThe paper is full of interesting observations and reports. For instance there is post-saturation generalization, even when a model achieves perfect accuracy on a single training sample, further training on the sample would still improve test performance. Subsequently, the paper shows the importance of the entropy term in boosting the performance of models. \n## Weaknesses\n- The explanations provided for the current observations are rather preliminary and there are still many unanswered questions (many of them may become future research questions, see below for some). \n- As authors have pointed out, a huge part of the gains are only due to format fixing. The gains due to format fixing may cause misinterpretation in the results of the paper. Here are some details. In Table 12, we observe that format fixing can increase model's performance to ~29% on the average score and to 65% on MATH. In Table 3, the 'performance increase' of several individual samples have been reported. We see that for samples {4, 7, 11} the performance reaches 65% after GRPO training and this is reported as 28% increase in performance. However, if we just compare this with format fixing, we can conclude these trainings are actually not doing anything beyond format fixing, but unfortunately the current presentation could be misleading. There is also another subtle phenomenon. The reader may compare performance increase (for MATH dataset) with the full set (39.4%) and the performance increase from one sample (38%) and concludes that they are almost the same. However, if we consider the baseline that only fixes the format (which already has 29% performance increase), we reach performance increase of 10.4% for the full training set vs. 9% for an individual sample. The point is there's much more difference between 10.4% and 9% compared to 39.4% and 38%. This baseline issue goes for most tables and figures presented in the table. \n\n\nI'd like to be extra clear. I think the paper's results are still significant and very interesting. However, I think they are slightly misleading in the current form, hence I will be happy to increase my score if the authors take proper actions to fix this issue. (One suggestion could be including format-fixing training as a baseline in tables and figures).", "was_revised": true, "final_justification": "I also believe this paper provides important insights into the role of RL in increasing the reasoning performance of language models. We do not yet fully understand, how one sample can be that effective. I also believe there might be scenarios (datasets + models) in which full RL training is required. Nevertheless, I think the community would benefit from this paper.", "rebuttal": "Thank you for your recognition of our work and your constructive feedback to improve our paper. We will revise our paper based on your advice. We detail our response below and please kindly let us know if our response addresses your concerns.\n\n> **Q1**: The explanations provided for the current observations are rather preliminary and there are still many unanswered questions\n\n**A1**:\nAs you mentioned, in our paper, we mainly focus on empirical findings.\nWhile more theoretical explanations would certainly be valuable, it would also require significant additional effort due to the complexity of the problem, especially if a rigorous proof is to be pursued. Such analysis could become an independent contribution and is difficult to fully include within the limited space of this rebuttal. Therefore, we believe it is reasonable to leave the theoretical investigation as future work.\n\nIn fact, we also discuss possible explanatory directions in Appendix D.3 (Lines 838–846), suggesting ideas for future research. For example, if a specific training example can encourage the model to output diverse Chain-of-Thoughts (CoTs) that generalize to other problems—especially when aided by entropy loss—then the policy gradient might serve as an implicit regularizer on these exploratory CoTs (e.g., punishing the wrong exploration). This could help the model leverage capabilities (e.g., diverse reasoning patterns) acquired during pretraining and midtraining. Additionally, it might be possible to draw on techniques used in analyzing double descent (note that the 1-shot RLVR on $\\pi_{13}$ shows a test curve somewhat similar to “double descent,” as in Fig. 2 middle), which is related to the implicit regularization effects of SGD, to better understand the behavior of 1-shot RLVR. We will make these points more clear in the main paper.\n\n> **Q2**: As authors have pointed out, a huge part of the gains are only due to format fixing… One suggestion could be including format-fixing training as a baseline in tables and figures\n\n**A2**: Thank you very much for bringing this up! We fully agree with your suggestion to prevent potential misinterpretation of format-fixing gains. To better disentangle the effects of format-fixing and non-format improvements, we plan to revise the main paper to include additional clarifications and baselines. Specifically, we will at least consider:\n\n- More clearly distinguish format and non-format improvements in abstract and introduction.\n- Add a baseline to Fig. 1, 6 and Tab. 3, 5, and 6, along with corresponding discussions. For example, as you suggested, we will revise the improvement values in Table 3 and show that some data do not yield improvements beyond format-fixing.\n- Include baselines in Table 4 for 1.5B and 7B. We omit Llama-3.2-3B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B, as these models already perform well with the default chat template due to instruction tuning and distillation. The format-reward baseline for Qwen2.5-Math-1.5B is in Sec. C.2.3, and that for Qwen2.5-Math-7B (new results) is as below.\n\n**Table R1**: RLVR with format-reward for Qwen2.5-Math-7B. \n| |Dataset Size|MATH500|AIME24|AMC23|Minerva Math|OlympiadBench|AIME25 | Avg.\n|-|-|-|-|-|-|-|-|-|\nBase  | 0 | 51.0 | 12.1 | 35.3 | 11.0 | 18.2 | 6.7 | 22.4 |\n+format-reward | 1209 | 65.8 | 24.2 | 54.4 | 24.3 | 30.4 | 6.7 | 34.3 |\n**+1-shot RLVR** | 1 | 79.2 | 23.8 | 60.3 | 27.9 | 39.1 | 10.8 | 40.2 |\n\nWe will incorporate the above changes in the final version of the paper.\n\n\n\n> **Q3**: if all rollouts receive reward 1 there will be no update from the policy gradient which happens to be the case after the first 200 iterations. we can get the same results with a combination of fixing the format (probably the first 100 iterations with 1 or more samples) and then just entropy training? (I.e., there is nothing coming from the policy update in post-saturation right?)\n\n\n**A3**: The rollout accuracy after 200 training steps is very close to 100%, but importantly, still slightly below it (e.g., around 99%). This means that incorrect rollouts can still have a large advantage due to small variance (as Eq. 6 in Appendix B.1), while lots of correct rollouts gets a minor but non-zero advantage. As a result, the policy gradient updates remain important after 200 steps, as the gradient norm shown in Fig. 18. Related discussions are in Appendix D.2.1. \n\nActually, policy gradients are essential to penalize outputs that fail to yield correct answers. Without this, training with entropy loss alone quickly leads to excessive entropy and degenerate outputs (As shown in Fig. 14, gets 0 accuracy after 20 steps). Similarly, in Tab. 6 and Appendix C.2.2 (Lines 709–715), we mention that entropy-only training is only effective for the initial few steps. Therefore, we believe combining only format reward and entropy training is insufficient. Without policy gradients, the model may degenerate into generating random outputs containing “\\boxed{}”, lacking meaningful CoTs. We will clarify these more explicitly in the main paper.\n\n\n\n> **Q4**: Why self reflection doesn’t increase when we train on the full data? How can one boost that?\n\n\n**A4**: Thanks for the good question. We think this phenomenon likely stems from two factors:  \n(1) the max training response length, and  \n(2) varying response lengths across data (often tied to problem difficulty).  \n\nA good illustration of this can be found in DeepScaleR [3] (Fig. 2).\nWhen the maximum training context length is limited (e.g., 8k), excessive self-reflection may lead to overly long responses that are eventually clipped. Additionally, GRPO loss tends to favor shorter correct responses over longer ones [2]. So model may compress its CoT, reducing response length and self-reflection.\nIn contrast, longer context windows (e.g., 16K in DeepScaleR) enable effective but not excessively long self-reflections on more difficult problems—the model has more room to reason and can obtain higher rewards. In this case, the average response length tends to increase. \n\nIn our case, Qwen2.5-Math-1.5B has a 4096-token context limit, and with a 1024-token prompt, responses are capped at 3072 tokens. This constraint likely contributes to the drop in both response length and reflection. But generally, when using models with longer context windows, adjusting max training response length and data difficulty may better support self-reflection on challenging tasks.\n\n\n\n> **Q5**: What response length did you use for the 7B and 3B models?\n\n**A5**: For both Qwen2.5-Math-7B and Llama-3.2-3B-Instruct, we use a response length of 3072 for both training and evaluation (Train: Line 144–145, Eval: Line 628). \n\n> **Q6**: Do you think there will be a way to select the data without training first for E epochs? In the paper do you use the historical variance score of Qwen2.5-Math-1.5B for all the models?\n\n**A6**: Yes, for simplicity, we use the same historical variance score from Qwen2.5-Math-1.5B to select data for all models. Although this score currently requires training models before compute, developing more efficient data selection or scheduling methods is an interesting future direction (Appendix D.3, Line 830-837). Potential approaches include:\n\n1. **Proxy model**: Similar to us, use a smaller but appropriate proxy model to compute the historical variance score,and then be applied to larger models. This is more efficient than training each target model individually.\n\n2. **Learnability-based metrics[1]**: For example, one can select a suitable teacher model and compute the accuracy gap between the teacher and target models for each problem. This gap serves as a proxy for learnability and can be used for filtering.  This approach only requires a few inference passes over the training data, and we have found it performs reasonably well in real-world RLVR scenarios.\n\nAlthough designing a new data selection method is not the main focus of our paper, it would be valuable to explore whether historical variance score can be combined with efficient selection techniques. The historical variance score may also be useful for data scheduling in RLVR, where training data is scheduled based on prior training steps, potentially avoiding additional costs when calculating the score.\n\n> **Q7**: elaborate \"To enable RLVR with one or few examples, we duplicate the chosen data until they reach the training batch size (e.g., 128) and store them as a new dataset.\"\n\n**A7**: This is a technical choice that doesn't affect our conclusions. For consistency, we use the same batch size (128) in 1-shot RLVR as in full RLVR with DSR-sub. Since `verl` sets `drop_last=True` for the training dataloader, the dataset must be at least as large as the batch size; otherwise, all data would be dropped. So we duplicate the selected examples until the dataset reaches size 128. We will clarify this in the revised paper.\n\n> **Q8**: How do you report accuracies for the experiments, because the accuracy is quite unstable across different iterations?\n\n**A8**: By default, in main Tables/Figures (e.g., Fig. 1, 6; Tab. 4, 6, 7), we report the checkpoint with the best average performance across 6 math benchmarks (saved every 20 steps). For comparisons or ablations on only MATH500 and AIME24 (e.g., Tab. 3, 5), we report the best performance on each benchmark separately. These details are noted in the appendix (Lines 633–638) and captions of Tab. 3 and 5. All methods, including baselines, follow the same setting for fairness. \n\n[1] Mindermann, Sören, et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2]  Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective, 2025b. URL https://arxiv.org/abs/2503.20783.\n\n[3] Luo, Michael, et al. \"Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl.\" Notion Blog (2025)."}, {"review_id": "DTBgWUbngG", "reviewer": "Reviewer_bR1Q", "summary": "This paper demonstrates that modern large language models can be fine-tuned via reinforcement learning on a single automatically verifiable training example to achieve dramatic improvements in mathematical reasoning: for instance, lifting accuracy on the MATH500 benchmark from 36 % to 73.6 %—on par with methods using thousands of examples. Through extensive experiments across multiple base models (Qwen, Llama, DeepSeek), RL algorithms (PPO, GRPO), and ablations, the authors show that both reward-based and entropy-only objectives yield large gains, that these gains transfer across problem domains, and that test performance continues to improve long after the model perfectly fits the lone example. This “one-shot” RL approach challenges prevailing assumptions about data requirements for RL tuning in LLMs and opens new avenues for data-efficient, exploration-driven fine-tuning.", "questions": "I greatly enjoyed the paper and find the empirical findings persuasive. To help researchers and practitioners translate these insights into day-to-day RLVR workflows, could the authors elaborate on a few near-term, “actionable” takeaways? This is a general question, which I wish to learn from the authors.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Timely focus on data efficiency – Tackles the under-explored but important question of how little data RL-based verification–reward pipelines really need, pushing the field toward more resource-friendly methods.\n- Non-trivial findings – Demonstrates that training on just one verifiable example can unlock large gains, an observation that is both surprising and likely to inspire follow-up work.\n- Thorough experimental coverage – Evaluates multiple base models, RL algorithms, training recipes, and a wide range of ablations, giving the results credibility and making the paper a useful reference point.\n\nWeaknesses\n- Limited explanatory depth – While the empirical phenomena are intriguing, the paper offers only light, largely speculative explanations; a clearer theoretical or mechanistic account would strengthen the contribution, which I understand is a bit too much to ask for but would significantly strengthen the contribution.", "was_revised": true, "final_justification": "The authors have answered my question properly and I believe the contribution of this work is important for understanding RLVR in LLM.", "rebuttal": "Thanks a lot for your recognition of our paper together with your valuable comments and suggestions. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.\n\n\n\n> **Q1**: Limited explanatory depth – While the empirical phenomena are intriguing, the paper offers only light, largely speculative explanations; a clearer theoretical or mechanistic account would strengthen the contribution, which I understand is a bit too much to ask for but would significantly strengthen the contribution.\n\n\n**A1**: As you mentioned, in our paper, we mainly focus on empirical findings and demonstrate some necessary conditions for the success of one(few)-shot RLVR, such as the strong capabilities stored in the base model due to pretraining/midtraining stages (Lines 810–819), the importance of policy gradient and exploration, etc. \n\nWhile more theoretical analysis would certainly be valuable, it would also require significant additional effort due to the complexity of the problem, especially if a rigorous proof is to be pursued. Such analysis could become an independent contribution and is difficult to fully include within the limited space of this rebuttal. Therefore, we believe it is reasonable to leave the theoretical investigation as future work.\n\nIn fact, we also discuss possible explanatory directions in Appendix D.3 (Lines 838–846), suggesting ideas for future research. For example, if a specific training example can encourage the model to output diverse Chain-of-Thoughts (CoTs) that generalize to other problems—especially when aided by entropy loss—then the policy gradient might serve as an implicit regularizer on these exploratory CoTs (e.g., punishing the wrong exploration). This could help the model leverage capabilities (e.g., diverse reasoning patterns) acquired during pretraining and midtraining. Additionally, it might be possible to draw on techniques used in analyzing double descent (note that the 1-shot RLVR on $\\pi_{13}$ shows a test curve somewhat similar to “double descent,” as in Fig. 2 middle), which is related to the implicit regularization effects of SGD, to better understand the behavior of 1-shot RLVR. We will make these points more clear in the main paper.\n\n\n\n> **Q2**: I greatly enjoyed the paper and find the empirical findings persuasive. To help researchers and practitioners translate these insights into day-to-day RLVR workflows, could the authors elaborate on a few near-term, “actionable” takeaways? This is a general question, which I wish to learn from the authors.\n\n\n**A2**: Thanks a lot for your recognition of our work! We are very glad to discuss our thoughts on RLVR with you.\n\n(1) **Design better exploration strategies**. In Sec. 4.1, we show that adding entropy loss and properly increasing the sampling temperature can improve the performance of one(few)-shot RLVR. We believe exploration is one of the key factors in RLVR. The current entropy loss evaluates token-level diversity, which may not be the best metric for exploration. Alternative metrics like semantic entropy or sentence-level embedding diversity could potentially perform better. It would also be interesting to explore more complex and effective exploration strategies beyond standard parallel rollouts.\n\n(2) **Investigate mid-training and RLVR jointly, and explore more domains**. The effectiveness of 1-shot RLVR suggests that the model already possesses strong capabilities from the pretraining or mid-training stages. Therefore, a more comprehensive investigation of RLVR could involve jointly controlling both mid-training and RLVR data. Since the math domain is often fully covered during mid-training, it would be valuable to apply comprehensive investigation to more diverse domains.\n\n(3) **Move from data efficiency to compute efficiency**. As shown in the paper, RLVR is highly data-efficient but computationally expensive. Given a strong base model, it is meaningful to explore how to better leverage its capabilities with minimal compute. For instance, in Tab. 14 of Appendix C.2.3, we interestingly find that using $\\pi_1$ and the model’s own CoT output (from step 500) as an in-context example can significantly improve Qwen2.5-Math-7B’s performance (37.4%, compared to 34.3% for the format-fixing baseline). This single example significantly outperforms the 4 official examples provided by the Qwen2.5 repository (21.7%). Although it does not work perfect on Qwen2.5-Math-1.5B, it is interesting to further explore such lightweight or zero-train methods for boosting performance.\n\n(4) **Better understand post-saturation generalization from the data perspective**. In Appendix C.2.5, we observe that simplifying the problem in $\\pi_1$ significantly reduces its effectiveness in 1-shot RLVR (Tab. 16), highlighting the importance of the problem statement and what kind of CoTs it can incentivize. It is also noteworthy that different data points vary widely in their impact on 1-shot RLVR performance (Tab. 3). A promising future direction is to use a similar 1-shot RLVR process to study how individual problems affect CoT generation and model behavior, which may also provide insight into post-saturation generalization and help with data curation.\n\nWe include some related discussions in Appendix D.3 and believe our findings will be helpful for future work."}, {"review_id": "F9d0fDW38W", "reviewer": "Reviewer_U7dr", "summary": "This paper finds that RL training on only one example can improve the reasoning performance of LLM.\nThe paper explores the effectiveness of different training examples and finds that they can all somehow improve the performance. The performance on the test set also increases as the training becomes longer, even if the accuracy on the training sample saturates.", "questions": "See weakness", "limitations": "See weakness.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The paper reveals an interesting phenomenon, that RLVR with only one example would improve the reasoning performance of LLM. The paper further explores different training samples, and the phenomena on training/test set during training.\n\n## Weaknesses\n1. The paper mainly presents empirical experimental results with hypotheses, but does not discuss why or provide convincing explanations with solid support.\n2. The training setting seems to be rarely used during real training, which makes the conclusion less applicable.\n3. The proposed method for selecting examples does not work, since training with samples $\\pi_{13}$ and $\\pi_{1209}$ also obtains decent improvements.\n4. The policy gradient loss is the basic functional part in RLVR. The reviewer does not quite understand the meaning of Rows 1-10 in Table 5. It is obvious that the policy gradient loss dominates the learning from the reasoning trajectories. The reviewer also does not understand the logic in line 258/259, why 1-shot RLVR is attributed to policy gradient loss can distinguish it from “grokking”.\n5. What does 'post-saturation generalization' mean? We might need further explanation. From Figure 2, it seems that training on only one example cannot surpass the best performance of training on many examples, and it seems training on many samples brings faster growth of accuracy on the test set, so what is the advantage of training on one example?", "was_revised": true, "final_justification": "It seems all the reviewers point out that more depth analysis is necessary. However, the author response does not resolve our concerns at this point. We agree that the phenomenon that training on only one of some examples could increase the performance is interesting; however, it is merely a discovery, as a scientific research project, only pointing out the phenomenon without solid explanation/proof/proposed method is incomplete. \nThe reviewer would expect more solid explanations or an effective method that exploits this phenomenon in this paper. Therefore, I believe this paper is not fully ready for this venue, although it is interesting.", "rebuttal": "Thank you for your recognition of our paper and your valuable feedback. We have responded to your concerns and will revise our paper based on the discussions. We would also appreciate it if you could let us know if our response addresses your concerns.\n\n\n\n> **Q1**: The paper mainly presents empirical experimental results with hypotheses, but does not discuss why or provide convincing explanations with solid support.\n\n**A1**:\n\nIn our paper, we focus on empirical findings and demonstrate some necessary conditions for the success of one(few)-shot RLVR, such as the strong capabilities stored in the base model due to pretraining/midtraining stages (Lines 810–819), the importance of policy gradient and exploration (but not weight decay, unlike grokking, see Sec. 4.1), etc. We have conducted extensive experiments to confirm and illustrate the key features of one(few)-shot RLVR (Sec. 3, Sec. 4, Appendix C).\n\nWhile more theoretical analysis would certainly be valuable, it would also require significant additional effort due to the complexity of the problem, especially if a rigorous proof is to be pursued. Such analysis could become an independent contribution and is difficult to fully include within the limited space of this rebuttal. Therefore, we believe it is reasonable to leave the theoretical investigation as future work.\n\nIn fact, we also discuss possible explanatory directions in Appendix D.3 (Lines 838–846), suggesting ideas for future research. For example, if a specific training example can encourage the model to output diverse Chain-of-Thoughts (CoTs) that generalize to other problems—especially when aided by entropy loss—then the policy gradient might serve as an implicit regularizer on these exploratory CoTs (e.g., punishing the wrong exploration). This could help the model leverage capabilities (e.g., diverse reasoning patterns) acquired during pretraining and midtraining. Additionally, it might be possible to draw on techniques used in analyzing double descent (note that the 1-shot RLVR on $\\pi_{13}$ shows a test curve somewhat similar to “double descent,” as in Fig. 2 middle), which is related to the implicit regularization effects of SGD, to better understand the behavior of 1-shot RLVR. We will make these points more clear in the main paper.\n\n\n\n\n\n\n\n> **Q2**: The training setting seems to be rarely used during real training, which makes the conclusion less applicable.\n\n\n**A2**: We would like to address this concern from the following points:\n\n(1) Few-shot reinforcement learning for LLMs remains valuable in **data-limited** domains where obtaining large amounts of high-quality, verifiable data is difficult and often expensive to annotate. For example, OpenAI’s reinforcement fine-tuning also supports training models with as few as 10 examples in fields like medicine [3]. Besides, we think one(few)-shot learning will also be helpful for other data-centric case studies in RLVR, because we can observe how each single data point or small amount of data affects the RLVR process and model output styles.\n\n(2) Most importantly, the main target of our paper is analyzing a new observation of RLVR, rather than proposing a new method. Except for the dataset size, all other experimental setups are very similar to other RLVR studies. And we think it conveys or confirms a lot of messages applicable for RLVR with general datasets, such as:\n\n(a) Supporting that the base model already has strong capability (discussed in Line 810–819), because one example brings limited new information and instead incentivizes the LLM’s pre-learned capabilities — emphasizing the importance of pretraining and midtraining for RLVR.\n\n(b) Showing that RLVR can be very hard to overfit to the data (as we train on 1 example thousands of times). This is a nice signal for training stability.\n\n(c) Showing that RLVR can be data-efficient, but still compute-intensive. The amount of RLVR data used in current RLVR recipes is much less than in other stages (e.g., SFT or midtraining). This implies the importance of data selection and curation for RLVR (discussed in D.3 Line 830–837). For example, Tulu3 [2] uses 939k examples for SFT, while Qwen3 [1] uses only 4k query-verifier pairs for reasoning RL. \n\n(d) Supporting the importance of encouraging exploration in the RLVR process (discussed in Line 847–858). In Fig. 5 we show that adding entropy loss is important for the post-saturation stage in 1-shot RLVR, and slightly increasing training temperature (e.g., from 0.6 to 1.0) can further improve results. Better exploration strategies are also important for future RLVR training.\n\nWe think these topics are important for real RLVR training and believe our findings would be insightful for future study. We will add these discussions in the revised paper to better clarify our position.\n\n\n\n\n\n> **Q3**: The proposed method for selecting examples does not work, since training with samples pi_13 and pi_1209 also obtains decent improvements.\n\n\n\n**A3**: We would reply to this question in the following points:\n\n(1) **Our selection method still outperforms random baselines.** In Table 4 (Qwen2.5-Math-7B), our selected set $\\{\\pi_1, ..., \\pi_{16}\\}$ achieves 42.5% accuracy—close to the 42.8% from full 1.2k DSR-sub RLVR, and clearly better than 40.2% from randomly selected examples.\n\n(2) We apply historical variance score for filtering examples with critical errors in ground truth or overly difficult problems (e.g., $\\pi_{1207}$ and $\\pi_{1208}$), which are not suitable for few-shot RLVR (Tab. 3). Random selection risks including such examples.\n\n(3) Most importantly, our paper focuses on analyzing a new feature of RLVR—not on proposing a SOTA data selection method (Lines 121–125). In fact, the success of $\\pi_{13}$ and $\\pi_{1207}$ **supports the generality** of 1-shot RLVR, rather than undermining our selection strategy. While better selection techniques are valuable, they are beyond our scope.\n\n\n\n> **Q4-1**: The reviewer also does not understand the logic in line 258/259, why 1-shot RLVR is attributed to policy gradient loss can distinguish it from “grokking”.\n\n**A4-1**: \n“Grokking” is a phenomenon that is strongly affected by explicit regularization methods like weight decay or dropout, as shown in the original grokking paper [4] and mentioned in Lines 241–248 of our main paper. However, from Row 2 vs. Row 3 or Row 8 vs. Row 4 in Tab. 5 (or Fig. 14), we can see that weight decay has little effect in 1-shot RLVR training. Therefore, 1-shot RLVR are different from \"grokking\".\n\n\n\n> **Q4-2**: The policy gradient loss is the basic functional part in RLVR. The reviewer does not quite understand the meaning of Rows 1-10 in Table 5. It is obvious that the policy gradient loss dominates the learning from the reasoning trajectories. \n\n\n**A4-2**: \nWe respectfully disagree with the question. There are four loss terms influencing RLVR training, and both weight decay and policy gradient could contribute to the observed post-saturation phenomenon. To draw firm conclusions, we believe the ablations in Tab. 5 are necessary.\n\nAs discussed in A4-1 and Lines 241–248, the post-saturation generalization shows similarities to the “grokking” phenomenon, which is known to be strongly affected by weight decay. However, we cannot conclude whether the success of 1-shot RLVR is due solely to weight decay, solely to policy gradient, a combination of both, or interactions with other terms such as entropy loss—without conducting proper ablation studies.\n\nWhile the policy gradient loss is indeed crucial for RLVR training, it is not necessarily the primary reason for the new phenomenon. For example, in the “grokking” paper [1], the training loss is critical, but regularization techniques like weight decay have a significant impact on the final outcome.\n\nAdditionally, we observed a ~4% improvement on MATH500 when combining entropy loss and policy gradient (Row 7) compared to using policy gradient alone (Row 2) in 1-shot RLVR. This supports the value of exploration (Fig. 5) and reinforces the need for our full ablation analysis.\n\n\n\n\n> **Q5-1**: What does 'post-saturation generalization' mean? We might need further explanation. \n\n\n**A5-1**: “Post-saturation generalization” refers to the phenomenon in 1-shot RLVR where, after the model reaches near 100% accuracy on the training data, its test performance continues to improve as training progresses.\nWe mention this in the title of Sec. 3.2.2, the caption of Fig. 2, and Lines 174–178. We could provide a more formal definition in the main paper.\n\n\n\n\n\n> **Q5-2**:... training on only one example cannot surpass the best performance of training on many examples, and it seems training on many samples brings faster growth of accuracy on the test set, what is the advantage of training on one example?\n\n\n**A5-2**: The main focus of our paper is to analyze a new observation in RLVR that we believe will be insightful for future research and real-world applications, rather than to directly propose a new training method. This response largely overlaps with A2 above.\n\nIn addition, although one(few)-shot RLVR indeed requires more computation to converge, it significantly improves data efficiency in RLVR training (using less than 0.1% of the data), which is particularly valuable for data-limited domains (refer to A2), given the high cost of annotating high-quality RLVR data.\n\n\n\n---\n[1] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025).\n\n[2] Lambert, Nathan, et al. \"Tulu 3: Pushing frontiers in open language model post-training.\" arXiv preprint arXiv:2411.15124 (2024).\n\n[3] OpenAI, “Reinforcement Fine‑Tuning,” OpenAI Platform Documentation, 2025. [Online]. Available: https://platform.openai.com/docs/guides/reinforcement-fine-tuning. [Accessed: Jul. 30, 2025].\n\n[4] Power, Alethea, et al. \"Grokking: Generalization beyond overfitting on small algorithmic datasets.\" arXiv preprint arXiv:2201.02177 (2022)."}, {"review_id": "hZkVIX0KA3", "reviewer": "Reviewer_URox", "summary": "The paper found that one training sample is sufficient to lead to substantive performance gains in RLVR. It discusses methods for selecting the sample, conducts experiments on different samples and models, and concludes that one or a few training samples can bring performance gains similar to training on the full dataset.", "questions": "1. In Figure 4, I observe that the response length of training on a single sample is much longer than on the full dataset. Would the response length in the evaluation dataset also be substantially longer? If so, this may be a significant drawback of training on a single sample that needs to be emphasized, due to the increased inference cost (even if the performance is similar).\n\n2. The evaluation mainly uses pass@1 with temp=0 and pass@1 with temp=0.6. I am interested in how pass@n (e.g., n=8, with a decent temperature such as 0.6) compares between training on a full dataset and a single sample. Given the large entropy shown in Figure 4 for 1-shot training, will pass@n be worse than when training on a full dataset?", "limitations": "Limitations are addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Novel finding: The phenomenon that one training sample can work with RLVR is surprising and counter-intuitive and should be investigated further by the research community.\n- Extensive experiments: Multiple models and different training samples are used, strengthening the claim that this phenomenon is not specific to a particular model or a particular training sample. The ablation analysis is comprehensive, such as including ablations on different components of the training loss.\n\n**Weaknesses**\n\n- A more in-depth analysis could strengthen the paper. Section 4.1 discusses that the underlying reason for the phenomenon is not due to grokking, while Section 4.2 shows that entropy alone may also help in the performance gain. However, neither section analyzes the underlying reason why a single sample is sufficient.\n- Slightly misleading narrative in Section 3.2.1: In Section 3.2.1, it makes the observation that \"The answer is not precise\"—i.e., the ground-truth answer is wrong. This may leave the reader with the impression that this slight inaccuracy is a necessary condition. However, the paper later shows that using the correct answer \"12.7\" (Row 11 in Table 5) works similarly to the wrong answer \"12.8\" (Row 5), with the difference likely being statistically insignificant. It would be smoother if the narrative started with the correct answer and then showed that the result is robust to slight deviations. I understand this would require a substantial re-running of existing experiments, so perhaps more clarification in the writing is sufficient.\n- Informal writing style: The writing tone can be more academic—for example, by using less italics and bolding. Page 8 is filled with bolded sentences, which is atypical of an academic paper. Informal phrases such as \"What's more\" should be removed.\n\nOverall, the results in the paper are novel and strong, and the weaknesses above are relatively minor, so I will recommend acceptance.", "was_revised": true, "final_justification": "I recommend acceptance, as the authors have adequately addressed my questions and the work provides novel insights into how RLVR operates in LLMs.", "rebuttal": "Thank you for your recognition of our paper together with your valuable comments and suggestions. We will revise our paper according to your comments. We respond to your comments below and would appreciate it if you could let us know if our response addresses your concerns.\n\n\n> **Q1**: A more in-depth analysis could strengthen the paper. Section 4.1 discusses that the underlying reason for the phenomenon is not due to grokking, while Section 4.2 shows that entropy alone may also help in the performance gain. However, neither section analyzes the underlying reason why a single sample is sufficient.\n\n\n**A1**: As you mentioned, in our paper, we mainly focus on empirical findings and demonstrate some necessary conditions for the success of one(few)-shot RLVR, such as the strong capabilities stored in the base model due to pretraining/midtraining stages (Lines 810–819), the importance of policy gradient and exploration, etc. \n\nWhile more theoretical analysis would certainly be valuable, it would also require significant additional effort due to the complexity of the problem, especially if a rigorous proof is to be pursued. Such analysis could become an independent contribution and is difficult to fully include within the limited space of this rebuttal. Therefore, we believe it is reasonable to leave the theoretical investigation as future work.\n\nIn fact, we also discuss possible explanatory directions in Appendix D.3 (Lines 838–846), suggesting ideas for future research. For example, if a specific training example can encourage the model to output diverse Chain-of-Thoughts (CoTs) that generalize to other problems—especially when aided by entropy loss—then the policy gradient might serve as an implicit regularizer on these exploratory CoTs (e.g., punishing the wrong exploration). This could help the model leverage capabilities (e.g., diverse reasoning patterns) acquired during pretraining and midtraining. Additionally, it might be possible to draw on techniques used in analyzing double descent (note that the 1-shot RLVR on $\\pi_{13}$ shows a test curve somewhat similar to “double descent,” as in Fig. 2 middle), which is related to the implicit regularization effects of SGD, to better understand the behavior of 1-shot RLVR. We will make these points more clear in the main paper.\n\n\n\n\n> **Q2**: Slightly misleading narrative in Section 3.2.1: In Section 3.2.1, it makes the observation that \"The answer is not precise\"—i.e., the ground-truth answer is wrong. This may leave the reader with the impression that this slight inaccuracy is a necessary condition. However, the paper later shows that using the correct answer \"12.7\" (Row 11 in Table 5) works similarly to the wrong answer \"12.8\" (Row 5), with the difference likely being statistically insignificant. It would be smoother if the narrative started with the correct answer and then showed that the result is robust to slight deviations. I understand this would require a substantial re-running of existing experiments, so perhaps more clarification in the writing is sufficient.\n\n**A2**: \nThanks for pointing this out! We acknowledge that the current narrative could be slightly misleading. We will revise the clarification in Sec. 3.2.1 to reduce the emphasis on imprecise answers in $\\pi_1$, as 1-shot RLVR using a single example with a precise answer (e.g., $\\pi_{13}$, Line 868) also works well.\n\nWe have discussed a related topic — the influence of random incorrect labels — in Sec. C.2.4 (Tab. 15), as an extension of the label correctness analysis in Tab. 5. However, as you mentioned, thoroughly investigating how varying degrees of label deviation affect RLVR would be an independent study requiring extensive experiments. We leave this as an interesting direction for future work.\n\n\n\n> **Q3**: Informal writing style: The writing tone can be more academic—for example, by using less italics and bolding. Page 8 is filled with bolded sentences, which is atypical of an academic paper. Informal phrases such as \"What's more\" should be removed.\n\n\n**A3**: Thanks for pointing this out! We will definitely revise our paper based on this feedback in the next version!\n\n\n\n\n> **Q4**: In Figure 4, I observe that the response length of training on a single sample is much longer than on the full dataset. Would the response length in the evaluation dataset also be substantially longer? If so, this may be a significant drawback of training on a single sample that needs to be emphasized, due to the increased inference cost (even if the performance is similar).\n\n\n**A4**: The response lengths in the evaluation set would **not** be substantially longer and still remain within a reasonable range. This aligns with the post-saturation generalization phenomenon described in Sec. 3.2.2, where the model begins to overfit to the single training example—mixing correct reasoning with excessively long, meaningless outputs on that example (Fig. 3, Step 1860, Left)—while still producing reasonable outputs on the test set (Fig. 3, Step 1860, Right).\n\nFor reference, the average response lengths (i.e., number of output tokens) on the test tasks are shown below. While 1-shot RLVR generally produces longer responses than RLVR with DSR-sub, the difference remains within a reasonable range and is consistent with our observations regarding the frequency of reflection (Fig. 4, Right). We would add these discussions in the revised paper.\n\n\n**Table R2**: Average response length of Qwen2.5-Math-1.5B on evaluation tasks. Here we use format-reward experiment (DSR-sub + Format in Tab. 12, Sec. C.2.3) as a baseline for eliminating the difference in tokens introduced by formats.\n\n| |MATH500|AIME 2024|AMC 2023|Minerva Math|OlympiadBench|AIME 2025|Avg.\n|-|-|-|-|-|-|-|-|\nFormat-Fixing|689.0|1279.7 |911.0|1018.1 |957.3|1177.1 |1005.4\n1-shot RLVR (step 100)|611.1 |1122.7 | 939.1 |1071.8 | 951.2 |1173.0 | 978.2\n1-shot RLVR (step 1500)|739.7 |1352.4 | 985.6 |904.7 |1088.9 | 1250.7 | 1053.7\nRLVR with DSR-sub (step 100)|635.9|1268.0| 873.7|797.0|953.6|1121.6| 941.6\nRLVR with DSR-sub (step 1500)|561.7|949.1|761.8|637.6|783.8|988.4|780.4\n\n\n\n\n> **Q5**: The evaluation mainly uses pass@1 with temp=0 and pass@1 with temp=0.6. I am interested in how pass@n (e.g., n=8, with a decent temperature such as 0.6) compares between training on a full dataset and a single sample. Given the large entropy shown in Figure 4 for 1-shot training, will pass@n be worse than when training on a full dataset?\n\n\n**A5**: This is a great question. As you suggested, we ran this analysis for AIME 24, AMC23 and AIME25 (pass@8, temperature = 0.6) on Qwen2.5-Math-1.5B, and the results are shown in Tab. R3 below:\n\n**Table R3**: Pass@8 results on 3 math evaluation tasks using Qwen2.5-Math-1.5B. We also include the performance of RLVR with format-reward (as in Table R2) as a stronger baseline.\n\n| |AIME 2024|AIME 2025|AMC 2023|Avg on 3 tasks\n|-|-|-|-|-|\nBase Model|26.6|20.0|72.5|39.7|\nFormat-Fixing (highest) |33.3|23.3|72.5|43.1|\n|\nRLVR with DSR-sub (highest, step 160) | 36.7| 26.7 | 87.5 | 50.3 |\nRLVR with DSR-sub (step 500) | 33.3|30.0|82.5|48.6|\nRLVR with DSR-sub (step 1000) | 33.3|20.0|75.0|42.8|\nRLVR with DSR-sub (step 1500) | 30.0|26.7|67.5|41.3|\n|\n1-shot RLVR (step 500)| 30.0|16.7|80.0|42.2|\n**1-shot RLVR (highest, step 980)**| 36.7 | 33.3 | 85.0 | **51.7** |\n1-shot RLVR (step 1500)| 26.6|23.3|87.5|45.8|\n\n\nInterestingly, we find that\n\n(1) 1-shot RLVR achieves similar or even slightly better best pass@8 performance (51.7%) across these three math tasks compared to full-set RLVR (50.3%). These results show that 1-shot RLVR will not result in worse pass@n performance, and also supports our claim about post-saturation generalization (Fig. 3): even though the model overfits to the single training example and shows high entropy on it (Fig. 4 middle), it still performs well on evaluation tasks.\n\n(2) There is a noticeable downward trend in pass@8 performance for full-set RLVR (with 1.2k DSR-sub) after 200 steps. This phenomenon is consistent with recent findings that RLVR can sometimes hurt a model’s pass@n performance [1].\n\n\nThanks for your insightful suggestion. We will include these new findings in the next revised version.\n\n\n\n[1] Yue, Yang, et al. \"Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?.\" arXiv preprint arXiv:2504.13837 (2025)."}]

Author rebuttal:

Meta-review: The reviewer finds this paper to present a set of significant and interesting observations regarding GRPO training, highlighting valuable findings such as post-saturation generalization and the surprising effectiveness of individual training samples. While the core results are deemed strong, the reviewer raises a crucial and valid concern that the current presentation of these results is potentially misleading. The primary weakness identified is the failure to properly baseline against the substantial performance gains, which may misrepresent the comparative performance of single-sample versus full-dataset training. However, this is an addressable issue, and the reviewers' consensus is to recommended acceptance. The authors are still advised to revising their results, tables, and figures to incorporate a more appropriate baseline comparisons that properly contextualize their findings in the final paper.
Decision: Accept (poster)