RESTRAIN: From Spurious Votes to Signals — Self-Training RL with Self-Penalization
Authors: ZHAONING YU
Zhaolun Su
Leitian Tao
Haozhu Wang
Aashu Singh
Hanchao Yu
Jianyu Wang
Hongyang Gao
Weizhe Yuan
Jason E Weston
Ping Yu
Jing Xu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
self-training
RL
unsupervised learning
self-penalization
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning with human-annotated data has boosted chain-of-thought
reasoning in large reasoning models, but these gains come at high costs in labeled
data while faltering on harder tasks. A natural next step is experience-driven learning, where models improve without curated labels by adapting to unlabeled data.
We introduce REinforcement learning with Self-resTRAINt training (RESTRAIN),
a self-penalizing RL framework that converts the absence of gold labels into a
useful learning signal. Instead of overcommitting to spurious majority votes,
RESTRAIN exploits signals from the model’s entire answer distribution: penalizing
overconfident rollouts and low-consistency examples while preserving promising
reasoning chains. This self-penalization mechanism integrates seamlessly into
policy optimization methods such as GRPO, enabling continual self-improvement
without supervision. On challenging reasoning benchmarks, RESTRAIN delivers
large gains using only unlabeled data. With Qwen3-4B-Base and OctoThinker
Hybrid-8B-Base, it boosts Pass@1 by up to +140.7% on AIME25, +36.2% on
MMLU STEM, and +19.6% on GPQA-Diamond, nearly matching gold-label
training while using no gold labels. These results demonstrate that RESTRAIN establishes a scalable path toward stronger reasoning without gold labels.

Published as conference paper at ICLR 2026
RESTRAIN: FROM SPURIOUS VOTES TO SIGNALS -ELF-DRIVEN RL WITH SELF-PENALIZATION
Zhaoning Yui:l.;;
Will Su Leitian Tao?,
Haozhu Wang",
Aashu Singh*, 4
Hanchao Yu", Hongyang Gao",
Jianyu Wang",
Weizhe Yuan 2.5
Jason 2eston 2.5, Ping Yu?!?,
Jing Xut.2
lowa State
FAIR at Meta Superintelligence Lab
UW--Madison *Meta
SNYU
ABSTRACT
Reinforcement learning with human-annotated data has boosted shain-of-thoughh reasoning in large reasoning models, but these gains come high costs in labeled data while faltering on harder tasks. A natural next step is experience-driven learn- ing, where models improve without curated labels by adapting to unlabeled data. We introduce REinforcement learning with Self-reSTRAING training (RESTRAIN), self-penalizing RL framework that converts the absence of gold labels into useful learning signal. Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution: penalizing overconfident rollouts and low-consisteng. examples while preserving promising reasoning chains. This self penalizaaioon mechanism integrates seamlessly into policy optimization methods such as GRPO, enabling continual self- imprrvemeoeentt without supervision.
challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data. With Qwen3-4B-Ban -BBase and Dhinkhinner Hybrid-8B- --brid-8--BBse,,, it boosts Pass by up to +140.7% on AIME25, +36.2%% % on MMLU.STEM, and +19.6% % on GPQA-Diamond, nearly matching gold-label training while using no gold labels. These results demonstrate that RESTRAIN es- tablishes scalable path toward stronger reasoning without gold labels.
Pess@l MATH500
Passo81 AIME25
Pase@1 on MMLU STEM
Pass@91 GPQ4-Dismond
Gold GAPO
TINL
20
Cold GIPO TIRL RESTRAN
Gold GEPO TTRI RESTRAI
+6.6
Go GIPD
TT%
RESTRAN
15
+13.7
3.8
25
AMC
AIMEZ4
MATH500
70.0
Accurary (%))
40.0
16
67.5
37.5
14
65.0
35.0
12
62.5
32.5
10
60.0
30.0
TTRL
ETMR
RESTRAIN
TTRL
ETMR
RESTRAIN
TTRL
ETMR RESTRAIN
Figure Performance of Label-free and Test-Time RL. Top: Pass of Qwen3-4B-Base and OctoThinker Hybrid-8B-Ban Base trained on DAPO-14K-MAND and nearly matches the Gold-label GRPO upper bound, even surpassing it on MMLU-STEM and GPOA-Diamond. Bottom: Test-time training Llama3. 1....11.6 1-8B-18---ee Instrrut using unlabeled test data from AIME24, AMC23, and MATH500, reporting Pass@ accuracy, RESTRAIN significantly outperforms TTRL and ETMR, especially AMC and MATH500.
RESTRAIN outperforms TTRI.
Work done during an internship at Meta
Joint first author 'Joint last author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in LLMs (uuo et al., 2025 Jaech et al., 2024 Yang et al. 2023) show that Re- inforcement Learning (RL) with human-annotated data and verifiable rewards (RLVR) greatly en- hances long chain-of-thought reasoning (Wei al. 2022), achieving strong performance
chal- lenging benchmarks. Yet RLVR remains limited: it depends on ever-growing quantities of high- quality labeled data. Achieving superhuman performance, models will eventually need to operate in environments where even humans lack definitive answers and cannot offer reliable feedback on out- puts. In these situations, models must develop the ability to self-improvy without direct supervision. This motivates exploring RL. on unlabeled data, where progress arises from self-improvement rather than curated labels, with large external corpora serving as training signal (Zuo al. 2025). In this work, we study RL in an unsupervised setting to advance reasoning generalization.
A central challenge in enabling self-improvement without labeled data is how model can generate its own learning signals. One natural direction is self relarren methods, where the model gener- ates its own reward signals-for instance, ranking or scoring its rollouts based on its own judgments Yuan et al., 2024). While these methods remove the dependence on gold labels, evidence remains limited that such methods consistently improve performance on complex reasoning tasks. second line of work leverages the model's internal agreement, such using majority voting across multiple rollouts (Zuo et al. 2025; Shafayat al. 2025; Liu et al. 2025a Prasad et al. 2024). Yet this ap- proach sulfers from relibility and robustness issues that can cause model training collapse: models frequently generate responses with low self- consisesstt or low confidence across multiple attempts, and for challenging reasoning tasks, the majority-vote voted answer itself can be systematically flawed. In such cases, minority rollouts can contain the correct solution (Stahlberg & Byrne 2019 [Stahlberg al. 2022), but these ignored when overconfident spurious majorities dominate. Training on such distorted reward signals limits scalability as task diversity and complexity increase. The key challenge, therefore, is not merely generating self-derived rewards, but ensuring that they provide robust signals that drive genuine reasoning improvement.
To address this gap, we introduce RESTRAIN. framework for self delf-driven RL with self self- penalization. Instead of relying on gold labels or external supervision, RESTRAIN leverages the model's ' own predictions by (I) considering all predicted answers rather than only majority votes, (2) penal- izing low-confidence rollouts with negative advantages, and (3) down-weighting low-agreement prompts with fragile majority votes. By integrating ceff-penalization directly into the RI. objec- tive, RESTRAIN turns the absence of labels into rollout-level and prompt-level learning signals. We evaluate RESTRAIN on two base models and two tasks across six benchmarks. Notably, RESTRAIN raises Pass @1 by 140.7% AIME25, 36.2% on MMLU and 19.6% on GPQA-Diamond. Even more striking, its performance nearly matches gold-label cupervision-lageee by only 0.4 points. These results establish RESTRAIN scalable approach to self-driven RL, pushing reason- ing models beyond supervised limits.
RESTRAIN
We introduce the main ideas of RESTRAIN below and in Figure
Preliminaries We adopt Grouped Relative Policy Optimization (GRPO) (Shao et al. 2024) as our main RL algorithm. GRPO optimizes a policy no by sampling n rollouts per prompt æ with gold label y. and updating with PPO-style objective against fixed reference policy mais using group- mean baseline for variance reduction. For each rollout ys, we denote by reward r; R(y,, 2/2z) with advantage A,, The GRPO objective for each prompt x with gold label y is:
T
(E
min (p:(0) A,, clip(a:(0)) -,,, 1.) 3I [To]|
(1)
2.1 PSEUDO-LABEL WEIGHTING
In unsupervised settings without gold labels, a model can give multiple predictions for a given prompt x, regardless of their correctness. Figure 2 reports accuracies model predicted answers
2

---

Published as a conference paper at ICLR 2026
100%
100%
90%
90%
80%
80%
70%
70%
Probability
Probability
60%
60%
50%
50%
40%
30%
20%
Pass@64
20%
Pass@64
10%
Majority-Anssee Correct Ratio
10%
Majority-Anse Answer Correct Ratio
0%
0%
5 10
25 30 Majority Size
45 50
65
40 Majority Size
20 25
30
35
45 50
(a) Qwen3-4B-BBl
(b) Octothinker Hybrid 8B basc
Figure 2: Majority- Vote Reliability. Pass@64 and the majority-voter accuracy over 64 samples are compared on the DAPO-MATH dataset for Qwen3-4B-Base (left) and OctoThinker Hybrid-8B- Base (right). The large gap between @4 and majority-vote shows that correct answers often diverge from majority votes. Accuracy also drops sharply when the majority size is small, revealing that majority votes can earry spurious signals. These observations motivate our self-penalizing framework, which seeks robust promising reasoning paths beyond unreliable majority votes.
for the Qwen3-4B-BBl Base model (a) and the Octokhinker Hybrid- Bendd-B-Baaal Base model (b) on the DAPO- MATH dataset. Although the Majority Correct Ratio rises with the majority vote size (number of solutions that agree). there remains a large gap between Pass @ 64 and the majority correct ratio, revealing that majority votes be spurious and often fail capture the true answer. To bridge this gap, we introduce a pseudo-label weighting scheme. Rather than collapsing all probability mass onto the most frequent answer (majority voting) or distributing it uniformly across candidates, our method assigns weights proportional to the observed vote counts. This produces consensus distribution that down-weight spurious low-frequency answers while avoiding the brittleness of requiring consensus, providing the foundation for our self self-penalization framework.
Construction Given a prompt n, we draw n rollouts {y Ta(- x) and collect the set of unique answers {a with counts ej. We treat each aj as a pseudo label and compute the weighted loss as follows:
LERPO(E; 0)
Σ my LORPO(X, (x;
(2)
where wy pseudo-label weight obtained by applying a monotonic function g to frequency fj 5
g(H))
Wi Emag((e))
We use Gaussian function centered at the & E [0, 1] with bias a > 0 as our shaping function
Interpretation Equation 3 prevents collapse to single majority answer while penalizing spurious low-frequency predictions through form of soft selection over answer frequencies: predictions with higher frequencies receive proportionally larger weights. The skewness of this weighting is controlled by the monotonic shaping function g(-): steeper g concentrates probability mass high-frequency answers, whereas a smoother g distributes weight more broadly across answers.
2.2 NEGATIVE ROLLOUT PENALIZATION
Existing methods Zuo et al. 2025 Shafayat al. 2025) often rely on the majority-voted answer being correct, making low self-consistency regions prone to spurious training signals. Our proposed pseudo-label weighting subsection 2. instead leverages control of Pass @nss@n: any rollout is correct, it provides a valid positive signal, yielding more robust learning under weak consensus. However, when the majority size is very low, Pass @n often degrades because the model may generate no correct rollouts all. As shown in Figure 2] prompts with very low majority size correspond to unreliable supervision where no answer can confidently trusted. handle such cases, we intro- duce negative rollout penalization, which assumes all responses incorrect and applies uniform
3

---

Published as a conference paper at ICLR 2026
RESTRAIN
Pseudo Label Weighting
Self-Driven RL with celf-Penalizatiin Majority 2
Model Predictions as Pscudo-Lahels
Label Weight W
Pacudo-Labe Confidence Weights
Majority Gating: choek consistency
Thrabald
Weight pseude label
confidence
Ew Losstrollouts y, pscudo-label
Final Self-iraining Loss = Prompt u." Label w. Penalized Loss(A)
Negative Rollout Penalizatiion
Penalize Rellouts
N Rolloutts
Reward:
Penolized Rollout Loss
Majority <
Negalive Advantage Offset: A
Policy
Model
Prompt Weighting
Prompt
Reference
N Rollouts
Stal
Prompt Weight U,
Adiakiay
Figure 3: Overview of Our Method RESTRAIN: RESTRAIN consists of 3 core components: 1. Pseudo Label Weighting which takes into account all possible model-predicted answers as candi- date pseudo-labels when calculating final losses. 2. Negative Rollout Penalization which penalizes rollouts with very low confidence by setting zero reward and applying negative advantage offsets to the losses. 3. Prompt Weighting which downweights entire examples where the reference model predicts with low self-consistency.
negative offset. This reduces explicitly penalizing rollouts and encourages the model to explore alternative reasoning paths.
Construction Consider the GRPO loss term LGRPO oy; 0) associated with pseudo-labe label aj- For each rollout ye. denote by fk.j Rly;, az) the reward and Ar, the corresponding advantage. Let (xx max, ej denote the majority count of prompt x, where ej is the vote count for label aj.
When the self-consistency is low (Mx k), we treat all candidate answers unreliable, zero their rewards, and apply a uniform penalty 8 2 0 to the advantages of all rollouts.
TTJ (() M(x) <
A
if M (x) k
A
(4)
O
A.,
5 if M(x) < K
In PPO/GRPO objectives, this means that all model predictions with (x) < contribute only negative updates, penalizing all rollouts with low self-consistency. This discourages reinforcement of spurious majority votes and steers the model away from unreliable reasoning paths.
2.3 PROMPT-LEVEL WEIGHTING
Previous penalizing schemes operate at a rollout level. In addition, we introduce a prompt-level penalty. For some prompts, the model exhibits high uncertainty, while for others it produces highly consistent responses. To account for this variation, we scale the update for each prompt by fixed weight that reflects the model's 's confidence: low-confidence prompts receive smaller updates, and high-confidence prompts receive larger updates. To prevent spurious feedback loops (e.g., inflated confidence during training), these weights are computed once using a frozen base model and kept constant thereafter.
Construction For each prompt x, we sample n rollouts from the reference policy गie and compute the majority count Czef. We define the prompt weight again using the monotonic function g(-):
Cref
(5)
We apply uy to each prompt for all training updates Unlike pseudo-label weights, prompt-level weights precomputed offline and remain fixed during the RI training. In Appendix E, we will
4

---

Published as a conference paper at ICLR 2026
Table 1: On DAPO-14K-Math: RESTRAIN outperforms all unsupervised baselines. All Pass results(%) are averaged over 16 seeds. The best results are highlighted in bold. RESTRAIN out- performs existing baselines without access to gold labels for both Qwen3-4-Bate and Octothinker Hybrid-8B Base. In particular, Qwen4-B-Bass trained without access to gold labels using RESTRAIN nearly matches the performance of GRPO with gold labels.
Model
math.
aime25
olym.
minerva.
mmlu.
gpqa-d.
Avg.
Qwen3-4B-Ban
68.0
7.9
35.4
26.0
58.3
32.2
38.0
w/ gold label
GRPO
85.0
20.8
50.1
40.1
73.7
38.7
51,4
w/o access to gold label
TTRL
76.3
8.3
39.6
35.9
59.4
33.6
42.2
SRT (easy prompt)
77.8
7.9
39.7
36.3
60.5
34.9
42.8
SRT (offline majority label)
76.9
12.0
39.8
34.2
59.4
34.5
43.1
RESTRAIN (Ours)
83.0
17.9
47.0
36.5
80.9
40.2
51.0
A(RESTRAIN TTRL)
+6.7
+99.6
+7.4
+0.6
+21.5
+6.6
+8.8
18.8%
1115.7%
118.7%
11.7%
136.2%
19.66
120.9%
OctoThinker Hybrid-8B-Ban
29.8
0.8
12.1
9.3
8.6
24.6
19.2
w/ access gold label
GRPO
71.7
6.2
35.2
31.3
62.0
31.0
39.6
w/o access gold label
TTRL
56.5
2.7
23.2
22.1
51.7
27.3
30.6
SRT (offline majority label)
58.5
1.7
23.6
27.6
56.4
29.3
32.8
RESTRAIN
62.1
6.5
24.0
26.1
65.4
30.8
35.8
A(RESTRAIN TTRL)
+5.1
+-388
+0.8
+4.0
+13.7
+3.5
+5.2
19.0%
1140.7%
13.4%
118.1%
126.5%
11288
117.0%
show fifline-computed prompt-level weights outperform online variants that are dynamically up- dated during training.
Final RESTRAIN loss
Jointly applying pseudo-label weights wy from Equation and negative roll-
out penalization Az from Equation 4, and the prompt-level weight u. from Equation 5 we derive our final RESTRAIN loss:
172
CRESTRAIN(;; Ux
Wy LORPO( ay: expund
(6)
LGRPo(2,0;;0)
Σ min (0) 1-c,,, A
DKL Teet]
EXPERIMENTAL SETUP
Datasets We evaluate the effectiveness of RESTRAIN on two mathematical and reasoning tasks:
We adopt the processed DAPO derived from DAPO-Math- 17k YYu et al. 2025b) which deduplicates prompts and standardizes the formatting of both prompts and reference answers. From this release, we further exclude 3k Chinese language prompts and use 14k English language prompts as training split, with further modifications.
5

---

Published as a conference paper at ICLR 2026
Synthetic Sik: A 5k synthetic reasoning dataset from CoT-Self Instrultt Yu et al. 2025a Starting from the curated SIk seed set (Muennighoff et al. 2025). ). Yu et al. (2025a) prompt LLMs to reason step by step and then synthesize new instructions of similar difficulty. Each synthetic example contains both a novel question and a verifialle target answer produced generated by L.M.. This dataset complements existing curated math datasets by providing a fully synthetic yet diverse set of reasoning problems, and allows us to systematically test our method under purely synthetic data generation setting.
Base Models To evaluate the generalizability of our method across different backbone models, we conduct experiments using the following models of various model families and sizes: we use Qwen3-4B-Base and Octothinker Hybrid 8B base (Wang et al. 2025b), which is specialized, highly optimized reasoning model midtrained from Llama3. (Dubey et al. (024), as well as the Llama3.1-8B -88--trruutt model. More details of experimental settings can be found in Appenndix D)
Benchmarks Our benchmark suite comprises six publicly available benchmarks spanning math- ematics (four) and science (two). (1) MATH-500 (Hendrycks et al., 202T), (2) AIME25 (Li et al, 2024), (3) OlympiadBench (math subset) (Yang et al. (024), we use the mathematics portion only. (4) Minerva math (Yang et 2024): the mathematies split from the Minerva quantiative-reasoning
suite. (5) MMLU_STEM (Yang et al. 2024). (6) GPQA-Diamont (Yang et al. 2024).
Metries We evaluate with averaged Pass@1 (Chen et al. 2021 across six benchmarks, sampling 16 predictions per question using temperature of 0.6 and a top-p value of 0.95 and averaging their 16 Pass@ accuracies. We use the official evaluation codebase of Qwen2.5-math (Yang et al. 2024).
Baselines We compare RESTRAIN against three recent label-free RLVR methods:
TTRL (Zuo et al. 2025): treats the majority-voted answer as the single pseudo-label, reinforcing it during RL updates. This makes training heavily dependent the majority being correct, and thus vulnerable to spurious votes.
Self- Reff-Rewarded Training (SRT) (Shatayat et al. 2025 proposes two heuristics to mitigate majority-
vote collapse:
Offline majority label: computes majority votes offline, reducing- but not eliminating the
risk of rewarding self-consistency instead of correctness.
Easy prompts: filters training to "easy" prompts with high vote ratios, discarding low- consensus prompts that often contain valuable but underrepresented reasoning paths.
Entropy-based Test Test-Time Reinforcement Learning (ETTRL) (Liu et al., 2025a is an entropy-based strategy that improves test-time reinforcement learning for LLM reasoning. We include ETTRL as a baseline only in our Test-Time RL experiments. As the original paper reports results only for test-time training (TTT) and no public implementation is available, we do not extend ETTRL to large-scale label-free RL training (e.g.. DAPO-MATH or synthetic S1k).
MAIN RESULTS
RESTRAIN outperforms unsupervised baselines On DAPO-MATH-144
Table
RESTRAIN
training without gold labels substantially outperforms existing unsupervised baselines TTRL and SRT. achieves 51.0%, compared to TTRL (42.2%, +8.8 pp), Offline Majority Label (43.1% +7.9 pp), and Easy Prompts (42.8%, +8.2 pp). A consistent trend appears on the 5k synthetic corpus (Table 2), where RESTRAIN remains the strongest label-free approach, exceeding the next-best base- line by at least 7.7 pp average. Notably, when excluding the two science-heavy benchmarks RESTRAIN nearly closes the gap with distilling the super- vised "reference target" by Qwen3-4B instruct 45.9% 47.7%, margin of only 1.8 pp. On Octo'hinkke Hybrid-8B, we observe the same effect: RESTRAIN consistently surpasses unsuper- vised baselines TTRL and SRT by large margins. These results underscore the power of self-driven RL with self- self-penalization, showing that label- and prompt-level penalties transform noisy unlabeled training into signals strong enough to rival gold-label supervision.
(MMLU STEM and EGQAA-Dimontl
RESTRAIN almost reaches the gold-label upper bound on Qwen3-4B-Base In Table
we treat the Gold-label setting an empirical upper bound for label-free RLVR, achieving average accu- racy of 51.4%. Remarkably, RESTRAIN reaches 51.0%, trailing by only 0.4 pp apseessitallly match- ing supervised GRPO without using labels. Even more striking, RESTRAIN surpasses the gald-label

---

Published as a conference paper at ICLR 2026
Synthetic S1k dataset: Our RESTRAIN outperforms all unsupervised baselines. All @ass@1 I results(%) are averaged over 16 seeds. The best results are highlighted in bold. When training from Qwen3-4B-Ban model on synthetic reasoning tasks without gold label, our method RESTRAIN also outperforms existing unsupervised baselines by 18%.
Table 2:
Model
math.
aime25
olym.
minerva.
mmlu.
gpqa-d.
Avg.
Bieen3-BB-Bae
68.0
7.9
35.4
26.0
58.3
32.2
38.0
w/ access to Qwen3-4B label
GRPO
83.7
18.9
48.4
39.7
83.6
43.5
53.0
w/o access to Qwen3-4B label
TTRL
76.0
9.2
39.3
35.9
57.6
32.8
41.8
SRT (easy prompt)
76.4
8.1
39.6
34.8
57.5
33.0
41.6
SRT (offline majority label)
75.8
10.4
39.2
33.1
57.1
33.1
41.4
RESTRAIN (Ours)
81.7
20.0
45.5
36.5
73.4
40.0
49.5
A(RESTRAIN TTRL)
+5.7
+10.8
+6.2
+0.6
+15.8
+7.2
+7.7
17.5%
1117.4%
115.8%
11.7%
127.4%
122.0%
118.4%
GRPO on MMLU STEM, scoring 80.9% vs. 73.7% and on GPQA-Diamond, 40.2% vs. 38.7%.
This suggests strong cross-domain generalization without gold-labels despite being trained solely on the math-focused DAPO-14K dataset. We hypothesize that gold-label supervision encourages overfitting to domain-speciff patterns, limiting transfer to science tasks, while self- self-penalization -eeiiis relies on distributional signals rather than gold answers, reducing overfitting and preserving generalization across domains.
through
RESTRAIN outperforms other Test Time RL Training methods Test Time RL training focuses the adaptation to test- time data. We compare our method with recent test-time RL methods like TTRL. (Zuo et al.) 2025) and Entropy-fork Tree Majority Rollout (ETMR) (Liu al. 2025a) on LLaam3.1-88B Instrrcct model. following the same setup as in Liu al.) (2025)). with all methods trained on test prompts without access to gold labels. In Table 3 our approach achieves con- sistent improvements across challenging math reasoning benchmarks. It surpasses TTRL and ETMR on AMC23 and MATH- 500 by margins of +13.0% and +13.3%, respectively, yielding overall +11.0% gain in average accuracy. These results demonstrate that our method can also scale very effectively at test time.
Table 3: Comparing RESTRAIN
Two Test Time RL Training Methods: TTRL and ETMR on Samma31-8B-Instruce All results(% are by greedy decoding following Liu et al. (2025a). RESTRAIN also outperforms the existing test-time scaling method by 11%.
Test-Tine Time Method
aime24.
ame
math.
Avg.
TTRL
10.0
32.3
63.7
35.3
ETMR (Liu et al.
2025a)
16.9
35.4
59.5
37.3
RESTRAIN (Ours)
16.7
67.4
41.4
A(RESTRAIN ETMR)
0.2
+44.6
+7.9
+4.11
Training Dynamics MATH500 (Smoothed)
RESTRAIN effectively prevent model collapse Figure 4 shows the averaged Pass@1 I on MATH500 across multiple supervised methods. The base model Qwen3-4B-BB
0.60
Accuracy
and all methods are trained on the 14k DAPO dataset. We
Nothed
Gold answer
SRT offline target
observe that TTRL improves at first but quickly collapses after 50 steps. In con- trast, our method RESTRAIN prevents this sudden collapse and keeps training stable throughout. We attribute this stability to RESTRAIN, which does not exclusively ward the majority-vote answer: instead, it assigns soft weights to all distinct answers
TTRL
ReSTrain (Ours)
SRT easy prompt
1r6a
1299
Training Step
Figure 4: RESTRAIN has more stable training dy- namics. In contrast to TTRL, our method RESTRAIN steadily improves model performances.

---

Published as a conference paper at ICLR 2026
in proportion their empirical frequencies. This frequency-aver aware weighting smooths the learning signal, curbs overconfident updates, and mitigates sudden collapse.
5 ABLATION STUDY
Effectiveness of each component in our RESTRAIN Table presents the impact each compo- of our proposed RESTRAIN. The removal of pseudo-label weighting results in the most sub- stantial performance degradation because training collapses quickly. Omitting negative rollout pe- nalization also hurts performance, reducing the average score from 51.0 42.1. Finally, removing prompt-level weighting leads to a more modest performance decrease, yet still validates its positive contribution the model. Taken together, these results show that all components are necessary for stable and effective unsupervised training.
Table 4: Each component in RESTRAIN is important. Each row represents the model's performance with one component removed. The best results are highlighted in bold.
Model
math.
aime25
olym.
minerva.
mmlu.
gpqa-d.
Avg. f
RESTRAIN
83.0
17.9
47.0
36.5
80.9
40.2
51.0
(-) Pseudo-label weighting
67.3
6.0
34.1
24.5
59.3
33.7
37.5
(-) Negative Rollout Penalization
77.3
9.6
39.9
36.2
56.4
33.0
42.1
(-) Prompt-level weighting
82.7
18.1
46.7
37.8
63.8
37.0
47.7
100
0.8
0-0.1
a-0,5(Ours)
a-1
MATH500 Accuracy
Pass@ Accuracy
0.6
60
0.4
40
Evenly label weighting
Ours w/o label weighting
Ours
20
0
100
200
300
400
Training Step
Olympiad.
GPQA-D.
(a) Accuracy curve on MATH500 benchmark.
(b) Performance for different values.
Figure 5: Effect of Pseudo-Label Weighting. Pseudo-label Weighting prevents training collapse, and the hyperparameter a can control the "skewness" the pseudo-label weight distribution.
Pseudo-label weighting is crucial to avoid training collapse To assess the impact of our pseudo- label weighting module training and performance, experiment, we apply prompt-level weighting, negative rollout penalization, and use the majority vote answer as pseudo label. In the second experiment, we replace the frequency-based soft weights with uniform weights over all targets for each prompt. Figure Sa reports the outcome: without pseudo-label weighting, training becomes unstable and eventually fails. Uniform weight- ing performs even worse, accelerating degradation and leading to an earlier collapse. This shows that merely considering all targets is insufficient-low-ffee low-frqueecc pseudo-labels are typically erro- neous/noisy, and assigning them the same weight as high-frequency (likely correct) pseudo-labels can steer the model in the wrong direction. In contrast, frequency-bane soft weighting suppresses rare noise and stabilizes training.
ablation experiments. In the first
Hyperparameter in Pseudo-label Weighting a controls the "skewness" or concentration of the prompt-level weight distribution. When is very small, the weighting approaches step-like function that sharply distinguishes majority from minority answers, effectively behaving like hard majority voting and largely ignoring less frequent responses. In contrast, large produces broad, flat distribution, leading to softer, more evenly spread weights across answers. From [Fiuure 5b, smaller 0.1) underperforms because gives much influence noisy, infrequent
8

---

Published as a conference paper at ICLR 2026
Benchmarks
Benchmarks
math.
math.
60
alme25
Accuracy yayacyy (Xg]
60
aime25
Accuracy
olym.
olym.
40-
minerva.
40
minerva.
mmlu.
mmlu.
gpqa-d.
gpqa-d.
0.1
5
3
8
(a) Performance of our RESTRAIN with different nega- tive advantage offict 6.
(b) Performance of our RESTRAIN with different ma- jority count threshold K, the number of rollout experiment 16.
Figure 6: Effect of Pseudo-Label Weighting. Model performance is sensitive to hyperparameters in Negative Rollout Penalization.
answers. Conversely, a larger a (a = 1) is also suboptimal as it fails to leverage valuable signals from correct minority responses. Thus, o 0.5 provides the best balance, effectively filtering noise while retaining the full distributional signal from the model's outputs.
Hyperparameters in Negative Rollout Penalization Figure 6a ablates the negative advantage offset 6, which dictates the magnitude of the penalty applied to low-consensus rollouts. The results demonstrate that the model's performance is sensitive 8. With the penalty disabled (5 0), the model simply ignores those low-confidence prompts. Performance is similar to the ablation without Negative Rollout Penalization (Table 4), indicating that simply discarding low-confidence prompts does not hinder training. The best accuracy occurs at ó 1, suggesting that a moderate penalty effectively discourages the model from generating noisy, low-confidence outputs, thereby stabilizing the training signal and enhancing reasoning capabilities. When the penalty magnitude is increased further to 6=2 and 5=5, a consistent and sharp decline in accuracy is observed across all benchmarks. This indicates that an excessively large penalty is detrimental, likely because it over-penalizes the model and may suppress potentially correct, albeit low-frequency, reasoning paths.
Figure 6b) varies the majority size threshold for triggering the negative penalty; the penalty is applicd if the count of the most frequent answer is less than k. The data reveals a similar trend where performance suboptimal at both low and high values of 6. peaking at value of к 3. threshold that is too lenient (k 2) fails to penalize many noisy, low-confidence training examples, thus limiting performance improvement. Conversely, threshold that too strict (w or 8) suppresses potentially valid reasoning paths in outputs with moderate consensus and causes significant drop in accuracy. Therefore, threshold value strikes a crucial balance, effectively filtering unreliable training signals without excessively restricting the model's learning process.
RELATED WORK
RL with Verifiable Rewards RL has shown great promise in improving LLMs, as demonstrated
by the success RL from human feedback (RLHF) and from Al feedback (RLAIF), which aligns
model responses with human preferences (Lee et al. 2023 Ouyang et al.) 2022 Liu al. 2025b
Yue ct al. 2025). More recently, reinforcement learning with verifiable rewards Gao et al.
2024
Shao et al.
2024 Guo al.) 2025 Yang al. 2025 Wen et al. 2025 Song al. 2025 Team
et al. 2025
Fatemi et al. 1025:/arm et al. 2025a] L et al. 2025b) has been developed to further
enhance reasoning capabilities
domains such mathematics and code. Despite its promise,
RLVR is largely limited to settings where a verifiable gold label or exhaustive validators exist, and
its outcome-based rewards may limit generalization to tasks that are out of distribution.
Unsupervised Reward Estimation Accurately capturing reward signals without relying on hu- man labels has been the focus of several recent studies. Early work like STaR (Zelikman et al. 2022] relies on repeated outcome evaluation. Self-Rewarding LMs (Yuan et al. 2024) explores using LLMM-as-a-Judge provide its own rewards to do self-training. SCPO (Prasad et al. 2024)

---

Published as a conference paper at ICLR 2026
introduced self celfisnnsst an alternative to human-annotatie rewards, demonstrating its effec- tiveness in improving reasoning tasks through (iterative) DPO training. Building on these ideas, TTRL (Zuo et al. 2025) further explored self-consistency signals in an online setting. which treats the majority-voted answer as a pseudo label and leverages the GRPO algorithm (Shao et al. 2024) update the model. However, TTRL was found suffer from overconfidence issues, resulting in mode collapse. To address this, SRT (Shafayat et al. 2025) proposed using offline-generated labels and curriculum learning; ETTRL (Liu et al., 2025a) proposed an entropy-based mechanism that enhances the balance between exploration and exploitation, thus mitigating overconfidence and improving overall performance; EVOL-RL (Zhou et al. (025) introduced novelty reward to in- crease exploration. Other unsupervised methods derive intrinsic rewards from model's internal Learning from Internal Feedback (RLIF). For example, some approaches measure the model's output certainty, using metrics like token- and trajectory-level entropy (Prab-) hudesai et al. [202
Beaubackk-Reifforceeee
Agarwal et al., 2025) or self-confidence (Li et al. 2025a). Along these lines, Intuitor (Zhao et al. 2025) utilizes model's 's internal confidence termed "selt-certainty" as its sole intrinsic reward. Another method, EMPO (Zhang et al. (2025a), uses clustering to extract semantic entropy across multiple rollouts and compute corresponding advantages. [Zhang et al. (2025b) the- oretically analyzes internal equivalence among RLIF methods and claims that the prior of the base model causes training collapse.
Unlikelihood Penalization Unlikelihood training is widely adopted technique in neural text generation to penalize undesirable outputs. (Welleck et al. 2019) reduces the probability of spe- cific "negative candidate" tokens. et al. 2019) later employed this approach to improve logical consistency, demonstrating its effectiveness as a general framework for mitigating known biases in dialogue by penalizing carefully selected set of negative tokens at each generation step. More recently, NSR (Zhu et al. (025)) extended this principle from the neural text generation model to LLMs post-training with their Negative Sampling Rejection (NSR) method. In the context of RLVR, they show that penalizing entire negative trajectories consistently improves performance, preserves generation diversity, and promotes generalization over the base model.
7 CONCLUSION
In this paper, we propose RESTRAIN, self-penalizing reinforcement learning framework that trans- forms the absence gold labels into a learning signal, enabling models to self-improve with- gold labels. By (i) weighting all predicted targets rather than only the majority, (ii) penal- izing low-confidence rollouts within the policy objective, and (iii) discounting prompts with low self-consistency, RESTRAIN enables robust self-improvement and mitigates the training collapse of majority-vote heuristics. Empirically, it delivers more stable optimization and stronger generaliza- tion on challenging reasoning tasks like math and science.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505. 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05 2025.
Jiaxuan Gao, Shusheng Xu. Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint 24Xi 2024.
10

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103. 103.03344, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint 1rXXi:24112.17200 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint
2023.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13(9):9, 2024.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprinl arXiv: 191 1.03860, 2019.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.0633 2025a.
Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lan- chantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.025: 2025b.
Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, FeiYang Shen, ShaoGuo Liu, and TingTing Gao. Eurl: Balancing exploration and exploitation 11m test-time reinforcement learning via entropy mechanism. arXiv preprint arXiv:2508. 1356, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 2.Xi::2503.20783 2025b.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time 19393
scaling, 2025. URL https::/arki.oor//as/////
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-277443 2022.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22666 2025.
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference opti- mization. arXiv preprint 24..:::441..0400000 2024.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint 20.xi:25005.2444/:/ 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint a0Xiv:2402.03300, 2024.
11

---

Published as a conference paper at ICLR 2026
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient train- ing rikkike reasoning models. arXiv e-prints. pp. arXiv-2503, 2025.
Felix Stahlberg and Bill Byrne On nmt search errors and model errors: Cat got your tongue? arXiv preprint arXiv: 190::::98.00000 10090, 2019.
Felix Stahlberg, Ilia Kulikov, and Shankar Kumar. Uncertainty determines the adequacy of the mode and the tractability of decoding in Soquence-to-sequenee models. arXiv preprint arXiv:2204.00471, 2022.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint (tti:2500.125999 2025.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057, 2025a.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Fei Xia, Ed Chi. Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, arXiv preprint wxXv:1908.00439, 2019.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An. Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. /0460, 2025.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Ping Yu, Jack Lanchantin, Tianlu Wang, Weizhe Yuan, Olga Golovneva, Ilia Kulikov, Sainba- yar Sukhbaatar, Jason Weston, and Jing Xu. Cot-self-instruet: Building high-quality synthetic prompts for reasoning and non-reasoning tasks. arXiv preprint arXiv:2507.2 23751,, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2rXhv:2503.14477 2025b.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXh::2401.1022.3 3, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar arXiv:2504. 133337 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-15488 2022.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. 255::25000008/2 2025a.
arXiv preprint
12

---

Published as a conference paper at ICLR 2026
Yanzhi Zhang. Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Ilm reasoning. arXiv preprint arXiv:2506.17211 2025b.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning reason without external rewards. arXiv preprini arXiv:2505. 19099990,,9 2025.
Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang. Haitao Mi, and Dong Yu. Evolving language models without labels: Majority 2025.
drives selection, novelty promotes variation. arXiv preprint arXiv:2509.11
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in Ilm reasoning. arXiv preprint arXiv:2506.013477 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.1608 2025.
13

---

Published as a conference paper at ICLR 2026
A
A PSEUDO CODE OF THE RESTRAIN LOSS FUNCTION
This section shows pseudo code of our RESTRAIN loss function calculation for one single prompt.
Listing 1: The pseudo-code of the RESTRAIN loss function for one prompt
def cestrain_lossaurtte prompt.weight, threshold, neg_offset):
# --- Extract answers
answers anttracttanserrrortttt (output) for output in outputs]
# --- Majority size M(x)
counts = Counter(answerrs,
counts. (mm.mtttcooooon
#
10
# Branch 1: Negative penalization
11
#
12
if Mx threshald:
13
rewards [0.0] (on(put)t
14
adv salculate_advantages adv [a neg_offset for in adv] loss calculate_loss(anl
15
16
17
return prompt_weight loss
18
19
20
# Branch 2:
Pseudo-label weighting
21
# Calculate label weights
23
freqs counts. values() lon(out)tt))
24
label_weights = weihlltth_hbllleeehhthht
25
26
# Calculate each label loss, then weighted to final loss final_loss 0.0
28
for i, label in (counts..
29
rewards [reward_fnnans, label) for ans in answers] adv calculate_ aaailaateeevvanteeeseeeeeaee
30
loss lorillateeoossaadee final_loss [abel_weights[i] loss
32
33
34
return prompt_weight final_loss
14

---

Published as a conference paper at ICLR 2026
B AN ALGORITHM OF THE PER-PROMPT RESTRAIN LOSS FUNCTION
Algorithm 1: Per-prompt RESTRAIN Loss
Input : Responses O = {o1.... 0, prompt weight negative offset 5 0.
0: majority threshold K;
Output : Loss L.
A = {ai,..... am) Set (e((ExtrractAnnwe((aa)))ll)l M(x) <- max(Count(a))
((r) < then
0 Vi; adv < Cal.ullateedvaartgessst...
adv, t adv; 5 Vi;
return L t U2 CClculateloss(advv)
else
for j = to do | fj - (t))/m;
" t (alccllateweighh((())
Z m Wyj
Wj /jj
Lfinal +- 0;
for j 1 to do
11
T. + RewardFn(AA]],, a,) Vi; adv +- Galculaaeeddvattaeessr???
oolculatelosssadyy Iitinal & Isinal + W {j
12
13
return T. <- U2 Teinal
15

---

Published as a conference paper at ICLR 2026
C DISCUSSION OF MOTIVATION
Distribut of Count Correct Base Step
1600
Count
Connect
Leaat Correct
3000
-94
-02
Coune
Figure 7: Statistics Majority Vote Count and Pass@32. The model is trained with Pseudo-label Weighting and Prompt-level weighting. We select checkpoint when the training converges and use the checkpoint to do inference on the training set to analyze the majority vote count and pass @ 32.
summarizes majority-vote statistics step 510 for Qwen3-4B-Base trained with pseudo-label and prompt-level weighting on the DAPO dataset. The x-axis represents the major- ity vote count. The chart highlights two key trends: (a) The red line shows the Majority Vote Correct Ratio. As the majority vote count decreases (moving left on the graph). the probability that the most frequent answer actually correct drops almost linearly.
Figure 7
(b) The orange line shows the At Least One Correct Ratio (i.e. @ass@k). This is the probability that at least one of the gener- ated responses was correct, even if wasn't the majority answer. This distinction is important for understanding different training methods. A method like TTRL is highly dependent vote being correct (the red line). When the consensus is low (a low majority vote count), TTRL receives an unreliable and often incorrect training signal. Our proposed method, however, relies on the principle of at least one correct answer being present (the orange line). As long as one of the generated responses is correct, our model receives valid positive signal for training. This makes it more robust, especially in cases where there isn't 't a strong consensus on the correct answer. How- ever, the chart also reveals critical weakness. For very low majority vote counts, the orange line shows a dramatic drop. This indicates that when the model's consensus is extremely low, it's highly probable that none of the generated responses are correct. In this scenario, our method is exposed to significant training noise because there is no positive signal to learn from. To address this specific problem, we introduce our negative rollout penalization to discourage the model from generating sets of answers where none are correct.
the majority
16

---

Published as a conference paper at ICLR 2026
D DETAILED RESULTS
D.1 BENCHMARKS
Our benchmark suite comprises six publicly available datasets spanning mathematics (four) and science (two). (1) MATH-500@Heddrycks et al. (021): 500-problem subset of the MATH cor- pus, emphasizing competition-style problems across algebra, geometry, number theory, and com- binatorics. (2) AIME25 (Li al. 2024): the official 2025 American Invitational Mathematics Examination questions. (3) OlympiadBench (math subset) (Yang et al., 2024): olympiad-level problems sourced from nationall/ntermationn contests;
use the mathematics portion only. (4) Minerva math (Yang et al., (024):: the mathematics split from the Minerva quantitative-reasone
suite. (5) MMLU STEM (Yang al., 2024): the STEM categories of MMLU (e.g.. physics, chem- istry, biology, mathematics-adjacee
subjects). (6) GPQA-Diamond (Yang et al. [024]] the highest- difficulty split of GPQA with expert-written, graduate-level science questions spanning physics, chemistry, and biology.
In addition the benchmarks reported in the main paper, we evaluate on three additional bench- marks. They are (1) AMC23 et al. 2024): prompts drawn from the 2023 American Mathematics Competitions, covering core high-school
mroblem-solving domains. (2) AIME24 (Li et al.) 2024);
the official 2024 American Invitational Mathematics Examination questions. (3) s1k (verifiable
subset) (Muennighoff et al. 2025): a subset of 893 s1k examples with verifiable answers from Yu
et al. 2025a).
D.2 IMPLEMENTATION DETAILS
We implement TTRL, SRT, and RESTRAIN using the VERL codebase. To validate correctness, we reproduce representative experiment from the original papers with implementations and verify that the resulting accuracies match. Since ETMR has not released code, we report its results as stated in the original paper. For hyperparameters, we use a learning rate of 1 10 and adopt the AdamW W optimizer for the policy model. We set kl loss coefficient to 0.001, and the entropy coefficient to For rollout, we sample 16 responses using temperature of 1.0 for training. The maximum generation length is set to 4096 for Qwen3-4B-Base and Llama3. 1-8B-Instruct and 8192 Octothinker Hybrid 8B base model. We employ unified hyperparameter configuration for RESTRAIN across all experiments. Specifically, we set the mean for the pseudo-label/prompt weight to 1.0, the bias o = 0.5, the negative advantage offset = 1.0, and the majority size threshold 3. We set the number of epochs to 20. All experiments were conducted on 32 NVIDIA A100 80GB GPUs.
D.3 ADDITION RESULTS
Table[5land show full results of our RESTRAIN on nine benchmarks. Results show that our method can outperform all unsupervised methods on both Qwen3-4B-Ban and Octothinker Hybrid 8B base models with two different training datasets.
Table
show experimental results on three different capacity models: base model: Qwen3-1.77 1.7B- Base, math-specific model: Qwen2.55math-775 and an instruct model: Llama-3. 1-.----IInsttec cross two training datasets(DAPO- 1---P-4--mmthhh and NuminaMath-100k Consistent with our findings in the main result section, RESTRAIN outperforms the TTRL. baseline under all settings. This confirms that our self-penalizztion mechanism is also effective for instruct models. By validating across Qwen (Base and math), Octo Thinker (Specialized Mid-traine tid-trained), and Llama (Instruct), we demonstrate that RESTRAIN generalizes across model families and training datasets.
17

---

Published as a conference paper at ICLR 2026
Table 5: The table shows the evaluation results of training Qwen3-4B-Ban on 14k DAPO dataset, all results(%) averaged over 16 seeds. The best results are highlighted in bold.
Model
math.
ame.
aime24ime22olym. miner. mmlu. gpqa.
slk
avg
Qwen3-4B-Bal B-sa
68.0
45.6
10.4
7.9
35.4
26.0
58.3
32.2
5.1
32.1
w/ access gold label
GRPO
85.0
69.3
21.2
20.8
50.1
40.1
73.7
38.7
12.2
45.7
w/o access to gold label
TTRL
76.3
52.6
12.0
8.3
39.6
35.9
59.4
33.6
4.6
35.8
SRT (easy prompt)
77.8
52.3
13.5
7.9
39.7
36.3
60.5
34.9
5.6
36.5
SRT (offline majority label)
76.9
51.8
10.4
12.0
39.8
34.2
59.4
34.5
4.7
36.0
RESTRAIN
83.0
60.2
20.3
17.9
47.0
36.5
80.9
40.2
10.3
44.0
Oct.Hybrid-8B-BB
29.8
16.1
1.9
0.8
12.1
9.3
8.6
24.6
2.1
15.0
w/ access gold label
GRPO
71.7
49.4
10.8
6.2
35.2
31.3
62.0
31.0
33.9
w/o access to gold label
TTRL
56.5
32.2
3.9
2.7
23.2
22.1
51.7
27.3
3.5
24.8
RESTRAIN
61.6
33.6
6.0
8.5
24.6
25.0
64.6
29.9
4.4
28.7
Table 6: The table shows the evaluation results of training Dwen3-4B-Bass 5k Synthetic S1k dataset, all results(%) are averaged over 16 seeds. The best results are highlighted in bold.
Model
math. amc. aime24ine25lym. miner. mmlu, gpqa.
slk
Avg. 1
Qwen3-4B-Bal
68.0
45.6
10.4
7.9
35.4
26.0
58.3
32.2
5.1
32.1
w/ access Qwen3-4B label
GRPO
83.7
64.5
23.7
18.9
48.4
39.7
83.5
43.5
11.5
46.4
w/o access to Qwen3-4B label
TTRL
76.0
50.2
10.8
9.2
39.3
35.9
37.6
32.8
4.8
35.2
SRT (easy prompt)
76.4
52.3
12.1
8.1
39.6
34.8
57.5
33.0
4.9
35.4
SRT (offline majority label)
75.8
53.3
11.9
10.4
39.2
33.1
57.1
33.1
4.6
35.4
RESTRAIN
81.7
58.4
17.9
20.0
45.5
36.5
73.4
40.0
8.8
42.5
Table Additional results on base model, a math specific model and an instruct model cross two training datasets.
Model & Dataset
Method
MATH500
AMC23
AIME24
AIME25
Olym.
Avg
Base
52.8
44.0
16.6
3.3
17.8
26.9
Qwen2.5-Math-7B
Gold
79.7
64.0
28.0
14.6
39.6
45.1
(1Ok NuminaMath)
TTRL
77.9
61.5
18.0
8.0
37.2
40.5
RESTRAIN
79.3
62.5
26.7
13.9
40.7
44.6
Base
56.4
30.1
4.5
3.9
22.8
23.5
Qwen3-1.7B-Base
Gold
69.9
42.0
9.7
3.1
31.5
31.2
(10k NuminaMath)
TTRL
63.2
36.2
5.4
3.5
26.0
26.8
RESTRAIN
66.9
41.0
8.3
5.2
29.6
30.2
Base
49.6
24.4
5.4
0.8
17.2
19.4
Llama-3.1-8B-I (14k DAPO Math)
Gold
53.4
28.9
6.4
0.2
20.2
21.8
TTRL
47.5
25.0
6.1
0.8
17.3
19.3
RESTRAIN
51.2
26.4
7.1
1.1
17.3
20.6
18

---

Published as a conference paper at ICLR 2026
E ABLATION STUDY OF PROMPT-LEVEL WEIGHTING
In this section, we evaluate the effect of prompt-level weighting on training. We ablate by two experiments, one is comparing our oflline prompt weighting with online prompt weighting, another is setting all prompt weights to I ("w/o prompt weighting"), As shown in Figure 8) in the first experiment, online prompt weighting quickly collapses while offline prompt weighting can continue to improve. For the second experiment, both methods' accuracies improve quickly at the start. Initially, the model without prompt weighting learns slightly faster. However, our method soon overtakes it and consistently maintains higher accuracy. Notably, the performance of the model without prompt weighting becomes unstable and drops sharply after 1,500 training steps. contrast, method's 's accuracy remains stable and continues to improve. This suggests that offline prompt- level weighting is key to achieving both higher final accuracy and greater training stability.
MATH500 Accuracy
0.6
0.4
0.0
0.2
0.6
Weighting
Online Prompt Weighing
100
140
160
promgt wegttng
Training Step
Thiniin
000
2000
(a) Online Prompt Weighting collapses very quickly at (b) Without Prompt Weighting, the model will ulti- around 100 steps.
mately collapse.
Figure 8: Offline Prempt-weighting can help model train stable.
19

---

Published as a conference paper at ICLR 2026
F STUDY OF HYPERPARAMETER WEIGHT BIAS IN PSEUDO-LABEL WEIGHTING
In this section, examine the bias parameter used in pseudo-label weighting. Table reports the tuning results. When a is small, the scheme effectively reduces to selecting the majorit-vott answer as the pseudo label; when values near zero or above lead to training collapse and substantially worse performance, whereas a moderate setting (e.g., a 0.5) yields the best stability and accuracy.
large, approaches uniform weighting. We observe that
Table 8: Ablation of Pseudo-label Weighting. The table shows the evaluation results of training Qwen3-4B-Base 14k DAPO-Math dataset by varying the hyperparameter weight bias, all re- sults(%) are averaged over 16 seeds. The best results are highlighted bold.
Target Level Weighting Bias o Small o skewed on majority label
Large d evenly dist. on all labels
math.
aime25
olym.
minerva.
mmlu.
gpqa-d.
Avg.
(0 weights on non-majonn non-majority labels)
67.8
7.7
34.7
24.1
58.6
32.1
37.5
= 0.1
73.4
2.7
36.0
34.4
62.4
31.3
40.0
0.25
76.5
9.6
39.7
32,6
60.52
34.4
42,2
= 0.5
83.0
17.9
47.0
36.5
80.9
40.2
51.0
= 1
65.1
7.3
33.4
24.3
59.0
32.8
37.0
2
66.2
6.2
33.1
23.8
58.9
31.4
36.6
5
61.1
5.8
32.6
23.7
58.4
33.3
35.8
(evenly distributed)
66.8
6.9
34.6
24.6
59.8
32.9
37.6
20

---

Published as a conference paper at ICLR 2026
G STUDY OF HYPERPARAMETER NEGATIVE ADVANTAGE OFFSET Ó AND MAJORITY COUNT THRESHOLD K
In this section, examine how the negative-advantage offset ő and the majority-count threshold k influence performance. The offset 5 scales the penalty applied to low-consensus rollouts; if set high, oevo-vveeallleee penalizess the policy and induces a sharp accuracy decline. The threshold к decides which prompts are treated as low-consensus: a strict threshold discards many informative examples and hurts accuracy, while an overly loose threshold admits noisy cases and weakens the intended penalization. Appropriate, balanced choices of and к suppress noise without sacrificing useful signal.
Table Results on Different Negative rollout Penalty. The table shows the evaluation results of training Owen3-4B-Bais on 14k DAPO-Math dataset varying the negative advantage offset, all results(% %) are averaged over 16 seeds. The best results are highlighted in bold.
Negative Advantage Offset
math.
aime25
olym.
minerva. mmlu.
gpqa-d.
Avg. 1
76.5
9.8
40.5
31.8
58.3
33.2
41.7
0.1
78.7
13.3
40.8
36.5
59.3
35.5
44.0
=
83.0
17.9
47.0
36.5
80.9
40.2
51.0
= 2
70.6
10.4
36.7
27.9
56.6
32.6
39.1
= 5
70.4
7.1
36.1
25.0
56.9
31.6
37.9
Table 10: Results Different Majority Count Threshold. The table shows the evaluation results of training Qwen3-4B-BB 4B-Base on 14k DAPO-Math dataset by varying the weight bias, all results(% averaged over 16 seeds. The best results are highlighted in bold.
Majority Count Threshold (for negative rollouts)
math.
aime25
olym.
minerva.
mmlu.
gpqa-d.
Avg. f
2
77.4
8.8
41.3
33.8
59.8
34.2
42.5
K 3
83.0
17.9
47.0
36.5
80.9
40.2
51.0
K 5
78.2
13.3
40.4
29.2
61.4
35.6
43.0
K
67.3
6.0
34.1
24.5
59.3
33.7
37.5
To further diseuss K, in experiments, fixed the rollout number pling temperature at 1.0. Crucially, we utilized a fixed x = 3 across all different training datasets (DAPO-14K, Synthetic S1k) and models (Qwen3-4B-Base single configuration consistently outperformed baselines, suggesting that k 3 serves as a robust generalist setting within the standard rollout regime (n 16).
16 and the sam-
DctoThinker-BB-Hyorrrd Sa-------bdddBBaas
This
However, & is naturally coupled with the rollout number (n) and the model's capability. x acts as gate for Negative Rollout Penalization, defining the minimum consistency required to consider prompt's signal "reliable" enough to avoid penalization. Relation to Rollout Number (n)) If n increased significantly (e.g.. from 16 to 100), should likely be scaled represent similar ratio of celf-consistency. Relation to Difficulty/Assupption of Negative Trajectories: Increasing represents a stronger assumption regarding negative trajectories. higher treats a larger portion of low-consensus outputs "noise" to be penalized. For extremely hard tasks where answers rarely achieve consensus, lower A might be necessary to avoid suppressing the rare correct signal. Conversely, for easier tasks where the model is generally confident, higher & could further enforce strict consistency. While our results show that x = 3 is empirically stable, we acknowledge that remains tunable hyperparameter that governs the trade-off between suppressing noise and preserving minority signals.
correct
21

---

Published as a conference paper at ICLR 2026
H DISCUSSION OF PASS @ vs. MAJORITY VOTE PERFORMANCE
In this section, we want to study how the gap between Pass@ and Majority Voting evolves. We conduct experiments on the OctoThinker--B--Hyrrrd TTRL, and RESTRAIN across four benchmarks.
model. We compared the Base model,
The results (detailed in Table reveal three distinct critical findings:
RESTRAIN bridges the "Consistency Gap": The Base model exhibits massive discrepancy be- tween Pass and Majority Vote (e.g., a 34.6% gap on MATH500). This indicates the model often possesses the knowledge in its latent distribution but fails output reliably in single attempt. RESTRAIN drastically reduces this gap (e.g.. to 12.7% on MATH500), effectively converting the model's latent "majority potential" into reliable, single-shot performance.
RESTRAIN expands the "Knowledge Boundary" (Raising the Ceiling): key limitation of TTRL is that it often only aligns the model with its existing majority, yielding minimal gains in the up- per bound. MATH500, TTRL. only improved the Majority Vote by 3.0% (64.4% - 67.44 67.4%) At the same time, RESTRAIN increased the Majority Vote by 10.4% (64.4% - 74.8%). This demonstrates that RESTRAIN significantly enhances the model's reasoning capabilities, generat- ing correct reasoning paths that were not dominant in the base model.
RESTRAIN prevents "Ceiling Collapse" on Hard Tasks: On the most challenging benchmark, AIME24, we observe a critical failure in TTRL. TTRL caused the Majority Vote to drop below the Base model's performance (Base: 10.0% + TTRL: 6.67%). This suggests TTRL overfitted to spurious signals or easy patterns, degrading the model's ability to solve hard problems. In contrast, RESTRAIN successfully raised the ceiling to 16.67%. This proves our RESTRAIN protects against the model collapse often seen in TTRI. on difficult training tasks.
Table 11: @ass@1 Majority Vote (Maj) Performance. We calculate the Gap Maj Vote @ass@] I to highlight consistency.
Benchmark
Metric
OctoThinker--B--Hyyrii TTRL RESTRAIN (Ours)
Pass@1 1
29.8
56.5
62.1
MATH500
Maj Vote
64.4
67.4
74.8
Gap
34.6
10.9
12.7
@ass@1 1
12.1
23.2
24.0
OlympiadBench
Maj Vote
29.3
33.9
35.7
Gap
17.2
10.7
11.7
Pass@1 1
9.3
22.1
26.1
Minerva Math
Maj Vote
25.0
35.3
37.9
Gap
15.7
13.2
11.8
Pass 1
1.88
3.94
6.46
AIME24
Maj Vote
10.0
6.67
16.67
Gap
8.12
2.73
10.21
22

---

Published as a conference paper at ICLR 2026
I DISCUSSION OF COMPUTATIONAL COST
In this section, we want to discussion the computational cost of RESTRAIN. We claim that RESTRAIN has nearly identical computational overhead TTRL. Both methods share the same training hy- serparameters-sseciffeae
rollout number, maximum sequence length, batch size, and number of epochs. Consequently, the most ienturceeittensive operations (LLM generation and policy for- ward/backward passes) are identical.
RESTRAIN introduces only negligible overhead through lightweight operations on the generated rollouts: grouping unique answers, computing normalized pseudo-label weights, and applying consensus-gated offsets. These are simple operations that do not require additional model passes or parameters. Furthermore, the prompt-level weights are derived from one-time offline computa- tion, adding only a small constant setup cost rather than a recurring per-step burden.
To validate this, we conducted a runtime analysis on the Qwen3-4I-Base model, which was trained on the DAPO- 44-MMTTH dataset (see Table 12). The results confirm that RESTRAIN maintains training time per step comparable both TTRL and standard gold-label training. While TTRL may exhibit shorter total training time due to early stopping caused by model collapse, the computational cost per step remains equivalent.
Table 12: Computational Cost Analysis. Comparison of training time per step and average sponse length.
Method
Training Time (Step 1)
Avg, Response Length
Train with Gold Label
144s
758.0
TTRL
216s
746.8
RESTRAIN (Ours)
155s
776.8
23

---

Published as a conference paper at ICLR 2026
J DISCUSSION OF ADAPTING RESTRAIN TO PPO
The RESTRAIN framework inherently designed for group-based reinforcement learning meth- ods like GRPO. Since these methods already compute baselines relative a group of rollouts, integrating RESTRAIN's soft-weighting and penalization logic is seamless. However, adapting this framework value-based RL methods like PPO requires more intricate design choices. Specifically, because PPO relies on learned Critic (V₂) rather than group averages for variance reduction, we must explicitly define how train this Critic interpret RESTRAIN's signals without destabilizing the advantage estimation. We show potential adaption below as example.
To apply RESTRAIN to PPO, we must translate its group-level signals into scalar rewards that learned Critic (V)) can predict. The key shift is replacing the standard unsupervised "hard majority" baseline, where the most frequent answer gets a reward consensus" approach. The PPO Critic (V₂) is tasked with learning the expected consensus score rather than a binary success probability. This allows the advantage function A(s, a) = r ((s) to correctly capture nuance: rollout matching strong consensus yiclds positive advantage, while a rollout matching a weak consensus yields a smaller signal. To prevent the model from reinforcing "hallucinated majorities" where the group is confused (i.e., the majority count M(x) is below a threshold s), we intervene directly in the advantage estimation to apply a "penalization". Specifically, when the model confused (M k), we override the standard calculation with a penalty. We zero out the rewards for the group and inject negative offset 6, resulting in a final advantage calculation:
and others with RESTRAIN's "soft
Atnal (G)
Crucially, the Critic must be trained the unpenalized rewards rather than the penalty itself. This ensures (s) z 0, preserving the pure negative signal - the policy update. Finally, the en- tire PPO loss is scaled by an offline prompt-reliability score uz, derived from reference model, ensuring gradients are only applied on solvable prompts:
= U CPro(A)
24

---

Published as a conference paper at ICLR 2026
K THE USE OF LARGE LANGUAGE MODELS
In this work, we use LLM for writing polishing and do not use it for any other purpose.
25


Reviews:
[{"review_id": "mkZfbOEu21", "reviewer": "Reviewer_Fb6c", "summary": "This paper proposes a new GRPO-based loss for self-training. Specifically, it does not rely solely on majority-voted answers but instead treats all answers as potentially correct, assigning different weights and penalizing low-consistency examples. Through extensive experiments on math and science datasets and two different models, the authors demonstrate that their self-training method outperforms previous approaches.", "questions": "1. How can TTRL be adapted to other RL framework and what are the results?\n2. Could you provide results using Llama-3.1-8B-Instruct and Qwen2.5-Math-1.5B or 7B models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is intuitive, well-motivated, and clearly described. It is simple yet effective.\n2. The method achieves significant improvements over comparative self-training approaches across different models and two task domains.", "weaknesses": "1. The proposed loss and experiments are too closely tied to GRPO. It remains uncertain whether the method is compatible with other RL algorithms, such as PPO and PRIME, as compared by TTRL.\n2. The tested datasets and models are relatively limited. Although the proposed method shows promising results in this paper, it is unclear whether it is biased toward inherently stronger models with certain specialized skills (both are base models), which may not generalize well in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5B9Lpq6Zor", "reviewer": "Reviewer_7W3F", "summary": "This work aims to solve the training instability and collapse caused by unreliable majority-vote signals in unsupervised RL methods like TTRL. The paper proposes RESTRAIN, a self-penalizing framework that replaces TTRL's reward scheme. Its novel loss function stabilizes training by using soft, frequency-based weighting for all potential answers, actively penalizing low-consistency outputs, and down-weighting unreliable prompts identified by a frozen model. Experiments show RESTRAIN significantly outperforms TTRL, avoids training collapse, and nearly matches the performance of a fully supervised, gold-label baseline.", "questions": "1. The hyperparameters are clearly critical. How were the optimal values selected?\n\n2. Computational Cost: This method inherits the high computational cost of TTRL while introducing additional computation. Are these additional overheads significant compared to the TTRL baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work directly addresses the critical and well-documented stability problem of majority-vote-based unsupervised RL.\n\n2. The three components are well-motivated and logically designed to counteract the specific failure modes of TTRL, such as signal smoothing and noise penalization.\n\n3. The empirical results are a significant strength, demonstrating performance that not only avoids collapse but also approaches the upper bound of fully supervised gold-label training.\n\n4. The paper is supported by a thorough set of ablation studies that demonstrate the necessity of each component for stable training and final performance.", "weaknesses": "1. High Hyperparameter Sensitivity: The method's performance appears highly sensitive to its key hyperparameters, including the weighting skewness, the penalty threshold, and the penalty offset. The ablation studies show that performance drops sharply outside a narrow range of these values, which may hinder its general applicability and reproducibility.\n\n2. Limited Novelty: The contribution appears to be more of a successful systems-level engineering effort than a fundamentally new paradigm. The core components (soft weighting, curriculum learning, and negative penalization) are all established concepts. Overall, this work is more like an incremental \"patch\" on TTRL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRvfcCEiFy", "reviewer": "Reviewer_hpMx", "summary": "The paper proposes RESTRAIN, an RLVR framework that does not need ground truth answers for questions. It tackles the problem that majority-voted answer may not be correct, especially for hard labels, and designed a weight-smoothing method to consider infrequent answers (pseudo-label weighting, negative rollout penalization and prompt-level weighting). The experiments show relatively large improvements over other unsupervised baselines.", "questions": "- How does the gap between pass@1 and majority voting change after RESTRAIN training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The motivation is very clear (majority voting != correct) and the designed soft weighting of infrequent answers seem reasonable and directly tackle the problem of majority-voting.\n- The negative rollout penalization is also convincing in down-weighting too infrequent/long-tailed answers (where the model is unsure about).\n- From the experiment results, it seems the proposed RESTRAIN can largely improve over existing unsupervised RLVR methods like TTRL. Besides, they test on both Qwen3-4B-Base model and Octothinker Hybrid 8B, which demonstrate that the method is applicable to recent SOTA base models and various model sizes.\n- They also provided very detailed ablation studies and useful details for reproduction.", "weaknesses": "- Not sure if the threshold kappa will be sensitive to rollout numbers and sampling temperature. Is it fixed over different datasets (which have different difficulty)?\n- Typo in Table 1: w/ access to gold labe -> w/ access to gold label", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)