How Far Can Unsupervised RLVR Scale LLM Training?
Authors: Bingxiang He
Yuxin Zuo
Zeyuan Liu
Shangziqi Zhao
Zixuan Fu
Junlin Yang
Cheng Qian
Kaiyan Zhang
Yuchen Fan
Ganqu Cui
Xiusi Chen
Youbang Sun
Xingtai Lv
Xuekai Zhu
Li Sheng
Ran Li
Huan-ang Gao
Yuchen Zhang
Lifan Yuan
Bowen Zhou
Zhiyuan Liu
Ning Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Unsupervised Reward
Reinforcement Learning
Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We revisit unsupervised RLVR through intrinsic and external rewards, unifying existing methods, analyzing their impact on confidence and failure modes, discussing their potential applications.

Abstract: Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels.
Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.
In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments.
We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution
This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned.
Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices.
Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability.
Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling.
Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.
Code is available at \url{https://github.com/PRIME-RL/TTRL}.

Published as a conference paper at ICLR 2026
How FAR CAN UNSUPERVISED RLVR SCALE LLM TRAINING?
Bingxiang He*!, Yuxin Zuo Zeyuan Liu"!, .Shanziqi Zhao Zixuan Fu' Junlin Yang', Cheng Qian', Kaiyan Zhang 1,5, Yuchen Fan", Ganqu Cui?, Xiusi Chen', Youbang Sun', Xingtai Lv!, Xuekai Zhu Li Sheng', Ran Li', Huan-ang Gao', Yuchen Zhang", Lifan Yuan', Bowen Zhou Zhiyuan Liu Ning Ding
'Tsinghua University *Shanghai AI Lab " an Jiaotong University *VIUC
Ar Shanghai Jiao Tong University 'Peking University
hebx2 @ma ils testttinnggua edu. en, dingningemail tsinghua. edu.
ABSTRACT
Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish unified theoretical framework revealing that intrinsic methods converge toward sharp- ening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, show intrinsic rewards consistently follow rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving trainability. Finally, we explore external reward methods that ground verification in computational asymmetries. showing preliminary evidence they may escape the manideence-orreectenn ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives. Code is available https:// /ttt://///thh Com/PRIME RLLTTRL.
practical indicator for RL
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) has been central recent breakthroughs in enhancing reasoning capability in large language models (LLMs). In RLVR, models learn from rewards that can be verified against ground truth, such as correctness in mathematics or successful code execution. Recent leading models including DeepSeek-RI R1 (Guo et al., 2025), Gemini 2.5 (Comanici al., 2025) and the Qwen3 series (Yang et al., 2025; Team, 2025) have achieved remarkable performance mathematics, coding and science benchmarks by scaling supervised RLVR. However, on the path toward superintelligence, this approach faces a crucial limitation: scaling supervision requires prohibitively high human costs, and as models reach surpass human expertise in specialized domains, obtaining reliable ground truth supervision becomes increasingly infeasible (Burns al., 2023; Silver & Sutton, 2025).
This supervision bottleneck has spurred growing interest Unsupervised RLVR (URLVR) (Zuo al., 2025; Zhao et al., 2025a), which derives rewards without ground truth labels. Just pretraining scaling laws (Brown et al., 2020; Raffel et al., 2020) transformed large-scale computation into
"Equal Contribution. Orders are determined randomly.
PProject Lead. Corresponding Authors.

---

Published as a conference paper at ICLR 2026
intelligence vast amounts of unlabeled data, URLVR promises extend this paradigm to post- training, scaling reinforcement learning beyond reliance on human-provided labels.
Current URLVR methods have primarily relied on leveraging the model's s intrinsic signals as re- wards. Many works have proposed intrinsic reward methods, from majority voting across multiple rollouts (Zuo et al., 2025) to entropy-band -aroop metrics (Agarwal et al., 2025) as rewards, reporting encouraging early training gains. Yet these gains come with growing concern,, as several studies highlight critical failure modes including reward hacking and model collapse (Shafayat al., 2025; Agarwal et al., 2025; Zhang et al., 2025c). Moreover, diverse methodologies have been applied across different model families and evaluation settings without systematic comparison or consensus on what constitutes reliable unsupervised rewards. This raises fundamental question for the field: Can intrinsic rewards truly scale L.M training?
To answer this, we conduct a comprehensive study of URLVR, spanning taxonomy. theory and extensive experiments. We begin by classifying URLVR methods into intrinsic and external based on the source of rewards (Section 2). We then theoretically analyze the underlying mechanism of intrinsic URLVR, revealing that despite diverse design choices, all intrinsic methods converge toward sharpening the model's 's initial distribution, amplifying existing preferences rather than discovering new knowledge (Section 3). This sharpening mechanism has both strengths and limitations, depending on the model prior: whether the model's confidence aligns with correctness. To validate this, implement intrinsic methods and show that intrinsic URLVR consistently follows a rise-then-fal pattern across all methods, differing only in when rather than whether collapse occurs (Section 4).
Despite these scaling limitations, intrinsic rewards remain valuable on small and domain-specific datasets, avoiding collapse even when all initial preferences are wrong, making well-suited for test-time training (Section 5). Furthermore, we can leverage the rise-then-fall pattern to measure model prior, proposing Model Collapse Step serve as practical model prior indicator, predicting RL trainability without expensive training runs (Section 6). Finally, since intrinsic rewards face fundamental scalability limits rooted sharpening mechanism, we investigate the scalability of external URLVR methods, taking ieff-veification as an example, which generate verifiable rewards through generation generation-verificatoo verification asymmetries rather than internal model states, showing that it exhibits sustained improvement without the collapse patterns inherent to intrinsic methods (Section 7).
Overall, our analysis charts a clear path from understanding why current intrinsic URLVR fails to identifying how future methods can scale. Our findings reveal that intrinsic rewards work within well-defined boundaries determined by the model prior, enabling efficient and safe gains in test-time training while risking reward hacking when confidence misaligns with correctness. These limitations motivate exploration of external reward methods, from generation- -verifictiiooo asymmetries in structured domains to self-supervised signals from vast unlabeled corpora, which offer pathways toward more robust and scalable improvement.
RELATED WORK
Reinforcement Learning with Verifiable Rewards. Recent advances in language model reasoning leverage reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2024), where models receive binary rewards based on answer correctness verified against ground truth. Leading systems including OpenAl's ol and o3 (Jaech et al., 2024; OpenAl, 2025), DeepSeek-RI (Guo al., 2025), Gemini 2.0 (Comanici et al., 2025) and Qwen3 (Yang et al., 2025; Team, 2025) have achieved remarkable performance through scaling supervised RLVR. However, this approach faces a fundamental bottleneck: as models approach human expertise in specialized domains, obtaining reliable ground-truth supervision becomes prohibitively expensive (Burns et al., 2023).
URLVR with Intrinsic Rewards. To address the supervision bottleneck, URLVR extends RL scalability beyond labeled data by using self eeff-generated proxy rewards. We distinguish two intrinsic methods derive rewards from a single policy's 's confidence, employing metrics such as self-certainty (Zhao et al., 2025b), token-level entropy (Agarwal et al., 2025; Prab- hudesai et al., 2025), trajectory entropy (Agarwal et al., 2025), raw probabilities (Li et al., 2025a) or attention patterns (Kiruluta et al., 2025b;a). (2) Ensemble-bar based methods leverage agreement across multiple rollouts, typically through majority voting (Zuo et al., 2025) or semantic clustering (Zhang al., 2025b). Subsequent works have refined this via reward-hacking analysis (Shafayat et al., 2025),
paradigms: (I) Certainty-
2

---

Published as a conference paper at ICLR 2026
tree search (Liu al., 2025), paraphrasing (Zhang et al., 2025d), multi-model collectives (Yuan et al., 2025) and intermediate reasoning consistency (Zhang et al., 2025a). Crucially, these methods "sharpen" existing preferences rather than discovering new knowledge.
URLVR with External Rewards. An alternative paradigm within URLVR escapes the intrinsic knowledge ceiling by deriving rewards from external mechanisms. One approach leverages unlabeled data, where the corpus provides latent ground truth for next-token prediction (Dong et al., 2025; Li al., 2025b), reasoning-step prediction (Wang et al., 2025; Hatamizadeh al., 2025), dual reconstruction objectives (She et al., 2025) or autonomous QA generation (Zweiger et al.. 2025; Akter verification asymmetries, where verifying solution is computationally cheaper than generating it. This principle scales via deterministic verifiers in mathematics (Simonds & Yoshiyama, 2025; Simonds et al., 2025), code execution (Zhao et al., 2025a) and formal theorem proving (Shao et al., 2025b; Hubert et al., 2025). Unlike intrinsic signals, these external rewards are grounded in objective computation or vast corpora, ensuring reward quality does not degrade model capability scales.
al., 2025). second approach exploits generation--errffcattoo
3 THE SHARPENING MECHANISM OF INTRINSIC REWARDS
We start from intrinsic reward methods in URLVR, investigating their potential in scaling LLM training. Existing studies (Zhang et al., 2025b; Zuo et al., 2025; Agarwal et al., 2025; Shafayat et al., 2025) have empirically assessed the strengths and weaknesses of these intrinsic methods, but the underlying mechanisms of how and why they lead to the observed effects, have been underexplored.
One-Step Update Dynamics. Training with intrinsic rewards involves moving toward an optimal policy zg defined by the KL-regularized objective: max E Ey [r y)] - 3Dx (Tyl Ter)). For a binary majority voting reward r(ar, y) 1 (yns(y) maj(Y) (Y)), the optimal probability mass the majority answer Ping is given by:
1/8
(k++1)
maj
P maj
P Pmai
(R)
2/8
(1 P Pmaj)
(I)
In practice, single gradient update does not reach the optimum but moves partway toward it. The
actual probability mass after one update p(++1)), satisfies the ordering Prij p=,(k++11) 2 Pimaj (R++1)
(k)
2 P Pimj"
This holds because policy gradients inerease mass on rewarded trajectories (lower bound) but are
constrained by the maximum expected reward of the optimal policy (upper bound). See Appendix A.
for detailed derivations and empirical validation.
Convergence to Sharpening. The one-step update creates a "rich-get neth-gee-richer" dynamic that trajecto- ries leading to the majority have their probabilities consistently increased. Iterating this process, the policy converges toward a deterministic policy concentrated on the initial majority answer:
Theorem 1. (Geometric Convergence). Consider the training process where each iteration k, we:
(I) sample N rollouts Ys from T (k) (2) compute majority (Yk). (3) perform one-step update with (k)
reward ( v) [ans(yy) maj (Yx)) to obtain Tgg (k+1) LLt p Pmmjj
(k)
E ya pyanstt)=maja. (Yx) To (z/²z
denote the probability mass on majorit trajectories iteration k.
Suppose the following assumptions hold, validated empirically in Appendix A.2:
(A1) Majority stability: maju majo for all k (holds for sufficiently large N);
(k+1) (k)
(A2) Effective learning: Pmaaj Penal for all k (a standard assumption in policy gradient methods).
(k) Then Pmaj converges geometrically to with rate 11/3 and the policy converges to
Σ [eee(1) x))
if ann(sy) = majo (Yo),
lim (k) (y) x)
y' ans(y mmaig"")maijy)) (Yn) Feret Tyer(y (x)
k
0,
otherwise.
3

---

Published as a conference paper at ICLR 2026
Trained with Majority Voting Reward
Trained with Cround Truth Reward
Majority Voting Reward
Reward Accuracy
Actor Enlropy
1.2
0M
Training Step
Training Stup
Training Stup
AIME 2024 Accuracy (2vg5232)
AIME 2025 Accuracy (2vg@322
AMC 2023 Accuracy (avgg
20 10 H8 Traning Stop
16 Step
Training
Step
Figure 1: Training dynamics comparing majority- vooingg training and ground-truth training. Intrinsic rewards initially match supervised performance but eventually collapse: the proxy reward rises while validation accuracy falls, revealing divergence between optimizing confidence and optimizing conrectness.
Complete proof provided in Appendix A.3. For other URLVR intrinsic rewards, we leverage the sharpening intuition and propose unified reward framework in Appendix A.4, showing that all intrinsic rewards can be understood through single lens: manipulating cross-entropy between carefully chosen distributions. Built upon this unified perspective, generalize this convergence analysis in Appendix A.5, and provide optimal policies induced respectively in Appendix A.6.
This convergence behavior has profound implications depending the model prior. When the model's ' confidence (reflected in maio) aligns with correctness, convergence reinforces good solutions. Conversely, if confidence is poorly aligned, the same mechanism amplifies errors, leading model collapse. Next, we will evidence this from an empirical perspective.
4 WHEN DOES INTRINSIC URLVR WORK?
Takeaways
Intrinsic URLVR universally follows rise-then-fall pattern across all methods. Early gains reflect monfidence-correcttii alignment the model's prior, while eventual collapse is inevitable when this alignment breaks down.
As derived in Section 3, intrinsic rewards sharpen distributions by amplifying the model's initial preferences. Here, we empirically investigate the practical boundaries of this mechanism. We first trace the lifecycle of intrinsic URLVR (Section 4)1,, finding early gains followed by collapse across all methods. We then conduct a fine- mee-grained per-pre per-problem analysis (Section 4.2), revealing that training primarily amplifies existing preferences rather than correcting them. These results confirm that success depends fundamentally on whether initial confidence aligns with correctness.
4.1 THE RISE AND FALL OF INTRINSIC URLVR
In this section, we conduct experiments across different hyperparameters and methods for intrinsic rewards, revealing that intrinsic URLVR follows consistent rise-then-fall pattern.
Setup. In main text, we take majority voting reward from TTRL (Zuo et al., 2025) as an example to compare intrinsic reward training against standard RLVR using ground truth labels. We train Sween--17B--aalt on DAPO-17K (Yu et al., 2025) using default hyperparameters from Table 6, and evaluate on AIME 2024 (Li et al., 2024), AIME 2025 (Balunović et al., 2025) and AMC 2023 (Li et al., 2024). Following standard practice, we generate 32 solutions per problem at temperature 0.6 with top-p 0.95, reporting average accuracy (avg@ 32). Beyond validation performance, we track three training dynamics: Majority Voting Reward (the intrinsic reward signal), Reward Accuracy (whether pseudo-reward matches ground truth reward), and Actor Entropy. See Appendix B. for complete details. Unless stated otherwise, we utilize the default setup above for later experiments.
4

---

Published as a conference paper at ICLR 2026
Results. Intrinsic rewards initially match ground-truth training but eventually collapse, and this pattern exists across all hyperparameter settings we tested. Figure 1 shows that majority-voting training matches or even exceeds ground-truth training three benchmarks during the early phase. But continued training reveals that while the proxy Majority Voting Reward keeps rising, both Reward Accuracy and validation performance decline, exhibiting obvious reward hacking. We also observe that training with majority voting reward drives down Actor Entropy faster than ground-truth training, establishing connection between reduced uncertainty and improved performance.
We further thoroughly tune four key hyperparameters across five intrinsic reward methods, including training temperature, mini-batch size, KL regularization and rollout number (Appendix B.3). We find that some choices matter significantly (e.g. mini-batch. size and rollout number), but nearly all settings eventually degrade, differing only in when rather than whether collapse occurs. Even with the most stable settings, collapse occurs roughly 1,000 steps, suggesting that this rise-then-fall pattern may reflect fundamental limitation rather than an engineering problem.
While this pattern is universal, the specific pathologies of collapse differ by reward formulation. We find that certainty bassed methods like raw probability tend to collapse toward brevity, whereas entropy-bused methods often pad sequences with repetitive tokens to drive down entropy. We provide a detailed comparison of these failure modes and their associated scaling limits in Appendix B.4.
4.2 FINE-GRAINED PER-PROBLEN ANALYSIS
Next, we examine individual training samples in detail. Our analysis reveals that training amplifies ini- tial preferences rather than correcting errors within the same problem. Surprisingly, this amplification can still generalize unseen problems and improve performance.
Setup. We train Qwen3-1.77 1.7----- on 25 randomly sampled individual problems from MATH500 using REINFORCE with Trajectory- Entropy as the intrinsic reward. Each problem trains for 100 epochs with batch size and 8 rollouts for baseline estimation.
Figure 2 shows 8 represen- tative cases and the others appear in Figure 8. We track greedy decoding correctness (heatmap) and whether the highest-reward sample is correct (green binary square-wave indicator). We observe four distinct patterns:
Results.
Uncertoin
False Uncertain
Folse Certain
ID 262
ID 146
ID 258
420
Amplifying success (ID 262, 146, 258): These problems start with cor- rect greedy decoding (blue epoch 0), and the highest-reward sample remains correct throughout training (green wave stays at I). Training sim- ply increases the model's confidence around this aiready-corred solution, shown by the deepening blue color.
131
222
ID 422
50
100
Epoch
Amplifying failure (ID 222, 422):
Figure 2: Training dynamics on individual representative problems. For each problem, the heatmap shows greedy decoding correctness across epochs (blue correct, red wrong; darker higher confidence), and the green wave indicates whether the highest-reward rollout is correct.
The highest-reward sample is almost wrong throughout (green wave mainly stays at 0). Training amplifies the model's confidence in these incorrect answers, shown by the deepening red color.
Wrong - Correct (ID 76, 131): Greedy decoding initially produces wrong answers (red), but the highest-reward sample is usually correct during training (green wave mostly at 1). This guides the model from wrong to correct, as shown by the transition (red - blue).
Correct Wrong (ID 420): The model starts correct (blue), but the highest-rewart sample fluctuates between correct and wrong (green wave alternates between and 0). This inconsistency causes the model gradually degrade from correct wrong (blue red).
5

---

Published as a conference paper at ICLR 2026
DAPO-32
DAPO-128
DAPO-512
DAPO-2048
DAPO-8192
DAPO-16384
Ground Truth Reward
Majority Voting Reward
Reward Accuracy
424
020
20 Stap
23 Training Step
110
Training Step
23 21
Figure 3: Effect of training dataset size from {32, 128, 512, 2048, 8192, 16384). Training with 32 or 128 samples maintains stable performance without model collapse.
Among the 25 problems, training flipped greedy correctness in just 3 cases (12%). The remaining 22 simply sharpened the model's initial preference regardless of whether correct wrong. Even for the 3 that flipped, the color still deepens over training, showing that sharpening happens regardless of whether correctness changes. This reveals training amplification rather than correction.
The key factor appears to be whether the highest-reward sample is mostly correct. Surprisingly, even when intrinsic rewards amplify failures training problem, the resulting sharpening still (OOD) problems. This suggests
generalize and improve performance on unseen out-of-distribun that the "knowledge" gained is not necessarily problem-specifii but reflects broader alignment between the model's confidence and correctness that can be exploited in certain regimes. Detailed results and analysis of this OOD cross-problem generalization are provided Appendix B.5.
5 How CAN SHARPENING FROM INTRINSIC URLVR BE APPLIED SAFELY?
Takeaways
Small datasets induce localized rather than systematic policy shift, even training wrong problems can yield gains, making test-time training a safe and practical application.
The previous section establishes that intrinsic URLVR drive convergence to deterministic policies and may result reward hacking, limiting its application in scaling LLM training. But do intrinsic URLVR always lead to model collapse, or can they be safely applied under specific conditions? In this section, we demonstrate that model collapse can be prevented when training data is sufficiently small and domain-speciii making intrinsic URLVR particularly suitable for test-time training scenarios.
5.1 SMALL DATASETS PREVENT MODEL COLLAPSE
In Section 4.2, we have seen how training on one problem works. Now we want to see whether the training dataset size influences the intrinsic URLVR performance.
Setup. We train Qwen3-1.7P --B-Bae using majority voting reward on randomly sampled training subsets from DAPO-17k with (32, 128, 512, 2048, 8192, 16384) samples. To ensure fair comparison, we fix global batch size to 32 and adjust epochs so all settings complete exactly 600 optimization steps. We monitor Ground Truth Reward, Majority Voting Reward, and Reward Accuracy to detect reward hacking. Results are verified across 3 random seeds for subset sizes {32, 128, 512].
Results. Figure 3 reveals that training with 128 samples maintains stable performance without collapse, while larger datasets (2 512) consistently exhibit reward hacking. Notably, DAPO-32 achieves rapid consensus (Majority Voting Reward + 1) while preserving high Ground Truth Reward, demonstrating that the model converges these problems without collapsing. This distinction holds across all three seeds, where DAPO-32 never collapses, while DAPO-512 always does.
Analysis. We hypothesize that small datasets induce localized overfitting rather than systematic policy shift. With only 32 problems, even perfect overfitting learns isolated facts rather than generalizable patterns. To test this, we measure the KI. divergence from the reference model at each training step:

---

Published as a conference paper at ICLR 2026
(2) log To (ylar)
DKi. (=06
RewDiii (112)
(3)
T
As shown in Figure 4, smaller subsets induce far smaller distributional shifts. DAPO-32 reaches only 0.057 KL after 600 steps, while DAPO- 512 reaches 2x higher. This aligns with find- ings that catastrophic forgetting correlates with distribution shift between fine-tuned and base policies (Shenfeld et al., 2025). The limited drift suggests the model sharpens confidence specific samples through localized parameter up- dates (Carlsson et al., 2024) without altering its global policy, thus preserving general reasoning on AIME24/AMC23. In contrast, large datasets require dense parameter updates that cause global policy shift, causing model collapse.
KL Loss
DAPO-32
DAPO-128
0.09
DAPO-512
0.06
0.03
0.00
80 120 160 200 210 280 320 500 440 480 560 600 Training Step
Figure 4: KL. divergence at each training step.
Remarkably, this localized effect is so robust that holds even extreme cases where the initial majority votes are entirely incorrect. We find that training on 32 samples with near-zero label accuracy still produces gains on OOD benchmarks like AIME24. We provide the detailed dynamics this fincorrect-majority experiment in Appendix C.I.
5.2 TEST-TIME TRAINING AS A SAFE APPLICATION
Building the small dataset insights, we examine test-time training where models are adapted directly on the target evaluation domains without ground truth labels, which may fit the small dataset size constraint. We investigate whether bring improvement while not leading collapse.
Setup. We train Boven3-1.7B-Be using majority voting reward on two settings: AMC23 (40 problems, test-time) and DAPO-17K (~17,000 problems, train-time). Both use batch size 40. We track Ground Truth Reward, Majority Voting Reward, and performance on both AMC23 and AIME24.
Results. Figure 5 shows that test-time training on AMC23 avoids collapse. Both Ground Truth Reward and Majority Vor- ing Reward rise and stabilize, with per- formance improving on both AMC23 and AIME24. In contrast, training DAPO- 17k shows the familiar rise-then-fall pat- tern. This indicates that intrinsic URLVR may be safely applied test-time training. And this explains why many recent works using intrinsic rewards focus on test-time rather than train-time settings (Zuo et al., 2025; Prabhudesai et al., 2025).
AMC23
DAPO
Ground Truth Reward
Majorily Voling Reward
AIME 2024 Accuracy (2)
AMC 2023 Accuracy (avg9332)
Truining Step
Training Step
Figure 5: Comparison between training and test-time.
How CAN WE MEASURE MODEL PRIOR?
Takeaways
We propose the Model Collapse Step novel indicator of model priors, which measures standard RL trainability by tracking reward accuracy collapses during intrinsic URLVR. This indicator achieves accuracy in assessing trainability par with running standard RL itself, but with higher efficiency; it outperforms pass@k, requires no ground-truth labels and remains robust multiple-choice problems.
Previous sections suggest that intrinsic URLVR is effective only when the model's initial confidence is aligned with correctness. This raises practical question: Can the strength of this alignment
7

---

Published as a conference paper at ICLR 2026
06.o-2-1124-2
Owwn2 MMath----
Qwen2.5-78
Owee3-83-RBm 8-8--a
Meta Eiato--amm-333 3.1-83
Qwes2.5-1.3
Qwan3-1.7B.
Model Collapse Step
Ground Truth Gain
Pass@k Gain
20.0
400
352
17.5
-17.0
60
+d6.0
350
Caan (AIME24)
Pass@l Cain (AIME24)
150
Collapse Step
300
203
2500
12.5
40
+.66.77
200
10.0
30
-32.00
150
160
7.5
+7.03
20
100
5.0
+5.16
2.5
10
0.0
Model
Model
Model
Figure 6: Comparison between Model Collapse Step (left) and Pass@k Gain (right) as predictors of RI. trainability, with GT Gain (middle) as reference. Later collapse strongly predicts better RI. performance.
be used to estimate the model prior rather than running expensive RL training to select the base model? The community typically selects by running full RL training on multiple checkpoints and considers such models to have a stronger prior. However, this method is costly and inefficient. Another standard approach is pass@k (Wu et al., 2025a), measuring by sampling & solutions without training. This works but also has limitations. First, this method has a significantly lower accuracy. when is sufficiently large.
And it also struggles with multiple-choice questions, where pass@k
In this section, we leverage the rise-then-fall pattern to propose Model Collapse Step, defined as the training step where Reward Accuracy drops below 1%, measuring how long model can sustain intrinsic URLVR before it starts to collapse. We find that different base models lead to different outcomes (see Appendix C.2 for pilot study Qwen and LLaMA families). Models with stronger priors remain stable for longer before collapsing. indicating higher suitability as base models for RL.
6.1 MODEL COLLAPSE STEP ACCURATELY PREDICTS RL GAINS
To validate Model Collapse Step as an accurate predictor of RL trainability, we compare against the widely-used pass@k metric and standard RLVR training.
Setup. We evaluate models from families (OLMo, LLaMA, Qwen) on AIME24, using three indicators: (I) Ground Truth (GT) Gain, measuring performance improvement from standard supervised RLVR on 1 epoch DAPO-17K training with ground truth rewards; (2) Pass@k Gain, calculated as the difference between p55 and pass@1 on AIME24; and (3) Model Collapse Step. measured as the training step when Reward Accuracy falls below 1 % during intrinsic URLVR training with majority voting reward with default hyperparameters. We evaluate the correlation between these indicators and use them to measure the model prior.
Results. As shown in Figure 6, Model Collapse Step correlates strongly with GT Gain. Models that survive longer with a larger Model Collapse Step consistently yield better results in standard supervised RL. This metric matches and even surpasses pass@k's predictive power and it cannot be gamed by random guessing on multiple-choice questions, demonstrating its reliability as an indicator.
6.2 MODEL COLLAPSE STEP RAPIDLY PREDICTS RL GAINS
While Model Collapse Step provides an accurate prediction, can it be computed efficiently? Based on insights from the hyperparameter tuning (Appendix B.3), we hypothesize that certain hyperparameters, such as mini-batch size and rollout number, can accelerate convergence without sacrificing the accuracy of the model prior indicator for RL trainability. This is valuable for practitioners evaluating multiple model candidates before committing to expensive RL training.
Setup. We test whether it remains consistent across different training hyperparameters by varying mini-batcl size E {1,8, 64} and rollout count E (8, 16, 32). For each setting, we measure for same 7 models and assess whether can preserve predictive power while reducing computation time.
8

---

Published as a conference paper at ICLR 2026
Results. Figure 33 shows that while aggressive hyperparameters (MBS = or N 32) accelerate collapse in absolute steps, the ranking of models remains relatively stable. This enables more rapid assessment for models with strong prior (2 50 steps less than before), effectively shortening the time required to measure model priors, making Model Collapse Step not only accurate but also rapid.
Notably, using optimized hyperparameters (MBS 1, N 8), the Model Collapse Step requires only 1.19B inference tokens, 5.6x fewer than a standard RL. training run (6.66B tokens). Detailed computation costs are provided in Table 7. While pass@k offers an alternative static metrie, our experiments show it correlates less reliably with actual RL gains (Figure 6). By measuring training dynamics rather than pre-traue pre-training performance, Model Collapse Step captures the interaction between model priors and the learning process itself. Moreover, it requires no ground truth labels, making applicable even when verification unavailable. For practitioners evaluating multiple model candidates for RI, this provides a more reliable filter before committing to expensive RL training.
The predictive power of Model Collapse Step grounded Theorem 1. captures the point where geometric amplification shifts from beneficial (reinforcing correct) to harmful (amplifying errors). In practice, this enables efficient pre- ReeRRVR assessment of trainability via short diagnostic runs with aggressive hyperparameters and calibrated early stopping, capturing gains while avoiding collapse.
DISCUSSION
Takeaways
Intrinsic rewards are fundamentally bounded by what the model already knows. External rewards grounded in unlabeled data or inperation-verifiaatto asymmetry provide signals that scale with data and computation rather than saturating with model capacity, offering a more promising path towards scalable URLVR.
Preceding sections reveal that intrinsic URLVR faces fundamental scalability limits rooted in confidence-cooreeteess alignment. When this alignment is weak, intrinsic methods amplify isting biases rather than discovering new knowledge. This limitation is not merely a hyperparameter issue but stems from the nature of the reward signal itself, which derives entirely from the model's internal state and therefore cannot consistently push the model beyond what it already knows.
This motivates exploring alternative URLVR methods that escape this ceiling. Recent works have explored external reward methods that generate verifiable signals through mechanisms independent of the model's internal state. As surveyed in Section 2, we identify two promising directions that leverage unlabeled data structures to derive rewards from the corpus and exploit generation- verification asymmetries. Both paradigms offer rewards that scale with data or computation rather than saturating with model capacity, providing complementary paths toward scalable self-evolution.
To understand how external rewards differ in practice, we examine velf-verificttion as a concrete case study, which exploits key asymmetry in data that generation is hard but verification is easy. This asymmetry allows models to provide accurate rewards by themselves without ground truth labels, which is also validated in DeepSeekMath-V (Shao et al., 2025b)
Specifically, we train Qwen3-1.77 1.7--B and 4B-Base on a subset of the Countdown task?, where models must form arithmetic expressions that reach target value. For Countdown, generating correct expressions is challenging, but checking if an expression equals the target is trivial. We sample 4k problems for training and 1k for validation. For self-verification the model generates solutions and evaluates them using a verification prompt that outputs binary correctness (see Prompt 2 in Appendix C.3). For reward score computation, whether expressions correctly evaluate to the target and compare: (1) Self- Verification (our method), (2) Trajectory-Leve Entropy (reward from Table 1), and (3) Oracle Supervision (training with ground truth reward). We track validation accuracy, Ground Truth Reward and Reward Accuracy.
the ground truth scoring function' to determine
Same method name but different implementation.
http:::s//hgginggfeee http://ni.......a...aa.aa/aaaa/a e/ count6own-py

---

Published as a conference paper at ICLR 2026
Oracle Supervision
SelfVerficaatin
TnajectoryLevel-Enttrp
Q3-1.7B-Base
Countdown (avg@ 16)
Q3-4B-Base Countdown (avg@16)
0.75
0.8
0.8
0.45
0.15
100
200
300
100
500
600
100
200
300
100
500
Training Step
Training Step
Q3-1.7B-Ban Training Dynamics
Cround fruth Reward
Q3-4B-Base
Training Dynamics
Ground Truth Reward
Reward Accuracy
0.75
tos
Reward Accuracy
0.75
0..5
0.60
-0.3
0.3
0.45
0.45
0.2
0.2
0.30
0.1
0.30
0.1
0.15
0.15
400
200
300
400
500
Training Step
Training Step
Reward Accuracy
Ground Truth Reward
Figure 7: Top: Validation accuracy comparing Self- Verification with Trajectory-Level Entropy and Oracle Supervision. Bottom: Training dynamics for Self- Verification.
Our experiments show that Self- Verification works much better than Trajectory-Level Entropy (Figure 7, top). Both models reach higher validation accuracy with Self-Verification. The bottom figure shows an interesting pattern. Reward Accuracy initially drops around step 200 as the policy tries to exploit the verifier, then recovers and stabilizes above 0.5. Meanwhile, Ground Truth Reward keeps rising. This recovery shows the model is genuinely learning to solve problems, validating that supplies stronger signals than previous intrinsic rewards while resisting reward hacking, supporting generation-verificatoo asymmetry as promising direction beyond intrinsic methods.
We find that model's s ability to follow verification instructions Instruction-aligner models demonstrate higher starting performance and significantly greater robust- ness to different verification prompts compared to base models. We provide a comprehensive analysis of prompt sensitivity and the scaling benefits of instruction alignment in Appendix C.3.
key determinant of success.
These observations point to why external rewards offer a more promising path for scaling URLVR. First, external rewards grounded in procedures that do not degrade the model improves. verifier that checks arithmetic expressions, executes code, or validates proofs against formal specifications remains equally reliable regardless of how sophisticated the model's outputs become. This contrasts with intrinsic rewards, which shift as the model's 's distribution changes and can amplify arbitrary patterns. Second, external rewards Verification procedures are often cheap to run, and the unlabeled-data paradigm shows that even tasks without native verification asymmetries can be converted into reward-based learning by leveraging structure in existing text. The quantity of available data and the affordability of verification computa- tion are both scalable resources, unbounded by human labeling capacity. Where intrinsic rewards are bounded by what the model already knows, external rewards can provide a fresh signal drawn from data and computation.
be generated at scale without human annotation.
CONCLUSION
This work explores how Unsupervised RLVR scales LLMS via unified framework for intrinsic reward methods. We show that these rewards sharpen outputs around confident predictions, enabling efficient gains when confidence aligns with correctness but amplifying errors when it does not. Empirical results reveal distinct failure modes yet also show that collapse can be avoided in small, domain-specific settings, making test-time training a natural application. Beyond these findings, early training dynamics emerge to pass @k for assessing RL trainability. Together, these results outline the limits of intrinsic rewards and highlight the need for external signals and hybrid paradigms for robust, scalable gains.
lightweight diagnostic of model-task priors, offering fast alternative
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by National Science and Technology Major Project (20232D012140) Young Elite Scientists Sponsorship Program by CAST (2023NNR001), National Natural Science Foundation of China (No. 62406165), Shanghai Municipal Science and Technology Major Project, the high-quality development project of MIIT, the National Natural Science Foundation of China (No. 62236004) and Major Project of the National Social Science Foundation of China (No. 22&2D298). We thank anonymous reviewers for their insightful comments and suggestions.
ETHICS STATEMENT
This work investigates unsupervised reinforcement learning in large language models. While our work advances understanding of AI self-improvement, we acknowledge key ethical considerations. Our findings about reward hacking highlight risks if these methods were deployed without safeguards, so that systems might become overconfident incorrect solutions, potentially causing harmful outputs in critical applications. Our identification of "safe" conditions should not be interpreted as universal guarantees, as model behavior can be unpredictable novel contexts. We emphasize that our findings aim to improve understanding of limitations and appropriate use cases rather than encourage unconstrained deployment. Practitioners should carefully assess the confidence-correccness alignment in their applications and implement monitoring systems.
REPRODUCIBILITY STATEMENT
We provide complete materials for reproducing our theoretical and empirical findings. Theoreti- cal contributions include mathematical derivations of the unified framework and formal proofs in Appendix A.3. Experimental implementations use standardized frameworks (veRL/GRPO) with hyperparameters in Table and tuning procedures in Appendix B.3. Code for all five intrinsic reward methods is provided at https: ////ttuu cOm/PRIM, SITEEERR//A RRLRRLTTT EL/TTRL. Experiments use publicly available models (Qwen series) and datasets (DAPO- 17k, AIME, AMC) are shown in the supplementary materials. All evaluation metrics are defined in Appendix B.2, and our codebase enables reproduction results in all figures and tables.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effective- ness of entropy minimization in Ilm reasoning. arXiv preprint arXiv:2505. /5134, 2025.
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, et al. Nemotron- crossthink: Scaling self-learning beyond math reasoning. arXiv preprint arXiv:2504. 13941, 2025.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms on uncontaminated math competitions. arXiv preprint wwXiiv2505.233285 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 202..
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.093390 2023.
Fredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali, and Joakim Nivre. The hyperfitting phenomenon: Sharpening and stabilizing Ilms for open-ended text generation. arXiv preprint arXiv.24/2.043/8 2024.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arxiv preprint arXiv: 2507.0626/, 2025.
11

---

Published as a conference paper at ICLR 2026
Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforcement pre-training. arXiv preprint arXiv:2506.08007, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.011 0000:20000.0.7,,, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12245 2025.
Ali Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz, Mostofa Patwary, Moham- mad Shoeybi, Bryan Catanzaro, and Yejin Choi. Rlp: Reinforcement as pretraining objective. arXiv preprint arXiv:2510.01265, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXiv:2103.03887 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner--erro An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503. 242990 2025.
Thomas Hubert, Rishi Mchta, Laurent Sartran, Miklós Z Horváth, Goran Zužić, Eric Wieser, Aja Huang. Julian Schrittwieser, Yannick Schroecker, Hussain Masoom, et al. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature, pp. 1-3, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:24/2.16720, 2024.
Andrew Kiruluta, Andreas Lemos, and Priscilla Burity. History-awar aware cross-attention reinforce- ment: Self-supervised multi turn and chain-of-thought fine-tuning with vllm. arXiv preprint arXiv:2506. / /008, 2025a.
Andrew Kiruluta, Andreas Lemos, and Priscilla Burity. A self-supervised reinforcement learning approach for fine-tuning large language models using cross-attention signals. arXiv preprint arXiv:2502.10482 2025b.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 1xXi::2411115222 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint 2025a.
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Evander Yang, Kun Li, Haoyuan Wu, Jiajia Wu, Zihao Zheng, Chenchen Zhang, et al. Reinforcement learning pre-training data. arXiv preprint 2509v:2509. 19249, 2025b.
Jia Liu, Yhang Yi He, YingQiao Lin, MingMin Yang, FeiYang Shen, ShaoGuo Liu, and TingTing Gao. Eurl: Balancing exploration and exploitation in Ilm test-time reinforcement learning via entropy mechanism. arXiv preprint arXiv:2508. //56, 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol -revieww with a 1.5 model by scaling rl. Notion Blog, 2025.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Openai o3 and o4-mini system card. Blog, 2025.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 011533288-374441 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 2020.
Sheikh Shafayat. Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint arXiv:2505.214444 2025.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint 25X6::2506..94// 2025a.
Zhihong Shao, Yuxiang Luo, Chengda Lu, ZZ Ren, Jiewen Hu, Tian Ye. Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Deepseekmath-v2: Towards self-verifiable mathematical reasoning. arXiv preprint arXiv:2511.22570 2025b.
Shuaijie She, Yu Bao, Yu Lu, Lu Xu, Tao Li, Wenhao Zhu, Shujian Huang. Shanbo Cheng, Lu Lu, and Yuxuan Wang. Dupo: Enabling reliable IIm self-verification via dual preference optimization. arXiv preprint arXiv:2508. 14460, 2025.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.0423 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
Toby Simonds and Akira Yoshiyama. Ladder: Self-improving llms through recursive problem decomposition. arXiv preprint wwXiv:2503.00733: 2025.
Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier. Risr: Reinforcement learning from self reward, 2025. URL https::///// : wtwpt/.orrrbbb///////e/ 220.200055...2..
Qwen Team, Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL btctps://wwenlng github. io /ttl////////////oooooooooooooltt 320p//////w////////
Liang Wang, Nan Yang. Shaohan Huang. Li Dong, and Furu Wei. Thinking augmented pre-training. arXiv preprint arXiv:2509.2000 201886 2025.
Haoze Wu, Cheng Wang, Wenshuo Zhao, and Junxian He. Mirage or method? how model-task alignment induces divergent rl conclusions, 2025a. URL /nttts:://axiii vetaaiivvoooo .org abs 2508
22188.
Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao Senjie Jin, Xiaoran Fan, Yuhao Zhou, Yanwei Fu, Qin Liu, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint vrXiv:2507.1052. 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lx, et al. Qwen3 technical report. arxiv preprint arXiv: 2505.09360 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
13

---

Published as a conference paper at ICLR 2026
Wenzhen Yuan, Shengji Tang, Weihao Lin, Jiacheng Ruan, Ganqu Cui, Bo Zhang, Tao Chen, Ting Liu, Yuzhuo Fu, Peng Ye, et al. Wisdom of the crowd: Reinforcement learning from coevolutionary collective feedback. arXiv preprint ar08:: 12338, 2025.
Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning, arXiv preprint arXiv:2506.0855 08745, 2025a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint 2XXv::5504008822, 2025b.
Yanzhi Zhang, Zhaoxi Zhang. Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Ilm reasoning. arXiv preprint arXiv:2506. 17219, 2025c.
Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement. arXiv preprint arXiv:2508.0044 2025d.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.0333. 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025b.
Yuxin Zuo, Kaiyan Zhang. Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self-adapting language models. arXiv preprint anxiv:2506.1090 2025.
14

---

Published as a conference paper at ICLR 2026
Method
Estimator
Formula
RLIF
Self-Certainty
r(z, El (e
EM-RL
Trajectory-Level Entropy
r(a, v)
Eli log no (yel),
EM-RI, RENT
Token-LLvvel Entropp
r(z, 157 >e1e, (nse)
RLSC
Probubility
r(a, II xa(aile, la, stee)
RLSF
Probahility Disparity
r(a. 9) Ei max.
maxigg max Talo, (at, eic))
Table 1: Overview of certainty-based rewards, estimators and their formulas. All variants reward high-confidence predictions through different formalizations model certainty.
Method
Estimator
Formala
TTRI, ETTRI. ScRL,, R-Zero
Majority Voting
arg max, Σm= 1/v, v1]] (w.) saf(l>
r(x. x)
1ly arg maxt Σ 1lv: = stll (s: ~ rat-aa)
Majority Voting across Rephrased Question
Co-Reward
+ 1[y arg max N 1wv - M"ll. (151 ~ No( (rephaso(x))
max E 1 -w1) ΣK o.
Self-eomisteney Weighted Voring
aug 2
RLCCF
(or (1x)) = 1.....N
EMPO
Semantic Similarity
<(: 5)
G(p) E Sement le (( (at ~ mo(iar)
| Con(as) [cos(Vul (oadiiulgg.))ssnt(xooee) Fam
Trajectory Concistency and Volatility
CuVo
fx mo(-le), G [[ ((::a))())) mn(()))
Table 2: Overview of ensemble- rewards, estimators and their formulas. All variants opera- tionalize the assumption that consistency across independent samples correlates with correctness.
A DETAILS FOR SECTION 3
DYNAMICS OF ONE-STEP UPDATE
Training with intrinsic rewards involves multiple gradient steps toward convergence. In this section, take TTRL (Zuo et al., 2025) and its majority voting reward representative intrinsic method to demonstrate the dynamics one-step update.
Consider the standard KL-regularized RL objective:
max (y [r(,, v)] - 3Ki. ( T9
where Tee is the reference policy and 33 controls the strength of regularization. Refer to (Rafailov al., 2023), the optimal policy for this objective has the well-known closed form:
(er(y)a) exp (frle.0)).
T (u]x:) = Z(x)
(5)
where Z(x) Ey ( exp r v) the partition function. The majority voting reward used in TTRL at iteration & is defined as
(=( y) 1 (yn)(y)) majy (Y+)),
(6)
15

---

Published as a conference paper at ICLR 2026
where Ys (y".)..,,,, denotes N rollouts sampled from Tg
(8) and majg (Yi)
arg max, Ki E [M] ans(y a}l the most frequent answer among the rollouts. Then
applying Equation (5) with the majority voting reward rk. if we held constant and performed (k), infinite updates starting from reference policy Ta we would converge to the optimal policy:
(k) (u/z) exp "
refa.y)
(y(x)
(x(
Since rx is binary, the exponential term takes only two values: e//B for the majority-voted maj; (Ys)
and e = 1 for all others. This yields the explicit form:
(k) (vl
if aanssy) = (Y
*.(++)) 8
(y|z)
(x(=)
(8)
2 (ulz) (x(m)
otherwise,
where the partition function ensures proper normalization:
(x(x)= =
Σ
T (y|z)
e/8
Tg (y/z).
(9)
y:ans(y)= (XL)
grans (y) Ammji, (Yr)
For cleaner notation, let maj
Σ yeans y mujj (Ya) "0 (y)la) denote the current policy's total prob-
ability mass on trajectories leading to the majority answer. Then the partition function simplifies to:
(x(r) Pmaj e 1/8 + (1 - Pmaj
(10)
The structure of Equation (8) reveals the method's update mechanism. At the optimal policy T each trajectory where ans(y) maja (Y&) has its probability amplified by factor et/9, The probability mass on the majority trajectories would be:
((+++)
(k)
1/B
Pmai
(l++1)
P
(k) Pmaj
1/8
mmj
P Pma)
(11)
Actual Training Dynamics, In practice, training performs only gradient update per iteration, not reaching the optimum m/ <x((k) but moving partway toward it. The actual probability mass after one update Pmaj
(k+1) satisfies:
Pany
(k(++)
(k+1) P
Pmajj
Pmagi
(12)
This ordering holds because: (1) policy gradient methods with positive rewards on majority trajectories tend to increase their probability mass (lower bound), and (2) the optimal policy achieves maximum expected reward, one-step updates cannot exceed it (upper bound). See Appendix A.2 for detailed theoretical justification and empirical validation.
A.2 JUSTIFYING THE ORDERING PMАI
-.(k+1)
(R+1) 2 PMA
2
PUAI
(R)
From the optimal policy of the standard KL-regularized RL objective (Equation (4)), if we held
(k)
reward F fixed and performed infinite updates starting from n we would reach the optimal policy with probability mass:
Pmnj
(k++1)
Pm
(R)
a a==6//2 1
(13)
1 1 (a
1)pms)
Lower bound (
Pms)): The policy gradient is: VoJ Emo [r,( Vo log Ta (yl=)). Since T,, y) for majority trajectories and rx(xr,)) y) 0 for non-majority trajectories, the gradient
16

---

Published as a conference paper at ICLR 2026
increases log Ty (y(az) only for majority trajectories. Therefore, after gradient update with learning rate ry. log Tg (k+1))() (y]z) x log ( (y/x) + 77 (z, y) [advantage terms]. Here we use because
are conducting stochastic gradient descent. We will empirically validate this increase later.
For majority trajectories with positive advantage, this increases their probability. Hence:
Pmnj
(++1)
(u|z) 2 g
(1+1)
Σ
(k) (ular) Pmaj
(14)
grans(y) ma
grans (rmn)
mia (Y)
Upper bound mamaj
(
maj )) Since * maximizes the KL-regularized objective for fixed ri-
achieves the highest possible expected reward. Our single-step update only moves partway toward
this maximum, so we cannot exceed the optimal probability mass: Pmujj (I++1)
(.+4+1)
< Pmmj
(k) This ordering establishes that Pany is monotonically increasing and bounded. The key remaining question is: does the majority answer maje (Y) remain stable across iterations?
With N rollouts from Tg each rollout independently yields answer a with probability T (k) (a|x))
(k) By the Law of Large Numbers, N increases, the empirical frequency of each answer converges to its true probability: ([ans(("))=a)) N-+00, Th (k) (a|x:). Therefore, maja (Y) (the most frequent (k) (a(xx) (the most probable answer under the policy).
NV
answer in rollouts) converges to arg max, Rg
Since Pinaj increases monotonically, the most probable answer remains majo (Y) throughout training:
(k) (a|x)) (r)) majo (Y) for all k. In practice, even moderate N (we use 8) provides reasonable stability.
arg max, Te
A.2.1 EMPIRICAL VALIDATION 1: VALIDATION OF ORDERING AND MAJORITY STABILITY
Setup: We trained on single problem from MATH-500 with N = 1024 rollouts (reducing majority vote randomness) for 50 steps. We randomly selected 4 problems and monitored whether the majority answer maje (Vi) remains stable and whether maj increases monotonically.
Step
3
8
10
level3 id14
12.70
15.53
15.92
16.21
18.46
22.07
22.56
24.80
31.35
39.36
level1 id187
6.64
6.84
7.42
10.45
11.04
11.43
11.82
15.82
18.46
levell id262
levell_id262
15.14
17.19
17.48
18.85
20.02
22.07
24.12
25.20
34.67
39.84
level3_idll id122
11.33
12.01
12.40
14.06
17.87
18.46
20.31
21.29
33.59
33.89
Table 3: Monotonic increase of Pmaj (%) early training steps.
Results for monotonic increase of Pmaj- Table 3 shows Pmaj values for the first 10 steps. All 4 problems exhibit strict monotonic increase every single step, confirming the lower bound (k+1)
Pmaj 2 Pmaj of the ordering. We also found that the majority answer remained stable across all iterations.
Step
5
10
15
20
25
30
35
40
45
50
level3 id146
18,46
39.36
48.93
91.11
95.41
98.14
98.54
99.02
99.61
99.80
levell id188
11.04
18.46
26.37
79.88
89.84
93.07
96.09
97.66
98.54
99,02
levell id262
22.07
39.84
51.37
90.14
95.90
96.80
97.46
98.05
98.63
99.41
level3_id122
17.87
33.59
43.55
84.28
92.19
93.26
95.80
96.09
98.34
98.54
Table 4: Geometric convergence of Pmoj (%) to 1.0 over 50 training steps.
Results for convergence of Pmaj- Table 4 shows the same 4 problems trained for 50 steps. All problems converge from initial values toward near-complete concentration 008.54%-99.80 at step 50), demonstrating the convergence predicted by Theorem 1. This validates non-trivial progress and confirms that the iterative training procedure with policy-dependent rewards does indeed converge to deterministic policies.
17

---

Published as a conference paper at ICLR 2026
A.2.2 EMPIRICAL VALIDATION 2: FIXED REWARD CONVERGENCE VALIDATION
Setup. To validate that the closed-form optimal policy in Equation (8) achievable when reward is held fixed, we conducted an extreme off-policy experiment. We used global batch size 1024 with mini-batch size I, generated me-time rollout (with N for each of 1024 prompts), and performed 1024 gradient updates using rewards computed solely from the initial rollout majority. This setup tests whether solving a single Kl.rregularized RL objective can converge to the theoretical optinum when the reward signal remains constant.
Results, After 1024 mini-updates using the same fixed reward signal, the Majority Voting Reward reached 1.0 (complete convergence), while validation performance AIME24, AIME25, and AMC23 dropped to zero. This confirms that the convergence point predicted by Equation (8) is achievable with sufficient updates.
A.3 PROOF OF THEOREM 1
Geometric Convergence Deterministic Policy
Consider the training procedure where each iteration k: (1) sample N rollouts Y2 from m (k) (2) compute majority majx (Y+). (3) perform one gradient update with reward ( y) (yns(y)) maj, (Yx))-
Under assumptions (A1) stable majority maję maio and (A2) 7k Mmni 0, the
(R)
probability mass Pmaj converges geometrically to As oo:
[ur(a)))) E /mnt Tnrlyr
if (aanssy) = majo (Yo);
lim To T (y)2z) k-+00
(15)
otherwise
Step 1: Effective Update Rule.
We model the actual update with step efficiency nh E (0, 1]:
Pmaj Pmaj 1T 1.(++)
(k+1)
(k)
Pmmjj
(1:)
(16)
Substituting Equation (13):
(k+1)
(k)
Ot PPmi
(k)
Pmnj
Pinjj
17k
1 + (ax 1) 1)p maj
(R)
P maj
(a - 1)(1 P Pmnaj Pmni
(k)
(K))
Pinaj
(k)
1Jk
1 (ax 1) 1)p (k)
(17)
mm)
Step Error Dynamies.
Define the error from the fixed point 1 as:
(k)
(k) P mu
(0, 1)
(18)
Substituting into Equation (17):
(k+1) = Pmnj
(k+1)
(a 1)(1 (k))e())
[1)) m7
1 (a - (a]1 E(1))
(k) (1 Mh
(a 1)(1 (A))
(19)
a (a - (N)(
Step 3: Monotonic Decrease.
18

---

Published as a conference paper at ICLR 2026
Since 1, c(0) E (0, 1). and ne (0, 11, have:
aa 1)(1 ((k))
<1
(20)
a (a 1)e(k)
Therefore:
(a- 1)(1 c(K))
0 < Ilk
<1
(21)
(a - 1)e(k)
This implies (k+1) < (8), proving the sequence {c())) is strictly decreasing and bounded below by
Step 4: Convergence to Zero.
Let = limg-soe (R) > O. Under assumption (A2), Ulk Mmin 0. If 0, then for large k; the
multiplier in Equation (19):
(a 1)(1 c(,)) 1)c(k)
7 a (a
I 7min
(22)
a
is bounded away from 1, causing continued decay. The only consistent limit is f = 0. Therefore:
(k) + 0
equivalently
Pmaj
(k)
(23)
Step 5: Geometric Convergence Rate.
From Equation (19), for large ls when (() small:
(k+1) so (b) (1 (1-
(24)
Under assumption (A2):
1 a --11 (k)
(k+1) < 11 7min
(25)
This establishes geometric convergence with rate depending on Mmin and a = 2/8 In the ideal case where 7/&- = 1 for all & (each update reaches the optimum), the convergence rate is exactly p = e 1/B
Step 6: Limiting Policy.
Given assumption (A1) that the majority remains stable at mass concentrates on trajectories with ans(y) = mYo) The limiting distribution is:
Pmn + 1, all probability
[ if ans(sy) majo (Yo),
lim To (vlar) 80000
(26)
otherwise
This completes the proof.
Remark on Assumptions.
(A1) Majority stability: By the Law of Large Numbers, with N rollouts, the empirical majority (k) (a|z) as + oo. Since pmoi increases monotonically. the argmax remains majo throughout training. We validate this empirically with = 1024 rollouts in Appendix A.2, where the majority never flipped across 200 iterations.
(Yk)(()) converges to arg max, Tg
(A2) Non-trivial progress: We assume 7) Muin 0, meaning each gradient update makes non-trivial progress. We validate this empirically: our experiments show consistent monotonic increase in Pmaj and convergence 1.0 under extreme off off-policy settings (Appendix A.2).
19

---

Published as a conference paper at ICLR 2026
A.4 UNIFIED REWARD FRAMEWORK
Despite varied implementations of intrinsic rewards, they can be understood through a single lens: the manipulation of cross-entropy between carefully chosen distributions. We consolidate these diverse rewards into the following unified paradigm:
Unified Reward Framework
Most intrinsic rewards can be expressed as:
Tuni (2., y) = vb
Σ
HIQ'' - :b)): IEI
a E (+1,-1))
(27)
where rewards derive from cross-entropy H between anchor distributions g and model distributions Tz. aggregated over granularity Z, with sign and monotonic transformation vé.
A.4.1 FRAMEWORK COMPONENTS
To understand how different intrinsic methods fit into this framework, we define each component:
Key Components:
Given question æ and generated response (a sequence of tokens y1,.... M/yl we can derive rewards from the model's 's internal distributions at different levels of granularity.
Aggregation granularity I: Determines the level to compute distributions. For token-level methods, I (1, (1,.... [y]} where each element corresponds to a position in the sequence. For answer-level methods, I {AA represents single distribution over complete semantic answers.
Model distribution oh at granularity i: For token-level granularity at position i, this is r²((-) xo(- 2, yet), the distribution over the next token given the context, For answer-level granularity, this is Tg = (- (- xx), the distribution over complete answers.
Anchor distribution oº granularity Serves as reference point. Different reward estimators use different anchors: uniform distribution Uy for Self-Certainty or one-hot distribution 8' centered the generated token for Trajectory-Level Entropy.
Cross-entropy H(y", mb): Cross-entropy between anchor distribution q' and model distribution z granularity i, defined HH('', mT) Evev. q (v) log (v).. For token-level granularity (i t), Vi is the token vocabulary, and the cross-entropy measures divergence between distribu- tions over next tokens. For answer-level granularity (i = A), V is the set of distinct semantic answers, and the cross-entropy measures divergence between distributions over complete answers.
{+1, -1: Determines the optimization direction. When the anchor is uniform, we set a = +1 to reward divergence from uniformity (encouraging peaked distributions). When the anchor y is sharp (e.g., one-hot the model's own distribution), set -1 reward alignment (reinforcing confident predictions).
Sign factor
Monotonic transformation uh: Reshapes the reward signal while preserving ordering. Common choices are identity (+²(=) 2) exponential (19(2) exp(z)), with exponential transformations amplifying the sharpening effect.
A.4.2 INSTANTIATIONS THE FRAMEWORK
We next demonstrate how most intrinsic rewards instantiate this framework. Each method's distinctive characteristics emerge from specific choices of I, q, σ,, and v. as shown in Table 5, which reveals that despite surface-level differences, all methods manipulate cross-entropy to achieve distribution sharpening.
Remarks. We highlight two special cases. First, the formulation of Self-Certainty includes an additional log V term. Since this constant is independent of model parameters, does not affect gradients during RL training. Second, the expression of rMv corresponds to the asymp- oc,
totic case where the number of rollouts co. By the law of large numbers,
20

---

Published as a conference paper at ICLR 2026
Method
Estimator
Formula
Monotonic transformation 1
Anchor distribution, y
Model distribution my
l H(U m8) + log M
RLIF
Self-Certainty
I log [V]
(Uv) ( (1r
EM-RL, RENT
Token-Level Entropy
TH
1v1
m(mb)
(15)
EM-RL
Trajectory-Level Entropy
Plng
1vl
E(8",
m
lv|
Probability
Thab exp(
Σ
816 m))
exp(12]
{6')]
EMPO
Semantic Entropy
TSE
exp( EE(( _
exp(x)
64
TTRI, SRT, ETTRI. Majority Voting
lim exp(- (1 mg'))
exp(x)
8a
Table 5: Instantiations for the unified reward framework of representative intrinsic rewards. Each method is specified by its estimator, anchor and model distributions, and monotonic transformation of the cross-entropy between them. Token-level (!) and answer-level (A) variants capture different granularities of aggregation.
the majority vote almost surely selects the answer with the highest probability under TT'. i.e. arg max, T (a). To make this limit computationally tractable, we use the tempered distribution Th (a) m (a) (a)/T)/ /S/ exp(mT which avoids the undefined log issue; as - 0+, collapses to the hard majority indicator 1 (yns(() = arg max, T6 (a)] thereby recovering the same limiting behavior as majority voting.
Key Observations:
Token-level methods (Self-Certainty, Token-Level Entropy, Trajectory-Level Entropy, Probabil- ity) operate at I = {1,..., local confidence.
......,wl]. |u]} manipulating distributions at each generation step to encourage
Answer-level methods (Semantic Entropy, Majority Voting) work at I = {A}, focusing on global answer consistency where TT represents the distribution over semantic answers and #g" denotes the empirical distribution from multiple rollouts.
Anchor choices reveal the core mechanism: uniform distributions (Uv) encourage departure from randomness, while sharp distributions (8', 84) reinforce high-probabilltyy paths, both leading to increased determinism.
Exponential transformations (exp(=)) amplify the sharpening effect by exponentially reward- ing low cross-entropy, while identity transformations (z) provide more gradual reinforcement.
A.4.3 MONOTONICITY ANALYSIS
The key insight comes from analyzing the monotonicity of the exponent in Equation (27). Since & strictly increasing by design, the behavior depends entirely on a:
Case a = +1: The reward increases with cross-entropy. Sequences where diverges from (typically uniform) receive higher rewards, pushing the policy toward more peaked distributions.
Case a = -1 The reward decreases with cross-entropy. Sequences where aligns with (typically sharp) receive higher rewards, reinforcing existing confident predictions.
Both cases lead to the same outcome: progressive sharpening the model's 's distribution, either by moving away from uniformity or by reinforcing peaked predictions.
A.5 GENERALIZED SHARPENING ANALYSIS VIA UNIFIED REWARD FRAMEWORK
To address the concern that Theorem 1 applies only to Majority Voting, and to demonstrate the analytical utility of our unified framework, provide generalized sharpening analysis. We show
21

---

Published as a conference paper at ICLR 2026
that methods with -1 share critical structural property, Reward-Confidence Monotonicity, which creates persistent pressure toward distribution sharpening.
Note: The following is proof sketch demonstrating the key convergence mechanism shared by o - - methods. A fully rigorous treatment requires additional technical conditions that we validate empirically. Methods with +1 Seeff-Certainty require separate analysis they reward away from uniform distribution.
Proposition (Sharpening Dynamics for a = -1 Methods). Consider any intrinsic reward with -1 in the unified framework w'mi =b(-H(q. xt)) where ub is strictly increasing and q is sharp anchor. These methods satisfy Reward-Confidence Monotonicity;
To (y (x) > (8((6l)) = Tani (T, ya) >> Tami yo)
(28)
For dominant trajectory y² (e.g., majority) and a non-dominant competitor y'. this inequality is strict: r(y") > r(y/). ). Under iterative Kl-rregularieed updates, this property creates self-reinforcing feedback loop that drives geometric concentration.
Proof Sketch:
We analyze the dynamics for dominant trajectory y" and competitor y/ (for ensemble methods, not in the same class as y²) where the model initially prefers y² (i.e., TT(3") )²²²== > TA (v(')) and assigns it strictly higher reward (rx(y") rx()
Step 1: Existence of a Positive Reward Gap
Using the unified formula, we justify why the gap is positive for -1
Self-Reinforcing Anehors (e.g.. Probability): r(y) = (log (v)). Since n (v²) > TT(l') and is strictly increasing, r'x (y²) rx(l').
Answer-Level Anchors (e.g., Majority Voting): belongs to the dominant answer class a", while y' does not. By construction, r(y") = and r(y/) =
In both cases, the intrinsic reward gap is strictly positive: A rw(x") rx((') > 0.
Step 2: The Optimization Target
We consider the optimal policy x² for the current fixed reward landscape Tk. The optimal solution implies target ratio:
x (v") (v"))) n (B) (x('
AA(A)
exp
(29)
B
Since A (A)) > 0, the target ratio is strictly larger than the current ratio.
Gradient Assumption: The gradient V E,,, (y) V log ro(y]] assigns positive weight to high-reward trajectories. We assume that policy gradient updates with positive learning rate n satisfy: if r(y") ) > r(y') and both have positive probability, then the updated policy satisfies
(x+i(y") 2 me(y")
(v)) This aligns with standard policy gradient convergence properties.
*x+i(v()
Step 3: The Reinforcement Loop
The unified framework reveals why this process spirals into determinism. As the policy updates to increase the probability mass the dominant trajectory:
For Relf-Reinforcing Anchors (e.g., Probability), because r(y) = (log (t)). increasing π Tz²" directly increases its reward r(y").
For Answer-Level Anchors (e.g., Majority Voting). increasing the total probability mass the dominant answer class a increases the reward for all trajectories in that class (since r oc log"(a")).
This creates a positive feedback loop: the update increases the probability of the dominant path, which maintains widens the reward gap A,, ensuring the pressure sharpen (A, 0) persists.
22

---

Published as a conference paper at ICLR 2026
Utility of the Framework: This derivation demonstrates that the "rich-get-richer" dynamic is structural inevitability for any method where the reward function is monotonically aligned with the model's confidence (o -)). The framework allows us to identify this shared property and predict that all such methods will drive the policy toward deterministic outputs, regardless of whether this leads success (when aligned with correctness) failure (when misaligned).
Remark on 0 = +1 Methods: Self-Certainty (a = +1) rewards higher when away from uniform distribution. Therefore, π (ya) T (30)) does imply r(y.) r(ws). A high-probability output and a very low-probability output could both have high KL-divergence from uniform, violating direct Reward-Conine Corard--oof Monotonicity, Its sharpening mechanism requires separate analysis.
While methods with = +1 do not strictly align reward with raw confidence, they still induce sharpening by penalizing high-entropy distributions. By maximizing the distance from uniform anchor, the optimization landscape naturally favors peaked, low-entropy policies, effectively driving the model toward determinism.
To substantiate the assumptions in this proof sketch, we provide empirical validation for different intrinsic reward methods in Figure and Appendix B.3, confirming that Reward-Confidend Monotonicity is not just a theoretical construct but the actual driver of the observed training dynamies.
Empirical Validation:
A.6 OPTIMAL POLICIES INDUCED BY OTHER INTRINSIC REWARDS
Optimal Policy of the Reward Function Tsc- For the Self-Certainty reward function rsc. it instantiates our unified framework with token-level granularity I = (1, 2..,, |y]) anchor distribution
q =Uv ||ll (uniform distribution over vocabulary), model distribution
{ { sign factor = =1,, and transformation (5x) = 2. As established previously. for any input ær the token-level predictive distribution of the model is evaluated against the current policy
+1, the farther this distribution deviates from the uniform distribution (i.e., the higher the model's confidence), the larger the reward rsc (2, y). Consequently, after single step of policy update, the optimal probability (b(2z) increases for such high high-confidence sequences, whereas it decreases when the per-token distribution close to uniform (low confidence). Thus, Nsc encourages the model to generate answers that are already preferred by the prior policy.
Due to
detailed derivation is provided below. The Self- Self-Certaitty based reward is defined as:
Ivl
M
V
1
rsc(s,y)
(U (U]
|| Te(. x.y+t)) -lg
V
Σ
log n (vt = v). (30)
1
y
V
Within the KL-regularized RL framework, dropping the constant term log VV. the one-step optimal policy becomes:
M log x = v)
To (o(()x) x < exp
(31)
3 lul IV
Therefore, whenever the model assigns concentrated probabilities to every token of (high confi- dence), the exponent grows, thus increasing the probability of the sequence Te (z)zz) In summary, the Self-Certainty based reward systematically enhances the model's "self-confidence" with respect to its prior policy.
Optimal Policy of the Reward Function Tи. For the token-level entropy-based reward TH. it instantiates our unified framework with token-level gramularity I = {1, 2.... [ }. anchor distribution -1, and transformation x(2)) 2. Maximizing ти is equivalent to minimizing the predictive entropy at every position, thereby discouraging the model from spreading its probability mass across multiple candidate tokens and hence increasing its decisiveness.
( model distribution
( sign factor
q Mi
detailed derivation is provided below. The entropy- bbase reward is defined as:
23

---

Published as a conference paper at ICLR 2026
lel H(ma(- x, (,,ye)))
191
V
(=( v)
Tb - v) log =
(32)
Within the KL-regularized RL framework, the one-step optimal policy becomes:
To(y)²)) x merlylz) exp
TU(yv - log = v)
(33)
3
Consequently, if the predictive distribution of an output sequence exhibits high entropy (i.e., the per-token distributions are close uniform), the negative-entropy reward r' strongly negative, which suppresses the exponential weight and reduces (y)zz)) Conversely, low entropy (highly peaked per-token distributions) yields ти O, thus the sequence probability enhanced after normalization. Therefore, the entropy-based reward encourages the model to generate answers whose token-level distributions are sharply concentrated, effectively boosting its "self-confidence" under the prior policy.
Optimal Policy of the Reward Function "hajj For the trajectory-level antropy-based reward Fiajj instantiates unified framework with token-level granularity I = (1, .2.,..... [y anchor distribution
Jul q = ={8"" 1t=1, model distribution = { sign factor . = -1, and transformation (b(2) = 2. For given input v, the model's predictive distribution is evaluated every token. With -1, the closer the distribution is to the one-hot reference 8 (i.e., the higher the model's confidence in each ground-truth token), the larger the reward "Traj (2r, y)- Hence, after policy-update step, the optimal probability Te (y((z) increases for such high- nigh-confidence trajectories, and decreases otherwise. Thus, PTnj encourages the model generate sequences that already enjoy high probability under the prior policy.
The trajectory-leve reward is defined as:
lyl log No ( x,y4e)
1 lvl
TTrij (Ir, y)
log no x)).
(34)
Within the KL-regularized RL framework, the one-step optimal policy becomes:
(y)ax) x Tee (v)(( exp
log To | m1)
< [To (=o(y) x)] क..
(35)
Bly
Consequently, whenever the model assigns higher prior probability to a sequence the weighted product term amplified, thereby increasing its normalized probability ne (y(xx) Therefore, the trajectory-level entropy reward boosts the probability of sequences that are already likely under the current policy my.
Optimal Policy of the Reward Function F'rob- For the grobability-base reward function Fprob. instantiates our unified framework with token-level gramularity I = 2.... w. anchor distribution {88) 1= model distribution = { f sign factor a -1, and transformation vh(2) exp([Z] 2). For given input æ. the model's predictive distribution is evaluated every token. With 1,, the closer the distribution is to the one-hot reference 8 (i.e., the higher the model's confidence in each ground-truth token), the larger the reward FProb (2., y) will be. Hence, after one policy-update step, the optimal probability (e(y)ar) increases for such high- cogh-confidence trajectories, and decreases otherwise. Thus, TProb encourages the model to generate sequences that already enjoy high probability under the prior policy.
The probability-ban -ased reward is defined as:
lyl
TProb (x, y)
no (ye ve,))))= mo (y/ x).
(36)
24

---

Published as a conference paper at ICLR 2026
Table 6: Default hyperparameters for training.
Advautage Estimator
Temperuture
Training
Batch Size
Global
Mini Batch Size
Rollout Number
Regularization
Max Prompt Length
Max Respouse
Length
Learning Rate
Epoch
GRPO
64
8
w/o KL/Eairopy
1024
7168
1e-6
Within the KL-regularized RL framework, the one-step optimal policy becomes:
To(u(æ> x )Te(()|z) exp g ( mn).
(37)
Consequently, whenever the model assigns high joint probability a sequence y, the exponential weight is amplified, thereby increasing its normalized probability (b((a)) The probability prroduct reward thus directly reinforces sequences that are already likely under the current policy, enhancing the model's preference for nigh-likelihood" trajectories.
Optimal Policy of the Reward Function TEMPO For the answer-space probability-distrie distributionn reward TEMPO employed by the EMPO algorithm, it instantiates our unified framework with answer- level granularity I {A], anchor distribution g 84, model distribution æ xg" a 1, and transformation (z) exp(z). For given input multiple roll-outs are used to estimate the current policy's distribution over the answer space. With a -1, the closer this distribution is to the one-hot reference 8 (i.e.. the more probability mass is assigned to the extracted answer), the larger the reward (EMPO(;; y) will be. Hence, after one policy-update step, the optimal probability me (v/zz increases for sequences that endorse the high-probability answer, while it decreases for all others. Maximizing TEMPO is therefore equivalent to driving the model to become more decisive at the answer level, thereby improving the consistency and determinism of the generated outputs.
Formally, the reward is defined as:
(EMPO(;; y) To (an((n(y) x)) where (a( xx)
Σ
To x).
(38)
ams(y ans(s)
Within the KL-regularised RL framework, the one-step optimal policy is:
TTp (ans(s( (y)
Te(y x)) x Trerly | ()) exp
(39)
As evidenced by Equation (39), a single EMPO update re- re-whights each sequence by a factor of exp(Tal (xp(s)(ans(y)) | B) that depends the current answer-level probability. After normalization. answers that already enjoy high probability under the prior policy gain additional mass, whereas low- probability answers suffer decrease. Consequently, the optimal policy at each step systematically shifts the overall probability mass toward the high-proe grgg-probbbilityy region of the prior policy.
B DETAILS FOR SECTION 4
B. EXPERIMENTAL SETUP
Implementation Details. All experiments are conducted using the veRL framework (Sheng et al., 2025) with the GRPO algorithm. Unless stated otherwise, we utilize the default configuration outlined Table We implement five representative intrinsic rewards by customizing the RewardManager module of VeRL, following the reward formulations in Table and Table 2:
Knsemble-Based Reward Estimators: Majority Voting
Certainty-Based Reward Estimators: Self-Certainty, Token-Level Entropy, Trajectory-Leve
Entropy, and Probability
Training Dynamics Monitoring, To monitor reward hacking and validate our theoretical predictions from Section 3, we implement specialized metrics to track the evolution of pseudo-rewards and their
25

---

Published as a conference paper at ICLR 2026
True Certain
True Uncartain
False Uncertain
False Certain
50
100
Epoch
Figure 8: Examples of per-problem training dynamics from MATH-500.
alignment with ground truth. These metrics help identify when and how these intrinsic methods transition from beneficial sharpening to pathological collapse.
Ensemble-Basse Metries: For methods using majority voting, separately track the accuracy of the chosen label and the accuracy of the rewards it generates.
Label Accuracy: Prompt-level accuracy of majority-voted answers against ground truth, measuring ensemble quality
Reward Accuracy: Sample-level agreement between pseudo rewards and oracle rewards, capturing "lucky hits" (Zuo et al., 2025) where individual rewards align despite incorrect majority votes
Ground Truth Reward: Average oracle reward (supervised baseline), computed using actual correctness
Maajoity Voting Reward: Average pseudo reward from majority voting, the divergence from Ground Truth Reward indicates reward hacking
Certainty-Based Metries: For certainty- methods, measure the correlation between this proxy reward and the actual correctness.
Label Accuracy: Ground-truth accuracy of the highest-coffidence response per prompt, testing whether maximum certainty implies correctness
These metrics collectively diagnose (1) pseudo-label quality degradation via Label Accuracy and (2) reward signal corrpption via the gap between Majority Voring Reward and Ground Truth Reward. Mathematical defimitions and implementation details are provided in Appendix B.2.
B.2 CALCULATION OF TRAINING DYNAMICS
We provide mathematical definitions for the metrics used to monitor training dynamics. These metrics diagnose reward hacking and validate theoretical predictions about distribution sharpening.
26

---

Published as a conference paper at ICLR 2026
B.2.1 NOTATION
Let D [(r,, af) denote the training dataset with M prompts, where æ; is the i-th prompt and at is its ground-truth answer. For each prompt 24, generate N rollout responses {ys, j=1 from the current policy Te, where each response Vi.j contains a trajectory and an extracted answer (ys(
Define the following:
[+]: Indicator function returning 1 if the condition is true, otherwise
maj(z;): Majority-voted answer for prompt Z2. computed arg max, Σ (lans(yr.)) a]
(V())) Ground-truth reward for response Vi.j, equals 1 (ans()).)) = 07.
mmv (: Majority-voting pseudo-reward equals 1 (yts( maj(
Teen (V Certainty-based reward (e.g., self-certainty, entropy) for response Vi.
B.2.2 ENSEMBLE-BASEE METRICS
Label Accuracy Measures the prompt-level accuracy of majority-voted answers:
Label Accuracy
M Σ 1 (maj(x.)) =a.]-
1
(40)
This metric ranges from 0 to 1, where indicates perfect pseudo-label generation.
Reward Accuracy Quantifies sample-level agreement between pseudo-rewards and oracle rewards:
M
N
Reward Accuracy
1 (rmn(( (Us.g) = Tg. (1s.))))
(41)
M
==
This captures "lucky hits" where individual rewards correct when the majority vote is wrong. For example, the majority vote incorrect but minority response is correct, that response still receives the appropriate (zero) pseudo-reward.
Ground Truth Reward
Average oracle reward across all generated responses:
Ground Truth Reward
Ta(())))
(42)
M
N
This represents the true quality generated responses and serves as the supervised baseline.
Majority Voting Reward Average pseudo-reward from majority voting:
Majority Voting Reward
Tmv (yis)-
(43)
M. N
The divergence between this metric and Ground Truth Reward indicates reward hacking: when the model learns to maximize pseudo-rewards at the expense of actual correctness.
B.2.3 CERTAINTY-BASED METRICS
Label Accuracy For certainty- baased methods, we identify the highest-confidence response per prompt and measure its aceuracy:
M Σ
Label Accuracy = M
(ans(())))) = a;},
(44)
where j = arg (a, Teen (V:.)) is the index of the highest-confidence response for prompt Tz.
27

---

Published as a conference paper at ICLR 2026
T- 0.6
-
1.1.0
--12 1.2
Label Accuracy
Ground Truth Reward
Majority Veting Reward
0.4
WAA
0.32
3.8
024
1.3
016
513
0:8
9.9
3.0
21720729901 Training Step
Training Stpp
Truining
Reward Accuracy
Training PassoN
Actor Entropy
3.00
9.9
938
30
Tanta TO
202222122
3.00
27
2E0 TRIORRE 300300
20
Trainin
Training Step
Tmmming Stop
2024
2025 (2vg@32)
2023 (avg@32)
0.33
E052
0.32
0.44
0.12
0.22
121 1211271911 220200228 247280 Training Step
1011000888111800 200501200
0.63
20110166-000111 10012014-000000 Training Stop
21 212202222
Traising Step
Figure 9: Effect of training temperature for Majority Voting method.
Mini BS =
Mini BS
Mini BS
16
Mini BS 32
Mini BS 64
Label Accuracy
Ground Truth Reward
Majority Voting Reward
0.3
3.6
0.0
6:10
033
043
0.0
0.30
0.00
10022 H6:6 220240250
00121100111 303220 263230
183 12012 388180
Reward Accuracy
Training Pass@
Actor Entropy
2.4
0.1
23
50 03
300:36 16 186366 200260266
12 40
100120 1916 18 201230 303300
20
60 50 180 93 S88
200
Training
Trasning Step
Training
Accuracy (2vg@:32)
Accuracy (avg@32)
Accuracy (avg@32)
2.04
0.45
0.2
032
002
120 100 30 203
100120140110 10000000000000000000
160 200 2202000200 Training Step
Training Step
Training Step
Figure 10: Effect of mini-batch size for Majority Voting method.
B.3 HYPERPARAMETER TUNING
Setup. We study four hyperparameters, including training temperature, mini-batch size, KL di- vergence regularization, and rollout count, that directly influence performance of intrinsic reward training. We vary one parameter time while keeping others fixed baseline values (see Table 6).
B.3.1 MAJORITY VOTING
Training Temperature. Temperature directly controls exploration during rollout generation and affects the quality of pseudo-labels via voting diversity. From our convergence analysis in Theorem 1, lower temperature reduces the effective 3 the KL regulariattion term, accelerating convergence. As shown in Figure 9, low E (0.6, 0.8} quickly sharpens logits, causing unstable Label Accuracy, consistent with premature convergence to an early majority that may be incorrect. Higher temperature (T = 1.2) maintains stability longer by preserving exploration, but the increased noise reduces peak performance. We find T 1.0 provides optimal balance, showing steady early gains with delayed degradation.
28

---

Published as a conference paper at ICLR 2026
w/o KL
w/ KL
Label Accuracy
Ground Truth Reward
Majority Veting Reward
0.40
0.54
0.32
318
0.63
016
0.88
309
1 EO 240255
Training Step
Truising
Triining Stop
Reward Accuracy
Training PassoN
Actor Entropy
0.50
0.64
64
0.56
62
10
Tanta TOTRE 2022221220
23
20 DE0 D01EO RETR 201-91000
Training Step
Truining Step
Trining Stop
2024
ATME 2025 (2vg@32)
2023 Accuracy (2vg@32)
0.37
0.36
6852
0.33
6003
0.52
12112711111 100200228 247288
100 10:10004- 5821810 200260200 Traising Step
10112118-0000000 10-00100 Training Step
Training Step
Figure 11: Effect of KL divergence regularization for Majority Voting method.
N
8
N
16
N 32
Label Accuracy
Ground Truth Reward
Majority Voting Reward
14
034
0.30
64
0.00
DOT 10
THE 201000011000
25
L60 338 100 pertars9 209210
00
10122111 200222
Step
Reward Accuracy
Training Passū)
Actor Entropy
3.6
4.5
6.15
0.3
107 123 1822022220000 2010022022
19018 20 200
10122110 200 228
Training Step
Training Slap
Training Stap
Accuracy (2vg@32)
Accuracy (avg@32)
Accuracy (2vg@32)
0.375
0.30
0.12
0200
3,000
Training Step
Trininng Stop
Trainin Step
Figure 12: Effect of rollout number for Majority Voting method.
Mini-batch Size. This parameter controls the on-policy nature of updates, directly affecting the validity of our optimal policy assumptions. Our theoretical derivation in Equation (8) assumes rewards are computed under the current policy wa. Small mini-batches violate this assumption through reward staleness: pseudo-rewards computed under become misaligned when applied to samples from Taas As shown in Figure 10, mini-batch size drives rapid collapse within 20 steps, while pure on-policy training (mini-bate batch = 64, matching global batch size) provides maximum stability. The intermediate sizes (16-32) show gradual improvement, confirming that maintaining policy-reward alignment is crucial for stable convergence.
KL Regularization. Our theoretical analysis suggests that KI. regularization should slow convergence by increasing the effective / parameter in Equation (5). However, empirical results in Figure 11 show that adding KL regularization (3 = 0.005) yields only marginal benefits: small early gains but increased training variance and minimal delay in collapse (~ 40 steps). This discrepancy arises because intrinsic rewards create competing optimization pressures, where the intrinsic signal drives sharpening while KL pulls toward the reference policy. Rather than smoothly balancing these forces, the optimization oscillates between them, increasing variance without providing durable stability. The marginal gains do not justify the additional memory overhead and training instability.
29

---

Published as a conference paper at ICLR 2026
T 0.6
T T 0.8
T 1.0
T.22
Label Accuracy
Mean Response Length
Actor Entropy
0.30
0.24
3000
0.00
Training Step
Training Stop
Training Stop
AIME 2024 Accuracy (2vg@:22)
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (2vg@:32)
0.335
0.330
1/00
6.25
0,25
23 90 60 1001200 H003E 101 266 Training Stop
40 00 100001100 100 10 200-882202222000000
60 20 10022000100000000000
Freining Stop
Training Step
Figure 13: Effect of training temperature on Self-Certaine Certainty performance, Note that Correlation is replaced with Mean Response Length due to Self-Certainty's 's scoring characteristics.
T 0.6
0.8
T
T 1.2
Label Accuracy
Mean Response Length
Actor Entropy
0.2
4500
3080
034
203528 3033
20
130
10S1RIGNT
20
24 281120022220000
AIME 2024 Accuracy (avg@)
AIME 2025 Accuracy (2)
AMC 2023 Accuracy (avg@32)
0.38
0.36
0.24
0.04
0.53
0.52
0.24
0.22
6.02
0.50
0.02
8:01
Triining Step
Training Step
Triining Step
Figure 14: Effect of training temperature on Token-Level Entropy performance.
T= 0.6
T=
T=
T= 1.2
Label Accuracy
Mean Response Length
Entropy
035
020
0:5
4502
010
3380
0.30
035
1580
105130 200 203230
130 12312310000
503180201 260200
100120 2011112010000000 240250
AIME 2024 Accuracy (2)g@32)
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (mvg@32)
0.10
0.58
0.03
034
6.14
0.3
0.50
6,02
100120:
201500200
Training Stop
Traising Stop
Training Step
Figure 15: Effect of training temperature on Trajectory-Level Entropy performance.
Number of Rollouts. The rollout count N affects both vote reliability and signal strength. While more rollouts improve statistical reliability of the majority vote, they also amplify the majority signal strength. From Equation (8), each update amplifies majority probability by factor e1/3. With more rollouts, this majority becomes more confident, accelerating convergence. Figure 12 shows this effect: N 32 collapses within 180 steps, = 16 within 220 steps, while < S remains stable over the full epoch. Although N 4 shows competitive performance in some metrics, we recommend N = 8 as it provides better statistical reliability for the voting mechanism while maintaining reasonable convergence control. The slight performance difference suggests that for this specific experimental setup, the trade-off between reliability and stability favors slightly smaller N, but = 8 offers more robust behavior across diverse problem types.
30

---

Published as a conference paper at ICLR 2026
T- 0.6
- 0.8
T-
1.2
Label Accuracy
Mean Response Length
Actor Entropy
7207
8003
0.8
6907
0.2
036
1507
Training Step
Traiting Step
Training Step
AIME 2024 Accuracy (2)
AIME 2025 Accuracy (2)g@32)
AMC 2023 Accuracy (avg@32)
0.34
6623
6623
0.22
033
63 00
100 100 10020022 262224
20 50
100 100:30380-00 00100000000000000000000000000000 200200000
20 60 00
100 100120160-00 10000000002011110110100000000001
Training Step
Training Step
Imuming Step
Figure 16: Effect of training temperature on Probability-based certainty performance.
Mini BS
Mini BS
Mini 16
Mini
32
Mini 64
Label Accuracy
Mean Response Length
Actor Entropy
0.55
0.2
0.60
300328 201101020222
120120 207201200280
10030020 Training
AIME 2024 Accuracy (avg@322
AIME 2025 Accuracy (avg832)
AMC 2023 Accuracy (avg@:32)
0.38
0.312
0.40
1/00
6.354
0.64
E.235
0340
0338
1/64
202031200
Training Stop
Training Slop
Training Step
Figure 17: Effect of mini-batch size on Self-Certainty performance.
B.3.2 CERTAINTY-BASED METHODS
Training Temperature. Temperature effects on certainty-based methods reveal distinct behavioral patterns compared to ensemble-based approaches. Unlike Majority Voting, certainty-base methods generally benefit from higher exploration temperatures, with notable method-specife variations in optimal configurations and convergence characteristics.
Results in Figures 14 to 16 demonstrate that higher temperature (T' 1.2) significantly delays model collapse across Token-Level Entropy, Trajectory- Entropy, and Probability methods. Higher temperatures initially maintain elevated Actor Entropy. facilitating extended exploration phases with gradual improvements across validation benchmarks. Importantly, these methods also exhibit relatively higher Point-Biserial Correlation values at T 1.2, indicating stronger alignment between certainty estimates and actual correctness crucial property for effective uncertainty-bast reward assignment.
However, Figure 13 reveals that Self-Certainty exhibits contrasting behavior. Higher temperature (T = 1.2) leads excessive exploration without convergence, maintaining persistently high Actor Entropy while achieving lower validation scores and Label Accuracy. The moderate temperature T 1.0 provides more stable and superior performance for Self Saff-Cetaainty. This divergence suggests that while different pertainty-basee methods converge toward similar sharp distributions, they exhibit distinct convergence rates requiring method-specific temperature tuning. Among all certainty-basee approaches, Token-Level and Trajectory-Leve Entropy methods demonstrate the greatest benefits from higher temperature exploration, likely due to their more robust entropy- based uncertainty estimation mechanisms.
Mini-Batch Size. Mini-batch size effects on certainty- based methods largely parallel those observed in Majority Voting, confirming that on-policy ratio critically affects training stability regardless of the underlying reward computation mechanism. However, method-specifie sensitivities reveal important distinctions in robustness to off-policy learning.
31

---

Published as a conference paper at ICLR 2026
Mini BS - 1
Mini BS
Mini BS 16
Mini BS 32
Mini BS 64
Label Accuracy
Mean Response Length
Actor Entropy
0.6
0.2
1387
0.88
1380
024
1380
Training Stap
Training Step
Training Stop
AIME 2024 Accuracy (2)
AIME 2025 Accuracy (2vg@32)
AMC 2023 Accuracy (avg@i32)
0.38
0.50
0.58
0:03
0.24
0.02
6.10
0.32
6.0
20 43 20
100120:46 160:66200 Training Step
240204
20 60 100:233 220-000222222000
Training Step
23
02
1.0 143188 101330 240200 Troining Step
Figure 18: Effect of mini-batch size on Token-Leve Entropy performance.
Mini BS - 1
Mini BS 8
Mini BS
16
+ Mini BS 32
Mini BS 64
Label Accuracy
Mean Response Length
Actor Entropy
0.30
0.00
1900
Training Stop
Training Slop
Trainin Step
AIME: 2024 Accuracy (2)g@:32
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (avg@32)
0.34
0.08
23 90
1001220 100-0000000 12--8) 200244222 Traiing Stop
28 01 36 83 L00 120 100-10 20000220222000000
60
31 100:221 160 10 2000000000000000000
Trasning Stop
Training Step
Figure 19: Effect of mini-batch size on Trajectory-Level Entropy performance.
Mini BS =
Mini BS 8
Mini BS 16
Mini BS 32
Mini BS 64
Label Accuracy
Mean Response Length
Actor Entropy
3.7
1000
2.4
750
500
Training Stop
Traning Stop
Truining Step
AIME 2024 Accuracy (avg@32)
ATME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (avg@32)
5.00
0,32
100130 1011001100 191 191.0600111100222220000 2022202
91 03 002 001211 10 100 2012200000000
00 98 100 221221193 1011000200020
Stop
Training Step
Training Stop
Figure 20: Effect of mini-batch size on Probability performance.
Figures 18 to 20 consistently demonstrate that larger mini-batch sizes prevent premature model collapse across Token-Level Entropy, Travectory-Level Entropy, and Probability methods. This pattern mirrors Majority Voting behavior, where pure on-policy training (mini-bate) size = 64) maintains optimal coupling between samples and their corresponding certainty- based rewards. The underlying mechanism remains consistent: certainty estimates computed from current policy states become unreliable when applied to samples generated from earlier policy iterations.
Notably, Self- Self-Certainty exhibits exceptional robustness to mini-batch size variations, as shown in Figure 17. This method demonstrates minimal sensitivity to on-policy ratio changes, suggesting that KL divergence-basel certainty computation may be inherently more stable across different temporal policy alignments. This robustness likely stems from Self- Seff-Cettnnty''s 's reliance on logit distribution comparisons rather than explicit probability estimates, making it less susceptible the temporal inconsistencies that destabilize other certainty- based approaches. Among the certainty- methods, Self-Certainty thus offers superior stability but at the cost of lower overall performance improvements.
32

---

Published as a conference paper at ICLR 2026
w/o KL
w/ KL
Label Accuracy
Mean Response Length
Actor En Entropy
0136
1280
0.00
1300
1120
1960
1.0
0.25
960
1.03
Training Step
Training Stop
Training Step
AIME 2024 Accuracy (2vg@:22
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (2)g@:32
0.300
6.315
0.355
1/00
6.302
6,289
1.20
0.255
23 40 00 D0 10012010022-00 2100200220000 260
42 30 02 100 1001221000000 10 200-882002220000000
60 20 35
100
Training Stop
Frenning Step
Training Step
Figure 21: Effect of KL divergence regularization on Self Self-Certainty performance.
w/o KL
w/ KL
Label Accuracy
Mean Response Length
Actor Entropy
0.2
1.22
0.26
5000
0.33
0.00
0.04
0.60
1520
0.00
1901300 100
co 203299 Training Step
Training Stop
Training Step
AIME 2024 Accuracy (2vg
AIME 2025 Accuracy (avg832)
AMC 2023 Accuracy (2v
0:53
6:32
0.68
0. 23 40 63 00 14 10 2002202022022
25 63
00
20 60 00
06
Training Step
Treining Step
Triinin Step
Figure 22: Effect of KI. divergence regularization on Token-Level Entropy performance.
w/o KL
w/ KL
Label Accuracy
Mean Response Length
Actor Entropy
0.0
1.25
5307
1.02
012
1383
6.75
0.88
0.24
6.25
0.30
1580
6.01
Training Stop
Traising Step
Training Stop
AIME 2024 Accuracy (avg@32) 32)
AIME 2025 Accuracy (2vg@32)
AMC 2023 Accuracy (avg@32)
0.45
0.76
0.53
0:24
0.22
0.32
0.8:
6.08
20 43 0E 01 100120:46 380-86 200 524 240204
20 41
100133 140-63110000000 201320000000000
23 03
1,0-28 180 100 2402000 Training Step
Training Step
Traising Step
Figure 23: Effect of KL divergence regularization on Trajectory-Level Entropy performance.
w/o KL
w/ KL
Label Accuracy
Moan Response Length
Actor Entropy
2.3
3.5
300
0.00
6.3
100
91 19 20125001036
Ean1al 30122020011
190 201 Training Stop
Training Stop
Training Step
AIME 2024 Accuracy (avg&32)
AIME 2025 Accuracy (avg832)
AMC 2023 Accuracy (avg@32)
0.340
1.23
0.2
0.32
23 40 00112040:2000 1800220110000000000000 221202220244220
Training Step
25 43 07 100 1000
Training Step
100-080 2012020002222000000
20 40 50 31 100-3314015100 11100000000000000000000
Training Step
Figure 24: Effect of KI. divergence regularization on Probability performance.
33

---

Published as a conference paper at ICLR 2026
N
N
N 16
N 32
Label Accuracy
Mean Response Length
Actor En Entropy
1100
14
13
000
Training Step
Training Stop
Training Step
AIME 2024 Accuracy (2vg@:32)
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (2vg@:32)
0.335
6.330
1405
0.50
6.303
03 40 00 100120 100300 266
60 30 00 103 1ed 200-888-022220000000
60
Training Stop
Fresning Stop
Training Step
Figure 25: Effect rollout number on Self-Certainty performance.
N=
Label Accuracy
Mean Response Length
Actor Entropy
0.3
1000
0.00
20 60
Co
BE 100120 100 100 100366002 360303
40 do ES 10012000110:000 20120000
20
6)
100:20300-00 LB0 20020) inumigg Step
Fremning Step
AIME Accuracy (2v
Accuracy (avg@32)
AMC 2023 Accuracy (avg@32)
6:15
0.000
0346
6:33
D:24
0.38
0.30
0316
0:51
0300
0:00
0.65
Training Stop
Training Stop
Training Stop
Figure 26: Effect of rollout number Token-Level Entropy performance.
KL Divergence Regularization. KL regularization effects on certainty-based methods mirror the limited impact observed Majority Voting, confirming that this regularization technique fails to address the fundamental instabilities inherent in training. However, subtle differences in method responses provide insights into the interaction between regularization and different uncertainty estimation approaches.
Results across all certainty- based methods (Figures to 24) show minimal impact on both training dynamics and downstream performance. KL regularization neither prevents eventual model collapse (except for Self-Certainty) nor significantly improves validation scores, consistent with our findings for Majority Voting. The underlying issue persists: regularization techniques designed for fixed reward signals cannot effectively stabilize systems where rewards themselves evolve with policy changes.
Interestingly, Token-Level and Trajectory-Leve Entropy methods exhibit slightly more pronounced benefits from KL regularization, evidenced by modest improvements in Label Accuracy curves. While these improvements remain insufficient to prevent collapse, they suggest that entropy- based certainty estimation may have marginally better compatibility with KL-based stabilization approaches. This observation aligns with the superior temperature robustness of these methods, indicating that entropy-based uncertainty measures may inherently more amenable to regularization techniques than probability-based KL-based certainty estimates.
Number Rollouts. Rollout count effects reveal consistent patterns across most certainty-based methods, with one notable exception that highlights fundamental differences in underlying reward computation mechanisms. These findings provide crucial insights into the sample size requirements for reliable uncertainty estimation.
Figures 26 to 28 demonstrate behavior parallel to Majority Voting: larger rollout counts (N 2 16) accelerate model convergence and premature collapse, evidenced by rapid degradation in validation benchmarks and Label Accuracy. This pattern suggests that the self-reinforcing dynamics observed in ensemble voting also manifest in certainty-based reward assignment, where higher sample
34

---

Published as a conference paper at ICLR 2026
N
N
N 16
N
Label Accuracy
Mean Response Length
Actor Entropy
0.6
1.00
0.2
1520
0:0
0.53
0.00
1526
0.60
Training Stup
Training Stop
Training Step
AIME 2024 Accuracy (2vg@32)
AIME 2025 Accuracy (2vg@32)
AMC 2023 Accuracy (2vg@32)
6:15
6:51
0.08
23 90 03 90 183121 143100 10:10010000000020000000000000000 200200020036616
0. 2- 62 60 00 100 1081022100111 20000000000000000000000000000000
20 60 00
10112010-000100
Traiiing Step
Fresning Stop
imuming Step
Figure 27: Effect of rollout number on Trajectory-Leve Entropy performance.
Label Accuracy
Mean Response Length
Actor Entropy
0.0
1257
0.2
0.6
034
507
0.2
0:0
10
TaTTa TOTRET 20020221220
31
Training
10110 2ET8123 201-91001
AIME 2024
AIME 2025 Accuracy (avg@32)
2023 Accuracy (2vg@32)
0.23
0.24
6602
E015
031
020
6000
0.63
Training Step
Training Stop
Training Stop
Figure 28: Effeec of rollout number on Probability performance.
sizes amplify confidence potentially incorrect assessments, leading to faster convergence toward suboptimal solutions.
However, Self-Certainty exhibits markedly different behavior, as shown in Figure 25. This method demonstrates remarkable stability across all rollout configurations, maintaining consistent perfor- mance without collapse or significant improvement. This unique characteristic stems from Self- Certainty's 's reliance on KL divergence between uniform and logit distribution. This fundamental difference in reward computation makes celf-Certainty inherently more robust to sample size varia- tions, though at the cost of limited performance improvements throughout training.
B.4 DIFFERENT METHODS, DIFFERENT FAILURES
We've ve shown that intrinsic rewards eventually collapse regardless of hyperparameter tuning. But do all methods fail the same way? We find that not all failures are equal. In this section, we reveal three distinct failure patterns, each exposing different weaknesses in how rewards reinforce confidence.
Setup. We compare five intrinsic rewards on Qwen3- 1..3--1..--Baae Base trained on DAPO-17K each with separately tuned byperparameters (Appendix B.3). For ensemble- rewards we use the Majority Voting estimator, and for certainty-basee we test Self- Celf-Certainty, Trajectory-Lewe Entropy, Token- Level Entropy, and Probability. We use the formulas in Tables and 2 and evaluate on three benchmarks, tracking Label Accuracy (whether pscudo-label matches ground truth label). Actor Entropy and Mean Response Length. The calculation is detailed in Appendix B.I.
Results. Figure 29 shows that different methods lead to different failure modes:
Self-Certainty and Majority Voting degrade most slowly, maintaining higher validation performance and higher Label Accuracy without collapsing within one epoch. Self-Ca Ceff-Certainty sharpens against uniform distribution each position (Table 1), making it less aggressive than direct probability maximization. Majority Voting operates at the answer level rather than the token level (Table 2), avoiding token-level artifacts.
Gradual degradation:
35

---

Published as a conference paper at ICLR 2026
Majority Voting
SelFCertaintyy
Token-Level Entropy
Trajectory-Leval Entropy
Probabllty
Label Accuracy
Actor Entropy
Mean Response Length
0.0
14 35 1
Tratiin Step
Stop
Training Step
ATME 2024 (2vg@:22
AIME 2025 (2)
AMC 2023 (avg@32)
030
0.00
Training Stop
Troining Stop
Training Stop
Figure 29: Five intrinsic reward methods exhibit distinct failure patterns. Self-Certainty and Majority Voting degrade gradually while maintaining label accuracy. Probability collapses toward brevity and entropy-based based methods drive entropy down through repetition rather than correctness.
Training Label Accuracy
Test Label Accuracy (ID 76)
Test: Label Accuracy (ID 131)
Trained on ID 14
Trained on TD 222
Trained on ID 249
Accuroyy
M
0.5
W WL
0.0
20
Trained on 279
Trained on ID 422
Trained on ID 498
Accuracy
Mn
201
40
810
100
201
803
1.00
40
90
100
Training Step
Traiiing Stop
Training Step
Figure 30: Training Label Accuracy (blue) on six MATH500 problems and Test Label Accuracy on two OOD problems: ID 76 (orange) and ID 131 (green).
Length collapse: Probability rewards brevity because it multiplies token probabilities, which naturally favors shorter sequences. The model becomes confident (lower Actor Entropy) but produces overly brief answers (shorter Mean Response Length), creating a distinct reward hacking pattern focused length rather than content quality. Length normalization like using geometric mean or average log-probabiliyy would likely mitigate this bias.
Repetition collapse: Both entropy- bbasep methods (Token-Level Entropy, Trajectory- En- tropy) average entropy across tokens, which is minimized not only by confident predictions but also by repeating high-probability tokens. Unlike probability multiplication (which rewards brevity), averaging across sequences encourages the model to pad sequences with repetitive text.
These patterns reveal that while all methods sharpen distributions in theory, their practical failures diverge substantially.
B.5 OUT-OF-DISTRIBUTERR CROSS-PROBLEM GENERALIZATION
To investigate the generalization of intrinsic URLVR, we design a targeted experiment focused on out-of-distribution (OOD) cross- cross-problem generalization.
Setup. We train Dwwen3-1.7B-BB on 6 problems from MATH500 using the same setting as in Section 4.2, representing a constrained one-shot RI scenario. For each problem, the highest-reward sample mostly wrong, and the model is trained on it individually for 100 epochs with batch size and 8 rollouts for baseline estimation. For evaluation, ID 131), ensuring no overlap with the training set. We track Training Label Accuracy (the correctness of the highest-reward sample) and Test Label Accuracy (the correctness of greedy decoding test problems). All results are smoothed using a moving average with a rolling window 5.
selected two unseen problems (ID 76 and
Figure 30 shows that all training problems have low Train Label Accuracy but for unseen test problems, they successfully turn from wrong to correct, with Test Label Accuracy increasing steadily from 0 to 1, This indicates that even though all training problems have wrong initial answers
Results.
36

---

Published as a conference paper at ICLR 2026
and intrinsic URLVR amplifying their failures, the sharpening still generalize OOD problems. As long as the confidence aligns well with correctness on these unseen problems, sharpening still works.
C OTHER EXPERIMENTAL DETAILS
C.I INCORRECT MAJORITY VOTES STILL IMPROVE REASONING
Following Section 5.1, are incorrect on a non- non-trivial proportion examples. We want to see at this extreme case, shouldn's the small subset also collapse? Surprisingly, we observe that even when training amplifies errors this incorrect subset, test-time training can still safely yield gains on OOD benchmarks, aligned with our cross-problent generalization findings in Appendix B.5.
consider extreme where the initial majority votes in training
Setup. We first perform offline filtering on DAPO-17k by sampling 64 responses per prompt from the base model and comput- ing maj @64. Note that during training, we actually use maj@s 8 with temperature 1.0,
Label Accuracy
Majority Voting Reward
which introduces some randomness. To en-
sure that the majority vote is incorrect in
Training Stop
Training Stop
a non-negligible portion, we deliberately filter offline with maj @4 and control for
AIME 2024 Accuracy (avg@32)
AMC 2023 Accuracy (avg@ 32)
higher majority ratios (>40%). We then trained 32 filtered samples using the same setting as DAPO-32. We monitor Label Accuracy (measure whether maj@ @8 during training match ground truth label), Majority Voting Reward and the actual per- formance on validation benchmarks.
630
Training
Training
Us
Figure 31: Training dynamics of extreme DAPO-32 setting, where almost all initial majority votes are incor- rect.
Results. As shown in Figure 31, we ob- serve non-zero accuracy at few early steps, with only 1-2 maj@ @8 matching ground truth labels. After 100 steps. it consistently shows zero Label Accuracy with a convergent trend Majority Voting Reward. Even when almost all 32 samples have incorrect initial majority votes, training still produces effective learning without catastrophic collapse. The gains observed on AIME24 and AMC23 demonstrate that small-scale training operates under fundamentally different dynamics than large-scale training. This validates that small subsets may avoid collapse through localized overfitting rather than systematic policy shift.
C.2 PILOT STUDY OF DIFFERENT MODELS
We first observe that intrinsic URLVR behaves differently across model families and training stages.
Q-w.25511.8
DusuS Stw-Seenn------utili Qwan-1.55 Actor Entrcgy
1.1-8-3.1-68
3-1--ul.-3-8-----0 SPT
Bowand Accuracy
Reward Accurucy
Actor Estropy
Accurary (ang5:32)
(20p)
Trissg 118
Trasig 145
Trunng 388
Tradnng S88
Figure 32: RL Training dynamics after different training stages in Qwen (left) and LLaMA (right) family. In Qwen, SFT enables stable training while base collapses by step 200. In LLaMA, both eventually collapse but SFT delays failure.
37

---

Published as a conference paper at ICLR 2026
(a) Varying Rollout Count
(b) Varying Mini-batch Size
400
N-32
MBS=1
Model Collapse Stap
350
Nm16
350
MB5m4
300
N-
300
MB 54
2550
Collapae
200
150
100
100
50
0LM-3-715
LS.148
O2S4.38
Q255-78
QU-177 Rosa
93-0 Base
ULM662715
13.142
02..Mbb AN
02.51.55
Q2 02:5-72
Q3
Q54m8 Basa
Figure 33: Consistency across different rollout counts (left) and mini-bated batch sizes (right). Aggressive hyperparameters accelerate collapse but preserve relative model rankings and predictive power.
Table 7: Computation cost comparison between Model Collapse Step and the gold standard (GT Gain) for assessing RL trainability across models.
Indicator
Computation Cost
Total Tokens
Requires GT
GT Gain
7k x 8 x 17k 7
6.66B (baseline)
Yes
(response rollouts X problems models)
Model Collapse Step
7k: x x 662 32
1.19B
No
(response x rollouts x total steps x batch)
(5.6x faster)
Qwen3-1 7B-Base-P1
Qwee3-1.2 1BB-Base-P2
Qwen3-1.7B-PP
Qwen3-1.7B-B
Countdown (avg@
Reward Accuracy
Self-Verify
1.0
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
100
200
300
400
300
600
100
200
300
6.00
6000
300
200
300
400
500
Training Step
Training Step
Training Step
Figure 34: Prompt sensitivity across base and instruction-aligneed models.
Setup. We compare four models: Qwen2.5-1.5B and DeepSeek-R1 -I--stil-----ssiill---- Seatill-Qwen-1.55B from the from the LLaMA family. This lets us
Qwen family; Llama-3.1-88 and separate architectural effects from training stage effects. All models trained DAPO-17K with majority voting reward. We track Reward Accuracy, Actor Entropy, and validation performance.
Results. The families show strikingly different patterns (Figure 32). In Qwen family (left), the SFT variant maintains Reward Accuracy above 0,8 throughout training while the base model drops near zero by step 200, despite starting with higher Actor Entropy. In LLaMA family (right), both variants eventually collapse but at different rates. The base model fails by step 40, while the SFT variant shows an initial performance rise before collapsing later. This architectural difference highlights that Qwen models appear fundamentally more stable, aligned with Shao et al. (2025a).
Zhang et al. (2025c) explains these dynamics through entropy minimization that early gains come from learning correct output formats, while later degradation stems from suppressing high-entropy transitional words that support reasoning. While this view explains how intrinsic rewards sharpen distributions, it cannot predict when sharpening helps versus hurts. Figure 32 reveals the missing piece that base models from both families start with higher Actor Entropy yet perform worse, with lower Reward Accuracy and faster collapse. If high entropy enables better reasoning, base models should outperform their SFT models. The opposite pattern suggests entropy is a consequence of the sharpening process, not its determinant and not enough predict RI. trainability and model prior.
38

---

Published as a conference paper at ICLR 2026
C.3 INSTRUCTION ALIGNMENT ENHANCES VERIFIER ROBUSTNESS
Prompt 1: Adapted from RLSR (Simonds et al. 2025
Verify:
"[expr)" only contains numbers from (nums).
2. Every number from [nums) is used in "(expr)" and is used only once.
3. "[expr)" is valid arithmetic expression and not an equation. "(expr)" equals [target].
If all checks pass, return \boxed (True): otherwise, return \boxed(False:)
Prompt
You strict mathematical verifier. Your task is to check whether the given expression correctly solves the arithmetic puzzle.
Do NOT attempt to find or generate new expression yourself. You must only analyze and evaluate the provided expression "[expr)".
Verification steps:
"(expr)" missing, emply, or not valid arithmetic expression (for example, contains words instead of numbers and operators), immediately output (boxed and your task is over.
2. Check that "[expr)" only uses numbers from (nums). Each number from [nums) must appear exactly once in "(expr)".
4. The expression must contain only valid arithmetic operators: +, / and parentheses.
5. Evaluate "(expr)" numerically, If the computed result equals (target) (within a tolerance
of le-6), passes this check.
If and only if all checks pass, output \boxed True). \boxed (False).
Otherwise, output
We find that the ability of instruction following is the key to the success of self-verification.
Setup. We compare Owen3-1.7B-Baa against its instruction-aligned version (Qwen3-1.7B) to how model capability affects velffverification. We test both with two verification prompts (Prompt 1 and 2) to measure prompt sensitivity. We track the validation accuracy, Reward Accuracy and Self-Verify Reward (proxy reward used during training).
Results. From Figure 34, find that instruction alignment helps with higher starting performance and robustness to prompt choice. The instruction model starts above 60% accuracy (surpassing the base model's final performance) and improves further to over 80%. More importantly, it succeeds with both prompts, while the base model only works with P2. The middle figure shows instruction models maintain stable Reward Accuracy with both prompts, while base models highly sensitive. These results suggest self seff-verfiiatiion can scale beyond test-time training (Section 5) when combined with instruction alignment, offering robust path for scaling RL.
C.4 IMPACT OF BACKBONE MODEL
We investigate how backbone models influence training stability and performance across three key dimensions: training stage, model size, and architectural generation. Our analysis employs 11 models from Qwen and Llama families (detailed configurations in Table 8), selected to provide systematic coverage of these factors. This selection is motivated by recent findings showing distinct architectural behaviors (Gandhi et al., 2025) and potential data contamination concerns (Wu et al., 2025b), making architeccc comparison essential. All models are trained on DAPO-17k using optimal hyperparameters from Section B.3 with Majority Voting as the representative intrinsic reward.
C.4.1 HORIZONTAL. ANALYSIS: TRAINING STAGE IMPACT
Training stage progression reveals distinct stability patterns between architectures. For the Qwen family (Figure 36), math-specialized and SFT models demonstrate superior stability, maintaining
39

---

Published as a conference paper at ICLR 2026
01Mo-2-1124-72
Owwnz. 5-Math-1.55
2-78
Owan3-5BB-B
Meta 3.1--8
Qwea2.5-1.5
Qwend-1.78.Buse
AIEE 2024 (Pass(ik)
AIME 2025 (Pass@-k)
AMC AMC 2023 (Pass@ik
100
80
Paospk
Figure 35:
Pass@k performance across different validation benchmarks and models, showing how trends vary as & increases from to 256.
Table 8: Model configurations for backbone experiments. Models are categorized by family, training stage, and size.
Family
Model
Abbrev.
Stage
Size
Qwen2.5-1.Sp
Q2.5-1.5B
Base
1.SB
Q2.5-Math- 1.55
Math Base
1.5B
PrepSeck-RI-Distile
D0--istil--Qwen--.... Qwen-1.5B
DS-RI-1.5B
SFT
1.5B
Qwen
Owen2.5-1..B-Imme
Q2.5-1.5B-Im
Instruct
1.5B
Qwen3-1.7B-Ball Base
Q3-1.7B
Base
1.7B
Qwen3-4B-Ban
Q3-4B
Base
4B
Meta-1 Meta-I.lama-3.1 1ama-3.1-88
1.3.1-88
Base
8B
OctoTThinker-8B-Shor Base
Octo-8B
Math Base
8B
Llama
Octu-3B
Math Base
3B
Llama-3.1-Tuile -Tuu------------S
L.3.18BB-Tlu-SFF
SFT
SB
Meta- Deta-Llama- 3.1 18-8 Instruct
L3.1-8B-Jne Inst
Instruct
8B
02.55-15B (Owen Base)
02.5-Math-1133 .5B (Qwen Math Bose)
DS-RI-1.58 (Owen SFD)
Majoctty Q2.5-1.5B-Inst (Qwen Inst)
Accuracy
Traning Step
Training Sinp
Training Step
Entropy
Tmininng Steg
Training Slep
Training Step
AIME 2024
AMC 2023 Accunaey
Traning Stop
Sraining Stop
Truinig Stop
Figure 36: Training dynamics across different training stages in Qwen family models.
Majority Voting Reward within 0.3-0.6 while base and instruct variants reach saturation (1.0) by step 180. Math specialization and strong supervised fine-tuning (DS-R1-1.5B) create robust foundations for optimization compared to raw base models non-math aligned instruct variants.
The Llama family exhibits contrasting behavior: all variants eventually succumb to reward hacking with different collapse timing, where base models fail earliest (step 40), followed by math-specialized,
40

---

Published as a conference paper at ICLR 2026
13.1-88 (Liama Base)
Octo-8B (Liama Math Base)
13.1-8B-T (Laam SFT)
L3.1 L3.1-8B-last (Liama lnst)
Label Accuracy
Ground Truth Reward
Majority Voting Reward
Touning Step
TE Triining Step
Training Siep
Reward Accuracy
Training PassgN
Actor Entropy
Training
Training Stop
Training Step
AIME: 2024 Accuracy (2)
AIME 2025 Accuracy (avg@32)
AMC 2023 Accuracy (1vg
3012
144
0301
300
5.1
608
0202
6/6
Towniig Step
Training Step
Training Step
Figure 37: Training dynamics across different training stages in Llama family models.
Q3-1.7B-Base
Q3-4B-Base
Octo-38
Octo-8B
Label Accuracy
Ground Truth Reward
Majority Voting Reward
8:00
Training Step
Traicing Step
Step
Accuracy
Training Pass
Entropy
0.5
033
96
060
Training Stop
Zraining Step
Training Stop
AIME 2024 Accuracy (avg8)32)
AIME 2025 Accuracy (avgo) 32)
AMC 2023 Accuracy (avg832)
0.39
060
0422
062
Training Stup
Training
Truining Stup
Figure 38: Effect of model size stability across both Qwen and Llama families.
SFT, then instruct versions (detailed analysis in Figure 37). This architectural difference highlights Qwen's 's fundamental ad advantage in providing genuine stability.
C.4.2 VERTICAL ANALYSIS: SCALE AND GENERATION EFFECTS
Model size analysis (Figure 38) reveals counterintuitive scaling effects: smaller models consistently outperform larger variants. Q3-1.7B maintains stability significantly longer than Q3-4B, while Octo-3B outlasts Octo-8B by about 40 steps. This suggests larger models' increased capacity amplifies sensitivity to noisy pseudo-rewards, accelerating convergence toward degenerate solutions and challenging conventional scaling assumptions.
Architectural generation comparison shows clear improvements in newer versions. Qwen3 models exhibit superior stability compared to Qwen2.5 counterparts, with D3-1.7B-Bass demonstrating more controlled Majority Voting Reward progression (comprehensive comparison in Figure 39). These improvements likely stem from better-calibrated uncertainty estimates and enhanced representation learning supporting more reliable pseudo-reward computation.
41

---

Published as a conference paper at ICLR 2026
Q2.3-1.5B Q3-1.7B-Base
Label Accuracy
Ground Truth Reward
Majority Voting Reward
800
Training Step
Traising
Truining Slup
Reward Accuracy
Training Pass@
Actor Entropy
100 28
140 1. 200 22, 360
121
189
18 1E 1/ 22
(avg@322
2025 Accuracy
AMC (avg@32)
0.770
404
5032
0328
1004
0450
020
Truining Step
Training Stup
Truining Stop
Figure 39: Comparison Qwen 2.5 and Qwen 3 generations across comprehensive training metrics. Results reveal improved stability in the newer generation, with Qwen3 models demonstrating more gradual and controlled training dynamics compared to Qwen2.5 counterparts.
MATH-8k
ORZ-56k
DeepScaler-40k
DAPO-17k
Label Accuracy
Reward Accuracy
Training Pass@N
Training Stop
Training Stop
Training Stup
Ground Truth Reward
Majority Voting Reward
Actor Entropy
12
t00
24
Step
300
420
400
20
Lo
200
Teaining Stop
30 12
100
Training
201
500
13
400
540
AIME 2024 (2)
AIME 2025 (avg@32)
AMC 2023 (avg@32)
0.0
1/0
080
080
150
Training Stop
Training Stop
Training Stop
Figure 40: Comparison of different training data sources.
C.5 IMPACT OF TRAINING DATASET
Setup. We investigate how different training dataset influence training stability and performance, focusing on math reasoning, utilizing MATH-8K (Hendrycks al., 2021), DeepScaleR-40kk (Luo et al., 2025), DAPO-17K (Yu et al., 2025) and ORZ-56k (Hu et al., 2025), all settings are trained on Qwen3-1.7B-Base with I epoch using optimal hyperparameters from Section B.3, and also evaluated on three validation benchmarks.
Results. We can see from Figure 40, much larger datasets (DeepScaler-40kk and ORZ-56k) exhibits clear reward hacking trend, while smaller datasets settings are on its steady rise stage, indicating that current intrinsic methods may see short-sighted incremental improvements the early stage, while extending it much larger training corpora, it inevitably encounter the reward hacking.
C.6 DOES INTRINSIC REWARD METHODS TRULY IMPROVE ?APARILITIES?
42

---

Published as a conference paper at ICLR 2026
Section 4 demonstrates that prevailing in- trinsic reward approaches predominantly leverage uncertainty reduction mech- anism for enhancing performance. This observation motivates critical ques- tion: do such approaches truly enhance model's capability, or do they merely improve the self-consistency of its out- puts? Take the TTRL method as an example. TTRL explicitly models the self-consistency across m model outputs through majority voting and leverages as a supervisory signal. This design seems to suggest that TTRL may simply push the model toward the performance upper bound implied by the base model under majority voting. In other words, while TTRL might steadily improve the pass@1 metric, it would be unlikely surpass the base model's 'sssi maj @mj@m performance, thereby failing to deliver substantive gains beyond consistency alignment.
Table 9: Comparisons before and after TTRL. The results show that ITRL-traines models significantly surpass the base models' majority-vot performance in accuracy.
Metric
Qwen2.5-Mathh1.55 Qwen2.5-Math-7
muy
28.09
33.23
maj04
33.68
41.20
maj@8
37 23
45.73
maj 16
38.10
47.98
maj 32
38.43
49.19
may 64
38.17
49.87
may @ 128
37.86
50.14
maj 256
37.55
50.40
m12
37.41
50.60
maj 1024
37.30
50.79
avg (w/TLLL
48.90
68.10
+11.60
+17.31
However, our experiments reveal the opposite, as shown in Table 9. Specifically, we applied TTRL to Qwen2.5-Math-I 1.5B and Qwen2.5-Math-77 AIME 2024 (30 samples) with a train batch size of 30 for 100 epochs, and directly compared the base models' maj@1022 with the pass@1 (aug@32) of the TTRL-trained models. Since the majority voting performance converges rapidly once the sample size reaches 32, maj @ 1024 can be reasonably regarded as close approximation maj@oo. Strikingly, our results show that even the pass@1 metric of the TTRL-trained models significantly exceeds the @aj@o performance of the base models. This finding demonstrates that TTRL does far more than enforce internal coliisonsit: it genuinely enhances the model's ability to generate accurate predictions. Put differently, than the base model, thereby delivering meaningful improvements in real-world performance.
the model solve broader range of problems
43


Reviews:
[{"review_id": "5VWflcWvFZ", "reviewer": "Reviewer_Kjs9", "summary": "This paper presents a broad investigation into recent RL methods for LLMs, that operate completely without external rewards and instead use the model's own internal signals. It introduces a framework, arguing that diverse intrinsic rewards (such as majority voting or self-certainty) share a common mechanism: they \"trade uncertainty for performance\" by \"sharpening output distributions\" around the model's initially confident solutions. This convergence, the paper shows, enables the model to amplify its existing knowledge, but also risks bias lock-in if the model's confidence is misaligned with correctness. Empirically, the paper confirms this trade-off, revealing that different intrinsic reward methods fail in distinct ways; for example, some collapse into overly brief responses, while others promote repetitive verbosity. The authors demonstrate that in small, domain-specific regimes, such as test-time training, intrinsic rewards can drive stable adaptation without collapse. Finally, the paper proposes a practical application for these findings: using the early training dynamics of intrinsic rewards as a fast, lightweight indicator to assess a model's suitability and \"RL trainability\" for a specific task, complementing traditional metrics like", "questions": "Has the term URLVR been used before? My search came up empty and I find it confusing to have “Verified Reward” prominently in the name, but negate it by prefixing it with “unsupervised”. Isn't Reinforcement Learning from Internal Feedback (RLIF) the established terminology at this point?\n\nRegarding the prior (almost concurrent) work by Y. Zhang et al (No Free Lunch: Rethinking Internal Feedback for LLM Reasoning): Do you see any discrepancies between your results and conclusions and theirs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Proposes to view the changes to the policy through the lens of trading uncertainty for confidence and provides a systematic, empirical analysis of the distinct failure modes of different reward types, such as length collapse for probability rewards and verbosity for entropy rewards. A practical strength of this paper is its investigation of scaling limits, which concludes that while large-scale training leads to collapse, these methods are stable and effective in small, domain-specific settings, identifying test-time training as an ideal application.", "weaknesses": "A closely related analysis was published recently (Jun 2025) by Y. Zhang et al (No Free Lunch: Rethinking Internal Feedback for LLM Reasoning). Arguably, the core insights of these two publications are very similar; with Y. Zhang et al provide a stronger grounding in theory and provide a more in depth analysis of learning dynamics for different base-models; while this paper provides more insight into the dynamics and failure-modes of the different reward signals. Overall, the authors cite this prior work but do not discuss and contextualize  the relevance, the overlap, and potential differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h2YGRSVReo", "reviewer": "Reviewer_pWtu", "summary": "This paper investigates the potential and limitations of Unsupervised Reinforcement Learning with Verifiable Rewards for scaling Large Language Models , specifically focusing on intrinsic reward derived from the model's own internal signals addressing the \"supervision bottleneck,\" where obtaining human-verified labels becomes expensive. The study presents a unified theoretical framework for intrinsic rewards stating that all such rewards share a core mechanism: they improve performance by trading uncertainty for performance by leveraging the models priprs knowledge and sharpen the model's output . The authors present empirical analysis  confirming this. The authors also show that model collapses is avoidable in small domain specific settings  such as test time training.", "questions": "how does this scale to long training on large datasets? is reward hacking inevitable? \nDo you see any issues with the limited domain? as the domain this has been tested on is quite limited to math reasoning tasks", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "the paper presents a consolidated mathematical perspective that connect different intrinsic rewards methods under a single framework which is very nice. The authors also present a solid evaluation of intrinsic rewards - they analysed different failure modes for different methods. I think the paper is very insightful and written clearly and I think it's very interesting to investigate these novelity driven rewards within RL for LLM's.", "weaknesses": "this is not necessarily a weakness but the paper really focueses on the analysis of these different intrinsic rewards and does not so much contribute any new method ontop of this. While I personally really enjoyed reading this paper I am not 100% sure this is the right venue for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MD29joNDlM", "reviewer": "Reviewer_um7o", "summary": "This paper investigates the mechanisms underlying Unsupervised Reinforcement Learning with Verifiable Rewards (URLVR), which finetunes a language model through RL over intrinsic rewards (i.e., rewards derived based on the model itself without any external supervision). It first introduces a theoretical perspective that unifies the goal of ensemble-based and certainty-based methods. Then, the theoretical insights are demonstrated empirically. In particular, the paper shows that existing intrinsic rewards trade uncertainty for performance by sharpening the language model distribution, yet this comes at the risk of distribution collapse and reward hacking. Furthermore, it characterizes how such collapse manifests in different methods and shows that in some specific regimes it can be avoided.", "questions": "--", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The topic is timely, given the increase in popularity of methods using RL with intrinsic rewards for improving the capabilities of language models.\n\n2. The empirical analysis sheds light on the difference between intrinsic rewards and their potential failure patterns.\n\n3. I find the suggestion of using intrinsic reward dynamics in the initial time steps as an indicator for the potential success of URLVR, as opposed to using measures such as pass@k that require access to a verifier or labels, to be interesting. However, it is worth noting that the evaluation of this method is somewhat lacking. There is no qualitative guidance as to how one should use the intrinsic reward dynamics and there is no comparison to pass@k in terms of predictiveness or computational efficiency.", "weaknesses": "1. The unified reward framework in Section 3.1 is currently not well-defined and its usefulness is not substantiated in the paper. Specifically, how does the right hand side of Equation (1) depend on $y$? Should the cross-entropy term $h$ be $-q^i (y | x) \\ln \\pi_\\theta^i (y | x)$? Moreover, the significance of such a unified perspective greatly depends on whether it allows characterizing similarities and differences between different instances. However, the unified framework is not really used in the paper after its definition, which raises the question of why it is necessary or helpful.\n\n2. The theoretical analysis contains potentially incorrect claims. \n    - The reward in majority voting depends on the current policy. It therefore changes during training. The closed form solution to the KL-regularized RL objective in Equation (3) holds for a fixed reward, and so it is not clear what it implies for a policy-dependent reward such as majority voting. In particular, without proof, there is no reason to believe that Equation (4) holds. Do the authors have a proof for this claim? Also, the majority reward is not formally defined: what does $maj (Y)$ stand for? How many rollouts are considered in this majority?\n    - Theorem 1 is not rigorously stated. It is therefore difficult to evaluate what it means or whether it is sound. Specifically, what does \"majority trajectories\" refer to? What are \"$k$ updates\"? Does this refer to $k$ steps of policy gradient or rather $k$ iterations of solving perfectly the KL-regularized objective and each time updating the reference policy to be the current policy?\n\n3. One of the main claims made in the paper is that the success of URLVR stems from sharpening the language model’s distribution on correct outputs, in case it already assigns such outputs a reasonable probability. However, as far as I am aware, this is already the existing conventional wisdom behind why methods such as majority voting as a reward can work (c.f. [1,2]).\n\n4. Relation to prior work is often not adequately discussed. In particular, the results of Section 5.1 are extremely similar in nature to Section 4.1 of the TTRL paper [2]. What contribution does Section 5.1 of this paper provide beyond what was already reported by the TTRL paper? Furthermore, I believe it is worth mentioning the relation to [1], which theoretically analyzes the sharpening mechanism of URLVR.\n\n[1] Zuo, Yuxin, et al. \"Ttrl: Test-time reinforcement learning.\" arXiv preprint arXiv:2504.16084 (2025).\n\n[2] Huang, Audrey, et al. \"Self-improvement in language models: The sharpening mechanism.\" ICLR 2025.\n\nReview Summary and Recommendation\n---\nOverall, I believe that the paper does not meet the requirements for publication at ICLR. The most significant limitation of the current manuscript is that it contains underspecified and potentially unsound theory. Beyond fixing this issue, it would greatly strengthen the contributions of the paper to elaborate on the unified perspective and its uses, clarify relation to prior work and conventional wisdom regarding how URLVR works, and provide quantitative evidence for the predictiveness of initial reward dynamics of the success of URLVR.\n\n\n\nAdditional (More Minor) Comments\n---\n1. The notation of Table 1 is defined only in the appendix, which makes it difficult to parse. I would recommend having the necessary notation in the main text (e.g., in the table caption).\n\n2. There are some missing implementation details, which make it difficult to gauge the significance of some empirical contributions.\n    - In Figure 4, how are the subsets chosen? Are these just chosen at random? Also, is the behavior consistent across random subsets? Intuitively, while some small subsets will not suffer from a collapse in ground truth reward, for others they should if the majority vote is incorrect in a non-negligible portion of the examples.\n    - The setup in Section 4.3.2 is unclear. For example, how many samples are used?\n    - The setup in Section 5.1 is unclear. For example, on which dataset are these experiments ran?\n\n3. The definition of reward hacking mentioned in line 355 does not seem to accord with its conventional use. If the intrinsic reward is maximized and the ground truth reward also keeps increasing, then why should this be considered hacking?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2QwGhRR55r", "reviewer": "Reviewer_t95z", "summary": "The paper unifies several intrinsic reward modeling approach under the proposed unsupervised RLVR (URLVR) framework. Using majority voting as a prototypical approach, the paper observes that the trained model converges to a deterministic policy (Theorem 1). The paper hypothesizes that the performance of the learned policy depends on the confidence of the pre-RL model. URLVR approaches improve sampling efficiency but may not necessarily add any new capability. The paper tests this hypothesis via experiments on a a few language models.", "questions": "- Would it be possible to share any insight on why Qwen3 models may have the confidence-correctness alignment shown in the paper? Is there a way to show some language models family are more aligned than others in the sense defined in the paper?  \n\n- What is RLIF mentioned in the paper? There is another paper not cited in the draft that shows up when the reviewer did a literature search : https://arxiv.org/abs/2311.12996\n\n- I did not see an LLM usage statement in the paper or appendix. Please include one as required by ICLR 2026 guidelines", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper unifies a variety of methods that use implicit reward modeling. This setup provides a framework for the community to understand the tradeoffs involved with these methods\n\n- The analytical approach used in Theorem 1 clarifies the confidence vs performance trade-off. \n\n- Experimental validation and analysis is mostly sound and supports the observation made in Theorem 1 and claim related to model collapse. I do have a question on the setup that I will note under weakness/question", "weaknesses": "- The use of Qwen model in the analysis could use more justification. The central claim in the paper is that  the success (or lack thereof) of URLVR methods depend on the \"base/pre-RL's\" model's confidence-correctness alignment. The empirical analysis is conducted on Qwen3-family of models. No justification is provided on why these models were chosen. I would be curious to understand whether there is something the reader can learn about how this choice is better than others that may be accessible to them.\n\n- The writing in the paper could be improved\n  - The paper uses math symbols (for example Table 1) without defining all of the symbols in text. While the appendix may have the info the reader needs, it would be useful if this information is available to the reader in caption/nearby text\n  - The paper covers related papers but uses short-form etc for their names. This practice can be confusing to readers that are not as familiar with the field (like this reviewer) as the authors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)