Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
Authors: Wenlong Deng
Yi Ren
Yushu Li
Boying Gong
Danica J. Sutherland
Xiaoxiao Li
Christos Thrampoulidis
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
Math Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning with verifiable rewards has significantly advanced the reasoning capabilities of large language models, yet how to explicitly steer training toward exploration or exploitation remains an open problem. We introduce Token Hidden Reward (THR), a token-level metric that quantifies each token’s influence on the likelihood of correct responses under Group Relative Policy Optimization (GRPO). We find that training dynamics are dominated by a small subset of tokens with high absolute THR values. Most interestingly, tokens with positive THR strengthen confidence in correct outputs, thus favoring exploitation, while tokens with negative THR preserve probability mass for alternative outputs, enabling exploration. This insight suggests a natural intervention: a THR-guided reweighting algorithm that modulates GRPO’s learning signals to explicitly bias training toward exploitation or exploration. We validate the efficacy of this algorithm on diverse math reasoning benchmarks. By amplifying tokens with positive THR value and weakening negative ones, our algorithm improves greedy-decoding accuracy, favoring exploitation. The reverse strategy yields consistent gains in Pass@K accuracy, favoring exploration. We further demonstrate that our algorithm integrates seamlessly with other RL objectives such as GSPO and generalizes across architectures including Llama. These findings establish THR as a principled and fine-grained mechanism for dynamically controlling exploration and
exploitation in RL-tuned LLMs, providing new tools for targeted fine-tuning in reasoning-intensive applications.

Published as a conference paper at ICLR 2026
TOKEN HIDDEN REWARD: STEERING EXPLORATION- EXPLOITATION IN GROUP RELATIVE DEEP REINFORCE- MENT LEARNING
Wenlong Dengl.2, Yi Ren', Yushu Li', Boying Gong', Danica J. Sutherland Xiaoxiao Li Christos Thrampoulidis"
'University British Columbia, Vector Institute, 3 AAmii, 4UC Berkeley
'Corresponding author
ABSTRACT
Reinforcement learning with verifiable rewards has significantly advanced the reasoning capabilities large language models, yet how to explicitly steer training toward exploration exploitation remains an open problem. We introduce Token Hidden Reward (THR), a token-level metric that quantifies each token's 's influence the likelihood of correct responses under Group Relative Policy Optimization (GRPO). We find that training dynamics are dominated by a small subset to- kens with high absolute THR values. Most interestingly, tokens with positive THR strengthen confidence in correct outputs, thus favoring exploitation, while tokens with negative THR preserve probability mass for alternative outputs, en- abling exploration. This insight suggests a natural intervention: a -Hiiggddeed reweighting algorithm that modulates GRPO's learning signals to explicitly bias training toward exploitation or exploration. We validate the efficacy of this algo- rithm on diverse math reasoning benchmarks. By amplifying tokens with positive THR value and weakening negative ones, our algorithm improves greedy-decoding accuracy, favoring exploitation. The reverse strategy yields consistent gains Pass @ K accuracy, favoring exploration. We further demonstrate that our algorithm integrates seamlessly with other RL objectives such as GSPO and generalizes across architectures including Llama. These findings establish THR as prin- cipled and Lnn-grrinne mechanism for dynamically controlling exploration and exploitation in RL-tuned L.Mss, providing new tools for targeted fine-tuning in -easoning-intensive applications.
INTRODUCTION
The integration of reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models (LLMs) (Guo et al. 2025 Jaech et al. 2024
Team al. 2023). Group Relative Policy Optimization (GRPO) (Shao et al. 2024) and its variants
(i.e., GSPO Zheng et al. (2025) have emerged as a widely adopted and empirically successful method for training LLMs complex reasoning tasks. Models like DeepSeek-RI (Guo al.
2025), DeepSeck-Math (Shao et al. 2024), Med-R1 (Lai et al. 2025). and Search-R1 (Jin et al.
2025) have leveraged GRPO to achieve state-of-the-art performance across diverse domains. Despite these successes, central and persistent challenge in RL-driven LLM training is managing the trade-off (Tang et al. 20244) Harris Slivkins (025). Exploration, sampling uncertain actions to acquire novel information, is crucial for tasks demanding creativity (Lu ett al. 2024) and enabling generalization to unseen test cases via scaling algorithms (Snell al.
inherent exploration-exploiaaaie
2024). Conversely, exploitation prioritizes optimal decision-making based on current knowledge, a preference in applications requiring high-confidence, low-variance responses, such as medical diagnosis (Wu et al. 2025). However, effectively shifting the training objective between exploration and exploitation remains an underexplored challenge.
Recent work has begun addressing this pressing challenge through various approaches. Chow et al. (2024) examine how to steer the balance between exploration and exploitation via a best-of- training objective, but their approach relies external verifier select the best candidate

---

Published as a conference paper at ICLR 2026
among generations. Contemporaneous works (Chen al. 2025 Mahdavi al., 2025 Walder & Karkhamis 2025) introduce Pass @ K-ttrining to encourage exploration, though their methods primarily reweight questions based hardness. Similarly, contemporaneous work (Cui et al. 2025) steers exploration by controlling entropy, but the analysis is limited to a token's influence on itself. In parallel, Deng et al.) (2025) examines the learning dynamics of GRPO, showing how training alters the confidence of correct responses. By downweighting penalties on tokens that reduce this confidence, their method improves greedy decoding performance better exploiting model capabilities. However, their analysis is limited to negative gradients and their role in exploitation.
Motivated by Deng et al. (2025), we examine the intrinsic contribution of each token in the generated responses to the confidence of correct responses and connect this to the exploration-exploite
ExploitaiionExxllraaa (Qwen2.5 5. 10122-5--at--1550
Exploration perforance (Mosn on all Kfror aug PasgK %)
(p-00)
$0.0
p- botter Exploration
trade-off. We introduce Token Hidden Reward (THR), a token-level metric that quan- tifies how individual tokens influence the change in the likelihood of correct responses within the GRPO frame- work. Our analysis shows that a small subset of tokens car- ries disproportionately high absolute THR values, while most have negligible impact. Even more interestingly, leveraging the sign of THR, we design a reweighting strat- egy that explicitly adjusts learning signals : (1) Positive THR tokens amplify the likelihood change of correct re- sponses, strengthening confidence and improving greedy decoding (exploitation); (2) Negative THR tokens pre- serves probability mass for alternative (than the correct) re- sponses, boosting Pass @ K performance (exploration). We specifically compare THR's 's token-level reweighting with question-level reweighting approaches such as Pass @ K- training, showing that THR provides finer-grained and more effective guidance. Finally, we establish THR's theoretical and empirical connection to entropy-based ex- ploration methods, while highlighting THR's efficiency in capturing cross-token interactions. In summary, our main contributions are threefold:
48.0
CAPO
p- botter Exploitarbon
48.5
Thep
45.00 45.25 45.50 45.75 46.00 46.25 46.50 46.75 Gready performance (Standard Aug,
Figure 1: Our THR algorithm identi- fies high-influence tokens and reweights their learning signals based on sign: when p O, positive THR tokens amplified (exploitation); when p < 0, negative THR tokens are amplified (ex- ploration). The figure demonstrates con- trol of
trade-off.
We introduce Token Hidden Reward (THR) and conduct a thorough analysis. uncovering that a small subset of tokens disproportionately influences training and that the sign of THR correlates with the exploration-ent -etrrattoon-xpliiaatiinn trade-off.
We propose a THR- guided advantage reweighting strategy that effectively directs the fine-tuning process, enabling targeted emphasis on either exploitation or exploration. Fig. [ for visualization.
Empirical evaluations on math benchmarks confirm the effectiveness of THR -gidddd reweighting, resulting in the successful realization of desired performance improvements.
RELATED WORK
Reinforcement Learning for LLM Reasoning. Recent works have explored the use of model- generated solutions a form of bootstrapping to strengthen the reasoning capabilities of large
language models (LLMs) et al.] 2024 Guo et al.) 2025 Team et al.) 2025). These methods typically generate candidate solutions using pre-trained model, then filter them based intermediate correctness signals(Setlur et al. 2024) or final answer correctness (Guo et al.) 2025 Team et al.) 2025). producing high-quality data to train new model. Building on the success of reinforcement learning from human feedback (RLHF) (Ouyang et al.) 2022). follow-up works such as GRPO (Shao et al.
2024 Guo et al. 2025) use online training to further enhance reasoning. Moreover, reinforcement learning directly incorporates the model's incorrect outputs into training, which has been found to further boost reasoning performance (Seed et al. 2025). Despite these advances, the role of model-generated outputs during training remains underexplored.
Optimizing for inference time objectives. An increasing number of finetuning methods seek to align training with inference-tine objectives. Some approaches treat inference-ting computation as a flexible post-hoe design choice (Snell et al. 2024), while others explicitly optimize best-of-n performance during training (Huang et al. 2025
The latter, however, depends an external verifier
2

---

Published as a conference paper at ICLR 2026
to select the best output, which complicates scalability. Another direction emphasizes exploitation: Deng et al. (2025) reduce penalties on tokens in incorrect responses that positively contribute to correct responses, thereby strengthening the model's ' most confident predictions. Their analysis, however, is restricted to negative gradients and does not address exploration. In parallel, several works focus exploration. @ass@K K training (Chen et al. [025 Mahdavi al.] 2025 Walder & Karkhanis 2025) encourages exploration by reweighting questions based on hardness, but operates only at the question level and overlooks token-level dynamics. Similarly, entropy-based regularization methods such as COV-KL (Cui et al. (025) promote exploration by adjusting token entropy, yet they model only a token's 's self-influence. By contrast, our work directly targets token-level contributions and cross-token interactions, showing how they govern the
balance in GRPO.
3 PRELIMINARY
Notations. W. wz, and h denote token unembedding matrix, unembedding of a token 2 E V, and hidden embedding of -eken-seueence E V 2/ is the k-th token in and z< is the first k 1 tokens in For question æ, the old policy Thw generates group of G' positive/negative responses (Lv NE(N+); {y; hE(N-)) with N+ + N = G. Lastly, TR [V] is one-hot vector for token 2.
3.1 GROUP RELATIVE POLICY OPTIMIZATION
Group relative policy optimization, introduced in DeepSeek-Math (Guo et al. 2025) and DeepSeek- R1 (Shao et al., (024), simplifies RLVR by eliminating the value function estimation required in PPO (Schulman et al. 2017). Instead of learning a separate value network, GRPO computes group- relative rewards within each training batch, reducing training complexity while maintaining stable policy updates. For query pair 2. the policy Ta samples G responses (y, ] Each y: consists of a sequence of ly:l tokens. Given rewards r; E {0, }} for each response, GRPO computes normalized advantages Ai, == fー where and a are the empirical average and standard deviation of the rewards. Specifically for binary rewards r; E {0, 1), denoting g = N+ /G the fraction of correct
(r; = 1) responses per group, GRPO's advantage scores become:
if
A
if 0.
Note that this is constant across all tokens k 1.....l.:
| in the i-th response. GRPO minimizes:
1ad min (0), A. clip (7,, 1-,,1--,,110 - (2)
E (w,
1=
k=
( eo (-12))
where 8 is a clipping hyperparameter, clip(-) is the clipping operation, and 7 (0)
=
muly: sa) is the likelihood ratio between the current policy ue and the old policy Te
Te & (..ck)))
3.2
LIKELIHOOD CHANGE OF CORRECT RESPONSE IN GRPO
A recent study (Deng et al. 2025) analyzed the learning dynamics of GRPO, examining how the likelihood correct responses y evolves during training. They proved the following theorem using the unconstrained features framework (Yang al. 2017 Mixon et al. 2022 Razin al., 2024
Theorem 3.1. For any question x. at any time 2 O of training, and any correct response y [N+], in addition its dependence on the token unembeddings, the likelihood change d In (acs)( (ut (z)) decreases as the following quantity increases:
dt
v N
Σ
O (ha.v, h 2.
K (e. h a.
(3)
Negative Token Hilden Reward
Positive Token Hidden Reward
Here, the weights + KK quantify the similarity of token-level prediction errors across responses:
o KK = <(eyta - - x,, vr e 20 T.(e) xx, vt.ek)))
0k = (Cuttz - -. x. 2.. vi Oz TT(e) (:1 x. ))
3

---

Published as a conference paper at ICLR 2026
where q" 11-11)/44 Vg/(1-1))) and recall q N NN++/G. /G.
This theorem provides the theoretical foundation of our analysis by explaining how individual tokens of both correct and incorrect responses influence training dynamies of correct response likelihood.
TOKEN HIDDEN REWARD
Using the log-likelihood change dt introduce Token Hidden Reward (THR) to isolate and quantify each token's 's specific contribution to the model's confidence in correct outputs. We then establish how THR values encode exploration- exploitation dynamics in model training.
In m (y,
x) as proxy for the GRPO objective, we now
4.1 DEFINITION OF THR
Definition 4.1. Given question and correct response y for any token yj k E [ly; in another (positive or negative) response yy, the THR quantifies that token's 's contribution to the change d In Te(o æ)) in the likelihood of the correct response. Formally, the hidden reward for the dr
k'tth token defined as:
1
THR(U kj.k) = (2r, - 1). ON (b
k=1
Note the negative sign for incorrect responses (r(y) 0) reflecting that GRPO penalizes those responses. In view of Theorem 3.1 higher THR associated with larger increase in likelihood.
Since GRPO operates on groups of responses (thus, there can be multiple correct answers), we extend THR to the group setting by marginalizing over all positive responses:
Corollary 4.2. Given question and the set of correct responses ( for any token Yj.k in response Yj (where Yj E {y.' JE[N+1 | U {yi hE(N--) -)), the token hidden reward is defined as its
contribution to the likelihood change of the group of correct responses N
In To(s)) (y x).
de
Formally, the k' -ttth token's contribution to likelihood change of the group of correct responses is:
N
THR THR(y,, K)
THR K),
y
In Corollary 4.2
the magnitude of THR quantifies the strength of each token's 's influence on the likelihood. The sign of THR indicates whether token positively or negatively contributes to the likelihood of generating the correct response.
4.2 CONNECTING THR WITH EXPLORATION AND EXPLOITATION.
Since the likelihood of correct responses reflects the model's confidence, we interpret changes in this likelihood, driven by token-level contribu- tions, as signals of exploitation or exploration. In our context, we define these as follows:
Correct Responses THi Density
Incorrect Responses Thil Density
05-
0.5
Exploration is encouraged by lower increase in the likelihood of observed correct responses since this preserves some probability mass for alternative outputs.
Exploitation is encourages by a higher increase in the likelihood of observed correct responses, since this strengthens confidence in those ob- served correct outputs.
(4)
(0)
Figure 2: Density of THR scores for Qwen2.5- Math-1.5B. For both correct responses (a) and in- correct responses (b), observe that only small subset of tokens exhibits significantly high THR values. Notably, both types of responses contain tokens with both positive and negative THR scores.
Since THR values quantify the amount by which likelihood of correct responses increases, we can modulate the trade-off between exploration and
4

---

Published as a conference paper at ICLR 2026
exploitation through reweighting THR tokens: Amplifying positive THR tokens (by increasing their advantage weights) reduces the quantity in Eq. (3). boosting correct response likelihood and favoring exploitation. Conversely, amplifying negative THR tokens increases this quantity, reducing correct response likelihood and encouraging exploration. We validate these insights through our detailed
analysis in Section aaa exhaustive experiments in Section 6
THR-GUIDED TOKEN ADVANTAGE ADJUSTMENT
In this section, we first analyze tokens' THR values and then propose a THR-based adjustment of token advantages to steer exploitation and exploration.
THR Analysis. Having defined THR, we now analyze its behavior in practice by examining the distribution of token-level THR scores in Fig. 2 where we observe:
Dominant Tokens, For both correct and incorrect responses, the majority of tokens have THR scores clustered around zero. However, a small subset of tokens exhibit significantly larger THR values, indicating that these tokens dominate the training dynamics.
Sign of THR. Both correct responses (a) and incorrect responses (b) contain tokens with both positive and negative THR scores, revealing that tokens in either response type can either strengthen or weaken confidence in correct outputs.
Then we use THR to guide the training from two complementary perspectives: magnitude, by focusing on the most influential tokens, and sign, by steering exploration versus exploitation.
Dominant Token Training. We define dominant tokens as those whose absolute THR score exceeds a threshold, i... THR > т. We detail the selection of r Section 6 To isolate the contribution of these tokens, we construct training objective that masks out all others by setting their advantage to zero. The modified token-level advantage becomes:
ATHR = [| > 7] A
We refer to this setup as THR only training. As shown in Table this strategy achieves similar performance to the original GRPO method, which utilizes all tokens. This observation supports our claim that a small set of highly influential tokens largely determines performance.
Steering Exploration and Exploitation via THR Sign. To further exploit the information captured by THR, we introduce a token-level reweighting strategy that adapts training dynamics based on the sign of each token's 's THR score. Specifically, we modulate the advantage based whether token positively or negatively contributes to the correct response. To encourage exploitation, we increase the weight of tokens with positive THR and reduce that of tokens with negative THR. Conversely, to promote exploration, we reverse this weighting. This yields token-level reweighted advantages:
TH)(P) A
[||HRR r). (1 (ign(THR,.)) p) A
When p O, this scheme boosts positive THR tokens while dampening negative THR tokens, thus reinforcing exploitation. In contrast, setting p < O reverses this behavior, shifting the training focus toward exploration. Experimental results for this reweighting approach are reported in Section 6.1 See also Fig. [ for visualization of the tradeoff
6 EXPERIMENTS & ANALYSIS
We evaluate THR's empirical effectiveness through comprehensive experiments across four dimen- sions: (1) Demonstrating exploitation (p > 0) and exploration (p < 0) capabilities as measured by greedy accuracy and Pass@1 performance, (2) comparing our fine-grained token-level control against coarser-grained question-level baselines, (3) analyzing the relationship between THR and prediction entropy, and (4) validating generalizability across GRPO variant (i.e., GSPO-token (Zheng et al. 2025) and Llama architectures,
Experimental settings. For all experiments, we follow Zeng et al. (0025) and train on the MATH dataset (levels 3-5) (Hendrycks et al.] (02T). To accelerate training, we adopt dynamic sampling (Yu ctt al. 2025), which discards samples with zero advantage and resamples until full batch is formed. Unless otherwise specified, all models and methods are fine-tuned with identical reinforcement learning hyperparameters. Specifically, we use four A100 GPUs with a prompt batch size of 256 and 8 rollouts per prompt. We use learning rate of le 6, and mini batih size of 64, resulting in 32 gradient updates per step. Training runs for 40 steps, which corresponds to more than two effective epochs given the higher throughput from dynamic sampling. We set the sampling temperature
5

---

Published as a conference paper at ICLR 2026
1.0, the clipping ratio to 0.2, and the KL loss coefficient to x 10 4. For the threshold т, follow Deng et al. (2025) Eq. (8)), defining it as the average influence of the i'-th correct response's 's tokens on the likelihoods of other correct responses. Additional details are provided in Appendix [ Evaluation setup. Since exploitation focuses on making the best decisions based on existing knowledge (Harris & Slivkins) 2025), we assess exploitation ability of fine-tuned models by measuring their greedy decoding accuracy. Here we adopt six widely used math benchmarks: three "Hard datasets" (AIME 2025, AIME 2024 (Veeraboina) 2023), AMC23) and three "Stan- dard datasets" (MATH500 (Hendrycks et al., 2021), Olympiad (He et al., (024), and Minerva Math (Lewkowyez et al. 2022 To evaluate exploration, we report the unbiased Pass K ac- curacy (Chen et al.) 2021 I) using temperature 1.0 on the challenging AIME2024, AIME2025 and AMC23 datasets, which require more exploration during attempts. The Pass K' metric is defined
as Pass K E
(Mx / //))) where M 2 K is the number of generated responses per
question I, and C' denotes the number of correct responses. For all Pass K evaluations, we use M 256 and report results for K 2
Model
Method
Hard Dutasets
Standard Datasets
Avg.
AIME25
AIMP24
AMC23
Hard Avg.
MATH500
Minerva
Olympiad
Standard Aug.
0.0
2.5
14.9
7.9
GRPO
0.0
0.0
7.5
2.5
33.8
8.8
9.9
17.5
10.0
Qwee2.5-0,588Ims
THR
0.0
15.0
10.9
THR 0.2)
0.0
0.0
20,0
6.7
34,0
9.9
8.9
17.6
12.1
THR Gr 0.2)
0.0
0.0
17.5
5.8
35.6
11.0
6.5
17.7
11.8
Base GRPO
0.0
3.3
20.0
7,8
39.6
7.7
24.9
24.1
15.9
3.3
13.3
57.5
21.7
71,8
29.0
21,1
15.0
31,8
Qwee2.5-Matb-1558
13.3
55.0
23.9
32.4
34.1
45.8
34,5
THR 0.1)
10.0
13.3
60.0
27.8
70.6
32.0
32.7
45.1
36.4
THR G+ = 0.1)
3.3
13.3
62.5
26.4
71.4
33.1
34.5
46.3
36.3
Base GRPO
13.3
6.7
42.5
20.8
64.6
15.8
26.7
35.7
28.3
13.3
10.0
62.5
28.6
82.2
46.0
42.1
$6.8
42.7
Qwen2.5-Math-77
11.1
43.1
THR (p 0.1)
23.3
16.7
62.5
33.9
82.2
36.8
42.4
53.8
44.0
THR (P 0.1)
20.0
16.7
75.0
56.3
Table 1: Exploitation Results hard and standard math datasets. Pass accuracy (%) using greedy decoding across different methods and datasets. Bold is best performance, underline is second-best.
Mathol
755
AIME 2005
03
2.5
31.5
129
8.1
5-
154
315
41.5
J9.6
56.7
16.1
21.
119
56.5
THE
0.2
8.6
12
2.5
4.
93
171
91.
5.4
9.
14.1
1944
25.0
$1.7
30.3
42.0
36.7
15.2
21.9
21.
35.1
42
403
36.3
KL.3
0.5
09
3/7
3.3
14.2
26.3
12.5
18.7
25.6
35.7
438
52.5
46.9
15.2
22.4
203
36.5
42.
3599
011
0.2
0.4
95
1.6
311
56
9.
163
3.3
633
113
18.5
274
16.4
48.3
29.6
53.3
7.5
13.5
22.0
32.5
4L0
43.
53.7
964
66.7
1.2
380
17-2
27.3
3677
$1.4
11.2
24-3
43.6
60.3
144
267
25.5
43.0
47.3
767
THR
391
51.9
THR (2
0.4
1.5
200
54
0.4
149
21.5
30.
11.9
18.2
349
31.
31.0
45.9
539
61.3
76A
17.
33.6
28.5
35.5
493
51.
66.3
13.3
AMC23
CRIO
11:4
16.7
76.
56.
57.3
64.5
T10
21.3
85.
05.6
56.
700
76
65.5
603
05.7
65.5
68.8
13.7
81.3
361
896
00
06.5
05.9
67.5
50.
443
0911
75.2
90.1
00.9
98.3
99.5
19.1
11.
1947
THR
3016
+2.7
12.
92.5
912
12.6
18.3
THR 9-3-01 41
11.11
19.6
29.2
01.5
56.0
2611
67.5
93.5
54.6
06.5
766
542
66.5
65.2
95.8
91.5
57.0
100
79.8
91.3
97.3
91.5
Arcraae
Itae
4.6
150
197
250
117
6.6
19-1
46.1
43.1
63.6
11.7
197
485
667
THl
4.9
7.4
117
183
11
161
30
78
191
617
675
725
1599
45.
91
568
035
678
733
16.7
74
11.
15.5
319
21.
63.2
767
393
356
120
250
555
72.
Table 2: Exploration Results. Pass K results for 1-......5-..---ntttett Qwen2.5-Math-11 and Qwen2.5-Math are reported on the AIME (24,25) and AMC23 datasets, along with their average.
6.1 EFFECTIVENESS OF THR IN EXPLOITATION AND EXPLORATION
We use varying-sized Qwen2.5 models Yang et al. 2024): ) OnsB-Ins, Math-1.5B. Math-7B. Impact of Dominant Tokens. Training exclusively with THR-dominant tokens (Eq, 4 results in performance comparable to original GRPO. In Table vanilla THR (p 0) matches GRPO in greedy accuracy across models. Similarly, in Table it also performs on par with GRPO with respect to Pass@A K. Thus, THR-dominant tokens play critical role in guiding the training process. Exploitation (p > 0). Setting p > 0 amplifies positive THR tokens while suppressing negative ones. As shown in Table THR(p- 0.1) increases the total average greedy accuracy over vanilla THR

---

Published as a conference paper at ICLR 2026
(p 0) by 1.9% on Owen2.5-Math-LL35 and 3.5% Qwen2.5-Math- It further outperforms
GRPO by 1.1% and 4.0% on the same models, highlighting > as the most effective configuration for exploitation. Moreover, despite prioritizing exploitation, p 0 maintains competitive Pass K results at larger K, staying close to both vanilla THR and GRPO (Table 2)
Exploration (p O). To encourage exploration, we upweight tokens with negative THR values while down-weighting positive ones, leaving more probability mass for alternative generations. As shown in Table 2 p < 0 consistently delivers strong Pass @ performance across all model sizes. For example, on Swwen.5-Math-1.. and 5.0% Pass 256, while Qwen2.5-Man --aahh-- shows steady gains of about 1% on average across all K. In addition, p < 0 maintains competitive greedy accuracy, outperforming vanilla THR and GRPO on several datasets (Table I Although weaker than p > on standard benchmarks, it excels on hard datasets such as AIME and AMC, with Qwen2.5-Math-1 Maath-1..5B 1.5B even exceeding the p > 0 configuration. This suggests that allowing greater exploration can be beneficial for hard datasets.
THR(p = -0.1) surpasses the best baseline by 2.4% at Pass@ 128
6.2
THR VS. PASS @K K TRAINING: TOKEN-LEVEI VS. QUESTION-LEVER REWEIGHTING
Pass @ K Training as Question-Level Reweighting, Chen et al. (2025): Mahdavi et al. (2025):
Walder & Karkhanis (2025) develop RLVR objectives that directly target Pass@K optimization. For GRPO, these amount to re-weightings of the advantage scores in a way that favors "rare suc- cesses" i.e.. responses associated with "hard" questions. Crucially, the reweighting is uniform across all tokens and responses for a given question. which we term question-leve reweighting. To
be concrete, As we show in Appendix D.I] that Chen et al. (2025)'s 's question-level reweighting of vanilla GRPO advantages takes the following simplified form (assuming G 2 K):
K //() (K )/
A
A
1-4
In practice, we adopt convex combination q Ar, + q) Aak of vanilla GRPO advantage and the above Pass K advantage, termed Pass @ K-mixed (Chen et al. 2025), to avoid overly suppressing easy questions and preserve valuable learning signals. Empirically, Pass @ K-mixed outperforms
GRPO on both Qwen2.5 -atn2. 1,5B (Table and Llama3.2-33 2-33-..-33B- (Table For training, we use K- 4, G throughout our experiments.
Method
Saath25-Math-L.5 Puss&
(wen2.5-Math-7 P
16
32
128
256
16
128
256
ATME 2025 GRIO
5.9
9.9
15.0
20.5
26.5
33,6
41.5
49.8
56.7
10.5
16.4
23.2
30.2
37.4
43.9
49.7
$5.6
63.3
8K aixed
9.6
33.3
50.0
56.7
16.5
23.1
43.3
56.3
66.7
THR (0)
10.1
15.3
28,9
26.8
33.9
50.0
50.0
17.9
24,9
38.9
46.7
50.7
$7.9
66.7
THKiP -O-Pank Mixed
8.3
12.9
25.6
30.2
37.9
46.5
56.7
15.3
22.5
36.0
42.2
47.9
54.6
63.3
THKP OHeyPookell xIGRPO
9.6
14.4
19.3
24.7
31.9
$1.2
63.3
17.4
24.7
38.4
46.6
50.9
57.2
63.3
AIME 2024
GRPO
11.4
17.7
24.3
30.5
36.7
43.4
S00
56.0
63.3
14.4
20.7
27.5
34.7
43.0
49.6
58.1
67.3
76.7
Puss aidieddd
10.6
16.7
23.5
30.3
37.1
44.3
51.2
57.5
63.3
14.9
20.7
26.8
33.8
41.2
49.1
59.0
67.9
76.7
THR ()
11.9
18.2
24,9
31.2
37.9
45.3
52.9
61.2
70.0
17.3
32.6
28.5
35.5
43.2
51.9
58.6
3
75.1
THR(p- 01 -Paek-Minnd
10.4
16.5
23.4
30.0
36.4
41.8
49.8
59.0
70.0
13.7
19.4
25.7
33.2
41
49.8
57.3
64.8
75.1
THR (He XXRRO
11.0
17.0
23.5
30.4
37.0
44.2
52.0
59.8
66.7
18.1
24.3
31.2
38.4
46.5
52.6
60.7
69.8
76.7
AMC23
GRPO
16.6
59.1
70.0
75.9
85.5
90.2
93.7
96.0
97.5
60.8
12.7
$1.3
86.8
89.8
92.0
94.2
95.9
97.5
Pass mised
45.2
58.1
69.4
75.4
85.2
90.8
95.2
98.5
160.0
61.3
73.5
$1.3
35.8
59.1
89.6
91.1
93.1
95.0
THR (p <00)
47.9
61.0
72.2
81.1
87.3
91.6
95.1
98.0
160.0
60.2
12.2
80.7
$5.9
89.5
92.8
95.9
98.3
108.0
THRip () +)
43.9
57.5
69.2
75.6
85.9
91.4
95.6
98.3
100.0
55.11
71.2
80.5
86.4
90.1
93.0
96.0
98.7
108.0
THRip PossPll XIGRPO
46.8
59.6
70.5
79.4
36.4
91.8
95.8
98.6
160.0
61.4
12.3
80.2
85.3
53.8
92.0
95.1
97.1
97.5
Average
GIRF
21.3
28.9
16.4
43.3
49.6
55.7
61.2
3
73.5
28.6
16.
44.0
50.6
56.4
61.8
67.5
73.9
79.3
Poos 20 K-nised
20.5
25.
35.5
42.9
49.5
56.1
62.7
68.7
73.3
28.9
36.0
43.7
49.
55.5
60.7
66.0
72.4
79.5
THR (p <0)
21.9
29.8
37.5
44.4
50.7
57.3
63.2
69.7
76.7
20.7
37.6
44.7
51.2
57.2
62.9
68.5
74.2
M0JI
THRip- GpPassi-Mised
19.7
27.4
35.2
42.2
48.6
54.5
61.1
67.9
75.6
27.3
35.5
43.8
49.6
55.9
61.7
67.1
72.7
7k.9
THRRe 0HegProkk4( XIORP
21.2
28.7
36.3
43.0
49.4
56.0
62.9
69.9
76.7
36.2
38
45.4
$1.9
$7.6
63.1
68.9
74.7
79.2
Table 3: Comparing exploration ability with Pass K. Results for Qwen2.5- 1.5B and Qwen2.5-
Math-7B are reported on the AIME 2024, AIME 2025, and AMC23 datasets, along with their average.
THR as Token-Level Modification within a Question. Contrasting to the question-level reweighting in Eq. (6). our THR algorithms in Eq. 4 and Eq. (5) operate at the token-level by reweighting the advantage with factors that are specific to tokens across responses within a question æ. As formalized
in Corollary 4.2 THR adjusts the advantage of each token based on whether it contributes positively or negatively to the likelihood. By setting < 0 in Eq. (5). THR effectively reserves probability mass for alternative responses within the same question, thereby encouraging exploration.
7

---

Published as a conference paper at ICLR 2026
Comparing THR with @ training. We compare the performance of THR with p < 0 to Pass@ K-mixed training. THR consistently outperforms Pass@ K-mixed across all Pass @ K metrics on Qwen models. With average improvement > 1.1% across most K values on both Qwen2.5-Math- 1.5B and Qwen2.5-Math-75 this highlights THR's 's stronger ability to promote exploration.
Directly combining THR with @ass K training is Suboptimal. We also investigate whether directly combining THR(p < 0) with Pass @ K-mixed yields additional benefits but found it underper- forms compared to plain THR(p < O). We hypothesize that this is because Pass @ K-mixed tends to assign excessively low weights to "easy" questions (for those, N and thus the first reweighting factor in Eq- (6) is small), thereby weakening THR's ability to explore sill-preeent and valuable token-level variations within them. To validate this hypothesis, we combine THR with "static" version of Pass @ K-mixed training where advantages become: X Pass@K + (1 - x) GRPO, for constant (question-independeee
X- Setting X 0.2 helps preserve the influence of easy questions. This modification leads to consistent improvements over THR(p < O)+Pass@ @ K-mixed and even outperforms THR (p < 0) Qwen2.5-Math-77 with Pass @ K performance increases by up to 0.7% for K = 4, 8 and shows steady gains across K = 21:7. These results suggest that while Pass@ liss@K K training and THR target different aspects of exploration, maintaining adequate weight for easy questions allows THR to complement Pass@ K training effectively,
In summary, both THR and Pass@ K training employ what Whar al. (2025) term implicit advantage design to steer exploration. However, THR provides more fine fine-grained control by operating at the token level, enabling more targeted and effective exploration management.
6.3
ON THE RELATION OF THR WITH ENTROPY
In this section, we study the relation between THR and entropy because entropy has long served as proxy for exploration in RL (Wang et al. 2018 Cui et al. 2025
Qwen2.S-Math. L.Sth Pusc
Qwen2.S-Math-7 PassOK
Method
2
4
8
16
32
64
128
256
8
16
32
64
128
256
AIME 2025
Cov-KL
9.1
14.0
19.4
25.1
37.8
44.2
50.0
30.8
37.6
43.6
THR (p < ()
6.0
10.1
15.3
20.9
26.8
33,9
41.7
50.0
60.0
11.7
17.9
24.9
32.1
38.9
44.7
50.7
57.9
66.7
AIME 2024
Cov-KL
23.8
36.6
43.1
54.6
20.4
26.7
33.9
41.5
48.7
55,1
THR (p < 0)
11.9
18.2
24.9
31,2
37.9
45.3
52.9
61.2
70.0
17.3
22.6
28.5
35.5
43.2
51.3
58.8
66.3
73.3
AMC23
Cov-KL
59.3
70.3
79.3
86.1
96.8
86.7
89.9
92.2
97.5
THR ip < 0)
47.9
61.0
72.2
81.1
87.3
91.6
95.1
98.0
100.0
60.2
72.2
80.7
85.9
89.5
92.8
95,9
98.3
100.0
Average
Cov-KL
36.0
43.0
65.2
29.5
37.1
50.5
56.3
61.5
66.2
75.8
THR (p < 0) 21.9
29.8
37.5
44.4
50.7
57.3
63.2
69.7
76.7
29.7
37.6
44.7
51.2
57.2
62.9
68.5
74.2
80.0
Table 4: Comparing exploration ability with Pass@ K. Results for Qwen2.5 Mathhhh 1.5B and Qwen2.5- Math-7B are reported on the AIME 2024, AIME 2025, and AMC23 datasets, along with their average.
Dominant tokens overlaps with high entropy to- kens. For a confident (low-entropy) token ey,, (1|ær, )<kk) has small magnitude, thus the resuiting a., in Definition 4.1 tends to be close to zero, lead- ing to low THR. We analyze the overlap between tokens with high THR scores and those with high entropy. For each sample, we select the same number of high-entropy tokens as high-THR tokens, compute their overlap rate, and plot the kernel density esti- mate (Chen 2017) of the resulting overlap scores in Fig. 3 We find consistently high overlap ratio, often around 90%. indicating a strong correlation between THR and entropy. This finding is consistent with the observation of contemporaneous work (Wang et al. 2025), demonstrating that training on only the top 20% of high-entropy tokens is sufficient to achieve performance on par with GRPO using all tokens. Relation between THR and entropy regularization. In Appendix (.2, establish, under mild assumptions, a link between reweighting and entropy regularization at the token level. In particular, reweighting token advantages with THR implicitly regulates the dynamics of token entropy, with
085
(8)
Figure 3: Overlap between high THR and high entropy tokens. For each sample. we quantify the overlap between tokens with high THR and high entropy, and plot the resulting density. The distribution shows a pronounced peak near 90%, highlighting a strong token- level association between these two metries.
8

---

Published as a conference paper at ICLR 2026
Fassgg Average Performance with (Qwen2: 5-Math-1158)
--- D3PO
COPO-TIR cer GSPC4TRR/pe)
Kei-4
K-B-22
5-54-22 2-64-226
32
24
22
120
256
Figure 4: Mean of AIME 2024, AIME 2025, and AMC23 datasets' Pass@ performance of THR GSPO using Qwen2.5-Math-In 1,5B across different K.
Passig Average
besgttorbed
T1B (111)--611
K-114
K-B-32
K-66-256
24
2500
20-
32
61.
32
65.
22
156
256
Figure 5: Mean of AIME 2024, AIME 2025, and AMC23 datasets' Pass @ performance of different
methods on 2et.aa3..----sttttttee across different K.
both strength and direction determined by the hyper-parameter Besides the conceptual similarity, we argue below that THR is more efficient alternative to entropy-based methods.
Comparison with Cov-KL. Cui et al. (2025) propose Cov- KL entropy- based regularization approach focusing on how each token affects the update of itself during training. In contrast, THR, as formalized in Definition 4.1 explicitly captures the cross-token interactions that arise throughout the learning process. As shown in Table THR(p < 0) consistently outperforms Cov-KI. in all Pass @ settings, underscoring the importance of modeling cross-token influence for guiding exploration.
6.4 GENERALIZING THR TO OTHER RL OBJECTIVES AND MODEL FAMILIES
Combining with other RL objectives. We further show that THR can be seamlessly integrated with other group relative RL objectives. For demonstation, apply THR to the token level variant of group sequence policy optimization (GSPO-token) (Zheng et al. (025),, which optimizes at the sequence level while allowing token level advantage adjustment (details Appendix Aj. Fig. shows that THR(p < 0) boosts @ass@ K performance across all K with an average improvement ~0.9% to THR(p 0) and 1.4% to GSPO. See Apx. for detailed results.
Performance on Llama. To further demonstrate the generality of THR across model families, evaluate on Llama3.2-33 I-n-ruuttt Unlike Qwen, Llama exhibits weaker mathematical knowledge, limited cognitive behaviors (Gandhi et al.) 2025). and faces reduced reasoning length during training. Despite this, as shown in Fig. 5 THR still substantially boosts exploration, achieving up to 7% Pass @ improvement compared to GRPO. Setting p O amplifies these exploration gains further. While baselines such as COV-KL and Pass K-mixed also provide exploration improvements, they consistently underperform relative to THR. Reduced response length, results on exploitation, exploration results on each dataset, and more training details are provided in Appendix C.3
CONCLUSION
We introduced THR, demonstrating that fine-grained analysis of learning dynamics can yield novel practical algorithmic insights steering exploration-exploitttio in RLVR. Our findings suggest that RL for LLMs benefits from token-level interventions that leverage the unique structure of language generation, revealing new opportunities for principled algorithmic design. Our analysis connects THR with contemporaneous approaches, from Pass @ K optimization's 's question-leve reweighting to entropy-based exploration methods, reinforcing that multiple perspectives on the same underlying
'The strength and direction controlled by the value and sign of hyper-parameter p

---

Published as a conference paper at ICLR 2026
dynamics complement and inform each other. As the field matures, combining insights from different analytical lenses (dynamics- based, nutropy-based, objective- -bsed) could yield even more sophisticated training methods. Specifically, directions itself, such as adaptíve tuning of THR's s parameter based on training progress or question difficulty and exploring similar token-level interventions in other RLVR domains from code generation to scientific reasoning.
dynamicc-first approach opens several promising
This work was partially funded by the NSERC Discovery Grant RGPIN-202 1- 03677, Alliance Grant ALLRP 581058999999 (98-22,, the Natural Science and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs program, the Canada Research Chair program, an μTP grant funded by MSIT, and the Digital Research Alliance of Canada. This work was partially funded under the Horizon Europe grant 101213369 DVPS.
Acknowledgments:
REFERENCES
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Yen-Chi Chen. tutorial on kernel density estimation and recent advances. Biostatistics Epidemi- ology, 1(1):161-187, 2017.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass @ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508. 100.: 2025.
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. arXiv preprint arXiv:2412./5285 2024.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models, arXiv preprint wwxxiv:5005226117 2025.
Wenlong Deng, Yi Ren, Muchen Li, Danica J Sutherland, Xiaoxiao Li, and Christos Thrampoulidis. On the effect of negative gradient in group relative deep reinforcement optimization. NeurIPS, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.0 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins 2rXiv:2501.12948, 2025.
Keegan Harris and Aleksandrs Slivkins. Should you use your large language model to explore or exploit? arXiv preprint 2rXiv:2502.00225, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 1400 2024.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
10

---

Published as a conference paper at ICLR 2026
Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? coverage, scaling, and optimality inference-time alignment. arXiv preprint wrXiv:2503.21881 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, al. Openai ol system card. arXiv preprint arXiv:24/2. 16720, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms reason and leverage search engines with reinforcement learning. arXiv preprint 1rXiv:2503.095/6, 2025.
Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in vision-language models, arXiv preprint arXiv:2503./3939, 2025.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung-yi Lee, and Shao-Hua Sun. Lim discussion: Enhancing the creativity of large language models via discussion framework and role-play. arXiv preprint arXiv:2405. 06373, 2024.
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Renjie Liao, and Christos Thrampoulidis. Beyond accuracy: A policy gradient reweighting approach for pass@ k maximization in 1lms. In 2nd Al for Math Workshop @ ICML 2025, 2025.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unaligmment: Likelihood displacement in direct preference optimization. arXiv preprint drX1v:2410.08877, 2024.
Yi Ren and Danica J Sutherland. Learning dynamics of Ilm finetuning. In International Conference on Learning Representations, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-vi. Advancing superb reasoning models with reinforcement learning, arXiv preprint arXiv:2504. 139000 4, 2025.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Il reasoning. arXiv preprint arXiv:2410.08146, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. meepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.0333 13 2024.
11

---

Published as a conference paper at ICLR 2026
Hao Tang, Keya Hu, Jin Zhou, Si Cheng Zhong, Wei-Long Zheng. Xujie Si, and Kevin Ellis. Code repair with Ilms gives an ieploraaioonexploiitatiinnn tradeoff. Advances in Neural Information Processing Systems, 37:11795--11799 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.1259 2025.
Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL httttt::s:/wwt
http://::/wwww.kgkll..ooo kaggle. com/
Christian Walder and Deep Karkhanis. Pass@ policy optimization: Solving harder reinforcement learning problems. arXiv preprint 2rXiv:2505.1522 2025.
Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Exploration versus exploitation in reinforce- ment learning: stochastic control approach. arXiv preprint arXiv: 18 18/.00/52, 2018.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXi::2506.0193) 2025.
Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et Medreason: Eliciting factual medical reasoning steps in Ilms via knowledge graphs, arXiv preprint arXiv:2504.00993; 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical expert model via self- eol-oimproeement. arXiv preprint arXiv:2409. 2024.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank rnn language model. arXiv preprint arXiv:1711.0338 2017.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing Hc, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar Xiv: 18892, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint 1887, 2025.
12

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
3
Preliminary
3.1
Group Relative Policy Optimization
3.2
Likelihood Change of Correct Response in GRPO
4
Token Hidden Reward
Definition of THR
4.2
Connecting THR with Exploration and Exploitation.
THR-Guided Token Advantage Adjustment
6
Experiments & Analysis
6.1
Effectiveness of THR in exploitation and exploration
6.2
THR P Training: Loven-Level vs. Question-Level Reweighting
6.3
On the Relation of THR with Entropy
6.4
Generalizing THR to other RL objectives and model families
7
Conclusion
Appendix
13
Additional Preliminary
14
Additional Experiment Details,
14
Additional Experiments
15
C.I
Ablation Study Positive and Negative-Only Training.
15
Additional Results on GSPO
16
Additional Results on I.ama
16
Additional THR Token Analysis
17
C.5
Running time of each module,
18
Detailed Proofs
18
D.I P as the question level reweighting
18
D.2 Relationship between THR and Entropy Regularizer
19
More Studies
22
Ablation study on p
22
Gradient Steps and Convergence
22
13

---

Published as a conference paper at ICLR 2026
E.3
Comparison with clip-high
23
Study on crror-correction behavior,
24
E,5
Tokens retained
25
Usage of Large Language Model
25
A
ADDITIONAL PRELIMINARY
Group Sequential Policy Optimization. Recently, Zheng al. (2025) introduce group sequence policy optimization (GSPO), a new reinforcement learning algorithm for training large language models. Following the basic principle of importance sampling. GSPO defines importance ratios based on sequence likelihood and performs sequence-leve clipping, rewarding, and optimization. The GSPO objective (0) is then defined as:
E
min (s:(0)ALa: Are (s.(() 1 -,,10))))++++)))))
(7)
(mnaa~~
(y: [-) ~ m (:1an)
where the defined the importance ratio (0) is based on sequential likelihood:
91
(0) Te yo x (inl
exp(
(a)
(8)
maa (3: (z)
y
The token-level objective variant of GSPO, namely JasPo-token (0) allows token-wise advantage customization and is defined
Mel (O), clip(s:. - +, 1)
E (R,a)~ (Br N (1a))
min
(9)
G
U/a
1=1
1=1
where
To Wi.k xc, vs.ck)
81 (0) sg(s
(10)
sg no Ur.blar yi.ck))
and sg[-] denotes only taking the numerical value but stopping the gradient, corresponding to the detach operation in PyTorch. The gradient of GSPO-token can be derived as:
GSPO demonstrates notably superior training stability, efficiency, and performance compared to GRPO and exhibits particular efficacy for the large-scale RL training of MoE models. To he specific,
B ADDITIONAL EXPERIMENT DETAILS.
Additional Details for Qwen2.5-0.5B.
For the 0.5B model, training is conducted
A6000
GPUs with a batch size of 32, a maximum rollout length of 2500 tokens, a learning rate of 5 and mini-batch size of 16-resulting in two iteration updates per training step. For the greedy decoding performance, we report the best accuracy across multiple checkpoints due to significant fluctuations during training. For all other settings, we report the performance the final checkpoint. In addition to high-THR tokens, we also include those within the top 20% highest-entropy tokens that do not overlap with high-THR (approximate 4.1 % tokens), and keep their advantage unchanged being Á, For formatting, we follow Zeng et al. (025),, adopting simple prompts since the model struggles with complex instructions. We p 0.2 and p -0.2 for exploitation and exploration respectively.
Additional Details for Qwen-Math: The Qwen-Math model Yang et al, (022)) uses its full context length of 3072 tokens for rollouts. For format, we folow Zeng et al. (2025) use Qwen Chat template and require final answer to be enclosed in a latex command \boxed[] Unless otherwise specified, set p 0.1 for exploitation and p -0.1 for exploration.
14

---

Published as a conference paper at ICLR 2026
Base Model
Method
AIME25
AIME24
AMC23
MATH500
Minerva
Olympiad
Avg.
Base
0.0
3.3
20.0
39.6
7.7
24.9
15.9
GRPO
3.3
13.3
57.5
71.8
29.0
34.1
34.8
Qwen2.5-Math-1158
Pos Only
3.3
57.5
70.6
30.1
33.8
THR (p 0.1)
3.3
13.3
62.5
71.4
33.1
34.5
36.3
Table 5: Exploitation Results. Pass@ accuracy (%) using greedy decoding across different methods and datasets. Bold indicates the best performance, while underline marks the second-best.
Additional Training Details for Llama: For the Llama3.2- 2-3B-Inntruce InstructDuueey al. (2024) model, training is carried out on 8 A100 GPUs with a batch size of 256, a maximum rollout length of 3000 tokens, a learning rate of 1 10 and a mini-batch size of 16. For greedy decoding, we report the best accuracy across multiple checkpoints due to the substantial fluctuations observed during training, while for all other settings we report results from the final checkpoint. In addition to high-THR tokens, we also include those within the top 20% highest-entrop! tokens that do not overlap with high-THR (approximate 3.5 % tokens and fix their keep their advantage unchanged being A, For formatting, we follow Zeng et al. (2025), adopting simple prompts since the model struggles with complex instructions.
C ADDITIONAL EXPERIMENTS
C.I ABLATION STUDY ON POSITIVE AND NEGATIVE-ONE TRAINING.
We further investigate the impact training with only positive or negative tokens by modifying Ar In the "Pos Only" setting. we set all values where A, < to O, thereby increasing the confidence of correct responses only. Conversely, in the "Neg Only" setting, we set all values where A,, > 0 to 0, which reduces the confidence of incorrect responses without reinforcing correct ones. As shown in Table "Pos Only" results in a 1.3% drop in average performance compared to GRPO, indicating that negative gradients also contribute to boosting confidence in correct responses.
Qwen2.5-0. 5-0..B--Instruee Instruct Paxs G
Qwen2, S-tthh. Puxs
Method
2
16
32
64
128
256
2
16
32
64
128
256
AIME
GRPO
0.2
0.4
0.6
2.5
4.8
9.2
17.1
30.0
5.9
9.9
15.0
20.5
26.5
33.6
41.5
49.8
56.7
Neg Only
0.2
0.4
0.7
1.4
2,8
5.3
9.5
16.2
26.7
4.7
8.1
12.7
17.8
23.4
30.2
38.2
46.2
56.7
(p
AIME 2024
GRPO
0.4
0.8
1.5
2.9
5.4
10.0
17.2
27.3
36.7
11.4
17.7
24.3
30.5
36.7
43.4
50.0
56.0
63.3
Neg Only
9.7
36.7
$6.7
THR (p < ()
0.4
0.8
1.5
2.9
5.4
9.4
14.9
21.5
30.0
11.9
18.2
24.9
31.2
37.9
45.3
$2.9
61.2
70.0
AMC23
GRPO
90.2
Neg Only
7.7
13.7
22.6
34.4
48 4
63.2
76.6
87.5
95.0
44.0
56.9
68.0
76.5
83.0
88.5
92.3
94.3
95.0
THR (p <
12.0
20.1
30.6
42.7
56.5
70.8
82.7
89.6
92.5
47.9
61.0
72.2
81.1
87.3
91.6
95.1
98.0
100.0
Average
GRPO
4.0
6.6
10.1
14.6
20.1
26.4
39.8
42.1
50.6
21.3
28.9
36.4
43.3
49.6
55.7
61.7
67.3
72.5
Neg Only
2.7
4.9
8.1
12.5
18.2
24.8
31.9
39.5
48.3
9.5
0
34.6
41.5
47.7
53.8
59.5
64.5
68.4
(p
7.4
28.3
35.5
21.9
57.3
Table 6: Comparing exploration ability with Pass K. Results for Qwen2.5-Math- 1.50 and Qwen2.5- Math-7B are reported on the AIME 2024, AIME 2025, and AMC23 datasets, along with their average. Bold indicates the best performance.
As also shown in T "Neg Only" underperforms most cases. For example, on AMC23 with Qwen2.5- --thh---a it achieves @256 of 56.7%, compared to 63.3% for both GRPO and vanilla THR. While "Neg Only" yields moderate improvements over the Base model on average average-indicatiinn indicating that suppressing incorrect responses provides some exploratory value-positive tokens still play critical role enhancing exploration. By selectively incorporating informative tokens, THR with p ( achieves substantially better exploration performance than "Neg Only" alone.
15

---

Published as a conference paper at ICLR 2026
Method
Qwen2.5-Math-1.55 Pass @ass@K
2
16
32
64
128
256
AIME 2025
GSPO
5.2
9.0
13.9
19.3
24.9
31.0
36.9
41.4
46.7
GSPO+THR
4.4
7.8
12.5
18.0
23.9
31.1
39.0
46.4
50.0
GSPO+THR (p -.11)
5.1
8,9
14.3
20.4
26.6
33.3
39.9
46.9
53.3
AIME 2024 GSPO
10.4
16.8
24.1
31.3
38.5
45.6
52.4
59.4
66.7
GSPO+THR
10.0
16.2
23.6
30.8
37.7
44.8
52.8
60.8
66.7
GSPO+THR -0=00.1
11.0
17.2
24.2
31.0
37.8
44.9
51.8
59.1
66.7
AMC 2023
GSPO
44.9
58.0
69.0
77.7
84.3
89.1
92.0
93.6
95.0
GSPO+THR
44.9
58.0
68.7
77.0
83.5
88.8
93.3
97.2
100.0
GSPO+THR (p= 0.1)
-01)
45.4
58.2
69.1
77.9
84.6
90.1
95.0
98.7
100.0
Average
GSPO
20.2
27.9
35.7
42.8
49.2
55.2
60.4
64.8
69.5
GSPO+THR
19.8
27.3
34.9
41.9
48.4
54.9
61.7
68.1
72.2
GSPO+THR (p 0.1)
20.5
28.1
35.9
43.1
49.7
56.1
62.2
68.2
73.3
Table 7: Performance with GSPO
response_lengthinee
horizontally
equivalent
somewhee
minimum
relative
1400
levels
had
confirms
juice
chosen
systematic
cent
nour
intersects
configurations
specifically
presens
coreers
amount
transfer
convers
conversion
alternuing
acule
1200
1000
others structure
approaches
convenience
cov
cover"s
hed
pa
raccoutting
paper
data
involving
Lat
instead
n
raising
natur
askes
dedeermine
answer
follows
quadra
ugging
expersiiee
Foncapt
minus
Figure 7: Word cloud of the top 50 tokens ranked by THR, generated from Qwen2.5-Math, Font size is proportional to each token's average THR. Tokens with high THR represent the key rea- soning steps most critical in the model's problem- solving process.
on AMC23.
Figure 6: Response length dynamics of Llama3.2-30 Jlama3.2-3B-Instruct across different stages of GRPO training.
C.2 ADDITIONAL RESULTS ON GSPO
We further show that THR can be seamlessly integrated with other group relative reinforcement learning objectives. In particular, we apply THR to token level variant of group sequence policy optimization (GSPO-token) Zheng al. (2025). which optimizes the sequence level through clipping, rewarding, and optimization while allow token level advantage adjustment (more details in Appendix Appendix [ As reported Table incorporating THR with p < 0 yields substantial improvements, boosting Pass @K performance across all K with average improvement by around 0.9% to THR and 1.4% to GSPO.
C.3 ADDITIONAL RESULTS ON LLAMA.
Reduced response length. As shown Fig. the response length Llama3.2 1--33.2--BB declines rapidly after a few epochs, with the average length dropping from about 1.5K tokens to roughly 650. This reduction may stem from the model's limited cognitive behaviors Gandhi et al. (2025). Exploitation Results Llama We report the greedy decoding performance of Llama in Table As shown in table, while GRPO achieves the best performance, setting > can improve the greedy decoding performance compared with vanilla THR by 1.1%.
16

---

Published as a conference paper at ICLR 2026
Base Model
Method
AIME25
AIME24
AMC23
MATH500
Minerva
Olympiad
Avg.
Base
0.0
3,3
22.5
40.2
16.3
11.9
15.7
GRPO
0.0
26.7
30.0
54,4
22.1
18.1
25.2
THR
0.0
13.3
32.5
51.8
22.1
19.9
23.3
Llama3.2-3-Instte
THR (p = 0.22
3.3
6.7
27.5
51.4
20.6
16.3
21.0
THR 0.05)
3.3
13.3
40.0
50.6
22.4
16.7
24.4
Table 8: Exploitation Results. Pass@ accuracy (%) using greedy decoding across different methods and datasets. Bold indicates the best performance, while underline marks the second-best.
Method
Baamaa.2-3B-Inaae Pass @ K
8
16
32
64
128
256
AIME 2025
Base
0.2
0.3
0,6
1.2
2.4
4.6
8.45
14.2
20.0
GRPO
0.3
0.7
1.25
2.4
4.3
7.0
10.2
13.2
16.7
Cov KL
0.4
0.7
1.4
2.5
4.5
7.4
11.2
23.3
Pass K-mixed
0.7
1.3
2.3
3.9
6.3
9.1
12.6
16.7
20.0
THR
1.0
1.8
3,4
5.7
8.6
12.0
16.7
24.0
30.0
THR (p = 0.1)
1.1
2.1
3.8
6.7
10.7
15.3
19.7
24.2
30.0
THR (p 0.2)
0.9
3.4
11.1
17.8
36.7
AIME 2024
Base
1.4
2.6
8.3
13.4
20.3
28.4
35.9
40.0
GRPO
12.7
17.5
22.4
27.4
31.0
33.3
34.9
36.7
40.0
Cov KL
11.9
15.9
20.4
25.6
30.6
33.8
35.8
38.3
43.3
Pass@ K-mixed
12.2
22.4
27.4
32.8
35.1
43.3
THR
9.8
15.0
20.5
25.7
29.8
32.6
35.0
38.2
43.3
THR
0.1)
9.2
13.9
19.0
24.2
29.3
33,5
36.5
40.0
46.7
THR (p
0.2)
9.4
13.6
18.2
23.1
27.9
32.5
37.1
41.6
46.7
AMC 2023
Base
9.6
17.0
27.7
41.0
55.7
69.2
80.1
86.4
90.0
GRPO
26.7
36,9
47.3
56.4
63.6
69.5
74.8
85.0
Cov KL
28.9
39.3
49.6
57.9
64.7
70.8
76.2
81.1
85.0
Pass@ K-mixed
28.6
39.3
49.9
58.9
65.8
71.3
76.3
81.4
87.5
THR
26.8
37.9
57.9
75.2
82.3
90.0
THR
00.1)
26.1
36.4
56.4
65.5
74.2
81.5
87.0
90.0
THR
0.2)
26.5
36.7
47.6
57.8
66.9
74.4
80.2
84.3
87.5
Average
Base
3.7
6.6
11.0
16.8
23.8
31.4
39.0
45.5
50.0
GRPO
13.2
18.4
23.7
28.7
33.0
36.6
40.0
43.2
47.2
Cov KL
13.7
18.6
23.8
28.7
33.3
37.3
41.1
50.5
Pass@ K-mixed
13.8
19.3
24.9
30.1
34,3
37.7
41.3
45.4
50.3
THR
12.5
18.2
24.1
29.8
35.1
39,9
44.7
49.9
54.4
THR (p = 0.11)
12.1
17.5
29.1
35,2
41.0
45.9
50.4
55.6
THR
12.1
17.1
22.5
28.1
33.7
39.3
45.0
50.7
57.0
Table 9: Pass performance of different methods using
Exploration Results on Llama As shown in Table 9 THR still substantially boosts exploration, achieving over 7% Pass @ K improvement compared to GRPO. Setting < 0 amplifies these exploration gains even further. While baselines such as COV-KL and Pass K-mixed also provide exploration improvements, they consistently underperform relative to THR.
C.4 ADDITIONAL THR TOKEN ANALYSIS
We further analyze tokens with high THR values using a word cloud visualization, as shown in Figure The representative tokens can be organized into five functional categories that correspond to step-by-step reasoning:
Stating the Given Information: tokens that capture the initial conditions or input facts (present,
data, paper).
Transformation and Operations:
tokens that describe conversions, equivalence, or transfers of knowledge (conversion, transfer, equivalent).
Constraints and Relationships: tokens indicating dependencies, limitations, or structural relations (relative, intersects. amount, dimensions).
Decision and Selection: tokens reflecting choices among alternatives or branching reasoning paths (determine, instead, alternating, allern, others).
17

---

Published as a conference paper at ICLR 2026
Verification and Conclusion: tokens signaling validation consolidation of results (confirms, systematic, answer).
C.5 RUNNING TIME OF EACH MODULE.
We also track the average time cost of each module during training, as reported in Table 10 Notably, the data generation (Data Gen) module that using dynamic sampling accounts for the majority of the total training time. In contrast, the overhead introduced by THR is minimal. e.g. 37 seconds for
Qwen2.5-Math-l1. contributing only small fraction to the overall cost.
Model+dataset
Data Gen
Model Upd
THR
Ref
Old Prob
Total (Sec)
Owen2.5-Math-1
347
210
37
120
120
834
Owan2.5-Math-7B
422
371
39
187
187
1206
-lamma32-3B-Instmaatti
625
139
26
89
89
968
Table 10: Average running time (per step, in seconds) of each module for different models and tasks.
D DETAILED PROOFS
D.1 PASS@K AS THE QUESTION LEVEL REWEIGHTING
Chen et al. 22025): Mahdavi et al. (2025) Walder Karkhanis (2025) develop RLVR objectives that directly target Pass @ K optimization. Starting with GRPO's ancestor, REINFORCE, Mahdavi ctt al. (2025); Walder Karkhanis (2025) derive reward rescalings by directly optimizing the Pass @ K objective. Mahdavi et al. (2025) apply the same rescaling advantages giving GRPO version of their approach. These rescalings upweight the gradient contribution of correct responses that constitute "rare successes"
responses associated with "hard" questions. Crucially, the reweighting is uniform across all tokens and responses for a given question, which we term question- level reweighting. More recently, [Chn et al. (2025) introduce appealing alternative optimizing Pass K by incorporating the design directly within GRPO's 's group structure. Here, we simplify the formulas in Chen al. (2025) and arrive an explicit formulation of advantage shaping that reveals its question-level nature. Starting from the defined advantages in Chen et al. (2025):
Remau 1-
K
group
ReroUP x (1 Reroup)
K
A QK
1
Rgroup
1
Reroup
(N--1)
(K1)
x (aotuup)-
Ang
o group
(x-1)
Since N (1 G then we can obtain:
A a pos
K
(2) 08ooup
1==0 ((1 9)G - i)
I1--0 (G- i)ouu
())) (% )/()
(E)
q
4
(
q V1-9
A Apos
(11)
)/()
18

---

Published as a conference paper at ICLR 2026
then harder question will have larger 1 q thus larger advantage, then derive the negative advantage.
N
A ARK
K
K
1
@eroup
K
K
K-1
(N- i)
I (N--1)
(N i)
I (N -i)
geroup
G II k-1 (N - i)
N
(G i)
o
q
A RR pos
(Apps
9
1-9
-
(2)
Aneg
(12)
(%x (/(2)
V
By combining Equation [1] and Equation
12) we arrive at Equation (6) completing the derivation.
D.2 RELATIONSHIP BETWEEN THR AND ENTROPY REGULARIZER
Under some mild assumptions. optimizing THR plays a similar role as regularizing the evolution of the token entropy more efficient way. Because, stated in the main context, THR considers cross-token influence while current analysis on token entropy consider the influence of learning observing token on itself al. (2025). We start from Lemma proposed in Cui et al. (2025). which is how the Cov-KL regularizer is derived.
Lemma in Cui (2025 Let the actor policy wy be tabular softmax policy, the difference of information entropy given states between two consecutive steps satisfy:
AH' & H(Ta((t1) - HH(TT())) - Cov -CVV y (log To( (y x)) - -1,) (13)
where is the logits vector provided by the model after feeding the input æ. For notational simplicity, we use the superscript Ê to denote the training step, rather than an exponent. The equation above holds long as first-order Taylor expansion valid the logits level, independent of the specific model under consideration. In other words, this lemma is agnostic to the mechanism by which evolves, which depends on the particular model architecture or parameterization.
Recall the definition of the covariance:
Covyna(X,X)) = Ey [X Y] - Eynu [X] [Y].
"The strength and direction controlled by the value and sign of hyper-paramcter
19

---

Published as a conference paper at ICLR 2026
Equation 13 can then be written as:
AH' (x) = Cov ywone (-xx) (log (V x), 1/ - -1y)
E Towe [log To( (y By) m - -1y] Byy=ooc [a - -1,) log Toce) x)]
= - - BynTor 11tt 1y] Bynaoce [(1t 1 log Tacc) (v x)]
-(((TT()) Σ Ty u 1x)(+t1 14)
To(n = 11)00 1,) (y =
Σ (y = 100+ + +1 (H((T)()) + log Te(e) (y = v x))
==1
(H(TTCC))TTON) x) (- x) 2 log To(a) x),1't1 12)
-(((a(() TocC x) H(TT()) T Totol 2)) C log To(e) (- x)) 1 -
Vx1.deffned Q(x)
= =c(-Q(x) - - 1) 1 ((x)))
(14)
where the operator @ is the element-wise multiplication of two vectors, X _ æ, Yek is the context for the prediction of the k-th token, and e a constant for notation conciseness. In the last equation, we reintroduce the input X to the notation remind readers that the entire equation is conditioned given context sequence X. That is an important extension, because most existing works on entropy regularization (e.g.. Cui et al. 2025)) only focus the influence introduced by updating the observing token on itself. In other words, the x for Q and are identical. The Cov-KL method compared in Table just applies the quantity above select tokens with high covariances, and then uses the KI. penalty to restrict the update them.
We here connect THR to entropy in more systematic way by showing that THR control the rate of entropy growth (x) through the choice of p. Beyond the simplified tabular softmax setting, our analysis extends more realistic models with shared parameters across tokens. In this case, THR naturally captures the cross-token influences that arise throughout the learning process. In other words, when tracking the confidence change Ta(e) (y x). THR accounts for the learning dynamics of all other tokens across all responses, i.e.. Vi for varying i and k.
To make the notations concise, we follow the settings in Ren & Sutherland (025) and use Xo and X to denote the "observing" token and "updating" context, respectively. Then, Equation (4) becomes:
AH' (x) = c(-Q(x,) - Te (- xo),, 1 (x.) - (xxx))
Following Deng al. 2025), and under the unconstrained features assumption Deng et al. (2025); Mixon et al. (2022), we then represent (xx) = W'h,, where W e Vdd xd denotes the shared read-out layer and h, E d is the feature vector produced by the LLM backbone, conditioned on the context sequence Xufo = x, Uu/o,<k- Note that while ('(x)) shares the same W the feature vector h differs across contexts due to variations in input sequences. The difference vector | (x.) - (x.)) E IR Vx1 can then be expressed as:
r++ (xx) - (xxx) = (W++1 W - V C(a (Wh,), e, h.
where is the learning rate, a is the softmax function, and e., the one-hot distribution determined by the label of yu- When the cross-entropy loss is considered, the equation above can be simplified to
1++ (xx) ((xx)) = (e To(e) Xx))
h ho
VX1
1x1
Substituting this back Equation 14p. we can get
AHL (-Q(xo) T Xo); eu -
Xu)) h hT ho
(15)
20

---

Published as a conference paper at ICLR 2026
Curve -xlog
Curve of -xlog Ulog-scaled x-axis)
0.35
0.35
0.30
0.30
0.25
3.20
0.20
0.10
0.10
0.55
0.03
0.00
0.0
02
0.5
1.0
10
X.00g
Figure 8: The shape of =logg for E (0, 1), shown both the original and logarithmic scales.
0.2
ST"
8.10
0.50
0.3
0.23
10
0.2
6.2
0.00
3430
QCxo)+
0.2
0.5
0.00
0.0
03
10
17
V
Figure 9: Four examples of distribution of n. e, - π and Q + m.
Now, recall our definition of THR in Definition 4.1 where for each in the summation, the term has the format (h (hx.ttex hx,yev), which is just h h, above. Combining the definition of a and using the notations in this section, we can rewrite the signed-THR as follows:
sygn(ww) THR (yo: Yo. k))
Σ (c. ap Xo), eu T-(r) Xw)) h ho.
(16)
where sygn depends whether the completion is correct not. Now, comparing the inner product in Equation (15) and Equation (16). it is clear that the directional similarity between (x()))
and e, determines the effect introduced by THR and the entropy regularizer.
We now show that, under mild assumptions (which typically hold during LLM fine-tuning). -x.x) and e, point very similar direction (measured by their cosine similarity).
This observation follows from the shape of the function - log æ, illustrated in Fig. g In distribution where most probability mass is concentrated on few dimensions, the dominant entry T
Xo) @ log (- Xx.) is significantly larger than the rest. To validate this, we randomly generate distributions and compute the cosine similarity between - -(xx)) and e, in Fig. andd Fig. I0 The results show a clear trend: as both the vocabulary size and the peakiness of the distribution increase, the alignment between the two vectors becomes stronger.
We now examine the relationship between THR and entropy. Recall that THR defined as
TH)(P) 1||THR,,A| 7) (1 sign p) A A A
When p < O, tokens with larger THR values receive stronger penalties. Since, most cases, AH' (x) and THR point in similar directions. this implies that tokens with higher potential entropy change penalized, closely aligning with the intuition behind Cov-KL. However, as shown in our experiments, THR achieves greater improvements in exploration performance because it explicitly accounts for cross-token influence, rather than relying solely on entropy-buse based signals on a token't 's self-influence. as in COV-KL Cui al. (2025)
21

---

Published as a conference paper at ICLR 2026
0.8
0.8
0.05
mean V:20
6.7
LM
0.6
mean
0.45
mion V:50
0.5
0.55
meen V:70-80
0.6
V180 90
V:30
meann Vi110-120
probubility
Figure 10: We sweep the value of vocabulary size V and argmax probability of the distribution n².
The distribution is generated by fixing x¹ and randomly assign the extra probability mass to other dimensions. The results show that the cosine similarity between C, and Q is indeed very large when V and x² are large enough.
E MORE STUDIES
E.1 ABLATION STUDY ON
In this section, we conduct ablation study on
for exploitation: For exploitation, evaluated p E {0, 0.05, 0.1, 0.2). The results in Table I show that decreasing from 0. to 0.05 achieves the higher greedy accuracy, outperforming GRPO by 2.8%. This suggests that milder exploitation strength more suitable for the Siven.5-MMath 1arth-1.5t 1.5B model. In contrast, increasing to 0.2 leads to a slight drop in greedy accuracy compared with p 0.1, likely due to excessive exploitation.
A blation Study
Hard Dataeets
Standard Datasels
Base Model
Method
Total
Avg.
Olympiad
Aug.
Basc
0.0
3.3
20,0
7.8
39.6
7.7
24.9
24.1
15.9
GRPO
13.3
57.5
21.7
71.8
29.0
34.1
15.0
31,8
Qwen2.5-Matb-1158
THR
3.3
13.3
55,0
23.9
70.8
32.4
34.1
45.8
34.8
THR (p 0.1)
10.0
13.3
60.0
27.8
70.6
32.0
32.7
45.1
36.4
0.05)
THR 0.1)
3.3
13.3
62.5
26.4
71.4
33.1
34.5
46.3
36.3
THR (+ 0.2)
60.0
33.9
35.7
Table 11: Exploitation Results hard and standard math datasets. Pass@1 I accuracy (%) using greedy decoding across different methods and datasets. Bold is best performance, underline is second-best.
Ablation Study on p < for exploration. For exploration, we evaluate p E {0, 0005, -0.1, -..1.-1. -02)). As shown in Table [2] we observe consistent exploration trend where all three p can consistently improve the pass performance over GRPO, thus reinforcing the conclusion that < can enhance exploration.
E.2 GRADIENT STEPS CONVERGENCE
Effective Gradient Steps. We note that a "step" our setup corresponds to 32 gradient steps. We follow standard GRPO practice and with prompt batch size of 256 and 8 rollouts per prompt. Then we use mini-batch size of 64, resulting in 32 gradient steps per step. Therefore, 40 steps corresponds to 1280 gradient steps.
Validation accuracy along Steps. We show the convergence of training by demonstrating the
accuracy of validation dataset of MATH (levels 3-5) Hendryeks et al. (2021), shown in Figure II] the validation performance continues to improve gradually until around 30-35 steps, after which the increasing flat, indicating that the model is convergence, thus 40 steps for consistency.
22

---

Published as a conference paper at ICLR 2026
Mothod
Qwen2.5-maan 5-math--m-555 @
32
64
128
AIME
GRPO
5.9
9.9
15.0
20.5
26.5
33.6
41.5
49.8
56.7
THR (p 0.05)
5.4
9.2
14.1
19.4
25.0
31.7
39.5
48.0
56.7
5.9
10.1
34.7
THR (P -0.1)
6.0
10.1
15.3
20.9
26.8
33.9
41.7
50.0
60.0
THR (P 0.2)
6.0
10.2
15.6
21.4
28.1
36.2
44.0
49.8
$3.3
AIME 2024
GRPO
36.7
43.4
50.0
THR
10.6
16.7
23.4
30.2
37.2
44.8
51.9
58.5
63.3
THR (p 0.05)
11.9
18.2
24.8
31.0
37.5
44.8
52.9
61.6
70.0
THR (p 00.1)
31.2
37.9
45.3
52.9
61.2
THR (P 0.2)
12.2
18.3
24.7
31.2
38.6
47.4
56.6
64.0
70.0
AMC 2023 GRPO
46.6
59.1
70.0
78.9
90.2
93.7
96.0
THR
44.8
57.8
69.1
78.2
85.1
90.1
93.6
95.9
97.5
THR (p 0.05)
48.1
60.6
71.2
79.5
85.3
89.8
93.6
97.1
100.0
(p 0.11
95.1
THR (P -0.2)
50.3
62.4
72.4
80.4
86.4
91.3
95.4
98.3
100.0
Average GRPO
21.3
28.9
36.4
43.3
49.6
55.7
61.7
67.3
72.5
THR
20.3
28.0
55.5
61.7
67.5
THR (p 0.05)
22.0
29.6
37.2
43.9
50.1
56.4
62.8
69.4
76.7
THR (p -0.1)
21.9
29.8
37.5
44.4
50.7
57.3
63.2
69.7
76.7
THR (p
22.8
30.3
37.6
51.0
58.3
65.3
70.7
Table 12: @ performance of different p for Qwen2.5-math-1.55)
Convergence Analysis on Validation Accuracy
Convergence Analysis on Validation Accuracy
0.725
0.8
Validation Accuracy
0.675
Velidation Accuracy
0.7
0.650
0.625
0.6
0.5
0.575
0.550
1.58 GRPO
7B GRPO 78 THR (p-0)
(p=0)
Step
Step
Figure 11: Validation accuracy along training of Qwenn.5-Math- 1.5B and Qwen2.5 --at5-
Reward along Steps. For completeness, we include the reward curves in Figure 12 As shown, the reward rises during the early phase and then stabilizes around 0.55 the 1.5B model and 0.6 for the 7B model, demonstrating that training remains stable throughout. Although dynamic filtering prevents the reported reward from capturing the true correctness model outputs, it remains a useful proxy for assessing training stability.
E.3 COMPARISON WITH CLIP-HIGH
In this section, we compare against the high baseline Yu et al. (2025) using the recommended
clipping value of 0.28. As shown in Table [3] clip-high improves exploration for K 32 relative to GRPO. Nevertheless, despite its strength, THR (p < 0) consistently surpasses clip-high across all K, highlighting the effectiveness of THR (p in enhancing exploration.
23

---

Published as a conference paper at ICLR 2026
Reward along training
Reward along training (Dynamic Filtering)
0.59
0.55
Reward
Reward
0.50
0.50
0.45
0.45
1.58 THR (p=0)
78 THR (p=0)
0.30
10
20
30
40
40
50
Step
Step
Figure 12: Reward (dynamic filtering applied) along training of Qwen2.5-Math-1.55 and Qwen2.5- Math-7B.
Qwen2.5-Math-1a M.th Pass@ K
Method
2
32
64
128
256
AIME 2025
GRPO
5.9
9.9
15.0
20.5
26.5
33.6
41.5
49.8
56.7
Clip-High
5.6
9.6
14.8
20.5
26.6
33.7
41.6
48.4
53.3
THR (p < 0)
6.0
10.1
15.3
20.9
26.8
33.9
41.7
50.0
60.0
AIME 2024
GRPO
11.4
17.7
24.3
30.5
36.7
43.4
50.0
56.0
63.3
Clip-High
10.8
16.7
23.2
29.8
36.5
44.0
52.1
60.7
70.0
THR (p
11.9
18.2
24.9
31.2
37.9
45.3
52.9
61.2
70.0
AMC23
GRPO
46.6
59.1
70.0
78.9
85.5
90.2
93.7
96.0
97.5
Clip-High
47.3
59.9
70.5
78.8
84,9
89.8
93.8
97.3
100.0
THR (p < 0)
47.9
61.0
72.2
81.1
87.3
91.6
95.1
98.0
100.0
Average
GRPO
21.3
28.9
36.4
43.3
49.6
55.7
61.7
67.3
72.5
Clip-High
21.2
28.7
36.2
43.0
49.3
55.8
62.5
68.8
74.4
THR (p < 0)
21.9
29.8
37.5
44.4
50.7
57.3
63.2
69.7
76.7
Table 13: Comparing exploration ability with Pass@ Qwen2.5-Math-1.5B across AIME 2024,
AIME 2025, and AMC23.
E.4 STUDY ON EEROR-CORRECIIO BEHAVIOR.
In this section, we investigate how THR (p < 0) relates to corrective and self-verifyig behaviors. To quantify this, compute the ratio of reflection-related tokens to the total number of generated tokens. The full list of reflection-relater words used for this analysis is provided in Table 14
Reflection Words
actually correct
although
but error fix
alternating
Qwen2.5-Mtth-1158
despite
Method
#Reflection Token / #Token
however
incurrect
instead
mistake realize still
nevertheless
nonetheless
note
GRPO
0.34%
realized
rethink
reconsider
THR
0.36%
thinking
think
though
wait
THR (p 0.1)
0.55%
whereas
otherwise
wrong
unless
Table #Reflection Token / #To- ken ratio for Qwen2.5- Owwnn.55Math-11 M-Mth-11555
Table 14: List of reflection-related words.
24

---

Published as a conference paper at ICLR 2026
We then report the frequency of these tokens in Table 13 which shows that setting p < 0 increases the presence of reflection tokens. This indicates that THR (p < 0) can encourage more verification and correction behavior.
E.5 TOKENS RETAINED
The threshold is inherently adaptive, as it is defined as the average influence of a correct response's ken on the likelihoods of all correct responses. We
Model
Avg. Ratio Retained
report in Table T6 the average proportion of tokens retained under this threshold for Qwen2.5-Math-1J.. and Owen2.5-Matt-77B Notably, the 7B model re- tains fewer high-THR tokens, which is expected: stronger model possesses more knowledge, is more
Qwen2.5-Math- Qwen2.5-Math-7B
18%
14%
Table 16: Average fraction of tokens retained under the adaptive threshold т. confident in its answers, and therefore relies fewer influential tokens.
F USAGE OF LARGE LANGUAGE MODEL
In preparing this paper, made limited use of ChatGPT to support writing and editing. Specifically, LLMSs were employed for language polishing, grammar refinement, and rephrasing sentences to improve clarity and readability. Importantly, all technical content, including theoretical analysis, algorithm design, and experimental results, was conceived, implemented, and validated by the authors. LLM outputs were always critically reviewed, verified, and revised before inclusion. No LLM-generated text, figures, or tables were incorporated without careful human oversight.
25


Reviews:
[{"review_id": "9BZ1Ep59tq", "reviewer": "Reviewer_KUZ2", "summary": "The paper introduces Token Hidden Reward (THR), a token-level metric that quantifies each token’s impact on correct-response likelihood in popular GRPO frameworks. The authors show that a small subset of high-THR tokens dominates training, enabling fine-grained control of the exploration–exploitation trade-off: positive THR drives exploitation, while negative THR promotes exploration. The proposed THR-guided reweighting steers learning toward exploration/exploitation, improving greedy accuracy (p > 0)/Pass@K (p < 0) on math reasoning benchmarks.", "questions": "- Why does training on THR-dominant tokens yield comparable, instead of better performance compared to GRPO given its potential effectiveness in proving stronger learning signals to guide the training process? Alternatively, does it lead to faster convergence?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduced THR framework is principled and fundamental, and connects learning dynamics in GRPO-tuned LLMs to the exploration–exploitation trade-off. This formulation offers a conceptually grounded and theoretically principled perspective that deepens understanding of how token-level interactions shape model behavior during reinforcement learning.\n\n- The paper is generally well written, logically structured, and clearly motivated, making it easy to follow. The authors conduct extensive experimental studies across multiple model scales (0.5B–7B) and architectures (Qwen and Llama), complemented by additional analyses and ablation experiments. Collectively, these results provide solid empirical support for the proposed approach and its underlying claims (while there are some inconsistencies and potential concerns; see weakness section).", "weaknesses": "- The empirical improvements reported between variants and baselines are relatively modest. Incorporating statistical significance analyses would help validate the robustness of these gains and ensure that the observed effects are not attributable to stochastic randomness.\n\n- The reported results indicate that the optimal choice of the hyperparameter p varies across model sizes and model families (e.g., Qwen vs. Llama), suggesting that the method’s efficacy may depend strongly on careful tuning. A more systematic discussion or ablation study on how p is selected—and whether its value generalizes across architectures or tasks—would strengthen the methodological soundness (please correct me if I missed such discussion).\n\n- In Table 1, the observation that p > 0 fails to consistently outperform p < 0 for smaller models (0.5B and 1.5B) appears inconsistent with the claimed interpretation that positive p promotes exploitation. This discrepancy weakens the theoretical linkage between the sign of p and behavioral control.\n\n- While experiments on Llama are included to demonstrate the generalizability of the proposed THR method, the reported results appear to hold primarily for the exploration configuration (p < 0). The exploitation experiments on Llama (Table 8) show that THR with p > 0 underperforms the GRPO baseline, suggesting that the proposed mechanism may not generalize robustly across architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsKB8zBoxP", "reviewer": "Reviewer_H57a", "summary": "The paper proposes THR, a token-level signal derived from the training dynamics of GRPO-style RLVR that estimates whether a token will increase or decrease the likelihood of correct responses. The method operationalizes THR in two ways: (i) dominant-token selection that keeps only tokens with large |THR| and (ii) sign-based reweighting that tilts updates toward positive-THR tokens (exploitation) or negative-THR tokens (exploration). The stated goal is to provide a principled, fine-grained knob to trade off exploitation vs. exploration.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is anchored in a theoretical motivation (training-dynamics view under GRPO) rather than ad-hoc heuristics. It is intuitive that upweighting positive-THR tokens should push the model toward already promising trajectories, while emphasizing negative-THR tokens should diversify candidates.\n- The structure of the paper is clean and logical, which makes it straightforward to map claims to evidence. Notation is compact, figures/tables are readable, and the roles of the sign-weighting scalar are explained sufficiently to re-implement. The exposition makes the method feel accessible to practitioners who already run GRPO/GSPO.", "weaknesses": "- The paper mentions training for ~40 steps, which is typically short for convergence in RLVR settings. From an evaluator’s perspective, this makes it hard to judge stability and whether gains persist or are transient. It would strengthen the paper to include training curves.\n- Prior work (e.g., DAPO) shows that increasing clipping threshold (clip higher) can implicitly encourage exploration by preserving higher-entropy tokens. Given the paper’s token-level framing, a “clip-higher” baseline in Table 2 would be an informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkbaEeArxa", "reviewer": "Reviewer_zvLX", "summary": "This paper introduces Token Hidden Reward (THR), a token-level metric that quantifies how individual tokens influence the change in the likelihood of correct responses in LLM-RL training. First, the paper analyzes how THR is distributed in correct and incorrect responses and it shows that majority of tokens have 0 THR values and small subset of tokens have large positive and negative values. Based on this insight, the paper proposes two training scenarios with THR, THR-only, where advantages of only tokens with sufficiently large THR values are used, and THR with exploration or exploitation, where reweighting advantages based on THR values. In experiments, THR-only shows comparable performance to the original GRPO variant, which indicates that only small subset of tokens influence training performance. The experiments also show that reweighting advantages based on THR values can steer exploration and exploitation and lead to improved performance in math tasks. Lastly, the paper shows connection between THR and entropy, which indicates that high THR tokens and high entropy tokens are overlapped.", "questions": "`Weakness` section includes questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and comprehensive. The concept of THR is interesting and generic to LLM-RL settings. And, the nice thing is that the THR values can be computed without additional training components so that it can be essentially used in a plug-and-play style. Also, the paper shows that the THR has an interesting property, which is that small subsets of tokens with large THR values dominate training performance. This fact is a new insight along with how token entropies involve LLM-RL training. The paper provides extensive experiments to show how THR can steer exploration and exploitation and how it can improve performance with math tasks.", "weaknesses": "Although I couldn't find obvious flaws in this paper, there some comments that should be addressed:\n- How much does THR calculation add computation cost? I imagine that computing THR every time is not a trivial thing to do. The authors should provide information about this.\n- This is a fundamental question about the proposal in the paper. Since THR and entropy are high correlated, can we just use the entropy as a proxy of THR? If so, the implementation of algorithms with THR could be simplified. Is there a practical reason to keep using THR over entropies? This seems clarified in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9Qt9AJUcp", "reviewer": "Reviewer_on7i", "summary": "This paper introduces Token Hidden Reward (THR), a token-level metric that quantifies how each generated token contributes to the change in the likelihood of correct responses under GRPO-style RL with verifiable rewards. The key idea is to decompose the learning dynamics into per-token “hidden rewards” whose magnitude identifies a small set of dominant tokens driving updates, and whose sign aligns with the exploration–exploitation trade-off: positive THR tends to strengthen confidence in correct outputs (exploitation), while negative THR tends to preserve probability mass for alternative outputs (exploration). Building on this insight, the authors propose a simple token-level advantage reweighting scheme that:\n\n- masks low-influence tokens (dominant-token training),\n- and amplifies tokens by sign to bias toward exploitation (p>0) or exploration (p<0).", "questions": "Interesting work, I wonder how this ties to goal-conditioned RL and if there some applications there.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality\n\n- A clear, token-level decomposition of learning dynamics under GRPO that cleanly ties per-token influence to exploration vs. exploitation via the sign of THR. This is a novel perspective beyond question-level hardness reweighting or per-token self-entropy methods, and extends prior analysis (e.g., negative-gradient focus) to both signs and cross-token effects \n\n- The framing of “dominant tokens” and the empirical finding that masking to high-|THR| tokens retains performance is a crisp, interpretable contribution that could inform more efficient training \n\nQuality\n- Theoretical grounding connects THR to likelihood dynamics and relates it to entropy regularization; the analysis explains why the sign of THR modulates exploration–exploitation \n\n- Broad experimental sweep across models (Qwen 0.5B/1.5B/7B; Llama3.2-3B) and RL objectives (GRPO, GSPO-token). The p>0 vs p<0 behavior is consistently demonstrated; THR compares favorably to Pass@K-mixed and COV-KL on Pass@K \n\n- Practicality: THR’s compute overhead is small relative to data generation, suggesting the technique scales. The paper explicitly reports module-wise runtimes, which is valuable for practitioners \n\nClarity\n\n- Clear definition of THR and reweighting, with equations and intuitive illustrations (e.g., sign interpretation, density plots of THR, overlap with high-entropy tokens). The narrative links theory → algorithm → empirical behavior in a readable way \n\nSignificance\n\n- A fine-grained knob to trade-off greedy accuracy vs. Pass@K with minimal engineering changes can be valuable in RLVR pipelines, where one may want different behaviors by domain or deployment constraints (e.g., high-confidence single-shot vs. BoN sampling) \n\n- The “dominant tokens” lens might inspire further token-level curricula, diagnostics, or adaptive schedules.\nAdditionally reflecting the user’s perspective: the method is appealing because computing THR is relatively lightweight compared to rollouts, making it amenable to scaling and combination with existing RLVR loops.", "weaknesses": "Magnitude and consistency of gains\n\n- While some settings show non-trivial improvements (e.g., up to ~4 points Pass@1 on Qwen-7B), in several cases improvements over strong baselines like GRPO are modest and sometimes within typical training variance bands, especially at smaller model sizes or across certain datasets. Emphasize variance estimates (e.g., CI/error bars across seeds) to quantify significance of the deltas \n\n- The exploration benefits at large K are clear, but the practical importance of very large K (e.g., 128–256) may be less relevant for many real-world budgets. Provide a sharper focus on K in the 4–16 range and on success-vs-cost trade-offs \n\nGenerality beyond verifiable domains\n\n- The method and analysis are anchored in RLVR with binary/verifiable rewards and GRPO/GSPO-style group structures. It remains unclear how to port THR to non-verifiable domains (e.g., preference models, open-ended generation) where correctness signals are noisy/subjective. The paper mentions generalization across RL objectives but not across reward types or unverifiable tasks \n\nInteraction with error-correction behavior\n- The paper interprets negative THR as “preserving probability mass for alternative (than the correct) responses” to encourage exploration. It is less clear how this interacts with error-correction/repair behaviors, where one typically wants to actively down-weight error-inducing trajectories and prioritize corrective steps. More analysis is needed on whether amplifying negative-THR tokens inadvertently reinforces patterns that impede self-correction or verifier-guided repair \n\nScope of analysis and ablations\n\n- Threshold τ selection follows a prior influence-based heuristic. Sensitivity analyses on τ, the fraction of tokens retained, and p schedules are limited in the main text. Given the centrality of these choices, ablate:\nabsolute vs. percentile thresholds for |THR|,\ndynamic τ over training, adaptive p tied to q (group accuracy) or per-question difficulty \n\n- Baselines could include token-level reweighting heuristics that don’t require THR (e.g., top-|grad| tokens, per-token loss magnitude, or simple entropy-top-k), to isolate THR’s unique value beyond “focus on hard tokens.” Some overlap with entropy is discussed, but stronger head-to-head controls would increase confidence \n\nReporting and diagnostics\n\n- The paper makes strong claims about cross-token interactions; more direct diagnostics (e.g., intervention studies that swap token subsets, or causal tests across positions) would bolster this claim beyond correlation/overlap with entropy \n\n- Calibration and stability analyses are missing. Since THR aims to shape confidence, report calibration metrics, entropy dynamics over training, and instabilities (e.g., length collapse, variance across seeds) more prominently for each setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)