Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Authors: Shenzhi Wang
Le Yu
Chang Gao
Chujie Zheng
Shixuan Liu
Rui Lu
Kai Dang
Xiong-Hui Chen
Jianxin Yang
Zhenru Zhang
Yuqiong Liu
An Yang
Andrew Zhao
Yang Yue
Shiji Song
Bowen Yu
Gao Huang
Junyang Lin
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Reasoning
Reinforcement Learning
Entropy
Primary area: deep_learning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), yet its underlying mechanisms remain insufficiently understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction (approximately 20\%) of tokens exhibit high entropy, and these tokens semantically act as critical forks that steer the model toward diverse reasoning pathways. We further demonstrate that moderately increasing the entropy of these high-entropy tokens via decoding temperature adjustments leads to improved performance, quantitatively confirming their role as decision points in reasoning. We ultimately refine RLVR by restricting policy gradient updates to these forking tokens. Despite utilizing only 20\% of tokens, our approach achieves comparable performance to full-gradient updates on the Qwen3-8B base model. Moreover, it demonstrates remarkable improvements on the larger Qwen3-32B base model, boosting AIME'25 scores by 11.04 and AIME'24 scores by 7.71. In contrast, training exclusively on the 80\% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that dictate key reasoning directions. Collectively, our results suggest promising avenues for optimizing RLVR algorithms by strategically leveraging the potential of these high-entropy minority tokens to further enhance the reasoning abilities of LLMs.

Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Shenzhi Wang*.* Le Yu*, Chang Gao Chujie Zheng*, Shixuan Liu Rui Lu Kai Dang Xiong-Hui Chen Jianxin Jianxin Yang*, Zhenru Zhang Yuqiong Liu An Yang* Andrew Z Yang Yue*, Shiji Song Bowen Yut. Gao Huang Junyang Lin Department of Automation, BNRist, Tsinghua University
Qwen Team, Alibaba Inc.
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a pow- erful approach enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we un- dertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence rea- soning performance. By examining token entropy patterns in Chain-of- Chounhof- (CoT) reasoning, we observe that only small fraction of tokens exhibit high en- tropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base models entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.c., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% the tokens while maintaining performance comparable to full- graddint updates on the Qwen3-8B base model and significantly surpassing full- full-gradient updates the Qwen3-32B (+11.04 AIME'25 and +7.71 AIME'24) and Qwen3- 14B (+4.79 on AIME'25 and +5.21 on AIME' 24) base models, highlighting a strong scaling trend. In contrast, training exclusively the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.
Introduction
The reasoning capabilities of Large Language Models (LLMs) have advanced substantially in do- mains like mathematics and programming, propelled by test-time scaling methodologies employed in OpenAI ol [241], Claude 3.7 [U]] DeepSeek R1 [6]], Kimi K1.5 [64]] and Qwen3 [AI]]. A piv- otal technique driving these improvements is Reinforcement Learning with Verifiable Rewards (RLVR) [17 6 []] where models optimize outputs through RL objectives tied to automated cor- rectness verification. While recent RLVR advancements have stemmed from algorithmic inno- vations 22 44 [I0], cross-domain applications [40, 21, 26), and counterintuitive empirical in-
B: Corresponding Authors f: Project Lead
Emails: wangshenzhh1990mmai.. com yubowen. www.wnen.ph@mmal..ommm com gachuangttsinghua.c edu. Project Page: https:///shezhiii wwreenhhi-wang.... wwn...igtuu................. www..//hh.o.ooooooooooooo------.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) high-entropy minority tokens fork the path
low-entropy majority
(b)
AIME24
AIME25
60
tokens follow the path
60-
all
Tork only
50
26.7
Acc@11 [%)]
50-
25.8
40-
45.6
50.4
62.9
40-
20.1
30-
33.324.5
24.5
20
20
8B
14B
328
88
148
328
RL training on all tokens
RL training on forking tokens
reasoning path
Q: What is 1 in base 2?
A: In decimal, + 1= But how does that translate to base 2? Well, in binary [...]
reasoning path
Figure 1: (a) In CoT', only a minority of tokens exhibit high entropy and act as "forks" in reasoning paths, while majority tokens are low-entropy. (b) RLVR using policy gradients of forking tokens delivers significant performance gains that scale with model size. With 20k maximum response length, our 32B model sets new SoTA scores (63.5 on AIME'24 and 56.7 AIME'25) for RLVR on base models under 600B. Extending the maximum response length to 29k further boosts the AIME'24 24 score 68.1.
sights [37 3 Ad]. existing implementations directly train over all tokens with limited understanding of which tokens actually facilitate reasoning. These approaches neglect the heterogeneous functional roles tokens play in reasoning processes, potentially hindering further performance gains by failing to prioritize critical decision points in sequential reasoning trajectories.
In this paper, we analyze the underlying mechanisms of RLVR through an innovative lens of token entropy patterns, investigating how tokens with varying entropy impact reasoning performance. We first point out that in the Mhain-of-Thhugle (CoT) processes of LLMs, the entropy distribution ex- hibits a distinct pattern where the majority of tokens are generated with low entropy, while a critical minority of tokens emerge with high entropy. Through comparing the textual meanings of these two parts of tokens, we observe that the tokens with lowest average entropy primarily complete the ongoing linguistic structures, while the tokens with highest average entropy function as pivotal de- cision points that determine the trajectory of reasoning among multiple potential pathways (referred to as forks), as depicted in Figure I[aa). In addition qualitatively anslysis, conduct controlled experiments by manually modulating the entropy of forking tokens during decoding. Quantitative results reveal that moderately increasing the entropy of these high-entropy forking tokens leads to measurable improvements in reasoning performance, while artificially reducing their entropy results in performance degradation, confirming the importance of maintaining high entropy and the role as "forks" for these high-entropy tokens. Furthermore, by analyzing the evolution of token entropy dur- ing RLVR training, we find that the reasoning model largely retains the entropy patterns of the base model, exhibiting only gradual and relatively minor changes as training progresses. Additionally, RLVR primarily changes the entropy of high-entropy tokens, while the entropy of low-entropy to- kens varies only within a small range. The above observations highlight the critical role high-entropy minority tokens may play in CoTs and RLVR training.
Building upon the discovery of forking tokens, we further refine RLVR by retaining policy gradi- updates for only 20% of tokens with the highest entropy and masking gradients of the remain- ing 80%. We observe that although solely utilizing 20% of tokens, our approach can still achieve competitive reasoning performance on Qwen3-8B base model compared to full-gradient updates. Moreover, its effectiveness increases with model size, yielding reasoning improvements of +1 +11.04 on AIME'25 and +7.71 on AIME'24 for the Qwen3-32B base model, and +4.79 on AIME'25 and +5.21 on AIME 24 for the Qwen3-14B base model, as shown in Figure (ī(b). Notably, the 32B model trained with only 20% high-entropy tokens attains scores of 63.5 on AIME'24 24 and 56.7 on AIME'25, setting new state-of-the-an art (SoTA) for reasoning models trained directly from base models with fewer than 600B parameters. Extending the maximum response length from 20k to 29k further elevates our 32B model's AIME'24 score from 63.5 to 68.1. Conversely, training exclusively on the 80% lowest-entropy tokens results in severe performance degradation. These observations show that 20% of tokens achieve performance comparable to or exceeding 100%, even surpassing

---

the 80/20 rule. The results demonstrate that the high-entropy minority tokens, functioning as critical decision points in reasoning trajectories, account for nearly all performance gains in RLVR.
Finally, we explore why retaining small fraction of the highest-entropy tokens leads to strong per- formance in RLVR via series of ablation studies. We adjust the chosen fraction of forking tokens, either by decreasing it from 20% to 10% or increasing it to 50% or 100%, and report the corre- sponding reasoning metrics and overall entropy during the training process. Experimental results demonstrate that retaining approximately 20% of the highest-entropy tokens optimally balances ex- ploration and performance, while deviating from 20% reduces the overall entropy with diminished exploration and incurs worse performance. This suggests that only a critical subset of high-entropy tokens meaningfully contributes to the exploration during RL while others may be neutral or even detrimental. Reducing the proportion to 10% removes certain useful tokens, which weakens explo- ration. Increasing the proportion to 50% or 100% adds low-entropy tokens, which also reduces the effectiveness of exploration. Last but not least, retaining the top 20% of high-entropy tokens results in the largest performance gains for the 32B model, followed by the 14B model, and the smallest gains for the 8B model. This may be due to the insufficient capacity of the smaller model, which restricts its ability to benefit from increased exploration. These findings highlight the importance of preserving appropriate proportion of high-entropy tokens in RLVR. As model size increases, the strategy of selecting high-entropy tokens appears to scale effectively.
In summary, our findings emphasize the pivotal role of high-entropy minority tokens in shaping the reasoning abilities of LLMs. We hope this inspires further analyses from the perspective of token entropy and informs more effective RLVR algorithms that strategically leverage these tokens to enhance reasoning performance. The key takeaways of our paper are as follows:
In CoTs, the majority of tokens are generated with low entropy, while only small subset exhibits high entropy. These high-entropy minority tokens often act as "forks" in the reasoning process, guiding the model toward diverse reasoning paths. Maintaining high entropy at these critical forking tokens is beneficial for reasoning performance. (S2)
During RIVR training, the reasoning model largely preserves the base model's entropy patterns, showing only gradual and minor changes. RIVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens fluctuates only within narrow range. (SB)
High-entropy minority tokens drive nearly all reasoning performance gains during RLVR, whereas low-entropy majority tokens contribute little may even hinder performance. One pos- sible explanation is that, prior to performance convergence, a subset (~ 20% in our experiments) of high-entropy tokens facilitates exploration, while low-entropy tokens offer minimal benefit or may even impede it. (SE))
Based on the insights above, we further discuss (i) high-entropy minority tokens as a potential reason why supervised fine-tuning (SFT) memorizes but RL generalizes, (ii) how prior knowledge and readability requirements shape the different entropy patterns seen LLM CoTs compared to traditional RL trajectories, and (iii) the advantage of clip-higher over entropy bonus for RLVR. (Deferred to Appendix (D)
2 Analyzing Token Entropy in Shain-of-Though Reasoning
Although prior works [I [Bll have highlighted the importance of generation entropy in chain- of-thought reasoning, they typically analyze the entropy of all tokens collectively. In this section, we take closer look at generation entropy in chain-of-thought reasoning by examining it at the token level. To this end, we use Qwen3-8B [II]], one of the most recent and capable reasoning models within comparable parameter scale, to generate responses for queries from AIME'2 24 and 25ME'25, using a decoding temperature of T = 1.0. We enforce the use of the thinking mode for every question and collect over 106 response tokens. For each token, the entropy is computed according to the formulation in Equation (G). The statistical analysis of the entropy values of these 10 tokens is presented in Figure Furthermore, visualization of token entropy for an entire long CoT response is provided in Figures I to M in the Appendix. From these analyses, we identify the following entropy patterns:
'Due the page limit, the preliminaries and related work are deferred to Appendix A and B respectively.
3

---

10
PussSnn Since going Thus which re
However ier walt " ussume specifip express and
The 80th percentile: 0.672
maybe This lir uging actarin "it per haps,
Supp
de
Frequency (log scale)
10
ust.. unless S ving ac Ectual y by what Maybe
denotes uppose se 2n Sholr oa
We
which
10
de f
let The how dermutyy mac
dok
10a
(b) Frequent tokens with high average entropy
102
B
sin radius coss
ed right
-y -11
(B
10
empty
k6
NB
theta
7 of
Term
Z
100
type
Entropy
3
- implify
3/
cos
ab
Trac
(a) Distribution of token entropy
(c) Frequent tokens with low average entropy
Figure 2: Entropy patterns in the chain of thoughts of LLMs. (a) Token entropy distribution. The Y-axis frequency is on log scale. A minority of tokens exhibit high entropy, while the majority have low entropy, often approaching zero. (b) (c) Word clouds of the top 100 tokens with the highest and lowest average entropy, respectively, selected from the set of frequently occurring tokens. A larger font size indicates a higher average token entropy. Tokens with the highest average entropy typically function as "forks" to determine reasoning directions, whereas tokens with the lowest average entropy tend to execute reasoning steps along the established path.
Typically, only minority of tokens are generated with high en- tropy, while a majority of tokens are outputted with low entropy. We can observe in Figure ((a) that the entropy of large amount of tokens are quite small, and only small amount of tokens have high entropy. Specifically, the entropy of over half the tokens (approximately 50.64%) is below 1002 while only 20% of tokens have entropy greater than 0.672.
Entropy Pattern 1 in CoTs:
Entropy Pattern 2 in CoTs: Tokens with the highest entropy typically serve to bridge the logical connection between two consecutive parts of reasoning, while tokens with the lowest entropy tend to complete the current part of a sentence or finish constructing a word. Other tokens combine these two functions to varying degrees. In Figure (l(b) and (c), we select the 100 tokens generated with the highest average entropy and the lowest average entropy from total of 106 tokens, respectively. To mitigate the impact of noise on the average entropy, we only consider tokens with frequencies above 100. High-entropy tokens often act as logical connectors within and across sentences, such as "wait," " "however," " and "unless" (indicating contrasts or shifts), "thus" and "also" (showing progression or addition), "since" and "because" (expressing causality). Similarly, tokens like "suppose," " "assume," "given," and "define" frequently appear in mathematical derivations to introduce assumptions, known conditions, or definitions. Conversely, low-entropy tokens are often word suffixes, source code fragments, or mathematical expression components, all of which exhibit high determinism. Additionally, Figures D to II provide detailed visualization of token entropy in a long CoT, showing that most tokens outside the highest-entrope or ewwest-entropy groups blend bridging and continuation functions varying degrees.
High-entropy tokens as "forks" in thain-of-thoughhe we refer to high-entropy tokens as "forking tokens", as they often lead to different potential branches with high uncertainty in the reasoning process. To further confirm the role of forking tokens in a quantitative way, using Qwen3-8B [II]]] with maximum response length of 28,672, we assign different decoding temperatures to the forking tokens and the other tokens in the evaluation on AIME 2024 and AIME 2025. Specifically, to analyze the effects of varying combinations of these temperatures on their behavior, we adjust probability distribution p' E IRV for each token / as follows:
Based on the two observed patterns above,
p Softmax
Zt Tи
where T
[Tigh Tiow
if H, > harreshold: otherwise.
Here, 2, E RV denotes the pre-sofumax logits for token t, and T3 E R represents the adjusted
temperature for token i; hutreshold = 0.672 the entropy threshold used to distinguish forking tokens from the other tokens, and is estimated by calculating the 80th percentile among the sampled 106 tokens above; Thigh E and Tiw E correspond to the temperatures for forking tokens and the other tokens, respectively.

---

The effects of varying Thigh and Too are pre- sented in Figure It can be seen that lower- ing Tiigh significantly degrades performance compared to lowering Taw contrast, in- creasing high results in substantially better performance than increasing Tow. which can even cause LLMs generating nonsensical out- puts. These results suggest that forking to- kens benefit from being assigned relatively higher temperature compared to other tokens. Given that forking tokens naturally exhibit higher entropy than other tokens, this further supports the need for them to operate even higher entropy level. This observation aligns with their role "forks," " where high entropy enables them to branch into diverse reasoning directions.
72
71.04
71.11 71.7
72.22
70.52
70.76
70.94
70.73
AIME Score
68
68.33
68.96
66
64-
62.71
25
20.
0.14
0
0.0
0.05
1.0 2.03.0
Temperature (symlog scale, linthresh=0.2)
Figure Average scores of AIME 2024 and AIME 2025. Red curve: varying Thgh with Too = 1. Blue curve: varying Tow with Thigh 1.
RLVR Preserves and Reinforces Base Model Entropy Patterns
In this section, building the observations of entropy patterns in CoTs discussed in Section Q further investigate how these patterns evolve throughout RLVR training.
RLVR primarily preserves the existing entropy patterns of the base models To analyze the evolution of entropy patterns during RLVR training, we apply DAPO [[2]] to the Qwen3-14B base model (details Section BD. Using the reasoning model after RLVR, we generate 16 responses question across the six benchmarks in Table D. For each token in these responses, we compute logits using reasoning models from various RLVR stages and identify those in the top 20% entropy. We then calculate the overlap ratio (i.e., the fraction of shared top 20% high-entropy positions) between each intermediate model and both the base and final RLVR models. As shown in Table although overlap with the base model gradually decreases and overlap with the final RLVR model increases, the base model's overlap still remains above 86% convergence (step 1360), suggesting that RLVR largely retains the base models entropy patterns regarding which tokens exhibit high or low uncertainty.
Table 1:
The progression of the overlap ratio in the positions of the top 20% high-entropy tokens, comparing the base model (i.e., step 0) with the model after RLVR training (i.e., step 1360).
Compared w/ Step 0 Step 16 Step 112 Step 160 Step 480 Step 800 Step 864 Step 840 Step 1280 Step 1360
Base Model
100%
98.92%
98.70%
93.04%
93.02%
93.03%
87.45%
87.22%
87.09%
86.67%
RLVR Model
86.67% 86.71% 1% 86.83%
90.64%
90.65%
90.64%
96.61
97.07%
97.34%
100%
RLVR predominantly alters the entropy high-entropy tokens, whereas the entropy of low- entropy tokens remains comparatively stable with minimal variations. Using the same setup as Table II, we compute the average entropy change after RLVR for each 5% entropy percentile range of the base model. It observed in Figure E that tokens with higher initial entropy in the base model tend to exhibit larger increases in entropy after RLVR. This observation could also further reinforce that RLVR primarily preserves the entropy patterns of the base model. More experimental results and analyses are presented in Appendix GNl
High-Entropy Minority Tokens Drive Effective RLVR
Reinforcement learning with verifiable rewards (RLVR) has become one of most widely used approaches for training reasoning models [11]] R A2, Ez]. However, there is lack of research on which types of tokens contribute the most to the learning of reasoning models. highlighted in Section B and Section E. high-entropy minority tokens particularly important. In this section,
5

---

Average Change (log scale)
10
10-2
10-1
10
5%
10% 15% 20%
25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%100%
Percentile Boundaries
Figure 4: Average entropy change after RLVR within each 5% entropy percentile range of the base model. 2% percentile means that 2% of the tokens the dataset have entropy values less than or equal to this value. It is worth noting that the Y-axis is presented on a log scale. Tokens with higher initial entropy tend to experience greater entropy increases after RLVR.
investigate the contribution these high-entropy minority tokens, also referred to as forking tokens, on the development of reasoning capabilities during RLVR.
4.1 Formulation of RIVR Using Only Policy Gradients of the Highest-Entropy Tokens
Building on DAPO's s objective in Equation (.), we discard the policy gradients of low-entropy tokens and train the model using only the policy gradients of high-entropy tokens. For each batch B sampled from the dataset D, we calculate the maximum objective as:
10'1
JighhEn (0) 1 Nenal
+
Ee Eee
H
-min (ri(0) clip(r:(0), Elow; 1 thigh Ai) s.t.
f0' is (s_equivalent(a.o))) G.
(2)
Here, Hi denotes the entropy of token in response i, I[-] is the indicator function that evaluates (0, 1] is predefined ratio specifying the top proportion of high-entropy tokens to be selected within batch, and r is the corresponding entropy threshold within the batch B such that only tokens with Hi 2 Tp. comprising the top-p fraction of all tokens in the batch, used to compute the gradient.
1 if the condition inside holds and () otherwise,
Comparing Equation (2) with Equation (E), there are only two differences, as highlighted in red in Equation (Z): (i) The advantage term is multiplied by II H 2 r ensuring that within each Amiero-)bated B, only tokens of whose corresponding entropy H; 2 TO involved in the pol- icy gradient loss calculation; (ii) The normalization term for the number of tokens is adjusted to EE Ei H 2 - meaning that only tokens whose entropy is at least the threshold B are p
considered.
4.2 Experimental Setup
We adapt our training codebase from verl [32] and follow the training recipe of DAPO [22], one of the state-of-the-art RI. algorithms for LLMss. Both configurations, RLVR with full gradients (vanilla DAPO depicted in Equation (E)) and RLVR with only policy gradients on fork- ing tokens (described in Equation (2)), employ techniques such as clip-higher, dynamic sampling, token-level policy gradient loss, and overlong reward shaping [32]] For fair comparisons, we apply the same hyperparameters as recommended by DAPO: for for overlong reward shaping, the maximum response length is 20480 and the cache length is 4096. Furthermore, we use training batch size of 512 and a mini-bated size of 32 in verl's configura- tion, resulting 16 gradient steps per training batch, with a learning rate of 10 and no learning rate warmup scheduling. Importantly, the training process excludes both KL divergence loss and entropy loss. To evaluate the scaling ability of these methods, we perform RLVR experiments across the Qwen3-32B base and Qwen3-8B base models, using DAPO-Math-11 17K [> as the train-
Training details
Chigh 0.28, Cjow = 0.2;

---

Table 2: Comparison between vanilla DAPO using all tokens and DAPO using only the top 20% high- entrppy tokens (i.e. forking tokens) policy gradient loss, evaluated on the Qwen3-32B, Qwen3-14h and Qwen3-8B base models. "Acc@ 16" and "Len 16" denotes the average accuracy and response length over evaluations per benchmark, respectively.
DAPO w/ All Tokens
DAPO w/ Forking Tokens
Improvement
Benchmark
Acc@
Len@ 16
Ace@
Len@
Acc@ 16
Len 16
RLVR from the Qwen3-32B Base Model
AIME'24
55.83
9644.15
63.54
12197.54
+7.71
+2553.39
AIME'25
45.63
9037.48
56.67
11842.25
+11.04
+2804.77
AMC'23
91.88
5285.03
94.22
5896.47
+2.34
+611.44
MATH500
94.36
2853.51
94.88
3366.01
+0.52
+512.5
Minerva
45.70
2675.28
45.82
2759.88
+0.12
+84.6
Olympiad
66.16
5597.37
69.02
7300.01
+2.86
+1702.64
Average
66.59
5848.80
70.69
7227.03
+4.10
+1378.22
RLVR from the Qwen3-14B Base Model
AIME'24
45,21
7945.15
50.42
11814.36
+5.21
+3869.21
AIME'25
38.13
7056.98
42.92
12060.48
+4.79
+5003.5
AMC'23
89.53
4509.37
91.56
7095.13
+2.03
+2585.76
MATH500
92.23
2348.22
93.59
3970.10
+1.37
+1621 1.8
Minerva
42.16
1.16
43.20
2959.32
+1.03
+948.
Olympiad
61.14
4642.07
64.62
7871,25
+3.48
+3229.18
Average
61.40
4752.16
64.39
7628.44
+2.99
+2876.28
RLVR from the Qwen3-8B Base Model
AIME'24
33.33
6884.89
34.58
9494.29
+1.25
+2609.40
AIME'25
25.42
5915.91
26.25
8120.20
+0.83
+2204.29
AMC'23
77.81
3967.91
77.19
5450.62
-0.625
+1482.71
MATH500
89.24
2059.00
89.70
2672.91
+0.46
+613.91
Minerva
39.77
1450.68
40.26
2068.41
+0.48
+617.73
Olympiad
56.67
3853.55
57.43
5241.54
+0.76
+1387.99
Average 53.71
4021.99
54.23
5508.00
+0.53
+1486.01
ing dataset. For main results, we set = 20% in Equation (2). meaning that the policy is updated using only the gradients of the top 20% highest-entropy tokens within each batch. The chat tem- plate we use for Qwen3 models is "User:\n(question [nneettio]]nPleasee InPPlase reason step by step, and put your fi- nal answer within \boxed[ Assssasistannt with "<lendoftextb>" serving as the EOS token, where "[question]" should be replaced by the specific question.
Evaluation We evaluate our models on standard mathematical reasoning benchmarks commonly used for assessing reasoning capabilities: AIME'24 24, AIME' 25, AMC'23, MATH500 [[2),, Minerva, and OlympiadBench [[ All evaluations are conducted zero-shot setting. For each question, generate 16 independent responses under decoding temperature 7 1.0, and report the average accuracy and the average number tokens per response.
4.3 Main Results
We present the main experimental results below. Additional results are provided Appendix Q
High-entropy tokens drive reinforcement learning for LLM reasoning. Figure E and Table D compare vanilla DAPO which uses all tokens, and our approach that retains only the top 20% high- entropy tokens in the policy gradient loss. Surprisingly. discarding the bottom 80% low-entropy tokens does not degrade reasoning performance and can even lead to improvements across six bench- marks. On the Qwen3-32B 32B base model, this approach delivers gains of 7.71 points on AIME'24 24 and
7

---

63.54
Accuracy (avg@
50
55.83
Mag
Response Length
8k
6k
40
4k
30
DAPO w/ only forking tokens
DAPO only forking tokens
20
DAPO
DAPO
500
1000
1500
500
1000
1500
Gradient Steps
Gradient Steps
(a) A.MI 24 scores trained from Qwen3-32B base.
(b) Response lengths trained from Qwen3-32B base.
50
50.42
Aod
af Are uy unp Applle A
8k
45
45.21
Accuracy (avg@16)
Response Length
40
6k
4k
30
25
DAPO only forking tokens
2k
DAPO only forking tokens
20
DAPO
DAPO
250
500
750
1000
1250
1500
250
500
750
1000
1250
1500
Gradient Steps
Gradient Steps
(c) AIME'24 scores trained from Qwen3-141 base.
(d) Response lengths trained from Qwen3-14DD base.
34.58
35.0
6k
32.5
33.33
Accuracy (ovg@16)
Response Length
30.0
27.5
4k-
25.0
22.5
20.0
DAPO only forking tokens DAPO
2k-
DAPO w/ only forking tokens
DAPO
100
200
300
400
500
600
100
200
300
400
500
600
Gradient Steps
Gradient Steps
(c) AIME'24 scores trained from Qwen3-8B base.
(f) Response lengths trained from Qwen3-8B base.
Figure 5: A comparison of vanilaa DAPO with full tokens and DAPO with top 20% high-entropy (forking) tokens in policy gradient loss was conducted on @wen3-32B, Qwen3-14B. and Qwen3-8B models. (a) & (b) Qwen3-32B: Dropping the bottom 80% low-entropy tokens stabilizes training and improves the AIME'24 score by 7.73. (c) & (d) Qwen3-14B: Similarly, removing 80% low- entropy tokens yields a 5.21 increase in the AIME 24 score. (e) & (f) Qwen3-8B: Retaining only the
top 20% forking tokens maintains performance. Additionally, using only the top 20% high-entropy tokens increases response length across all model sizes.

---

Top 10%
Top 10%
0.35
Top 20%
Top 20%
Top 50%
Top 50%
Accuracy (avg@:16)
35
Bottom 80%
0.30
Bottom 80%
100%
100%
Entropy
0.25
25
0.20
20
0.15
15
0.10
100
200
300
400
500
600
100
200
300
400
500
600
Gradient Steps
Gradient Steps
(a) AIME'24 24 scores trained from Qwea3-8B base.
(b) Overall entropy trained from Qwen3-8B basc.
0.50
50
Top 10%
0.45
Top 20%
100%
Accuracy (avg@16)
0.40
0.35
35
Entropy
0.30
30
0.25
0.20
Top
20
Top 20%
0.15
100%
0.10
250
500
750
1000
1250 1500
250
500
750
1000
1250
1500
Gradient Steps
Gradient Steps
(c) A1ME'24 scores trained from Qwen3-14B base.
(d) Overall entropy trained from Qwen3-14BB base.
70
Top 10%
Top 10%
Top 20%
Top 20%
100%
100%
Accuracy (avg@166)
Entropy
0.5
40
0.4
0.3
30
0.2
20
0.1
0
500
1000
1500
500
1000
1500
Gradient Steps
Gradient Steps
(c) AIME'2 24 scores trained from Qwen3-328 base.
(f) Overall entropy trained from Qwen3-322 base.
Figure 6: Comparison among DAPO using different range of tokens in policy gradient loss. Top x% means only using the v% of the tokens with highest entropy (x = 10, 20, 50), bottom 80% means only using the 80% of the tokens with lowest entropy, and 100% means using all tokens (i.e., vanilla DAPO). Furthermore, "overall entropy" refers to the average entropy over all tokens.
11.04 points AIME'25. Similarly, the Qwen3-14B base model shows improvements of 5.21 points on 2IME'24 and 4.79 points on A1ME'25. For the Qwen3-8B base model, performance re- mains unaffected. These findings suggest that the gains in reasoning ability during RLVR driven

---

primarily by high-entropy tokens, while low-entropy tokens may have little effect on or could even hinder reasoning performance, particularly on the Qwen3-32B and Qwen3-14B base models.
To conduct a deeper analysis, we vary the proportion, denoted as p in Equation (2), for experiments, as shown in Figure a() The results show that the performance of the Qwen3-8B base model remains relatively consistent across different proportions, such as 10%. 20%, and 50%. For the Qwen3-1 and 32B base models, Figure Bic) and (e) reveal that reducing p from 20% to 10% leads to slight drop in performance, while increasing it to 100% results in a notable decline. These observations indicate that within a reasonable range, reasoning performance is largely insensitive to the exact value of p. More importantly, they suggest that focusing on high-entropy tokens, rather than using all tokens, generally preserves performance, and could even offer substantial gains larger models.
Low-entropy tokens contribute minimally to reasoning performance. As illustrated in Fig- ure ((a) and (b), retaining only the bottom 80% of tokens with low entropy during RLVR leads to a substantial decline in performance, even though these tokens account for 80% of the total to- ken count used in training. This finding indicates that low-entropy tokens contribute minimally to enhancing reasoning capabilities, highlighting the greater importance of high-entropy tokens for effective model training.
The effectiveness of high-entropy tokens may lie in their ability to enhance exploration. Our analysis reveals that focusing on a subset of high-entropy tokens, approximately 20% in our experi- ments, strikes an effective balance between exploration and training stability in RLVR. As illustrated in Figure ((.)). (d) and (f), adjusting the ratio p from 20% to either 10%, 50%, 100% leads to per- sistently lower overall entropy starting from the early training phase and continuing up to the point where performance begins converge. Moreover, training with the bottom 80% of low-entropy to- kens results in significantly reduced overall entropy. These findings indicate that retaining a certain proportion of high-entropy tokens may facilitate effective exploration. Tokens outside this range could be less helpful or possibly even detrimental to exploration, particularly during the critical phase before performance convergence. This might explain why, on the Qwen3-32B base model, DAPO using only the top 20% high-entropy tokens outperforms vanilla DAPO, as shown in Fig- ure Bia). However, on the Qwen3-8B base model, probably due to the models lower capacity, the benefits of enhanced exploration appear limited.
5 Conclusion
In this work, we analyze RLVR through a novel perspective of token entropy, providing fresh in- sights into the mechanisms of reasoning in LLMs. Our study of CoT reasoning shows that only small subset of tokens exhibit high entropy and serve as forks in reasoning paths that influence rea- soning directions. Additionally, our analysis of entropy dynamics during RLVR training reveals that the reasoning model largely retains the base model's entropy patterns, with RLVR mainly modifying the entropy of already high-entropy tokens. Building on these findings, which underscore the signif- icance of high-entropy minority tokens, we restrict policy gradient updates in RLVR to the top 20% highest-entropy tokens. This approach achieves performance comparable to, or even surpassing, full-token RLVR training, while exhibiting a strong scaling trend with model size. In contrast, di- recting optimization toward the low-entropy majority results in a significant decline in performance. These findings indicate that RLVR's 's effectiveness stems primarily from optimizing this high-entropy subset, suggesting more focused and efficient strategies for improving LLM reasoning capabilities.
We believe there still room for improvement in our work. First, our experiments could be extended to models beyond the Qwen family. Although we attempted to evaluate our ap- proach on LLaMA models, they struggled to achieve meaningful performance on the AIME bench- marks. Additionally, the scope of our dataset could be expanded to encompass domains beyond mathematics, such as programming or more complex tasks like ARC-AGI [3. EA]. Furthermore, our findings are based on specific experimental settings, and it is possible that the observations and conclusions presented in this paper may not generalize to all RLVR scenarios. For instance, in a different RLVR setting, the effective proportion of 20% observed in our experiments may need to be adjusted to a different value to achieve optimal results. Future directions involve developing new RLVR algorithms to better leverage high-entropy minority tokens and exploring how these in- sights can enhance not only RLVR but also other approaches, such as supervised fine-tuning (SFT), distillation [[3], inference, and multi-modal training [22, 20, 27].
Limitations
10

---

References
[1] Anthropic. Claude 3.7 Sonnet. 01-05-20251.
http://www.aaarroo...cc/oco/o/// 2025. [Accessed
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint (XXi:22502.13923 2025.
[3] Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv: 2505.11831, 2025.
[4] François Chollet. On the measure of intelligence. arXiv preprint arXiv: 1911.01547, 2019.
[5] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quac V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.1761,, 2025.
[6] DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Rumxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu. Yu Wu. Z. Wu, Zhibin Gou, Zhihong Shao. Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjic Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Pampan Huang. Peng Zhang. Qiancheng Wang. Qinyu Chen. Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li. Shuang Zhou. Shaoqing Wu. Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xic, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su. Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li. Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaoleng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong. Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu. Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wa, Zihui Gu, Zijia Zhu Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang, Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 250::2501.12448 2025.
[7] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model align- ment prospect theoretic optimization. arXiv preprint 2rX0::240.001306, 2024.
[8] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025.
[9] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn. Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra. Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua
11

---

Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Paviova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He. Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy. Ramon Calderer, Ri- cardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel. Romain Sauvestre. Ronnie Polidoro. Roshan Sumbaly, Ross Taylor. Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennahasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gu- rurangan, Sydney Borodinsky, Tamar Herman. Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whimey Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Sri- vastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenherg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, An- drew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong. Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel. Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Bcau James, Ben Maurer, Benjamin Leonhardi, Bemiie Huang. Beth Loyd, Beto De Paola, Bhar- gavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Sto- jkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mcjia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiame Chu, Chris Cai. Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erie-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspe- gren. Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Ilias Leontiadis. Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings. Jon Carvill, Jon Shepard, Jonathan Mc- Phie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khan- delwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Mauhias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko. Michelle Restrepo, Mihir Patel, Mik Vyatskov. Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nand- hini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo. Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez. Rafi Ayub, Raghotharn Murthy, Raghu Nayani, Rahul Mi- tra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sar- gun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan. Saurabh Verma, Sciji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang. Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehae, Steve Satterfield, Sudarshan Govin- daprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Mathews. Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria
Albiero,
12

---

Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang. Wes Bouaziz, Will Constable. Xi- aocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang. Ying Zhang, Yossi Adi, Youngjnn Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of modcls. arXiv preprint arXiv:2407. 21787. 2024.
[10] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math; Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.4552 2025.
[11] Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-Ieve bilingual multimodal scientific problems. Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[12] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacoh Steinhardt. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 503.:1500000331,,202 2015.
[14] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang. Dehao Zhang, and Yu Cao. Openrlhf: An easy-to- use, scalable and high-performance rihf framework. arXiv preprint arXiv:2405.1 111,, 2024.
[15] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open- zeaso:er-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint or 250::253..2299 2025.
[16] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang. Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoic. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0797 2024.
[17] Nathan Lambert, Jacoh Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang. Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith. Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training. arXiv preprint M..i::2411.111244 2025.
[18] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, al. Lims can easily learn to reason from demonstra- tions structure, not content, is what matters! arXiv preprint arXiv:2502.077 073744 2025.
[19] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang. and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhence Ilm's 's reasoning capabil- ity. arXiv preprint arXiv:2411. 1.1441111999.... 2024.
[20] Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3491 2023.
[21] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470 2025.
[22] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. In Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 124198-124235. Curran Associates, Inc.. 2024.
[23] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
[24] OpenAL.
Learning to Reason with LLMs.
com/ index/
learning to whtt-ooo-t-oo----h-------------- 11mm/ 2024. [Accessed 01-05-2025).
13

---

[25] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[26] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang. and Daniel Rueckert. Medvim-rl: Incentivizing medical reasoning capability of vision-language models (vIms) via reinforcement learning, 2025.
[27] Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang. Yanbin Wang. al. ART: Anonymous region transformer for variable multi-layer transparent image generation. CVPR, 2025.
[28] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023.
[29] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 2021.
[30] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. arXiv preprint arXiv:1707.06344 2017.
[31] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open lan- guage models. arXiv preprint 20Xi::2402.03300. 2024.
[32] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 19256, 2024.
[33] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint wxXi:2405.08444 2024.
[34] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang. Congcong Wang. Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma. Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei. Sihan Cao, Siying Huang. Tao Jiang. Weihao Gao, Weimin Xiong. Weiran He. Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei, Xianging Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu. Xinyu Zhou, Xuehai Pan, Y. Charles. Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo I. Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wa, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 12599, 2025.
k1.5:
[35]
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. March 2025.
[36] Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning tuning. arXiv preprint arXiv:2502.00m 06533, 2025.
[37] Yiping Wang. Qing Yang, Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai Hc, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang. Simon Shaolci Du, and Yclong Shen. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.200 2504.0 2025.
[38] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforeement learning library. Journal of Mackine Learning Research, 2022.
[39] Ronald Williams.
Simple statistical ing. Machine learning, 8:229-256, 1992.
algorithms for connectionist reinforvement learn-
[40] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen. Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint 2505 2025.
14

---

[41] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mci Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09m: 2025.
[42] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin. Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangje Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Wei-Ying Ma. Ya- fa-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-soures source Ilm reinforcement learning system at scale. arXiv preprint arXiv: 14476, 2025.
[43] Yang Yue. Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang. Does reinforcement learning really incentivize reasoning capacity in ilms beyond the base model? arXiv preprint 2504 2025.
[44] Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. Tian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingx- uan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint 250::2550400111.. 2025.
[45] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models wild. arXiv preprint arXiv:2503. 18892, 2025.
[46] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, arXiv preprint 2335: 2025.
15

---

Appendices
Contents
Infroduction
Analyzing Token Entropy in Chain-of-Thought Reasoning
3
RLVR Preserves and Reinforces Base Model Entropy Patterms
High-Entropy Minority Tokens Drive Effective RLVR
Formulation of RLVR Using Only Policy Gradients of the Highest-Entie Entropy Tokens
Experimental Sctup
Main Results
5 Conclusion
10
Appendices
16
Preliminaries
17
A. Token Entropy Calculation
17
A.2 RLVR Algorithms
17
Related Work
18
More Results
18
Additional Results Figure
18
Scaling Irend of Retaining Only Forking Tokens
19
Generalization Ability Other Domains
19
Unlocking potential of RLVR with only forking tokens
20
Results models other than Owen
20
6
Qualitative Results
21
Diccussions
21
Compute Resources
22
Broader Impacts and LLM Usage Disclosure
22
16

---

A Preliminaries
A.1 Token Entropy Calculation
The token-level generation entropy (referred to as token entropy for brevity) for token defined as
H
E Pt.j log Pt.j where (Pt,1,'.. .PLV) Pp = = q. 0ct) Softmax
(T) T (3)
Here, no denotes an LLM parameterized by 0, is the input query, and Ocr (01, 02, ( represents the previously generated tokens. is the vocabulary size, 2; E R denotes the pre- softmax logits time step t, P: RV is the corresponding probability distribution over the vo- cabulary, and T E IR is the decoding temperature. In off- off-policy settings, sequences are generated by rollout policy α while the training policy is Ta. with * 9. The entropy is still calculated using To. as defined in Equation (3), to measure the uncertainty of the training policy in the given sequence.
"Token entropy" corresponds to the token generation distribution, not a specific token. Throughout our paper, we clarify that the token entropy H; refers to the entropy at index t, which determined by the token generation distribution p; rather than by any specific token o, sampled from p.. For brevity, when discussing the token or sampled from the distribution P,, we describe its associated entropy as H, and refer to He as the token entropy of or. However, if there exists another index l' + such that os, the token entropy of is not necessarily equal to Hy.
A.2 RLVR Algorithms
Proximal Policy Optimization (PPO) PPO [50] is widely adopted policy gradient algorithm in RLVR. To stabilize training, PPO restricts policy updates to remain within a proximal region of the old policy Te, using the following clipped surrogate to maximize the objective:
(f) =EB-D (B.a 9)
min (+) clip(r. - c, 1 c)).))
with re())= no (o q, Oct)
Te (01lg, 0z1)
Here, D is dataset of queries q and corresponding ground-truth answers a, B is a batch sampled
from D, e E R is hyperparameter typically set to 0.2, and A, is the estimated advantage computed using a value network.
Group Relative Policy Optimization (GRPO) Building on the clipped objective in Equation ()). GRPO [GI]] discards the value network by estimating advantages using the average reward within group of sampled responses. Specifically, for each query q and its ground-truth answer a, the rollout policy Toold generates group of responses {o') ) with corresponding outcome rewards (Ri] J= where
the group size. The estimated advantage A then computed as:
x mean (R') (
[ 1.0
if is "ss_equivalennt(,o)) o'),
A
where R
0.0
otherwise.
In addition to this modified advantage estimation, GRPO adds a KL penalty term to the clipped objective in Equation (H)
Dynamic sAmpling Policy Optimization (DAPO) Building on GRPO, DAPO [22 removes the
KL penalty, introduces a clip-higher mechanism, incorporates dynamic sampling, applies a token- level policy gradient loss, and adopts overlong reward shaping, leading to the following maximiza- tion objective, where (() is defined as in Equation (9), and A is computed as in Equation (5))
1o'l
J (0) = (BB-D.((a)) (le)
min (0)
D 0
(l))(r((0), <low, + Enigh) A:)
s.l. 0 0< fo' [ jis_equivalent(a.o]) o') <G. (6)
17

---

DAPO is one of the state-of-the-a RLVR algorithms without a value network. In this work, we use DAPO as the baseline for our RLVR experiments.
B Related Work
Reinforcement I. Learning for LLM Before the advent of reasoning-capan models like OpenAl's ol [24]]. reinforcement learning (RI.) was widely used in reinforcement learning from human feed- back (RLHF) to improve large language models' (LLMs) mnstruction-follviin and alignment with
human preferences [ RLHF methods are broadly categorized into online and offline preference
optimization. Online methods, such as PPO [BO], GRPO [[]]], and REINFORCE [19]] generate re- sponses during training and receive real-time feedback. Offline methods like DPO
[28]]
SimPO [22],
and KTO [ optimize policies using pre-collected preferences, typically from human annotators
or LLMs. While offline methods are more training-efffcient they often underperform compared to online approaches [| Recently, RL with verifiable rewards (RLVR) has emerged
promising approach for enhancing reasoning in LLMs, particularly in domains like mathematics and programming((11. K AII, [Z]]- OpenAl o1 [24]] was the first show that RL. effectively in- centivize reasoning at scale. Building on o1, models such as DeepSeek R1 [B]] QwQ [35]], Kimi k1.5 [34]], and Qwen3 [2II] have aimed to match exceed its performance. DeepSeek R1 stands for showing that strong reasoning can emerge through outcome-based optimization using the online RL algorithm GRPO [[ It also introduced zero RL paradigm, where reasoning abilities are elicited from the base model without conventional RL fine-tuning, Inspired by these results, subse- quent methods such as DAPO [92], VAPO [[E]], SimpleRLZoo [155], and Open- Open-Reaaoner-Zerr [TS] have further explored RL-based reasoning. In this work, we use DAPO as our baseline to investigate key aspects of RL applied to LLMs.
Analysis on Reinforcement Learning with Verifiable Rewards Recently, Reinforcement I Learn- ing with Verifiable Rewards (RLVR) has emerged as a prevalent approach to enhance the reasoning capabilities of large language models (LLMs). Several studies have analyzed the characteristics of RLVR and its related concepts. Gandhi et al. [S] find that the presence of reasoning behaviors, rather than the correctness of answers, is the key factor driving performance improvements in reinforce- ment learning. Similarly, Li et al. [α]] show that the structure of long chains of thought (CoT) is critical to the learning process, while the content of individual reasoning steps has minimal impact. Vassoyan et al. [ل+]] identify "critical tokens" in CoT's, which are decision points where models are prone to errors, and propose encouraging exploration around these tokens by modifying the KI. penalty. Lin et al. [I9]] also identify critical tokens that significantly influence incorrect outcomes and demonstrate that identifying and replacing these tokens can alter model behavior. Our finding that RLVR primarily focuses on forking tokens in reasoning paths may share some common ground with the observations of Gandhi al. [[S and Li et al. [[8], who suggest that RLVR primarily learns the format rather than the content. However, our analysis goes further by identifying the finding at the token level. Moreover, the concept of critical tokens in Vassoyan et al. [66]] and Lin et al. [19]] is closely related to the high-entropy minority tokens we introduce. In contrast to prior work, which judges token importance based on correctness of the output, we propose token entropy as a criterion that may more accurately reflect the underlying mechanisms of LLMs.
C More Results
In this part, we include some additional results as supplement to Sections to
C.1 Additional Results to Figure !!
Additional to Figure R. Figure further illustrates the evolution of entropy percentiles during RLVR training using the Qwen3-14B 14B base model. The figure reveals that as we move from the Oth to the 100th percentile, the range of fluctuations during the whole RLVR training steadily diminishes. These observations suggest that throughout the whole training process, RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens exhibits minor variation and remains relatively stable.
18

---

8.00
100th pereentile
0.80
80th pereentile
60th percentile
0.04
6.00
0.60
Eatropy
4.00
0.40
mM
0.02
2.00
0.20
0.00
0.00
75
50
25
75
x10
x10 -1
2.00
x x
40th percentile
20th percentile
Oth percentile
1.00
1.00
Entropy
1.00
0.50
0.50
0.00
0.00
0.00
-1.00
75
50 Gradient Steps
25
50 Gradient Steps
75
Gradient Steps
Figure 7: The evolution of entropy percentiles during RLVR training. x-th percentile means that x% of the tokens the dataset have entropy values less than or equal this value. In other words, represents the threshold below which the entropy values of the lowest x% of tokens fall, allowing us to track how different segments of the entropy distribution change throughout training.
C.2 Scaling Trend of Retaining Only Forking Tokens
55
60
+7.71
+11.04
Accuracy (avg@16)
Accuracy lavg@166
50
55
45
50
40
+4.79
45
35
DAPO w/ only forking tokens
30
DAPO w/ only forking tokens
35
+41.25
DAPO
+0..8
DAPO
25
14
32
14
Parammeter (B)
#Parameters (B)
(a) Scaling Trend on AIME 2024
(b) Scaling Trend on AIME 2025
Figure 8: Scaling trend of DAPO using only forking tokens (i.e., top 20% of high-entropy tokens) in policy gradient loss.
In this part, present the scaling trend when utilizing only forking tokens, illustrated in Fig- ure 8 On the AIME'24 and AIME'25 benchmarks, we observe that as the model size increases, the performance gain over vanilla DAPO becomes increasingly significant. This suggests promising conclusion: focusing solely on forking tokens the policy gradient loss could offer greater advantages in larger reasoning models.
C.3 Generalization Ability to Other Domains
As outlined in Section 1.2, we used the DAPO-Math-177 dataset, which primarily consists of math- ematical data, for RLVR experiments. Here, we test whether DAPO, when trained on math
19

---

dataset and using only small fraction of high-entropy tokens in the policy gradient loss, can still surpass vanilla DAPO on ontoof-distribution sets, such as LiveCodeBench [T
Top 10%
Top 20%
27
100%
Accuracy (avg@16)
26
25
24
200 400
600
800
1000
1200
1400
Gradlent Steps
Figure 9: Comparison among DAPO using different range tokens in policy gradient loss trained from the Qwen3-32B base model on the cut-of-distribution LiveCodeBench Benchmark [[6] (v5, Aug. 2024 to Feb. 2025). Top x% means only using the x% of the tokens with highest entropy (x 10, 20), and 100% means using all tokens (i.e. vanilla DAPO). Due to the high variance of the accuracy curves, we smooth the curves using window smoothing with a window size of 10.
The results comparing DAPO with only top 10% 20% tokens with highest entropy vanilla DAPO (which uses 100% tokens) on the Qwen3-32B base are illustrated in Figure E. using the same setup described in Section TV 10% or 20% tokens with highest entropy, DAPO still significantly outperforms vanilla DAPO on the out-of-distribution
From these results, we observe that even when retaining only top
dataset LiveCodeBench. This finding suggests that high-entropy tokens may be associated with the generalization capabilities of reasoning models. Retaining only small subset of tokens with the highest entropy could potentially enhance the generalization ability of reasoning models.
C.4 Unlocking more potential of RLVR with only forking tokens
we set maximum response length of 20480. As shown in Figure IIA we increased the maximum response length for DAPO w/ only forking tokens (depicted in Figure () and (b))trained from the Qwen3-32B base modelto 29696. This adjustment resulted in an improvement of the already SoTA performance on AIME'24, increasing from 63.54 to 68. 12. These findings suggest that the full potential of our approach may not yet be realized, and with longer context length or potentially more challenging training data, even greater performance gains could be achieved.
In the experiments described in Section 1
C.5 Results models other than Qwen
We compare DAPO using only forking tokens (i.e., the top 20% tokens with the highest entropy) against vanilla DAPO models other than the Qwen series, specifically the Llama-3.1-8h model. When DAPO is applied to the Llama-3. 1-88 base model, we observe that achieves very low accuracy (approximately 1%) on the training dataset (i.e., DAPO-MATH-I7K erates responses with repetitive words early in the RL training process. To address this, we use the Qwen3-32B model [[I]] as teacher generate responses for DAPO-MATH-
[[2]) and often gen-
queries. From the generated queries, we randomly sample 10,000 with correct answers to serve as cold-start data and perform supervised fine-tuning (SFT) on the Llama-3.1-8B base model [Н]] The remaining 7,398 queries are reserved for RL after the cold-start phase. The AIME'24 24 score and response length during RL training are plotted in Figure III The results indicate that DAPO with only forking to- kens still surpasses vanilla DAPO. while also producing longer responses on average. However, given the relatively low performance of both configurations on AIME'24, we believe the results on Llama-3.1-88 3.B are less convincing compared to those observed on the Qwen3 models.
20

---

70
16k
68.12
14k
60
Accuracy (avg@166)
Response Length
12k
50
10k
8k
40
6k
30
4k
DAPO w/ only forking tokens longer context
DAPO w/ only forking tokens
20
2k
DAPO only forking takens
0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Gradient Steps
Gradient Steps
(a) AAMA' scores
(b) Response length
Figure 10: By extending the maximum response length from 20,480 to 29,696 and continuing train- ing from the SoTA 32B model shown in Figure (a and (b), the AIME' 24 scores improve further from 63.54 to 68. 12, alongside notable increase in response length.
20000
10.21
DAPO only forking tokens
10
18000
Accuracy (avg@16)
7.92
Response Length
16000
14000
6-
12000
10000
DAPO w/ only forking tokens DAPO
8000
2
100
200
300
400
500
600
100
200
300
400
500
600
Gradient Steps
Gradient Steps
(a) AIME 24 scores trained from Llama-3. 1-88BB after cold-start.
(b) Response length trained from Llama-3, 1-8B after cold-start.
Figure 11: Comparison of DAPO using only forking tokens and vanilla DAPO, both trained from Llama-3.1-80 after cold-start.
C.6 Qualitative Results
Visualization of token entropy of full CoT response We visualize the token entropy of long, randomly generated Chain of Thought (CoT)) response by Qwen3-8B [А]]] in Figure I2 to Figure I From the visualization, we observe that high-entropy tokens are relatively sparse among all tokens. These high-entropy tokens often act as branching points in reasoning paths, influencing the direction of reasoning. This observation further complements the entropy patterns discussed in Section Z.
D Discussions
High-entropy minority tokens (i.e,, forking tokens) could play key role in ex- plaining why RL generalizes while SFT memorizes. [5S] demonstrated empirically that RL, par- ticularly with outcome- based rewards, exhibits strong generalization to unseen, rule-based tasks, whereas supervised fine-tuning (SFT) is prone to memorizing training data and struggles with gen- eralization outside the training distribution. We hypothesize that one critical factor underlying the differing generalization capabilities of RI. and SFT may related entropy in forking tokens. Our experiments (e.g., Figure Ø and Figure B suggest that RL tends preserve even increase the
Discussion 1:
21

---

entropy of forking tokens, maintaining the flexibility of reasoning paths. In contrast, SFT pushes output logits towards one-hot distributions, leading to reduced entropy forking tokens and, con- sequently, a loss of reasoning path flexibility. This flexibility may be a crucial determinant of reasoning model's ability to generalize effectively to unseen tasks.
Discussion 2: Unlike traditional RL, LLM reasoning integrates prior knowledge and must produce readable output. Consequently, LLM CoTss contain a mix low-entropy majority tokens and high-entrops minority tokens, whereas traditional RL can assume uniform action entropy throughout a trajectory. As shown in Figure 2ia,, most LLM CoT tokens have low entropy, with only small fraction exhibiting high entropy. In contrast. traditional RL typically formulates each action distribution as Gaussian with a predefined standard deviation [EO, 29 BS) resulting in uniform entropy across actions. We attribute this distinct entropy pattern in LLM CoTs to their pretraining on large-scale prior knowledge and the need for language fluency. This forces most tokens align with memorized linguistic structures, yielding low entropy. Only small set of tokens that are inherently uncertain the pretraining corpus allows for exploration, and thus exhibits high entropy. This deduction is consistent with our results in Table MI.
Discussion 3: In RLVR, entropy bonus may be suboptimal, as it increases the entropy of low- entropy majority tokens. In contrast, clip-higher effectively promotes entropy in high-entropy minority tokens. In RL, entropy bonus is commonly added to the training loss to encourage explo- ration by increasing the entropy of actionsa well-established practice in traditional tasks [600, 59, 25)) and recently applied to LLM reasoning [22, D4]]. However, as discussed above, unlike typical RL trajectories, LLM CoT's display distinct entropy patterns. Increasing entropy across all tokens can degrade performance by disrupting the low-entropy majority, while selectively increasing the en- tropy of high-entropy minority tokens improves performance (Figure E). Thus, uniformly applied entropy bonuses are suboptimal for CoT reasoning. Instead, clip-higher [22]] which moderately raises Ehigh in Equation (60, better targets high-entropy tokens. Empirically, we observe that tokens with high importance ratios (f)) (as defined in Equation (E)) tend to have higher entropy. By in- cluding more of these tokens in training, clip-higher increases overall entropy without significantly affecting low-entropy tokens, as supported by [12]] and illustrated in Figure 7
E
Compute Resources
We utilize NVIDIA A100 80G GPUs as our compute resources for conducting experiments. Each experiment involving 8B, 14B, and 32B models is executed on 8 nodes, with each node equipped with 8 GPUs. Our training codebase is adapted from verl [62]] For the 8B model, the initial process- ing time for each mini-batch is approximately 0.5 hours, which increases around 1.5 hours the response length expands during training. Similarly, for the 14B model, the runtime per mini-batch batch starts about 0.8 hours and grows to approximately 2.2 hours the response length increases. In the case of the 32B model, processing each mini-batch initially takes about 1 hour, but this extends to roughly 3 hours the response length continues grow during training.
F Broader Impacts and LLM Usage Disclosure
Broader impacts Our study analyzes entropy patterns in CoTs and enhances LLM reasoning by retaining only the top 20% of high-entropy tokens in policy gradient loss during RLVR training. While this improves reasoning abilities, it may also increase the risk of hallucinations or unsafe outputs. We urge developers to responsibly apply our findings to build safe reasoning models and advise users to exercise caution when using them.
Declaration of L.LM Usage L.MSs are the primary subject of research in this paper, and all of our experiments are conducted on L.MMs. Additionally, we also used L.MSs for polishing the text during the writing process.
22

---

Okay tacide problom stop by stop problem about finding
list of positive integers with specific properties the sum the uniquue
and the median positive Intager we need to find the sum of the squares all the Items in the list Inin First
recall what each Term mears The mode is the number that appears most trequently
Since specified appears more Limes then any other number in the list The median b the middle
unique made
ordered smallest largest number elements the median the middle one even the average
list
the two middle numbers However the problem the median positive Integer
Integer appear
So
which implies that the list must have en odd number of elements because it
average integer Therefore the list must have an odd number elements say $
median
numbers could
where odd Iniu Let me note dawn the points Jynin Sum of appcars more
any
number An Median is positive integer not in the list Antn Our goal to
find squares Since Uhe median is not in the list and mediann en integer
let
the
demote the number of elements Where term the ordered list Since the median not In the list that term must appear list
which odd denote
Integer Then the median the
not be In the list Let me think For example the list Is median which
the
median which
the list
how can median bo
Maybo median
number that Is between two numbers
in the
list
Wait
but
the
median
Is
0,000
1.316
2.632
3.948
5.264
6.580
Token Entrapy
Figure Visualization of token entropy (part 1).
23

---

specttic value ordered
that element is not in the list that seerro impossible Wait but maybe the
list duplicates Wait hold
excmple suppose list
The median Is which Is In the list the list
median which the sist So how the median not
case the median present Tinin maybe the list
has
multiple irstances of the median But Lhen the median would be present Wait perhaps madian
element
Walt that Impossible
Decause the median the middle element of the ordered list So must an element of the list Therefore how the mediaan not be the list tnyn Wait maybe problem saying
not present
but that the median positive integer that does not oppeat in the list Wait that exactly what says median
en integer 50 must be odd as discussed before and it does not pear
the list But the median the middle element of the list then the median contradiction unless the list empty But the problem says it 1s list of positive integers
maybe mistaake The median Lhe middle value when Uhe list is ordered for example the
again
Iist Is
the median If the list Is
the
can
definition median not be the list Vnnn Wait unless the median is not element ? But
misunderstanding Maybe
the problem Is referring to the median value not the median element Far example the has number elements Uhen Uhe median average the two middle numbers prablem says median positive Integer So the list has even number of elements the average of
0.000
1.316
2.6.32
3.948
5.264
6.580
Entropy
Figure Visualization of token entropy (part 2).
24

---

up the list Imin integer they
the median on Intagen gar but not necessarily an element
perhaps has mber elements and the median
average the middle numbers which integer not present the
That would make Therefore the list
have even number of elements
Therefore my provious assumption
was wrong Vnin me
aminne that The problem says the median positive integer that does not
appear the
alement
then
list must have an even number elements 50 that the median the average
Two numbers Let me check with example Suppose the list
Integer
1. the mediaan is M which an integer and is in the list If median
list
which is an integer However is not the list Therefore this case the median which number elements median can be an integer not present in the list Therefore
list is
median Is
yes
So my our list must have
Therefore
number elements Is oven Let me denote where pasitive Integer
Then median the the and elements For the
an Intager
elements even
Therefore they are both even both odd Vntn So to recap :tnin The list has
elements say
The unique mode Is 50 occurs more times than any other number Jnnnn The
median integer not list. Antn Our find Uhe
.nin Therefore
figure out possible
number elements and the composition of the list
Jnin First
let
consider
0.000
2.632
3.948
5.261
6.580
Taken Entropy
Figure 14: Visualization token entropy (part 3).
25

---

M 2
which
is not
Uhe list Wail
this
is
the
samee
as
the
list 5
but
with
and
km/n But sum 1
2
0 So need
possible
pairs
as before For example
gives list
median is
Walt
but
can be So only those pairs
Vnin yes the only vellid
list Is 7 Inin Therafare sum squares Is
sure
list with different numbers but same properties Anin Sup pose the list has three and another
another arrangement where the median Is not the list For example suppose the list which not in the with and the median which gives only found
leads
knin Alternatively ppose the two not consecutive Wait in sorted
then there are numbers between them For example 5 consecutive Inin But
list they
that
the median
Which not in the list Inin betore
Alternatively Anti ence think this Is the only list vnin Therafore
sorted
other possibilities
thorough other tiss where the mode is Inin Sup pose the list is
with and other numbers but
other numbers
same Wait but then the mode would and the other number making two modes
Which is Invalid Win For example
The sum 5 Not
sum
appear once So no mode Therefore Involld Anio
9. sum Is D... As
before
median
knin
and
different
number
0.000
1.316
2.632
3.948
5.264
6,580
Taken Entropy
Figure 15: Visualization of token entropy (Part 4). For brevity, we omit the CoT following Figure ?
and preceding Figure IT.
26

---

Atternatively
6 9 The median Is ( with
1 2
and are samee then
1. with
Then list
Not Intager Viln But the
appearing
unique
invalid
Vinn
Theretore
only the list
valid Inin Thus the sum ot
aquates
plus
plus
is
3 Vnin Therefore the answer 2 236 In/n Final Answer *In baxed
3 Min
</think>
solve
problem
determine the of the
squares
Items
positive integers that satisfles
fol following
conditions
<> Sum of the list 0 **in The unique mode: appear # Step Determine the Number of Elements in the List trin
Anin
wnyn
positive Integer
itself
*Myytn
--iin
Sinco the median positive integer not in the list this implies that the list must have an even number of elements tnin For odd number elements the median Therefore cannot be integer not the list Thus the list must have an even
denate number elemants
the middle element the sorted which
the validity or the conditions Anin .--iin ##2 Stop Consider $ $inin
the has elements Let the elements sorted
median :trinn In Median frac ] Pi ss
are told that the ** median positive integer not in the list and that unique mode more often than any other number Since only have elements the only possbiliity appears
For
unique
appeaa
and numbers appear
Thus the list must be the form
5. where and
$.
Jnin We also
require Antk
3
Rightarrow
1.316
5.264
Token Entropy
Figure 16: Visualization of token entropy (part 5).
27

---

$1n trac
9
15
Integer
hence
Is odd
Xi
$1 frac
) is
not
in the list
Inim
Let
test
integer
and ne
5,
ne
Antn
$:
List
Median frac
not integer Yin
List is
15 Median
Integor
Uhe list **Vrin This list salisfies all the conditions Annnm
Sum *+:
mode
Median
which Is not the Inin Thus the list
15 valid Annn
...yii #o Step Compute the Sum ares Inin Now calculate the of the
squares the elements
Vn
81 236 In $$ Inin --hn/
# Final Answer Inim $$ In boxed Yin $5
0.000
1.316
2.632
3.948
5.264
6.580
Entropy
Figure Visualization of token entropy (part 6).
28

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] " generally preferable to "[No] is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] question, in the justification please point to the section(s) where related material for the question can be found.
used"). In
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: all backed up in the experiments section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
29

---

Justification: in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: no theory assumptions and proofs in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
Answer: [Yes]
Justification: the experiment section.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-soures dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome deseribe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We still cleaning up our codes and will open-source later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
/ /maps..c/public/guddee Sto.edSeeee/eeSeSbiiica. wwstmisssoonnii ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
31

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to the big resources for experiemnts, we cannot repeat for several times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
32

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: we conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We haven't release data models now.
Guidelines:
33

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode, com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re- e-packagedd both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
34

---

Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (tte::///aarrrreccc//oooeeereeee for what should or should not be described.
35


Reviews:
[{"review_id": "VTQ9VQu3IF", "reviewer": "Reviewer_2VsV", "summary": "This paper investigates the underlying mechanisms of Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy patterns in reasoning models. The authors analyze token entropy distributions in LLM reasoning processes and identify that approximately 20% of tokens exhibit high entropy and function as \"forking tokens\" that determine reasoning directions. They propose restricting policy gradient updates to only these high-entropy tokens during RLVR training. Experiments on Qwen3-8B and Qwen3-32B models across mathematical reasoning benchmarks show that this selective approach can match or exceed full-gradient RLVR performance while using only 20% of tokens.", "questions": "- Simple experiments on GRPO or PPO may help clarify whether the proposed method can generalized to other RL algorithms. (If not, what is the reason?)\n- The response length is also increasing with the improved accuracy. How does this phenomenon related to the training of High-Entropy Minority Tokens?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Novel analytical perspective: The token-level entropy analysis of RLVR mechanisms provides a fresh and interpretable lens for understanding how reinforcement learning improves reasoning capabilities. The identification of \"forking tokens\" as critical decision points is intuitive and well-motivated.\n- Solid empirical validation: The paper provides multiple lines of evidence supporting the importance of high-entropy tokens.\n- Practical significance and Simplicity: The finding that RLVR effectiveness can be improved while reducing computational requirements has practical implications for both performance and training efficiency. \n- Comprehensive experimental design: The evaluation spans multiple mathematical reasoning benchmarks and includes appropriate ablations varying token proportions.\n\nWeaknesses\n- Lack of experiments on GRPO and PPO: The experiments are mainly conducted on DAPO. It will be helpful to also include experiments on GRPO or PPO to see if the proposed approach generalize to other RL algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions, which have contributed to improving our manuscript.\n\n---\n\n> Q1: Simple experiments on GRPO or PPO may help clarify whether the proposed method can generalized to other RL algorithms. (If not, what is the reason?)\n\nA1: Thank you for the insightful suggestion. We selected DAPO over GRPO as the baseline because DAPO is an enhanced version of GRPO, incorporating four techniques (i.e., clip-higher, dynamic sampling, token-level policy gradient loss, and overlong reward shaping) that strengthen training stability in RL and substantially improve model performance. As a result, DAPO serves as a more robust and informative baseline for comparison.\n\nAs for PPO, it relies on a separate critic model, which increases VRAM usage and adds complexity to the training process. For these reasons, many recent state-of-the-art reasoning models [1–4] have adopted critic-free RL methods such as GRPO and DAPO.\n\nConsidering both of these factors, we chose DAPO over GRPO or PPO as our baseline.\n\nHowever, we agree that including an additional baseline can provide more insight into whether the proposed method generalizes to other RL algorithms. To this end, we removed the clip-higher, dynamic sampling, and token-level policy gradient loss techniques from DAPO to revert to the GRPO algorithm. We then compared training using only the top 20% high-entropy tokens against training with all tokens, using the Qwen3-14B base model. The results, shown in the table below, indicate that even with the GRPO algorithm, training on the top 20% high-entropy tokens still significantly outperforms training on the full set of tokens.\n\n| # of Gradient Steps     | AIME'24 (w/ top 20% high-entropy tokens) | AIME'24 (w/ full tokens) | AIME'25 (w/ top 20% high-entropy tokens) | AIME'25 (w/ full tokens) |\n|----------|------------------------------------------|---------------------------|------------------------------------------|---------------------------|\n| step 160 | 23.96                                    | 23.54                     | 25.83                                    | 31.25                     |\n| step 320 | 26.25                                    | 14.58                     | 29.58                                    | 15.83                     |\n| step 480 | 26.46                                    | 19.38                     | 30.83                                    | 22.50                     |\n| step 640 | 28.33                                    | 18.54                     | 31.67                                    | 12.08                     |\n| step 720 | 30.21                                    | 16.67                     | 32.92                                    | 18.75                     |\n| step 800 | 27.50                                    | 19.79                     | 32.92                                    | 18.75                     |\n| step 960 | 27.50                                    | 17.50                     | 33.33                                    | 20.00                     |\n\n[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[2] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025).\n\n[3] Chen, Aili, et al. \"MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention.\" arXiv preprint arXiv:2506.13585 (2025).\n\n[4] Rastogi, Abhinav, et al. \"Magistral.\" arXiv preprint arXiv:2506.10910 (2025).\n\n-------------\n\n> Q2: The response length is also increasing with the improved accuracy. How does this phenomenon related to the training of High-Entropy Minority Tokens?\n\nA2: Thank you for your insightful observation. We have found through experiments that focusing training only on high-entropy minority tokens—specifically the forking tokens shown in Fig. 2(b), which often act as logical connectors within and between sentences—can make the chain-of-thought (CoT) logic more complex, especially with more self-verifications and logical turns. As a result, response length tends to grow alongside this more complex logic.\n\nDue to the character limits of the rebuttal response and long length of CoT, we cannot provide comparison of CoTs of training with all tokens and with only top 20% high-entropy tokens. We will provide a comparision of CoTs of different methods in the next version of our paper.\n\nTo further support the qualitative observation above, we include quantitative results. After convergence in RLVR training, we calculate the average number of words strongly associated with self-verification and logical reasoning per CoT. As shown in the table below, these words appear significantly more frequently when training is restricted to high-entropy tokens compared to using all tokens. This reinforces our qualitative finding: training with only high-entropy tokens fosters more complex logical reasoning, with a particular emphasis on self-verification.\n\n| Training Method                         | Number of \"wait\" per CoT | Number of \"but\" per CoT | Number of \"however\" per CoT | Number of \"alternatively\" per CoT |\n|----------------------------------------|---------------------------|--------------------------|------------------------------|------------------------------------|\n| Training w/ top 20% high-entropy tokens| 17.94                     | 29.13                    | 1.84                         | 3.68                               |\n| Training w/ full tokens                | 11.39                     | 11.67                    | 0.57                         | 1.49                               |\n\n---\n\nWe sincerely appreciate your expertise and careful evaluation. **If you find our responses satisfactory, we kindly ask you to consider adjusting the score accordingly.** We welcome any further discussion and are happy to address any additional questions or concerns you may have."}, {"review_id": "Dl5jVryxBO", "reviewer": "Reviewer_g3nW", "summary": "The work explores the mechanism behind  RLVR by zooming from sequence-level to token-level entropy in CoT traces of LLMs. An empirical analysis reveals that only about 20 % of tokens carry high entropy, and these “forking tokens” act as pivotal decision points that steer reasoning paths. Controlled decoding experiments show that selectively increasing temperature at these tokens boosts accuracy, whereas manipulating low-entropy tokens has little effect. The results suggest that RLVR’s efficacy stems almost entirely from optimizing this high-entropy minority, pointing toward more targeted and efficient training strategies for future reasoning-oriented LLMs.", "questions": "As discussed in weakness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strenghts\n1. This work shifts the discussion from sequence level entropy to token level patterns, revealing an actionable 20/80 split that had not been quantified before.\n2. The authors conduct temperature manipulation and gradient masking studies to convincingly identify the contribution of forking tokens and reduce confounds.\n3. The proposed method achieves equal-or-better accuracy with 80 % fewer gradient updates, promising for large-scale RLVR where compute is scarce.\n4. The authors conduct comprehensive studies, including varying the kept token ratio multiple base model sizes, and six public math benchmarks, to support robustness claims.\n\n# Weaknesses\n\n1. All experiments target mathematical reasoning; it remains unclear whether the 20 % rule or performance gains hold for code, commonsense, or general reasoning tasks.\n2. Results are restricted to Qwen3 variants; LLaMA or Mixtral models are not included. It would strengthen this work by demonstrating the generalization of this idea with different model families.\n3. No error bars or significance tests are reported, so improvements could partly stem from stochasticity in RL runs.\n4. While the idea is intuitively appealing, the work stops short of formalizing why high-entropy tokens dominate the policy gradient signal. Discussing with more insightful analysis from optimization or learning dynamics may further boost this work.", "was_revised": true, "final_justification": "I have checked the reviewers-author discussion and the  response to my questions. I believe most of my concerns have been resolved. I will keep my original rating.", "rebuttal": "Thank you for your careful review and constructive suggestions, which have helped us improve our manuscript.\n\n---\n\n> Q1: All experiments target mathematical reasoning; it remains unclear whether the 20 % rule or performance gains hold for code, commonsense, or general reasoning tasks.\n\nA1: Thank you for your valuable feedback. Based on your suggestion, we compared DAPO on the Qwen3-14B base model using either all tokens or only the top 20% high-entropy tokens on a code dataset. For evaluation, we followed the methodology used in Qwen3 [1] and assessed model performance on code benchmarks including Codeforces Ratings from CodeElo [2] and LiveCodeBench (v5, 2024.10–2025.02) [3].\n\nEach method was trained for a total of 960 gradient steps. The performance comparison is presented in the table below. We observed that training with only the top 20% high-entropy tokens leads to a significant improvement on Codeforces, while achieving performance comparable to using all tokens on LiveCodeBench. These findings suggest that training with just the top 20% high-entropy tokens can deliver better or at least equivalent performance across different benchmarks compared to using full-token training.\n\n| Trained from        | Methods                          | Maximum Score on Codeforces | Maximum Score on LiveCodeBench v5 |\n|-------------------|----------------------------------|------------------------------|------------------------------------|\n| Qwen3–14B–Base     | w/ top 20% high-entropy tokens   | 1387                         | 30.36                              |\n| Qwen3–14B–Base     | w/ full tokens                   | 1259                         | 30.03                              |\n\n[1] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025).\n\n[2] Quan, Shanghaoran, et al. \"Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.\" arXiv preprint arXiv:2501.01257 (2025).\n\n[3] Jain, Naman, et al. \"Livecodebench: Holistic and contamination free evaluation of large language models for code.\" arXiv preprint arXiv:2403.07974 (2024).\n\n---\n\n> Q2: Results are restricted to Qwen3 variants; LLaMA or Mixtral models are not included. It would strengthen this work by demonstrating the generalization of this idea with different model families.\n\nA2: Thank you for your helpful suggestion. We have already included experiments on LLaMA-3.1-8B in Appendix A.4, where training with only the top 20% high-entropy tokens also demonstrates an advantage over training with all tokens. For your convenience, we summarize the performance comparison in the table below. We will also make this experiment more prominent in the next version of our paper.\n\n| Trained from           | Methods                          | Maximum Score on AIME'24 |\n|----------------------|----------------------------------|---------------------------|\n| Llama–3.1–8B–base     | w/ top 20% high-entropy tokens   | 10.21                     |\n| Llama–3.1–8B–base     | w/ full tokens                   | 7.92                      |\n\nFurthermore, to verify that training with the top 20% high-entropy tokens also provides an advantage on other types of LLMs, we conducted additional comparison experiments using the Qwen2.5-7B-base and Qwen2.5-7B-instruct models. The results, shown in the table below, indicate that training with only the top 20% high-entropy tokens consistently outperforms training with all tokens.\n\n| Trained from             | Methods                          | Maximum Score on AIME'24 | Maximum Score on AIME'25 |\n|---|---|---|-------|\n| Qwen2.5–7B–Base        | w/ top 20% high-entropy tokens   | 20.21                     | 16.25                     |\n| Qwen2.5–7B–Base        | w/ full tokens                   | 19.79                     | 15.42                     |\n| Qwen2.5–7B–Instruct    | w/ top 20% high-entropy tokens   | 18.33                     | 17.50                     |\n| Qwen2.5–7B–Instruct    | w/ full tokens                   | 17.29                     | 16.25                     |\n\n---\n\n> Q3: No error bars or significance tests are reported, so improvements could partly stem from stochasticity in RL runs.\n\nA3: Thank you for your feedback. As shown in Appendix B, each experiment requires substantial computational resources, making it difficult to run multiple trials with different random seeds. For instance, a full RLVR run on the Qwen3-32B model demands 64 A100 80GB GPUs for approximately 220 hours. Given the computational constraint, it is standard practice to perform only a single run per setting, as noted in references [1–5] mentioned by Reviewer q52N.\n\nHowever, we can quantify variability in an alternative way: by computing error bars based on the test scores from 11 mini-batches surrounding the peak score (five before and five after). We calculate the average and standard deviation of these scores, and the results are as follows. They indicate that, for the 32B and 14B models, training with only the top 20% of high-entropy tokens yields a statistically significant improvement over using all tokens. For the 8B model, however, the improvement is not statistically significant. This observation aligns with the claim made in our paper.\n\n| Model                                      | avg score ± std     |\n|---|----|\n| 32B w/ top 20% high-entropy tokens         | 59.00 ± 1.07         |\n| 32B w/ full tokens                         | 53.42 ± 1.17         |\n| 14B w/ top 20% high-entropy tokens         | 49.33 ± 1.79         |\n| 14B w/ full tokens                         | 42.17 ± 1.74         |\n| 8B w/ top 20% high-entropy tokens          | 32.31 ± 1.58         |\n| 8B w/ full tokens                          | 31.28 ± 1.23         |\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Yu et al. 2025.\n\n[2] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. Cui et al. 2025.\n\n[3] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025.\n\n[4] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025.\n\n[5] Reinforcement Learning for Reasoning in Large Language Models with One Training Example. Wang et al. 2025.\n\n---\n\n> Q4: While the idea is intuitively appealing, the work stops short of formalizing why high-entropy tokens dominate the policy gradient signal. Discussing with more insightful analysis from optimization or learning dynamics may further boost this work.\n\nA4: Thank you for the insightful comment. We clarify here why, under common policy parameterizations, high-entropy tokens tend to yield larger policy gradients, thereby more strongly influencing the update.\n\nRecall the (unclipped) policy gradient term for a single sample:\n\n$$\ng_t = A_t \\cdot r_\\theta(t) \\cdot \\nabla_\\theta \\log \\pi_\\theta(a_t \\mid s_t),\n\\quad \\text{where } r_\\theta(t) = \\frac{\\pi_\\theta(a_t \\mid s_t)}{\\pi_{\\text{old}}(a_t \\mid s_t)}.\n$$\n\nWhile all three terms affect the gradient magnitude, in practice:\n\n(1) The advantage estimate $A_t$ is treated as fixed during optimization (precomputed from rollout), and is often normalized (i.e., normalized within each group in DAPO or GRPO) per update.  \n(2) The importance ratio $r_\\theta(t)$ affects the gradient only multiplicatively, and in DAPO is further clipped to $[1 - \\epsilon_{low}, 1 + \\epsilon_{high}]$, limiting its variability.  \n(3) The score function $\\nabla_\\theta \\log \\pi_\\theta(a_t \\mid s_t)$ is the only term directly affected by the policy entropy and dominates the direction and magnitude of the update.\n\nSpecifically, under a softmax policy for discrete actions, assume the last network layer outputs a logit vector $z\\in\\mathbb R^{|A|}$.  \nThe soft-max policy is  \n\n$$\np_i =\\pi_{\\theta}(a=i | s) = \\frac{\\exp(z_i)}{\\sum_{j}\\exp(z_j)}, \\quad i=1,2,\\cdots, |A|.\n$$\n\nFor a sampled action $a$ the log-probability is  \n\n$$\n\\log \\pi_\\theta(a\\mid s)= z_{a}-\\log\\bigl(\\textstyle\\sum_j\\exp(z_j)\\bigr).\n$$\n\nGradient w.r.t. the logits.  Denote $g_k=\\partial\\log\\pi/\\partial z_k$:\n\n$$\ng_k = \\delta_{ka}-p_k,\n\\qquad k=1,\\dots,|A|.\n$$\n\nHence for the component $k=a$\n\n$$\ng_{a}=1-p_{a},\n$$\n\nwhile for every $k\\neq a$\n\n$$\ng_k=-p_k.\n$$\n\nMagnitude for one sample.  The (squared) Euclidean norm of this sample gradient is  \n\n$$\n\\lVert \\nabla_{z}\\log\\pi_\\theta(a\\mid s)\\rVert_2^2\n=\\sum_{k}(g_k)^2\n=(1-p_a)^2+\\sum_{k\\neq a}p_k^{\\,2}.\n$$\n\nWhat happens at different entropy levels?\n\n- Near-deterministic policy, e.g. $p_a\\approx1$: $\\quad  1-p_a\\approx0$ and all other $p_k\\approx0$  $\\Rightarrow$  gradient nearly vanishes.\n\n- High-entropy (near-uniform) policy, $p_i\\approx 1/|A|$: $\\quad$  All terms stay away from 0  $\\Rightarrow$  gradient norm reaches its maximum.\n\nHence a soft-max policy automatically produces larger raw policy gradients when its output distribution is high-entropy and smaller gradients when it becomes low-entropy.\n\nAlthough our primary interest lies in $\\nabla_\\theta \\log \\pi_\\theta(a \\mid s)$, our analysis focuses on $\\nabla_z \\log \\pi_\\theta(a \\mid s)$, justified by the chain rule:\n\n$$\n\\nabla_\\theta \\log \\pi_\\theta(a \\mid s) = \\nabla_z \\log \\pi_\\theta(a \\mid s) \\cdot \\nabla_\\theta z(s; \\theta).\n$$\n\nThis expression highlights that the structure and magnitude of the score function in parameter space are shaped by the gradient with respect to logits. While the Jacobian $\\nabla_\\theta z$ is architecture-dependent, $\\nabla_z \\log \\pi$ reflects how the output distribution’s shape — especially its entropy — affects the gradient signal. Under mild assumptions on the Jacobian norm, $\\nabla_\\theta \\log \\pi$ and $\\nabla_z \\log \\pi$ remain aligned through backpropagation, making the latter a valid and informative proxy for studying entropy's influence on policy gradient magnitude.\n\n---\n\nThank you again for your expertise and thoughtful review. **If you find our response helpful, please consider raising the score.** We're happy to continue the discussion and address any further questions."}, {"review_id": "C3jIziihb8", "reviewer": "Reviewer_q52N", "summary": "This paper investigates the mechanisms underlying Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing LLM reasoning capabilities through the lens of token-level entropy analysis. The authors demonstrate that in Chain-of-Thought reasoning, only approximately 20% of tokens exhibit high entropy and serve as critical \"forking points\" that determine reasoning directions, including some common tokens indicating test-time scaling like \"wait\". They propose a modified RLVR approach that restricts policy gradient updates to only these high-entropy tokens, achieving comparable or superior performance to full-gradient methods while using only 20% of tokens. The key finding is that low-entropy tokens contribute minimally to reasoning improvements, with the high-entropy minority tokens being the primary drivers of RLVR effectiveness.", "questions": "How did you determine that the high-entropy tokens are forking tokens? In the paper, this is a very strong causal claim (\"We refer to high-entropy tokens as forking tokens, which function like forks in the road by determining which path to follow for continuation\"), so more convincing and comprehensive evidence should be provided.", "limitations": "The experimental setups. See weakness.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- Novel analysis perspective. Although there have been recent analyses of entropy for LLM reasoning, they have mostly focused on sequence-level [1], whereas this work focuses on the token-level. As token-level entropy is adopted as the proxy to gauge entropy, the angle in this work may be more suitable.\n- Strong empirical validation. To investigate the entropy issue, this paper starts with an inspection of the distribution of token entropy. To further validate the hypothesis on the existence of high/low-entropy tokens, a decoding experiment is performed, showing that high-entropy tokens are pruned to high entropy while low-entropy tokens require lower temperature. **My only complaint is that the conclusion that high-entropy tokens are \"fork tokens\" is mainly based on qualitative analysis through word clouds.** As this terminology has repeatedly occurred and seems to be an important concept introduced in this paper, I expect there is a more rigorous derivation and evidence.\n- Useful insights in practice. This work provides valuable insights into preserving the exploration ability in RL, following previous practices [2]. Together with other recent works, it calls for attention to entropy and exploration in general during RL. Also, if the authors can more scientifically validate that high-entropy tokens are \"fork tokens\", this will be even more interesting/important as it helps people attribute the underlying reasons.\n- The paper is well-written, and the experiments present solid improvements.\n- Biggest concern: the experiments are mainly focused on Qwen models on math, which has been proven to be a \"noisy\" setup, as we can not disentangle whether the effectiveness comes from the base model or the algorithm itself [3,4,5]. \n\n[1] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. Cui et al. 2025.\n[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Yu et al. 2025.\n[3] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025.\n[4] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025.\n[5] Reinforcement Learning for Reasoning in Large Language Models with One Training Example. Wang et al. 2025.", "was_revised": true, "final_justification": "The authors said they will solve the writing issues in the next version, which is good. Given that I already gave them a high score and the change does not fundamentally improve paper quality, I will maintain my rating.", "rebuttal": "We sincerely thank you for your thorough review and valuable suggestions, which have significantly contributed to improving our manuscript.\n\n---\n\n> Q1: My only complaint is that the conclusion that high-entropy tokens are \"fork tokens\" is mainly based on qualitative analysis through word clouds. As this terminology has repeatedly occurred and seems to be an important concept introduced in this paper, I expect there is a more rigorous derivation and evidence.\n\nA1: Thank you for highlighting this potential ambiguity. In our paper, we use the term \"forking token\" to describe high-entropy minority tokens. These tokens naturally have multiple plausible alternatives due to their high entropy, making them interchangeable at their respective positions. As such, they resemble \"forks\" in a decision path, where several viable candidates can lead to different CoTs.\n\nMoreover, as shown quantitatively in Fig. 3, the high-entropy characteristic of these tokens is essential—removing or diminishing this property results in a significant drop in performance. This supports the idea that the \"forking\" function of high-entropy minority tokens is critical: when we suppress the probabilities of all candidates except the most likely one (thereby reducing the forking effect), performance degrades markedly.\n\nThanks again for pointing this out. We will clarify and emphasize this explanation in the next version of our paper.\n\n---\n\n> Q2: How did you determine that the high-entropy tokens are forking tokens? In the paper, this is a very strong causal claim (\"We refer to high-entropy tokens as forking tokens, which function like forks in the road by determining which path to follow for continuation\"), so more convincing and comprehensive evidence should be provided.\n\nA2: Thanks for your pointing out. We make this claim based on the observations in `A1` above. We will add the explanations above to the next version of our paper to avoid potential misunderstanding.\n\n---\n\n> Q3: Biggest concern: the experiments are mainly focused on Qwen models on math, which has been proven to be a \"noisy\" setup, as we can not disentangle whether the effectiveness comes from the base model or the algorithm itself.\n\nA3: Thank you for pointing out the potential issue. We'd like to clarify the following two points:\n\n(1) According to a recent study [1], which discusses or references the papers you listed [2, 3, 4], there is some degree of data contamination in the Qwen models on older benchmarks such as MATH500, AMC, and AIME'24. This may be unavoidable when pre-training on large-scale web corpora that include publicly available benchmark problems, as noted in [1]. \n\nHowever, newer benchmarks like AIME'25 and MinervaMath are likely free from such contamination. Importantly, our method—training on the top 20% of high-entropy tokens—consistently outperforms full-token training on both the older benchmarks (MATH500, AMC, AIME'24) and the newer ones (AIME'25 and MinervaMath), demonstrating the robustness and effectiveness of our approach.\n\n(2) In addition to experiments on Qwen models, we also present results in Appendix A.4 using LLaMA-3.1-8B, which is believed to have less data contamination according to [1]. In this case as well, training on only the top 20% high-entropy tokens yields better performance than using all tokens. For your convenience, we summarize the performance comparison in the table below. We also plan to highlight this experiment more prominently in the next version of our paper.\n\n| Trained from           | Methods                          | Maximum Score on AIME'24 |\n|----------------------|----------------------------------|---------------------------|\n| Llama–3.1–8B–base     | w/ top 20% high-entropy tokens   | 10.21                     |\n| Llama–3.1–8B–base     | w/ full tokens                   | 7.92                      |\n\n[1] Wu, Mingqi, et al. \"Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.\" arXiv preprint arXiv:2507.10532 (2025).\n\n[2] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025. \n\n[3] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025. \n\n[4] Reinforcement Learning for Reasoning in Large Language Models with One Training Example. Wang et al. 2025.\n\n---\n\nWe greatly appreciate your thoughtful evaluation and expertise. **If our responses address your concerns adequately, we would be grateful if you could consider increasing the score accordingly.** We remain open to further discussion and are more than willing to clarify any remaining questions or issues you might have."}, {"review_id": "E0ybq5iMg0", "reviewer": "Reviewer_TtMU", "summary": "The paper analyzes the token entropy patterns in Chain-of-Thought (CoT) reasoning and observes that only a small fraction (approximately\n20%) of tokens exhibit high entropy. These high-entropy tokens seem to function as \"forks\" to determine reasoning directions, and low-entropy tokens tend to execute reasoning steps along the established path. Based on these observations, the paper proposes to only learning on the high-entropy tokens during reinforcement learning with verifiable rewards and surpasses the baseline DAPO. The experimental results verify the importance of these high-entropy tokens.", "questions": "1. See weakness 1. The reviewer would expect more explanations and analysis about why the CoT length becomes longer, and how the patterns change of these high/low entropy tokens before and after RLVR on these high-entropy tokens, and why the \"fork\" tokens have high-entropy?\n2. In the setting of ZERO training, some meaningless and garbled tokens will also have high entropy, it seems that the proposed method will encourage these garbled tokens in training, how is this issue fixed?", "limitations": "See weakness 1.\nThe paper mainly present empirical observation and experimental results. More insights would help improve the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper identifies an interesting phenomenon, where the tokens in the long chain-of-thought trajectories exhibit different entropy patterns, points out the difference between these tokens, and applies the analysis in RLVR.\n2. The experimental results show that only learning on these high-entropy tokens can improve performance based on baseline DAPO, which is a strong baseline for RLVR.\n\n### Weaknesses\n\n1. The mechanism of high/low-entropy tokens lacks further analysis. For example, after RLVR on high-entropy tokens, the length of CoT becomes significantly longer. Furthermore, it is a bit counterintuitive that the tokens have high entropy on some \"fork\" tokens that do not exactly execute reasoning. Thus, it might be more insightful if the paper could bring deeper analysis and explanation about the high/low entropy tokens, and their corresponding changes after RLVR only on these high-entropy tokens.", "was_revised": true, "final_justification": "The paper addressed my concerns. According to other review comments and the corresponding responses, I tend to raise my score to 5.", "rebuttal": "We sincerely thank you for your thorough review and valuable suggestions, which have significantly contributed to improving our manuscript.\n\n---\n\n> Q1: The mechanism of high/low-entropy tokens lacks further analysis. For example, after RLVR on high-entropy tokens, the length of CoT becomes significantly longer. The reviewer would expect more explanations and analysis about why the CoT length becomes longer.\n\nA1: Thank you for your valuable question. We offer the following observations—both qualitative and quantitative—when comparing training using only high-entropy tokens versus training with all tokens:\n\n(1) Qualitative: In CoT reasoning, we empirically find that models trained with only the top 20% high-entropy tokens exhibit more complex logical structures compared to those trained on all tokens. This complexity includes increased occurrences of self-verification and a higher number of logical reasoning steps. (Due to space limitations in the rebuttal and the length of CoTs, we're unable to include a comparison between CoTs from full-token training and top 20% high-entropy token training. We will provide this comparison across methods in the next version of our paper.)\n\n(2) Quantitative: After convergence in RLVR training, we compute the average number of words strongly associated with self-verification and logical reasoning per CoT. The results, presented in the table below, show a significant increase in these words when training is limited to high-entropy tokens, compared to full-token training. This supports our qualitative finding: training with only high-entropy tokens leads to more complex logical reasoning, particularly with greater emphasis on self-verification.\n\n| Training Method                         | Number of \"wait\" per CoT | Number of \"but\" per CoT | Number of \"however\" per CoT | Number of \"alternatively\" per CoT |\n|----------------------------------------|---------------------------|--------------------------|------------------------------|------------------------------------|\n| Training w/ top 20% high-entropy tokens| 17.94                     | 29.13                    | 1.84                         | 3.68                               |\n| Training w/ full tokens                | 11.39                     | 11.67                    | 0.57                         | 1.49                               |\n\n**A possible explanation:** We hypothesize that this phenomenon stems from the model’s increased focus on logic-related tokens when trained on high-entropy inputs, as illustrated in Fig. 2(b). As a result, when faced with challenging problems that the model cannot initially solve, it prioritizes enhancing the logical components of the CoT to arrive at the correct answer. This process typically results in longer, more logically intricate CoTs compared to training with all tokens.\n\n---\n\n> Q2: Furthermore, it is a bit counterintuitive that the tokens have high entropy on some \"fork\" tokens that do not exactly execute reasoning. Why the \"fork\" tokens have high-entropy? \n\nA2: Thank you for your insightful question. \n\nRegarding why \"fork\" tokens typically do not directly execute reasoning, this may be more intuitively understood through the metaphor illustrated in Fig. 1(a). When a person is walking along a mountain path and reaches a fork in the trail, they do not actually walk to decide which way to go. Instead, they simply change direction—turning or adjusting orientation based on a decision. Similarly, in CoT reasoning with LLMs, when the model decides on the next direction of reasoning, it doesn’t carry out the reasoning itself at that moment. Instead, it outputs logically suggestive tokens—such as “wait,” “but,” “however,” or “alternatively” (as illustrated in Fig. 2(b))—that signal a shift or branching in the reasoning path. While these tokens don’t perform the reasoning directly, they critically shape the logical flow of subsequent CoT steps.\n\nAs for why the \"forking\" tokens have high entropy, it is because that, at positions in the CoT where multiple plausible candidate tokens exist, each plausible candidate tokens tends to have a relatively high probability. This results in a comparatively flat probability distribution. According to the token entropy definition in Equation (1):\n\n$$\nH_t = -\\sum_{j=1}^{V} p_{t,j} \\log p_{t,j},\n$$\n\na flatter distribution typically yields higher entropy. This explains why \"forking\" tokens exhibit high entropy.\n\n\n---\n\n> Q3: Thus, it might be more insightful if the paper could bring deeper analysis and explanation about the high/low entropy tokens, and their corresponding changes after RLVR only on these high-entropy tokens. \n\nA3: Thank you for your constructive suggestion. We conducted a deeper analysis on how token entropy evolves after RLVR, and surprisingly found that the overall entropy pattern (i.e., which tokens exhibit high entropy and which exhibit low entropy) remains largely unchanged. Specifically, we use the RL-trained LLM to generate responses, then compute the logits for these responses using both the pre-RL model (i.e., the base model) and the post-RL model (i.e., the RLVR model). We then calculate the overlap in token positions within the top 20% entropy. The results are presented in the table below.\n\n| Compared w/  | Step 0 | Step 16 | Step 112 | Step 160 | Step 480 | Step 800 | Step 864 | Step 840 | Step 1280 | Step 1360 |\n|--------------|--------|---------|----------|----------|----------|----------|----------|----------|-----------|------------|\n| Base Model   | 100%   | 98.92%  | 98.70%   | 93.04%   | 93.02%   | 93.03%   | 87.45%   | 87.22%   | 87.09%    | 86.67%     |\n| RLVR Model   | 86.67% | 86.71%  | 86.83%   | 90.64%   | 90.65%   | 90.64%   | 96.61%   | 97.07%   | 97.34%    | 100%       |\n\nFurthermore, we observe that this finding holds regardless of whether full tokens or only high-entropy tokens are used during training. The only difference is that training exclusively with high-entropy tokens amplifies the changes in entropy for those tokens, resulting in a higher overall entropy level as shown in Fig. 5.\n\nThanks for your valuable suggestion again, and we will add the analysis above to the next version of our paper.\n\n---\n\n> Q4: In the setting of ZERO training, some meaningless and garbled tokens will also have high entropy, it seems that the proposed method will encourage these garbled tokens in training, how is this issue fixed?\n\nA4: Thank you for your valuable feedback. We also observed that in CoT reasoning, certain seemingly meaningless or garbled tokens—such as `.`, `,`, or `\\n\\n`—can still exhibit high entropy. However, it is difficult to determine whether these tokens are truly meaningless or garbled. For instance, although they may appear insignificant to humans, such tokens could potentially serve as transitional elements in CoTs, helping to link preceding and subsequent thoughts.\n\nThat said, we acknowledge that entropy is only a proxy metric for identifying tokens that are more valuable for training. This metric is not perfect and can sometimes include genuinely meaningless or garbled tokens in the training process. We believe exploring more effective proxy metrics for identifying important tokens is a promising direction for future research.\n\nThank you for pointing this out, and we will incorporate this discussion into the limitations section of the next version of our paper.\n\n\n---\n\nWe sincerely appreciate your expertise and careful evaluation. **If you find our responses satisfactory, we kindly ask you to consider adjusting the score accordingly.** We welcome any further discussion and are happy to address any additional questions or concerns you may have."}]

Author rebuttal:

Meta-review: This paper first analyzes the existence of high-entropy minority tokens as forking tokens during the CoT reasoning of LLMs and observes that restricting policy gradient updates to these forking tokens during Reinforcement Learning with Verifiable Rewards (RLVR) improves mathematical reasoning performances of Qwen3-32B.

Overall, the entropy-based analysis and the corresponding simple modification of RLVR seem to be interesting and technically sound. Lack of experiments regarding different reasoning tasks and LLMs are somewhat (not fully) addressed during the rebuttal. Further analysis showing that RLVR with only 20% of high-entropy tokens leads to longer responses with more use of self-verification words is also interesting and helpful in supporting the claims of the paper. Based on the consensus of all positive ratings between the reviewers, I would recommend the paper to be accepted.

However, more empirical validation with diverse reasoning tasks and various LLMs would be necessary. Also, current theoretical analysis would be still insufficient. In particular, it remains unclear why updating only the influential tokens can directly lead to performance improvements. Moreover, the choice of 20% as the decision threshold is unlikely to be universal; from the perspective of the exploration–exploitation transition, a more thorough theoretical analysis on what constitutes an appropriate setting would be necessary.
Decision: Accept (poster)