Title: Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Reasoning

Reinforcement Learning

Entropy
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), yet its underlying mechanisms remain insufficiently understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction (approximately 20\%) of tokens exhibit high entropy, and these tokens semantically act as critical forks that steer the model toward diverse reasoning pathways. We further demonstrate that moderately increasing the entropy of these high-entropy tokens via decoding temperature adjustments leads to improved performance, quantitatively confirming their role as decision points in reasoning. We ultimately refine RLVR by restricting policy gradient updates to these forking tokens. Despite utilizing only 20\% of tokens, our approach achieves comparable performance to full-gradient updates on the Qwen3-8B base model. Moreover, it demonstrates remarkable improvements on the larger Qwen3-32B base model, boosting AIME'25 scores by 11.04 and AIME'24 scores by 7.71. In contrast, training exclusively on the 80\% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that dictate key reasoning directions. Collectively, our results suggest promising avenues for optimizing RLVR algorithms by strategically leveraging the potential of these high-entropy minority tokens to further enhance the reasoning abilities of LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the token entropy patterns in Chain-of-Thought (CoT) reasoning and observes that only a small fraction (approximately\n20%) of tokens exhibit high entropy. These high-entropy tokens seem to function as \"forks\" to determine reasoning directions, and low-entropy tokens tend to execute reasoning steps along the established path. Based on these observations, the paper proposes to only learning on the high-entropy tokens during reinforcement learning with verifiable rewards and surpasses the baseline DAPO. The experimental results verify the importance of these high-entropy tokens.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper identifies an interesting phenomenon, where the tokens in the long chain-of-thought trajectories exhibit different entropy patterns, points out the difference between these tokens, and applies the analysis in RLVR.\n2. The experimental results show that only learning on these high-entropy tokens can improve performance based on baseline DAPO, which is a strong baseline for RLVR.\n\n### Weaknesses\n\n1. The mechanism of high/low-entropy tokens lacks further analysis. For example, after RLVR on high-entropy tokens, the length of CoT becomes significantly longer. Furthermore, it is a bit counterintuitive that the tokens have high entropy on some \"fork\" tokens that do not exactly execute reasoning. Thus, it might be more insightful if the paper could bring deeper analysis and explanation about the high/low entropy tokens, and their corresponding changes after RLVR only on these high-entropy tokens.\n\nQUESTIONS:\n1. See weakness 1. The reviewer would expect more explanations and analysis about why the CoT length becomes longer, and how the patterns change of these high/low entropy tokens before and after RLVR on these high-entropy tokens, and why the \"fork\" tokens have high-entropy?\n2. In the setting of ZERO training, some meaningless and garbled tokens will also have high entropy, it seems that the proposed method will encourage these garbled tokens in training, how is this issue fixed?\n\nLIMITATIONS:\nSee weakness 1.\nThe paper mainly present empirical observation and experimental results. More insights would help improve the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the mechanisms underlying Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing LLM reasoning capabilities through the lens of token-level entropy analysis. The authors demonstrate that in Chain-of-Thought reasoning, only approximately 20% of tokens exhibit high entropy and serve as critical \"forking points\" that determine reasoning directions, including some common tokens indicating test-time scaling like \"wait\". They propose a modified RLVR approach that restricts policy gradient updates to only these high-entropy tokens, achieving comparable or superior performance to full-gradient methods while using only 20% of tokens. The key finding is that low-entropy tokens contribute minimally to reasoning improvements, with the high-entropy minority tokens being the primary drivers of RLVR effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n- Novel analysis perspective. Although there have been recent analyses of entropy for LLM reasoning, they have mostly focused on sequence-level [1], whereas this work focuses on the token-level. As token-level entropy is adopted as the proxy to gauge entropy, the angle in this work may be more suitable.\n- Strong empirical validation. To investigate the entropy issue, this paper starts with an inspection of the distribution of token entropy. To further validate the hypothesis on the existence of high/low-entropy tokens, a decoding experiment is performed, showing that high-entropy tokens are pruned to high entropy while low-entropy tokens require lower temperature. **My only complaint is that the conclusion that high-entropy tokens are \"fork tokens\" is mainly based on qualitative analysis through word clouds.** As this terminology has repeatedly occurred and seems to be an important concept introduced in this paper, I expect there is a more rigorous derivation and evidence.\n- Useful insights in practice. This work provides valuable insights into preserving the exploration ability in RL, following previous practices [2]. Together with other recent works, it calls for attention to entropy and exploration in general during RL. Also, if the authors can more scientifically validate that high-entropy tokens are \"fork tokens\", this will be even more interesting/important as it helps people attribute the underlying reasons.\n- The paper is well-written, and the experiments present solid improvements.\n- Biggest concern: the experiments are mainly focused on Qwen models on math, which has been proven to be a \"noisy\" setup, as we can not disentangle whether the effectiveness comes from the base model or the algorithm itself [3,4,5]. \n\n[1] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. Cui et al. 2025.\n[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Yu et al. 2025.\n[3] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025.\n[4] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025.\n[5] Reinforcement Learning for Reasoning in Large Language Models with One Training Example. Wang et al. 2025.\n\nQUESTIONS:\nHow did you determine that the high-entropy tokens are forking tokens? In the paper, this is a very strong causal claim (\"We refer to high-entropy tokens as forking tokens, which function like forks in the road by determining which path to follow for continuation\"), so more convincing and comprehensive evidence should be provided.\n\nLIMITATIONS:\nThe experimental setups. See weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work explores the mechanism behind  RLVR by zooming from sequence-level to token-level entropy in CoT traces of LLMs. An empirical analysis reveals that only about 20 % of tokens carry high entropy, and these “forking tokens” act as pivotal decision points that steer reasoning paths. Controlled decoding experiments show that selectively increasing temperature at these tokens boosts accuracy, whereas manipulating low-entropy tokens has little effect. The results suggest that RLVR’s efficacy stems almost entirely from optimizing this high-entropy minority, pointing toward more targeted and efficient training strategies for future reasoning-oriented LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n# Strenghts\n1. This work shifts the discussion from sequence level entropy to token level patterns, revealing an actionable 20/80 split that had not been quantified before.\n2. The authors conduct temperature manipulation and gradient masking studies to convincingly identify the contribution of forking tokens and reduce confounds.\n3. The proposed method achieves equal-or-better accuracy with 80 % fewer gradient updates, promising for large-scale RLVR where compute is scarce.\n4. The authors conduct comprehensive studies, including varying the kept token ratio multiple base model sizes, and six public math benchmarks, to support robustness claims.\n\n# Weaknesses\n\n1. All experiments target mathematical reasoning; it remains unclear whether the 20 % rule or performance gains hold for code, commonsense, or general reasoning tasks.\n2. Results are restricted to Qwen3 variants; LLaMA or Mixtral models are not included. It would strengthen this work by demonstrating the generalization of this idea with different model families.\n3. No error bars or significance tests are reported, so improvements could partly stem from stochasticity in RL runs.\n4. While the idea is intuitively appealing, the work stops short of formalizing why high-entropy tokens dominate the policy gradient signal. Discussing with more insightful analysis from optimization or learning dynamics may further boost this work.\n\nQUESTIONS:\nAs discussed in weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the underlying mechanisms of Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy patterns in reasoning models. The authors analyze token entropy distributions in LLM reasoning processes and identify that approximately 20% of tokens exhibit high entropy and function as \"forking tokens\" that determine reasoning directions. They propose restricting policy gradient updates to only these high-entropy tokens during RLVR training. Experiments on Qwen3-8B and Qwen3-32B models across mathematical reasoning benchmarks show that this selective approach can match or exceed full-gradient RLVR performance while using only 20% of tokens.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Novel analytical perspective: The token-level entropy analysis of RLVR mechanisms provides a fresh and interpretable lens for understanding how reinforcement learning improves reasoning capabilities. The identification of \"forking tokens\" as critical decision points is intuitive and well-motivated.\n- Solid empirical validation: The paper provides multiple lines of evidence supporting the importance of high-entropy tokens.\n- Practical significance and Simplicity: The finding that RLVR effectiveness can be improved while reducing computational requirements has practical implications for both performance and training efficiency. \n- Comprehensive experimental design: The evaluation spans multiple mathematical reasoning benchmarks and includes appropriate ablations varying token proportions.\n\nWeaknesses\n- Lack of experiments on GRPO and PPO: The experiments are mainly conducted on DAPO. It will be helpful to also include experiments on GRPO or PPO to see if the proposed approach generalize to other RL algorithms.\n\nQUESTIONS:\n- Simple experiments on GRPO or PPO may help clarify whether the proposed method can generalized to other RL algorithms. (If not, what is the reason?)\n- The response length is also increasing with the improved accuracy. How does this phenomenon related to the training of High-Entropy Minority Tokens?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}