Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective
  Reinforcement Learning for LLM Reasoning
Authors: name: Shenzhi Wang; name: Le Yu; name: Chang Gao; name: Chujie Zheng; name: Shixuan Liu; name: Rui Lu; name: Kai Dang; name: Xionghui Chen; name: Jianxin Yang; name: Zhenru Zhang; name: Yuqiong Liu; name: An Yang; name: Andrew Zhao; name: Yang Yue; name: Shiji Song; name: Bowen Yu; name: Gao Huang; name: Junyang Lin
Published: 2025-06-02 17:54:39

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
powerful approach to enhancing the reasoning capabilities of Large Language
Models (LLMs), while its mechanisms are not yet well understood. In this work,
we undertake a pioneering exploration of RLVR through the novel perspective of
token entropy patterns, comprehensively analyzing how different tokens
influence reasoning performance. By examining token entropy patterns in
Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of
tokens exhibit high entropy, and these tokens act as critical forks that steer
the model toward diverse reasoning pathways. Furthermore, studying how entropy
patterns evolve during RLVR training reveals that RLVR largely adheres to the
base model's entropy patterns, primarily adjusting the entropy of high-entropy
tokens. These findings highlight the significance of high-entropy tokens (i.e.,
forking tokens) to RLVR. We ultimately improve RLVR by restricting policy
gradient updates to forking tokens and uncover a finding even beyond the 80/20
rule: utilizing only 20% of the tokens while maintaining performance comparable
to full-gradient updates on the Qwen3-8B base model and significantly
surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71
on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models,
highlighting a strong scaling trend. In contrast, training exclusively on the
80% lowest-entropy tokens leads to a marked decline in performance. These
findings indicate that the efficacy of RLVR primarily arises from optimizing
the high-entropy tokens that decide reasoning directions. Collectively, our
results highlight the potential to understand RLVR through a token-entropy
perspective and optimize RLVR by leveraging high-entropy minority tokens to
further improve LLM reasoning.