From Uniform to Heterogeneous: Tailoring Policy Optimization to Every
  Token's Nature
Authors: name: Zheng Liu; name: Mengjie Liu; name: Siwei Wen; name: Mengzhang Cai; name: Bin Cui; name: Conghui He; name: Wentao Zhang
Published: 2025-09-20 09:30:25

Abstract: Reinforcement Learning has emerged as the fundamental technique for enhancing
reasoning in LLMs. However, existing algorithms apply uniform optimization to
all tokens, ignoring their different roles in reasoning process. To address
this limitation, we introduce Heterogeneous Adaptive Policy Optimization
(HAPO), a comprehensive token-aware algorithm that dynamically adapts
optimization based on token entropy. For rollout sampling, we propose Adaptive
Temperature Sampling, which adjusts sampling temperature in real time,
promoting exploration at high-entropy tokens while preserving coherence at
low-entropy ones. For advantage calculation, we introduce Token Level Group
Average that normalizes advantages at token level, jointly accounting for
sequence-length as in token-mean loss while preserving non-biased treatment. We
then develop Differential Advantage Redistribution that leverages entropy and
importance ratios to modulate rewards-adjusting updates for tokens with clear
signals. For clipping loss, we design Asymmetric Adaptive Clipping, allowing
aggressive probability reduction for noisy low-entropy tokens while enabling
exploration for high-entropy tokens. Through systematic investigation between
entropy and training dynamics, we embedded token-level treatment into every
stages to achieve fine-grained control. Extensive experiments demonstrate that
HAPO consistently outperforms DAPO across multiple model scales. Our code can
be found in https://github.com/starriver030515/HAPO.