CE-GPPO: Controlling Entropy via Gradient-Preserving Clipping Policy
  Optimization in Reinforcement Learning
Authors: name: Zhenpeng Su; name: Leiyu Pan; name: Minxuan Lv; name: Yuntao Li; name: Wenping Hu; name: Fuzheng Zhang; name: Kun Gai; name: Guorui Zhou
Published: 2025-09-25 03:22:04

Abstract: Reinforcement learning (RL) has become a powerful paradigm for optimizing
large language models (LLMs) to handle complex reasoning tasks. A core
challenge in this process lies in managing policy entropy, which reflects the
balance between exploration and exploitation during training. Existing methods,
such as proximal policy optimization (PPO) and its variants, discard valuable
gradient signals from low-probability tokens due to the clipping mechanism. We
systematically analyze the entropy dynamics and reveal that these clipped
tokens play a critical yet overlooked role in regulating entropy evolution. We
propose Controlling Entropy via
Gradient-Preserving Policy Optimization
(CE-GPPO), a novel algorithm that reintroduces gradients from clipped tokens in
native PPO in a gentle and bounded manner. By controlling the magnitude of
gradients from tokens outside the clipping interval, CE-GPPO is able to achieve
an exploration-exploitation trade-off. We provide theoretical justification and
empirical evidence showing that CE-GPPO effectively mitigates entropy
instability. Extensive experiments on mathematical reasoning benchmarks show
that CE-GPPO consistently outperforms strong baselines across different model
scales.