Title: Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Reinforcement Learning with Verifiable Rewards, Proximal Policy Optimization, Advantage Estimation, Long-Horizon Reasoning

Abstract: Training Large Language Models (LLMs) for reasoning tasks is increasingly driven by Reinforcement Learning with Verifiable Rewards (RLVR), where Proximal Policy Optimization (PPO) provides a principled framework for stable policy updates. However, the practical application of PPO is hindered by unreliable advantage estimation in the sparse-reward RLVR regime. This issue arises because the sparse rewards in RLVR lead to inaccurate intermediate value predictions, which in turn introduce significant bias when aggregated at every token by Generalized Advantage Estimation (GAE). To address this, we introduce Segmental Advantage Estimation (SAE), which mitigates the bias that GAE can incur in RLVR. Our key insight is that aggregating $n$-step advantages at every token(as in GAE) is unnecessary and often introduces excessive bias, since individual tokens carry minimal information. Instead, SAE first partitions the generated sequence into coherent sub-segments using low-probability tokens as heuristic boundaries. It then selectively computes variance-reduced advantage estimates only from these information-rich segment transitions, effectively filtering out noise from intermediate tokens.  Our experiments demonstrate that SAE achieves superior performance, with marked improvements in final scores, training stability, and sample efficiency.  These gains are shown to be consistent across multiple model sizes, and a correlation analysis confirms that our proposed advantage estimator achieves a higher correlation with an approximate ground-truth advantage, justifying its superior performance.

TL;DR: We propose Segmental Advantage Estimation (SAE) as a superior alternative to GAE, which improves PPO training for LLMs by calculating advantages at the boundaries of semantic segments rather than at the noisy token-level.