Geometric-Mean Policy Optimization
Authors: name: Yuzhong Zhao; name: Yue Liu; name: Junpeng Liu; name: Jingye Chen; name: Xun Wu; name: Yaru Hao; name: Tengchao Lv; name: Shaohan Huang; name: Lei Cui; name: Qixiang Ye; name: Fang Wan; name: Furu Wei
Published: 2025-07-28 09:54:05

Abstract: Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.