Geometric-Mean Policy Optimization
Authors: name: Yuzhong Zhao, name: Yue Liu, name: Junpeng Liu, name: Jingye Chen, name: Xun Wu, name: Yaru Hao, name: Tengchao Lv, name: Shaohan Huang, name: Lei Cui, name: Qixiang Ye, name: Fang Wan, name: Furu Wei

Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.