Group Sequence Policy Optimization
Authors: name: Chujie Zheng; name: Shixuan Liu; name: Mingze Li; name: Xiong-Hui Chen; name: Bowen Yu; name: Chang Gao; name: Kai Dang; name: Yuqiong Liu; name: Rui Men; name: An Yang; name: Jingren Zhou; name: Junyang Lin
Published: 2025-07-24 03:50:32

Abstract: This paper introduces Group Sequence Policy Optimization (GSPO), our stable,
efficient, and performant reinforcement learning algorithm for training large
language models. Unlike previous algorithms that adopt token-level importance
ratios, GSPO defines the importance ratio based on sequence likelihood and
performs sequence-level clipping, rewarding, and optimization. We demonstrate
that GSPO achieves superior training efficiency and performance compared to the
GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and
has the potential for simplifying the design of RL infrastructure. These merits
of GSPO have contributed to the remarkable improvements in the latest Qwen3
models.