Title: SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Post-training, GRPO

Abstract: Group Relative Policy Optimization (GRPO) introduces a new paradigm for reinforcement learning in Large Language Models (LLMs), modifying PPO by eliminating the value model for efficient post-training. However, vanilla GRPO assigns equal weight to all prompts during policy updates, overlooking that some prompts lie beyond the LLMs’ current knowledge and can harm performance. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs’ uncertainty and uses it to modulate the learning process. This enables conservative updates for high-uncertainty prompts (e.g., beyond model knowledge) while preserving relatively higher signals for confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization. The code, implementation details, and pre-trained models will be publicly released.