SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy
  Optimization
Authors: name: Minghan Chen; name: Guikun Chen; name: Wenguan Wang; name: Yi Yang
Published: 2025-05-18 10:20:59

Abstract: Large language models (LLMs) exhibit varying levels of confidence across
input prompts (questions): some lead to consistent, semantically similar
answers, while others yield diverse or contradictory outputs. This variation
reflects LLM's uncertainty about the input prompt, a signal of how confidently
the model understands a given problem. However, vanilla Group Relative Policy
Optimization (GRPO) treats all prompts equally during policy updates, ignoring
this important information about the model's knowledge boundaries. To address
this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which
explicitly measures LLMs' uncertainty of the input prompts semantic entropy.
Semantic entropy measures the diversity of meaning in multiple generated
answers given a prompt and uses this to modulate the magnitude of policy
updates. This uncertainty-aware training mechanism enables dynamic adjustment
of policy update magnitudes based on question uncertainty. It allows more
conservative updates on high-uncertainty questions while maintaining the
original learning signal on confident ones. Experimental results on five
mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva
34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new
state-of-the-art performance in average accuracy, validating the effectiveness
of uncertainty-aware policy optimization.