GCPO: When Contrast Fails, Go Gold
Authors: name: Hao Wu; name: Wei Liu
Published: 2025-10-09 05:09:06

Abstract: Reinforcement learning has been widely applied to enhance the reasoning
capabilities of large language models. Extending the inference limits of
smaller models has become a prominent research focus. However, algorithms such
as Group Relative Policy Optimization (GRPO) suffer from a clear drawback: the
upper bound of a model's rollout responses is entirely determined by the model
itself, preventing the acquisition of knowledge from samples that are either
all incorrect or all correct. In this paper, we introduce Group Contrastive
Policy Optimization (GCPO), a method that incorporates external standard
reference answers. When the model cannot solve a problem, the reference answer
supplies the correct response, steering the model toward an unequivocally
accurate update direction. This approach offers two main advantages: (1) it
improves training efficiency by fully utilizing every sample; (2) it enables
the model to emulate the problem solving strategy of the reference answer
during training, thereby enhancing generalization in reasoning. GCPO achieves
outstanding results across multiple benchmark datasets, yielding substantial
improvements over the baseline model. Our code is available at:
https://github.com/AchoWu/GCPO.