Evolving Language Models without Labels: Majority Drives Selection,
  Novelty Promotes Variation
Authors: name: Yujun Zhou; name: Zhenwen Liang; name: Haolin Liu; name: Wenhao Yu; name: Kishan Panaganti; name: Linfeng Song; name: Dian Yu; name: Xiangliang Zhang; name: Haitao Mi; name: Dong Yu
Published: 2025-09-18 17:50:04

Abstract: Large language models (LLMs) are increasingly trained with reinforcement
learning from verifiable rewards (RLVR), yet real-world deployment demands
models that can self-improve without labels or external judges. Existing
label-free methods, confidence minimization, self-consistency, or majority-vote
objectives, stabilize learning but steadily shrink exploration, causing an
entropy collapse: generations become shorter, less diverse, and brittle. Unlike
prior approaches such as Test-Time Reinforcement Learning (TTRL), which
primarily adapt models to the immediate unlabeled dataset at hand, our goal is
broader: to enable general improvements without sacrificing the model's
inherent exploration capacity and generalization ability, i.e., evolving. We
formalize this issue and propose EVolution-Oriented and Label-free
Reinforcement Learning (EVOL-RL), a simple rule that couples stability with
variation under a label-free setting. EVOL-RL keeps the majority-voted answer
as a stable anchor (selection) while adding a novelty-aware reward that favors
responses whose reasoning differs from what has already been produced
(variation), measured in semantic space. Implemented with GRPO, EVOL-RL also
uses asymmetric clipping to preserve strong signals and an entropy regularizer
to sustain search. This majority-for-selection + novelty-for-variation design
prevents collapse, maintains longer and more informative chains of thought, and
improves both pass@1 and pass@n. EVOL-RL consistently outperforms the
majority-only TTRL baseline; e.g., training on label-free AIME24 lifts
Qwen3-4B-Base AIME25 pass@1 from TTRL's 4.6% to 16.4%, and pass@16 from 18.5%
to 37.9%. EVOL-RL not only prevents diversity collapse but also unlocks
stronger generalization across domains (e.g., GPQA). Furthermore, we
demonstrate that EVOL-RL also boosts performance in the RLVR setting,
highlighting its broad applicability.