SPICE: Self-Play In Corpus Environments Improves Reasoning
Authors: name: Bo Liu; name: Chuanyang Jin; name: Seungone Kim; name: Weizhe Yuan; name: Wenting Zhao; name: Ilia Kulikov; name: Xian Li; name: Sainbayar Sukhbaatar; name: Jack Lanchantin; name: Jason Weston
Published: 2025-10-28 17:46:16

Abstract: Self-improving systems require environmental interaction for continuous
adaptation. We introduce SPICE (Self-Play In Corpus Environments), a
reinforcement learning framework where a single model acts in two roles: a
Challenger that mines documents from a large corpus to generate diverse
reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics,
the Challenger creates an automatic curriculum at the frontier of the
Reasoner's capability, while corpus grounding provides the rich,
near-inexhaustible external signal necessary for sustained improvement. Unlike
existing ungrounded self-play methods that offer more limited benefits, SPICE
achieves consistent gains across mathematical (+8.9%) and general reasoning
(+9.8%) benchmarks on multiple model families. Our analysis reveals how
document grounding is a key ingredient in SPICE to continuously generate its
own increasingly challenging goals and achieve them, enabling sustained
self-improvement.