SPICE: Self-Play In Corpus Environments Improves Reasoning
Authors: name: Bo Liu, name: Chuanyang Jin, name: Seungone Kim, name: Weizhe Yuan, name: Wenting Zhao, name: Ilia Kulikov, name: Xian Li, name: Sainbayar Sukhbaatar, name: Jack Lanchantin, name: Jason Weston
Organization: AI at Meta

Self-improving systems require environmental interaction for continuous
adaptation. We introduce SPICE (Self-Play In Corpus Environments), a
reinforcement learning framework where a single model acts in two roles: a
Challenger that mines documents from a large corpus to generate diverse
reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics,
the Challenger creates an automatic curriculum at the frontier of the
Reasoner's capability, while corpus grounding provides the rich,
near-inexhaustible external signal necessary for sustained improvement. Unlike
existing ungrounded self-play methods that offer more limited benefits, SPICE
achieves consistent gains across mathematical (+8.9%) and general reasoning
(+9.8%) benchmarks on multiple model families. Our analysis reveals how
document grounding is a key ingredient in SPICE to continuously generate its
own increasingly challenging goals and achieve them, enabling sustained
self-improvement.