OpenSIR: Open-Ended Self-Improving Reasoner
Authors: name: Wai-Chung Kwan, name: Joshua Ong Jun Leang, name: Pavlos Vougiouklis, name: Jeff Z. Pan, name: Marco Valentino, name: Pasquale Minervini
Organization: EdinburghNLP - Natural Language Processing Group at the University of Edinburgh

Recent advances in large language model (LLM) reasoning through reinforcement
learning rely on annotated datasets for verifiable rewards, which may limit
models' ability to surpass human-level performance. While self-play offers a
promising alternative, existing approaches depend on external verifiers or
cannot learn open-endedly. We present Open-Ended Self-Improving Reasoner
(OpenSIR), a self-play framework where an LLM learns to generate and solve
novel problems by alternating teacher and student roles without external
supervision. To generate novel problems, OpenSIR optimises for both difficulty
and diversity, rewarding problems that challenge appropriately while exploring
distinct concepts, enabling open-ended mathematical discovery. Starting from a
single trivial seed problem, OpenSIR substantially improves instruction models:
Llama-3.2-3B-Instruct advances from 73.9 to 78.3 on GSM8K, and from 28.8 to
34.4 on College Math, while Gemma-2-2B-Instruct rises from 38.5 to 58.7 on
GSM8K. Our analyses reveal that OpenSIR achieves open-ended learning through
co-evolving teacher-student roles that adaptively calibrate difficulty and
drive diverse exploration, progressing autonomously from basic to advanced
mathematics.