SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
Authors: Bo Liu
Simon Yu
Zichen Liu
Leon Guertler
Penghui Qi
Daniel Balcells
Mickel Liu
Cheston Tan
Weiyan Shi
Min Lin
Wee Sun Lee
Natasha Jaques
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Self-Play
Large Language Models
Reasoning
Multi-Agent Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: Self-play on multiple zero-sum language games teaches LLMs transferable reasoning skills that improve mathematical and general reasoning benchmarks by up to 10%, without requiring any domain-specific training data.

Abstract: Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approaches depend on human-curated problem-answer pairs and domain-specific reward engineering. We introduce SPIRAL, a self-play framework where models learn by playing multi-turn, zero-sum games against continuously improving versions of themselves, generating an automatic curriculum of stronger opponents, and eliminating the need for human supervision. To enable this self-play training at scale, we implement a fully online, multi-turn, multi-agent reinforcement learning system for LLMs and propose role-conditioned advantage estimation (RAE) to stabilize multi-agent training. SPIRAL produces reasoning capabilities that transfer broadly, improving performance by up to 10\% across a suite of 8 reasoning benchmarks on 4 different models spanning Qwen and Llama model families, outperforming supervised fine-tuning on 25,000 expert game trajectories. Multi-game training (TicTacToe, Kuhn Poker, Simple Negotiation) yields the strongest results, with improvements observed across both base and instruction-tuned models. Analysis of chain-of-thought traces reveals that games develop distinct cognitive patterns that transfer to improve reasoning performance, with different games developing complementary strengths. Even models which have already been trained on reasoning tasks using RLVR, like DeepSeek-R1-Distill-Qwen-7B, still benefit from our approach. These results demonstrate that zero-sum games naturally develop transferable reasoning capabilities across diverse model architectures and training stages, highlighting a promising direction for autonomous reasoning development.

Published as a conference paper at ICLR 2026
SPIRAL: SELF-PLAY ON ZERO-SUM GAMES INCEN- TIVIZES REASONING VIA MULTI-AGENT MULTI-TURN REINFORCEMENT LEARNING
Bo Liu*1 Simon Yu*2 Zichen Liu#1.3, Leon Guertler#1
Penghui Qi Daniel Balcells", Mickel Liu", Cheston Tan", Weiyan Shi?, Min Lin Wee Sun Leek Natasha Jaques
National University of Singapore Northeastern University
Sea Al Lab
*Centre for Frontier Al Research (CFAR), A* Plasticc Labs Uniierrsity of Washington
ABSTRACT
Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable re- wards, but these approaches depend human-curated problem-answer pairs and domain-specific reward engineering. We introduce SPIRAL, a self-play frame- work where models learn by playing multi-turn, zero-sum games against tinuously improving versions of themselves, generating an automatic curricu- lum of stronger opponents, and eliminating the need for human supervision. To enable this self-play training at scale, we implement a fully online, multi- turn, multi-agent reinforcement learning system for LLMs and propose role- conditioned advantage estimation (RAB) to stabilize multi-agent training. SPI- RAL produces reasoning capabilities that transfer broadly, improving perfor- mance by up to 10% across suite of 8 reasoning benchmarks on 4 different models spanning Qwen and Llama model families, outperforming supervised fine-tuning on 25,000 expert game trajectories. Multi-game training (TieTacToe, Kuhn Poker, Simple Negotiation) yields the strongest results, with improvements observed across both base and instruction-tuned models. Analysis of chain-of- thought traces reveals that games develop distinct cognitive patterns that transfer improve reasoning performance, with different games developing complemen- tary strengths. Even models which have already been trained on reasoning tasks using RLVR, like DeepSeek-RI -i-essiR--istilleeeeee
still benefit from our approach.
These results demonstrate that zero-sum games naturally develop transferable rea- soning capabilities across diverse model architectures and training stages, high- lighting promising direction for autonomous reasoning development. Our code can be found in https: / thub. com commspiiee paprrall.. -spprairrrrrraaal
INTRODUCTION
Recent breakthroughs in language model reasoning, including OpenAl ol (OpenAI, 2024) and DeepSeek-R1 (DeepSeek Team, 2024), reveal that reinforcement learning (RL) can unlock dramatic improvements in Chain-of-Thhuulhe reasoning (Wei et al., 2022). Through outcome- rewards, RL enables models to develop generalizable reasoning strategies and consistently solve complex problems where supervised fine-tuning shows limited progress.
However, current approaches face a fundamental scalability bottleneck: dependence carefully engineered reward functions, domain-specific datasets, and expert supervision (DeepSeek Team, 2024; Ouyang et al., 2022; Bai et al., 2022). Each new reasoning domain requires experts craft evaluation metrics, curate training problems, and validate reasoning traces. This manual process becomes increasingly unsustainable we pursue more general intelligence, limiting both scale and diversity of reasoning challenges that models can learn from.
Equal contribution, order randomly decided by dice roll. Corresponding author.

---

Published as a conference paper at ICLR 2026
Owen3-4B-Base
Cwen3-48-SFT
80
Owen3-4B-Mistrl
Swen3-4B-Gemin
Score (%)
4w (SPIRAL)
30
Game Avg
Benchmark Avg
MATH530
AIME24
AIME25
OlympiadBench
AMC-23
Minerva Math
GPOA
AMMLU-PIOO
Figure I: SPIRAL achieves consistent improvements over base models across game performance and reasoning benchmarks. also surpasses SFT on expert game trajectories and RL baselines trained against fixed opponents (Mistral and Gemini).
Self-play on games offers solution by eliminating human supervision in training data creation (Sil- ver et al., 2017; Tesauro, 1995). In game-based self-play, models learn by competing against copies of themselves, where game outcomes provide automatic feedback and opponents improve equally, maintaining consistent challenge that drives continuous learning. Although many prominent suc- cesses past AI research relied on self-play-from TD-Gammon (Tesauro, 1995) to AlphaGo (Sil- et al., 2016; 2017) to OpenAl Five (Berner et al., 2019) far, applying self-play games to enhance language model reasoning remains largely unexplored. Prior attempts have been limited to simple word games with offline updates (Cheng al., 2024), LoRA adaptations (Dettmers al., 2023; Park et al., 2025), or single-turn tasks (Zhao et al., 2025), falling short of leveraging multi-turn competitive dynamies for extended strategic reasoning.
We introduce SPIRAL (Self-Play on zero-sum games Incentivizes Reasoning via multi-Agent multi-turn reinforcement Learning), which applies self-play to two-player zero-sum language games for developing reasoning capabilities. SPIRAL offers two key advantages: unlike traditional RLVR approaches depending on human-curated problem-answer answer pairs, generates unlimited train- ing data through game dynamics alone; compared to fixed-opponent training (see Fig. 2), self-play prevents overfitting to static strategies by continuously evolving challenge level. However, imple- menting this for LLMs presents significant challenges. The computational demands of multi-turn, multi-agent autoregressive generation require sophisticated distributed systems, while standard RL algorithms suffer from high variance in multi-agent settings. We address these through fully online, multi-turn, multi-agent reinforcement learning system with distributed actor-learner architecture and introduce oel-connitiooeed advantage estimation (RAE), which stabilizes training by normalizing wards relative to each player's expected performance.
Key Findings. Training on zero-sum games produces reasoning capabilities that transfer broadly across diverse model architectures. Multi tullti-ammeea SPIRAL training (TicTaccToe, Kuhn Poker, Simple Negotiation) achieves up to 10% improvement across 8 reasoning benchmarks, outperforming super- vised fine-tuning on 25,00 25,000 expert trajectories. On Swen3-4B-Baas (Yang et al., 2025), multi-game training reaches 44.5% average performance versus 34.0% baseline (+10.5% absolute gain), while Qwen3-8B-Base (Yang et al., 2025) improves from 39.5% to 49.6% (+10.1 11.0..15. %). The approach gener- alizes across model families: base models (Qwen3-4B/S)) Baa//BBBBsse,. Octothinker-8B. 8B-Base; Wang et al. (2025a)) and instruction-tunde models (Llama-3, 3.t.a....--B3 --8----I-snttttt Dubey et al. (2024)) all show consis- tent improvements, with Octothinker-88 2.0% despite already being instruction- -unee.. Each game develops complementary cognitive skills: TicTacToe for spatial reasoning. Kuhn Poker for probabilistic thinking, and Simple Negotiation for strategic optimization, which combine synergistically in multi-game training. Using post-hoc analy- sis, we find examples of three patterns learned from gameplay that transfer to improve math perfor- mance: case-by-case analysis, expected value calculation, and pattern recognition. These patterns develop effectively through self-play's 's adaptive curriculum, as opponenp training fails while self-play continuously improves. Role-conditioned Advantage Estimation proves critical: without
gaining 8.0% and Llama-3.1 1-B---3.--88--tttut improving
2

---

Published as a conference paper at ICLR 2026
REWARD FUNCTION rluss))) sky Attrrnyyt. vinie))
X win
think
i..thhen
O X O
XIXIO
Figure 2: From human-designa rewards self-discovered reasoning through SPIRAL. Left: Tra-
ditional RI requires human experts to design complex reward functions. Middle: Fixed opponent training leads to exploitation of static strategies. Right: SPIRAL. enables continuous reasoning improvement through self-play, where both players develop increasingly sophisticated strategies without human supervision.
RAE, models abandon reasoning after 200 steps, progressively generating empty thinking traces that destroy generalization. Building on these findings, our work makes the following contributions:
1. A Fully Online, Multi-Turn, Multi-Agent RL Framework for LLMs: We develop dis- tributed actor-lcarner architecture enabling online self-play with aall-parmettrr updates across multiple two-player zero-sum language games. The multi-turn aspect trains models to reason through sequential decisions, directly preparing them for complex multi-step problem solving. Unlike prior offline approaches, this provides continuous curriculum the model adapts to an ever-improving opponent. We release our implementation to facilitate further research.
2. Role-conditioned Advantage Estimation (RAE): We introduce a variance-reduced advantage estimator specifically designed for multi-agent settings. By normalizing rewards relative each player's expected performance, RAE prevents the degradation of the model's 's reasoning capa- bilities, a failure mode we term "thinking collapse". Without it, models progressively abandon reasoning traces after 200 steps, which is critical for generalization.
3. Empirical Discovery of Transfer: We demonstrate that self-play zero-sum games improves both out-of-distribution game performance and academic reasoning benchmarks by up to 10% without domain-specific training data. Our analysis identifies reasoning patterns (systematic decomposition, expected value calculation, case-by-case analysis) that transfer from games to mathematics at measurable rates, with different games developing specialized skills that combine synergistically in multi game training.
RELATED WORK
Reinforcement Learning for LLM Reasoning. Reinforcement learning (RL) in LLMs has pro- gressed from alignment tasks using RLHF (Jaques et al., 2019; Ouyang et al., 2022; Bai et al., 2022) directly improving reasoning capabilities. Recent models like OpenAl ol (OpenAI, 2024) and DeepSeek-R1 (DeepSeek Team, 2024) demonstrate that RL. with verifiable rewards (RLVR) can unlock thain-of-thought reasoning using rule-based rewards (Lightman al., 2023; Uesato al., 2022). However, these approaches depend on human-curated problem sets and domain-specific reward engineering. SPIRAL eliminates this dependency by using self-play games to generate un- limited reasoning challenges without human supervision.
Self-Play and Multi-Agent RL for LLMs. Self-play in LLMs initially focused on aligmment ob- jectives (Chen et al., 2024; Yuan et al., 2024) before recent work applied it to enhance model ca- pabilities. SPAG (Cheng et al., 2024) applies self-play to Adversarial Taboo using offline updates on single game; SPC (Chen et al., 2025) and Genius (Xu et al., 2025a) require predefined human task distributions: Absolute Zero (Zhao et al., 2025) generates single-turn coding tasks; Founda- tion Model Self-Play (Dharna et al., 2025) uses foundation models to evolve code-based policies rather than direct gameplay; Prover- Verifier Game (Kirchner et al., 2024) improve output legibility through adversarial training. Implementing multi-agent RL (MARL) for full-scale LLMs presents significant technical challenges (Wan et al., 2025: Liu et al., 2025b;a). Prior work circumvents these challenges by using RNNs instead of transformers (Sarkar et al., 2025), restricting to simpli-
3

---

Published as a conference paper at ICLR 2026
Game Trajectories
SPIRAL
Actor
Parallel
Self-Play
Learner
Rotlout
GPL2
GPU
Zero-Sum Games
PLAYER
0g- [Re16]]]
Trjjectory'
Toe Kuhn Poker
PLAYER
(TT)
Σ Rofx)
Σn.0)
XlO10
PLAYER 0 (X2)
4i- Cp-ICalll
Role-comditionel
O
Bewadd
[Bei
Advantage Estimation (RAB
Rewand
mTg
Role 0 Advaniaee Ad6. a)
Role Adcendagy Ails, a)
Simple Negotiation
PLAYFIR 0 (82)
4y "Bol 30
Rewaad 0
PLAYER OT
Emsi Terminated
RL Training
Trajectory z
Vectorized Game Enn
Acmes GPLS
Multi- Agulti-Agent, Multi Turn Rollout
Weight Sync
Figure 3: The SPIRAL Framework. SPIRAL employs actor-learner architecture for scalable self-play training. Parallel actors sample trajectories from a diverse set of games using vectorized environments. single policy n, plays both roles, generating zero-sum, sparse reward game tra- jectories. The centralized learner processes these trajectories using Role-conditioned Advantage Estimation (RAE) to compute separate advantages, Ao(s, a) and Ai a), for each role. These then used for on-policy reinforcement learning updates.
fied enviromments (Jacob et al., 2022; Sukhbaatar et al., 2018), or applying supervised fine-tuning on trajectories from proprietary models (Liao et al.. 2024). In contrast, SPIRAL implements fully online. full- full-paraeeter MARL through distributed actor-learner architecture, enabling continuous adaptation to evolving opponents across multiple games.
LLMs in Gaming, Games serve as both evaluation benchmarks (Paglieri et al., 2024; Ruoss et al.,
2024: Zhang et al., 2024; Duan et al., 2024) and training domains (Feng al., 2024; Verma al., 2025). LMRL-Gym (Abdulhai et al., 2023) and RAGEN (Wang et al., 2025b) both employ single- agent multi-turn RL, with LMRL-Gym providing 8 benchmarking tasks and RAGEN focusing on trajectory-level optimization. ViGaL (Xie et al., 2025b) shows that single-agent RL on visual-spatial games transfers to mathematical reasoning without explicit math training. Logic-RL RL (Xic et al., 2025a) trains on synthetic puzzle games;
(Zhang et al., 2025) applics offline learning to grouped games; Boundless Socratic Learning (Schaul, 2024) uses language games for continual learning: Code2Logic (Tong et al., 2025) synthesizes reasoning data from game code. We also distinguish SPIRAL from works designed for achieving super-human performance at specific games. Cicero (FAIR et al., 2022) integrates a language model with a separate strategic planning algorithm to achieve human-level performance in Diplomacy. Similarly, agents developed for Were- wolf (Xu et al., 2024; 2025b) and Avalon (Wang et al., 2023) focus on optimizing strategic communi- cation and hidden-role deduction to win within their respective game rules. While these works target in-domain victory, SPIRAL treats competitive pressure as training scaffold to develop reasoning patterns (e.g., case analysis) that transfer to out-of-domain reasoning tasks. SPIRAL uniquely com- bines three elements: (1) multi-agent self-play where both players share parameters, (2) fully online learning with continuous opponent evolution, and (3) demonstrated transfer from zero-sum language games to academic reasoning benchmarks achieving up to 10.5% improvement without exposure to benchmark-re related problems during training.
THE SPIRAL FRAMEWORK
We present SPIRAL, a framework enabling language models to develop generalizable reasoning through multi-turn competitive self-play on games, illustrated in Figure 3.
Formulation. SPIRAL implements self-play through turn-based zero-sum language games from collection g (G1, G2 Gn). Each game G; is two-player zero-sum Markov game (Littman,
4

---

Published as a conference paper at ICLR 2026
1994) built turn-level MDPs where states E S represent complete contexts (e.g.. game figurations), actions a E A are complete multi-token responses, and transition function T; deter- mines state dynamics after full turn completion. The zero-sum property ensures ro(s, a!") aa)) + (s, (0) ( = for all states and actions where (P2) denotes the action of player p {0,1}, creating competitive dynamics. See Appendix C for detailed formulations.
Benefits multi-turn, zero-sum games. Zero-sum dynamics create continuous improvement pres- sure through rewards given only at game termination: Ti (1 4 "a.") = 0 for all non-terminal states, with terminal rewards (T) (s(s) and (T)
-pr(sT) where Pt Sterminal + {-1,0, determines the outcome and r represents the complete trajectory. This forces robust strat- egy development as models only receive feedback upon game completion. The multi-turn structure mirrors sequential reasoning problems: players alternate turns with t mod acting at time t while the opponent waits, training models maintain context, plan ahead, and adapt strategies.
Self-play. Rather than training separate policies Te, and πT, for each player, SPIRAL uses single shared policy тe with parameters 0, setting Ou = 02 = Role conditioning through system prompts enables the model to learn distinct strategies for each position (see Appendix D.1). At each turn, the active player generates a complete response y/2 ~ (e(-|st: p. G.) conditioned on current state St, player role p. and game Gj. From this response, we extract the action dz (p) to update the game state via 8t-1 = (a, Of az ()) where a; (1-p) for the inactive player. This parameer approach ensures efficient use of GPU memory while also guaranteeing that as the model improves at one role, it simultaneously faces a stronger opponent, creating an automatic curriculum. Algorithm presents the complete training procedure.
(0)
RL objective. To optimize this shared policy, we apply Monte Carlo policy gradient methods. Using REINFORCE (Williams, 1992), the gradient becomes:
(0) = EE--BBrnTxxole
Σ Vo Polog (yv O, Ro(T) + Σ Vo Vog 1s, Ri(r)
(1)
where T, = {t t mod = p} denotes turns where player p acted. This formulation uses Monte Carlo returns which suffer from high variance, particularly problematic in self-play where the oppo- nent's strategy continuously evolves, making the environment non-stann con-stationary.
Role-conditioned advantage estimation. Self-play zero-sum games implies using the same model to optimize for opposing objectives, since (T) = - -Ro(T). This can lead to unstable training dynamics which impedes learning. To reduce the high variance inherent in multi-agent REINFORCE, we introduce Role-conditioned Advantage Estimation (RAE). In two-player games, with a shared policy, different roles may have different expected returns due to game asym- metries (e.g., first-move advantage in ToeT,cToe, information asymmetry in Kuhn Poker). RAE maintains separate baselines bu., for each game G & and role p {0, 1}, estimating expected return [[Rp(T) for that role in that game. We update these baselines using exponential moving average with decay rate E [0, 1]:
ba.p abc,, + +(1 (1-a) AG, Rp(T) bG.
This provides better variance reduction than global baseline by accounting for role-specifie asym- metries. The variance-reduced policy gradient becomes:
V JSPIRAL (0) = BrecgErr
Σ Ac,((T) Vo log Ta( 8t, | G) (E CET,
Σ
(3)
By centering returns around role-specific expectations, RAE ensures gradient updates reflect genuine learning signal rather than inherent positional advantages. We do not normalize by response length to avoid length bias (Liu et al., 2025c). The complete procedure in Algorithm 1.
Implementation. To implement SPIRAL, develop truly online multi-agent, multi-tur RL system for finetuning LLMs. Our training framework builds on Oat (Liu et al., 2024), which provides interfaces of distributed actor-learner architecture (Espeholt al., 2018). We instantiate actors
5

---

Published as a conference paper at ICLR 2026
Algorithm SPIRAL: Role-Balanced Multi-Turn Self-Play
Require: Policy Te. Games 9 [G...... Gn decay rate [0.1]
1: Initialize baselines ba.p = for all G; E G. E {0,
2: while not converged do
3: // Self-Play Trajectory Collection
4: B +
5:
for k = to K actors in parallel do
6:
Sample game G; 9, initialize 8g G; for turn t = 0, 1, 2,... until terminal do
7:
8:
p - mod 2
b Determine active player
9:
Di p TT( st. Gi)
Generate '[reasonin]]
[action]
di (p) + extract action(y:
10:
11:
di (1-p) 0
Inactive player
(2))
12:
8:41 f- Ti(st, di az end for
13:
14:
Ro < Pi(ST). R1 t Ro
15:
Define (Su, yo), (s1, vn). (ST, vr)}
16:
Add (r, G+) to batch B
b Store trajectory with its game
end for
18:
// Role-Balanced Policy Optimization
for (T, G.) do
20:
for p 10, do
bG.F 5- obb + (1 (p(r) AG. +- Rp(T) bG..P
22:
23:
end for end for
24:
25:
Update on full sequences y, using REINFORCE with advantages AGEp (Eq. 3)
26:
end while
to execute the self-play loop, using vLLM (Kwon et al., 2023) for efficient model inference and TextArena (Guertler et al., 2025) to simulate the language games. The resulting multi-turn, multi- game self-play experiences used to update the LLM via policy gradient methods (Sutton & Barto, 2018), incorporating our proposed Role-conditioned Advantage Estimation in the collocated learner.
4 EXPERIMENTAL RESULTS
We evaluate SPIRAL across diverse model architectures and game environments understand how self-play develops transferable reasoning capabilities. We train on three games from TextArena (Guertler al., 2025): TicTacTToe (spatial reasoning). Kuhn Poker (probabilistic reason- ing), and Simple Negotiation (strategic optimization). Models include Qwen3-4B/8BB-Ban BaaBB-BBBeBee (Yang al., 2025), Llama-3.1-80 In-Irustttt (Dubey al., 2024), and Octothinker -B-heer--BBBseee (Wang et al., 2025a). Training spans 400 steps with 128 samples per step 8 H100 GPUs, using Adam opti- mizer with learning rate 10-6 and temperature 1.0. We evaluate on eight reasoning benchmarks (MATH500. OlympiadBench, Minerva Math, AIME24/25, AMC23, GPQA-Diamond. MMLU-Pro) and seven cut-of-distribution games. Complete implementation details are in Appendix D.
Self-play games transfers to improve reasoning. The central results of this paper are shown in Table 1, which demonstrates that multi-game SPIRAL training achieves up to 10.5% improve- ment reasoning benchmarks without domain-specific data. with Qwen3-4B-B 34.0% to 44.5% average performance (-10.5%). We also compare with supervised fine-tuning (SFT) 25,000 expert game trajectories, generated by Qwen3-32B models, which improves per- formance on several benchmarks including AIME24 and AIME25, revealing that games themselves contain skills relevant to reasoning. However, SPIRAL consistently outperforms SFT across all benchmarks, demonstrating that self-play discovers more effective reasoning strategies than imitat- ing expert demonstrations.
improving from

---

Published as a conference paper at ICLR 2026
Table 1: Reasoning benchmark performance. The "-Kuhn" suffix denotes fine-tuning solely on single game (Kuhn Poker), while the "-Multi" suffix indicates fine-tuning on all three games. SPI- RAL improves reasoning without any domain-speciff training data. "Few evaluation following Qwen3 technical report.
Model
Math500
AIME24
ATME25
Olymplad
AMC-23
Minerva
GPQA-D
MMLUUPro
Owen3-4B-Base
73.4
9.6
42.4
29.4
SFT-KKuhn
74.0
11.0
10.4
36.7
48.6
36.8
33,0
48,8
37.4
-SFT-Multi
74.2
13.7
11.7
37.6
51.1
40.1
37.8
51.3
39.7
+ -SPIRAL-Kubn (Qurs)
76.4
18.2
15.6
38.4
61.2
42.4
37.0
$7.7
43.4
+ SSPIRAI MM-lli (Ours)
78.2.44
19.7.181
13.3,31
41,8,
61.6,192
42.6.132
40.1
58.3.11.
44.5,
Qwen3-SSlBBuxe
77.0
12.1
11.2
33.5
30.6
38.2
38.0"
55.7*
39.5
+ SFF--Mulii
82.8
19.9
15.6
45.9
63.5
40.8
41.6
58.8
46.1
SPIRAL-Mulii (Ours)
86.6.0.
26.2.141
16.8.56
49.6.16.1
65.2.114
46.3.u.
61.1.a4
49.6.
Octothinker-ST-iee
1.7
25.7
-SFT-Multi
66.0
3.3
31.0
23.8
24.9
39.1
27.0
SPIRAL-Mulá (Ours)
68.6.3a
5.3.34
4.8.1a
33.7.ta
43.2,
32.0.c.
33.8.11.
49.3.18.
33.8.
Llama-3.1-SBB -.Bt--33--BBInnttrucc
46.4
4.6
0.7
13.8
23.3
22.8
30,2
49.1
23.9
SFlt-Multi
51.8
4.6
0,7
19.1
23.3
21.7
30,0
48.9
25.0
+ SPIRAL-Muli (Ours)
49.9 49.8.34
4.9a
188,1
17.3
26.0..7
24.6.1.
32.2,20
50.4.1.
25.9.2m
DeepSeek--Dstee Seiei-iiitti---------
90.8
53.0
39.5
$6.9
89.3
48.2
48.6
57.1
60.4
+ SETT-Multi
91.8
49.3
36.6
52.4
88.2
48.2
44.5
55.6
58.3
+ -PPRAL-Mulii (Ours)
93.0...
54.1.1a
40.8.
57.9.
89.3.am
51.1.29
49.6,1.
58.9.14
61.8.14
Recogn
Expected Value Carculation
Cass-by by Case Analysa
Reasoning (Game)
Reasoning Pattern (Math)
Benchmarks
71
G2
30.6
Traces (%)
60
60
48
45
37.5
Score
36
40
51
30
28
18
31.2
12
10
Early
Mid
Late
Early
Mid
Late
Early
Mid
Late
Training Stage
Training Stage
Training Stage
Figure 4: Evolution of reasoning patterns during SPIRAL training and their transfer to math- ematical reasoning. We track three core reasoning patterns (Pattern Recognition, Expected Value Calculation, and Case-by-Case Analysis) across 290 game trajectories and 46,792 math solutions. Left: Patterns increase via training on games, with Expected Value Calculation reaching 78% by late training. Middle: These patterns transfer to mathematical reasoning with varying effectiveness: Case-by-Case Analysis maintains high transfer (72% to 71%). Pattern Recognition shows amplifi- cation (35% to 45%), while Expected Value Calculation transfers more selectively (78% to 28%).
Right: Math benchmark scores improve from 31.2 to 39.6 as these reasoning patterns develop, demonstrating that game game-leamed strategies enhance mathematical problem-solvin capabilities.
4.1 UNDERSTANDING WHY SPIRAL WORKS
Emergent reasoning patterns learned on games transfer to math questions. To understand how games can improve reasoning performance, we analyzed chain-of-thought patterns using GPT- 4.1 (OpenAI, 2025) to classify 290 game trajectories and 46,792 math solutions. Table 2 illustrates three core patterns that emerge during gameplay and transfer to mathematics. Figure 4 tracks their evolution. In the late training stage, the transfer from games to math is clear: Case-by-Cat Analysis transfers near- serrereefeetly (72% to 71%), Pattern Recognition amplifies (35% to 45%). while Expected Value Calculation transfers selectively (78% to 28%). The success of this transfer may stem from two factors. This transfer likely works because evolving opponents prevent memorization while games teach pure reasoning strategies without requiring mathematical knowledge.

---

Published as a conference paper at ICLR 2026
Table 2: Representative reasoning patterns and their manifestations in games and mathematics. We use an L-M-aa-a-judge framework to evaluate the frequency of these patterns over training.
Patiern
Game Context (Kuhn Poker)
Math Transfer
Case-by-Case
<output>>
<output>>> To salve m² +223" = mm(2*1 1):
Analysis
Case Fold:
You lose chip. You have chips
now.
Case 0: m²
m, real solutions
Case 2- Call: You have 0% chance of winning, so
Case n = 1: m. 6 = 3m,, no real solutions
you will have chip you lose.
Case 2: m" 18 7m, real solutions
Since losing I chip (fold) is better than losing 2 chips (call and lose), the best action fold </output>
Case = 3: m 54 15m gives m = 6,9
(valid)
Case 4: m 162 31m, no integer
solutions
Case 5: m² + 186 63m gives
(valid) </ //output>>
Expected
Value
souttput>>> pust With againstt opponent's 's bet:
<output>> average A+ BC DEF over
Calenlation
EVicall) = P(win) x 2- P(lose) 2
permutations:
-0x 1xx22 2-1 1 x =2
E[A] Zgi =
(V(fll)) -I (lose ante)
EICC| (E(B))EIC) (-
Since EV(tol)) > EV(call), fold optimal.
E[DEF] = (EDD)NE))()))))))))E)E))))E>))
</output>
Total = - 27
</output>>
Pattern Recogul- tion
<out.pu>> Previous rounds; Player had K both times and won. Pattern identilied: Player likely has strong cards bluff's consistently. Given have (weak), betting might exploit their uggressive calling pattem. </output>
coutput> Sum of three consecutive integers (n - 1) + + (n 1) = 3n.
Pattern recognized: sum is always divisible 3. For perfect cube; 3n = A.a, so k must be divisible by 3.
Smallest:
3n 27 </ </output>
Adaptive curriculum beats static opponents. Self-play creates an auto- matic curriculum that adapts to model capabilities. Figure compares self- play against fixed opponents (Random, Mistral-Smalla- Gemini-2.0-Flasse Flash--l---
Random opponents cause collapse: al- though they provide randomized rewards with positive expected value, similar to spurious rewards that might upweight cer- tain base model behaviors (Shao et al., 2025), we observe this is insufficient to improve performance. Fixed model op- ponents enable initial learning but plateau once exploitable strategies are found. That fixed opponents like Gemini yield smaller gains reveals the effects are not merely from learning game mechanics spurious rewards (Shao et al., 2025), but specifically from the adaptive curriculum. Unlike static baselines, self-play's 's con- tinuously evolving challenge forces genuine reasoning development rather than exploitation of static patterns. Table 3 confirms this: self-play maintains 50-52% win rates while fixed-opponent training rises from 0% 62.5%, indicating exploitation rather than continued learning.
T: Win rates at different training stages of Gemini Opponent and Self-Play vs its opponent.
Gemini Opponent Win Rate vs Germini-2.0-lunsh-...
Self-Play Win Rate vs Self (L-16)
Training Stuge
Step 16 Step
0.0%
$2.3%
37.5%
51.7%
Step 384
62.5%
50.9%
Different games develop complementary skills. Each game cultivates distinct cognitive abilities that transfer related domains; specifically, selected TicTac target spatial reasoning, Kuhn Poker for probabilistic inference, and Simple Negotiation for strategic optimization. Table 4 tests how well agents trained on a specific game ('specialists') transfer to novel out-of-distribun games. We find specialists transfer effectively to similar out-of-distribution games: Tic Tac spe- cialists achieve 56.0% on Snake (spatial), Poker specialists dominate Pig Dice at 91.7% (proba- bilistic), Negotiation specialists win 55.8% on Truth and Deception (strategic). Multi- training combines these skills synergistically, as shown in Table 5, which shows win-rate against Gemini- 2.0-Flash². Multi-game agents achieve 59.5% average performance, outperforming all single-game
(OOD)
Accessed via httthht:////eneeeee a //og//ooooloooo/ooooooo // /emim//mmnnn//m/ ni -2- -0-s-ssh---- -2.0-91aah-1ite-000.. lite-001.
Accessed via https://cpenrooterr a /googl eeeooggle///mm/ii gemi ni -2.0- 22.0-11ahh 001.
8

---

Published as a conference paper at ICLR 2026
Game evaluation
Math evaluation
General evaluation
Average win rate (%)
40
Everage score
30
Average score
20
20
10
10
100
200
300
400
100
200
300
400
100
200
300
400
Pollcy iteration step
Policy iteration step
Policy iteration step
Random Opponent
Mistral Opponent
Gemini Opponent
Self-Play (Ours)
Figure 5: Performance comparison of self-play training and fixed-opponent baselines. All evalu- ations are averaged multiple games/benchmarks (see App. D.3). Mistral Opponent refers to against Mistral-Small-3; Gemini Opponent refers to against Semini-2.0-Flash-LLic
specialists (best: 52.9%), using fixed budget of 400 training steps determined by convergence analysis.
Table 4: Game specialists excel at both their training games and unseen games requiring similar cognitive skills. Each cell shows the win rate in head-to-hear competition between specialists (c.g., 57.5% means Toe specialist wins 57.5% of games against the other two specialists Tic- ToeT)o)). Bold indicates best performance in each column.
Training Games
OOD Games (Similar Skills)
TicTucToe
Poker Simple Negotiation
Pig Truth Deception
(Sputial)
(Probabilistic)
(Strategic)
TicThcToe Specialist 57.5%
45,1%
30.4%
56.0%
56.7%
48.75
Poker Specialist
45.5%
64.2% %
37.7%
42.5%
91.7%
Negotiation Specialist 40.5%
40.2%
62.7%
41.0S.
1.1%
55.8%
Table 5: Multi-game training achieves competitive performance across all training games while excelling at novel composite challenges. All win rates shown are against Gemini-2.0-Fian as a fixed opponent. The multi-game model outperforms all specialists on average, demonstrating that diverse game training develops more flexible reasoning.
Training Games KuhnPuker
OOD Games
Model
TicTucToe
Simple Negotiation
Snuke
Dice
Truth and Deception
Average
Base Model
17.5%
21.5%
15.6%
7,8%
0.2%
49.6%
18.7%
Random Policy
24.5%
31.3%
8.2%
N/A
N/A
N/A
N/A
Model
46.2%
Single-Game Specialists
TicTacToe Specialist
24.4%
30.5%
Kuhn Poker Specialist
31.0%
48.5%
28.7%
27.7%
98. 8%
81.6%
52.7%
Simple Negotiation Specislist
27.7%
16.8%
39.1%
26.4%
98.6%
82.8%
48.6%
Multi-Game Model
54.3%
53.9%
33.2%
31.6%
99.8%
84.0%
59.5%
Role-conditioned Advantage
Response length (char)
General evaluation
Estimation proves essential
48
for stable training. Figure 6
4000
REINFORCE with RAE REINFORCE
Average score (%)
shows that without RAE,
3000
models suffer catastrophic
44
thinking collapse within 100
2000
42
policy iterations-rcasoning trace
1000
lengths plummet from ~2,000 to near-zero characters, with models generating degenerate outputs like \boxed ; General reasoning performance correspondingly drops from 44% to 40%. In contrast, RE- INFORCE with RAE maintains
80
0
100
200
300
400
400
Policy iteration step
Policy iteration
Figure 6: Training dynamics comparing REINFORCE with RAE (orange) versus vanilla REINFORCE (gray). RAE maintains sta- ble performance across all metrics while vanilla REINFORCE suffers catastrophic thinking collapse. Left: Response length re- veals thinking collapse where models stop generating reasoning traces; Right: Performance on general reasoning benchmarks.

---

Published as a conference paper at ICLR 2026
stable response lengths around 1,300-1,500 characters and improves performance from 40% to 47%. RAE achieves this stability by centering returns around role-specific baselines, preventing gradient variance from driving policies toward degenerate solutions.
CONCLUSION
We introduced SPIRAL, enabling language models to develop reasoning capabilities through com- petitive self-play without human-curated data. Our technical contributions include fully online multi-turn MARL system for LLMs and Role-conditioned Advantage Estimation (RAE), which prevents thinking collapse in zero-sum games. Empirically, multi-game SPIRAL training improves reasoning benchmarks by up to 10.5% across diverse model architectures, surpassing supervised fine-tuning on 25,000 expert game trajectories. Different games develop distinct transferable skills (spatial, probabilistic, strategic) that combine synergistically. Analysis reveals that competitive gameplay forces discovery of reasoning patterns (case-by-case analysis. expected value calculation, pattern recognition) that transfer to academic domains.
SPIRAL demonstrates that simple games can unlock complex reasoning without domain-specific data. Future work could expand to cooperative games, incorporate partial observability, and design games targeting specific reasoning weaknesses. Understanding game-skill mappings could enable principled environment design for autonomous reasoning development.
ACKNOWLEDGEMENT
We thank Xidong Feng and Runji Lin for their helpful discussions and support throughout this project. We thank John Schulman for insightful feedback and encouragement. We would like to thank Thinking Machine Lab and Modal Lab for providing compute credits that supported periments. We also thank the TextArena team for maintaining the game environments used this work. This research was supported by the Cooperative AI Foundation, the UW-Amazon Science Gift Hub, Sony Research Award, UW-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAD,, the Microsoft Accelerate Foundation Models Research Program, Character.AI, DoorDash, and the Schmidt A12050 Fellows program. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): FA8650-23-C-7733 Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA.
REPRODUCIBILITY STATEMENT
We provided the experiment code at https: http:///mmmpaaaattssppeeeee.e also provided the training settings in the experiment section ($4) and Appendix D. The experiments are run with an 8 H100 GPU cluster, and all calls for proprietary LLMS are via the official API or OpenRouter'.
We have
REFERENCES
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023. URL https:
org. abs awas////. 2211 18232.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Richard Bellman. markovian decision process. Journal of mathematics and mechanies, 1957.
hhttps://pperrouter.tt
10

---

Published as a conference paper at ICLR 2026
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1ww.1199..0600, 2019.
Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yer Yee K Wong. Spe: Evolving self-play critic via adversarial games for Ilm reasoning. arXiv preprint arXiv:2504.191 19162, 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. In ICML, 2024.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Xiaolong Li, et al. Self- playing adversarial language game enhances IIm reasoning. Advances in Neural Information Processing Systems. 37:126515-126543, 2024.
DeepSeek Team. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learn- ing. arXiv preprint arXiv:2401.0000 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-1011111 2023.
Aaron Dharna, Cong Lu, and Jeff Clune. Foundation model self-play: Open-ended strategy innova- tion via foundation models. arXiv preprint 250.:::22270066666. 2025.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gthench: Uncovering the strategic reasoning limitations of Iims via game-theoretic evaluations. arXiv preprint arXiv:2402. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018
FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 578(6624))1067-10777 2022.
Xidong Feng, Bo Liu, Yan Song, Haotian Fu, Ziyu Wan, Girish A Koushik, Zhiyuan Hu, Mengyue Yang, Ying Wen, and Jun Wang. Natural language reinforcement learning. arXiv preprint arXiv:2411.1422 14251, 2024.
Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. Textarena, 2025. URL https: arxiv. www.orrggooo.... abs 2tabs/2555 2200/2504000 442.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Athul Paul Jacob, Abhishek Gupta, and Jacob Andreas. Emergent linguistic phenomena in multi- agent communication games. arXiv preprint arXiv:2205.0500 059844, 2022.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu. and Rosalind Picard. Way off- off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 000666 2019.

---

Published as a conference paper at ICLR 2026
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. vrover-verifier games improve legibility of Ilm outputs, 2024. URL https://axxiv..ott abs/2407..1 13692..
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
John Langford and Tong Zhang. The epoch-greeds algorithm for information. Advances in neural information processing systems, 20, 2007.
bandits with side
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self- self-play in non-zero- sum games, 2024. URL https:/ arxiv. org/ abs/ 18872....
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.220 20050,, 2023.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sain- bayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510. 2025a.
Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models, 2025b. URL https: /arxiv.vvvv/////a///aaa//aaaavva org/ /20b.///55... 64.
Zichen Liu, Changyu Chen, Xinyi Wan, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research- friendly framework for IIm online alignment. https: /tttt/gbtbb//t//g///gg//g.g.hh com/ sail-sg/oat 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint MxXiv:2503.20783, 2025c.
MAA. American invitational mathematics examination (AIME). Mathematics Competition Series, n.d.a. URL https: //tpa/////aaaaaaaa wwt.:///aaaa..comeettt..o.omm aime.
MAA. American mathematics competitions (AMC 10/12). Mathematics Competition Series, n.d.b.
URL httph:tt//// /maa.aaaaaa// org /matt/mmah-aaa wwwp:/oottoooooooiiiioieeeeeeeee www.:/ttttttttteeeeoeeeooiotttte
OpenAl. Learning to reason with 1lms. OpenAl Blog, 2024. URL //pp//ai.com//nde/
OpenAl. Introducing deep research.
https:
introducing-deee deep---eeepeaeee research/ wwwpeeeeaeereeeereeeeeeeeo 20255 Accessed: 2025-09-24.
OpenAl. GPT-4.1. OpenAl API, 2025. URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, et al. Balrog: Bench- marking agentic Im and vlm reasoning games. arXiv preprint arX1v:2411.13543, 2024.
12

---

Published as a conference paper at ICLR 2026
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo- Moo-Kyung Kim. for collaborative large language models with reinforcement
Maporl: Multi-agent learning. arXiv preprint S1.ii::2522.14453 18439, 2025.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly q-function. In Conference Language Modeling, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024.
Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations, arXiv preprint arXiv:2412.01441, 2024.
Bidipta Sarkar, Warren Xia, C Karen Liu, and Dorsa Sadigh. Training language models for social deduction with multi-agent reinforcement learning. arXiv preprint arXiv:2502.0606) 2025.
Tom Schaul. Boundless socratic learning with language games. arXiv preprint ari1v:4111.1660 2024.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint
102:2206611997 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 11X.011712.01815, 2017.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations (ICLR), 2018
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38 (3):58-68, 1995.
Jingqi Tong, Jixin Tang, Hangcheng Li, Yurong Mou, Ming Zhang, Jun Zhao, Yanbo Wen, Fan Song, Jiahao Zhan, Yuyang Lu, al. Code2logic: Game-code-driven data synthesis for enhancing vims general reasoning. arXiv preprint arXiv:2505. 1388. 2025.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-basee feedback. arXiv preprint urXiv:2211.14273 2022.
Vivek Verma, David Huang, William Chen, Dan Klein, and Nicholas Tomlin. Measuring general intelligence with generated games. arXiv preprint arXiv:2505.0720 .02000...2...... 2025.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song. Hanjing Wang, Linyi Yang. Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, et al. Rema: Learning to meta-think for llms with multi-agent reinforcement learning. arXiv preprint MxXxv:2503.09501 2025.
13

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation, 2023. URL https: /awxivvvviv.v////i//.e.///aaaav/ wwweii.rrg/.oeeeo. org/orgg/ao http////////// 22100 01320.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arxiv:250 20512. 2025a.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, arXiv preprint arXiv:2504.20073, 2025b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of- of-o-hughttttt prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837, 2022.
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8:229-256, 1992.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing 11m reasoning with rule-based reinforcement learning, 2025a. URL https: /aaess//aarxaee /waa/aaavvvoo/o//.///aaa org/ abs/ /ww./552..a. 2tt.//222...1..
Yunfei Xie, Yinsong Ma, Shiyi Lan, Alan Yuille, Junfei Xiao, and Chen Wei. Play to generalize: Learning to reason through game play. arXiv preprint arXiv:2506.0888 2025b.
Huajian Xin, ZZ Ren, Junxiao Song. Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Deepseek-prover-vl.
Zhang, Xuan Lu, Qiushi Du, al. Deepseek prover-v1. 5: Hamnessing proof assistant feedback
for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408.08 08140000081..... 2024.
Fangzhi Xu et al. Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning, 2025a.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study werewolf, 2024. URL https: httttp:://axxii.o.// arxiv. org abs 2209/339..000 04658.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game, 2025b. URL https: h //tp:///a/iviii// org/ /31C. 18940.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.093: 009.8888, 2025.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 2024.
Xiaoqing Zhang, Huabin Zheng, Ang Lv, Yuhan Liu, Zirui Song, Flood Sung, Xiuying Chen, and Rui Yan. Divide-fuse-conueee Eliciting" aha moments" in multi-scenario games. arXiv preprint arXiv:2505.1640) 2025.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as mastermind: survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230 2024.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.083 003355. 2025.
14

---

Published as a conference paper at ICLR 2026
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers, 2025. URL https: //axxiv.rrg/aaa/205.. 21493.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-22: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 11931, 2024.
15

---

Published as a conference paper at ICLR 2026
This appendix provides comprehensive details supporting main findings. App. A documents the use of large language models in our analysis. App. discusses limitations of our approach including reliance designed game environments, computational requirements, evaluation constraints, and potential reward hacking risks. App. C provides the detailed formulations of turn-level MDPs and two-player zero-sum Markov games referenced in the main paper, showing how SFT and RLVR adapt to these frameworks. App. D presents complete implementation details including game en- vironment observations, hyperparameter configurations, and evaluation settings for all benchmarks. App. E provides extended benchmark results across multiple base models, an extended analysis of our RAE ablation study with additional training dynamics, and a detailed case study showing the evolution of case-by-case analysis in mathematical problem-solving. App. F describes our system- atic bottom-up approach for discovering and quantifying reasoning pattern transfer, including our GPT-4. lassissed analysis framework. Finally, App. G specifies all game environments, detailing both training games ToeeTacToo, Kuhn Poker, Simple Negotiation) and out-of-distribution evalua- tion games (Snake, Pig Dice, Truth and Deception).
A LARGE LANGUAGE MODEL. USAGE
We used large language models (LLMs) only for language refinement tasks, including grammar checking, phrasing adjustments, and enhancing readability. We also used Deep Research (OpenAI, 2025) to assist with related work search. Besides these, all scientific ideas, experiments, analyses, and results are the sole contributions of the authors.
B LIMITATIONS
Our study, while promising, has several limitations that offer avenues for future research.
Reliance on Designed Game Environments A core limitation is the dependency on engincered game environments. Although SPIRAL eliminates the need for human-curated problem datasets, it shifts the dependency to well-designed games. The games used in our experiments, such as the Tic- tactoe and Khun Poker, relatively simple and feature dense rewards. It is open question how well this approach scales to more complex, open-ended environments with sparse rewards, such Minecraft or realistic roboties simulations. The design of the game environment itself may im- plicitly encode biases or heuristics that influence the agent's 's learned reasoning strategies, potentially limiting their generality.
Computational Cost and Scalability The computational requirements for training are substan- tial. Each experimental run demanded 8 H100 GPUs for approximately 25 hours, which may be prohibitive for many research groups. Furthermore, we observed that performance gains began to plateau after extended training periods. This suggests that simply scaling up the training dura- tion with the current framework may yield diminishing returns, and more efficient algorithms or architectural improvements are necessary for further progress.
Evaluation and Transferability Our evaluation, while comprehensive, has two key constraints:
Focus on Academic Benchmarks: We primarily assessed reasoning on established aca- demic benchmarks like MATH and GPQA. These benchmarks are excellent for measuring formal and scientific reasoning but do not capture the full spectrum of self-play.
Zero-Shot Evaluation: The strict zero-shot evaluation setting tests for direct transfer but may not fully reveal the model's potential. Fine-tuning on small set target domain ex- amples could potentially unlock significantly better performance, possibility not explored in this work.
Potential for Reward Hacking Like many reinforcement learning systems, SPIRAL is suscepti- ble reward hacking. An agent might discover policies that maximize the in-game score without learning the intended underlying reasoning skill. For instance, could exploit a bug in the game physics find repetitive, degenerate strategy that succeeds for narrow set of problems. While
16

---

Published as a conference paper at ICLR 2026
did not observe significant instances of this, it remains risk, especially in more complex envi- ronments where robust reward shaping is challenging.
C
PRELIMINARIES
This section provides the mathematical foundations and formal definitions underlying the SPIRAL framework, including turn-level MDPs, two-player zero-sum Markov games, and how existing train- ing paradigms adapt to these formulations.
C.I TURN-LEVEL MARKOV DECISION PROCESSES (MDPS).
Language model training traditionally formulates generation as a token-level MDP (Bellman, 1957; Rafailov et al., 2024) where each action is a single token from vocabulary V. For multi-turn rea- soning and game-playing, we instead adopt a turn-level MDP formulation .M Here, states S represent complete contexts (e.g., game configurations, problem states, or conversa- tion histories), actions are complete responses (containing many tokens), the transition function T S x A + (SS) determines state dynamics, S A + R provides immediate rewards, and 7 E [0. 1] is the discount factor. The return is defined as the discounted sum of rewards:
R(T) = Ei=
The key distinction: in token-level MDPs, each decision outputs one token; in turn-level MDPs, each decision produces complete multi-token response before transitioning. At each turn t, the language model observes state 8, and generates:
Vt (think) (/think) (ttnn////aswerruu(aansseee)
where externalizes reasoning and a E AA is the executable action. (See App. C for how existing SFT and RLVR paradigms adapt turn-level MDPs.)
C.2 TWO-PLAYER ZERO-SUM MARKOV GAMES.
We extend the single-agent MDP competitive settings with two-player zero-sum Markov game (Littman, 1994) G = (S, As, A1, T, 7),, where Ao and A1 are the action spaces for player and player respectively. The zero-sum property requires:
ru(s, (0), (l) + ri(s,, (0)) ()) Vs, (0)
(5)
where (0)) E Ag and (1) E A denote actions taken by each player. Given trajectory т (11,, (0) (1) aat )T the returns satisfy (T) = -R((r).
=
C.3 SUPERVISED FINE-TUNING (SFT) IN TURN-LEVEL MDPS
In the turn-level setting, SFT requires a dataset Dsrr = {(s;, ci at > of states with expert reasoning traces of and actions of. The model learns imitate complete turn-level responses:
(S) - (s a D>D [log To(c", [[s]].
6
Note that in single-turn settings where each state s appears only once, SFT reduces to standard behavior cloning. The key limitation remains: SFT requires expensive human annotation of both reasoning traces and final answers.
C.4 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS (RLVR) IN TURN-LEVEL MDPS
RLVR (DeepSeck Team, 2024) eliminates the need for reasoning supervision. requiring only state- answer pairs DRiVR = ((*i, a N the turn-level formulation:
(R) Be-Pava.v~rr(lee) [r(s,a)),
where 0) T(a a*) indicates answer correctness and y contains both reasoning and action.
17

---

Published as a conference paper at ICLR 2026
In single-turn settings without subsequent interactions, RLVR reduces contextual bandit prob- lem (Langford & Zhang, 2007). Recent works on mathematical (Shao et al., 2024; DeepSeek Team, 2024) and code reasoning (Zhu et al., 2024; Xin et al., 2024) show that even this simplified bandit- style RIVR can unlock sophisticated reasoning. However, these approaches still require human- curated problem sets DREVR which SPIRAL eliminates through self-play.
D EXPERIMENTAL SETUP DETAILS
This section provides complete implementation details for reproducing our experiments. We begin with visual examples of game environments, followed by our hyperparameter configurations.
D.1 GAME ENVIRONMENT OBSERVATIONS
The language models receive structured text observations from each game environment. Fig. shows example observations from our three training games: TacTe, Kuhn Poker, and Simple Negotiation. These observations serve as the input prompts at each turn, providing complete game state information in natural language format.
Plaeer
opponertts stnnes slones appeario
(CAMMt] Veu ane Itayer i a round game kahh Poher.
Yourar Player Negotyttion
have
back made
On your dhocee one call it sumbersd
Kihnn Toller uses & Braed deck with (.O, X lowest highea))
Out the kital value your resourees increses
Index your stone them.
resornee and succated values you commity hove
poresomph, board
player
and
consinees sourds
|Woodd
Qle
Valoe:
Your ofjective to lorn continuoca lis three your
The player with most chips ofter all nounds wins
Nosepl
Value:
alcmes any now, oxumm, clagoral.
Qh
Valuer21
Camen Boand
[etokT Pae aithout betting (onty if but tabaa)
At Mrccc timm tilk orrroppreeccor malee
[eet add chip to the por lorly no bet the tabk))
trado crddecllez.
[cal] oppenant's adding ship
hollooologggggggeee fer
[[Hbe]]
33:415
Fomoo (Cilee Offired Resvicess Requested
Recornee]
Tromple
Sheep)
[CAME
To [Dwey] to deey on incoring (debuit). inckude addittonal tex betooo and theoe
inceniing
Acsibire Meree (0) DLL (SL 17.
Your to &
wwwialle artioe (cheels] (bwe)
The gune ladt for turn sotal.
TicTacToe
Kuhn Poker
Simple Negotiation
Figure 7: Example observations of three training game environments.
For games with partial observability such as Kuhn Poker and Simple Negotiation, we maintain Markovian state representations by concatenating historical actions into the current state st- This ensures the model has sufficient information for decision making despite hidden information.
Similarly. Fig. 8 presents observations from five evaluation environments used to test out-of- distribution generalization. These games were never seen during training, allowing us to assess whether learned skills transfer novel game mechanies.
D.2 HYPERPARAMETER CONFIGURATION
Table 6 presents the complete hyperparameter settings used across all experiments. These configura- tions were selected through preliminary experiments to balance training stability and computational efficiency. Please see our open-source codebase for complete and reproducible experiment exam- ple.
These hyperparameters remain fixed across all game environments and model scales to ensure fair comparison. The distributed training infrastructure utilizes 8 H100 GPUs, with parallel actors gen- erating game trajectories while a centralized learner performs synchronous policy updates. On this hardware setup, the compute budget per experiment was approximately 25 hours for Qwen3-4B, 28 hours for Qwen3-8B, and 16 hours for both Llama3.1-88 1--.---8--nnsteee Instrutttt and Octothinker-80 For SFT, we use Qwen3-322 (Yang et al., 2025) models to self-play on the targeted games, generating 25k winning trajectories and crafting them the SFT dataset.
18

---

Published as a conference paper at ICLR 2026
[CAMRF) biyee Srake on Bet grid.
[GAMI You Playcrollayaae
(GAMR] Your are Playert, and you we the Desiverin the
You cerpal mulecd Yalld mono.
firre
Trruts and Douptiai ganee
your tume you on ether [nol] thoal]
Obpective begant leelggg the
Bull
Fadt l: The blanetingd odopua humnze humara.. twnarel
Nahhon sor then lass 40 lunaa
full Lose trm weal and tum
your
aivegggge octopus
Yourgeal goali canvin the thodle player dhoce the
Ihha itss your tum, you wu
and turn
feds
Rosporal with [oal] toalll the due [bold] boni
Arke that the other player will make their gress
paett.
your slayly type your massoge
This fine rdll your
Acattotle actione pood or "hoad""
Snake
Pig Dice
Truth And Deception
are Player Spleyer game.
GaME You Playerd Gonnect Four.
liudec
Your die symbett x.
OOrycen tumn you may celece
higher quantit Higher
Player take deuppig thair dacana uncul calamno 1o
botho thecurrent
Call the lest bell
TCAlI?
to someet ther discvertiall.
On your timm enter colurte number equared brccum
la actual count of tuat taca valee all dics
make
than the full the bikler leos une tie
GaME Board date
payer ches tettmmnttte The renaining player wire.
01 25458
Remaining Your curent Dheearee 1.6.&.
Liars Dice
Connect Four
Figure Example observations of five evaluation game environments.
Parameter
Value
ACTOR
Maximum response length
8192 tokens
Sampling temperature (top top k)
1.0
(1.0,-1)
LEARNER
Optimizer
AdamW
Adam parameters (31, 32)
(0.9, 0.95)
Weight decay
0.0
Gradient norm clipping
1.0
Batch size
128
Discount factor
1.0
EMA decay rate
0.95
Learning rate scheduler
Constant
Learning rate
10-6 6
Inner proximal update epoch
2
KL loss coefficient
0.0
KL penalty coefficient
0.0
Policy clipping parameter
0.2
Table 6: Hyperparameter configurations used in all experiments.
D.3 EVALUATION SETTINGS
To investigate whether the reasoning abilities developed through gameplay could transfer to non- game contexts, we evaluate models suite of established benchmarks. All evaluations these benchmarks are conducted in a zero-shot setting* to determine if game-induced reasoning could be successfully transferred general problem-solving We use sampling temperature of 0.6 and top-p of 0.95 for all evaluations.
"Except for the base model, for which provide few-shot examples that follow the Qwen3 Report settings.
19

---

Published as a conference paper at ICLR 2026
Math Benchmarks. For mathematical reasoning, use MATH500 (Hendrycks et al., 2021), OlympiadBench (He et al., 2024). Minerva Math (Lewkowyez et al., 2022). AIME24, AIME25 (MAA, a). and AMC23 (MAA, b) datasets, which cover wide range of topics includ- ing algebra, geometry, and competitive mathematics. Following the settings Zhou et al. (2025), report AVG @ 32 for AIME24, AIME25 and AMC23; and PASS @ for other math benchmarks.
General Reasoning Benchmarks. For general reasoning, we utilize GPQA-Diamond (Rein al., 2024), which consists of graduate-level science questions, and MMLU-Pro (Wang et al., 2024), benchmark for multidisciplinary knowledge.
E ADDITIONAL. RESULTS AND ANALYSIS
E. DETAILED EVOLUTION OF CASE-BY-CASH ANALYSIS
To understand how reasoning patterns develop during training, we tracked the evolution of case-by- case analysis across checkpoints. Table 7 shows a concrete example from Minerva Math Problem 135, illustrating how models progressively develop structured reasoning.
This progression demonstrates how competitive self-play forces models to develop increasingly structured approaches. Early attempts show unorganized reasoning, while later checkpoints hibit clear case separation and systematic analysis, pattern that emerges from game playing and transfers mathematical problem solving.
E.2 EXTENDED ANALYSIS OF RAE ABLATION STUDY
Our ablation study of Role-conditioned Advantage Estimation reveals additional insights beyond those presented in Section Figure 9 shows three additional metries that further demonstrate RAE's critical role in stable self-play training.
Game evaluation
Math evaiuation
Policy gradient norm
0.20
30
Averrae score
0.15
Awerage win
25-
0.10
20
ORCE
0.05
Vanilla REINFORCE
15
0.00
200
100 200
100
200
300
400
Policy Reeration stop
Policy iteration step
Policy iteration step
Figure 9; Extended training dynamics comparing REINFORCE with RAE versus vanilla REIN- FORCE (continued from Figure 6). Left: Game win rates showing RAE achieves faster initial learning compared to vanilla REINFORCE. Middle: Math reasoning performance crashes from 35% to 12% without RAE. Right: Policy gradient norms exhibit instability then collapse to near- zero without RAE, while RAE maintains stable gradients around 0.1.
Game Performance. The left panel shows that REINFORCE with RAE learns significantly faster, rapidly reaching 35% win rate while vanilla REINFORCE exhibits slower initial learning. RAE's superior learning efficiency demonstrates the benefits of role-conditioned advantage estimation for accelerating policy improvement.
Math Reasoning Collapse. The middle panel reveals the most dramatic failure: math reasoning performance without RAE crashes from 35% to 12% at around step 150 (a 66% relative decrease). This collapse occurs precisely when models stop generating reasoning traces, confirming that think- ing collapse directly causes reasoning failure.
Gradient Stability. The right panel shows policy gradient norms, revealing the underlying optimiza- tion dynamics. Without RAE, gradients exhibit high variance with erratic spikes before collapsing to near-zer zero after step 200, indicating convergence to a degenerate policy. RAE maintains stable gradient norms around 1.1 throughout training, enabling continuous improvement.
20

---

Published as a conference paper at ICLR 2026
Table 7: Case-by-Case Analysis in Minerva Problem 135
Stage
Problem: Round cosmic ages to nearest power of 10
Early(Ckpt 16)
Approach: Lumped reasoning without explicir case separation
<output> The age when helium formed is approximately 10 minutes... The age when hydrogen formed is 377,000 years... current age is 13.8 Gyr. Rounding to nearest
power of we get: 14 Gyr </output>>
Case Structure: No explicit cases subproblems handled in continuous text
Result: Subproblem 2: 14 Gyr x (should be 10)
Mid(Ckpt 128)
Approach: Explicit enumeration of cases with structured analysis <output>
Case 1 Subproblem 0:
Understanding the Context: BBN at minutes
Nearest Power 10: 10 10
3. Final Answer: minute
Case 2 Subproblem 1:
1. Understanding the Context: Recombination at <>77,,000 years 10 100, 000
Nearest Power
Final Answer: 400000 years
Case 3 Suuproblemm 2:
Understanding the Context: Current age 13.8 Gyr
2. Nearest Power of 10: Identifies _0th correctly
3. Final Answer: 13.8 Gyr </output>
Case Structure: Clear case separation with consistent 3-step process
Result: Still incorrect final rounding X
Late(Ckpt 400)
Approach: Systematic case-by-case with correct execution
<output>
Case 1 Subproblem 0:
1. Understand Context: He formation in carly universe
2. Estimate Time: ~02 minutes after Big Bang
3. Round to Power of 10: 20 minutes
Case 2 Subproblemm 1:
Understand Context: atom formation (recombination)
Estimate Time: ~380,000 years
Round to Power 10: 100000 years
Case 3 Subproblem 2:
1. Understand Context: Current universe
Convert Units: 13.8 billion years 13.8 Gyr
Round to Power of 10: 13.8 10 10 Gyr </cutput>>
Case Structure: Complete systematic enumeration with correct logic
Result: All cases solved correctly
21

---

Published as a conference paper at ICLR 2026
These additional results reinforce main finding: self-play alone is insufficient for stable reason- ing development. Proper variance reduction through role-specific baselines is essential to prevent models from converging to degenerate policies that abandon reasoning in favor of minimal outputs.
E.3 GAME GENERALIZATION
We conducted additional experiments on higher-complexity variants of our training games (5x5 Tic- Tac-Toe, 5-Card Kuhn Poker, and 8-Resource Negotiation) to assess scalability.
Table 8: Generalization performance on increased complexity environments. SPIRAL retains sig-
nificantly higher performance on out-of-distribution (OOD) tasks compared to SFT.
Environment
Setting
Qwen3-4B Base
SFT
SPIRAL (Ours)
3x3 (Train)
17.5
46.9
54.3
Tic-Tac-Toe
5x5 (OOD)
5.5
17.8
27.3
3-Card (Train) 5 5-Card (OOD)
21.5
43.6
53.9
Kuhn Poker
21.9
28.6
50.1
5 Resources (Train)
15.6
26.7
33.2
Simple Negotiation
8 Resources (OOD)
5.5
8.7
31.0
Average (Train) Average (OOD)
18.2
39.1
47.1
11.0
18.4
36.1
Table shows that although SFT has comparable improvement over the base model in in-domain environments (averaging 39.1% vs. 18.2%), it struggles adapt to OOD cases with increased complexity. The SFT model's performance drops to only 18.4%. contrast, SPIRAL demonstrates better generalizability. It not only outperforms SFT in the training settings (averaging 47.1 47.1%) but maintains robust performance in the more complex OOD environments, achieving an average of 36.1%. This suggests that SPIRAL is more capable of handling increased complexity.
E.4 COMPREHENSIVE BENCHMARK RESULTS
Table presents extended results showing SPIRAL's performance across different training configu- rations and base models.
These results reveal several important insights. First, single-game SPIRAL training (40.0-41.4 average) outperforms supervised fine-tuning 25,000 expert examples (38.4% average), validating that self-play can discover more effective reasoning strategies than imitating expert demonstrations. Second, multi-game training (42.3-42.7%% average) consistently outperforms single-gan ganle-game variants, suggesting that diverse cognitive challenges create more robust reasoning capabilities. Third, SPI- RAL improves strong models like SeepSeek-Distil--Oee BBB (from 59.7% to 61.7%), demon- strating that competitive game self-plat elay training can enhance models that already excel at reasoning tasks.
We curated an additional 27k dataset with Qwen3-32B self-play and evaluated SFT on a total of 52k trajectories but training with epoch. As shown in the table, doubling the SFT data yields major improvement (e.g., Qwen3-4B Average 39.7% vs 19,7%, while SPIRAL consistently outperforms both SFT basclines. This confirms that the benefits of SPIRAL stem from the reinforcement learning dynamic rather than simply dataset size.
E.5 STATISTICAL ROBUSTNESS
To address concerns regarding statistical significance, dom seeds (seeds 14, 42, 100). As shown in Table 10, SPIRAL consistently outperforms the SFT baseline with narrow confidence intervals, confirming the robustness of gains.
main experiments with 3 ran-
22

---

Published as a conference paper at ICLR 2026
Table 9: SPIRAL training improves reasoning benchmarks for different base models. We include an additional baseline of SFT on 52k trajectories to demonstrate that simply scaling supervised data does not match the gains from self-play.
Model
MATHES00
AIME'24
AIME'13 Olsmpiallench
AMC-23
Minerv Math
GPQA-D
MMLU-Pro
Average
Qwend Buse Family
3-4B
734
9.6
6.2
33.3
42.4
29.4
30.5
47.3
34.0
SET (Kuha)
74.0
36,7
48.6
33.0
SET (Mall. 25%
74.2
13.7
11.7
37.6
51.1
37.8
29.7
73.4
16.2
323
48.
51.2
Masual Opponent (KalaPakke)
64.0
4.3
298
316
35.6
Gemini (KahnPoker)
69.2
5.2
33M8
29,8
35.3
SPIRAL (TiTTaTTo))
75.6
38.5
55.0
(KulaPoker)
(Negolaaion)
(Mulli-Game)
Fomuly
33.5
38,0
19.9
Game)
33.5
SPIRAL (Multi-Gina
68.6
5.3
33.8
Family
SFT
$1.0
23.2
(Mulii (lute--Gnnn)
49.8
32.2
DeepSrok--Dail---e www.---Daill-ooeeee.. Fumily
DeepSeek
91.8
(Muli--Gane)
93.0
54.1
Table 10: Comprehensive performance comparison across multiple seeds. SPIRAL demonstrates consistent improvement over baselines across diverse benchmarks.
Model
Math500
AIME24
Olympiad
AMC-23
GPQA-D
Average
Owen3-4B-Base
30.6
47.2
SFT-Mulni
74.0
0.6
12.4 1.5
11:2111
37.8
52.2 + 0.8
40.7 40.
37.7
309
39.6
SPIKAL-Mult (Ours)
78.7 1 2.0
18.82:25
15.0
418113
62.0 1.6
4211133
39.1
0.54
$4..0.
E.6 GAME TRAJECTORY STATISTICS
Table 11, we added a comprehensive table with the average number of game lengths, reasoning tokens per step, P1/P2 self-play win rates, and average win-rate and win-rate per game against Gemini-2.0-Flan
Table 11: Game trajectory statistics across training checkpoints. We observe increasing game length and reasoning tokens alongside improved win rates against the fixed opponent.
Training Steps
Arg, (Moses)
Ava. Resouuing Step
Self
Salf-Play Win-Raan
Geminii (Ti/TacToc)
Win-Rate vs. Gemini (Simpll Negotintion)
(Kuhn Poker)
4061
57.3%
128
7.633
1669
31.3%
2M.51
37,555
11.216
256
8:47
1255
$1,255
46.8%
51
42.9%
68 28.815
45.315
Step 384
9.43
1921
39.15
40.0%
66.7%
75.0%
68.5%
56,3%
Stop 410
9.55
2012
07.45
17.6%
67
$1.98
67.552
$6
E.7 SPIRAL AS PART OF THE MID-TRAINING STAGE
We're rumning Base Base-SSPRAL--RVVV experiments to compare with Base-RLVV in terms of con- vergence speed and final performance. Specifically, we use Math- 12th-12k (Lightman et al., 2023) for the standard Math RLVR training, and examine the two variants: Base Sire--SPRALL-RRII and Base -ase-+RLVR- -ALLVR--PPIRAL.....
Our results in Table 12 validate SPIRAL's effectiveness integrating part of the mid-training SPIRAL - RLVR outperforms the Base + RLVR baseline, while applying SPIRAL as a post- RLVR stage (RLVR SPIRAL) yields the highest performance (Avg 48.1). Furthermore, the
23

---

Published as a conference paper at ICLR 2026
Table 12:
Performance comparison of SPIRAL integrated into different training stages. SPIRAL acts robust performance booster both before and after RLVR.
Model
Math500
AIME24
AIME25
Olympiad
AMC-23
Minerva
GPQA-D
MMLU-Pro
Average
Qwen3-4B-Base
73.4
9.6
6.2
33.3
42.4
29.4
30.6
47.2
34.0
RLVR (Math)
83.0
18.4
44.6
62.8
43.4
43.7
36.8
SPIRAL
78.2
19.7
13.3
41.8
61.6
42.6
40.1
58.5
44.5
23.1
59.6
57.4
RLVR SPIRAL
86.1
22.6
18.1
46.0
62.5
44.3
44.7
60.8
47,9
SPIRAL-Multi improves the average score to 59.3, whereas standard SFT leads to performance regression. These findings demonstrate that SPIRAL serves as a robust booster that mitigates the alignment tax often seen in standard fine-tuning stages.
stronger
E.8 INSTRUCT MODEL RESULTS
To better study our method's effectiveness, we perform additional experiments with Qwen3-4B- Instruct-2507.
As shown the Table 13, standard SFT on reasoning trajectories slightly degrades performance. In contrast, SPIRAL successfully reverses this trend. SPIRAL achieves a 2% improvement over the original model and a substantial average gain of 3.6% % over the SFT baseline, reaching a total average of 75.51 %. This demonstrates that self-play refine genuine reasoning capabilities even models that have strong performance.
Table 13: Performance comparison on (a-n3--B-Innstree While standard SFT leads to regression, SPIRAL improves reasoning capabilities across most benchmarks.
Model
Math500
AIME24
AIME25
Olympiad
AMC-23
Minerva
GPQA-D
MMLU-Prr
Average
Swen3-4B-Instruct (Base)
91.2
64.6
47.4
89.6
82.1
87.6
62,0
69.6
74.10
90.8
82.3
71.88
+ SPIRAL-Molti (Ours)
93.5
67.9
49.8
88.8
83,3
83.5
61.8
71.5
75.91
F CASE STUDY METHODOLOGY
This section details our systematic approach to discovering and analyzing reasoning pattern transfer from games to mathematics. Rather than searching for predetermined patterns, we employed bottom-up discovery process to identify what reasoning strategies naturally emerge and transfer between domains.
F.1 DATA COLLECTION FRAMEWORK
Our analysis examined reasoning traces from two sources across three training checkpoints:
Game Trajectories: We collected 290 complete Kuhn Poker games, focusing on winning trajec- tories to identify successful reasoning strategies. Each trajectory includes the complete thought process from initial card observation through final decision.
Mathematical Solutions: We analyzed 46,792 solution attempts across MATH500, AIME, OlympiadBench, and Minerva Math benchmarks. Solutions were categorized by success (score=1) or failure (score=0) to understand which reasoning approaches prove effective.
Temporal Analysis: Checkpoints at steps 0 (initial), 128 (intermediate), and 400 (final) capture the evolution reasoning complexity throughout training.
24

---

Published as a conference paper at ICLR 2026
F.2 BOTTOM- UP PATTERN DISCOVERY PROCESS
Rather than searching for predefined patterns, we employed GPT-4.1 to discover patterns that natu- rally emerge in the data. This bottom-up approach ensures capture the actual reasoning strategies used rather than imposing our expectations.
Pattern Discovery Prompt
Analyze these (domain) reasoning traces using BOTTOM-UP approach. Don't look for predefined patterns. Instead, discover what reasoning patterns actually exist.
REASONING TRACES ((len (sample) } samples from {len (traces) total); (json.dumg) ([t reasoning') for t in sample[:40]], indent-2) Your task:
Read through Ald, the reasoning traces carefully
2. Identify RECURRING patterns structures that appear multiple times
3. Group similar reasoning approaches together
4. each discovered pattern based what actually does
5. Count how many traces use each pattern
6. Provide example quotes for each pattern
Format your response as:
PATTERN 1:
[Descriptive Name]
Description: [What this pattern does|
Count: </tien(sample) traces
Example quotes: 12-3 actual quotes showing this pattern]
Be speci fic and grounded in the actual data. If you see pattern only once, don't include it.
This discovery process revealed three dominant patterns that emerged independently in both do- mains:
1. Case-by-Case Analysis: Systematic enumeration of scenarios
2. Expected Value Calculation: Probabilistic decision-making
3. Pattern Recognition: Identifying regularities and structures
F.3 PROSS-DOMAIY TRANSFER QUANTIFICATION
After discovering patterns in each domain, we compared them to identify which strategies transfer between games and mathematics:
Pattern Comparison Prompt
Compare the reasoning patterns discovered in games mathemat
GAMK PATTERNS:
Igame patterns!
MATE PATTERNS:
[math pacchh_pattrrss
Analyze:
1. Which patterns appear in BOTH domains? (These show transfer)
Waich patterns are unique to each domain?
3. Calculate transfer rates for shared patterns
Identify the most successfully transferred reasoning strategics
5. Explain WEY certain patterns transfer well
Pocus on concrete evidence of transfer, not speculation.
25

---

Published as a conference paper at ICLR 2026
The transfer analysis revealed:
Case-by-Case Analysis shows near-perfect transfer (72% in games to 71% in math) because sys- tematic enumeration represents domain-agnostic structured thinking. Whether analyzing opponent possibilities in Poker or solution branches in mathematics, the core cognitive skill remains identical.
Expected Value Calculation exhibits limited transfer (78% in games to 28% in math) because explicit probabilistic decision-man making appears primarily in probability and optimization problems. Most mathematical domains lack the decision-theorett structure that makes this pattern universally applicable in games.
Pattern Recognition demonstrates amplification during transfer (35% in games 45% in math). Mathematics inherently requires pattern identification, so game training enhances an already- essential mathematical skill, producing stronger pattern recognition than games alone develop.
F.4 PATTERN EVOLUTION ANALYSIS
To understand how reasoning develops during training, tracked pattern emergence across check- points:
Evolution Analysis Prompt
Analyze how reasoning patterns evelve across training checkpoints: [json. dumps petpluuioonnnaallle indent=2) For each checkpoint:
1. Describe the complexity of reasoning
2. Identify patterns that emerge
3. Track how patterns become more sophisticated
4. Show concrete examples of improvement
Focus on the actual evolution you can see in the data.
F,5 CONCRETE TRANSFER EXAMPLE IDENTIFICATION
To validate transfer claims, we identified parallel reasoning structures across domains:
Transfer Example Prompt
Find the clearest examples of reasening transfer from games mathematicos (json.dumps (examples, indent-2) For each complexity level (short/medium/lon)
Identify parallel reasoning structures
2. Quote speci fic passages that show transfer
Explain what cognitive skill being transferred
4, Rate the clarity of transfer (1-10)
Focus examples where the same reascning approach clearly appears both domains.
F.6 PATTERN CLASSIFICATION AT SCALE
After discovering patterns through bottom-up analysis, we classified all traces to measure transfer rates:
26

---

Published as a conference paper at ICLR 2026
Pattern Classification Prompt
Analyze these (domain) reasoning traces and find examples of each pattern.
PATTERNS TO FIND:
1. Casc-by-Case Analysis: Systematic enumcration of different scenarios/case:
2. expected Value Calculation: Explicit probabilig lity cal calculations, computing expected outcomes
Patternn Recognition: Identifying recurring structures, noticing trends
REASONING TRACES:
(json. dumps (batch, indent-2))
For EACH pattern, identify All examples that clearly demenstrate it.
Return in this EXACT format:
CASE BY CASE INDICES: 1iist of indicesl EXPECTED BAATTTDDD__ALEEEEDDIIIII PATTERN CEATIILLECGGGNTIONNNIII
[list of indices]
[list of indices]
Be strict only include clear examples of each pattern.
F.7 VALIDATION METHODOLOGY
To ensure robust findings, we implemented multiple validation steps:
Sampling Strategy: We analyzed 50 random trajectory samples per checkpoint to avoid selection bias while maintaining computational feasibility.
Success Stratification: Separate analysis of successful and failed attempts revealed which reasoning strategies genuinely contribute to problem-solving rather than merely appearing frequently.
Manual Verification:
Spot-checking GPT-4.1's Is pattern classifications against raw traces confirmed the accuracy of automated analysis.
Scale Validation: After discovering patterns through focused analysis, we classified all 46,792 mathematical traces to verify that observed transfer rates hold at scale.
This methodology ensures our findings reflect genuine cognitive transfer rather than superficial pat- tern matching, providing quantitative evidence that competitive gameplay develops reasoning skills applicable far beyond the training domain.
G GAME ENVIRONMENT SPECIFICATIONS
This section provides detailed specifications for all game environments used in our experiments, including both training and evaluation games.
G.1 TRAINING GAME ENVIRONMENTS
TieTacToe tests spatial pattern recognition through perfect information gameplay. Players alternate placing marks on 3x3 grid, aiming create lines of three. The deterministic nature isolates pure strategic reasoning from uncertainty management. Success requires recognizing winning patterns, blocking opponent threats, and creating fork positions that guarantee victory. We hypothesize these skills transfer to geometric reasoning and spatial visualization tasks.
Kuhn Poker introduces probabilistic reasoning through minimal hidden information. With only three cards (Jack, Queen, King), one per player plus one undealt, the game distills poker to essential elements of bluffing and value betting. Players can check, bet, call, or fold, with outcomes deter- mined by card strength. Success requires calculating expected values, modeling opponent behavior, and making decisions under uncertainty. These capabilities should transfer to probability problems and strategic decision-making.
27

---

Published as a conference paper at ICLR 2026
Simple Negotiation is game that develops multi-constraint optimization skills through resource trading. Two players exchange Wheat, Wood, Sheep, Brick, and Gold tokens. Since the utility of these resources varies for each player, there is natural incentive to trade. Each player aims to max- imize their portfolio's value by making proposals and counteroffers. Success requires understanding opponent's preferences, planning multi-step trades, and communicating strategically. We expect these skills to improve performance on optimization problems and multi-constraint reasoning tasks.
G.2 OUT-OF-DISTRIBUTIN EVALUATION GAMES
Our evaluation suite tests whether learned skills generalize to novel mechanics:
Snake extends spatial reasoning to dynamic environments. Players control snakes navigating grids to collect apples while avoiding collisions with walls, themselves, or opponents. This tests whether static pattern recognition from TicTaacToe transfers to trajectory planning and dynamic obstacle avoidance.
Pig Dice isolates risk-reward decision making. Players repeatedly roll dice to accumulate points but lose all turn points when rolling 1. This tests whether probabilistic reasoning from Kuhn Poker extends sequential risk assessment and expected value calculation in different contexts.
Truth and Deception focuses on asymmetric information and persuasion. One player knows the true fact among options and misleads through conversation while the other must identify truth through questioning. This evaluates whether negotiation skills transfer to pure communication strat- egy.
These diverse evaluation games prohe different aspects of transfer learning, revealing which cogni- tive skills generalize beyond their training context and confirming that SPIRAL develops fundamen- tal reasoning capabilities rather than game-specific tactics.
28


Reviews:
[{"review_id": "foAwrccTBZ", "reviewer": "Reviewer_TDEZ", "summary": "This paper presents SPIRAL, a self-play reinforcement learning framework where LLMs learn reasoning by playing multi-turn zero-sum games against themselves. It introduces Role-conditioned Advantage Estimation (RAE) to stabilize multi-agent training. Experiment results show that SPIRAL improves models' performance on reasoning benchmarks like math and reasoning.", "questions": "compare them with the corresponding base model, to better analyze how much of the improvement on base models comes from genuine reasoning enhancement?\n2. Please analyze why training on games improves QA benchmarks. The paper provides detailed analysis on why SPIRAL enhances performance on math benchmarks, but the improvement on QA seems less straightforward and lacks detailed explanation.\n3. During training, the base model cannot directly use the <think> special token, but the pattern analysis section mentions the “thinking token.” How was this implemented in practice—was it through CoT prompting or another mechanism?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid idea and algorithm design: Using self-play to construct an automatic curriculum for improving LLMs’ reasoning ability is a reasonable idea, and the proposed multi-turn, multi-agent RL framework is a novel approach.\n2. Good empirical results: The method is thoroughly evaluated across multiple models and benchmarks, showing consistent and meaningful performance improvements.\n3. Good clarity and presentation: The manuscript is well written and easy to follow. Figures and tables effectively illustrate both the proposed method and the experimental outcomes.", "weaknesses": "1. The second contribution (RAE) is relatively weak: Normalizing advantages separately for different agents in multi-agent settings is a common practice in classical MARL [1, 2]. Applying it to LLM-based multi-agent systems is quite straightforward, which makes the second contribution less substantial.\n2. Limited performance on instruct models: In Table 1, SPIRAL shows a significant improvement on base models but much smaller gains (1–2%) on instruct models. This raises the question of whether the improvement on base models truly comes from enhanced reasoning ability or merely better instruction-following.\n3. Missing related work: For LLMs in gaming, the authors do not discuss closely related work on LLM agents in multi-agent games such as Diplomacy [3], Werewolf [4, 5], and Avalon [6]. Some of these works also use self-play with RL to train agents, and should be referenced.\n\n[1] Lowe, Ryan, et al. “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments.” arXiv preprint arXiv:1706.02275 (2017).\n\n[2] Yu, Chao, et al. “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games.” Advances in Neural Information Processing Systems (2022).\n\n[3] Bakhtin, Anton, et al. “Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning.” Science 378.6624 (2022): eade9097.\n\n[4] Xu, Yuzhuang, et al. “Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf.” arXiv preprint arXiv:2309.04658 (2023).\n\n[5] Xu, Zelai, et al. “Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game.” arXiv preprint arXiv:2310.18940 (2023).\n\n[6] Wang, Shenzhi, et al. “Avalon’s Game of Thoughts: Battle Against Deception through Recursive Contemplation.” arXiv preprint arXiv:2310.01320 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "91EZ0x5L1C", "reviewer": "Reviewer_9i3G", "summary": "The paper uses online policy optimization (REINFORCE) with LLM self-play in zero-sum games (Tic-Tac-Toe, Kuhn Poker, and Simple Negotiations). The authors demonstrate how their approach improves downstream reasoning across multiple models (including a model that was already trained using RLVR - DeepSeek-R1-Distill-Qwen-7B). While games offer verifiable, human-free rewards, they shift complexity to game environment design and validation, which can shape emergent strategies, generalization, and scalability of this method.\nThey introduce a role-conditioned advantage to reduce variance and adjust it for asymmetric reward expectations (e.g., first-move effects in Tic-Tac-Toe and Kuhn Poker) and show it prevents policy collapse.\nThey present a systematic bottom-up method for automated evaluation of reasoning traces to discover common (\"transferable\") patterns (Appendix F).\nOverall, the authors demonstrate up to 10% gain in average on downstream tasks and a considerable margin compared to SFT on “expert” trajectories.", "questions": "Main:\n* Can the authors provide multi-seeded evaluation for Table-1 models (at least partial coverage for one or two models, including SFT and SPIRAL)? \n* Can the authors provide mean and STD using multi-seeded plots in Figs. 5. 6. (and 9.)?\n* Can the authors clarify which model sourced reasoning traces for Section 4.1?\n* Can the authors add a random policy and a simple RL expert to Tables 4 and 5 to contextualize self-play/Gemini Flash win-rates? \n* Other frameworks exist for large-scale multi-turn trajectory sampling (e.g., Nemo RL). Can the authors explain how this framework is different (since the authors consider their framework part of their main contribution)?\n* How well does the SPIRAL model’s reasoning generalize in increased complexity environments (e.g., impact on winrate in 5x5 grid Tic-Tac-Toe, 5 card Kuhn Poker, etc)? Discussing this sort of generalization can help understand how well SPIRAL performs in a smaller controlled domain, before leaping into academic reasoning and math benchmarks.\n\nSecondary:\n* Can the authors provide some statistics on the game trajectories during training? (e.g., game length (number of moves), avg reasoning tokens per step, self-play player 1 win-rate, win-rate vs. baseline agent, etc.)\n* Please explain how “transferable patterns” were discovered in the main text, or add explicit appendix pointers.\n* Please define the τ notation in the text and in Algorithm 1. Also, specify the exact input structure used to train the policy model (and which part of the input tokens gets a gradient during policy optimization)\n* Please report GPU hours per model and game/multi-game (reproduction compute budget).\n* Can the authors provide some justifications for the choice of games (including OOD games)? Also, justify the chosen training regime?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper systematically shows transfer from simple games to harder reasoning tasks, strengthening prior evidence that self-play improves LLM reasoning. \n\n* At face value, this paper provides strong gains across models, as shown in Table 1, with a relatively simple online policy optimization algorithm. These gains present an interesting avenue to future research, leveraging more robust and sample-efficient RL algorithms and game environment design.\n\n* The automatic evaluation methodology could be refined to address some of its current weaknesses (mentioned below) and serve as a blueprint for evaluating fuzzy LLM statistics in future work.", "weaknesses": "* Major issue: the authors did not report seeded policy optimization, and results do not include mean±STD for critical experiments. This is a must to assess reproducibility and statistical robustness, while results may appear cherry-picked.\n\n* Concerns about SFT dataset quality and size for a fair baseline:\n  * Quality: The “expert” is unevaluated. Compare the generator to SPIRAL-trained models and to classic RL agents to establish competence. Report diversity of winning traces (e.g., % unique trajectories vs. coverage seen during SPIRAL training).\n  * Size: With GBS=128 and 200 steps (or 2 epochs over 400 steps), the SFT baseline is underpowered versus multi-model self-play data.\n  * Example for a fair setup: sample 400×128 trajectories from multiple “expert” models to match self-play scale and diversity.\n\n* Section 4.1, LLM-as-verifier limits qualitative reliability. The paper is also missing key details to evaluate logic and reproduce the results.\n  * Reliability issues are known to be exacerbated by long context: F.2. suggests the authors insert multiple full traces into the judge model's context to find patterns, while F.6 suggests the authors used \"batch-mode\" for classification of multiple traces together. This seems like a loose method for a central claim in the paper without additional guardrails.\nPerhaps the authors can mitigate this using the judge to evaluate trace-by-trace, identify distinct reasoning approaches in each trace, then cluster similar patterns together as opposed to the full context approach used in F.2. Finally, reclassify each trace independently to detect the dominant patterns. This approach should allow the authors to report clear, verifiable statistics.\n  * F.3 cross-domain transfer: what model sourced the traces for the analysis? If the authors used a SPIRAL model to produce math traces, it is reasonable to assume these traces would be biased to common patterns seen in the game traces (as opposed to observing the reasoning traces of an unrelated strong baseline).\n  * How are traces evaluated under sequence-length limits?\n\n* Clarity: Although the method is relatively simple to understand, the paper feels a bit chaotic to read, forcing back and forth between the appendix and different sections of the paper to find and extrapolate critical information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G8ADlKiyye", "reviewer": "Reviewer_Q6Ug", "summary": "This paper introduces SPIRAL, a novel and compelling framework for improving the reasoning capabilities of large language models through multi-agent, multi-turn self-play in zero-sum games. The core contribution is the demonstration that complex, transferable reasoning skills can be developed without reliance on human-curated problem-answer datasets, which represents a significant step towards more autonomous and scalable AI development. By leveraging an adaptive curriculum of ever-improving opponents, SPIRAL effectively teaches models to reason strategically. The technical innovations, particularly Role-conditioned Advantage Estimation (RAE), and the extensive empirical results provide strong evidence for the viability of this approach. This is a high-quality paper with major findings, and I recommend its acceptance.", "questions": "1.\tWhat do you think about the results on training with SFT (Qwen3-4B-SFT) is better than RL against Gemini and Mistral models (Qwen3-4B-Gemini and Qwen3-4B-Mistral)?\n2.\tHow do you see the SPIRAL framework extending beyond the zero-sum setting? For example, could it be adapted for cooperative or mixed-motive games to elicit other valuable skills like collaboration and negotiation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is written in a clear and understandable manner, with a well-defined methodology and simple yet effective improvement strategies that are easy to follow.\n\n●\tThis work makes a major contribution toward the goal of self-improving LLMs by reducing the dependence on human-curated data. By using self-play as a source of unlimited training signal, SPIRAL facilitates significant model self-improvement. This approach could represent the next paradigm for RLVR, moving beyond the domains of math and code reasoning.\n\n●\tThe experiments are comprehensive, conducted on five different models (Qwen3-4B, Qwen3-8B, Octothinker-8B, DeepSeek-Distill-Qwen-7B, and Llama-3.1-8B) from two model families (Qwen, Llama), including base models, instruction-tuned models, and those already fine-tuned for reasoning. The consistency of improvements across eight challenging benchmarks is a testament to the generalizability of the skills learned through SPIRAL.\n\n●\tThe paper provides valuable resources for the community by delivering a complete, fully online, multi-turn, multi-agent RL framework. The core technical contribution, RAE, effectively solves the critical \"thinking collapse\" problem and is broadly applicable to future MARL experiments. Furthermore, the framework is orthogonal to other emerging techniques; for instance, the auto-curriculum from SPIRAL could be combined with prolonged RL methods [1] to continuously push reasoning boundaries.\n\n●\tThe paper goes beyond simply reporting benchmark scores by providing a deep and insightful analysis of skill transfer. Identifying and tracking specific cognitive patterns [2] (e.g., Case-by-Case Analysis), provides a compelling explanation for why the method is effective and builds confidence that the model is learning genuinely useful strategies.\n\n[1] https://arxiv.org/abs/2505.24864\n[2] https://arxiv.org/abs/2503.01307", "weaknesses": "While this is a great paper, there are several areas where further discussion or exploration could enhance its contribution. These points are primarily intended as constructive feedback instead of reasons to reject.\n\n●\tThe training resources are demanding, which could be a barrier to broader adoption. A discussion on potential avenues for improving computational efficiency (i.e. LoRA) would be a valuable addition.\n\n●\tIn line 412, it would be helpful if the authors could further elaborate on how the fixed-opponent baselines (vs. Gemini and random) address the issue of spurious rewards. The connection is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mjYzY7dtUY", "reviewer": "Reviewer_2RMj", "summary": "This paper proposes SPIRAL, a self-play reinforcement learning framework that enhances LLM reasoning by training models through multi-turn, zero-sum language games (e.g., TicTacToe, Kuhn Poker) against evolving versions of themselves. The key insight is that competitive game dynamics generate an automatic curriculum of increasingly challenging reasoning tasks without requiring human-curated problems or domain-specific reward engineering. To stabilize training, the authors introduce Role-conditioned Advantage Estimation (RAE), which mitigates variance in multi-agent policy gradients. Experimental results show significant improvements on mathematical reasoning benchmarks, with evidence of reasoning pattern transfer from games to academic tasks. The approach is evaluated across multiple model families, demonstrating broad applicability.", "questions": "1) Has the impact of SPIRAL training on general capabilities been evaluated? Specifically, what happens to performance on MMLU, AlpacaEval, or MT-Bench after SPIRAL post-training? Is there any degradation in instruction-following or commonsense reasoning?\n\n2) Have the authors conducted experiments where SPIRAL is used as a pre-training stage before RLVR (e.g., Base → SPIRAL → RLVR)? If so, could you include a comparison with Base → RLVR in terms of final performance and convergence speed?\n\n3) When stating that models like DeepSeek-R1-Distill still benefit from SPIRAL, does this refer to further training after RLVR? If yes, how does it affect performance on the original RLVR benchmarks?\n\n4) Can the authors provide ablation studies on game diversity? For example, does combining multiple games (TicTacToe + Kuhn Poker) outperform single-game training or simple data mixing? Do different games induce complementary reasoning skills?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**The work proposes a novel training paradigm**: instead of relying on human-curated problems with verifiable answers, SPIRAL uses self-play in zero-sum language games to create an autonomous curriculum of increasingly complex reasoning challenges. While the game outcomes are still verifiable, the strategies and subproblems emerge dynamically from competition, enabling the model to develop transferable reasoning skills without direct exposure to target domains.\n\n**Empirically Promising Results**: SPIRAL achieves substantial gains on math reasoning benchmarks compared to base models, even without exposure to target problems during training. The qualitative analysis of reasoning pattern transfer (e.g., case analysis, expected value calculation) provides mechanistic insights beyond mere performance metrics.\n\n**Broad Applicability**: The method demonstrates consistent benefits across diverse model architectures (Qwen, Llama, Octothinker) and training stages (base, instruction-tuned, and even RLVR-trained models), suggesting strong generalization potential.", "weaknesses": "**Limited Benchmark Coverage**: Evaluation is narrowly focused on mathematical reasoning. There is no evidence of improvement on other critical reasoning domains such as code generation, logical deduction, causal reasoning, or multi-hop QA. This limits claims about general reasoning enhancement.\n\n**Missing Pipeline-Level Experiments**: The paper suggests SPIRAL can serve as a foundational or complementary stage in modern training pipelines (e.g., pre-training before RLVR or fine-tuning after). However, no experiments compare:\n\n1) Base → SPIRAL → RLVR vs Base → RLVR\n\n2) RLVR-Trained Model → SPIRAL vs continued RLVR/SFT Without these, the practical value of SPIRAL as a booster or enhancer remains speculative rather than demonstrated.\n\n**No Assessment of General Capability Preservation**: The paper does not evaluate whether SPIRAL training degrades performance on non-reasoning tasks (e.g., instruction following, commonsense reasoning, language modeling). This raises concerns about catastrophic forgetting or capability misalignment, which are crucial for real-world deployment.\n\n**Claims Outrun Evidence**: While the title and abstract imply broad applicability to \"reasoning,\" the experimental validation is largely confined to math. The claim that SPIRAL develops \"transferable reasoning capabilities\" would be stronger with more diverse downstream evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)