MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
Authors: Huining Yuan
Zelai Xu
Zheyue Tan
Xiangmin Yi
Mo Guang
Kaiwen Long
Haojia Hui
Boxun Li
Xinlei Chen
Bo Zhao
Xiao-Ping Zhang
Chao Yu
Yu Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Self-play
Multi-Agent System
Strategic Games
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a self-play framework for enhancing general multi-agent capabilities of LLMs via reinforcement learning on strategic games.

Abstract: Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce **MARSHAL**, an end-to-end RL framework that incentivizes **M**ulti-**A**gent **R**easoning through **S**elf-play wit**H** str**A**tegic **L**LMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agents trained from Qwen3-4B develop strong strategic abilities, with up to $28.7$\% performance improvements in held-out games. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant zero-shot performance gains of up to $10.0$\% on AIME, $7.6$\% on GPQA-Diamond, and $3.5$\% on average across all benchmarks. These results establish self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs.

Published as a conference paper at ICLR 2026
MARSHAL: INCENTIVIZING MULTI-AGENT REASON- ING VIA SELF-PLAY WITH STRATEGIC LLMS
Huining Y Zelai Xu²* Zheyue Tan Xiangmin Yi', Mo Guang', Kaiwen Long', Haojia Hui', Boxun Li", Xinlei Chen', Bo Zhao Xiao-Ping Zhang", Chao Yu Yu Wanga)
SIGS, Tsinghua University, EEE Tsinghua University, AAalto University, 'Li Auto Inc., IIfinigencce AI
Project Page
n Code
Models
ABSTRACT
Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is critical step towards more advanced in- telligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizen credit assignment and agent-specific advantage estimation. To address these challenges, we intro- duce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agen Reasoning through eelf-play witH strAtegic LLMs both cooperative and com- petitive games. MARSHAL features turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-speeific advantage normalization to stabilize multi-agent training. By learning with self- play across coeperative and competitive games, MARSHAL agents trained from Qwen3-4B develop strong strategic abilitics, with up to 28.7% performance im- provements in held-out games. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MAR- SHAL agent achieves significant zero-shot performance gains of up 10.0% on and 3.5% on average across all benchmarks. These results establish self-play in strategic games as powerful approach for developing generalizable multi-agent reasoning capabilities in LLMss
AIME, 7.6% on
Strategic Ability
Generalie
Generalization Multi-Agent Systems
MATH 100
MATH
Tee
Fornect
GSMBK
GPGA
GSMBK
GPQA
Train
Test
90
Kunn
Ledue
MAD
AutoGen
Pekor
Hole'om
(Comprtiiva)
(C /Cooperative)
AQUA
MMLU
AQUL
20
MMUU
Miri
Simple
Honols
Hondls
Cwen3-4B
SPIRAL
MARSHAL
Figure 1: Evaluation of MARSHAL and baselines on strategic games and reasoning benchmarks. MARSHAL incentivizes multi-agent reasoning ability via self-play in strategic games and generalizes improvements of multi-agent systems like MAD and AutoGen math and QA benchmarks.
"Equal contribution: fyuanhuiningo, zelai lecss @gmail f.tggall.cmm "Coresppnndigg authors: xpzhang0i eee. org, yuchao0sz. tsinghua. letiggua.edu...ee yu-wang@tsinghua. edu..ccuu.eedccc edu.cn

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The remarkable capabilities of Large Language Models (LLMs) have revolutionized numerous domains, enabling strong performance a wide range of tasks from question answering to code generation (Achiam et al., 2023; Team et al., 2023). However, many real-world scenarios, such as negotiation (Bianchi et al., 2024), strategic gameplay (Silver al., 2016; FAIR et al., 2022), and collaborative software development (Qian et al., 2023; Wu et al., 2024a) inherently involve multiple agents interacting over long horizons. Enabling LLMs to cooperate and compete effectively within multi-agent systems (MASs) is a critical frontier for advancing artificial intelligence.
While reinforcement learning (RL. has demonstrated remarkable success in enhancing the reasoning capabilities of individual LLMs (Guo et al., 2025; Team et al., 2025), its extension to multi-turn, multi-agent tasks faces two critical challenges. First, the problem of long-horizon credit assignment arises in multi-turn interactions. As a sequence of actions can each result in an immediate reward while collectively leading the final sparse reward, accurately attributing the contribution of the action in each turn is inherently more complex than standard single-ture settings. Second, multi- training couples heterogeneous game roles with asymmetric information and different payoff scales, which introduces variance in advantage estimation, destabilizing the training process.
In this work, we present MARSHAL (Multi-Agem Reasoning through Self-play witH strAte LLMs), an end-to-end RL framework that incentivizes multi-agent reasoning capabilities via self- in strategic games. We introduce two novel techniques to address the challenges in multi-turn, multi-agent self-play with Group-Relativ Policy Optimization (GRPO) (Shao al., 2024). First, we propose a simple yet effective turn-level advantage estimator to enable fine- fine-grained credit assignment. This allows the model to accurately attribute long-term outcomes to individual actions and provide learning signals across multiple turns and agents. Second, we propose an agent-specific advantage normalization that stabilizes the training process by calibrating advantage estimates relative the performance of each agent. This normalization accounts for the heterogeneous roles in multi-agent systems and ensures stable policy updates. By integrating these components, MARSHAL enables robust self-play learning from game outcomes, empowering LLMs to develop strategic abilities and multi-agent reasoning skills through cooperation and competition with themselves.
To evaluate the performance and generalization ability of MARSHAL agents, we conduct extensive experiments by training Qwen3-4B a diverse range of cooperative and competitive games. Specifi- cally, MARSHAL agents exhibit strong strategic ability across all game environments, with up to 28.7% performance improvements in three held-out games. More importantly, the capability acquired through self-play in games further generalizes to improvements of multi-agent systems in reasoning benchmarks. When integrated into both cooperative and competitive multi-agent systems, including AutoGen (Wu et al., 2024a) and MAD (Liang et al., 2023), our MARSHAL agents achieve zero-shot performance improvements of up to 10.0% on AIME, 7.6% on GPQA, and 3.5% on average across all benchmarks. We further conduct ablation studies to validate the effectiveness of our algorithmic design, complemented by a comprehensive analysis of reasoning patterns and failure modes to understand the successful generalization. The evaluation results establish MARSHAL powerful approach for developing generalizable multi-agent reasoning capabilities in L.MMs.
In summary, contributions are follows:
We propose MARSHAL, an end-to-end RL framework that enhances multi-agent reasoning through self-play in diverse range of strategic games.
We introduce two novel techniques of turn-level advantage estimator and agent-specific normal- ization to address the credit assignment and advantage variance in multi-turn, multi-agent RL training for LLMss.
We perform extensive experiments and ablation studies show that MARSHAL incentivize strong strategic ability and multi-agent reasoning capability that are generalizable to held-out games and multi-agent LLM systems.
2 PRELIMINARY
In standard RL fine-tuning for tasks like math, the environment is static. The model generates a complete Chain-of-Thoure response, and only then is a terminal reward assigned. This process is
2

---

Published as a conference paper at ICLR 2026
Player Loop
Player Loop
GRPO
MARSHAL
Env
Trojactorias
Player Traj
Player 0 Iraj
Tumn
Turn
Tam3
Tunn5
As playee X. aboadd... My
You ase Sagal
atate..
Tou ace Czzenes state.. begal
Nov, as aboozd...
Trav- Rewords
Rewrrdd
Batchhormm
CuaBuxi
LLM
Returns
Batchhlorn
episode
Tumil
Tumi
Tumb
Traj -1 aiall Advantages
Advantages
Turn-Loval
Player trojectory
Player trojectory
Token-l evel Actvve
Aovev
Figure 2: Overview of MARSHAL. Left column: generating player trajectories through self- self-play in strategic games, Middle column: naive advantage estimation by GRPO. Right column: advantage estimation by MARSHAL. for accurate credit assignment in multi-turn, multi-agent setting.
typically modeled as a token-level Markov decision process (MDP). Here, a trajectory consists of a single "turn", which is the generation of one complete response with multiple tokens. The state step t is the sequence of previously generated tokens (oI.... 0t- ) and the action is the selection of the next token or. Given a task question q, the goal is optimize the token-level policy क (oelg, dct) to produce sequence that maximizes the final sparse reward.
In contrast, strategic games introduce more complex, hierarchical deciisin-mmaknng structure. An entire game, not a single response, constitutes an episode. This is best modeled as a turn-level MDP, where decisions occur at two levels. At the high level, the state 8І represents the state of the game at the beginning of turn k (e.g., the board configuration, the cards in hand, etc.). A high-level action, Og, corresponds to the agent's entire output for that turn (e.g.. place my in the top-left corner and here is why..."). This action dg is itself sequence of multiple tokens generated by the
LLM's low-level, autoregressive policy, In this case, the goal is to optimize the turn-level policy (au(sk) = I T (Ok,, 8k, ok,ct) to maximize the total return of all turns R = Σ k=
3
METHOD
In this section, we introduce MARSHAL, an end-to-end RL framework that enhances multi-agent reasoning ability of LLM through self-play in diverse range of cooperative and competitive games. We begin by outlining the overall architecture of our training framework, building upon Group- Relative Policy Optimization (GRPO) (Shao et al., 2024). We then detail our primary technical contributions for addressing the credit assignment and advantage estimation challenge in multi-tury turn, multi-agent training. Finally, we describe the selection of our game environments and design of reward structures. An overview of our proposed method is shown in Fig. 2.
3.1 SELF-PLAY WITH GRPO
To eliminate the extensive computational cost introduce by the critic model in classic Proximal Policy Optimization (PPO) (Schulman al., 2017), GRPO rollout each query for multiple times, constructing a group of G responses {ot) and their corresponding outcome rewards r = Jizi= [r"]
then estimate the advantage of each response by their relative reward to their corresponding group, which assigns equal advantage to the whole sequence (Fig.2 middle), yielding
JGRPO (0) = (g
Tsur (To; no To Ai,
(1)
(01))
where (Tu; Teass Ai, min
is the r'-mean(t) token-level PPO surrogate objective, and the advantages estimated by Ai std(r)
no(ofla A clip (011.02))
mo(oilg.o Rewee (o (.)))
1 E, 1 E )}
3

---

Published as a conference paper at ICLR 2026
In multi-agent self-play, where all players in strategic game is controlled by the same model, each episode of game results a multi-turn trajectory for each player role. respectively. As a naive generalization of the original GRPO to our multi-turn scenario, start by considering all trajectories from game environment as group of response, i.e. {(s), /= terminal reward for each trajectory r = RE Then, with an additional summation for multiple turns, GRPO directly generalized to
1 with total return as the
K
1o Jsure (To; Moria A E), (2)
Tempi (0) 16,3 P(S).ot, "oo (0 C
where the advantages are estimated by A in the multi-tumn trajectory, similar to the original GRPO. For multi- game training, trajectories for each game are naturally considered separate groups and normalized independently.
12" meem(r))
This assigns equal advantage to all tokens
sid(r)
3.2 ADVANTAGE ESTIMATION FOR MULTI-TURN, MULTI-AGENT LEARNING
While GRPO has demonstrated remarkable efficiency and stability in single-turn settings, such direct application to the multi-turn, multi-agent structure of self-play introduces significant challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these issues, we introduce two novel modifications to GRPO.
Turn-level advantage estimator. To enable fine-gratd tineggrined credit assignment across a long trajectory, we incorporate the sequence of turn-level rewards r = {(ri)= 10 This setup is analogous to the "Process Supervision" setting the original GRPO paper (Shao et al., 2024). In that setting, the proposed advantage estimation involves first normalizing all rewards across the entire batch, (rcan(r)/sst((1))) and then computing the cumulative sum of these normalized values,
T = (ri A - S F.. However, since the distributions of intermediate rewards can vary significantly, treating the entire set of rewards as a single distribution for global normalization is likely inappropriate and potentially problematic.
To address this problem, we propose crucial inversion of these two steps: we first sum, then normalize (Fig.2 right). Specifically, we begin by calculating the standard Monte Carlo return (or cumulative reward) from each turn i onwards: Rg = > r We then compute the advantage
by normalizing these returns to their mean: A/., = R2 mean(R), where R is the collection of all cumulative rewards in the group. This formulation is equivalent to Generalized Advantage Estimation (GAB) (Schulman al., 2015) with y and À = 1, where the value function V(s)) is approximated by simple yet effective bascline: the empirical mean of the batch returns E R]. By ensuring the final advantages are properly centered, this method provides much more stable learning signal for multi-turn decision- makin...
Agent-specific advantage normalization. In many games, the expected return can be highly dependent a player's role (e.g., player 0 vs. player 1, or different roles in cooperative game). Normalizing advantages across different roles would force all players toward shared baseline, which is statistically inappropriate and can obscure role-specific learning signals.
To address this, we refine our advantage calculation further. We partition the batch of trajectories into sub-groups based player role and apply the turn-level advantage estimator described above independently within each sub-grout group GP (Fig.2 right), where p denotes the player index. This ensures that the advantage for given action is calculated relative to the average outcome for that specific role, providing a more accurate and stable credit assignment in multi-agent settings. This agent-specifie, tornn-allee advantage estimation forms the final objective for MARSHAL
JMARSHAL (0) E B P(S) NTO ( (Ols 2.50)
P
Gp
R
1o
(3)
(m Tomer R e),
1=1
where AP. RR.. mean(R")) (R"). RR denotes the total set of cumulative rewards from subgroup GP.
4

---

Published as a conference paper at ICLR 2026
3.3 GAME ENVIRONMENTS
To incentivize comprehensive multi-agent reasoning ability for MARSHAL, we select a range of six strategic, two-player games. These games are partitioned into training and more complex, held-out testing set for out-of-distribttion (OOD) evaluation:
Perfect-information competitive games: we train on TiceTac-Toe, requiring the agent to recognize its strategic position (e.g., first-mover second-mover) and plan accordingly. For evaluation, Connect Four serves as a more complex test due to its vastly larger state space.
To enable deterministic planning and role adaptation,
Imperfect-inte information, competitive games: under imperfect information and uncertainty, we train on Kuhn Poker, simplified poker variant. We evaluate generalization on the more sophisticated Leduc Hold' em.
To foster robust reasoning and decision-making
inperfectiiformmtionn cooperative games: To develop social intelligence like intent recogni- tion and Theory of Mind, we consider the classic cooperative card game Hanabi. Specifically, we train on Mini Hanabi and evaluate on a more challenging variant Simple Hanabi to test the generalization of cooperative strategies.
Collectively, this diverse range of both competitive and cooperative games ensures the LLM agent develops generalizable multi-agent reasoning capabilities.
3.4
REWARD DESIGN
While our primary learning signal is derived from the intrinsic game outcomes to minimize reward engineering, incorporate auxiliary rewards ensure stable training. The final reward signal is composed of three components:
The primary signal is the intrinsic game outcome. This is +/-1 reward for a win/loss/draw in
Intrinsic game rewards:
chips won lost in Kuhn Poker (max of 2), and shared +1 reward per successfully played card in Mini Hanabi (max of 4). To balance these varying scales during multi-game training, we normalize the maximum reward across games to by scaling the Ti--Ta-Toee reward with factor 2f
To ensure parsable outputs, we provide small positive reward (+0.05) for validly formatted actions and a large negative penalty (-10.0) that terminates the game for invalid ones, similar to the approach in DeepSeek-RI (Guo et al., 2025).
Format reward:
Response length penalty: To encourage conciseness, we apply turn-level penalty for verbosity. inspired by Kimi kl.5 (Team et al., 2025), which scales linearly for responses longer than a set threshold. The penalty is calculated as
1 mmin Tmax
Tength (1) mnax
0,
Imin
where we set min 11, Imax 2048, and the scaling coefficient a = 0.5.
4 EXPERIMENTS
In this section, present extensive experiment results validate MARSHAL's ability to obtain generalizable multi-agent reasoning capability. Additional results can be found in the Appendix.
4.1 EXPERIMENTAL SETUP
We use Qwen-4B (Yang et al., 2025), state-of-the-art reasoning model, as our foundation to measure the multi-agent reasoning capabilities of MARSHAL. We train two model types: specialist agents on each game (Tic-Tac- Toe, Kuhn Poker, Mini Hanabi) and a generalist agent on all three simultaneously. Implementation details and hyperparameter settings are provided in Appendix B and C. Our primary baseline is SPIRAL, a recent work focused on self-play in purely competitive, zero-sum games (Liu al., 2025). Our evaluation is structured in four stages (see Appendix D for details):
5

---

Published as a conference paper at ICLR 2026
Tic-Tac-Toe
Kuhn Poker
Mini Hanabi
Testing Corees Cenerotis Troining Games
233
772
321
780
22.5
60-
655
sp-
60-
37A
s2-
18A
18
40-
300
20
20
20-
Connect Four
Leduc Hold'em
Simple Hanabi
30-
269
15-
60-
20-
R.S
40-
20-
135
130
10-
20-
Fist move
second move
Firtt move
second move
Qwen3-4B 4B
SPIRAL
MARSHAL Specialist
MARSHAL Generalist
Figure 3: Average normalized game returns. Specialist agents not only master their training domains but also generalize effectively to their more complex, held-out counterparts (e.g., from Tic-Tac-Toe Connect Four). The generalist model achieves consistently high performance across the entire suite of games, establishing it as the most robust and versatile agent.
1. Strategic ability: We first evaluate strategic competence by benchmarking agents against strong Monte Carlo Tree Search (MCTS) Nash Equilibrium (NE) opponents. Performance is measured over 1000 games on both the training games and a suite of more complex, held-out games. For the cooperative Hanabi games, we report the standard self-play score.
2. Generalization to multi-agent systems: This is the primary MARSHAL agents into two established systems: the competitive MAD framework (Liang et al., 2023) and the cooperative AutoGen framework (Wu al., 2024a), measuring zero-shot generalization standard math and QA benchmarks.
of this work. We integrate
3. Pattern analysis: We conduct qualitative analysis of the agent's 's reasoning process and quantitative analysis of failure modes to understand the successful generalization to MASs.
4. Ablation studies: Finally, we perform ablation studies validate our key algorithmic designs, particularly our novel advantage estimation technique.
4.2 STRATEGIC ABILITY
Tic-Tac-Toe Specialist on Tic-Tac-Toe
Our analysis begins with strategic ability on games, with normalized game return detailed in Fig 3. The MÁRSHAL framework proves highly effective: specialist agents not only master their training do- mains and outperform baselines, but also generalize effectively more complex, held-out counterparts (c.g.. from Tic- Te--TT-Toe to Con- nect Four). We also observe evidence of cross-category skill general- ization. For example, the Tic-Tac- Toe specialist model demonstrates smooth improvement not only on Tic- Tr--Ta--Toe but also the OOD Mini Hanabi (Fig. 4), suggesting MARSHAL cultivates foundational skills like turn- based planning that are broadly beneficial.
Game Return
-0.4
-0.6
-0.7
Tic-Tac-Toe Toe Specialist
on Mini Hanabi
Game Hetum
1.2
Crucially, the generalist model, trained all environments simulta- neously, achieves high performance across the entire suite of games, achieving 28.7% improvement Leduc Hold' em and 22.9% on Simple Hanabi. Its broad competence across competitive and coop- erative settings establishes it most robust agent overall.
0.8
40
160
200
Step
Figure 4: Eval curves of the Tic- Hee-Tac-Tor specialist in Tic- Tac-Toe and Mini Hanabi.
4.3 GENERALIZATION TO MUETI-AGENT SYSTEMS
The ultimate test of MARSHAL is whether skills honed in games generalize to practical, out-of- domain challenges. We evaluate this on suite demanding mathematics and QA benchmarks in zero-shot manner, including MATH500 (Cobbe et al., 2021), GSM8K, AQUA-RAT (Ling et al., 2017), AIME24, AMC23, MMLU-STEM (Hendrycks et al., 2020), and GPQA- -iimmond (Rein et al., 2024). As preliminary step, investigate how our game tasks enhance foundational reasoning in

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results downstream reasoning benchmarks within multi-agent systems. Competitive game-trained agents excel in the competitive MAD framework, while the cooperative- trained agent excels in the cooperative AutoGen framework. The generalist model performs robustly across both. Bold and underlined indicate the best and second-best scores, respectively.
Math
QA
Setting
Model
Average
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
Qwen3-4B
60.74
87.60
94.60
39.80
36.70
70.00
57.10
39.39
SPIRAL
63.75
87.50
94.80
51.20
36.70
80.00
58.70
37.37
MARSHAL
Single
Tic-Tac-Toe
63.54
89.10
95.20
46.50
40.00
77.50
57.60
38.89
Agent
Kuhn Poker
61.38
87. 80
94.50
48.40
33,30
72.50
59.30
33.84
Mini Hanabi
62.05
88.10
94.70
48.00
43.30
65.00
58.90
36,36
Generalist
62.79
89.90
94.60
52.00
33.30
75.00
59.90
34.85
Qwen3-4B
72.45
90.20
95.91
80.71
40.00
75.00
87.42
37.88
SPIRAL
73.41
91.60
95.45
81.89
40.00
77.50
87.01
40.40
MAD
MARSHAL
(Compe-
Tic-Tac-Toe
75.01
92.20
96.06
83.07
43.33
82.50
86.76
41.12
titive)
Kuhn Poker
74.54
91.60
96.21
82.68
40.00
82.50
87.39
41.41
Mini Hanabi
73.70
91.40
95.60
82.68
43.33
77.50
87.04
38.38
Generalist
75.96
92.80
95.60
83.86
46.67
80.00
87.36
45.45
Qwen3-4B
79.14
93.40
94.69
85.04
56.67
87.50
89.21
47.6 47.47
SPIRAL
80.05
94.20
94.47
86.61
60.00
87.50
91.60
45.96
AutoGen
MARSHAL
(Coope-
Tic-Tac-Toe
80.15
94.40
94.69
87.01
60.00
90.00
89.53
45,45
rative)
Kuhn Poker
81.54
95.80
94.39
86.61
63.33
92.50
89.65
48.48
Mini Hanabi
81.54
94,40
94.54
86,22
66.67
95.00
88.98
44,95
Generalist
82.15
95.20
94.54
86.61
66.67
92.50
89.53
50.00
standard single-agent setting. Notably, MARSHAL models achieve notable improvements over both the Qwen-4B baseline on number of math benchmarks, on par with SPIRAL (Table 1).
We further embed our agents into established multi-agent systems to directly measure their cooper- ative and competitive capabilities. In the competitive MAD debate framework, agents trained on competitive games show clear advantage. Notably, the generalist agent is able to achieve average of 3.51% over the original Qwen3-4B across all benchmarks. Conversely, in the cooperative AutoGen framework, the agents trained for cooperation the Hanabi specialist and the generalist excel across numerous benchmarks. In particular, the generalist model achieves a striking gain of 7.57% on GPQA-Diamond in the MAD framework, and 10.00% on AIME in the AutoGen framework.
These results provide compelling evidence that MARSHAL forges distinct, generalizable skills for both competition and collaboration. These capabilities directly translate to improved performance in downstream multi-agent applications, with the generalist model proving to be the most robust.
4.4 REASONING PATTERN AND FAILURE MODE ANALYSIS
To understand the mechanisms behind MARSHAL's successful generalization, we analyze the agent's reasoning (<think>) traces. This qualitative analysis reveals the emergence of key multi- reasoning patterns cultivated in our mix of strategic games. We highlight two representative patterns corresponding to the competitive and cooperative training environments, as showcased in Table 2.
First, reflecting the skills honed competitive games, MARSHAL. develops role-aware strategy. In Tic- Tic-Tac-Toe the agent explicitly identifies itself the "second-mov move player" and adopts defensive strategy. This skill generalizes directly to the MAD debate framework, where the agent recognizes its role the "negative side assistant" and adopts negative stance rather than just solving the problem.
More profoundly, MARSHAL cultivates intent recognition, a key component of Theory of Mind. In the cooperative Hanabi, the agent learns to interpret teammate's 's actions as communications with hidden intent (c.g.. "Maybe they want me to play this card?"). This sophisticated skill generalizes to the AutoGen framework, where the agent infers collaborator's uncertainty from subtle cues, such

---

Published as a conference paper at ICLR 2026
Table 2: Qualitative analysis of emergent reasoning patterns.
Skill
Manifestation in Game-Play
Generalize to Multi-Agent Systems
Role
The Tic- Te-Ta--Toe specialist recognizes its role as the second player (O) and adopts a defen- sive strategy.
The same agent, acting as the "negative" de- bater in MAD, adapts its strategy to refute the opponent.
Undorstanding
<think>
<think>
Okay, so am playing the game of Tic-
Okay, so I need to figure out minimum number of... The affirmative side just ar- gued for 4. As the negative side assistant, I should try to find an alternative solution and convince the judge... </think>
Toe as the mark O... As the second-mov player, I should prioritize blocking the X marks from forming a horizontal, vertical, or diagonal line...
</think
Intent Recognition
The Hanabi specialist infers the intent be- hind teammate's 's ambiguous hint.
The same agent, acting as a user proxy in AutoGen, infers uncertainty from collabo- rator's missing 'TERMINATE' token.
<think
Okay, so need to figure out next move. Player just revealed my only red card. Wait, but the rank remains unknown. Maybe they want me play this card to stack?
<think
Okay, lct's sec. The assistant gave the an- swer 17.5 m/s, but did not end the conversa- tion with "TERMINATE', Maybe the assis- tant is not sure with this answer? </think>
</ </think
Failure Categories
Breakdown of Category 2: Inter-Agent Misalignment
Fralure Rote
00-
5A2
T1A
Cotagery Sysemm Oeoign lquuss
Cotegory Intte Agant
Categgary Tosi Verffcotton
Commenattoo Roved
Fall Ask Ciardicttoon
Took Deraimmnt
Information Witnadding
Ignosed Oeheer
Ageni's Input
Rearoniin Morrnng-Acctto Actiion Homattch
Muslighment
Gwen3-48
MARSHAL Generalist
Figure 5: Percentage of different failure modes in GPQA-Dimond Through MARSHAL training, the generalist agent significantly reduces the occurrence of Inter-Agent Misalignment.
as a missing TERMINATE token, instead of treating as a simple error. Together, these patterns provide qualitative evidence that MARSHAL goes beyond improving benchmark scores; it equips the agent with the cognitive toolkit essential for effective multi-agent interaction.
To further substantiate these qualitative observations with quantitative evidence, performed failure mode analysis on the GPQA-Diamond benchmark within the MAD framework. Adopting the taxonomy of Cemri et al. (2025), we categorized failures into System Design Issues (e.g., formatting errors, loop repetition), Inter-Agent Misalignment (c.g., multi-agent reasoning failures like ignoring peers task derailment), and Task Verification issues.
As shown in Fig 5 (left), while MARSHAL improves basic instruction following (reducing System Design Issues by ~7%), the reduction in Inter-Agent Misalignment is significantly larger (11.5%). To understand the drivers of this strategic improvement, we further decompose the Inter-Agent Misalignment category into its sub-categories. The breakdown reveals that the performance gains primarily driven by reductions in Task Derailment and Ignored Other Agent's 's Input. This indicates that MARSHAL agent are actively listening peers and maintaining focus on its objective, validating our hypothesis that game- teme-thhorettc self-play cultivates generalizable multi-agen reasoning skills.
4.5 ABLATION STUDIES
To validate algorithmic designs, conduct two targeted ablation studies: (1) comparing self-play to training against a fixed opponent, and (2) sequentially removing the two key algorithmic components: our turn-level advantage estimator and agent-specific advantage normalization.
8

---

Published as a conference paper at ICLR 2026
Table 3: Generalization comparison between MARSHAL (self-play) and its fixed-opponent variant.
The latter exhibits significant overfitting to static environments and opponents. Values denote average normalized game returns; for competitive games, entries indicate first-move / second-move returns. Underlined scores indicate performance degradation compared to the standard MARSHAL model.
Training Games
Testing Games Ledue Hold em
Model
Mini Hanabi
Simple Hanabi
Tic-Tac-Toc
Kuhn Poker
Connect Four
MARSHAL TTe-TT
75.30 / 32.10
74.15 / 3.42
50,48
30.65 / 14.85
58.36
27.65
29.75
fixed opponent
88.00 / 41.95
63.15 / 28.84
34.93
20.35 / 5.65
47.38
35.55
12.22
MARSHAL (Kuhn Poker)
69.85 / 25.50
79.04 / 22.49
44.98
27.60 / 12.70
63.94
/ 62.10
29.35
w/ fixed oppoment
0.00 / 0.00
76.19 / 15.64
0.00
0.00 / 0.00
0.00
0.00
0.00
Table 4: Ablation results for algorithmic design. Both our turn-level advantage estimator and agent- specific advantage normalization prove essential for performance. Notation follows Table 3.
Training Games
Testing Games
Model
Mini Hanabi
Leduc Hold' em
Simple Hanabi
Tic-Tac-Toe
Kuhn Poker
Connect Four
MARSHAL TTc--TccTToe)
75.30 / 32.10
74.15 / 3.42
50.48
30.65 / 14.85
58.36
27.65
29.75
w/o Turn-Level.
74.60 / 24.15
80.26
28.35
34.80
26.75
12.30
48.34
41.34
19.05
w/o Agent-Spenn Spent-Specffic.
82.70 / 31.20
70.89
11.24
44.10
25.40 / 10.50
51.04
49.88
21.72
MARSHAL (Kuhn Poker)
69.85 / 25.50
79.04 / 22.49
44.98
27.60
12.70
63.94
62.10
29.35
w/o Turn-Level.
63.35 / 19.65
92.49
21.02
41.65
29.60
10.85
32.26
31.23
22.98
w/o Aaget--Specipp
69.55 / 24.55
75.37 / 19.55
40.18
27.00
10.50
35.73
21.50
22.42
MARSHAL (Hanabi)
71.90 / 7.35
72.52
9.29
55.55
26.75
5.75
37.36
55.12
33.93
w/o Ti Level
67.55 / 10.60
68.45
/ 31.78
53,20
25.25
/ 3.05
54.79
47.77
30.68
w/o Agent- Specific.
68.15 / 13.40
74.15 / 10.27
52.50
32.10 / 5.10
44.30
56.41
32.08
Self-play vs. fixed-opponent learning. Training agents against fixed, expert opponents reveals an intuitive and critical flaw: the agents overfit. As shown in Table 3, these models fail to generalize, exhibitinga significant performance drop on games outside their direct training domain. This is particularly evident with the Kuhn Poker specialist, which suffers a severe degradation on non-poker games clear case of strategic mode collapse. These results confirm that the adaptive curriculum provided by self-play is essential for the development robust, generalizable policies.
Analysis of advantage estimation design. We then ablate our two-part advantage esti- mation technique. First, reverting from our fine-grained turn-level advantage estimator to coarse, trajectory-level reward signal causes a significant performance drop, especially for the Tic-Tac-Toe specialist, which is trained a long-horizon game. Second, removing agent- specific advantage normalization also degrades performance, particularly in competitive games- trained model like the Tic-Tac- Toe specialist and the Kuhn Poker specialist where player expe- riences and agent return are distinct (Fig. 6). Conversely, the effect is mild in the Hanabi spe- cialist, which is trained on a cooperative game with similar return distribution between players (Fig. 6). Together, these results confirm that both components of our advantage estimation are crit- ical for effective learning in complex multi-turn, multi-agent environments.
Return Distribution Tic-Tac-Toe
Player
player me
Denstty Tonntyy
Player
player 0.228
-1.0
-0.5
0.0
0.5
1.0
1.5
Return
Return Distribution Mini Hanabi
Player
pllaerr meon=1.3
Density
Player
player meare1111
Raturn
Figure 6: Return distribution of the generalist MARSHAL agent as different player roles Tic- Tac-Toe and Mini Hanabi.
9

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
LLM-based multi-agent systems. To further extend the ability of LLMs with collaborative in- telligence, LLM-based multi-agent systems have been proposed for various tasks, AutoGen (Wu et al., 2024a) and CAMEL (Li et al., 2023) design cooperative agents for general reasoning and question answering. MetaGPT (Hong et al., 2024) and ChatDev (Qian et al., 2023) propose static multi-agent workflows with specialized roles for software development tasks. Multi-Agent Debate (MAD) (Liang et al., 2023) takes a competitive approach by letting LLMs propose and criticize solutions over multiple rounds for the final answer. These works mainly focus on designing workflows for multi-agent interactions with fixed LLMs. Other works have proposed to generate workflows automatically (Zhang al., 2024a). In comparison, our work takes complementary approach by training LLMs to enhance their multi-agent reasoning capabilities and build stronger foundation models that can be integrated with these frameworks.
Reinforcement learning for LLMs. Reinforcement learning (RL) has emerged as a prominent approach enhance LLMs' ability from instruction following to reasoning. To align LLMs with human values, RI from human feedback (RIHF) (Quyang et al., 2022) and from AI feedback (RLAIF) (Bai et al., 2022) have been widely used in post-training steer models towards favorable behaviors. The recent success of reasoning models (Jaech et al.. 2024; Guo et al., 2025) further popularizes RL from verifiable rewards (RLVR) for incentivizing long Chain-of- Thhoughht reasoning capability of LLMs and has been applied in mathematical reasoning (Shao et al., 2024), coding (Wei et al., 2025), and tool-using (Jin et al., 2025). However, these works primarily consider single-turn or single-agent tasks, while our approach extends RL training to multi-turn, multi-agent scenarios. Recently, MT-GRPO (Zeng et al., 2025) also address the turn-level credit assignment issue for multi- turn RL. However, they propose to normalize the turn-level rewards turn-by-turn, which can introduce variance in strategic games with varying numbers of turns. We propose a "sum-ther then-hhn-- wet-tteeennomaazzzt approach that takes advantage of all turns in the normalization, increasing robustness.
Self-play training of LLMs. The great success of achieving superhuman performance in multi-agent games (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019) has spurred the use of self- self-play play to train LLMs by playing with themselves (Zhang et al., 2024b). Some work (Chen et al., 2024; Wu al., 2024b) combines the game-the property of self-play to overcome the intransitivity of human preference in LLM alignment. Another line of work (FAIR et al., 2022; Xu et al., 2023; 2025a) uses self-play improve the strategic ability of LLM agents in specific games. The most relevant work to ours is SPAG (Cheng et al., 2024) and SPIRAL (Liu et al., 2025), which also train LLMs via self-play to show generalization to reasoning tasks. However, this work focuses on zero-sum games and considers generalization in single-agent evaluations. In contrast, our work consider both cooperative and competitive games and demonstrate generalized improvement in multi-agent systems. As concurrent work to ours, SPIRAL also addresses the heterogeneity of player roles in advantage estimation and introduces a Role- CaleConnitioned Advantage Estimation (RAE), analogous to our agent-specific normalization. We advance this finding by revealing that role separation is only critical in games with distinct return distributions between players.
6 DISCUSSION
In this work, we introduced MARSHAL, a framework utilizing self-play in a diverse range of cooper- ative and competitive strategic games to cultivate multi-agent reasoning capabilities in LLMs. Our findings demonstrate that skills honed in strategic games directly translate to enhanced performance in general multi-agent systems, establishing self-play as scalable paradigm for training LLM agents.
Despite these promising results, limitations remain. First, our study utilizes two-player games as efficient "prototypes" to cultivate foundational skills. While we demonstrate that these constrained settings are sufficient for strong generalization to standard reasoning tasks, scaling to larger N N-player environments introduces significantly greater challenges regarding non-stationarity. population diver- sity, and credit assignment that warrant dedicated future investigation. Additionally, moving beyond classic games to complex "social sandboxes" " (e.g., simulated software engineering, collaborative research) represents compelling direction for training agents in more practical domains.
Ultimately, our work provides strong evidence that the principles of self-play are a powerful engine for progress, paving the way for the next generation of sophisticated L.MM agents.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All code, model checkpoints, and training scripts required to reproduce the findings of this paper publicly available https://gi thub. com/thu-nics/MARSHi The repository includes all necessary configurations to replicate our key experiments.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. The primary ethical consideration of our work is the dual-use nature of enhancing LLM agent capabilities. While our goal to foster beneficial multi-agent reasoning skills like cooperation and competition, the resulting models could potentially be applicd to malicious ends. We release our code and models to the research community to encourage further study into the safety and alignment of more capable agents.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (No.62406159 62325405), Ant Group, Beijing National Research Center for Information Science and Technology (BNRist), Beijing Innovation Center for Future Chips, Shenzhen Key Laboratory of Ubiquitous Data Enabling (No. DDSY$2022052711440000 and Tsinghua Shenzhen International Graduate School-Shenzher Pengrui Endowed Professorship Scheme of Shenzhen Pengrui Foundation.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0000 08774, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from ai feedback. arXiv preprint arXiv:22/2.00007 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large seale deep reinforcement learning. arXiv preprint arXiv:/9/2.060 2019.
Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue, Dan Jurafsky, and James Zou. How well can Iims negotiate? negotiationarena platform and analysis. arXiv preprint arXiv:2402.05883. 2024.
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, al. Why do multi-agent I1m systems fail? arXiv preprint arXiv:2503. (tt.:25033.367, 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint vrXiv:2401.01332 2024.
Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances I1m reasoning. Advances in Neural Information Processing Systems, 17:122515-226543 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint
14168, 2021.
FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 3786662) 2022.
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12245 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0338 03300, 2020.
Sirui Hong, Mingchen Zhuge. Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang. Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. International Conference on Learning Representations, ICLR, 2024.
Aaron Jaceh, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms reason and leverage search engines with reinforcement learning, arXiv preprint 1rXiv:2503.09516 2025.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:141..66 1412.6980. 2014.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Marc Lanctot, Edward Lockhart, Jean- Saan-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis Open- Spiel: A framework for reinforcement learning in games. CoRR, abs/ 0/s//998.09444 9999009880,., 2019. URL http:://axxvv.org// aarrg///b//99....../ 09453.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 06:51991-52008 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19/18, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. ACL, 2017.
Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.. 241 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
12

---

Published as a conference paper at ICLR 2026
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXhv:2307.07924, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. 2024.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1.Xiv:550.02444 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XXiv:7777.03477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro, Megatron-Im: Training multi-billion parameter language models using model parallelism. 2019.
arXiv preprint
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-88, 2016.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint 2rXiv:2501.12599, 2025.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-354 2019.
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang. Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. arXiv preprint ar arXiv:2506.06/22 2025.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Il reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502. 18449, 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 1lm applications via multi-agent conversations. In First Conference on Language Modeling. 2024a.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0067: 2024b.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.1888 18940 2023.
Zelai Xu, Wanjun Gu, Chao Yu, Yi Wu, and Yu Wang. Learning strategic language agents in the werewolf game with iterative latent space policy optimization. arXiv preprint arXiv:2502. 04686, 2025a.
Zelai Xu, Zhexuan Xu, Xiangmin Yi, Huining Yuan, Xinlei Chen, Yi Wu, Chao Yu, and Yu Wang. Vs- bench: Evaluating vlms for strategic reasoning and decision makinn in multi-agent environments. arXiv preprint arXiv:2506.02233 02388, 2025b.
13

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprins arXiv:2407.10671, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122 121222. 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Rui Ye, Keduan Huang, Qimin Wu, Yuzhu Cai, Tian Jin, Xianghe Pang, Xiangrui Liu, Jiaqi Su, Chen Qian, Bohan Tang, al. Maslab: unified and comprehensive codebase for llm-basee multi-agent systems. arXiv preprint arXiv:2505. 16988, 2025.
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Reinforcing multi-turn reasoning in Ilm agents via turn-level credit assignment. arXiv preprint /182 2025.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng. Xionghui Chen, Jiaqi Chen, Mingehen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv 2024a.
preprint arXiv:2410.1076
Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Wenhao Tang, Shiyu Huang, Deheng Ye. Wenbo Ding, Yaodong Yang, et al. A survey on self-play methods in reinforcement learning. arXiv preprint arXiv:2488.01072 2024b.
14

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
In the preparation of this paper, LLMs were utilized writing assistants. Their use were limited to improving grammar, clarity, and overall readability. The core research, including the methodology. experiments, and analysis, represents the original work of the authors.
B IMPLEMENTATION DETAILS
Framework and software stack. Our implementation of MARSHAL is built upon ROLL (Wang al.. 2025), a robust open-source codebase for post-training LLMs with reinforcement learning. ROLL's native support for agentic, multi-turn rollouts provided a strong foundation for our system. To achieve high performance, ROLL leverages vLLM for efficient inference during phase (Kwon et al., 2023) and is built on Megatron-LM for distributed training (Shoeybi et al., 2019). All game environments used in this work were implemented using OpenSpiel (Lanctot et al., 2019) and VS Bench (Xu et al., 2025b), ensuring the correctness and standardization of game logic.
rollout
Training settings. We use Qwen-4B as base model for all experiments reported in our main results. The training is conducted in fully online manner, where self-play trajectories are immediately used for policy updates. Specialist agents are trained with batch size of 128 trajectories, while the generalist agent uses a batch size of 384 (128 from each game). All models were trained for total of 200 optimization steps. For optimization, we employ the Adam optimizer (Kingma, 2014) with cosine annealing learning rate schedule, which warms up for 10 steps to a peak of 1 x 10-6 before gradually decaying to O.
Generation parameters. During the self-play rollout phase, text is generated via nucleus sampling. We use temperature of 0.6, Top-P of 0.99, and Top-K of 100. These parameters were selected to encourage a diverse yet coherent exploration of strategies.
Hardware configuration.
Models were trained on single server with 8 NVIDIA H100 GPUs.
C HYPERPARAMETERS
To ensure reproducibility and consistency, we maintain unified hyperparameter configuration for all trained models. The complete settings provided in Table 5.
D EVALUATION DETAILS
Strategic ability. To rigorously assess game-playing proficiency, we evaluate our agents against suite of strong, fixed opponents. For competitive games, this includes:
Tic- Te--ac-Toe and Connect Four: two MCTS agents with varying simulation counts (100/1000 for Tie-Tac-Toe, 10/100 for Connect Four) to test against different strengths.
Kuhn Poker: The exact Nash Equilibrium (NE) policy.
Leduc Hold'em: NE policy approximated by 5 x 10 iterations of Counterfactual Regret Minimization (CFR).
In all competitive games, models evaluated both the first-move and second-move player, measured by the average game return. For the cooperative Hanabi variants, performance measured by the standard self-play game return. In all game settings, agents evaluated for 1000 games.
For the results on strategic ability reported in the main text, we normalize the scores with respect to the wosst-ob performance and the theoretical optimal performance.
For the -ixeentpponent ablation studies, the MCTS agent with 100 simulations is used for Tic-Tac Toe the opponent, and the NE policy is used for Kuhn Poker.
15

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters
Category
Parameter
Value
Max Response Length
4096
Model Configuration
Max Sequence Length Dtype
32768
bf16
Max Steps
200
Train Batch Size
128
Optimizer
Adam
Adam parameters (31, 82)
(0.9, 0.95)
Training Settings
Learning Scheduler
Cosine Annealing
Learning Rate
x 10
Weight Decay
0.05
Warm-up Step
10
Gradient Norm Clip
1.0
Sampling Temperature
0.5
PPO Epochs
1
(top P, top k)
(0.99, 100)
KL Loss
Truc
Loss Coefficient
RL Settings
Entropy Coefficient
0
Dual Clip Loss
ture
PPO Policy Clip
0.2
Lambd
0.95
Gamma
Generalization to multi-agent systems. For downstream evaluations, we assess all models on comprehensive suite of mathematics and general QA benchmarks. In the single-agent setting, we utilize evaluation scripts from Qwen2.5-Man (Yang et al., 2024a;b) for MATH500, GSM8K, AQUA- RAT, AIME24, AMC23, and MMLU-STEM, while employing Im-evaluation harness (Gao et al., 2024) for GPQA-Diamont For multi-agent evaluations, adopt the MASLab framework (Ye et al., 2025) across benchmarks. To balance evaluation efficiency and accuracy, we set the maximum output length 8,192 tokens for all settings.
E SCALING TO LARGER MODELS
To validate the scalability of the MARSHAL framework and ensure findings not specific to the 4B parameter scale, we extended our training to the larger Qwen3-8B model. We trained an 8B-parameter MARSHAL generalist agent using the exact same of strategic games (Tic-Tac-Tore Kuhn Poker, and Mini Hanabi) and hyperparameters as the 4B experiments.
Strategic ability. First, we evaluate the model's proficiency on both training and held-out games. As shown in Table 6, the MARSHAL-SE agent achieves significant improvements over the Qwen3-88 WBB base model across all environments. Notably, we observe the same strong generalization pattern seen in the 4B model: the agent generalizes effectively to held-out, complex games, achieving a drastic improvement Leduc Hold'en em (7.26% to 53.89%) and Simple Hanabi (4.55% 37.27%)
Table 6: Strategic ability comparison on Qwen3-8B. The MARSHAL training yields consistent improvements on both training games and held-out testing games (Connect Four, Leduc Hold' em. Simple Hanabi).
Model
Tic- Ric-Tae--e Tue
Kuhn Poker
Mini Hanabi
Connect Four
Ledue Hold em
Simple Hanabi
Qwen3-8B
48.38
33.12 44.49
27.00 55.28
10.48 21.55
7.26
4.55
MARSHAL (Generalist, 8B)
54.05
53.89
37.27
We further evaluate whether these improved strategic capabilities translate general reasoning benchmarks within multi-agent systems. We integrated
Generalization multi-agent systems.
16

---

Published as a conference paper at ICLR 2026
the MARSHAL-SB agent into both the competitive MAD framework and the cooperative AutoGen framework.
The results, detailed in Table 7, confirm the scalability of our approach. The MARSHAL-trained 8B model consistently outperforms the base model:
In the competitive MAD setting, observe an average improvement of 2.60%, with substantial gains on challenging math benchmarks like AIME (70.00% 80.00%).
In the cooperative AutoGen setting, the improvement is more pronounced, with average gain 3.90%, and a significant boost on AIME (60.00% to 70.00%)
These results demonstrate that the MARSHAL algorithm and game selection scale stably to larger models, consistently unlocking cooperative and competitive reasoning capabilities.
Table 7: Generalization multi-agent systems using Qwen3-8B. The MARSHAL generalist consis- tently improves performance on reasoning benchmarks in both competitive (MAD) and cooperative (AutoGen) settings.
MAS
Model
Avg
MATH
GSM8K
AQUA
AIME
AMC
MMLJU
GPQA
Qwen3-8B
82,49
95.00
96.36
83,46
70.00
90.00
89.59
53.03
MAD
MARSHAL (Generalist, 8B)
85.09
96.40
96.59
83.46
80,00
95.00
90.70
$3,54
Qwen3-8B
79.68
88.80
95.91
83.07
60.00
89.19
89.30
51.52
AutoGen
MARSHAL
(Generalist, 8B)
83.58
94.40
95.00
85.04
70.00
95.00
90.04
$5.56
F COMPARISON WITH SPIRAL: DECOUPLING ALGORITHM AND GAME ENVIRONMENTS
To rigorously decouple the contributions of our proposed learning algorithm and our selection of game environments, we conducted set of controlled experiments comparing MARSHAL against SPIRAL baseline methods. We analyze two specific variations:
1. Ablation on algorithm (RAE + our games): We train an agent using SPIRAL's Role-
conditioned Advantage Estimation (RAE) our full selection of competitive and cooperative game environments. This isolates the impact of the advantage estimation technique.
2. Ablation on games (MARSHAL algorithm + competitive games): We train an agent using the MARSHAL algorithm but restricted to competitive-oni games (Tic-Tae- Toe and Kuhn Poker). similar to SPIRAL. This isolates the impact of cooperative games.
F.1 STRATEGIC ABILITY COMPARISON
Table 8 presents the performance on both training and held-out games. We analyze the results through the lens the two ablations:
Ablation on algorithm. When replacing the MARSHAL algorithm with RAE (row 2), we observe a sharp performance decline in the cooperative games. Specifically, performance Mini Hanabi falls from 54.33% 24.93%, and in Simple Hanabi from 36.75% to 5.53%. This confirms that RAE struggles with the dense, multi-turn credit assignment required for cooperative settings, validating the necessity of MARSHAL's turn-level num-then-normallin technique to unlock the potential of self-play learning.
Ablation games. When training only competitive games (row 3), the agent naturally fails to acquire cooperative skills (low scores in Hanabi). This serves as a crucial control baseline for analyzing the downstream generalization results below.
F.2 GENERALIZATION TO MULTI-AGENT SYSTEMS
We further evaluate how these variations affect generalization to standard reasoning benchmarks within multi-agent systems.
17

---

Published as a conference paper at ICLR 2026
Table 8: Strategic ability comparison decoupling algorithm and game environments. MARSHAL (row 1) outperforms both the RAE-based agent (row 2) and the competitive-only agent (row 3),
particularly cooperative environments.
Model
Tic- Dec-Ta-Too Toe Kuhn Poker
Mini Hanahi Connect Four Leduc Hold'en
Simple Hanabi
MARSHAI (Full Method)
53.13
40.05
54.33
19.98
54.56
36.75
SPIRAL (RAE) Our Games
48.35
42.05
24.93
9.70
6.11
5.53
MARSHAL Alg. + Adv. Games
48.80
47.55
24.48
17.90
29.98
11.48
Generalization to competitive systems (MAD). As shown in Table 9, the full MARSHAL method achieves the highest average performance (75.96%). Notably, the RAE-based agent underperforms (74.23%), confirming the algorithmic superiority of MARSHAL. The competitive-only agent per- forms comparably to the full agent, which is expected MAD is competitive framework.
Generalization to cooperative systems (AutoGen). Table 10 reveals the critical role of the game environments. The agent trained on the competitive-only games suffers significant performance drop in the cooperative AutoGen framework (Average 80.34% vs. 85% for MARSHAL). This confirms that cooperative games
merely "more data"; they necessary component for generalizing skills to cooperative downstream tasks, Additionally, the RAE-based agent also underperforms, again solidifying the novelty of the algorithmic design of MARSHAL.
Table 9: Generalization to the competitive MAD framework.
Model
Avg
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
MARSHAL (Full Method)
75.96
92.80
95.60
46.67
45.45
SPIRAL (RAE) + Games
74.23
92.40
36.67
77.50
86.79
45.96
MARSHAL Alg.
Adv. Games
75.24
92.20
83.46
46.67
41.92
Table 10: Generalization to the cooperative AutoGen framework. Note the drop in performance for the competitive-only games (row 3), highlighting the necessity of cooperative training.
Model
Avg
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
MARSHAL (Full Method)
82.15
95.20
94.54
86.61
66.67
92.50
89.53
50.00
SPIRAL (RAE) + Our Games
80,39
94.80
94.77
86.61
60.00
92.50
88.57
45.45
MARSHAL Alg. Adv. Games
80.34
94,20
94.24
86.61
60.00
90.00
89.37
47.98
Conclusion. These decoupling experiments confirms the superiority of MARSHAL's algorithmic design over SPIRAL's RAE, and that MARSHAL's successful generalization is not due one factor alone. It requires both our selection of competitive and cooperative games to provide necessary learning signals, and our novel credit assignment algorithm effectively learn from them.
G COMPARISON WITH MT-GRPO
concurrent work MT-GRPO (Zeng al. (2025)) also identifies the limitations of trajectory-level estimation in the original GRPO and proposes a fine aineigraneed turn-level strategy. While MT MT-GRPO shares our motivation, and also critical first step towards fine-grain credit assignment multi-ture GRPO, our "sum-the "sum-then-nen then-normalize approach differs fundamentally from their strategy, offering distinct advantages in stability and scalability in the game tasks that we consider in this work.
G.1 THEORETICAL DIFFERENCES
MT-GRPO relies on normalizing the turn-level rewards turn-by-tumn (i.e., computing the mean and variance for the reward at step across the batch). We identify two structural limitations with this approach that MARSHAL. addresses:
18

---

Published as a conference paper at ICLR 2026
1. The variable length problem: In real-world reasoning and gaming tasks, trajectory lengths vary significantly. With tum-by-turn normalization, the effective batch size shrinks as shorter trajectories conclude. This causes the variance of the normalization statistics to increase for later turns the sequence, introducing instability for training. In contrast, MARSHAL. normalizes the cumulative return, which remains defined for the entire trajectory regardless of length.
2. Immediate reward vs. long-term return: MT-GRPO's local normalization focuses on the immediate step-wise reward relative peers that specific step. MARSHAL calculates the full Monte Carlo return first. By normalizing the cumulative outcome, capture the long-term impact of actions relative to the global baseline, preserving critical long-term dependencies often obscured by local normalization.
G.2 EMPIRICAL COMPARISON
To validate this theoretical analysis, we implemented MT-GRPO and compared it directly against MARSHAL. We trained Tic- Tac-Toe specialist using both algorithms and evaluated their strategic proficiency across our full game suite.
Strategic ability. As shown in Table 11, the MT-GRPO agent exhibits notable performance drop across both the training game (Tic-Tac-Toe) and generalization held-out games (e.g.. Connect Four, Hanabi). In particular, the drop significant in Mini Hanabi (50.48% to 36.67%) game requiring consistent long-term coordination. This confirms that our 'sum-then-normalli formulation is more robust for the variable-length, multi-turn interactions inherent in strategic games.
Generalization MAS. We further extended this comparison to the integration within agent systems to assess if the stability issues in game training affect generalization. Table 12 present the results in the competitive MAD and cooperative AutoGen frameworks, respectively. MARSHAL consistently achieves higher average scores (75.01% % vs. 73.77% in MAD; 80. 155 vs. 79.10% in AutoGen) and outperforms MT-GRPO on key hard reasoning benchmarks like AIME and AMC. This indicates that the training proficiency provided by MARSHAL translates directly to better robust reasoning in general multi-agent tasks.
Table 11: Comparison between MARSHAL and MT-GRPO. The MARSHAL advantage estimation yields superior performance and generalization compared to the concurrent MT-GRPO method, validating the robustness the
approach.
Model TTc-Tac-Tor Specialist) Tic Tic-Tac-Toe Kuhn Poker Hanabi Connect Four Ledue Hold' em Simple Hanabi
MARSHAL (Ours)
53.70
38.79 40.05
50.48 36.67
22.75 18.55
43,00
29.75
MT-GRPO
50.10
39.94
20.08
Table 12: Comparison of generalization to multi-agent systems between MARSHAL and MT-GRPO (using Tie-Tac- specialists). MARSHAL achieves higher average performance and demonstrates
greater stability in downstream tasks.
MAS
Model (Tie-The Toe Specialist)
Avg
MATH
GSMSK
AQUA
AIME
AMC
MMLU
GPQA
MARSHAL (Ous)) MT-GRPO
75.01
92.20
96.06
83.07
43.33
82,50
86.76
41.12
MAD
73.77
92.60
95.91
84.65
36.67
77.50
86.91
42.13
AutoGen
MARSHAL (Ours) MT-GRPO
87.01
60.00)
89.53
79.10
95.40
94.62
85.04
56.67
90.00
89.02
42.93
H HYPERPARAMETER ANALYSIS ON RESPONSE LENGTH PENALTY
To ensure efficient and stable reasoning within the context window limits, reward design includes a response length penalty characterized by three hyperparameters: the minimum length bmins the maximum threshold / and the penalty weight a.
19

---

Published as a conference paper at ICLR 2026
Selection of length constraints. constraints:
The length bounds were chosen based practical engineering
This is the hard lower bound determined by the minimum number of tokens required to generate a syntactically valid response structure with the correct format (e.g.. iiiiikk-//thinn--actiii <//ctiion)))
min 11:
Imax = 2048: This soft upper bound is derived from the context window of the base model. Given Qwen3-4B's 32k context length, allocated budget of approximately 16 turns per game history (32, 768 /6 = 2048), which provides a sufficient horizon for the reasoning traces in our chosen game portfolio while preventing context overflow.
Ablation on penalty weight To assess the sensitivity of the framework to the penalty weight a, we conducted an ablation study training Tic-Tac- Tac- Toe specialists with a 1.0 (stronger regularization), 0.5 (default), and = 0.0 (no regularization). The results are detailed in Table 13.
Table 13:
Ablation study on the length penalty weight a. The default setting (a = 0.5) strikes balance between performance and conciseness. Removing the penalty (a 0) leads to overly verbose responses and performance degradation in cooperative games like Hanabi, "OL" denotes the percentage of games lost due overlong responses.
Strategic Performance
Response Statistics
Model Setup
Tic-Tac- Toe
Kuhn Poker
Mini Hanabi
Connect Four
Leduc Hold em
Simple Hanahi
Avg Len
OL in
Mini
Simple
OL
Hanabi
Hanabi
0.5 (Default)
53.70
38.79
50.48
22.75
43.00
29.75
1700
9.6%
14.3%
1.0 (Strong)
54.90
42.82
47.53
18.25
50.46
26.05
1657
7.7%
13.9%
0.0 (None)
54.65
40.95
38.18
23.35
28.61
20.10
1954
20.4%
27.3%
Analysis. The results yield two key conclusions:
1. Robustness:
The framework is robust reasonable variations in a. Increasing the regularization to 1.0 maintains comparable strategic performance to the default setting, indicating that the method not overly sensitive to the precise magnitude of the penalty.
2. Necessity of regularization: The length penalty is crucial for stability. When removed 0), the average response length increases significantly (from 1700 to 1954 tokens). This verbosity leads sharp performance drop in cooperative, Hanabi (50.48% to 18)1.) This degradation correlates directly with dramatic increase in the "Overlong Response" failure rate (9.6% 20.4%), confirming that the penalty effectively prevents the model from degenerating into computationally wasteful loops.
sooory-o--mindddmmmaee
games like Mini
T GAME OBSERVATION AND PROMPT
Tic- Tac-Toe For Tic-Tac-Toe, we provide the agent with a complete observation of the 3x3 game board. The state of each well-whether it empty, occupied by 'X', occupied by 'O'-is explicitly provided. The prompt clearly indicates which player's turn it is ("X' or 'O') and presents the current board state, asking the agent select coordinates for its next move from the available empty cells. For example, the game begins with a prompt that provides the empty 3x3 grid and asks the agent to make the first move (Listing 1).
Listing Prompt for Tic-Tac-Toe
system_prompt:
You are an AI agent that makes optimal decisions to win in the game of Toc-T.c-
user_prompt:
GAME RULES:
1. Tic-Tac- Toe is two -layer board game played on three-by- three grid The grid is 0 o-ineexdd, where (0 is the top-left corner and (2, 2) is the bottom-rice corner.
20

---

2. the start of each game, both player_0 and player_1 place chip Published as a conference paper at ICLR 2026
2. Two players take placing their marks X and in empty cells of the grid.
3. The player who first places three of their marks in a horizontal, vertical, or diagonal line wins.
4. If all cells are filled and no player wins, the ends in draw.
PLAYER INFORMATION:
1. Your mark is x. You are competing with another player controlling the mark 0.
2. each of your turns:
a. The game text grid, where 'X' and '0' are the marks of the two players, and represents empty cells.
demonstrates the current board with three-line
b. You need to chose an action to place your mark in empty cell, based the given game c. All legal actions for the current turn are provided in the format of "{(((ro),,ccouum)) (/colum))>'' where 'X' is your mark, and {row} and column} are integers indicating the row and column of the cell to place your mark.
and the history of your decisions.
RESPONSE INSTRUCTIONS:
Always choose only one action from the legal actions and output '<answer >yyour chosen (stion)</answers thinking process. For example, '<answer><x follow the above format and keep your thinking process concise. Responses that do not follow the format will result in immediate loss of the game.
with no extra text after you finish the
x)><</nswwrr>'. (Bsser>>x((.0)>>//nssseee Strictly
Information of Turn-1: This is your provided belox. Please choose your action and strictly follow given output format in the response instructions.
The game
and legal actions for this
GAME STATE:
LEGAL ACTIONS:
<I(0,0) <X 1)>, <)),2)), <(1,0)>, d(1,)), d(1,2)>, <I(2,)>,
(2.1)>, <x(22)>
Kuhn Poker For Kuhn Poker, provide the agent with the observation consists of its single private card (c.g., Jack, Queen, or King) and the complete history actions of two players. The prompt structured to provide this context clearly, asking the agent decide its next action (pass bet) based on its private information and the betting history. For example, the initial prompt at the beginning of game provides the agent's private card and the empty action history, asking for the first move (Listing 2).
Listing Prompt for Kuhn Poker.
system_prompt:
You are an AI agent that makes optimal decisions to win in the game of Kuhn Poker.
user_prompt:
GAME RULES:
1. Kuhn Poker is a two-player card game. The deck includes only three cards: King (K) > Queen (Q) Jack (J).
into the pot a blind ante.
3. Each player is dealt a private card, and the third card is set aside unseen.
4. The two players take turns acting, starting with player_0. A player choose to:
21

---

Published as a conference paper at ICLR 2026
a. <PASS>: place additional chips into the pot.
b. <BET>: place 1 additional chip into pot.
5. If a player chooses to <PASS> after the other player's <BET>, the betting player wins the pot.
6. If both players choose to <PASS> both players choose <BET>, the player with the higher card wins the pot.
PLAYER INFORMATION:
1. You are player_0 You are competing with player_1.
2. each of your turns:
a. The game
shows your private card and the betting history.
b. You need to choose an action based on your card and the current
game state.
All legal actions for the current turn are provided in the format of "<PASS>'
'<BET>'.
RESPONSE INSTRUCTIONS:
Always choose only one action from the legal actions and output 'canswer >yyour chosen action)</answer>> with extra text after you finish the thinking process. For example, '<answer><PAS>><</aae anssss//////////ss. Strictly follow the above format and keep your thinking process concise. Responses that do not follow the format will result in immediate loss of the game.
Information of Turn-1:
This is your turn. The game state and legal actions for this turn are provided below. Please choose your action and strictly follow the given output format in the response instructions.
GAME STATE:
1. Blind ante: both player_0 and player_1 place chip into the pot.
2. Deal: your card is Jack (J).
LEGAL ACTIONS: <PASS>, <BET>.
Hanabi For the Hanabi variants, a cooperative imperfect information game, we provide the agent with unique observation. An agent observes the cards held by all other players but remains unaware of its own hand. The observation also includes the number remaining information and life tokens, the cards successfully played the board (fireworks), and the contents of the discard pile. The prompt is highly structured, detailing the game rules, player-specific information, and strict response formatting. For training, we use Mini Hanabi, simplified 2-player version with colors, 2 ranks, card hands, and 3 information/life tokens. For out-of-distribution evaluation, we use a more complex variant, Simple Hanabi, which features colors, ranks, 5-card hands, 8 information tokens and life tokens. For example, an initial prompt details the starting setup of the chosen variant, including all visible cards and token counts, before asking the first player to make the first move (Listing 3).
Listing 3: Prompt for Mini Hanabi.
system_prompt:
You are an AI agent that makes optimal decisions to achieve the highest score in the game of Hanabi.
user :pe_ppromp::
GAME RULES:
1. Hanabi is a cooperative card game for players, player and player
1.
2. The deck consists of 2 colors: Red by R), Yellow by r Each color contains 4 cards: three of rank 1, and one of rank 2, for a total of cards.
with ranks ranging from
3. Each player holds 3 cards in hand. Players can observe the hand of the other player, but not their own.
4. There are 3 information tokens and 3 life tokens shared by both players.
22

---

Published as a conference paper at ICLR 2026
5. The objective is to play cards in ascending order of rank, from
2, their corresponding color stacks, hence achieving the 'Fireworks
6. The players take turns to take one of the following actions:
a. <Play 'i'>: play the 1-th card from the player's 's own hand (0- the card of its
indexed). If the card is sequential corresponding color stack, the the top of the stack, then both players receive point. Otherwise, a life token is lost.
valid and the card is added
b. <Discard 'i'>; discard the i-th card from the player's own hand information token.
and gain c. <Reveal player +1 color 'c'>: spend reveal all cards of color 'c' in the other player 's hand.
information token to
d. <Reveal player +1 rank 'r'x: spend one information token to reveal all cards of rank 'r' in the other player's hand. discarding, the player receives deck (if remaining).
7. After playing
card from the
8. The game ends when:
a. If all color stacks are completed (i.e., all cards of rank 2 are their corresponding color stacks), then both players finish the game with the highest possible total
played
b. deck is depleted, both players finish the game with a total score which equals the sum of the highest ranks of each color stack. c. If all life tokens are lost before the above two conditions are then both players lose all points they have earned far, and finish the game with total score of 0.
PLAYER IMFORMATION:
1. You will be playing as the player 8.
2. each of your turns, you will be provided with the current game state information, including the remaining life tokens information tokens, the current color stacks, the remaining deck size, the discard pile, the hand of the other player, and the revealed information on your own hand.
3. Known cards are denoted by their color and rank. For example, 'R2' means a red card of rank 2. The current color stacks are represented by the top card of each color stack. In particular, rank 0 denotes an empty stack. For example, 'Yo' means the yellow stack is still empty.
RESPONSE INSTRUCTIONS:
Always choose only one action from the legal actions and output '<answer >yyour chosen action)</answer> thinking process. For example, '<answer><Discard Card 0></answer>'. Strictly follow the above format and keep your thinking process concise. Responses that do not follow the format will result in immediate loss of all life tokens and end of the game.
with extra text after you finish the
Information of Turn-1:
This is your turn. The game state and legal actions for this turn are provided below. Please choose your action and strictly follow the given output format in the response instructions.
GAME STATE:
1. There are 3 life tokens and 3 information tokens remaining.
2. The top of the color stacks are: RO Yo.
3. cards remain in the draw pile.
4. The discard pile currently contains: None.
5. The other player's hand: Card (Y1): the other player believes it is one of the colors [R, Y] and one of the ranks [1, 2].
Card (R1): the other player believes it is one of the colors [R,
Y] and
of the ranks [1, 2].
Card (R1): the other player believes it one the colors [R,
Y] and one of the ranks [1, 2].
6. Your own hand, based on the revealed information:
Card 0: of the colors [R. Yl and of the ranks [1. 2].
23

---

Published as a conference paper at ICLR 2026
Card 1: of the colors [R. Y] and of the ranks [1, 2]. board, showing the positions of all 'X' and 'O' marks. The prompt is designed to comprehensive, initially presenting the rules of the game (connecting four pieces to win, draw conditions). It then informs the agent of its assigned mark ('X' or 'O') and the opponent's mark. For each turn, the prompt presents the current board state and asks the agent to choose column (0-6) to drop its piece. For example, the game begins with prompt that provides the empty 6x7 grid, explains the rules and player marks, and asks the first player choose column (Listing 4).
Card 2: one colors [R, and one the ranks [1, 2].
LEGAL ACTIONS:
<Play card 0>, <Play card 1>, <Play card 2>, <Reveal player +1 color R>, <Reveal player +1 color Y>, <Reveal player rank 1>.
Connect Four For Connect Four, we provide the agent with complete observation of the 6x7 game
Listing 4: Prompt for Connect Four.
system_prompt: You are an AI agent that makes optimal decisions to win in the game of Connect Four.
user_prompt:
GAME RULES:
1. Connect Four is two two-player board game played on a 6x7 grid. Players take turns dropping their pieces into columns.
2. The goal is connect four of your pieces horizontally, vertically,
or diagonally.
3. Pieces fall to the bottom of the column or stack on top of existing pieces.
4. The first player to connect four pieces wins. If the board fills up without winner, it's draw.
PLAYER INFORMATION:
1. Your mark is x. You are competing with another player controlling the
mark
2. In each your turns:
a. The game state shows the current board as 6x7 grid.
b. You need to choose a column (0-6) to drop your piece, where 0 denotes the leftmost column, 6 denotes the rightmost column. provided '<X(fcolumn))", where 'I' is
All legal actions your mark, and {column) is the column number.
RESPONSE INSTRUCTIONS:
Always choose only one action from the legal actions and output '<answer >yyour chosen column)</answer> with no extra text after you finish the thinking process. For example, <<ansver><d33> the above format and keep your thinking process concise. Responses that do not follow the format will result in immediate loss of the game.
Strictly follow
Information of Turn-l:
This is your turn. The game state and legal actions for this turn are provided belox. Please choose your action and strictly follow the given output format in the response instructions.
GAME STATE:
LEGAL ACTIONS:
24

---

Published as a conference paper at ICLR 2026
<I(0>> <x11>> <x(2>>, <x(3>>, <x()>> <(()>, <(
Leduc Hold'em For Leduc Hold' em, we provide the agent with the observation which contains all necessary information for decision-making. The observation is complex due two betting rounds and a public card. In the first round, the agent observes its private card and the betting history. In the second round, after a public card is revealed, the observation is updated to include this board card. The prompt provides the agent with its private card, the current betting history, and the public card if revealed, requiring to make an action the context of the evolving game state. For example, the initial prompt at the beginning of game provides the agent with its private card and asks for an action in the first betting round before any actions have been taken (Listing 5).
Listing 5: Prompt for Leduc Hold em.
system_prompt:
You are an AI agent that makes optimal decisions to win in the game of Leduc Paker.
user_prompt:
GAME RULES:
1. Leduc poker is tlo-er card game. The deck includes only six pairs of King (K), Queen (0), and Jack (J). 2. the start of each game, both player_ and player_1 place chip
cards:
into the pot as a blind ante.
3. Each player is dealt one private card from the deck, and the remaining cards are set aside unseen.
4. The game has
betting rounds. When the first round ends, public card from the remaining cards of the deck revealed to both players.
5. The two players take turns acting in the betting rounds, both starting with player_0. A player can choose to:
<FOLD>: stop betting and the other player wins the pot. b. <CALL>: match the current bet. If bet has been made in the current round, this is equivalent to checking.
c. <RAISE>: first match the current bet and then add 'n' chips to the bet, where 'n=2" in the first round and 'n=4" in the second round. If bet has been made in the current round, this is equivalent to betting 'n' chips.
6. maximum of two <RAISE>s are allowed in each round. Each round ends when both players have acted and their bets are equal.
7. If player chooses to <FOLD>, the other player wins the pot.
8. If neither player chooses <FOLD>, the second round ends with showdown:
a. If a player has pair (private card public card), the player wins the pot.
b. If neither player has pair, the player with the higher card (K Q J) wins pot.
c. two players have the same card the players split the pot.
PLAYER INFORMATION:
1. You player_0. You competing with player_1.
2. each of your turns:
a. The game state shows your private card, public card (if revealed),
and the betting history.
b. You need to choose action based your cards and the current game state.
с. All legal actions for the current turn are provided in the format of '<FOLD>', '<CALL>', or '<RAISE>'.
RESPONSE INSTRUCTIONS:
Always choose only one action from the legal actions and output '<answer >[your chosen action)</answen thinking process. For example, the above format and keep your thinking process concise. Responses that do not follow the format will result immediate loss of the game.
with no extra text after you finish the
Strictly follow
25

---

Published as a conference paper at ICLR 2026
Information of Turn-1:
This is your turn. The game state and legal actions for this turn are provided belox. Please choose your action and strictly follow the given output format in the response instructions.
GAME STATE:
1. Blind ante: both player_0 and player_1 place chip into the pot.
2. Deal: your card is J.
LEGAL ACTIONS: <CALL>, <RAISE>
FULL RESULTS
In this section, we present the raw (unnormalized) results for all experiments in the main text.
Table 14: Full results of game play return in training games
Tic-Tac-Toe
Model
MCTS (100 Simulations)
MCTS (1000 Simulations)
Kuhn Poker
Mini Hanabi
Qwen3-4B
0.403/-0.629
-0.374/-0.687
-0.148/-0.141
1,200
SPIRAL
0.470/-0.624
-0,297/-0.717
-0.301/-0.148
1,494
MARSHAL
Tie-Ta--Toe
0.506/-0.358
-0.278/00379
-0.119/00,
2.019
Kuhn Poker
0,397/-0,490
-0.888/005538
00,107/-0.100
1.799
Mini Hanabi
0.438/-0.644
-0.350/-0.702
-0.123/-0.
2.222
Generalist
0.544/-0.419
-0.302/-04450
-0.114/-0.141
2.173
MARSHAL (w/ Fixed Opponent)
Tic- Tae-Tac-Toe 0.760/-0.16 161
-0.378/-0.161
-0. 1466/000000
1.397
Kuhn Poker
-1.000/-1.000
-1.000/-1.000
-0.114/-0.117
0.000
Mini Hanabi
MARSHAL (w/o Turn-Level Advantage Estimator)
Tic-Tac Tae-Tae-Te
0.492/-0.517
-0.292/-0.567
0.1104/0.091
1.392
Kuhn Poker 0.267/-0.607
-0.426/-0.625
-0.074/-0. 106
1.666
Mini Hanabi
0.351/-0.788
-0.400/-0.811
-01133/0.004
2.128
MARSHAL (w/o
Advantage Normalization)
Tic- Tac-Toe
0.654/-0.376
-0.213/-03366
00.127/-0.126
1.764
Kuhn Poker
0.391/-0.509 0.363/-0.732
-0.446/-0.567
-0.1 16.116/-0. 109
1.607
Mini Hanabi
-0.364/-0.764
-0.119/-0.128
2.100
26

---

Published as a conference paper at ICLR 2026
Table 15: Full results on game-play return in testing games
Connect Four
Model
MCTS (10 Simulations)
MCTS
Leduc Hold'en em
Simple Hanabi
(100 Simulations)
Qwen3-4B
0,,462/-09922
0.902/-0.998
-0.629/-0.69 -0.870/-0.706
0.833
SPIRAL
-0.3833-0.833
-0.868/-0.995
0.836
MARSHAL
Tic-Tac-Toe The
-0.387/-0,
-0.858/-0.974
-0.518/-0.702
1.785
Kuhn Poker
00.448/-0,74 746
-0.874/-0999
-0.460/-0.321
1.761
Mini Hanabi
0,,465/008885
-0.880/00.99
-0.736/-0,400
2.036
Generalist
0.421/-0.780
-0.871/-0.979
-0.483/-04487
2.205
MARSHAAL (w/ Fixed Opponent)
Tic-Tac-Toe -0.593/-0.887
-0.939/-0.990
-0.632/-0.616
0.733
Kuhn Poker -1.000/-1.000
-1.000/-1.000
-1.124/-1.003
0.000
Mini Hanabi
MARSHAL (w/o Turn-Level Advantage Estimator)
Tic- Tic-Tac-Toe
-0.465/-0.755
0.843/-D.988
-0.622/-0.553
1.143
Kuhn Poker
-0.408/-07883 0..995/009999
0.899/-0.998
-0.789/-0.663
1.379
Mini Hanabi
0.886/-0.999
-0.555/-0.458
1.841
MARSHAL (w/o Agent-Specific Advantage Normalization)
Tie-Tac-Tos
-0.492/-0.790
0.8822/0.983
0.594/-0.460
1,303
Kuhn Poker
-0.460/-0.790 -0.358/-0.896
-0.908/-0.992
-0.753/-0.769
1.345
Mini Hanabi
-0.902/-0.998
-0.664/-0.389
1.925
Table 16: Full results on general benchmarks in standard single agent setting
Model
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
Qwen3-4B
87.60 87.50
94.60
39.80
36.70
70.00
57.10
39.39
SPIRAL
94.80
51.20
36.70
80.00
58.70
37.37
MARSHAL
Tic-Tac-Ted Toe
89.10
95.20
46.50
40.00
77.50
57.60
38.89
Kuhn Poker
87.80
94.50
48.40
33.30
72.50
59.30
33.84
Mini Hanabi
88.10
94.70
48.00
43.30
65.00
58.90
36.36
Generalist 89.90
94.60
52.00
33.30
75.00
59,90
34.85
MARSHAL (w/ Fixed Opponent)
Tic-Tac-Toe 89.10
94.80
53,90
50.00
72.50
63.00
37.88
Kuhn Poker
86.20
94.00
45.70
33.30
72.50
53.80
33.84
Mini Hanabi
MARSHAL (w/ Turn- Level Advantage
Estimator)
Tic-Tac Toc-Toe
88.80
94.80
48.80
40.00
72.50
57.70
36.36
Kuhn Poker 88.60
94.00
48.40
40.00
72.50
57.90
34.34
Mini Hanabi 87.80
94.50
48.40
36.70
70.00
59.30
36.87
MARSHAL (w/o Agent-Specipn Advantage Normalization)
Tic-Tac-Tos
88.20
94.50
53.10
36.70
77.50
58.20
39,39
Kuhn Poker
88.10
94.80
53.90
40.00
75.00
58.70
35.86
Mini Hanabi 88.60
93.90
46.10
40.00
75.00
57.00
38.89
27

---

Published as a conference paper at ICLR 2026
Table 17: Full results on general benchmarks using the MAD framework
Model
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
Qwen3-4B
90.20
95.91
80.71
40.00
75.00
87.42
37.88
SPIRAL
91.60
95.45
81.89
40.00
77.50
87.01
40.40
MARSHAL
Tic-Tac-Toe
92.20
96.06
83.07
43.33
82.50
86.76
41.12
Kuhn Poker
91.60
96.21
82.68
40.00
82.50
87.39
41.41
Mini Hanabi
91.40
95.60
82.68
43.33
77.50
87.04
38.38
Generalist 92.80
95.60
83.86
46.67
80.00
87.36
45.45
MARSHAL (w/ Fixed Opponent)
Tic-Tac-TIe The
94.80
94.54
85.04
53.33
90.00
88.57
42.93
Kuhn Poker 95.00
94.24
86.22
53.33
85.00
88.54
45.45
Mini Hanabi
MARSHAL (w/o
Turn-Level Advantage
Estimator)
Tie-Tac-Tee
93.80
94.77
85.04
60.00
85.00
88.98
47.98
Kuhn Poker
95.00
94.54
86.22
56.67
90.00
89.05
50.00
Mini Hanabi 94.20
94.62
85.43
53.33
90.00
88.51
46.97
MARSHAL (w/o Agent-Specipp Advantage Normalization)
Tic-Tac-Te
94.40
94.62
84.65
56.67
87.50
89.33
46.97
Kuhn Poker 95.00
94.69
84.25
56.67
85.00
89.62
47.47
Mini Hanabi
94.20
94.47
87.01
63.33
90.00
88.92
43.94
Table 18: Full results on general benchmarks using the AutoGen framework
Model
MATH
GSM8K
AQUA
AIME
AMC
MMLU
GPQA
Qwen3-4B
93.40
94.69
85.04
56.67
87.50
89.21
47.47
SPIRAL
94.20
94.47
86.61
60.00
87.50
91.60
45.96
MARSHAL
Tic-Tac-Toe
94.40
94.69
87.01
60.00
90.00
89.53
45.45
Kuhn Poker
95.80
94.39
86.61
63.33
92.50
89.65
48.48
Mini Hanabi
94.40
94.54
86.22
66.67
95.00
88.98
44.95
Generalist 95.20
94.54
86.61
66.67
92.50
89.53
50.00
MARSHAL (w/ Fixed Opponent)
Tic-Tac-Toe
92.40
95.38
83.07
43.33
80.00
87.11
43.94
Kuhn Poker
90.20
95.83
83.46
33.33
72.50
86.69
42.42
Mini Hanabi
MARSHAL (w/o Turn-Level Advantage
Estimator)
Tic-Tac- Toe
91.00
95.75
82.28
40.00
72.50
86.34
42.93
Kuhn Poker
91.80
95.45
82.68
36.67
82.50
86.79
41.92
Mini Hanabi 91.20
95.91
83.86
40.00
75.00
87.33
41.92
MARSHAL (w/o Agent-Specifie Advantage Normalization)
Tic-Tac Te--Tac-Te
91.20
95.83
83,46
40.00
80.00
87.14
38.58
Kuhn Poker
91.20
95.53
84.65
36.67
75.00
86.63
43.94
Mini Hanabi
92.60
96.06
82.68
33.33
80.00
87.11
43.43
28


Reviews:
[{"review_id": "U7kxUFcU22", "reviewer": "Reviewer_yhYi", "summary": "The paper proposes MARS, an end-to-end reinforcement learning framework to improve multi-agent reasoning in LLMs via self-play across both cooperative and competitive strategic games. The key issue solved is that, direct applying GRPO to the multi-turn, multi-agent structure of self-play will introduce significant challenges of long-horizon credit assignment and agent- specific advantage estimation. MARS propose a turn-level, sum-then-normalize advantage estimator for long-horizon credit assignment within multi-turn trajectories, and Agent-specific advantage normalization to handle heterogeneous roles and asymmetric payoff scales in multi-agent settings. Plenty of experiments show that MARS trains Qwen3-4B can bring Improved strategic performance.", "questions": "1. In eq.4, how the parameters alpha and l are choose? Is the performance is sensitive to values?\n2. For agent-specific normalization, if the sub-groups are sub-groups (e.g., one role acts more frequently or has lower variance rewards)? Is the formulation adaptive to weight or normalize per subgroup?\n3. In MAD/AutoGen, do competitive-trained agents consistently outperform cooperative-trained agents in debate, and vice versa? \n4. When generalized to multiagent system, do the MARS models need further optimization? Can the author give more explanation, why the MARS collaborative and competitive capabilities are useful for mathematics and general QA task? In tab.1 MARS(multi.) doesn’t have the best performance when testing.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem motivation and the solution is very clear, i.e. long-horizon credit assignment and role heterogeneity for multi-turn multi-agent RL with LLMs.\n2. The method is very simple as an improvement to GRPO, including Turn-level advantage estimator and Agent-specific advantage normalization, \n3. The experimental setting is diverse, including a portfolio of six strategic, two-player games to cultivate a mixed set of multi-agent capabilities, both debate (MAD) and cooperation (AutoGen), as well as fixed opponents (MCTS/NE/CFR) and detailed MAS benchmarks (MATH, GSM8K, AQUA-RAT, AIME24, AMC23, MMLU-STEM, GPQA-Diamond).\n4. When integrated into leading multi-agent systems,  MARS agent achieves significant performance gains of 10.0% on AIME and 12.5% on GPQA-Diamond.", "weaknesses": "1. The sum-then-normalize estimator centers on batch means across turns/trajectories. It can stabilize the result, but discards the value function structure and may be sensitive to batch composition across diverse games.\n2. Agent-specific normalization is defined over role-based subgroups. In more complex settings (N-player, heterogeneous roles, non-stationary curricula), subgroup granularity and sample efficiency could be problematic.\n3. The ability of transferring to MAS may conflate multiple factors. The MAS gains is driven by better turn-taking discipline, concise formatting, or truly strategic multi-agent reasoning? \n4. Scaling. 200 training steps and 4B base model are modest. Can the performance gain be obtained for the MAS on 7B/14B or larger models.\n5. The contribution is mainly about the advantage function computation, lacking of improvement about topology or collaboration method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P1ER3h3Ch8", "reviewer": "Reviewer_3fZ1", "summary": "MARS is an end-to-end reinforcement learning framework that enhances multi-agent reasoning capabilities of large language models through self-play in strategic games, achieving 28.7% performance improvement in games and 10.0% and 12.5% gains on multi-agent reasoning benchmarks AIME and GPQA-Diamond, respectively.", "questions": "Weak Transfer Learning Justification\n\nThe paper doesn't adequately explain why strategic game skills transfer to math problems (AIME, MATH-500....) or science questions (GPQA) (Reasoning problem). The connection between game-playing abilities and general reasoning is assumed but not validated\nNo analysis of which specific game skills contribute to reasoning improvements.\n\nThe paper doesn't discuss failure cases or when MARS performs poorly.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a genuinely novel approach by adapting GRPO (Group-Relative Policy Optimization) to multi-agent settings, which hasn't been explored before in this context\n\n2. The framework moves beyond traditional supervised learning approaches for multi-agent LLMs, establishing self-play as a viable training paradigm\n\n3. The experimental design is comprehensive, covering diverse game types and reasoning benchmarks", "weaknesses": "The paper significantly weakens its contribution by failing to compare against recent and relevant multi-agent reasoning methods that have shown strong performance. Missing Comparison with like AFlow, ToT...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1RTHUjuXqV", "reviewer": "Reviewer_dt93", "summary": "This paper introduces MARS, an RL framework for training LLMs via self-play using a mixed curriculum of cooperative (Hanabi) and competitive (Tic-Tac-Toe, Kuhn Poker) games.1 The authors claim this training, supported by two methodological tweaks to the GRPO algorithm (\"turn-level advantage estimation\" and \"agent-specific advantage normalization\") 1, leads to skills that generalize. The experimental results, particularly the transfer of these skills to downstream multi-agent systems like AutoGen and MAD 1, are quite strong and genuinely interesting.\n\nHowever, I have some *very significant* concerns about the paper's framing, its engagement with closely related prior work, and the experimental design, all of which muddy the central claims of methodological novelty.", "questions": "My overall rating is \"Borderline Accept\" because the *results* (the generalization to AutoGen/MAD) are genuinely exciting and significant. However, my final decision is entirely contingent on the authors' ability to provide satisfactory answers to the following questions, which directly address the weaknesses I've outlined.\n\n1. **\"Agent-Specific Normalization\" vs. SPIRAL's RAE:** I'm going to be very direct here. Your \"agent-specific advantage normalization\" appears to be functionally identical to the \"Role-conditioned Advantage Estimation (RAE)\" proposed in **SPIRAL (Liu et al., 2025)** 5, which is your main baseline.\n   - Can you please provide a clear, algorithmic explanation of how your method *differs* from RAE?\n   - If it is not different, this must be acknowledged. If it *is* different, why did you not compare against RAE in your ablations? This really needs to be addressed, as it undermines a core methodological claim.\n2. **Decoupling Method vs. Curriculum:** Your key comparison against SPIRAL is confounded. You changed both the algorithm *and* the game curriculum. How can you be sure that your performance gains are not *entirely* due to your superior curriculum (i.e., just adding Hanabi)? To support your methodological claims, you *must* provide evidence that decouples these factors. Can you provide results for either:\n   - **(a)** The SPIRAL algorithm (with RAE) trained on *your* mixed (Tic/Kuhn/Hanabi) curriculum?\n   - **(b)** Your MARS algorithm trained on SPIRAL's *adversarial-only* curriculum?\n3. **Regarding Concurrent Work on GRPO:** Your \"sum-then-normalize\" fix for GRPO's process supervision flaw is one solution. However, concurrent work from June 2025 (OpenReview `h83vIG5Hre`) 8 proposed an alternative, **\"MT-GRPO\"**. Could you please comment on why your solution is preferable to theirs (a weighted, multi-signal advantage)?\n4. **Regarding Novelty Framing:** You claim the \"inclusion of cooperative games is a key differentiator\". Given that you cite **Xu et al. (2023)** , which used the \"Werewolf\" game (a mixed cooperation/competition game), would you be willing to reframe your novelty? It seems to me the real, and more impressive, contribution is demonstrating *generalization from a simple, modular curriculum of game prototypes to complex, downstream MAS*.\n5. **Ablation Result in Table 5:** In Table 5, you ablate the \"agent-specific\" component. The performance drop for the Hanabi specialist is described as \"mild\" (55.55 -> 52.50). This seems counter-intuitive. Hanabi is an asymmetric, imperfect-information *cooperative* game where player roles and perspectives are fundamentally different and critical to success. Why is the effect so much smaller here than in the adversarial games? Does this suggest your component is less crucial for cooperative scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This is a solid piece of work, and I want to start by highlighting what I think is really excellent here.\n\n1. **Originality & Significance (The Core Idea):** The most significant contribution here, in my view, is the demonstration of *skill transfer from a simple game curriculum to complex, downstream multi-agent systems*. This is a really big deal. Prior work like SPAG or SPIRAL showed self-play can boost *single-agent* reasoning benchmarks, but this paper is the first I've seen to convincingly argue that these skills transfer to *cooperative and competitive software systems* like AutoGen and MAD. The 10.0% gain on AIME and 12.5% on GPQA-Diamond are non-trivial and suggest this is a very promising research direction.\n2. **Quality (Experimental Rigor):** The experimental setup is high-quality and, in one specific area, exceptionally well-designed.\n   - The evaluation is comprehensive: held-out games (Table 1) , single-agent benchmarks (Table 2) , and downstream MAS (Table 2).\n   - **Skill-System Mapping:** This is, frankly, the strongest part of the paper. The analysis showing that the agent trained on adversarial games (Tic-Tac-Toe) excels in the *competitive* MAD framework, while the agent trained on the *cooperative* game (Hanabi) excels in the *collaborative* AutoGen framework, is fantastic. This provides powerful evidence for the central thesis that MARS is teaching distinct, transferable skills, not just generically \"making the model smarter.\"\n   - **Ablations (at first glance):** The ablation studies in Table 4 (showing self-play beats a fixed opponent) and Table 5  (showing the method components are necessary) seem to validate the design choices. (I have major questions about these, see Weaknesses, but they are presented as being thorough).\n3. **Clarity:** The paper is very well-written and easy to follow. The central ideas are communicated effectively.\n   - **Figure 2** does an excellent job of visually explaining the flaw in the naive GRPO and how the proposed \"sum-then-normalize\" method is different.\n   - **Table 3** is a superb piece of qualitative evidence. Showing the \"Intent Recognition\" thought-trace from Hanabi (\"Maybe they want me to play this card?\") mapping to AutoGen (\"Maybe the assistant is not sure?\") makes the quantitative results in Table 2 so much more believable.\n4. **Significance (Broader Impact):** If these results hold, this work provides a scalable, data-efficient paradigm for \"social alignment.\" The idea of autonomously training agents in a \"social sandbox\" to develop complex, generalizable skills like \"theory of mind\" is highly significant for both the MARL and LLM-agent communities.", "weaknesses": "Despite the strong results, I have major problems with the paper's claims to *methodological* novelty. The authors have either missed or ignored extremely relevant, contemporaneous work, and the experimental design fails to decouple the most important variables.\n\n1. **Major Novelty Issue: \"Agent-Specific Normalization\" vs. SPIRAL's RAE.**\n   - **The Problem:** The authors claim \"agent-specific advantage normalization\" is a *novel technique* to solve the problem of \"heterogeneous roles\" in multi-agent training.\n   - **The Criticism:** The paper's *own primary baseline*, **SPIRAL (Liu et al., 2025)** , explicitly identifies and solves *the exact same problem*. SPIRAL calls its solution **\"Role-conditioned Advantage Estimation (RAE)\"**. Based on its description, RAE *also* computes separate advantages, $A_0(s, a)$ and $A_1(s, a)$, for each role\".\n   - **The Evidence:** From all available information, MARS's \"agent-specific normalization\" and SPIRAL's \"RAE\" appear to be functionally identical. Yet, the authors *never once mention RAE* in the entire paper, even when citing SPIRAL. The ablation study (\"w/o agent-specific\" in Table 5) only proves that *some* kind of role-based normalization is needed, it does *not* prove that the MARS version is new or superior to RAE. This is a major, major oversight and frankly misrepresents the methodological contribution.\n2. **Confounding Variables: The Method vs. The Curriculum.**\n   - **The Problem:** The authors present MARS as a complete package (Method A + Curriculum X) and show it beats SPIRAL (Method B + Curriculum Y).\n   - **The Criticism:** This is a classic confounding variable problem. The MARS \"package\" differs from SPIRAL in *two* key ways: (1) the advantage estimation algorithm and (2) the training curriculum (MARS adds cooperative games like Hanabi 1, while SPIRAL is purely zero-sum).\n   - **The Evidence:** The paper provides *no evidence* to decouple these factors. How do we know the impressive gains on AutoGen, AIME, and GPQA aren't *entirely* due to the superior *curriculum* (i.e., adding Hanabi was the only thing that mattered) and have *nothing* to do with the paper's proposed \"sum-then-normalize\" or \"agent-specific\" algorithms? The paper is just completely silent on this.\n   - **What's Missing:** The critical missing experiment is: **SPIRAL's algorithm (with RAE) trained on MARS's mixed curriculum.** If that setup achieves the same results as MARS, then this paper's methodological claims are largely invalidated. This is a huge hole in the experimental design.\n3. **Concurrent Novelty: \"Turn-Level\" Credit Assignment.**\n   - **The Problem:** The authors correctly identify a \"critical flaw\" in the original GRPO \"Process Supervision\" setting, which uses a \"normalize-then-sum\" approach, and propose a \"sum-then-normalize\" fix.\n   - **The Criticism:** They weren't the only ones to spot this. A paper on OpenReview (**`h83vIG5Hre`**, pub. **June 8, 2025**) *also* identified that trajectory-level estimation is \"inadequate\" and proposed a \"fine-grained turn-level advantage estimation\" strategy.\n   - **The Evidence:** This concurrent paper proposed **\"MT-GRPO\"**, which uses a *different* fix (a weighted sum of independently normalized turn-level and outcome-level advantages, e.g., $\\hat{A}_{i,1} = \\hat{A}^{T}_{i} + \\lambda\\hat{A}^{O}_{i}$). This means the \"best way to fix GRPO\" was an open question at the time of submission. MARS presents its solution as *the* solution, without acknowledging or comparing it to other concurrent proposals. This again weakens the claim of methodological novelty.\n4. **Misleading Framing: The \"Novelty\" of Mixed-Motive Games.**\n   - **The Problem:** The paper repeatedly frames its contribution as \"The inclusion of cooperative games is a key differentiator\" and contrasts this with the \"narrow, adversarial-only mindset of prior work\".\n   - **The Criticism:** This is an oversimplification and a bit of a strawman. The *concept* of using mixed-motive games is not new.\n   - **The Evidence:** The authors *themselves* cite **Xu et al. (2023)** , which used the \"Werewolf\" game. The \"Werewolf\" paper *explicitly* states the game \"involves both cooperation and competition\".The *real* novelty of MARS isn't the *idea* of using a non-adversarial game; it's the demonstration that a *simple, modular curriculum* of prototype games (pure-coop, pure-comp) can *generalize to downstream MAS*. This is a much stronger and more accurate claim, and I'd urge the authors to reframe their contribution this way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBE0ooexDt", "reviewer": "Reviewer_YdbT", "summary": "This paper introduces MARS (Multi-agent RL method for LLMs), a new framework for training Large Language Models in multi-agent environments. The method includes: \n1) the introduction of turn-level rewards to enable more fine-grained credit assignment, \n2) an agent role-based advantage normalization technique, which aims to reduce training variance by calibrating advantage signals relative to the performance of rollouts from the same role,\n3) self-play against a variety of partners opponents in competitive games.\n\nThe authors conduct experiments using Qwen3-4B as the backbone model. They train specialist agents on individual games representing diverse dynamics: Tic-Tac-Toe (competitive, deterministic), Kuhn Poker (competitive, uncertain), and Mini Hanabi (cooperative, uncertain). They also train a generalist agent (MARS-multi) on all games simultaneously. Evaluation is performed on held-out games with similar dynamics (Connect Four, Leduc Hold'em, Simple Hanabi). The results indicate that MARS specialist agents outperform baselines (SPIRAL, Qwen3-4B) on their respective held-out tasks. The paper also evaluates the agents on general STEM reasoning benchmarks, finding mixed results in a single-agent setting but small, consistent improvements when integrated into a multi-agent framework (AutoGen). The ablation suggests playing with mixed opponents is critical.", "questions": "- Ablation results in Table 5 shows that without either turn-level reward or agent-specific advantage normalization resulted in significant performance drop in Tic/Kuhn trained models, but not necessarily true for Hanabi-trained model in Kuhn Poker and Leduc Hold'em settings, do you have any intuition on why that is the case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The algorithmic contributions are very clear and well motivated.  Turn-level reward shaping for credit assignment and role-conditioned advantage normalization are simple, intuitive modifications to GRPO that target known pain points (variance, difficulty credit assignment).\n- The experimental design is solid, as the choice of training and testing environments is commendable. The authors systematically cover competitive, cooperative, deterministic, and uncertain dynamics. Evaluating on held-out games (e.g., training on Tic-Tac-Toe and testing on Connect Four) provides a solid test of generalization to similar dynamics, which is a key claim of the paper.\n- The results are positive when tested in case of generalization. We see that MARS agents outperform baselines on held-out games (Connect Four, Leduc Hold'em) is a significant result. This suggests the method is not just overfitting to the training games but is learning underlying strategies for that specific dynamics.", "weaknesses": "- All experiments use small, two-player environments with constrained action/communication spaces. This limits external validity; N-player settings introduce additional non-stationarity, credit assignment, and population-diversity challenges, so evidence scaling beyond 2-player cases is important to substantiate generality claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)