Title: MARS: Reinforcing Multi-Agent Reasoning of LLMs through Self-Play in Strategic Games
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Self-play, Multi-Agent System, Strategic Games

Abstract: Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARS, an end-to-end RL framework that incentivizes **M**ulti-**A**gent **R**easoning of LLMs through **S**elf-play in both cooperative and competitive games. MARS features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, the MARS agent trained from Qwen3-4B develops strong strategic abilities that generalize to held-out games with up to $28.7$\% performance improvements. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of multi-agent systems in reasoning benchmarks. When integrated into leading multi-agent systems, our MARS agent achieves significant performance gains of $10.0$\% on AIME and $12.5$\% on GPQA-Diamond. These results establish end-to-end RL training with self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs. Our code and models are publicly available at <https://anonymous.4open.science/r/MARS-LLM>.

TL;DR: We propose a self-play framework for enhancing general multi-agent capabilities of LLMs via reinforcement learning on strategic games.