Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
Authors: Yujie Zhao
Lanxiang Hu
Yang Wang
Minmin Hou
Hao Zhang
Ke Ding
Jishen Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
multi agent system
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Multi-Agent System (MAS) and Reinforcement Learning (RL) are both widely adopted to improve large language model (LLM) agentic performance. MAS strengthens task-specialized performance via role-based orchestration; RL leverages environment rewards to train stronger policies, such as Group Relative Policy Optimization (GRPO)-style optimization. Yet applying on-policy RL training to MAS is underexplored. While promising, it poses several challenges. On the algorithm side, Standard GRPO grouping assumptions fail in MAS because prompts differ by role and turn. On the system side, the training system needs to support MAS-workflow-based rollouts and on-policy updates for both single and multiple policy models. To address these issues, we introduce AT-GRPO, consisting of (i) an Agent- and Turn-wise grouped RL algorithm tailored for MAS and (ii) a system to support both single-policy and multi-policy training. Across game, plan, coding, and math tasks, AT-GRPO demonstrates substantial performance gains across diverse domains. Especially on long-horizon planning tasks, AT-GRPO boosts accuracy from a 14.0–47.0% single-agent RL baseline to 96.0–99.5%. Furthermore, it improves reasoning performance, with an average gain of 3.87–7.62% on coding and 9.0-17.93% on math. The code are available at https://github.com/pettingllms-ai/PettingLLMs.

Published as a conference paper at ICLR 2026
MARSNGER-MAS: MULTI-AGENT REINFORCEMENT LEARNING FOR COLLABORATIVE LLMs
Yujie Zhao'
Lanxiang Hu!
Yang Wang²
Minmin Hou?
Hao Zhang'
Ke Ding
Jishen Zhao'
Uniiersity of California, San Diego
Intel Corporation
ABSTRACT
Multi-Agent System (MAS) and Reinforcement Learning (RL) are both widely adopted to improve large language model (LLM) agentic performance. MAS strengthens task-specialized performance via role-based orchestration; RL lever- ages environment rewards to train stronger policies, such as Group Relative Policy Optimization (GRPO)-styy) optimization. Yet applying on-policy RL training to MAS is underexplored. While promising, it poses several challenges. On the al- gorithm side, Standard GRPO grouping assumptions fail in MAS because prompts differ by role and turn. On the system side, the training system needs to support wASS-workflo-bbas
rollouts and on-policy updates for both single and multiple policy models. To address these issues, we introduce A7-GRPO, consisting of (i) Agent- and /'urn-wiss grouped RL algorithm tailored for MAS and (ii) system to support both single-policy and multi-policy training. Across game, plan, cod- ing, and math tasks, AT-GRPO demonstrates substantial performance gains across diverse domains. Especially on long-horizon planning tasks, AT-GRPO boosts ac- curacy from a 14.0-4 1..00-4..00 .00 single-agent RL baseline 96.0-99.55 Furthermore. it improves reasoning performance, with an average gain of 3.87 on coding and 9.0-17.93% on math.
Q Code & Environment:
http://tt//eeetllmssssees ai/ Pes
INTRODUCTION
Large Language Model (LLM) agents are task-specific workflows Yao et al. 2023 Xi et al. 2023
Wang et al., 2023b) that utilize LLMs key components for decision making (Shinn et al. 2023),
action taking Wang et al. 2023a). and tool use (Qian et al. 2025 Schick et al. 2023) LLM agents
have demonstrated strong promises across various application domains, such embodied control
(Ahn et al. 2022: Wang et al. 2023a), software engineering (Tao et al. [0024] Yu et al. (2025). expert
drug discovery (Liu al.
2024 Inoue al., 2024), and scientific ideation and hypothesis testing
Ghafarollahi and Buchler
2024).
Today, two complementary approaches are widely used improve the performance of LLM agents: multi-agent systems (MAS) and reinforcement learning (RL). RL treats the LLM as a policy and iteratively updates its weights strengthen decision-making: interacts with the environment, collects rule-based rewards, and then computes a policy optimization loss to update the parameters (Shao al., 2024). In practice, this workflow requires training stack that supports both scalable rollouts and online updates, e.g., VERI. (Sheng et al. 20255 and AReal. (Fu et al.) (025)). MAS typically employs prompt-only augmentation on a shared L.LM policy for bole-based coordination; practical deployments instantiate diverse workflows. Recent studies (Belcak et al.) 2025; Chen et al. 2024; Wang et al. (2024) further highlight the potential benefits of role-specialized MAS, which adopts distinct models for different roles, enabling role- specialized policies inference. However, the effectiveness of RL training on role-specialized MAS is underexplored.
each iteration, the current model
natural next step is to combine the two: using RL to train MAS, such that we gain both stronger learned policies, role-specialized collaboration. However, bringing RL. into MAS raises two coupled

---

Published as a conference paper at ICLR 2026
MAS+RL (AT-GRPO)
100
55% (+25.0%)
Accuracy(%
20
MAS Augmentation
Accuracy(P9
75
(+560.0%)
50-
MAS (AT-GOOA
15
Single-agent RL (GRPO)
25
Augmentation
Single-agent RL (GRPO)
50
100
150
20
60
80
100
Steps
Steps
(a) Qwen3-1.7B on BiveCodBBench
(b) Qwen3-8B on Sokoban
Figure 1: MAS+AT-GRPO vs. Single-agent-GRPO. The gray line denotes the prompt-only MAS
baseline.
challenges. First, training a MAS may require concurrently launching multiple models, orchestrat- ing inter-agent environment interactions, and performing independent on-policy parameter updates. But most existing on-policy RL frameworks for LLM agents only support a single model (Volcano)
Engine| 2025 Sheng et al. 2024 Fu et al. 2025). Second, rollouts from MAS are difficult to group. The advantage must be conditioned on interaction history and role to ensure fair credit assignment.
Group-based RL objectives designed for a single agent (Volcano Engine
2025
Qian et al. 2025
Feng ct al. 2025) are not directly applicable to MAS.
To address these challenges, we first design AT-GRPO, an Agent- and Turn-wise grouped RL method that adapts group- rroup-relative optimization for MAS. Furthermore, we develop a novel training system to support on-policy RL for MAS. Our training system supports rollouts for diverse MAS workflows and enables on-policy RL training for both role-sharing policy and role-specific policies. We conduct extensive experiments on Qwen3 models across a range of representative agentic domains, including game, plamning, coding, and mathematical reasoning. As highlighted in Fig. D AT-GRPO (blue) significantly outperforms single-agent GRPO (red). For instance, it achieves a 5.0% higher accu- racy (+25.0% relative) on LiveCodeBench (with Qwen3-1.7B), while the improvement increases to 84.0% on Sokoban (with Qwen3-8B).
This paper makes the following key contributions:
AT-GRPO Algorithm. We introduce an agent- and turn-turii wise grouped RL algorithm, AT-GRPO, and identify the substantial benefits of applying on-policy RL to MAS across diverse domains: planning, gaming, coding and mathematical reasoning tasks.
MAS Training System. We design novel training system to support (i) executing rollouts for
diverse MAS workflows and (ii) performing on-policy RL updates for multiple policies.
Our method delivers consistent gains across diverse domains. On long-horizon planning tasks, it overcomes key bottleneck of single-agent RL, boosting accuracy from 14 14-47% baseline to 96.0-99.5% Furthermore, also demonstrates gains on code and math benchmarks, with average improvements of 187-7.62% and 9.0-17.93% respectively.
Our analysis shows that (1) RL training on MAS reinforces role-specific specialization; (2) with MAS AT-GRPO, whether to choose a role-sharing policy or role-specialized policies needs to be determined by the task characteristics.
2 RELATED WORK
Role-sharing vs. Role-specialized Policies in MAS. A predominant approach in LLM-based MAS centers on role-sharing architecture, where a single policy is shared across all agents. In these frameworks, such as AutoGen (Wu et al. (023) and MetaGPT (Hong et al. 2024]. role-specific behavior is elicited at inference time via prompt augmentation. More recently, research has begun to explore role-specialized policies. This shift is motivated by the observation that single L.M''s performance exhibits significant variance across domains (Chen et al., 2024 Wang al., 2024 Belcak et al. 2025). Consequently, assigning distinct and more suitable models to specialized roles, as demonstrated by Ye et al. (2025): Belcak et al. (2025). has emerged as a promising direction for enhancing performance. Despite this architectural evolution, recent surveys (Cemri al. 2025 Guo et al. 2024) indicate that most studies focus on inference-time design, leaving the potential of training MAS policies with RL largely underexplored.
RL Training for MAS. RL has become a key technique for LLMs agent training, using group- relative and nule-based rewards to enhance reasoning, long-horizon planning, game, and tool use
2

---

Published as a conference paper at ICLR 2026
Tritiol Problem
Turn
Turn
Dusnteal fower Theary (
Flreal "yiael "allatmme"))
Phar seteen "'cletmme"
if 1.11 satees bala
Tras fraeeter wo
Game/Plan
Code
Math
sparre
Conatraction
Cock Syrthas
Tid cill
Rocseeay
Tarchie dal Rusard
Alyo
Net Algee?
Recrenert
Coda Rafineest
Teel call Refremert Ansuaing Releameny Net wla??
Arrie
NOT 4lgo?
(V. M.th Agent Workflod for Somr. Por, Code and Math Tocke
Figure 2: MAS workflow across different domains. (a) Role-based coordination: code generation via a coder-tester loop. (b) Different task-specific workflows for Game/Plan, Code, and Math; see Sec. 5. and Appendix C.2 for workflow details.
(Feng et al. 2025 Wang et al., 2025b; Qian et al. 2025 Hu et al., 2025). These approaches,
however, predominantly operate within single-agent framework. While growing body of work attempts to extend RL to Multi-Agent Systems (MAS), most efforts remain confined to limited in- teraction settings fixed role structures. For instance, CURE (Wang al., 2025a) focuses co-evolving Coder and Unit- Tester using a role-sharing policy specifically for code generation. Similarly. SPIRAL (Liu al. (2025) employs self-play in zero-sum games using single LLM, while MHGPO (Chen et al. (025a) targets retrieval auggeeted generation. MAPoRL (Park et al.) 2025a b) and CoRY et al. (2024) train LLMS within fixed, homogeneous-role debate workflows.
More recent works also exhibit limitations: MARFT Liao et al. (2025) restricts agents to single- turn sequential interactions, and MARTI /hhnn et al. (2025) merely introduces basic single-agent RL algorithms (e.g., GRPO) to the MAS setting. Critically, the implemention for these works is concentrated on the single domain of math. We include more comprehensive comparison of these related works in Appendix A In contrast, our study offers more comprehensive solution. We pro- pose a general algorithm designed for multi-turn, multi-agent environments. Unlike prior works, we conduct thorough analysis and evaluation of both shared and role-specific policies across diverse MAS workflows and varying domains.
3 PRELIMINARIES
MAS Setting. The N-agent LLM system is modeled
Markov game M
(S, (ri),,,, T,, H). where S is the state space; A, is the action space of agent The transition function T induces intra-turn micro-transitions where 8t,0 = 8; and 8t, T(st,--1. (r..;),, i), culminating in $t+1 stN. The reward for agent i is given by r: A; + [0, 1], and the turn horizon T. the optimization step horizon H. At each turn t, agent receives an obser- vation summarizing the environment state and interaction history h4, Ot,i
(8t, he). Each agent i implemented with role-specific prompt template P.(.). Let θ {0(m) JMM denote the set of LLM parameter vectors, with 1 < M < N, and let {l..... .....,, N (I...., to LLM. We treat one LLM rollout (a token sequence) one full interaction in which all agents emit macro-actions to the environment. A step denotes one optimization update to the parameter set θ during training.
......,,a M) assign each agent single macro-action a,,i A twrn is
MAS Workflow. Following prior work (Wang al. 2025a Ahn al. 2022 Chen et al. 2025c
we employ domain-specifie MAS workflows, as shown in Fig 2 Our ex experiments confirm that this prompt-only method outperforms a single-agent bascline (see Tab. and in Sec. (.2).
We compare against MARFT v3, the latest preprint available prior the completion of this work.
3

---

Published as a conference paper at ICLR 2026
Group-based RL. Methods for LLM agentic training with group-relative advantages (Feng al. 2025 Wang et al. 2025b; Qian et al. 2025) operate by first sampling candidate actions {a ()) for a given prompt. Each action is evaluated to obtain rule-based reward R(a, forming { (a R(a,)))
Jeml
(a("), R(a, (((a{K)) }. For each action a in this group, the relative advantage is then defined as its mean-centered and normalized return.
comparison group: G =
R(at) (o) mean (R(a)
A
F
(Ba(a (3) |
Role-sharing vs. Role-specialized Policy Optimization. We distinguish between two optimiza- tion regimes, role-sharing and role-specialized, both of which initialize policies from the same base model. During rollouts, each agent generates a dataset D, which consists sample groups. A single group g is composed of shared observation context Og and K candidate actions with their corresponding advantages, denoted as g = {i, ag ( ]
Je=1- The core difference between the two regimes lies in how the training data is batched. A minibatch B.. = (A) D, for specific policy g(m) is constructed by pooling the datasets from all agents assigned to it:
[(O(m)) = - EEg
((am) (alm)
Alo) (m((r))))((mm)) 1 1 1,,,,111 1) ((0)
min
K
Role-sharing policy (M=1): All agents share a single policy 0%. The training batch is the union of data from all agents, B1 U Di, and is used for single joint update: 0¹ - 0 - V L.
where r(0) =o(0:1g)
Too (4:]g)
Role-specialized policies (M = N): Each agent has distinct policy g(i), such that (i) = i. Each policy is updated independently on B, D; and update policy: g(t) <- g(3) - occ ((B()))
4
METHOD
4.1 ALGORITIM DESIGN: AT-GRPO
GRPO's advantage calculation [I hinges on fair compari- son among all candidates within group. This faimess is en- forced by the reward mecha- nism itself. As illustrated in Fig. 2 (top), token-level scor- ing assigns credit the gen- erated response tokens (Reward Mask=l), while the prompt to- kens receive no credit (Reward Mask=0). Since the advantage is determined solely by the qual- ity of the response, a valid and fair comparison is only possible when all responses a group originate from an identical prompt. Consequently, single-agent LLM- RL methods Wang al. 2025b Qian et al. 2025 Feng et al. 2025) typically form groups by sampling multiple responses to the same question.
mnstitat untic Pythen sole (emedd, provest mode pramnter lene howver Take Turt (ood guancttd by oatterr add wring pluse rifire
Acfeccur
thoi be nehreed
Cule
Coffement
Cale
Derterment
Cook
Acteement
achnosent
Oole
bae
oottermnt
Shen
Agen
Grcup
Tree Scrpting
Figure 3: Two sampling schemes. (a) In parallel sampling,
trajectories are sampled but incomparable, leading to groups of size 1. (b) In tree sampling, branching each turn forms valid comparison group of size K.
In MAS, however, a "prompt" is not only question description, but also embeds the role-specific context and cross-agent interaction history. For example, in the code tasks depicted in Fig. 2 (a) where the workflow entails coder-tester loop: one agent synthesizing code and the other creating
4

---

Published as a conference paper at ICLR 2026
Algorithm AT-GRPO: Agent- and Turn-wise MAS RL Training
Require:
Markov game M policies è {((m)) Jm Imm=1 role mapping 0, sampling temperature
T Tamp, branches K, total steps S. batch size E, turn horizon T, termination condition Zeerm
1 Termination helper: returns true if horizon reached or env signals done -
1: for training step 8 ........ do
1 Phase 1: On-Policy Rollout & Data Collection -*/
2:
Initialize per-agent datasets {Dr) < D. Resample E environments.
3:
for each environment instance {1, (....., E} in parallel do
4:
for & = to T" do
5:
$t,0,e f- St,e for each agent E [1,...,NN do
Initialize micro-step state
6:
7:
Vc [I....,KK di Tplo(() (- Or,i,i Tamp); compute Ti.i.e (Eq. 3
(e)
(c)
Define group key g hhahh(e,,,,, i, and compute advantages (Ag) (Eq. 1K
8:
Append (g, oniee (c)
(u {Ag ( Je=1 to D.. (c')
9:
10:
e" + arg max. "iie 0r,i, + Gie (Treestrruturre sampling.)
St,i, t T (1 (
Agent-wise micro-transition
12:
end for
S1+1ic f- S,
End-of-turn state
14:
if TTrr (st+le) then break
endi if end for end for
16:
17:
1 Phase 2: Per-Model Policy Update -
18:
for each model m E {l...., , M) } in parallel do
19:
Construct per-model batch Bm, loss L(((m)) on B, using Eg
and update policy m.
20: end for
21:
end for
unit tests, and they iteratively refine the output until alignment (Fig. 3 top). the turn-2 refinement prompt already contains the turn-1 code, unit tests, and role-specific prompt format, so prompts differ across turns and roles. We therefore adopt agent-wise and turn-wise grouping extension of tabular-wise group-normalized advantages in GiGPO (Feng et al.) (2025) the multi- agent setting: candidates share the same role and turn position, ensuring prompt identity for valid GRPO advantage comparisons.
natural
However, agent- and turn-wise grouping introduces new question. If we follow the common parallel sampling used by prior agentic BLL-sample K full trajectories from the initial state/problem (Fig. 3 (a), bottom), each group size when > 1: other sample shares the identical prompt. GRPO therefore eliminates its variance-reduccion effect and yields unstable updates. To address these challenges, develop AT-GRPO (see Alg. I with three key ideas: tree-structured sampling, agent-and turn-wise grouping, and agent-wise credit assignment.
Tree-structured Sampling. At each turn f,, for each agent i, we sample K candidate actions and their corresponding rewards from the current state (Alg. line 7). The advantages for these K can- didates are then calculated within this group (line 9). Subsequently, the full data cople-containing the group key, observation, K actions, and their K advantages is added to dataset D; specific to the policy of the acting agent (line 10). To proceed with the environment rollout, we greedily select the candidate with the highest reward to be the executed action (line 11). This greedy selection strategy concentrates exploration on coordination-critical decisions and helps maintain a balanced mix of positive and negative samples, which stabilizes the learning optimization.
Agent- and Turn-wise Grouping. We group experiences based on the acting agent and the turn number within each parallel environment instance. Operationally, we implement this by defining a unique group key g for each agent i each turn t in each environment using lightweight hash function (Alg. [ line 8). All data generated from the K-branch sampling at that step, including the observation and the calculated advantages, is stored together under this group key (line 10). During
5

---

Published as a conference paper at ICLR 2026
the policy update phase, these collected data groups are used to construct per- per-model training batches for the final optimization step (lines 20-21).
Agent-wise Credit Assignment. Inspired by mixed-reward designs in cooperative Multi-Agent RL (Mao et al., 2020 Sheiik and Bölöni 2020), we assign credit using a mixture of global and local rewards. At each turn i, the environment provides a global team reward pleam and an agent- specific local reward r-oe that evaluates its subtask performance. These components are combined using a hyperparameter form the final reward for agent é:
Tτ, arream team + 22,
(3)
This formulation balances a shared team objective with role-specific incentives. For instance, in coder-tester MAS, the team reward pleam is the pass rate the generated program on a set of golden unit tests. The local rewards ! are tailored to each role: the coder is rewarded for its own code's pass rate, while the tester rewarded based on the pass rate of a golden reference implementation against its generated tests. Detailed reward designs for all tasks provided in Appendix C.I
4.2 MAS TRAINING SYSTEM
LLM Resource Pool 1 (GPU Group i)
Compute
Less
ROLLOUT
UPDATE WORKER
Mainstream RL post-training frameworks for
WORKER
Serve
(Medel
Porematers
LLMs, e.g.. TRL (von Werra al. 22200). VERL (Sheng et al., (2024), AReaL (Fu et al.
LLM Resource Pool (GPU Group j)
2025). and OpenRLHF (Hu et al.] 2024) pri
ROLLOUT WORKER
LLM Medel J!
UPDATE WORKER
marily support single-agent RL training, which
typically involves: single agent-environment interaction pattern, policy operating, and single LLM resource pool. This makes dif- ficult to (i) train multiple models in on-policy RL, (ii) maintain clean on-policy training data, and (iii) support diverse MAS workflow.
Ein Researce Fool ECPUS
conrrl
MAS
Wertont
Agent
Agent
Agent
Trajectory Data
Model Betches
ROUTING
We introduce a novel MAS training system to overcome these challenges and enable AT- GRPO By allocating an independent resource pool to each model, our system is designed to support the concurrent on-policy training of multiple policies. The system, depicted in Fig. consists of the following components:
Model Batcher
Figure 4: MAS training system. Each LLM m has a GPU-pinned model pool with a Rollout- Worker and UpdateWorker. A CPU environ- ment pool hosts envworkers that execute environ- ment steps. Trajectories routed to the corre- sponding UpdateWorte Worker.
LLM Resource Pools (GPU). Each policy is managed within an independent resource pool. Fol- lowing HybridFlow-styke (Sheng et al., 2025). each pool comprises two workers: a RolloutWorker for inference and an UpdateWorte Worker for optimization. During the rollout phase, all policies interact collectively according to the Alg. [[ MAS workflow; Once collected, each trajectory is routed to the corresponding UprateWorker, Worker, maintaining an on-policy learning regime for every policy.
Environment steps run in a fleet of CPU En- vWorkers, each managing a single sandboxed instance to ensure safety and reproducibility (seed- ing, wall-clock timeouts, I0 quotas, and deterministic tool hamessss). This one-actor-per-ie per-instance mapping efficiently supports thousands of concurrent rollouts in parallel. Env Workers stream obser- vations, tool logs, and rule-based rewards Router. The Router dispatches collected experience based on policy assignment: experiences generated by an agent i are sent to the Updateworker of its designated policy (1).
Environment Execution (CPU) and Data Flow.
EXPERIMENTS
5.1 DATASETS AND MODELS.
1. Experimental Setup. We train and evaluate Qwen3 models at 1.7B and 8B in the no-thinking mode YYang and the Qwen Team 2025). All runs single node with 8x H100 GPUs. The rollout

---

Published as a conference paper at ICLR 2026
sample size is K=4 and the turn horizon is T = 4 for both multi-agent (MA) and single-agent (SA) settings. The reward- eowardmmxxig coefficient is a=1 without further tuning. Full training details appear in Appendix C.I
2. Baselines. We evaluate five variants (all initialized from the same base model): (a) Single Agent (prompt-only): one frozen L.MM solves the task end-to-end; (b) Single Agent GRPO: as (a) but trained with GRPO (Shao et al. 2024); (c) MAS (prompt-only): role-specialized prompting over frozen, role-sharing backbone; (d) MAS + RL (role-sharing policy); all roles share one policy and pooled trajectories update it jointly; (e) MAS RL (role-specialized policies); samples are routed by role and each policy is optimized independently (no parameter sharing).
3. Task Setups and Baselines. To ensure a fair comparison. we align all environmental obser- vations and reward signals across both MA and SA settings. While both paradigms utilize the same role-specific reward functions, the sole distinetion that the MA framework involves multiple agents capable of discussion. Detailed prompt templates and reward specifications are provided in Appendix B und[..
Single Agent Variants. We also evaluated a multi-turn SA variant for both Code and Math (see Appendix F). where a single agent repeatedly revises its own output until self-consistency. This setup is inherently less natural: the agent receives additional environmental signal cross-agent feedback, and the interaction pattern deviates from the QA-style pretraining regime of LLMs. Re- flecting this mismatch, the multi-turn SA variant brought no empirical improvement and sometimes slightly degraded performance relative to the standard single-turn SA baseline.
Code. The environment observation is restricted to the problem description. The MA setting em- ploys a dual-role debating mechanism: a Tester generates unit tests and Coder generates the code. They refine their outputs iteratively until alignment is reached the maximum turn limit is met. The natural SA baseline employs Coder to generate the solution directly, as no other environmen- tal feedback and other agent's 's output is available.
The environment observation consists is restricted to the problem. MA uses Dual-role de- bating MAS: Tool-User (utilizing code interpreters) and a Reasoner (performing direct reasoning) until alignment or the maximum turn limit is met. The SA setting uses a Reasoner to derive the answer directly, as no other environmental feedback and other agents' response.
Math.
Planning and Game. The environment observations are game states of the current turn. The MA setting employs a collaboration mechanism: a Tool-User (executing the tools) and an Executor (verifying tool outputs and executing actions). The SA employs an Executor to perform actions. Both settings share identical termination conditions based on goal satisfaction or the turn budget.
4. Training and Evaluation Datasets. Sudoku and Sokoban. We evaluate our method on gam- ing tasks: a 4x4 Sudoku and a 6x 6 Sokoban. We use instances with an automatic checker, following the symbolic task setup of SYMBENCC (Chen et al. 2025c). To ensure fair evaluation, we generate distinct training and validation sets using different random seeds and verify there is no overlap.
Plan-Path. We use a 10x 10 grid-based Plan-Path environment. This follows the checker-backed symbolic task setup in CodeSteer's SymBench (Chen et al. 2025c). To separate training and valida- tion, we generate the two splits with distinct random seeds and verify no duplication.
Code Generation. For training, we adopt size-specific corpora: the 1.7B Qwen model is trained on the APPS training split cooouucctoy-difficuut subset) (Hendrycks et al.) (02T). while the 8B model is trained on CodeContests (DeepMind) 2024). For model-generated code, we use the dataset's s golden unit tests to score correctness; for model-generated UT, we use the dataset's golden ref- erence solutions compute the reward. For evaluation, we three widely adopted coding benchmarks spanning interview-style and contest-style settings: APPS (Hendrycks et al. 2021) LiveCodeBench-v6 (White et al. 2024), and CodeContests (DeepMind 2024).
Mathematical Reasoning. We train on the Dalaris-Dataset5333 (An et al.) 2025) and evaluate on several standard mathematical reasoning benchmarks. For validation, (Mathematical Association of America & AoPS Community, [022]] (2025) and OLYMPIAD (He et al.) 2024). All math tasks use verifier-checked numeric scoring.
A1ME24/AIME25
7

---

Published as a conference paper at ICLR 2026
Table 1: Qwen3 L.7B results game, planning. coding, and math.
Game
Plan
Code
Math
Method
Sudoku
Sokoban
Plan-Path
LiveCudeBench
APTS
CodeContests
AIME24
AIME25
OlympiadBench
7.00
0.00
5.00
11.60
16.20
3,60
13.40
9.80
22.20
Single agene
(40.00)
(40.00)
(+0.00)
(40,00)
(40.00)
(40.00)
(4000)
(40
(40.00)
29.00
3.00
11.00
18,90
17.00
3,00
10.00
6,70
23,80
Single agent GRPO
(+22.00)
(+3.00)
(+6.00)
(+7.20)
(+0.80)
(-0.60)
(3,40)
(-3110
(+1.60)
MAS
(+62.00)
69.00
(+0.00)
0.00
(+5,00)
10.00
(+7.40)
19.00
(+0:40)
16,50
(+0.00)
3.60
(+-0.10)
13.30
(+
13,00
(+13.70)
35.90
MAS GRPO
87.00
1,00
82,00
20,60
17:60
4.80
13.30
16:70
35,00
(+90,00)
(+1.00)
(+77.00)
(+9,00)
(+1,40)
(+1,20)
(+00110)
(+6.90)
(+12.80)
w/ shared policy
MAS+ + AT-GRPO
(+92.00)
99.00
(+10.00)
10:00
(-91.00)
96.00
(49.30)
20.90
(+1.40)
(+1,20)
4,80
(43.30)
(46.90)
(+16.8)
39,60
99.00
11.50
97.00
24,00
18.60
7,80
13:30
18.30
35,20
MAS + AT-GRRO
(+92.00)
(+11.50)
(+92.00)
(+12.40)
(+2.40)
(+4.20)
(+00.10)
(+8.50)
(+13.00)
w/ par-role policies
Table 2: Qwen3 SB results on game, planning, coding, and math.
Game Sokohan
Plan Plan-Path
Code APPS
Method
Sudrok
LiveCodellench
CodeContests
AIME24
AIME25
OlympiadRench
Single agent
(+0.00)
48.00
(+0.00)
9:00
(+0.00)
12.00
(+0.00)
22,80
(+0.00)
30.20
(+0.00)
15.75
(+0.00)
18.30
(+0.00)
20.00
(+0.00)
55.00
Single agent GRPO
54,00
14.00
47,00
25.70
37.00
12.12
18:30
26.67
54,80
(+6.00)
(+5.00)
(+35.00)
(+2,90)
(+6.80)
(-3663
(+0.00)
(+6.67)
(-0.20)
72.00
16.00
71.00
28.00
44.40
17,50
36,60
30.00
56.50
MAS
(+24.00)
(+7.00)
(+59,00)
(+5.20)
(+14,20)
(+1.85)
(+18.30)
(+10.00)
(+1.50)
MAS + GRIO
(+51,00)
99.00
(+21
30.00
(-84.00)
96.00
(+1,40)
24.20
(-10.00)
40.20
(:5.45)
10.30
(+115.00)
33.30
(46.67
26.67
(-1.80)
53,20
99.50
96.00
93:00
30.28
45.80
18.10
50.00
35,20
$6.80
MAS + AT-GRPO
shared policy
(+51,500
(+87 00)
(-81 (00)
(47,48)
(-15.60)
(+2.35)
(+31.70)
(+15.00)
(+1.80)
MAS+ + AT-GRPO
(151..00
99.00
(+89.00)
98.00
(+84.00)
96.00
(+10.30)
33.10
(+16.30)
46.50
(+2.35)
18.10
(+38.700)
$7.00
(+20.00)
40.00
(.1.60)
56.60
per-mie policies
Parentheses denote gain over the Single Agent baseline; best second-best results per column are highlighted.
5.2 RESULTS AND ANALYSIS
We evaluate AT-GRPO across four distinct domains (game, planning, code, and math) using two model scales (Qwen3 1.7B and 8B). To contextualize its performance, we benchmark against all the variants described in Sec. 5. Tab. 1 and Tab. summarize our main results.
MAS+ + AT-GRPO consistently yields substantial performance gains, especially in long-horizon planning tasks. MAS + AT-GRPO elevates the success rate from a 14-47% range for the single- agent baseline to 06.0-99.5%% By analyzing the dialogue records between agents, we find this dramatic improvement stems from an emergent collaboration: the tool agent learns to generate cor- rect algorithms (e.g.. BFS, A" search), while the plan agent provides crucial oversight, interpreting execution outcomes and delivering the corrective final action list. On-policy RL training within the MAS enhances inter-agent coordination. Conversely, training agents in isolation results in only lim- ited improvement, detailed in our ablation study (Sec. 5.4 Tab. 4 Furthermore, on the coding and math benchmarks, our approach yields consistent gains, with absolute gains over the baseline ranging from +2.35 (CodeContests) +16.30 (APPS) in coding, and from +1.80 (OlympaidBench) to +38.70 (AIME24) in math. We hypothesize two reasons: (1) Base models like Qwen3 have al- ready been extensively trained for these common domains, as noted in their official reports (Yang) and the Qwen Team 2025). potentially leading to performance saturation. (2) The diverse nature of problems within these domains presents greater challenge for improvement via RL training.
With MAS AT-GRPO, whether choosing a role-sharing policy or role-specialized policies should be determined by the task characteristics. Role-specialized policies involve a funda- mental trade-off: training each agent exclusively on its own data fosters deep specialization, but prevents access to potentially useful data from other roles. Our findings indicate that the opti- mal resolution to this trade-off depends on the task characteristics. We observe clear benefits for role specialization in the coding domain, where the Coder and Tester functions are highly distinct. This separation allows each agent to hone its specific skills, improving the average accuracy by 3.05 points with the Qwen3 1.7B.In contrast, the roles in the math domain exhibit greater functional over- lap, meaning shared policy sometimes be superior. For instance, with the Qwen3 1.7B model on OlympiadBench, the shared policy achieves a 39.60% accuracy, surpassing the 35.20% from per-role policies. This suggests the Tool agent, which must often perform reasoning to execute tool
8

---

Published as a conference paper at ICLR 2026
calls, benefits from the Reasoner's training data. For game/plan tasks, this choice becomes moot, as both configurations already achieve near-optimal, saturated performance (e.g., 99.50 on Sudoku).
Limitations of MAS-GRPO. Empirical re- sults in Tab. M indicate that directly apply-
Tool-User User
MAS (bbtrandd
MASHORPO
ing GRPO to MAS often results in performance degradation. Notably, Qwen3-8B exhibits sub- optimal results CodeContests (17.60 - 10.30) and OlympiadBench (56.50 - 53.20). We at- tribute this to the violation of the identical- state assumption: as multi-turn interaction his- tories diverge, the group-averaged baseline in- correctly aggregates heterogeneous states, This structural misalignment biases advantage esti- mation and destabilizes optimization.
Ressomer
MAS AASIAFO
Judge
Graph
Agents
M-N+l
21
Tool
Youl
Toel
Scalability Analysis with Collaborative Agents. To investigate scalability, we deploy modular MAS architecture comprising Rea- soners, Tool-Users, and Judge (Fig-5(a), a()). By varying the number of concurrent Reasoners (N) and Tool-Users (M), we scale the total agent count (M N 1) to expand the exploration space. Fig. 5(b) demonstrates the scalability of MAS+AT-GRPO. While the baseline MAS+GRPO fails to scale effec- tively-saturating at 34.1% accuracy in the 7-agent regime-our method successfully leverages the increased ensemble size, achieving a continuous performance gain from 18.2% to 47.7%. This confirms that MAS+AT-GRPO can effectively scale across multiple agents without hitting the coordination bottlenecks observed in baselines. For a detailed analysis of computational efficiency and complexity, please refer to Appendix G
Zelge
hedge 5 1hedge
Scalable Interaction Graphs, (b) Accurac Scaling wir. Agent
Figure 5: (a) The system aggregates outputs from an ensemble of N Reasoners and M Tool-Users into a Judge. The total agent count scales as M + N + 1, allowing for flexible resource allocation. (b) Evaluation on AIME24 (using Qwen3-8B).
5.3 COMPARISON WITH OTHER MARL FRAMEWORKS
To assess the efficacy of our framework. we conducted ablation studies against rep- resentative baselines: As summarized in
Table 3: Comparison with existing MARL frame- works. We report Accuracy (%) for math/logic tasks and Pass I (%) for code tasks.
Tab. 3 we benchmark against three repre-
sentative baselines: MAPORI (Park et al.)
Math: Accuracy (%)
2025b), MARFT (Liao et al.) 2025), and CURE Wang al. (2025a). For fair com- parison, we utilize identical base mod- els and dataset splits. Our analysis high- lights the advantages of two distinct MAS features: heterogeneous agent roles and multi-turn iterative interaction, as sum- marized in Tab. 3
Backhone
Mathod
Config
Phi-I-mini
Vanilla Roseline MAPORL
Zero-dhut
Trined
$1.0*
Ours (MAS)
Untrained
84.4
Ours (MAPOXTEGGRP)
Trained
M6.7
Qwen2 S-Cdee 311- Instrett
Vanilla Boseline MARFT
Zero
26.8"
Trined
Untrained
(MAS-AT-CRPO)
Trained
Code: Paec (%)
Hackbone
Mithod
CodeContests
LiveCodeBench
Comparison with MAPORL. We com- pare our approach with MAPORL us-
Quen 2.5 Instract
Biseline
22.5
CURE
(MAS)
30.4
Ours (+AT-GRPO)
34.2
ing the Phi-3-mini-12288 (3.4B) on gsm8k
orggnall papers
dataset[Cobbe al. (202T). While MAPORL relies on a debating mechanism among homogeneous agents, our framework realizes role searrognneity-yynerriieee
a Reasoning Agent with a Tool-use Agent for verification. This structural specialization proves superior: our untrained MAS achieves 84.4%, outperforming the trained MAPORL (81.0%). With AT-GRPO training, our performance further improves to 88.7%.
Comparison with MARFT and CURE. This comparison highlights the critical efficacy of itera- tive alignment over single-turn workflows. In math reasoning (Qwen2.5-Coder-31 3--str-3-Insttuuet) while MARFT relies single-turn preference optimization, our framework leverages multi-turn interac- tions to facilitate active error correction and ambiguity resolution. Consequently, our inference-onle MAS (84.4%) significantly outperforms the trained MARFT 78.7%) confirming that extended

---

Published as a conference paper at ICLR 2026
reasoning horizon contributes more to robustness than single-step alignment; AT-GRPO training further amplifies this to 87.1%. A similar structural advantage is evident against CURE in code generation (Tab. B) While CURE generates code and unit tests in single turn, without utilizing them for self-correction. Our framework establishes a self-refinement cycle. This enables iterative debugging using generated tests, boosting CodeContests accuracy from 22.8% (vanilla) to 30.3% (untrained), surpassing the CURE baseline (25.9%), and ultimately reaching 34.2% with training.
5.4 ABLATION STUDY
To further investigate the contributions of our core training components, We also conducted an ablation study with results summarized in Tab. and Fig. Our analysis yields several observations. First, on-policy RL training within MAS environment is critical for effective collaboration. As shown in Tab. 4 training agents in a single- agent (SA) setting offers limited ben- efits: while individual agents im- prove their specialized skills (achiev- ing 11.00 and 14.50 accuracy, respec- tively), their performance when com- bined in MAS is only marginally better, reaching just 16.00. In stark contrast, training the agents jointly within the MAS environment boosts accuracy to 96.00. This vast performance gap demonstrates that multi-agent training is essential. It not only allows agents to co-evolve highly specialized abilities but also fosters the crucial inter-agent alignment and collabo- ration required for success.
Table 4: Plan-Path (Qwen3-1.7B) ablation. Performance
gain A over the single agent baseline.
Method
Aec.(%)
A
Single agent Training tool agent in SA, eval in SA Training code agent SA, eval in SA Training in SA, eval MAS
5.00
11.00
+6.00
14.50
+9.50
16.00
+11.00
MAS RL (role specific policies), eval in MAS w/ Swapped Policies
96.00
+91.00
6.00
+1.00
Second, RL training on MAS reinforces role- specific specialization. We observe this across multiple metrics. As shown in Fig. 6 (a) for Qwen3 1.7B on Plan-Path, the learning rewards of both the planning and tool-using agents in- crease throughout training, suggesting coordi- nated co-evolution as each adapts to the other's improving policy. Consistent with the abla- tion, after training two role-specialized poli- cies with our full method, swapping them in- duces catastrophic drop from 96.0% to 6.0%, confirming that the agents have learned distinct and complementary functions that are not in- terchangeable. In our coding (LiveCodeBench) and math (AIME25) workflows, MAS interac- tion terminates when the two agents align (e.g..
Standariiied Reward
ielleeelee Liveodeflechh AIMERS AIME2S
0
Average Turns
1/ 4. do
Agrat
L.2
20
64
503
120
Steps
Steps
(6) of Standandieed Rewards Two Model MAS.
(bi Poclutiion of Averge Torns to Allgrmere
Figure 6: (a) Evolution of standardized rewards for the Tool and Plan agents in the role-specific MAS on Plan-Path with Qwen3 1.7B. Shaded bands show variability across runs. (b) Evolution of the average turns required to solve tasks.
tests pass or the reasoner and tool outputs agree). Accordingly, Fig- 6 (b) shows that the average number of turns needed to solve a task decreases over training, providing direct evidence that the agents achieve tighter alignment and collaborate more efficiently.
CONCLUSION
In this paper, we proposed AT-GRPO, an agent- and turn-wise grouped reinforcement learning algorithm tailored for on-policy training in MAS. To support this, we introduced novel training system capable of managing diverse MAS workflows and performing on-policy updates for multiple policies. Our extensive experiments demonstrate that our method delivers consistent gains across diverse domains. On planning tasks, overcomes a key bottleneck single-agent RI. by boosting accuracy from 14 14-47% baseline to 96.0-99.5% Furthermore, it improves reasoning performance with average gains of 3.87-7.62% on coding and 9.0-17.93% on math tasks. Our analysis reveals that RL training in MAS context reinforces role-specific specialization, with the choice between shared or specialized policy contingent on the task's 's characteristics.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
We study multi-agent reinforcement learning for large language models on planning, coding, and math tasks. Our experiments are purely computational and use public benchmarks (e.g., program- matically generated Plan- Pann-Pat//uuoku instances and widely available coding/mathe datasets) together with ceff-constructed simulators and verifiers. No human subjects, sensitive personal data, or propri- etary content are involved. Code execution is performed in a sandboxed environment with restricted file 1/O and no network access; tool calls are limited to deterministic checkers to prevent unintended side effects. While our methods are intended to improve reliability and sample-cfficiency of agen- tic LLMs, we recognize dual-use risks common to autonomous systems (c.g,, unsafe tool use or over-delegation) To mitigate these risks, avoid external system operations, log all actions for auditability, and refrain from releasing any configurations that grant networked or privileged execu- tion. We also note that base LLMs may encode societal biases that training does not remove; results should therefore not be used for high-stakes decisions. We will release prompts, generators, and evaluation scripts support reproducibility, subject dataset licenses and safe-use guidelines.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have made our datasets, code, and experimental details available. All datasets used in this study are publicly available; we provide detailed descrip- tions of these datasets and all data preprocessing steps in Sec. [52 S.c. and Appendix(C.. The source code used for our experiments is included in the supplementary material. Upon acceptance, we will release the complete, documented source code under a permissive open-source license to facilitate the reproduction of all presented results. Key hyperparameters, model architectures, and training configurations are also detailed in Appendix C.I
USE OF LLM
During the preparation of this manuscript, large language model was utilized to aid in polishing the grammar and improving the clarity of the text. The authors reviewed and edited all outputs to ensure the final content accurately reflects our original ideas and are fully responsible for all statements and conclusions presented.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv:2204.01691, 2022. URL httttss//aaaxvv.aaa httpg///aa/////11///0/ 2204401161
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, and Xuanjing Huang. Polaris: A post-training recipe for scaling reinforcement learn- ing on advanced reasoning models, 2025. URL
/log///oo////// blog/2025/ Polaris
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future agentic ai, 2025.
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Ti- wari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ranchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent Ilm systems fail? arXiv preprint arXiv:2503. 13657, 2025.
Guanzhong Chen, Shaoxiong Yang, Chao Li, Wei Liu, Jian Luan, and Zenglin Xu. Heterogeneous gased reinforcement learning for llm-based multi-agent systems, 2025a. URL https:// arrxvvorg/abby /5wp.//200622111
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, and Sercan Ö Ank. Sets: Leveraging velf-verifiaation and self-correction for improved test-time scaling, 2025b. URL https:: hatps://aaxiv.orggg http://////.a///.19999909
11

---

Published as a conference paper at ICLR 2026
Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. arXiv preprint arXiv:2407.07061, 2024.
Yongchao Chen, Yilun Hao, Yueying Liu, Yang Zhang, and Chuchu Fan. Codesteer: Symbolic- augmented language models via code/text guidance. arXiv preprint arXiv:2502.04350. 2025c. doi: 10.48550VarXiv.2552..4444
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
DeepMind. Codecontests. http://ggihna..ooooogggleeeeeeee 2024. GitHub repos- itory; archived Dec 6, 2024.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 109788 2025.
W. Fu et al. AReaL: large-scale asynchronous reinforcement learning system for llms. arXiv:2505.24298, 2025. URL http:: httppp/s/arri..oo////2222..22000
Alireza Ghafarollahi and Markus J. Buehler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning, 2024.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01668 2024.
Chaoqun Hc, Renjie Luo, Yuzhuo Bai, Shengding Hu, et Olympiadbench: A challenging bench- mark for promoting agi with olympiad-leve level bilingual multimodal scientific problems. In ACL, 2024. URL httts:///rxxx http:/axxxviorr/aaa abs/ 2402.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Andy Zou, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv:2105.099) 09938. 2021.
URL htttps://arxiv.ott /atp/aaairi./aaaaa/a wwwpsvvv.org/abs/1/...09323. 2105.09938
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. International Conference on Learning Representations, ICLR, 2024.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rihf framework. arXiv preprint arXiv:2405.11145 2024.
Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Imgame-bench: How good are 1mss at playing games? arXiv preprint arXiv:2505.15144 2025.
Yuki Inoue et al. Drugagent: Multi-agent large language model-based reasoning for drug-target interaction prediction, 2024.
Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang Marft: Multi-agent reinforcement fine- tuning. arXiv preprint arXiv:2504.16122 2025. URL https:/ http:///axivv.oro .rg. abs 25t.s..0.........
Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. Spiral: Self-play zero-sum games incentivizes reasoning via multi aulti-agent multi-turn reinforcement learning, 2025. URL
hntips:/arrxiv.rgg/ abs/ aataab///2255555 2506.24119.
Shengchao Liu et al. Drugagent: Automating ai- ai-aided drug discovery programming through Ilm
multi-agent collaboration, 2024.
12

---

Published as a conference paper at ICLR 2026
Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning Ilm with sequential cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Sys- tems (NeurlPS), 2024. URL https: http://oooddiss..eueee httpppppsccpppprrrrepppppp files/ Messpaper/// /202//222/ (0002//f/ll/ r
Hangyu Mao, Zhibo Gong, and Zhen Xiao. Reward design in cooperative multi-agent reinforcement learning for packet routing. arXiv preprint urXiv:2003.05433, 2020. URL https: arxiv.org/ abs abs/2003.03433
Mathematical Association of America AoPS Community. Aime 2024 problems (aops wiki).
com /wiki wriki/indezs /t///iidx...pi php 2002400242222 202L_AIME1 & httpps//
aatofproblemsoving.ccmmm 09-11.
http://ddxxpppe http:///iixx.x.pp///// AIME Problems, 2024. Accessed 2025-
Mathematical Association of America & AoPS Community,
Aime 2025 problems (aops www/:wiiiiiiiie www.iiiiiiixx..ooo phpp2025 2025 Problees & hhips:
wiki). https: http:///rrooprobbnoooiiinnggooe
/artofproblemsollinngggo com wiki php AIME II Problems 2025. Accessed
2025-09-11.
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman E. Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collahorative large language models with rein- forcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (ACL), Vienna, Austria, 2025a, Association for Computational Linguistics. doi:
10.18653/v 1/201/ www-lon............ 1459. URL https:/ http:///aaaaanoogog.ooooo 2025 2et///222222coo0m 1459/
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman E. Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Mapor12: Multi-agent post-co-training for collaborative Ilms with reinforcement learning. OpenReview preprint, 2025b. URL httpss///peenreview. net podfiid=f5TTQhhy Influence- aware verification reward; multi-agent PPO with learned verifier.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint urXiv:2504.1300 2025. URL httlss///axxivoorge. abs/ 2504. 13958
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXi::230.004761, 2023. URL. https: poutts://arxiin /ttp///arxrxiaaaabba .org/bs//////// 2300.
04761
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Hassam Ullah Sheikh and Ladislau Bölöni. Multi-agent reinforcement learning for problems with combined individual and team reward. In Proceedings of the International Joint Conference Neural Networks (IJCNN), 2020. URL https: battss//arnae http://axxxvoor.rge abs.iii.rrg/a//////eaae.s al//aabb/020000555
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv:2409. 19256, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv:2409. 19256, 2025. http:::ss////rriioorr/g//// 19256
Noah Shinn, Federico Cassano, Edward Berman, et Reflexion: Language agents with verbal reinforcement learning. arXiv:2303.1133 2023. URL https:/ /ttp://xivvvvo/// org/ /5303/23031136
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction, MIT Press, 2nd edition, 2018
13

---

Published as a conference paper at ICLR 2026
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng.
Magis: Llm-based multi-agent framework for github issue resolution. In NeurlPS
2024,
2024.
URL https: http///orcceeens.ee/eeee neurips.cc. httppps.ccppppppppppeeeeepp ales/paper/ /2024 /0le/file/ 388-- compeeecooneeeec.......
Volcano Engine. VERL: Volcano engine reinforcement learning for llms.
https: github.cohmm
volcengine/ /verl 2025.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforce- ment Learning.
htggluuub.ooo//uuggggiiaa//////e Lrl 2020. GitHub repository.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, et al. Voyager: An open-ended embodied agent with large language models. arXiv:2305.66291, 2023a. URL https:tttttxii...oaosa aaw//r//////////aa/aaa hatp///bbb//2111111 22005.1221
Junlin Wang. Jue Wang. Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances
large language model capabilities. arXiv preprint arXiv:2406. 2024.
Liang Wang, Qian Liu, Kun Song, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308. /432, 2023b. URL htttps:///// abss////////// 2308 11422.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving llm coder and unit tester via reinforcement learning, 2025a. httpst https///aaavo/vaaba/b.a 2506.03336
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-ture turli-tturnnnn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025b.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination free 11m benchmark. arXiv preprint arXiv:2406. 19314, 2024. URL httpss/ peatts://arxivv.out wwwpivooooo//////////// /bs////////// 2406.19314
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation, 2023.
Zhizheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Li, Gongshen Cui, Yong Dou, Junzhe Zhou, Bo An, et al. The rise and potential of large language model based agents: A survey, arXiv preprint 23xx:2309.078644 2023.
An Yang and the Qwen Team. Qwen3 technical report. arXiv:2505.09388, 2025. URL https:
//arxiv.org/aat htt../org//bb//5550003333 2958800388
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. arXiv:2210.03629, 2023. URL
https: /aaxiiivoorggg.... org/ abs/ 2210.03629
Rui Ye, Xiangrui Liu, Qimin Wu, Xianghe Pang, Zhenfei Yin, Lei Bai, and Siheng Chen. X-mas: To- wards building multi-agent systems with heterogeneous llss. arXiv preprint a5Xiv: 16997, 2025.
Zhongming Yu, Hejia Zhang, Yujie Zhao, Hanxian Huang, Matrix Yao, Ke Ding, and Jishen Zhao. Orcaloca: An I1m agent framework for software issue localization. arXiv preprint arXiv:2502.003300 2025.
Kaiyan Zhang, Runze Liu, Xuekai Zhu, Kai Tian, Sihang Zeng. Guoli Jia, Yuchen Fan, Xingtai Lv, Yuxin Zuo, Che Jiang, Ziyang Liu, Jianyu Wang, Yuru Wang. Ruotong Zhao, Ermo Hua, Yibo Wang. Shijie Wang, Junqi Gao, Xinwei Long, Youbang Sun, Zhiyuan Ma, Ganqu Cui, Lei Bai, Ning Ding, Biqing Qi, and Bowen Zhou. Marti: framework for multi-agent Imm systems reinforced training and inference. https:/
2025. Preprint.
14

---

Published as a conference paper at ICLR 2026
A EXTENDED COMPARISON OF RL TRAINING FOR LLM-BASED MAS
To complement the discussion on RL training for LLM-based multi-agent systems in Sec. provide an extended, axis-by-axis comparison of representative frameworks Table 5 We con- trast MAPoRL, MARFT, MARTI, CURE, and our method along several key design dimensions: (i) whether agents share a single policy or use role-specific policies; (ii) whether the interaction pattern sequential, parallel, or a hybrid of both; (iii) whether the framework supports multi-turn interac- tion; (iv) whether agent roles are heterogeneous, (v) the number of evaluation task domains; and (vi) whether the underlying RL algorithm designed as general-purpos. MAS training scheme rather than being tightly coupled to a single task.
Table 5: Comparison of RL-based LLM multi--gent training frameworks.
Methed
Policy sharing
Execution pallern
Multi-trm
Role beterogeneity
2 domains verifcctionn
Generic MAS RL algo.
MAPaRI (aak
201250 M0-
MARFI
Tonner
CURL
Wang & air 30z-0
$
Ours (SHvigerMAS)
S+8
S4P
Note. For MARFT (Liao al. 2025), report the characteristics of version v3 of the framework, corresponding to the preprint available prior to the completion of this work.
As summarized in Tab. 5 StrongerMAS is the only RL-based LLM multi-agent training framework that simultaneously supports both shared and role-specific policies, hybrid sequential-paralle exe- cution, multi-turn interaction, and heterogeneous agent roles, while being validated multiple task domains and implemented as a general-
general-purpose purpose MAS RL algorithm. In contrast, MAPoRL (Park) purely role-shared setting and are tailored to specific tasks, MARFT Liao et al. (2025) restricts agents to single-turn sequential interactions, and MARTI /Zhng al.
al. 2025a) and CURE (Wang et al.
2025a) restrict policies
(2025) ) merely introduces basic single-agent RI. algorithms
(e.g., GRPO) to the MAS setting. This combination of flexible workflow expressivity (S+R, S+P, heterogeneous roles) and broad, cross-domain evaluation makes StrongerMAS more faithful and scalable abstraction for training cooperative LLM-based MAS.
B REWARD DESIGN
B.1 MATH REWARD DESIGN
We consider math QA with horizon T and optional tool calls. Let h2 be the dialogue/tool history at turn We adopt MATH- VERIFIER as the checker front-end.
Define numeric comparator with tolerance 6:
NUMEQu(a, b) 1f10- la- b/ S or la b)
max(1,1b]] S so). 6-10-6
Team reward. Sparse pass at termination via numerical equality. We broadcast the same scalar reward all turns:
tram = ICCHECCHHNAAAMMm//eemseetentOD
E (0, 1), VI.
Local rewards. Each agent i at turn f uses masked convex combination of verifiable sub-scores $i E [];
Σ
si
ΣΦ =1
Tt. = m4,5
tool,
where m2, E (0, 1) verifiability mask.
M VERIFY (Hugging Face), GitHub: hugginnfaaceMMat--Vrrii front-end and then apply numeric comparator.
We use it as parsingnnormalization
15

---

Published as a conference paper at ICLR 2026
Reasoner local design. Coefficients:
Reasoner 0.20,
Reasoner = 0.00,
Reasoner 0.80. step
tool
Component scores (pure numerical check):
Reasoner
1 (required output schema matched at turn th,
fmut,
MATHVERIFIER extracts a numeric 9; NUMBQs(Ú v*),
Reasoner
S step..
0,
otherwise,
and we do not use a tool-related step, so the corresponding score is implicitly Stool,t mask is
Rensoner = 0. The
my
Reasoner
( available (MATH at turn t)-
los For the Reasoner, Reaessee is obtained by plugging these coefficients, scores and the mask into the generic form above.
B.2
CODE REWARD DESIGN
We consider code synthesis with unit tests. Let S be the active test suite and
Σ
1(RUN(t,, code) = pass} E [0, 1].
S|
LES
Team reward is dense; we again broadcast over turns:
team p.
Vt.
Local rewards. Each agent i at turn & uses masked convex combination of verifiable sub-scores ft E [];
loc E,a
mi Σ
E
Coder local reward. We define the Coder's local reward as weighted combination of basic sanity checks and the fraction of golden tests passed by the generated code. Coefficients:
Coder = 0.10, Coder 0.10, Cur
Coder 0.80.
Obuld
Let Tgold the fixed set of golden unit tests for this problem. Component scores at turn are
Coder = Shiidd,
1 the candidate code compiles/imports without syntax errors at f},
Coder smoke subset of Toold runs without uncaught exceptions/timeout f), rrun,
Coder S nr.f
Σ
1(RUN(u, code) = pass),
T
i... the fraction of golden tests passed by the current code code. We apply an availability mask Coder (build/run logs and golden-test results are available f). my
and define the Coder's local reward as mi
L,Coder
m
Coder Coode Coder Cuuld Suild, + Crun $run,t + Cur Sur,f
Coder
Codeer
Tester local design.
Coefficients:
Callid
Tester
= 0.20,
Tester 0.80.
Tester uur
Σ
1(RUN(,, code") = ppsss > Tmud):
U7
wEul
where Tmut 0.60.
Mask:
Tester 1 (test runner and mutation/coveragee reports available at t}.
my
The Tester local reward Tester is obtained by combining c. Tester s Tester and m. m local-reward formula.
loc
Tester via the generic
16

---

Published as a conference paper at ICLR 2026
B.3 SUDOKU REWARD DESIGN
We consider Nx N Sudoku. Let ht be the answer action at turn / and SOLVED(-) check row/colum- n/subgrid validity. Team reward a sparse success signal termination, broadcast across turns:
team
1f SOLVED(h) -truu E {0,
At.
Local rewards. Each agent i at turn t uses masked convex combination of verifiable sub-scores E, E [0,
Edsshe
E
t,a
mas
Reasoner local design.
Coefficients:
Cin
Reasoner
0.1,
Tegaa
Reasoner
0.1,
prog
Reasoner 0.80.
Component scores (let Ge be the current grid, Gt-1 1 the previous grid; 0 denotes empty):
Reasoner fint,t
faction format is valid (full Nx. grid or list of [r, c, u))}:
Heasoner
nno row/columm/subgrrd duplicates in G.]:
legal,t
1
Reasoner
1GG--11, G,(r,c)00).
prog,t
N2
Mask:
m
Reasoner
1( (we can parse the action and compute legality/progress at t).
Tool (executor) local design. Coefficients (fixed):
Cmt
0.10,
Gexec = 0.10,
Caan = 0.80.
Component scores:
Tool fint.!
AAP/sshema valid; values in [1, N]: indices in bounds}.
Tool exee,t
(no runtime error/timeout when applying edits),
1, if all applied edits satisfy local Sudoku constraints,
Tool san..
otherwise.
Mask:
Tool 1 (executor logs available and legality checks computed at t).
m
B.4 PLAN-PATH REWARD DESIGN
We consider 2D grid path planning on HxW map with horizon T and four-neighhorhood moves. Let de be the Manhattan distance from the current position to the goal at turn / and do max(1, initial distance) for normalization. Team reward is dense and distance-improving:
if goal at t, otherwise.
team
max(0, (d.-1 de)/
Local rewards masked convex combinations
Joc t.
= m4,i
Σ
of
Σ o =1
17

---

Published as a conference paper at ICLR 2026
Planner local design.
Coefficients (fixed):
lut
Planner
0.10,
Oeg
Planner 0.10,
sp
Planner 0.80.
Component scores at turn (action or RU],,L,R}; / denotes passable neighbors; SPNEXT is 1 if a lies on at least one shortest path from 8t-1 to goal, else 0):
Planner fitt
[ (C, R)):
Planner $ Fleg,
1 (next cell in-bounds and a wall);
if SPNExT(())=1,
Planner
sp,t
0, otherwise.
Mask:
my
Planner
map known and shortest-patt oracle available at l).
Tool (executor/simulator) local design.
Coefficients:
fmnt
0.10,
Tool 0.10,
Shape
Tool
0.80.
Component scores (let Φi de be the potential used in shaping):
Tool S fmt,t
|ction list parsable as ["U", "D",""""""
Tool exeectt
no invalid move applied; simulation advances),
shape,, (cr 2 Φ1-13;
i.e. the potential does not decrease. Mask:
m
Tool
(execution logs and potentials (ór-1s 6t) available).
B.5 SOKOBAN REWARD DESIGN
consider Sokoban with horizon T on a fixed grid. Let be the number of boxes and b, the number of boxes goal at turn f. Team reward is dense in box-on-goal ratio with terminal success at completion:
[1,
if all boxes on goals at t,
team
r
b,/B,
otherwise.
Local rewards are masked convex combinations
loe
Eeiseo
mi,
Planner local design. Coefficients (fixed):
lint
Planner
0.10,
Glg
Planner
0.10,
Cilk
Planner 0.80.
Component scores at turn & (action a, E (U, RJ: PUSHOK = if planned push does collide and stays in-bounds; DEADLOCKFREE 1 if the move avoids standard static corner deadlocks for boxes not goals):
Planner Stut,
1(a, (U.D,L,)),.
Planner leg..
[step is in-bounds and not into wall; if pushing, PUSHIK = 1},
1. if DEADLOKKRREE 1,
Planner
Palk,t
0, otherwise.
Mask:
Planner my
(grid known and deadlock heuristics evaluable 1).
18

---

Published as a conference paper at ICLR 2026
Tool (executor/simulator) local design.
Coefficients (fixed):
Tool Cint
0.10,
Gexec 0.10,
Tool
0.80.
'pot
Let 2/2 = - Ehosess min (|x,, 9rl IIc -9el) be the box-to-goal potential (larger is better).
Component scores:
8 f
Tool
1 (action list parsable; symbols match {U, [U,,,,,
Tool exxe,,
I illegal push; no wall/box collision),
Toal Spot,t = 1 2 w
Mask:
m
(execution logs and potentials (14--1, v/x)) available).
B.6 OUTCOME-ONLY REWARD DESIGN
The shaped rewards Sections B 1-5 provide rich, task-specific feedback (e.g., shortest-path signals in Plan-Path and deadlock heuristics in Sokoban). To isolate the contribution of such complex shaping from that of the AT-GRPO algorithm itself, we additionally consider simplified outcome- only reward design used in our ablation studies.
The team reward is strictly binary and episodic. Let I[Success) denote the environment success indicator. The team reward is defined and broadcast over turns as
team I(Success), r
Vt.
The per-agent local reward in this setting is an auxiliary binary signal that only checks whether agent produced validly formatted action (e.g., correct API call JSON structure). Let I(FmtValid)) denote the indicator that the output of agent turn satisfies all formatting straints. We define
loc f,i
Lout ((FmtValid")
The final per-agent reward re, then combined according to Eq.
= -rr
loc. team + Tt,s
where we use a fixed, task-independent a shared with the shaped-reward configurations. By uti- lizing sparse episodic rewards and simple formatting checks rather than dense shaping signals, this outcome-only configuration is significantly more general and provides a baseline for our algorithm ablation.
B.7 THEORETICAL JUSTIFICATION FOR GREEDY TURN-LEVEL TRANSITIONS
In this section, we formally justify the optimality of greedy selection based on environment-verife rewards. Consider the underlying MDP with optimal action-value function Q* a). The Bellman optimality principle implies that any policy n² satisfying n" (s)) arg max, Q* a) is optimal (Sutton and Barto 2018).
We operate in setting where the environment returns outcome-based verifiable reward, (s, a), for each action. We posit that this reward acts as a monotonic proxy for the true value function (*, higher verification score directly corresponds a higher probability of final success. Consequently, maximizing the immediate verifiable reward is structurally equivalent to maximizing the long-term optimal value. We formalize this alignment as follows:
Assumption 1 (Monotonicity of Verification Feedback). For any state and actions a1, 02 the verifiable reward preserves the ordering of the optimal action-value function:
Tver (8, a1) > (s, a2) Q* Z
(4)
This implies that rour and Q" (s. induce consistent rankings over the action space at any state
19

---

Published as conference paper at ICLR 2026
Lemma 1 (Equivalence of Maximizers). Under Assumption 7 the of actions maximizing the verifiable reward is a subset of the actions maximizing the optimal Q-function:
arg max (8, a) arg max (s)
(5)
Proof. Let aver E arg maxe Vver a). Suppose, for the sake of contradiction, that there exists an action a' such that (* a') >> (* ). By the contrapositive of Assumption [ strict inequality in Q" implies strict inequality in fver (given consistent rankings), which would imply Fver (8, a') 2 Tver (8, ) Since over is maximizer, strict inequality impossible. If equality holds, a' is also a maximizer of Tver, and by the consistency assumption, it must share the same optimal Q-value. Thus, any action maximizing Tver necessarily maximizes Q*
Proposition 1 (Optimality of Verifier- Verifier-Greedy Greedy Policy). Let Tvver be deterministic policy such that Tver (s) E arg max, (ver a) for all states s. Under Assumption
7 Tver an optimal policy.
Proof. By Lemma selecting an action that maximizes the immediate verification score ensures that (s) E arg max, Q" (s. a). Consequently, Tver satisfies the Bellman optimality equation at every state.
In implementation, approximate this policy by sampling candidate actions and greedily lecting the one with the highest Iver. [rpoosttionguurannnnnn that this strategy effectively performs greedy search the support of sampled actions with respect the true optimal value function Q' avoiding the myopic bias typically associated with greedy transitions.
C EXPERIMENT DETAILS
C.I TRAINING DETAILS
All methods share the same hyperparameters unless noted. The maximum response length is 4096 tokens, and the (task-specific) maximum prompt length is set to accommodate turn-by-turn dialogue history: 8192 tokens mathematics and code tasks, and 16384 tokens for all other symbolic tasks. Training uses global batch size of 128, with PPO mini-batch size 64 and gradient clipping 1.0. The actor is optimized with Adam at a learning rate of 1e-6 and weight decay 0.01. We adopt GRPO for advantage estimation with 1=0 and 11100. Entropy regularization off (entropy ccocff-0). The sample temperature Teample 1.0, top-p=1.0, top-k= 1, and sample per prompt; validation deterministic (temperature do sample=False). rewards computed by rule-based function (compute score) when provided. Both models are trained for 150 steps.
C.2 PROMPT DESIGN
Code MAS Workflow
PILASE 1: GENERATION
In the initial phase, both agents given problem description. The Coder is prompted generate a solution, while the Tester is prompted to generate a corresponding test case.
Code Agent (Coder): Turn
Input:
Problem: A natural language description programming task.
Prompt:
You are helpful assistant that writes Python to solve the problem. Think step by step, then output code. Important: Read all inputs via input(). Print all results with print(). not hardcode or fabricate inputs. Now solve: Problem: "'rroblem description" First, decide on the number and types of inputs required (c.g., x int(input(), int(input()), then
20

---

Published as a conference paper at ICLR 2026
implement the solution and print the result. Please answer in the following format: Code: python (your code here)
Output: Code
Unit Tester Agent (Test-Case Author): Turn 0
Input:
Problem: A natural language description programming task, e.g., {problem).
Prompt:
You are helpful assistant that creates unit test cases (input + expected output) for coding task.
Problem: problem "iscrption"
Provide one new high-quality test case. Before giving the test case, reason carefully to ensure the output is correct, then derive the output for your chosen input. Respond in the format: **Test Input:** - "ipput here **Test Output: - output here" Output: Test input, Test Output.
PHASE 2: REFINEMENT
In subsequent turns, the agents receive feedback based on mismatches between the generated code and test cases. They are prompted refine their previous outputs.
Code Agent (Coder): Turn
Input:
Problem: The original problem description, {problem).
Mismatch History: A record of previous code, test inputs, expected outputs, and
actual execution outputs, highlighting any differences,
Prompt:
You are a helpful assistant that corrects and refines code.
Important: Read inputs via input(); output with print(). hardcode inputs. Problem: {problem}
Use the history below guide your fixes:
(mismatch history)
your previous code crashed, first fix the bug. If execution succeeded but outputs mismatched the expected output, decide if the test case is correct. the test correct, refine your code to pass it. If the test is wrong, verify your program's logic and keep it.
Provide the final, corrected code. Respond in the format:
Code: python your code here
Output: Code
Unit Tester Agent (Test-Cass Author): Turn
Input:
Problem: The original problem description, {problem).
Mismatch History: A record showing the test and the differing execution output from the Coder's program, (mismatch history).
Prompt: You are an assistant that checks and refines unit tests for coding task.
Problem: problem
Analyze the history below:
21

---

Published as a conference paper at ICLR 2026
(mismatch history]
First, decide whether your previous test case was correct (watch for misunderstandings of the task). If it was wrong or unclear, provide corrected test case. Respond in the format: **Test Input:**, **Test Output:**
Output: Test input, test output.
Math MAS Workflow
PHASE I: GENERATION
In the initial phase, two complementary agents are given the same math problem. The Reasoning Agent produces step-by-step mathematical solution and boxed final answer. The Python Tool Agent writes executable Python that computes (and prints) the final answer.
Reasoning Agent: Turn 0
Input:
Problem: A mathematical problem natural language.
Prompt:
You are a helpful assistant that solves math problems via careful reasoning.
Problem: problem
First, outline the key reasoning steps. Then carry out the full solution. After solving, present the final answer in LaTeX box.
Before giving the full reasoning, summarize the steps clearly in: **Reasoning Steps:** "reasoning steps here
Then provide your complete solution concisely. Put your final answer in: ###
Rules:
* The boxed value must be a single number or expression (simplified if possible). * Do add words after the box; only the final value goes after ####.
Output format:
1. Your reasoning (short and clear). 2. Final line must contain only the boxed answer, e.g.,
##### 123.
Output: Reasoning solution and final answer after ####..
Python Tool Agent (Coder for Math); Turn 0
Input:
Problem:
The same mathematical problem, (problem).
Prompt:
You are helpful programming assistant that writes Python to solve the math problem.
Problem
{problem}
Requirements
Write correct, readable Python that computes the final answer.
Think step by step in comments if helpful.
Use only the standard library and deterministic math (no internet, randomness).
At the end, PRINT ONLY the final numeric or symbolic answer (nothing else).
Output: Code (the program prints the final answer).
22

---

Published as a conference paper at ICLR 2026
PHASE 2: REFINEMENT
From the second turn onward, agents receive feedback derived from mismatches between the Rea- soning Agent's boxed answer and the Python Tool Agent's printed output. Each agent uses the history to refine its output.
Reasoning Agent (Math Solver): Turn
Input:
Problem:
The original problem, {problem}.
Mismatch History: Prior reasoning crreasonnggsoouution its extracted answer ([reasoning.oo (Iressoning.exttrcteee.. cxtrracted_asswrrrr... the Python code and the code's printed output (fcode (fcode.extrrateed.. atteeeeeetrcteedassseeeeeeee summarized as (mismatch historry.......
Prompt:
You are a helpful assistant that refines mathematical solutions through reasoning.
Problem: problem
History (previous attempts and outputs): mismatch history
First, compare your previous boxed answer with the Python Tool Agent's ' printed output. If the code output corrects computational slip in your reasoning, adopt the corrected value. * If the code likely has a bug (c.g., mishandled edge cases, precision, domains), keep the mathematically correct answer and explain briefly.
Then solve the problem again, more robustly. Before giving the full reasoning, summarize the key steps clearly: **Reasoning Steps:** 'reasoning steps here
Finish with the final answer after: #####
Final line must contain only the boxed value (no extra text).
Output:
Updated reasoning and final answer after ####..
Python Tool Agent (Coder for Math): Turn
Input:
Problem: The original problem, {problem).
Mismatch History: Prior code and printed output, and the Reasoning Agent's 's so- lution and boxed answer, summarized as (mismatch_hister
Prompt:
You are a helpful programming assistant that refines Python solutions for math problems. problem
Problem:
History (reasoning vs. execution mismatches): mismatch history Tasks:
1. Judge whether the Reasoning Agent's boxed answer or your previous printed result likely correct (consider numerical stability, edge cases, exact vs. float). 2. Fix or rewrite the code so it reliably computes the correct final answer.
* Prefer exact arithmetic (fractions, integers, rational simplification) when possible. * Add minimal checks for domain/edge cases. Keep outputs deterministic. Respond in the format:
**
python # corrected code here print ONLY the final answer on the line
Output: Refined code (the program prints the final answer).
Sudoku MAS Workflow
In the initial phase, two complementary agents are given the same Sudoku-solving task on an nxn grid. The Tool Agent writes executable Python that outputs either completed grid list of fill steps. The Plan Agent inspects the task, the tool code, and its execution output, then decides the final solution.
23

---

Published as a conference paper at ICLR 2026
Tool Agent (Sudoku Coder)
Input:
Task Description: {task), including grid size, rules (rows/ollummssuu--grriss con- tain unique digits), and any constraints.
Env Context: (env_context) (e.g., {size), (subgrid
{puzzle),
{observation)).
Prompt:
You Al assistant designed be helpful. Utilize your programming expertise to ad- dress the task. Propose Python code (within a single python code block) for the user to run. Ensure each response contains only ONE code block. Use the "print" function output EI- THER: (A) the completed grid as a JSON array of arrays, OR (B) a JSON list of fill steps (I,c,v) using 1-based indices.
Formatting requirements:
* The program's output is the Sudoku solution:
eg:
[[5.3,4.6.788991122]]]
2,4.5,2.8,611,77999
* Print ONLY the JSON (no extra text, no comments).
Task:
Solve the sizexsize Sudoku. Fill digits I..size rows, columns, and sub-grids must have unique digits.
Current puzzle (dots denote blanks): observation
Environment: size subgrid size: subgrid size notes/constraints: constraints
Output: Code (program prints either the completed grid JSON JSON list of fill steps).
Plan Agent (Planner & Verifier)
Input:
Task Description: (task).
Tool Code: (too..code).
Tool Execution Output: (tool execution output).
Tool Proposed Solution: [tool_solution] (ISON grid or ISON steps).
Observation (for reference): {observation).
Prompt:
You are planning and reasoning agent. You will receive:
The original task description The Tool Agent's code The code execution output (a
ISON grid or JSON steps)
Your job is to reason carefully, decide the final Sudoku solution, and format your response
EXACTLY as specified.
Instructions:
Read the task, inspect the code, and verify the execution output against the Sudoku rules: rows, columns, and sub-gride -rbiggids must contain unique digits in I.n.. If the tool's output is a complete, valid solution, adopt it. If incomplete or violates constraints, correct it provide your valid.
* Keep reasoning concise but explicit: explain why the final result is
FORMATTING IS MANDATORY. Give the final answer AFTER the line that begins with #UHH. You may return EITHER: a completed grid as ISON, OR a JSON list of fill steps (r,c,v), 1-based indices.
Examples:
#### [[5.3,4.6.7.8..911222 .... [3.445.2.8.661.7.991]
#### [11.334].12.1.611999.1]]
Output: Final Sudoku answer (completed grid JSON or JSON steps).
24

---

Published as a conference paper at ICLR 2026
D PLAN-PATH MAS WORKFLOW
PHASE I: GENERATION
In the initial phase, two complementary agents are given the same world. The Tool Agent writes executable Python that outputs action list (e.g.. [U, R, D, L)). The Plan Agent inspects the task, the tool code, and its execution output, then decides the final action list.
task on a grid/-
Tool Agent (Path Coder): Turn
Input:
Task Description: {task), including grid/map, start, goal, obstacles, and con- straints.
Env Context: (env.context) (e.g.. {grid), (start), {goal), {obstacles). {constraints)).
Prompt:
You are an Al assistant designed to be helpful. Utilize your programming expertise to ad- dress the task. Propose Python code (within single python code block) for the user to run. Ensure each response contains only ONE code block. Use the 'print' function output the action list that moves from the start to the goal. You may output the full action list if you can reach the target, a partial list if uncertain. Formatting requirements:
* Begin the Python block with python and end with The program's output IS the action list (e.g.. [U.R,D,L). Print ONLY the action list (no extra text).
Task: task
Environment: env context
Output: Code (program prints an action list, egg. (U,RDDLL).
Plan Agent (Planner & Verifier): Turn 0
Input:
Task Description: {Lask).
Tool Code: (tool_code).
Tool Execution Output:
Tool Proposed Action: [tool_action].
Prompt:
You are a planning and reasoning agent. You will receive:
The original task description The Code Agent's (Tool Agent's) code The code execution
output
Your job is reason carefully, decide the final action list, and format your response EX-
ACTLY as specified.
Instructions:
Read the task, inspect the code, and verify the execution output against the task require- ments and environment constraints (bounds, obstacles, goal). * If the code/output correct and sufficient, adopt it. Otthewisse, improve or override with your own reasoning. Keep reasoning concise but explicit: justify why the final action is correct.
FORMATTING IS MANDATORY. Give the final action list AFTER the line that begins with
####. Example:
#### [U,R
Output: Final action list.
25

---

Published as a conference paper at ICLR 2026
PHASE 2: REFINEMENT
From the second turn onward, agents receive feedback based on mismatches between the Tool Agent's printed action list and feasibility checks from the environment the Plan Agent's assess- ment. Each agent uses the history to refine its output.
Tool Agent (Path Coder): Turn >> 0
Input:
Task Description: (task).
Mismatch/Trajeee Trajectory History: Prior code and printed actions, planner feedback, and (action, state) pairs, summarized as aaateonsstate. )
Prompt:
Refine your Python solution produce correct, executable action list.
Task: task
History (previous attempts, planner feedback, and trajectory): iitionsstate_hittee
Requirements:
* Output must be action list that reaches the goal without violating constraints (stay in- bounds, avoid obstacles). If certain, print the full list; if uncertain, print a safe partial prefix. Single python code block only; program output IS the action list, * Begin with 'python and end with '; print ONLY the action list (e.g., [U.R,D,LJ). Respond the format:
**
python
# corrected code here last line prints ONLY the action list
Output: Refined code (program prints action list).
Plan Agent (Planner & Verifier): Turn > 0
Input:
Task Description: (task).
Tool Code: (tool code).
Tool Execution Output: (Lool execution execution.oottuu)
Tool Proposed Action: {LooLaction).
Action State History (if any): For each step i, The i-th action is a. The i-th state is s;. Summarized action state history.
Prompt:
You are a planning and reasoning agent.
Task: task
Tool Agent's latest code and output:
* Code: tool code Execution output: tool execution output Proposed action: tool action
Trajectory/history: action Laatonsstte___ee _historysttryy Instructions:
Verify feasibility of the proposed action sequence step by step. If it collides, goes out of bounds, loops, fails reach the goal, correct it (you may shorten, extend, or replace the sequence). Prefer the simplest valid plan; if uncertain, provide the best safe prefix and explain briefly. Keep reasoning concise but explicit.
FORMATTING IS MANDATORY. Give the FINAL action list AFTER the line that begins
with ####. Example:
*ни# [U.R.D,L]
Output: Final action list.
26

---

Published as a conference paper at ICLR 2026
E ABLATION STUDY OF OUTCOME REWARD
Table 6: Performance Comparison with Sparse Outcome-Only Rewards. To address concerns only sparse outcome signals (Outcome-
regarding reward engineering, evaluate AT-GRPO using only), removing all intermediate heuristics. Even without dense guidance, our method maintains high performance and significantly outperforms the baselines.
Baselines
AT-GRPO (Ours)
Robustness (Drop A)
Task
MAS
Outcome-only Dense (Original)
Sokuban
48.0%
72.0%
93.0%
96.0%
-3.0%
Sudoku
9.0%
16.0%
99.5%
99.5%
0.0%
Plan-Path
12.0%
71.0%
89.0%
93.0%
-4.0%
A potential concern with the dense task-specific rewards (detailed in Appendix B is that they might provide "oracle" guidance (e.g., distance-to-goal heuristics in Plan-Path), thereby simpli- fying the reasoning challenge. To disentangle the contribution of the AT-GRPO algorithm from the reward design, we evaluate our method using the Outcome-only reward formulation defined in Appendix (B.6] In this setting, all intermediate heuristic signals removed, and the agents receive positive feedback only upon successfully solving the final task, exactly matching the sparse signal availability of the baselines.
Table 6 compares the performance of AT-GRPO under dense versus sparse outcome-only rewards against the SA and MAS baselines. We observe two key findings:
1. Independence from Dense Heuristics:
The removal of dense rewards results in only marginal performance degradation. For instance, on the Plan-Puth task-where the dense reward provided shortest-path information- accuracy drops by only 4.0% (from 93.0% to 89.0%). On Sudoku, the performance remains identical at 99.5%. This indicates that while dense rewards accelerate learning, they are not prerequisite for the model's cess.
2. Superiority over Baselines:
Even in the sparse outcome-only setting, AT-GRPO main- tains decisive advantage over the baselines. On Plan-Path, our sparse-reward performance (89.0%) vastly outperforms the SA baseline (12.0%) and the MAS baseline (71.0%). This dramatic gap (+77% vs. SA) under identical reward conditions strongly refutes the hypoth- esis that our results are confounded by reward engincering. Instead, it demonstrates that the cooperative group optimization mechanism intrinsically capable of solving complex planning tasks without oracle guidance.
F MULTI TURN SINGLE AGENT
Table 7: Single-agent ablations on Code and Math (Qwen3 1.7B).
Code
Math
Setting
LiveCodeBench
APPS
CodeContests
AIME24
AIME25
Olympiad
SA, single turn
11.6
16.2
3.6
13.4
9.8
22.2
SA + multi-turn
10.4
10.4
0.0
3.3
6.7
15.8
SA. single turn RL
18.8
17.0
3.0
10.0
6.7
23.8
SA. multi-turn +RL
17.7
13.3
1.2
6.67
3.3
16.9
On the effectiveness of multi-turn single-agent variants. Tab. and Tab. & report single-agent ablations on Code and Math. For both 1.7B and 8B models, introducing a multi-turn SA variant (i.e., letting one agent repeatedly revise its own answer) brings consistent benefit over the standard single-turn SA baseline and often degrades performance, which is align with the obeservation in Chen et al. (025b) For example, the SFT stage Qwen3-1.7B LiveCodeBench drops from I1.6 to 10.4 and AIME24 from 13.4 to 3.3 when switching from single-turn SA to multi-turn SA, with similar trends AIME25 and Olympiad. After RI, the single-turn SA policy still outperforms
27

---

Published as a conference paper at ICLR 2026
Table 8: Single-agent ablations on Code and Math (Qwen3 8B).
Code
Math
Setting
LiveCodeBench
APPS
CodeContests
AIME24
AIME25
Olympiad
SA, single turn
22.8
30.2
15.75
18.3
20.0
55.0
SA + multi-tur turn
7.8
20.3
5.12
16.7
16.7
53.4
SA. single turn RL
25.7
37.0
12.12
18.3
26.67
54,8
SA, multi-turn + RI.
16.8
35.4
11.1
16.7
23.3
51.2
its multi-turn counterpart across most Code and Math benchmarks for both model scales. These results support our claim in the main text: in the absence of additional environmental signals or feedback from complementary roles, multi-turn SA interaction is a contrived use of extra turns that departs from the QA-style pretraining regime and fails to translate into improved task performance, in contrast to our multi-agent workflows where multi-turn interaction with structured cross-agent feedback yields clear gains.
G SYSTEM COMPLEXITY OF AGENT- AND TURN-WISE GROUPING
Our on-policy RL framework operates by alternating between two distinct phases: inference (rollout generation) and training (loss computation and parameter updates). In this section, we analyze the computational and memory complexity of AT-GRPO (Alg. [ and discuss how it scales with the number of agents and the turn horizon under different MAS interaction patterns and system constraints.
Notation. Let N be the number of agents, T the turn horizon, E the number of parallel envi- romment instances, and K the sampling factor (number of candidate actions per agent-turn in tree sampling). Let L denote the average number of generated tokens per action.
G.1 INFERENCE TIME COMPLEXITY
G.1.1 SYSTEM DESIGN WITH ASYNCHRONOUS VLLM GENERATION
Our implementation uses vLLM-style asynchronous engine with continuous batching for both rollouts and evaluation: each agent-turn turn query (e, t, is submitted as an independent request, and the engine maintains token-level scheduler that dynamically adds finished ones. Compared to naive synchronous design that forms a fixed batch of agent responses and waits for the longest sequence finish, this asynchronous scheme largely eliminates long-tail stragglers, keeps the GPUs close to saturation, and naturally interleaves agent-turns from parallel and sequential MAS workflows into efficient pipeline.
sequences and removes
G.1.2 INFERENCE-TIME COMPLEXITY
During inference (K=1), analyze the per-episode wall-clock latency. The complexity depends on the execution schedule Sequential or the baseline Single-Agent (SA) complexity also varies by task: for Code and Math, the SA baseline typically single-turn (T=1), whereas for Plan and Game, the SA baseline involves multi-turn interactions (T>1). 1). We denote the baseline latency as SA infer
by the interaction logic. Crucially,
Sequential MAS. In this setting (e.g., Plan, Game), agents act serially within each turn con- dition updated history. While the SA baseline requires 7 sequential steps, the Sequential MAS requires N serial agent moves for each of the 7 turns, resulting critical path of N. T. Comparing this to the multi-turn SA baseline:
Time
infer
N.T
N.
Time
T
Thus, the latency overhead scales linearly with the number of agents N.
28

---

Published as a conference paper at ICLR 2026
Parallel MAS. In this setting (e.g., Code, Math), employ multi-round debate where all N agents act in parallel in each round. By leveraging continuous batching, the N concurrent queries the inference engine. However, the parallelism is not unbounded: for
processed together a fixed model and hardware budget, there exists a maximum number of concurrent sequences the engine can hold in memory.
Let Bmax denote the maximum number of concurrent sequences that can be served by the cluster (determined by GPU memory, model size, and the target context length). With E parallel environ- ments and K candidates per agent (e.g.. K GRPO samples), the number of sequences per MAS step E N. K. To keep all agents truly parallel, we must satisfy
E.N K & Buax-
Equivalently, the maximum parallelizable agent count is
Baax E K
N Wiax
When N < Maxx the N agents at each turn can be fully batched, and the latency scales primarily with the debate depth T:
Time
Para
infer
< T.
Time infer
When N Max. the engine automatically schedules the N agents in [N/Nmax] waves, and the latency bound becomes
Time infer SA Time infer
S T- [N/NNax]:
which smoothly reduces to the single-wave case when N S NMmxx
G.2 TRAINING-TIME COMPLEXITY
The computational bottleneck during training lies in the forward and backward passes for the col- lected candidate actions. For E environments, N agents, and T turns, with K samples each, the total rollout sample size is (D) MAS = E N. K.
The proposed agent- and turn-wise grouping introduces only a lightweight hashing overhead of O(D)MAs), which is negligible compared to the token-level model execution ( MAS L- Cmedel..
Therefore, the complexity relationship between our multi-agent approach and the standard single- agent GRPO (IDlsA = T. K) is defined by the ratio of their rollout sample sizes:
MAS
D MAS
grin
NT.
Time train
SA
Disa
This demonstrates that our method introduces no extra asymptotic complexity beyond linear scal- ing with the number of agents N.
G.3 EMPIRICAL LATENCY STUDY
We conducted latency profiling on cluster of four H100 GPUs with an effective decoding batch size of 32 x 8. For the Code task, one on-policy iteration for the single-agent baseline ( (N=1,7=1) requires approximately 4 minutes for rollout (inference) and minute for AT-GRPO training: thus, inference dominates roughly 80% of the total wall-clock time. Scaling to the MAS setting (2= multi-turn) approximately results in 8 minutes for rollout and 2 minutes for training.
In the Game domain, while training costs remain comparable to the Code task, we observe inver- sion in inference latency. The single-agent baseline averages 2.8 minutes per rollout, whereas the MAS inference time drops 1.5 minutes. This reduction is attributable to the superior performance of MAS: the group efficiently completes tasks in fewer turns (triggering early termination), whereas the single-agent policy frequently struggles and exhausts the maximum turn horizon.
29

---

Published as a conference paper at ICLR 2026
H CASE STUDIES OF MAS WORKFLOWS
This appendix presents two concrete multi-agent case studies, one in a box-pushing grid game and one in code generation with unit tests. For each domain, we include the original prompts and agent- facing messages, and we distinguish erroneous behaviors from successful ones using x and v, respectively.
H.1 MAS FOR GAME
Task. Task: Planner proposes the next action sequence; Executor calls environment tools (sim- ulator, legality checker, shortest- phortest-path/BFS helper) to apply actions and return effects/observations (updated grid, agent/box poses, success/failure flags). Episode ends when the goal is met (all boxes on targets) or the turn budget is reached.
The Plan Agent gets a valid path for the box from Tool agent but completely misses the point. It tries to follow the box 's path itself, runs straight into a wall, and fails instantly. It doesn't understand job is to push the box, not the box.
Before RL (x). Before RL:
After on-policy RL in MAS (V). After on-policy RL in MAS: RL teaches the agent the differ- ence. It learns that rewards come from moving the box along the designated path. This insight forces it discover the correct low-level strategy: first, navigate behind the box, then execute the push.
The Puzzle (Game Grid)
Input (environment prompt):
#
# ####
#
# wall, free, A agent, B box, G goal
Start: 188,1, B(6,6), G(1,8) 1-indexed (row,col).
Output: Initial grid state for the MAS.
Tool Agent (Box Path via BFS)
Input:
Task Description: Given the grid and object positions, compute an optimal path for the box from its start to the goal.
Prompt (Tool Agent behavior):
tool agent perceives obstacles and calculates an optimal path for the box using BFS: Output (Tool Agent response):
Box path: [R. U, U, U, U, R]
(Validated: free behind-space each step)
Plan Agent
Failed Interpretation
Input:
Task Description: Use the Tool Agent's 's box path to control the robot.
30

---

Published as a conference paper at ICLR 2026
Tool Output: RR R, U, U, U, R..
Prompt (narrative shown to the agent):
The planning agent directly applies the box path to itself, failing to understand that pushing box is different from moving into free space.
Response (Plan Agent thought and action list):
Thought:
"Great, I'll execute these moves."
Planned robot actions:
[R. R, U, U, Rj
Outcome: Blocked by walls; gets behind the box to push it. = Failure
Plan Agent
Correct Interpretation
Input:
Task Description: plan.
Move the box from its start the goal using the Tool Agent's
Tool Output: [R, R, U, U, U, Rj.
Prompt (narrative as shown to the agent):
The agent correctly translates the high-level plan (move the box) into sequence of agent- specific actions (get behind the box, then push).
Response (Plan Agent thought process):
Thought Process:
1. Translate each box push to a required pre-push robot cell.
Push Right > Robot must be Left of box.
Push Up - Robot must Below box.
(And so on for I. and D...)
2. For each step in the box plan, route the agent to the corresponding pre-push cell (avoiding walls), then move one final tile in the push direction.
Output: A valid sequence of robot moves that first routes behind the box and then pushes it along the planned path, achieving the goal V.
H.2
MAS FOR CODE
Code Generation Case Study. Task: Coder writes a solution; Unit-Tester writes tests. Termi- nate=all tests pass. Otherwise: each agent revises its own previous output using the environment feedback/results (Coder fixes code; Unit-Tester fixes unit test), then re-run.
The Plan Agent gets valid path for the box from Tool agent but completely misses the point. It tries to follow the box 's path itself, runs straight into a wall, and fails instantly. It doesn't understand job is to push the box, not be the box.
Before RL (x). Before RL:
After on-policy RL in MAS (V). After on-policy RL in MAS: RL teaches the agent the differ- ence. It learns that rewards come from moving the box along the designated path. This insight forces it to discover the correct low-level strategy: first, navigate behind the box, then execute the push.
The Prompt (Unit- Test Generation)
Input (environment prompt):
31

---

Published as a conference paper at ICLR 2026
A request is sent to unit test generator calculate the "prettiness" sum of an array of size n using the formula:
ged (0; az) ged(i.
1=1
The generator is instructed to produce the test input and output following a specific format:
**esst Input:** imput here
**Test Output:** output here
Output: Problem specification and required I/O format for the Unit-Tester.
Unit-Tester Failure 1: Missing Output
Prompt (failure description):
The generator only provided the test input, failing to include the required output. Response (incorrect test case):
**ess Input:**
n -5 3, 6, 1,
Test Output:** ** (is missing)
Outcome: The required **Test Output:** field is missing entirely, the test case is unusable x.
Unit-Tester Failure 2: Wrong Format
Prompt (failure description):
The output doesn't 't adhere to the specified **Test...* ** bolded format.
Response (incorrect test case):
test input: n
a
2,
test output:
77
Outcome: Although both input and output are present, the format violates the required bolded **Test Imput:* and **Tsst Output:** structure, so the checker rejects it x.
Unit-Tester
Success (MAS + AT-GRPO)
Prompt (success description):
Correct Case (MAS AT-GRPO)
The output correctly follows the format, provides both input and output, and ensures all variables consistent.
Response (correct test case):
** Test. Input:**
a 3, 6, 2, 1,44
** Test Output:**
77
32

---

Published as a conference paper at ICLR 2026
Outcome: The test fully respects the prescribed format and contains consistent input and output fields, enabling reliable automatic checking v.
33


Reviews:
[{"review_id": "vinuZbQiZx", "reviewer": "Reviewer_aB72", "summary": "This paper introduces AT-GRPO, an agent- and turn-wise grouped reinforcement learning method, alongside a novel training system, to enable effective on-policy RL for LLM-based multi-agent systems. Through extensive experiments across gaming, planning, coding, and math tasks, the authors demonstrate that their approach consistently enhances performance, particularly in long-horizon planning where it elevates success rates from 14-47% to over 96%.", "questions": "- The description of Figure 3 is unclear. Where is the “middle” section referring to the code debugging task (mentioned around line 214)? Also, what do the gray and blue colors represent in the figure?\n\n- The notation used across sections is inconsistent. For example, Equation 1 uses index  l \\in \\{1, \\dots, K\\}, while Algorithm 1 uses c \\in \\{1, \\dots, K\\}. This inconsistency hinders readability.\n\n- How is the number of agents \\( N \\) determined in the MAS setup?\n\n- What is the advantage of tree sampling over parallel sampling?  Additionally, in Algorithm 1, greedily selecting the action with the maximum turn-level reward at each turn t may lead to suboptimal final outcomes. Could this myopic selection affect the overall reward?\n\n- Equation 3 does not include a clipping mechanism to constrain policy distribution changes. Is there a specific reason for omitting this common GRPO stabilization technique?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The experimental design is comprehensive, and the results consistently demonstrate the effectiveness of the method. However, the model scales used (1.7B, 8B) are relatively small, leaving uncertainty about how well the approach would scale to much larger models.\n\n- The finding that the choice between a role-sharing policy and role-specialized policies depends on the task characteristics is a very interesting and valuable insight for the field.", "weaknesses": "- The paper is at times hard to read. Key explanations, such as the components of Figure 3, are unclear.\n\n- See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGZLgrQAb6", "reviewer": "Reviewer_SCZc", "summary": "This paper focus on developing on-policy RL to Multi-Agent Systems of LLMs. This work introduces AT-GRPO, which uses an \"Agent- and Turn-wise\" grouped tabular to store agent- and turn-wise state values. This work also developed on-policy training for shared/separated LM parameters, based on VeRL.\n\nExperiments across four domains (game, planning, coding, math) reveal a dramatic bifurcation: near-perfect accuracy on long-horizon planning tasks (96–99.5%), but modest gains on complex reasoning (3–18% avg. improve in code/math).", "questions": "1. What is the computational and memory complexity of maintaining the \"Agent- and Turn-wise\" advantage grouping (Algorithm 1, line 8)? And how does this approach scale as the number of agents ($N$) and the turn horizon ($T$) increase in more complex MAS environments?\n\n2. Tables 1 & 2 show that the \"MAS (prompt-only)\" baseline significantly outperforms the \"Single Agent (prompt-only)\" baseline, even without any RL training. Sec 5.1 mentioned that for code and math, the SA baseline is \"single-turn termination\" while the MAS baseline is allowed $T=4$ turns. Is this performance gap primarily an artifact of this unfair comparison, where the MAS baseline benefits from multiple turn iterative refine while the SA baseline does not?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript is well-written and easy to follow.\n\n2. The authors have invested considerable effort in designing an effective RL training framework for multi-agent systems (MAS), adeptly handling both shared and distinct LLM parameters.\n\n3. Comprehensive experiments were conducted across two model scales (Qwen-3 1.7B and 8B).", "weaknesses": "1. The experiments use an unfair reward engineering for the proposed method: AT-GRPO got 96-99.5% on planning/games while the single agent GRPO baseline only got 0-56%. A major reason is the local reward functions detailed in Appendix A.1, they provide lots of oracle information to the algorithm, e.g. in Plan-Path, the local reward is 1 if and only if the agent's chosen action \"lies on at least one shortest path from $s_{k-1}$ to goal.\" **This is not learning to reason; this is learning to imitate an predefined oracle.** In contrast, no detailed reward function is provided for the single-agent GRPO baseline, suggesting it was unfairly handicapped and may have only used a sparse final reward.\n\n2. The proposed Agent- and Turn-wise Group-normed advantage is a very natural extention of tabular-wise value estimate to MAS, like GiGPO[1] from GRPO, which is not mentioned in this paper. And consider the literature of cooperative MARL, tabular-wise value estimate may not be a reasonable credit assignment, a common practice in this field is to use CTDE methods like MAPPO and MADDPG which introduce a global value function for credit assignment.\n\n3. The author should conduct ablations on the Agent-wise and Turn-wise design of the advantage, e.g. an experiment of MAS + GRPO.\n\n4. This paper also missing some import MARL baselins for LLM, e.g. MAPoRL mentioned in the related work. Additionally this paper should also discussed some related works like MARTI[2] and MARFT[3]\n\n[1] Feng, Lang, et al. \"Group-in-group policy optimization for llm agent training.\" arXiv preprint arXiv:2505.10978 (2025).\n\n[2] https://github.com/TsinghuaC3I/MARTI\n\n[3] Liao, Junwei, et al. \"Marft: Multi-agent reinforcement fine-tuning.\" arXiv preprint arXiv:2504.16129 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LluXLCp8cG", "reviewer": "Reviewer_2wky", "summary": "This paper addresses the significant challenge of applying on-policy RL to Multi-Agent Systems composed of Large LLMs. The authors identify two primary bottlenecks. First, standard group-based RL objectives, such as GRPO, rely on an assumption of identical prompts for all samples within a comparison group. This assumption is fundamentally violated in MAS, where prompts are heterogeneous, differing by agent role and interaction turn. Second, existing RL-for-LLM training frameworks are predominantly designed for single-agent, single-policy optimization. They lack the architectural support for the complex, multi-agent rollouts and concurrent on-policy parameter updates for multiple, distinct policies required by MAS. To overcome these challenges, the authors propose a two-part solution, the AT-GRPO. This method creates valid GRPO comparison groups by branching $K$ samples at each agent's specific turn in the interaction. The algorithm is characterized by (i) tree-structured sampling, (ii) agent-and-turn-wise grouping, and (iii) an agent-wise credit assignment mechanism that utilizes a mix of global (team) and local (agent-specific) rewards.", "questions": "See Weaknesses", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper correctly identifies a key frontier in AI research. The field is actively moving beyond single-agent LLM fine-tuning and prompt-only MAS frameworks. The challenge of applying on-policy RL—which is often crucial for stable learning in complex, non-stationary environments—to heterogeneous, multi-policy MAS is both a real and difficult problem. The paper's attempt to provide a principled solution is commendable.\n\n2. The paper provides a nuanced and well-reasoned analysis of the trade-off between using a single, role-sharing policy versus multiple, role-specialized policies. The finding that specialized policies are superior for roles with highly distinct functions (e.g., the Coder and Tester in the code domain) is intuitive but empirically validated. Conversely, the finding that a shared policy can sometimes be superior for roles with overlapping skills (e.g., the Reasoner and Tool agent in the math domain, where the Tool agent still needs to reason) is a practical and useful takeaway for researchers and practitioners in this field. \n\n3. The practical engineering of the MAS training system, while derivative of prior work (HybridFlow), is a non-trivial and necessary contribution to enable this line of research. More importantly, the Appendix is exemplary in its thoroughness. It provides exhaustive details on the complex, task-specific reward functions for all five domains (A.1.1-A.1.5) and the complete prompt templates for every agent, role, and interaction phase (A.2.2). This level of transparency is critical for reproducibility and is highly commendable.", "weaknesses": "1. The paper's headline claim—that AT-GRPO boosts planning accuracy from ~14-47% to 96-99%—appears to be built on a comparison between two fundamentally different tasks.\n\n- Baseline Definition: The single-agent (SA) baseline for Plan/Game tasks is defined as \"one agent outputs a plan (same termination condition)\". For the Code baseline, it is \"one agent emits code; single-turn termination\". This implies a one-shot, open-loop plan generation.\n- MAS Workflow Definition: The Multi-Agent System, by contrast, is an iterative feedback loop: \"Planner proposes actions; Executor calls tools and returns effects/observations... termination when the goal is met or turns reach K\".\n\nThis is a comparison between apples and oranges. The SA baseline is given an exponentially harder open-loop planning problem, while the MAS method is given a standard, iterative, closed-loop planning problem where it receives state feedback at every turn. The massive performance gain (e.g., 14.0% -> 96.0% on Plan-Path ) is likely dominated by this difference in task workflow, not the RL algorithm. The paper is currently attributing the gains of the MAS workflow to the AT-GRPO algorithm. \n\n2. The paper's remarkable achievements on planning tasks (96-99% accuracy) are fatally confounded. These results are not the product of the AT-GRPO algorithm, but rather of large-scale, complex, task-specific reward functions. These reward functions (see Appendix A.1 for details) provide the agent with \"oracle-level\" guidance. The algorithm is merely learning how to optimize a pre-computed solution.\n- Plan-Path $s_{sp,k}^{Planner}$: The reward component is based on a function called SPNEXT. A positive reward is given if and only if the agent's action $a_k$ is \"on at least one shortest path from $s_{k-1}$ to the goal\". This is, by definition, a shortest path oracle. The agent is not \"learning to plan\"; it is simply rewarded at each step for following \"breadcrumbs\" provided by an external oracle that has already solved the problem.\n- Sokoban $s_{dlk,k}^{Planner}$: This component is based on a DEADLOCKFREE heuristic, rewarding the agent for moves that \"avoid standard static corner deadlocks.\" The whole difficulty of Sokoban lies in identifying and avoiding such deadlocks. It provides dense, step-by-step \"breadcrumbs\"—which, by definition, is a shortest path oracle—for each step. The agent isn't \"learning to plan\"; it's simply rewarded at each step for following \"breadcrumbs\" provided by an external oracle that has already solved the problem. This step-by-step reward makes the long-term credit allocation problem insignificant.\n- Code $s_{cov,k}^{Tester}$: Testers are rewarded based on the \"mutation score\" of their generated tests on \"golden code.\" This requires not only a golden reference implementation but also a complete mutation testing framework. This is an extremely powerful and expensive validator, far exceeding simple \"pass/fail\" signals.\n\n3. The paper's presentation of its reward mechanism is contradictory and misrepresents a critical, hand-tuned component of the method. Sec 5.1 explicitly states: \"The reward-mixing coefficient is $\\alpha=1$ without further tuning\". Equation 4 1 defines $\\alpha$ as the weight on the team reward ($r^{team}$). This claim implies the method uses 100% team reward and requires no task-specific reward tuning. This claim is factually incorrect. The appendix, which details the actual reward designs, reveals complex, hand-picked, task-specific mixing coefficients (which are renamed $\\lambda$): $\\lambda_{math} = 0.70$, $\\lambda_{sudoku} = 0.60$, $\\lambda_{plan} = 0.50$, $\\lambda_{sok} = 0.40$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymIs7GqVct", "reviewer": "Reviewer_GGUj", "summary": "This paper proposes AT-GRPO, which addresses the problem of applying on-policy RL to MAS with LLMs to improve their collaborative capabilities. This method achieves significant improvements on the Qwen3 model across four tasks: game playing, planning, coding, and mathematics.", "questions": "1.\tAll current experiments are based on two agents. Has the scalability to the number of agents been examined, including performance and computational overhead?\n\n2.\tSimilarly, the paper repeatedly emphasizes the significant gains on \"long-horizon planning tasks\" in the abstract and conclusions. However, the turn horizon in the experimental setting is set to T=4.\n\n3.\tThere is a lack of strong comparison with other related work on MAS-RL, such as MAPoRL and CURE.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this paper addresses the question of 'why standard GRPO fails in MAS' and proposes a novel sampling and grouping strategy based on the GRPO algorithm, which is applicable to MAS and provides a supporting engineering implementation.\n\nThe experiments are logically clear and demonstrate the advantages of the algorithm and the effectiveness of its components under different parameter base models and task domains.", "weaknesses": "In this paper, AT-GRPO uses K sampling and greedy selection of the optimal action in the rollout phase. This effectively provides a search budget of K for each decision step. In contrast, the baseline does not appear to have this budget. \n\nI have doubts about the fairness of this comparison. How much of the performance improvement is due to AT-GRPO's RL update algorithm, and how much is due to the K-fold increase in the search budget during rollout?\n\nFurthermore, the paper relies on GRPO's variance reduction properties, but does not theoretically analyze whether the new \"tree sampling + greedy execution\" strategy introduces new biases, or discuss the stability of its advantage estimate as T and K increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)