MARTI: A Framework for Multi-Agent LLM Systems Reinforced Training and Inference
Authors: Kaiyan Zhang
Kai Tian
Runze Liu
Sihang Zeng
Xuekai Zhu
Guoli Jia
Yuchen Fan
Xingtai Lv
Yuxin Zuo
Che Jiang
Yuru wang
Jianyu Wang
Ermo Hua
Xinwei Long
Junqi Gao
Youbang Sun
Zhiyuan Ma
Ganqu Cui
Ning Ding
Biqing Qi
Bowen Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Multi-Agent
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: We present MARTI (Multi-Agent Reinforced Training and Inference), an open-source framework designed to facilitate scalable and efficient learning of multi-agent LLM systems. MARTI supports centralized multi-agent interactions and distributed policy training, with the added capability of multi-turn asynchronous rollouts to enhance training efficiency. The framework includes dynamic workflows for multi-agent interactions, which integrate both rule-based verifiable rewards and LLM-based generative rewards. We validate the effectiveness of MARTI through comprehensive experiments on diverse mathematical tasks, demonstrating that multi-agent LLM-based systems outperform single-agent systems within the same inference budget after convergence. Our contributions lay the foundation for exploring scalable collaborations within LLM-based multi-agent systems and advancing the capabilities of large reasoning models.

Published as a conference paper at ICLR 2026
MARTI: A FRAMEWORK FOR MULTI-AGENT LLM SYSTEMS REINFORCED TRAINING AND INFERENCE
Kaiyan Zhang
Kai Tian Runze Liu Sihang Zeng'
Xuekai Zhu Guoli Jia
Yuchen Fan 266
Xingtai Lv
Yuxin Zuo' Che Jiang'
Yuru Wang'
Jianyu Wang'
Ermo Hua 16
Xinwei Long' Junqi Gao Youbang Sun'
Zhiyuan Ma?
Ganqu Cui
Ning Dinglo
Biqing Qi Bowen Zhou '61
"Tsinghua University ZShanghai Jiao Tong University University of Washington
"Beijing Institute of Technology
*Harbin Institute of Technology 'Huazhong University of Science and Technology
'Shanghai AI Laboratory
8 Frontis.Al
Project Leader
*Core Contributions *Corresponding Author
zhang-ky22
wwwignii.nnaee.. wwwphuaaeedd...ooo cn
MARTI
ABSTRACT
We present MARTI (Multi-Agent Reinforced Training and Inference), an open- source framework designed to facilitate scalable and efficient learning of multi- agent LLM systems. MARTI supports centralized multi-agent interactions and distributed policy training, with the added capability of multi-turn asynchronous rollouts enhance training efficiency. The framework includes dynamic workflows for multi-agent interactions, which integrate both rule-based verifiable rewards and I.LM-based generative rewards. We validate the effectiveness of MARTI through comprehensive experiments diverse mathematical tasks, demonstrating that multi-agent LLM-based systems outperform single-agent systems within the same inference budget after convergence. Our contributions lay the foundation for exploring scalable collaborations within LLM-based multi-agent systems and advancing the capabilities of large reasoning models.
INTRODUCTION
Large Reasoning Models (LRMs), such as DeepSeek-R1 (Guo et al., 2025) and OpenAl ol/o3 (El- Kishky et al., 2025), highlight the significant role Reinforcement Learning (RL) plays in enhancing the reasoning capabilities of Large Language Models (LLMs) for solving complex problems. Notably, LRMs can explore and generate extended chains of thought using only rule-based outcome rewards. This RL paradigm has also demonstrated considerable progress in other domains, including visual reasoning (Liu et al., 2025d; Zhou et al., 2025; Team et al., 2025) and agentic reasoning (Wang et al., 2025c: Jin al., 2025) tasks. These studies indicate the effectiveness of scaling up test-time inference computations using RL. However, further performance improvements through post-training RL typically demand substantial computational resources. Additionally, recent research suggests that RL primarily activates intrinsic capabilities and reflective patterns established during pre-training (Gandhi al., 2025; Yue et al., 2025a; Shah et al., 2025). Consequently, the initial model's passk performance sets an upper bound for RL-based enhancements (Yue et al., 2025a), which means the base model determines the reasoning limit. Therefore, the most viable approach for significantly boosting policy model performance remains within the scaling laws (Kaplan et al., 2020; Brown et al., 2020), either by training models on larger datasets or increasing the model's parameter size. Regarding the reinforcement learning stage, effectively leveraging the potential of exploration and environmental interaction remains a critical challenge (Silver & Sutton, 2025).
Meanwhile, LLM-based Multi-Agent Systems (MAS) (Han et al., 2024; Guo et al., 2024) scale inference computation by expanding the number of agents, each adaptively responding to specific tasks. Numerous open-source frameworks for LLM-based MAS currently available, including AutoGen (Wu et al., 2023a), CAMEL (Li et al., 2023), and MetaGPT (Hong et al., 2024). However, these frameworks predominantly rely LLM inference. This reliance makes their efficacy highly

---

Published as a conference paper at ICLR 2026
Table Comparison between Multi-Agent and RL Framework.
Framework
MAS Inference
Single RL
MAS RL
CAMEL (Li et al., 2023)
X
X
AutoGen (Wu et al., 2023b)
X
X
Meta-GPT (Hong et al., 2023)
X
GPTSwarm (Zhuge
2024)
TRL (von Werra et al., 2020)
OpenRLHI (Hl et al., 2024a)
/
Verl (Sheng
2024)
/
AReaL (Fu et al., 2025)
MARTI (Our)
dependent on the iestruction-followiii capabilities of the LLMs. factor that, as recent studies (Pan al., 2025) indicate, can readily contribute to operational failures. Concurrently, several RL frameworks (e.g., OpenRI.HF (Hu et al., 2024b), veRL. (Sheng et al., 2025), TRL (von Werra et al., 2020)), designed train LLMs, can enhance LLM reasoning abilities but do not support LLM-bbaed MAS. This observation prompts essential question:
leverage RL to improve LLM-based MAS and thereby achieve superior reasoning performance? Addressing this question requires mitigating the gap between inference and RL training in LLM-based MAS, which in turn necessitates a unified framework integrating multi-agent reinforcement learning with inference capabilities.
In this work, propose the Multi-Agent Reinforced Training and Inference (MARTI) framework for systems. MARTI is built upon the OpenRLHF framework (Hu et al., 2024b), which enables scalable and pigh-performance RL for LLMs. For multi-agent inference, integrate asynchronous workflows to facilitate dynamic interactions. MARTI employs a centralized interaction design for its built-in workflows (e.g., Multi-Agent Debate, Chain-of-Agentss and Mixture-of-Agents) and customizable workflows, while utilizing distributed policy training for individual agents. During inference, MARTI supports rule-based rewards used in DeepSeek-RI (Guo et al., 2025), along with generative reward models (Liu et al., 2025c). Prior to transferring rollout experiences to distributed agent policies, MARTI incorporates several reward shaping techniques (Park et al., 2025; Motwani et al., 2024) and credit assignment strategies to allocate rewards effectively.
LLM-based
Our preliminary experiments demonstrate that MARTI enhances multi-agent workflow performance, achieving higher upper bound than single-agent RL training under the same inference budget. For instance, our multi-agent debate workflow based on DeepScaleR-1.5B: 1.SB-Preview (Luo et al., 2025) attains score of 65.0 on the AIME benchmark, surpassing the single-agent baseline (53.5), which relics on large reasoning models with test-time RL. However, challenges remain in multi-agent RL, including the need for improved reward models for multi-agent systems (Pan al., 2025) and real-world applicability (Li et al., 2025: Zheng et al., 2025). We will continue optimizing MARTI to advance MAS training for high-value applications.
Our contributions can be summarized as follows:
We propose and open-source the Multi-Agent Reinforced Training Infrastructure (MARTI,, framework that facilitates centralized multi-agent interactions and distributed policy training, enabling scalable multi multi-agent agent learning. MARTI also supports multi-turn asynchronous rollouts during training to enhance the efficiency of multi-agent learning.
We implement dynamic workflows for multi-agent interactions that support both rule-based verifiable rewards and LLM- generative rewards.
We conduct comprehensive experiments various mathematical tasks, which demonstrate that multi-agent L.LM-based systems can achieve superior performance than single agent under the same inference budget after convergence.
2 MARTI: MULTI-AGENT REINFORCED TRAINING AND INFERENCE
2.1 FRAMEWORK DESIGN
We designed the MARTI framework based on the principle of centralized multi multi-agent interaction with distributed policy training, where all agent interactions and reward allocation occur centrally, while
2

---

Published as a conference paper at ICLR 2026
MARTI: Multi-Agent Reinforced Training and Inference
OpenRLHF
Multi-Agent Interactive World
Single Agent Policy Trainer
Agent
Agenr
Agent 3
Agent
(SFTGGRPO/PP))
Centralized Rewarding (Process & Outcome Verifier)
Credit Assignment
AG
Reward Shaping
Verifier
Verifier
Verifler
Verifier
Rule-based Verifier
Figure 1: Overview and motivation behind of MARTI.
policy training is distributed across individual agents. As illustrated in Figure 1, MARTI consists of three core modules for rollout generation and policy training: Multi-Agent World, Centralized Reward Models, and Agent Policy Trainer. The relationships and detailed descriptions of each module are provided in the following section.
Multi-Agent World. This module serves the environment for all multi-agent interactions and experience generation. Its core functions are to execute prompt-driven rollouts according to specified interaction workflows, manage the credit assignment mechanism for resulting trajectories, and perform the necessary data format conversion for downstream RL training of individual agents. To ensure maximal experimental flexibility, the system supports the dynamic injection of custom workflows, which define the multi- interaction logic and provide access to vLLM engines Trajectories collected in this environment are subsequently processed using Centralized Reward Models. A key architectural feature is the workflow's 's support for asynchronous generation, which significantly enhances data throughput. The abstract workflow interface is defined Code and complete code example is provided in Appendix B.3 for reference.
Centralized Reward Models. Following world interactions, this module collects trajectories and performs credit assignment and reward shaping. Initial global rewards computed using either rule- based strategies or generative reward models, which are then decomposed into agent-level rewards for subsequent agent training. Section 2.2.1 introduces rule-based rewards (e.g., DeepSeek-RI and influence-aware reward shaping for MAS. For open-domain applications, Section 2.2.2 presents generative reward models that extend to LLMM-asjudge approaches (Zheng al., 2023; Gu et al., 2024) for multi-agent reward allocation across roles and collaborations.
Agent Policy Trainer. After trajectory collection and reward allocation, MARTI distributes agent- specific trajectories and rewards to individual policy trainers. Here, backbone LLMs undergo supervised fine-tuning or reinforcement learning. Section 2.2.3 discusses policy training strategies. Leveraging distributed training capabilities. we implement various RL algorithms from OpenRLHF, including REINFORCE++ (Hu, 2025), GRPO (Shao et al., 2024), and PPO (Schulman et al., 2017), while maintaining extensibility for novel algorithms such as PRIME (Cui et al., 2025). We additionally integrate supervised fine-tuning during on-pan on-policy rollout training enhance stability and accelerate convergence. These dynamic training strategies warrant further investigation regarding on-policy and off-policy combinations (Yan et al., 2025; Tang et al., 2025).
3

---

Published as a conference paper at ICLR 2026
2.2 ALGORITHMS IMPLEMENTATION
In this section, we present the implementation details of reward allocation and policy training for multi-agent training in MARTI. For reward allocation, we first discuss rule-based reward shaping (Section 2.2.1), followed by generative reward models for open-domain applications (Section 2.2.2), and finally policy training strategies (Section 2.2.3).
2.2.1 1 RULE-BASED REWARD SHAPPNG
For mathematical problems with verifiable solutions, employ rule-based reward models such as DeepScek-R (Guo et al., 2025). This approach is particularly effective for mixture-of-agents (Wang et al., 2025a) and multi-agent debate (Du et al., 2024) scenarios, where each agent's ' output can be directly evaluated against the ground truth solution, enabling precise reward assignment based on predefined scoring rules. To improve temporal consistency and leverage historical information in multi-turn interactions, we introduce an inference-aware reward shaping strategy from MAPoRL (Park al., 2025). This method integrates past performance estimates with current rewards. Specifically, the approach combines an immediate correctness reward from task verifier with a dynamic adjustment derived from the agent's historical performance. This historical performance is calculated as the average reward across previous interactions.
We implement two variants: (1) a Quality Mode, which encourages consistency by aligning current performance with historical correctness, and (2) Margin Mode, which directly rewards agents for surpassing their historical average performance. Additionally, two historical evaluation scopes are provided: one considers only the most recent interaction, offering immediate but potentially variable feedback, while the other averages across all past interactions for more stable and reliable estimates. These modular and flexible strategies effectively reduce overfitting single-turn outcomes, enhancing long-term collaboration effectiveness in multi-turn scenarios.
Let R2 E [0, 1] denote the immediate correctness reward assigned by task verifier for agent i at turn t, and let Q; E [0, 1] represent the historical performance estimate of the agent, computed over a set of previous interactions:
Q
R REH;
7
where Hi {l....,,-
f-1 denotes the historical evaluation scope (c.g., most recent round or all previous rounds). We define the dynamic shaping term A under two modes:
Margin Mode:
A Ri Qi
(2)
Quality Mode:
A Qi Rj (1 -1- R2).
The final shaped reward R is then given by: RC R2 + a A; where a E R20 is a tunable hyperparameter controlling the influence of historical consistency.
2.2.2 GENERATIVE REWARD MODELS
Recent advances have demonstrated that LLMs effectively evaluate response quality, enabling their use as generative reward models (GenRMs) to enhance policy model reasoning capabili- ties (Zhang al., 2025e; Mahan al., 2024; Zhao al., 2025). Building on these developments, we implement GenRMs in MARTI for both verifiable and open-domain problems. Our framework supports GenRMs through either local vLLM engines or OpenAA--cmpptiine APIs, with defined GenRM that assigns scalar rewards to given problem-trajectee trajectory pairs.
Furthermore, we investigate specialized GenRMs for multi-agent systems (MAS) that explicitly address common failure modes identified in prior work (Cemri et al., 2025; Zhang et al., 2025f). These models show particular promise for improving collaborative behaviors MAS. We continue to optimize this functionality, with further discussion reserved for future work.
2.2.3 POLICY MODEL. TRAINING
Upon obtaining rollout experiences comprising individual trajectories and corresponding rewards for each agent, we initiate distributed training of agent policy models. The training leverages adapted
4

---

Published as a conference paper at ICLR 2026
implementations from OpenRLHF, supporting various reinforcement learning algorithms including REINFORCE+ (Hu, 2025), GRPO (Shao et al., 2024), and PPO (Schulman et al., 2017). Notably, all agent policies are trained using identical RL algorithms to maintain consistency.
Furthermore, we augment the training process by incorporating additional imitation learning strate- gies during on-policy rollouts. These include supervised fine-tuning (SFT) and direct preference optimization (DPO) (Rafailov et al., 2023), extending beyond OpenRLHF's native capabilities. This integration enables dynamic selection of training strategies tailored specific application requirements, such as stable training and faster convergence.
EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
Datasets. We utilize competition-level mathematical datasets for our experiments, including AIME24 (AI-MO, 2024a), AMC (AI-MO, 2024b), and MATH-500 (Lightman et al., 2024). All datasets are adapted from the publicly released DeepScaleR project materials
Models. For non-reasoning models, we use Qwen2 5B and Qwen2. 3--3--Innttaat (Yang et al., 2024). For reasoning models, we utilize Qwen3-1.75 (Team, 2025) and DeepScaleR-1- 5B-Preiiess 1 (Luo et al., 2025) for experiments. We also incorporate results for and OpenAI-ol- -minni..
DeepSeek-RR Ratekk-----we---111-
Inference Details. For multi-agent workflow inference, we use a temperature of 0.6 and top_p of 0.95 for all models. The max generation token is set to 8192 for non-reasoning models and 16384 for reasoning models. For reasoning models with outputs like "<think> reasoning </think><answee final answer<< /aasrrrr" agents exclusively exchange final answers without their intermediate thinking processes with each other.
Training Details. We employ the MARTT framework train both base and reasoning models, specifically 5 3B and DeepScaleR-1.. -B-Previe.. For Qwen2. 5B3 we imple- ment DeepSeek-RI zero-like reinforcement learning training using Level 3-5 samples from the MATH dataset (Hendrycks et al., 2021) like previous works (Zeng et al., 2025; Liu et al., 2025b). The DeepScalee- 5i--1e1B--rreieee model, which exhibits strong inherent reasoning capabilities but presents training challenges, undergoes test-time reinforcement learning (TTRL) (Zuo et al., 2025) adaptation on AIME benchmark data. We maintain the same maximum generation tokens and temperature settings as used during inference, while extending the maximum prompt token length to 8192. For multi-agent reinforcement learning, we employ a cluster configuration consisting of nodes, each equipped with 8 A800 80GB GPUs, allocating full node per agent.
We evaluate model performance using accuracy scores computed for all datasets with open-source scripts from Owen2. 5 5-Math3, Additionally, measure Pass@l 1 by averaging scores across multiple responses and compute Maje jeN (where = 4 or 6) under the same inference budget using multi aulti-agent reinforcement learning (RL). For conciseness, abbreviate multi-agents debate, mixture-of-agent and chain-of-agents as MAD, MoA, and CoA, respectively.
Evaluation Metrics.
3.2
MAIN RESULTS
We present comparative results for both non-reasoning and reasoning models across different training and inference configurations in Figure 2 (instruction models) and Figure (reasoning models), followed by multi-perspection analysis:
Failures of Multi-agent Workflow. Our experimental results demonstrate that both non-reasoning and reasoning models achieve superior performance through majority voting compared to multi-agen workflows under equivalent computational budgets. This observation aligns with existing literature documenting failures in LLM-based multi-agent systems (Cemri et al., 2025; Zhang et al., 2025ftc), which identifies two key limitations: (1) inability to adhere to role specifications, and (2) failure to effectively utilize inter-agent interaction information. We attribute these shortcomings to the
watpm//ageeic-pppjeeeeeett /eeppppeeeecceepscallll deepscaa 'https://gt thub. cwm /wwm//weniien M/ Qwwn2. 5-Math
5

---

Published as a conference paper at ICLR 2026
Untrained
RL Fine-luned
MARTI Fine-lured
Base and Models
50
Budgett +1
Budget =3
Buudet 4
Budget +
454
40.1
43.1
41.4
41.3
42.9
Accumity (%6
22.6
35,8
Oreng Eull preaegt!
Gevetz Madntrnn Poadh!
Ceeid presgh!
Oweit Solp Eorkar)
Cheec sial muse pokar!
Chevit Mot 2011
Govet Mingl!!
cheest tutiiii Mhigee
Gownt A-t Meai!!
Oirect a- Mot S-1
Oanit tel
Comnt S-B Bluget
Orenc &-d wwwehh Beggin
Cheec 8-t M460
Oomels lnsenan 8MD
Figure Average scores of Qwen2.5-3B base and instruct models under different budget and settings.
Untralned
Aured Reasoning
75.9
Y5.4
7E.1
76.5
75.3
71.1
71.1
AUME24 Aca.racy
Devdlatoi 18 PPssoa)
Qwedd (Poxsl))
Dayphnsel HB (e) md
dweck (RIGAN))
D to eMaidd)
Depdloaet 1-8 alI aug
Daspiaant 38 (teste)
Camal PB sog
Chap deale call MAD PND)
BassFiaknt 1.0 sanvie
Oreac 1.0 (Masde))
Desas 2B
Figure 3: Average scores of reasoning models under different budget and settings.
predominant single-agent training paradigm of current LLMs, which inherently lacks exposure multi-agent dynamics. These findings motivate our proposed MARTI framework, which implements MARL develop advanced reasoning capabilities through structured agent interactions.
MARTI Enhances Base Models Using Zero-like RL. We further investigate training base models with zero-like RL using MARTI. For the Qwen2. 5-3B model, compare standard RL training with MARTI-based training and observe that multi-agen systems trained with MARTI achieve higher performance upper bound than single-agent systems. Notably, multi-agent debate yields the best results under the same computational budget. Consistent with prior research, our results demonstrate that base models enhanced with reinforcement learning achieve comparable performance to instructed models. This finding further supports the established conclusion that RL primarily enhances a model's intrinsic pre-trained capabilities rather than imparting new knowledge. These findings suggest the need to explore novel reinforcement learning approaches that enhance individual model capabilities, such as through multi-agent interaction paradigms.
MARTI Enhances Large Reasoning Models. To explore the upper bound of multi- agent training, we apply test-time reinforcement learning (TTRL) to large reasoning models (DeepScaleR-1 5B- 5 ew). Our results demonstrate TTRL's effectiveness for LRMss, par-

---

Published as a conference paper at ICLR 2026
ticularly complex tasks. Notably, Multi-Agent Debates (MAD) achieve score of 66.7 AIME, significantly outperforming other same-cost configurations, including OpenAl-ol-min,
Multi-Agen RL Achieves Higher Performance Upper Bound Than Single-Agent Systemss. After analyzing the performance of both base models and large reasoning models trained with MARTI, we find that multi-agent RL consistently achieves higher performance upper bound than single-agent systems. This demonstrates that, under the same inference budget, reinforced multi-agent models attain superior benchmark scores compared their single-agent counterparts. Furthermore, these results suggest that reinforced multi-agent training can enhance advanced reasoning capabilities, presenting promising direction for future research in reasoning optimization.
Table 2:
Results for 1atta--..-3B-mmmm across various workflows and training configurations. Un- der equivalent inference budget, MARTI consistently outperforms both single-agent reinforcement learning and majority-vote baselines.
1lama-3.22-B-Innstit
AIME
AMC
MATH500
Avg
Single Agent (Pass@1)
3.3
12.4
32.2
16.0
+ RL
11.7
25.6
48.9
28.7
Single Agent (Maj@4)
6.6
18.1
36.6
20.4
+ RL
11.7
27.7
50.6
30.0
MAD 2x2
3.3
16.9
38.4
19.5
+ RL (MARTI)
13.3
29.5
53.6
32.1
MoA 3x1
6.6
16.9
37.2
20.2
+ RL (MARTI)
11.7
28.7
52.6
31.0
Table 3: Comparison of REINFORCE++ (RF++) and GRPO Qwen2.5-3B. Both algorithms
produce strong performance gains; GRPO achieves marginally better results on most evaluated metries.
Qwen2.5-3B
AIME
AMC
MATH500
Avg
Single-Agent RF++
10.0
36.1
66.7
37.6
Single-Agent GRPO
13.3
34.6
66.0
37.9
MAD 2x2 + RF++
16.7
49.4
70.8
45.6
MAD 2x2 + GRPO
16.7
50.0
71.2
46.0
Table 4: Ablation study on reward shaping for Qwen2.5-3B. Removing reward shaping results in substantial performance degradation for both MAD and MoA architectures.
Qwen2.5-3B
AIME
AMC
MATH500
Avg
MAD 2x2 w/ reward shaping
16.7
49.4
70.8
45.6
MAD 2x2 w/o reward shaping
6.6
36.6
66.7
36.6
MoA 3x1 w/ reward shaping
13.3
47.0
69.0
43.1
MoA 3x1 w/o reward shaping
10.0
38.9
65.4
38.1
3.3 ABLATION STUDIES
Different Model Families To examine whether MARTI generalizes beyond Qwen-based models, we apply both single-agent and multi-agent RL the 3lama-3.223B-Ine Instructt backbone. Table summarizes the results. Single-agent RL already brings large improvement over the supervised @ass@1 baseline (from 16.0 28.7 on average). On top this, applying MARTI to multi-agent workflows yields further gains: for example, MAD 2x2 with RL reaches average outperforming both single-agent RL (28.7) and majority-vot RL (30.0). MoA 3x1 1 with RL achieves similar average score of 3..0. These trends consistent across all three benchmarks, indicating that the benefits of MARTI are not tied to specific model architecture.
of 32.1,
Different Algorithms We further compare policy gradient estimators, REINFORCE++ (RF++) and GRPO, on Qwen2.5-3B. As shown in Table 3, both algorithms improve substantially over the supervised single-agent baseline, and GRPO provides slight but consistent edge. For instance, MAD
7

---

Published as a conference paper at ICLR 2026
0.42
0.70
MAD RL
0.40
Single
0.68
Single
0.38
0.66
Accuracy
0.36
0.64
0.34
0.62
0.32
0.60
0.30
0.58
0.28
0.56
25
125
150
175
100
125
Steps
Steps
(a) AMC
(b) MATH-500
Figure 4: Accuracy of MAD (Qwen2.5-3B) on AMC and MATH-500
Agent 1
Agent
Agent
Training Reward
Response Length
Total Length
2400
Training Reward
Response Length
900
Total Len gth
2250
840
780
1950
1800
80 Steps
Steps
Steps
Figure Training Dynamies of MAD (Qwen2.5-3B) with RL MATH.
2x2 with GRPO achieves an average score of 46.0, compared to 45.6 with RF++ These results indicate that MARTI is robust the specific choice of policy-gradient algorithm, and that the main gains arise from the multi-agent RL formulation itself.
Reward Shaping. MARTI employs a delta-style reward shaping mechanism that compares the current turn of an agent with its own historical performance, thereby rewarding relative improvements instead of only absolute correctness. Table 4 reports ablations on Qwen2.5-3B for MAD 2x2 and MoA 3x1 1. Removing reward shaping causes clear drop in average performance: from 45.6 to 36.6 for MAD and from 43.1 to 38.1 for MoA. This shows that reward shaping is essential for stabilizing multi-agent RL; purely outcome-based rewards multi-turn interaction, whereas our shaping provides smoother optimization signals that better align with collaborative reasoning quality.
prone to instability and reward hacking in
4 DISCUSSIONS
4.1 CASE STUDY: MULTI-AGENTS DEBATE
Experimental Setup. conduct multi-agent debate training using two model architec- tures: Qwen2 5-BB and DeepScaleR-1.. 1...-1..1--PP.. -B-.eieee The Qwen2.58 5-33 model is trained using REINFORCE+ on Level to samples from the MATH-500 dataset, while DeepScaleR-1. 110.-1..B--PP- 5B-Preview employs TTRL the AIME benchmark.
Training Dynamics. Model accuracy results are presented for both AMC and MATH-500 benchmarks in Figures 4a and 4b, respectively. Additionally, analyze the training dynamics of RL in Figure and TTRL optimization in Figure
architec- We present additional case studies analyzing training dynamics across various tures in Appendix C, including ofxture-of-AAentee (MoA) (Appendix C. Chain-of ofen---tsse (CoA) (Appendix C.2), and Judge-based Debate (Appendix C.3).
8

---

Published as a conference paper at ICLR 2026
Agent 1
Agent 2
Agent
Training Reward
Response Length
Total Length
Training Reward
Response Length
8700
Total Length
0.45
6600
8400
0.42
6400
8100
0.39
7800
Steps
Steps
Steps
Figure 6: Training Dynamics of MAD (DeepScaleR-II.D--Peeiiii with TTRL on AIME.
Table Statistics of asynchronous rollouts MARTI, (a) End-to-end rollout time vs. concurrency for Chain-of-Agents and MAD. (b) Total rollout time vs, number of interaction rounds for different concurrency settings.
(a) Time vs. concurrency (seconds)
Workflow
Syne
Asynex32
Asynex64
Asynex 128
Asynex256
Asyncx512
Chain
612.6
615.5
553.5
508.5
515.4
498.4
MAD
593.5
732.9
616.8
592.4
569.4
561.2
(b) Time vs. interaction rounds (seconds)
round
Sync
Asyne
Async
916.6
1074.1
887.5
2194.0
2186.1
2022.8
8
3308.0
3004.4
2928.0
4.2 STATISTICS OF ASYNCHRONOUS GENERATIONS
Previous rollouts in RL frameworks are typically performed batches for batch generation. However, this approach proves inefficient for multi-turn interactions, such multi-turn tool calls and multi- agent interactions, due to significant discrepancies in time costs during generation. As a result, asynchronous generation has become core feature in mainstream RL frameworks, particularly in agentic RL systems such as OpenRLHF (Hu et al., 2024a), veRL (Sheng et al., 2024), and AReaL (Fu al., 2025). To the best of our knowledge, MARTI is the first framework to support asynchronous generation for multi-turn, multi-agent scenarios.
We further analyze the effect of asynchronous rollouts in MARTL Table 5 summarizes the time costs for synchronous and asynchronous execution in both Chain-of- and Multi- Debate workflows under different levels of concurrency and interaction depth. With moderate concurrency, asynchronous rollouts consistently reduce end-to-end inference time, but very large numbers of parallel workers become compute bound and yield diminishing returns. When interaction depth is small (e.g., 2 rounds), trajectories are short and synchronization overhead is negligible, Async x 64 provides only a modest 3% speed-up over the synchronous baseline. As the number of rounds increases, rollout latency grows and the throughput advantage of asynchrony becomes more pro- nounced. Overall, asynchronous generation is most beneficial for deep, interaction heavy workflows, while shallow workflows primarily limited by raw compute rather than synchronization.
5 CONCLUSION
We present MARTI, unified framework integrating multi-agent reinforcement learning (RL) with inference for LLM-based systems. By combining scalable RL training (via OpenRLHF) with adap- tive multi-agent workflows, MARTI outperforms single-agent TTRL in reasoning tasks, achieving

---

Published as a conference paper at ICLR 2026
advanced performance AIME Challenges like reward modeling and real-world deployment persist, but MARTI advances MAS capabilities through built-in credit assignment and support for diverse reward models. Our work demonstrates that multi-agent RL elevates performance ceilings beyond single-agent approaches, offering pathway to enhance reasoning in practical applications. Future work will focus optimizing MAS training for broader adoption.
ETHICS STATEMENT
This work presents MARTI, a framework for LLM-based multi-agent reinforcement learning and inference. We use established public benchmarks to ensure transparent and unbiased evaluation, while minimizing computational waste through efficient configurations.
REPRODUCIBILITY STATEMENT
We provide comprehensive details to ensure reproducibility, including implementation specifics in Section 3.1 (models, inference details, training procedures, and evaluation metrics), Additionally, the anonymous MARTI codebase provided in https:/// //gg:/giitttt http://iog///ocCCCCC.... /MMRTT..
ACKNOWLEDGMENTS
This work supported by the National Science and Technology Major Project (2023DD0121403), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and National Natural Science Foundation China (No. 62406165). We thank anonymous reviewers for their insightful comments and suggestions.
REFERENCES
AI-MO. Aime 2024, 2024a.
URL https://hugiingaace.oo cot watae/seteaa/a/a//aaaaaaasas http:://hgggggg.e.e..oooooaaaaaa
aimo-validation dad.aaiiiiimm.....
AI-MO. Ame 2023, 2024b. aimo validation amc.
URL httpe://naggingaccc.c//tttttttt
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your I1m fully utilize the context. Advances in Neural Information Processing Systems, 37:62160-62188, 2024.
Antonis Antoniades, Albert Örwall, Kexun Zhang. Yuxi Xie, Anirudh Goyal, and William Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.2025 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: //arxiv. org/ abs 2000 14165.
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent Ilm systems fail?, 2025. URL https: //arxiv. org/abs// 250. 13657.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview www.:/eeettfteeeeeeee iiwtoruummi-Feeeppp d=FQepi sCUWu.
10

---

Published as a conference paper at ICLR 2026
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay. Börje Karlsson, Jie Fu, and Yemin Shi. Autoagents: a framework for automatic agent generation. In Proceedings of the Thirty-Thir International Joint Conference Artificial Intelligence, pp. 22-30, 2024a.
Pei Chen, Shuai Zhang, and Boran Han. CoMM: Collaborative multi-agent, multi-reasoning- path prompting for complex problem solving. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 7720-1738, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10. 12118663/11/2224..0 finni/4finnnigs--aaall findings-naaci 112/.
URL
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024c. URL https://cpenrewee
net/ forum? 2immmidd=iii SeeeummiddEEHSSSDyyA
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language 2023.
model generation. arXiv preprint arXiv:2311.1733
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint wrXiv:2502.01456, 2025.
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. arXiv preprint arXiv:2310.09520, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factu- ality and reasoning in language models through multiagent debate. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 11733-11633. PMLR, 21-27 Jul 2024. URL https: http://ppcceeennsseeeeee/ mlr prwssppsppeee..p....... v2 du24e. html.
Ahmed El- El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.068077 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint 2:XXi::5005.2228. 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411. 155594 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges, 2024. URL https:/ / arxia. org/ abs / 22022. 61080.
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Lim multi-agent systems: Challenges and open problems, 2024. URL https: / /axxiv.vvvooo./ao/aa////aaa org/abs/ 2402. 03578.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https::////... / openrevi ew net/ forum? d d=BBytt
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang. Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint 2rXivv2338..00:: 3(4)66, 2023.
Sirui Hong, Mingehen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang. Jinlin Wang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for multi-agent collaborative framework, 2024. URL https://arxiv. org/ abbs/330B 00352.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03260 2025.
Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang. Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and pigh-performance rlhf framework. arXiv preprint arXiv:2405. /113, 2024a.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: easy-to-use, scalable and high-performance rihf framework, 2024b. URL tittps://arxia brg/abs/2405 11143.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou, Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://opeereviiee
f d Id=IkkmEEKBFP
Dom Huh and Prasant Mohapatra. Multi-agent reinforcement learning: A comprehensive survey. arXiv preprint arXiv:2312.10256 2023.
Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou On large language models' hallucination with regard to known facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1041-1053, 2024.
Bowen Jin, Hansi Zeng. Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
junyou li, Qin Zhang, Yangbin Yu, QIANG FU, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //t///o///// //oppeeeeiew.eeeeeeee feet/foorine forum? wtt.iiddggggggeeeeeeeee
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: a arxiv. org/ abs 20000000..000 08361.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint avXiv:240.001502 2024.
Maxim Khanov, Jirayu Burapacheep. and Yixuan Li. Args: Alignment as swward-guieed search. arXiv preprint arXiv:2402.01694, 2024.
Guohao Li, Hasan Abed A1 Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: communicative agents for" mind" exploration of large language model society. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, pp. 51991-52008, 2023.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025.
12

---

Published as a conference paper at ICLR 2026
Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene le. Improving multi-agent debate with sparse communication topology. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp- 7281-7294, Miami, Florida, USA, November 2024. Asso- ciation for Computational Linguistics. doi: 12065/////0024...nnngs-----
URL
https: h www.anntooogg.oooooo.oo.o.. a.dd.......................-- emmlp 1p 427/.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang. Rui Wang. Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17889-17900 Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/v// 2024. 1124.emllp-miin 992. URL https: /aw.aaaaaanntoooooooaoaaooooooal org/ 2ttg/222 2224. emnlp haappmma........mmm 992/.
Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An open-source framework for building general ai agents, 2025. URL https:
[[doi..or//// ///////.og//...2////eeeeee
www.//22eenooo 15186407.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations (ICLR), 2024. URL https: opw.reveeeeee..eeeee net/ 6E0i.
forum?
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb I1m surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv.2502.06703, 2025a.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. arXiv preprint arXiv: 07,100151101 2019.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl -l-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025c. URL httpt:t///eatt / //txxi/.rxxx/o////// abs/ 2254. 02495.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0178: 2025d.
Sikai Lu, Yingfeng Cai, Ze Liu, Yubo Lian, Long Chen, and Hai Wang. A preference-based multi- agent federated reinforcement learning algorithm framework for trustworthy interactive urban autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o -preview with a 1.5b model by scaling rl. Notion Blog. 2025. Notion Blog.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self feedback. Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 12832, 2024.
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent I1m training. arXiv preprint arXiv:2412.01928.. 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 2021.
13

---

Published as a conference paper at ICLR 2026
Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, et al. Why do multiagent systems fail? In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025.
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv preprint arXiv:2502.184433 2025.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large language model- multi-agent collaboration. In International Conference on Learning Representations (ICLR), 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53777 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre- training, arXiv preprint urXiv:2504.04022 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint mxxi:2242.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 19256, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297, ACM, March 2025. doi: 00.1145/2688031336666777 URL http:/ /xx doi org /10 1145/ 3689033 3696075.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
Oliver Slumbers, David Henry Mguni, Kun Shao, and Jun Wang. Leveraging large language models for optimised coordination in textual multi-agent reinforcement learning. arXiv, 2023.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 4rXiv:2408.03314, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3022 2020.
Yunhao Tang, Taco Cohen, David Zhang. Michal Valko, and Rémi Munos. Rl-finetuning lims from on-and off-policy data with single algorithm. arXiv preprint 1xXiv:2503.19117 2025.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
Qwen Team. Qwen3, April 2025. URL
Raghav Thind, Youran Sun, Ling Liang, and Haizhao Yang. Optimai: Optimization from natural language using Imwppweeeed ai agents. arXiv preprint arXiv:2504 16918, 2025.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https: / //gi thub. www.wwwg..oo....oto.eo..o......t wtwee///..a///../oo///
2020.
14

---

Published as a conference paper at ICLR 2026
Ziyu Wan, Yunxiang Li, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu. and Ying Wen. Rema: Learning to meta-think for Ilms with multi-agent reinforcement learning. arXiv preprint arXiv:2503. 09500, 2025.
Junlin Wang. Jue WANG, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. In The Thirleenth International Conference on Learning Representations, 2025a. URL https: /ttpppppeeviieeeee net/ www.foorudddiiio iitrrmmmiddhh
Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. CharacterBox: Evaluating the role- -layyla capabilitiess of LLMs in text- based virtual worlds. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chupter of the Association for Compulational Linguisties: Human Language Technologies (Volume I: Long Pupers), pp. 6372-6391, Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics. ISBN 979-8-891787 19176-189-6 URL https: // //aclanthlloyyy / 2025 5. maaci-long 323/.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023a.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 6106-6133 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18////vll 1/2024.acl-ooog..33 URL https: / ac antthoothyyyyy wwg.oor//2222 acl- 331/-
Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang. Order matters: Agent-by-agent policy optimization. arXiv preprint axXi::2302.06205 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprini 2rXiv:2203.11177 2022.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang. Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlie,, et al. Ragen: Understanding self-evolution in 11m agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.2007,, 2025c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouugh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to veta-generation: Inference-time algorithms for large language models. arXiv preprint 2406::2300..68388 2024.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang. Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex instruction-folllwingg with multiple constraints composition. Advances in Neural Information Processing Systems, 37:137600-137645, 2024.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang. Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation, 2023a. URL
wtw./bs/////.oooooo.o....o....o. abs/ /2308. 08155.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 11m applications via multi-agent conversation. arXiv preprint arXiv:2308.081. 2023b.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint ar arXiv:2405.0044 2024.
15

---

Published as a conference paper at ICLR 2026
Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. Towards reasoning in large language models via multi-agent peer review collaboration. arXiv preprint arXhv:2311.08152. 2023.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- off-policy guidance. arXiv preprint arXiv:2504.14990 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.15115 2024.
Sen Yang, Yafu Li, Wai Lam, and Yu Cheng. Multi-llm collaborative search for complex problem solving. arXiv preprint arXiv:2502. 18873, 2025.
Yaodong Yang. Many-agent reinforcement learning. PhD thesis, UCL (University College London), 2021.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR), 2023.
Hai Ye, Mingbao Lin, Hwee Tou Ng. and Shuicheng Yan. Multi-agent sampling: Scaling infer- ence compute for data synthesis with tree search-based agentic collaboration. arXiv preprint arXiv:2412.170611 2024.
Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. trehange-of-thougeee Enhancing large language model capabilities through cross-model communication. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. December 2023. Association for Computational Linguistics. doi: 10. 26653/1/2222..coomo/ eemn-m-mannlnnnnn 936. URL hwtps: / /aaaa/aaaanttooooooooooo 0r2/2023. emnlp- ma in. 936/.
Singapore,
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01991, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in ilms beyond the base model? arXiv preprint ar5iv:2504. 13837, 2025a.
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning route 1lms for multi-agent systems, 2025b. URL huttps://aree htgg/bb//ae /t/aaabs//22/ 2502. 11133.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025, aaaiiviiii org/ abs 2503. 18892.
URL htth::/::
Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang. Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. Evoflow: Evolving diverse agentic workflows the fly. arXiv preprint arXiv:2502.07373.
2025a.
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet. arXiv preprint 2rXiv:2502.04// 180, 2025b.
Hangfan Zhang, Zhiyao Cui, Xinrun Wang, Qiaosheng Zhang, Zhen Wang. Dinghao Wu, and Shuyue Hu. If multi ageet debate is the answer, what is the question? arXiv preprint arXiv:2502.08788, 2025c.
16

---

Published as a conference paper at ICLR 2026
Kaiyan Zhang, Jiayuan Zhang, Haoxin Li, Xuekai Zhu, Ermo Hua, Xingtai Lv, Ning Ding, Biqing Qi, and Bowen Zhou. Openprm: Building open-domain process-bae based reward models with preference trees. In The Thirteenth International Conference on Learning Representattins, 2025d.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In International Conference on Learning Representations (ICLR), 2025e. URL https: /ttptp:::///peeeeeeee http///pppeeeeiieeeeeeeeeeeeeeet net forummid- Cowp4tFEtE.
Shao Zhang, Xihuai Wang, Wenhao Zhang, Yongshan Chen, Landi Gao, Dakuo Wang. Weinan Zhang, Xinbing Wang, and Ying Wen. Mutual theory of mind in human-ai collaboration: An empirical study with llm-driven ai agents in real-time shared workspace task. arXiv preprint arXiv:2409.08811 2024a.
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang. Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of Ilm multi-agent systems. arXiv preprint arXiv:2505.00212 2025f.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination large language models. arXiv preprint
2023.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances in Neural Information Processing Systems, 37:132208-132233 2024b.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504. 00891, 2025.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4665 2023.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160 2025.
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothnee Haohan Wang, and Yu- Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406 2023.
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. R1- zero's" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint a:Xiv22503.05132, 2025.
Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgenn Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference
Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62743-62767. PMLR, 21-27 Jul 2024. URL mlr. wwwpppeeeeeeeeeee v235/s//v33///22 2ttp55/hhge22a..aam / htmm..
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuckai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning, 2025. URL https: / wwwpviiorrr/aaaee.ee abwgg//b///ae 4504. 6084.
17

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A.1 MULTI-AGENT LLM SYSTEMS
Previous researches explore multi-agent L.MM workflows, focusing on multi-agent debate (Du et al., 2024; Liang et al., 2024; Xu et al., 2023; Yin et al., 2023; Wang et al., 2024; Chen et al., 2024b; Zhang et al., 2025c), communication topology (Chan et al., 2024; Chen et al., 2024c; Zhuge et al., 2024; Qian et al., 2025; Li et al., 2024; Yue et al., 2025b; Zhang et al., 2025b;a), and test-time scaling (junyou li et al., 2024; Wang et al., 2025a; Antoniades et al., 2024; Ye et al., 2024; Yang et al., 2025). These works demonstrate the potential of multi-agent systems to enhance collaborative problem-solving and scalability, effectively managing complex interactions across diverse tasks.
Multi-agent frameworks improve collaborative task-solving effectively (Li et al., 2023; Chen et al., 2024a; Liang et al., 2025). CAMEL (Li et al., 2023) uses a role-playing framework where user agent decomposes tasks and assistant agent executes them, guided by inception prompt. MetaGPT (Hong et al., 2024) simulates the collaboration a software company by assigning distinct roles for handling complex tasks. However, fixed agent roles restricts the adaptability of multi-agent frameworks. To address this, AutoAgents (Chen et al., 2024a) dynamically generates specialized agents and coordinates them through central planning module for complex tasks. AutoGen (Wu et al., 2023a) focuses on developing LLM applications through layered and extensible multi-agent design. Similarly, OpenManus (Liang al., 2025) provides modular framework with agents, flows, prompts, and tools, adopting a tool-centric ReAct (Yao et al., 2023) paradigm to support plan-then-act Docision-making effectively handling tasks requiring extended reasoning.
Despite the wide variety of existing multi-agent LLM systems, significant performance gains at the emergent level remain elusive (Liu et al., 2019; Chen et al., 2024c). In some tasks, multi- ageet frameworks exhibit only marginal gains over single-agent approaches (Pan et al., 2025). These limitations often stem from the inherent constraints of single LLM agents. When handling context, a single agent may fail to follow task or role instructions (Wen et al., 2024; Wang et al., 2025b), or lose focus long-context scenarios (Zhang et al., 2024b; An et al., 2024). Additionally, the model itself may generate outputs with factual hallucinations misinterpret contextual cues (Zhang et al., 2023; Jiang et al., 2024). On the other hand, the design of the workflow and inter-agent coordination mechanisms often plays a critical role in MAS failures. Common issues include overly complex system design (Kapoor et al., 2024), disorganized memory management (Han et al., 2024), and failure of verify-refine mechanisms (Huang et al., 2024). Our proposed MARTI framework provides a platform for testing, observing, and mitigating such failures through training.
A.2 REINFORCEMENT LEARNING FOR LLMS
Test-time scaling (TTS) is designed to enhance the capabilities of LLMs in handling complex tasks by increasing computational resources at the time of testing. Prior research (Snell et al., 2024; Liu et al., 2025a) indicates that TTS is more efficient than scaling during pre-training (Kaplan et al., 2020); thus, reallocating the same computational resources from pre-training to test-time could yield greater improvements in model performance. Current studies on TTS fall into two categories (Welleck et al., 2024): parallel generation and sequential generation. Parallel generation entails LLMs producing multiple candidate responses (self-consistency (Wang et al., 2022; Chen al., 2023), best-of-N (Stiennon et al., 2020; Nakano et al., 2021)), decision steps (Monte Carlo Tree Search (Zhou et al., 2023; Xie et al., 2024)), or tokens (Reward-guidee Khanov al., 2024)) during inference. Subsequently, an aggregation strategy is applied to integrate these candidates. commonly utilizing process reward models (Lightman et al., 2024; Wang et al., 2023a; Zhang et al., 2025d). Concurrently, sequential generation focuses on extending the L.MSs' output to include longer responses with reflective and chain-of-thouu processes (Wei et al., 2022; Madaan et al., 2023). Although prompting techniques are widely adopted, they are often constrained by the capabilities of the underlying models. Notably, DeepSeck-RI (Guo et al., 2025) represents a significant advancement in this area, achieving extended reasoning capabilities in pre-trained language models through outcome based RL, like group relative policy optimization (Shao et al., 2024). Compared to the first approach, which requires intensive process- supervision (Yuan et al., 2024), the second approach is more scalable due to its reliance rule-based rewards.
Search (Deng & Raffel, 2023;
18

---

Published as a conference paper at ICLR 2026
A.3 MULTI-AGENT REINFORCEMENT LEARNING
Multi-agen, auglt reinforcement learning has emerged as powerful framework for modeling strategic interactions, guided by game-theoretic principles that shape both learning dynamics and reasoning processes (Yang, 2021; Huh & Mohapatra, 2023). Recent research has focused on addressing its unique challenges such as non-stationarity, credit assignment, and scalability. Wang et al. (2023b) introduce sequential agent-wise update scheme with off-policy correction, ensuring monotonic improvement and enhancing performance in cooperative tasks. Slumbers et al. (2023) leverage shared policies, centralized training. and natural language communication to enhance performance in text-based environments. Zhang et al. (2024a) shows that LLM-driven agents with Theory of Mind improve perceived coordination in human-Al teams, though bidirectional communication performance. Wan et al. (2025) separates meta-thinking and reasoning into distinct agents, achieving improved generalization and performance on complex reasoning tasks. Park et al. (2025) jointly trains multiple LLMs via inference-awar rewards to foster effecctve, transferable collaboration in multi-turn tasks. Lu al. (2025) proposes integrates rule-based models and human preference signals in urban autonomous driving scenarios. Thind et al. (2025) translates natural language optimization problems into executable solvers through role-specialized agents.
hinder
enferrnce-ggider multi-agent federated framework that
B WORKFLOWS
B.1 WORKFLOW CODE
Listing 1: The pseudo-code of the Abstract Workflow.
asyne def workflow(
prompt: str, label: str,
agents: List /str, Anyll, tool_ manager,
task: str,
metadata: Optional [Dict] Nene, **kwargs
->
Dict[sta Any]:
Custemized Interactions
trajectory
13
14
"turn_id"; 0, pagent_index" 0,
"agent_name": ""agent",, "agent_role"; "generator", "agent_input": "input_example", "agent_output"" "output__xample", "metadata";
18
1
22
Add more turns
23
]
24
rewards = [0]
Add reward for each turn if exist
27
return
28
"prompt": prompt,
"Label": label,
"trajectory": trajectory,
31
"tinal reward"; rewards|-1
32
19

---

Published as a conference paper at ICLR 2026
Agent
Agent
P
[Rou Roo Red Ra
Verifier
Agent
Agaii Aggreaae (Dst) (Riz Ruy Rai + Reval
Prompt
Prompl
Agent
Agant
Agant
R
Agg
R
Agent
Agent
Agent
12
Rria
Agont Solvor P + Ru.
Feodbadk
Agent fiwliww Rs (t)) Rt
Verifier
Verifler
Verifler
Verifler
Trgestory: + R2 + R2 R4
Variflir
Varifler
Prompt
Agent
Agent
Agent
Verifier
Agent
R
Apent
Prompt
Agent
R
Agent
R
Agent
R
23
Trajectory P: R., + Ry.
+ Riz
Riy
Ri,
Ri
Agent 2
R
Trajectory 2:P Riz Ru
Figure MAS Examples for Typical Multi-Agent Workflows.
0.68
Single
Single RL
0.36
MoA RL
0.66
MoA RL
0.34
Accuracy
0.64
0.62
0.32
0.60
0.30
0.58
0,28
Steps
Steps
(a) AMC
(b) MATH-500
Figure 8: Accuracy of MoA (Qwen2.5-3B) on AMC and MATH-500
B.2 WORKFLOW EXAMPLE
We introduce and compare several workflows in Figure 7, including mixture-of-agents and chain-of- agents. For the chain-of-agents workflow, two typical credit assignment strategies considered: (1) assigning verifiable rewards at each turn or (2) assigning the final reward at the end, with the final reward distributed across the intermediate turns. These workflows are fully supported the MARTI framework for further experimentation.
B.3 CODE EXAMPLE
A full example for MathChat with three agents is provided in Figure 2.
C CASE STUDY
C.1 CASE MIXTURE-OF-AGENT
Experimental Setup. We evaluate mixture-of-agents approach using the Qwen2. model, trained on Levels 3 through the MATH-500 training dataset.
Training Dynamics. The model's accuracy results presented for both AMC and MATH-500 benchmarks in Figures 8a and 8b, respectively. Furthermore, we analyze the complete training dynamics in Figure 9, including training rewards, response length, and total length.
20

---

Published as a conference paper at ICLR 2026
Agent
Agent
Agent (Aggregator)
Training Reward
Response Length
Total Length
2400
800
Response Length
Training Reward
Total Length
0.6
600
1600
0.4
500
0.2
400
800
0.0
300
300
Steps
Steps
Steps
Figure 9: Training Dynamics of MoA (Qwen2.5-3B) with RL on MATH.
CoA reward shaping
CoA w/o reward shaping Agent (Verifier)
Agent 1 (Generator)
Agent 3 (Refiner)
0.75
0.8
Training Reward
0.50
Training Reward
Training Reward
0.6
0.50
0.25
0.4
0.25
0.00
-0.25
0.0
-0.25
50
25
Steps
Steps
Steps
Figure 10: Training Rewards of CoA (Qwen2.5-3BB with RI. on MATH.
CoA reward shaping
CoA w/o reward shaping
Agent (Generator)
Agent 2 (Verifier)
Agent 3 (Refiner)
880
Response Length
Response Length
800
Response Length
800
720
600
720
640
400
640
560
200
560
75
75
Steps
Steps
Steps
Figure 11: Training Response Length of CoA (Qwen2.5-3B) with RL on MATH.
C.2 CASE 2: CHAIN-OP-AGENTS
Experimental Setup. We investigate chain-of-agents reinforcement learning (RI.) using Levels 3-5 of the MATH-500 training set. Our evaluation compares standard RL training with a quality-aware reward shaping variant to assess performance improvements.
Training Dynamics. The training process characterized by three key metries:
Training reward curve in Figure 10.
Response length dynamics in Figure 11.
21

---

Published as a conference paper at ICLR 2026
Eval Reward
Response Length
Total Length
0.75
900
Response Length
Eval Reward
0.60
750
Total Length
900
0.45
750
450
0.30
600
300
20 Steps
30
10
20
Steps
Steps
Figure 12: Training Dynamics of Judge-based Two-player Debate.
Table 6: Average number of output tokens instance on Qwen2.5-3B across tasks and workflows. Multi-agent workflows operate under a comparable token budget Majority @ and can even be more efficient.
Output Tokens
AIME
AMC
MATH500
Avg
Single-Agent (Avg@44
4532
3706
2322
3520
MAD 2x2
2698
4268
2698
3221
MoA 3x1
3083
2146
1518
2249
C.3 CASE 3: JUDGE-BASED DEBATE
Experimental Setup. We conduct LLM training using GenRM GenRM-generated feedback under debate setting 4, Specifically, we use the round-robin tournament data from and train 1.ama-3.2-11 Instruct model debate against a Llama-3.1 1imaa3.--8B-Inttruu model. The reward is the win rate in the round-robin tournament judged by Qwen2.5- Instrutt model. Training Dynamics. We present the evaluation rewards, response length, and total trajectory length during training in Figure 12.
D THE USE OF LARGE LANGUAGE MODELS
All core research ideas, theoretical derivations, experimental designs, and algorithmic innovations were developed by the authors without LLM assistance. Additionally, all paragraphs in the paper were originally written by humans. LLMs were used solely to fix bugs in the MARTI framework, under human review, and to polish sections of the paper.
E COMPUTE ACCOUNTING AND INFERENCE BUDGET
Unified Definition of Inference Budget. To ensure fair comparisons between single- and multi- agent workflows, we standardize the inference budget terms of the number of model rollouts under fixed sampling configuration (temperature, top-p, maximum length, etc.). Concretely, single-agent Majority @4 evaluation and MAD 2x2 session both generate four trajectories and therefore consume equivalent rollout budget. Similarly, MoA 3x1 produces three trajectories plus one final aggregation step, which in practice is comparable to Majority @4 in terms of compute.
We also report untrained multi-agent baselines (with the backbone kept frozen) disentangle the benefit of test-time compute from that of learned collaboration. These baselines only change the interaction topology (c.g.. voting, MAD, MoA) while keeping the rollout budget fixed, and thus highlight that the gains of MARTI mainly come from reinforcement learning on collaborative behaviours rather than simply sampling more trajectories
22

---

Published as a conference paper at ICLR 2026
Token-Level Statisties Across Workflows. In addition to rollout counts, number output tokens per instance for different workflows on Qwen2.5-3B. Table 6 summarizes the results. MAD 2x2 consumes similar number of tokens to the single-agent Majority @4 baseline (within roughly 10% on average), while MoA 3x1 1 is even more token-efficient. This confirms that the accuracy gains reported in the main paper do not arise from substantially increased generation cost.
the average
Listing The pseudo-code of MathChat workflow.
from typing ng import Dict, sist, Any
asynе def workflow(
prompt: str,
label: str, agents: List (Dict Anyll,
7
tool_manager: Any,
task: str, **kwargs
Dict Any]: =**
11
12
Orchestrates an asynchronous multi-agent workflow and collects data for training.
This example defines three-step interaction:
15
1. 'generator' agent proposes a solution.
16
2. A 'coder' agent implements the solution in code, which is then executed.
3. 'refiner' agent verifies all outputs to provide final answer.
18
19
The collected 'trajectory' retains all inputs, outputs, and rewards, forming a complete data sample for reinforcement learning.
20
Initialize workflow and identify agents by their predefined roles
23
trajectory [] generator_agent, coder_agent, refiner_agent = agents(0), agents(i], agents
24
25
28
Turn 1: Generator proposes a solution
27
generator_input = f"Problem: ( hrap:mm___ppmppp/hheeeeeeeeeee reata/aeoeeseeeess_______ becasonnstee_Beee bts_s_tt_step...._________
generator_response await generator_agent ["1lm]] generate_assne. remote(
29
generator_input, generator_agent www.piiiinnnnannnnnnnm.........a paramss
30
31
gencrator_output = gencrator_rcgposse.uttooo [0] .tet
trajectory ap
33
"agent_role"; "generator", "agent_input", generator_input, agent_output": generator_outpic
34
1)
Turn 2: Coder writes and executes code based the generator's solution
37
coder input = ""Proolem; htsp:a/sttttooeeeettttttttt bet.seerrttppuuuu...eeeeetatoooe nWrite Python, code...code..."
coder_response await coder_agent ["11m") [eeean")sseeerrrtaaasssooooe coder_input, coder_agent ["saas_ig_nnng__rramms
40
41
coder_cutpup =
cutputs [0] .text
Use the tool manager execute the generated code bode_to__xxectee extract_code (codderoutpput) exccution_result, = await toca_
44
45
"code_ incee_iiter,,,eee"", ("code"; wewp_aaaaaeerreecuteeeeoooee execute_toolf
46
23

---

Published as a conference paper at ICLR 2026
trajectory append "agent_role"; "coder", "agent_input"; coder_input, "agent_output" coder_output, "motadata"; ["tool_output"; exccution_result; })
31
Turn 3: Refiner verifies all outputs to produce a final answer
55
axtiner_input LLVEroleeer.ppoooppp//sssseeeee generator_outpute
f [icodd/utrpustffeeeeeoooooorrree
57
refi refiner_response - await (agnee___ggppnnnnnnneeeeeeeeeee rentee
refiner input, refiner refiner_agentI/ agent (sampllnnnn [araiinn__ppa_]]]___________
refiner outppt refiner_ heapeeerrresscsseeuueee... [0] textt aopoctory.appendee
61
"agent_role"; "refiner", "agent_input"; refiner_input, agent_output": refiner_output
})
64
65
# 2. Evaluate the completed trajectory to assign rewards for RL training
all_ _ut_cus "aue""""ett_uupuut"
output"| for turn in trajectory] all rewards auto verity (task, all outputs, [Label] len( all_outputs))
67
68
for turn, reward in zip(trejy,,,ry, all_rewards): tural"agent_rewarc" reward
71
72
3. Return the structured data sample in the required format return
73
"prompt"; prompt, "label": label,
75
76
"trajectory"; trajectory, "final_reward"" all_rewards[-]
77
24


Reviews:
[{"review_id": "BackKvySgy", "reviewer": "Reviewer_U2kR", "summary": "MARTI is an open-source framework for scalable multi-agent LLM systems, integrating centralized interaction with distributed policy training to enhance reasoning via reinforcement learning. It introduces dynamic workflows with both rule-based and generative reward models, improving agent collaboration efficiency. Experimental results demonstrate that MARTI-trained multi-agent systems outperform single-agent baselines under equivalent inference budgets.", "questions": "Although the algorithm’s performance was validated on competitive mathematics problems (e.g., AIME, AMC), can the MARTI framework maintain comparable efficacy in diverse, non-mathematical testing scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "MARTI enables efficient, large-scale multi-agent learning through centralized interaction and distributed policy training.\nCombines rule-based and generative reward models to enhance reward accuracy and adaptability.\nMulti-agent systems trained with MARTI outperform single-agent baselines in reasoning tasks under equivalent resource constraints.", "weaknesses": "MARTI’s performance may degrade on tasks requiring unseen reasoning patterns, as pre-trained LLM capabilities still set an upper bound on performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "061e3hpGV7", "reviewer": "Reviewer_3yGA", "summary": "This work presents a framework for training \"multi-agent\" LLM systems with reinforcement learning. In the context of math word problems, the proposed RL framework outperforms single-agent RL and inference-only multi-agent systems.", "questions": "Why does the response length significantly increase for 2 agents and decrease for one agent over the course of training in figure 5?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "- The results confirm that the proposed method increases performance beyond that of baselines, whereas many multi-agent systems underperform majority voting", "weaknesses": "- This work only studies math \"reasoning\" problems instead of natively multi-agent LLM settings.\n  - Examples of multi-agent LLM settings would include zero-sum games in SPIRAL (SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning), social deduction games (Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning), or purely cooperative games (Ad-Hoc Human-AI Coordination Challenge)\n- Only Qwen-based models are trained and tested in this work. However, these models behave very differently from other base models (like Llama), especially in terms of their RL performance, making it hard to determine if the findings are generalizable to LLMs in general.\n- The results from \"asynchronous generations\" are underwhelming, underperforming synchronous behavior at a concurrency of 32 and having an unimpressive improvement for larger concurrency values.\n- There is no ablation over the reward shaping terms or novel components of MARTI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8K08TKSAJV", "reviewer": "Reviewer_Nbtr", "summary": "The paper introduces MARTI (Multi-Agent Reinforced Training and Inference), a framework that unifies multi-agent LLM inference workflows (Mixture-of-Agents, Chain-of-Agents, Multi-Agent Debate) with distributed RL training and centralized reward/credit assignment. It supports asynchronous multi-turn rollouts, rule-based verifiable rewards for math tasks, and LLM-based generative reward models, then trains per-agent policies with PPO/GRPO/REINFORCE++ (optionally mixing in SFT/DPO). Experiments on AIME24/AMC/MATH-500 claim that, under equal inference budgets, multi-agent RL reaches a higher performance ceiling than single-agent RL.", "questions": "Please see the weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s key strength lies in providing a unified and scalable framework that bridges multi-agent LLM inference workflows with distributed RL training, filling a clear tooling gap. The modular design (multi-agent world, centralized rewards, agent-policy trainer) supports asynchronous rollouts, flexible agent compositions (MoA/CoA/MAD), and both verifiable and generative reward models, making it practical for real multi-agent RLHF experimentation. The reward shaping approach is intuitive and history-aware, improving consistency in multi-turn reasoning. Empirically, the framework demonstrates that multi-agent RL can achieve a higher ceiling than single-agent RL under matched inference budgets, with strong gains on math benchmarks (e.g., AIME) and evidence of good scaling behavior with concurrency.", "weaknesses": "1. The paper states that multi-agent settings are compared under “equivalent inference budgets,” but the budget is not formally defined (e.g., tokens, rollouts, GPU hours, or wall-clock time). A clear definition and a standardized reporting format would strengthen the fairness of comparisons.\n\n2. Although substantial compute is implied (e.g., 8×A800 GPUs per agent node), the manuscript does not report training time or convergence duration. Providing training wall-clock time, GPU-hours, and throughput would help readers assess scalability and practicality.\n\n3. The repository includes multiple RL algorithms, yet only REINFORCE++ results are shown. Including results for other supported algorithms (e.g., PPO/GRPO) would help confirm that the observed improvements are not algorithm-specific.\n\n4. Experiments are conducted only with Qwen-based models. Adding results on another model family (e.g., LLaMA or Phi) would improve confidence in the framework’s generality.\n\n5. Experimental figures lack narrative context.\nSeveral plots are presented without a clear accompanying explanation of the experimental question and takeaway. Adding brief guiding paragraphs before each experiment would greatly improve clarity.\n\n6. When running the released code, the MA-MAD results appeared noticeably lower than the reported values. This may be due to configuration differences; confirming the default settings and expected performance ranges in the repo would be helpful.\n\nThis work provides a very valuable and well-designed repository. I have tested multiple components and found the implementation largely correct and practical for multi-agent RL research. Addressing the points above — especially clarifying budgets, reporting compute costs, expanding baselines, and improving result presentation — would further strengthen the contribution. With these refinements, I would be pleased to give a higher score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWIJppnEbh", "reviewer": "Reviewer_JbEm", "summary": "The paper proposes MARTI (Multi-Agent Reinforced Training and Inference), an open-source framework that unifies reinforcement learning (RL) and multi-agent systems for large language models (LLMs). MARTI integrates centralized multi-agent interaction and distributed policy training, supporting asynchronous rollouts and both rule-based and generative reward models to improve training efficiency and scalability. MARTI also introduces modular reward shaping and credit assignment strategies to enhance long-term collaboration and stability during training. Experiments on mathematical reasoning benchmarks (AIME, AMC, MATH-500) show that multi-agent systems trained with MARTI outperform single-agent models under the same inference budget.", "questions": "Please refer to weaknesses.\n\nDiscussion (won't affect the rating):\n1. I'm curious whether the credit assignment in MARTI could be implemented using a counterfactual baseline proposed in COMA [1], and how this modification might influence the training stability, interpretability, and overall performance of the multi-agent reinforcement learning process.\n\n[1] Foerster, Jakob, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. \"Counterfactual multi-agent policy gradients.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The framework provides a well-structured MARL formulation that combines centralized multi-agent interaction with distributed policy training, ensuring scalability and modeling consistency.\n2. The introduction of a credit assignment mechanism reasonably decomposes global rewards, allowing more stable and interpretable multi-agent learning.\n3. The inclusion of off-policy training enables the reuse of historical samples, improving sample efficiency and training stability.\n4. The presentation of this paper is clear.", "weaknesses": "1. The paper lacks ablation studies on key components such as credit assignment or off-policy training, making it unclear how these mechanisms individually contribute to performance upper bounds or optimization efficiency.\n2. Figure 1 presents the overall MARTI framework but remains too abstract, lacking detailed illustration of key components (credit assignment, reward shaping, and the rule-based verifier mechanisms).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YltmdedEb9", "reviewer": "Reviewer_Pa5A", "summary": "MARTI is a framework for multi-agent LLM training and inference, featuring centralised coordination, distributed policy learning, and asynchronous rollouts. It combines rule-based and LLM-generated rewards to optimise multi-agent collaboration. Authors claim that MARTI is achieving increased performance over single-agent systems on mathematical and reasoning tasks.", "questions": "Please see \"Weaknesses\" section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-", "weaknesses": "- pseudo-code in Listing 1 does not provide value or helps understanding\n- Rule- and Generative Reward Shaping seems to be quite important, but is not part of the main contribution and does not show up in experiments, therefore it seems unnecessary.\n- Untrained results of models that have not been RL Fine-tuned or MARTI Fine-tuned seem to be unnecessary and blur the picture\n- Not showing MARTI Fine-tuned for Budget 1 and 6 in \"Figure 2: Average scores of Qwen2.5-3B base and instruct models under different budget and settings.\" seems cherry-picked\n- Same goes for \"Figure 3: Average scores of reasoning models under different budget and settings.\" - no mention of MARTI Fine-tuned for Budget = 1, 3, 4\n- There is quite some focus on Multi-Agent Debate (MAD) results, while the discussion and impact on the main hypothesis seems minimal\n- The authors state \"comprehensive experiments\" in the Abstract; however, only a few benchmarks have been evaluated (AMC, AIME, MATH)\n\n-> What I would have expected in the results section is the following experiments and metrics:\n- Generally you should aim for 3-10 training runs and report average performances for this to be significant\n- Single-Agent as a baseline is fine, but need to look at time/steps to convergence and final performance\n- But mainly you should compare MARTI vs other MARL setups without the features and contributing advancements so we can see the effectiveness of such. For example MARTI asynchronous VS MARTI synchronous etc. or a collaborative setup VS non-collaborative setup", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M2SPqYVTfK", "reviewer": "Reviewer_HGkq", "summary": "The paper introduces MARTI (Multi-Agent Reinforced Training and Inference), a unified and open-source framework that integrates multi-agent reinforcement learning (MARL) with LLM-based reasoning systems. The framework bridges the gap between multi-agent interaction (e.g., debate, chain-of-agents, mixture-of-agents) and reinforcement training by combining centralized reward modeling and distributed policy optimization. MARTI supports both rule-based verifiable rewards and LLM-based generative reward models, enabling flexible integration of verifiable and open-domain reasoning tasks. Experiments on several mathematical reasoning datasets (AIME24, AMC, MATH-500) show that multi-agent systems trained with MARTI achieve higher performance ceilings than single-agent baselines under equal inference budgets. The work provides a valuable infrastructure for scaling reasoning through multi-agent coordination and RL-based training", "questions": "- Could you clarify how MARTI handles credit assignment when generative reward models introduce noisy or inconsistent feedback across agents?\n\n- Have you tested MARTI on non-mathematical reasoning or tool-use tasks, to validate that the improvements generalize beyond verifiable domains?\n\n- How does scalability behave when increasing the number of agents or asynchronous rollouts—does training stability or throughput degrade?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper stands out for its conceptual clarity and systems-level contribution. It proposes the first general framework that unifies multi-agent inference with reinforcement-based training, addressing an increasingly relevant problem at the intersection of MARL and LLM reasoning. The design of centralized reward modeling with distributed training is elegant and pragmatic, allowing various workflows (debate, mixture-of-agents, chain-of-agents) to plug into the same infrastructure. The authors also provide strong experimental validation with detailed comparisons and visualizations, showing tangible improvements in reasoning benchmarks. The open-source commitment adds further value, making this paper a meaningful step toward reproducible MAS + LLM research. Overall, the work is technically sound, well-positioned in current literature, and likely to have significant community impact.", "weaknesses": "While the framework is impressive, the evaluation scope remains narrow, focusing primarily on math reasoning benchmarks. It would be helpful to see more open-domain or tool-use tasks to demonstrate generality beyond symbolic reasoning. The paper could also discuss stability, scalability, and cost-efficiency—for example, how well MARTI handles growing agent populations or asynchronous rollouts at scale. Additionally, while reward modeling is a core innovation, the comparison between rule-based and generative rewards is underexplored; more ablations or sensitivity analyses would make the contribution stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)