CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
Authors: Xiangyuan Xue
Yifan Zhou
Guibin Zhang
Zaibin Zhang
Yijiang Li
Chen Zhang
Zhenfei Yin
Philip Torr
Wanli Ouyang
LEI BAI
Venue: ICLR 2026 Poster (2026)
Keywords: LLM-based Agent
Multi-agent System
Primary area: foundation or frontier models, including LLMs

Abstract: Self-evolution is a central research topic in enabling large language model (LLM)-based agents to continually improve their capabilities after pretraining. Recent research has witnessed a transition from reinforcement learning (RL)-free to RL-based methods. Current RL-based methods either rely on dense external reward signals or extract intrinsic reward signals from LLMs themselves. However, these approaches diverge from the self-evolution mechanisms observed in human intelligence, where individuals learn and improve through mutual discussion and collaboration. In this work, we introduce Co-Evolving Multi-Agent Systems (CoMAS), a novel framework that enables agents to improve autonomously by learning from inter-agent interactions without external supervision. CoMAS generates intrinsic rewards from rich discussion dynamics, employs an LLM-as-a-judge mechanism to formulate these rewards, and optimizes each agent's policy through RL, thereby enabling decentralized and scalable co-evolution. Experimental results demonstrate that CoMAS consistently outperforms untrained agents and achieves state-of-the-art performance across most evaluation settings. Ablation studies confirm the necessity of interaction-based reward signals and reveal promising scalability as the number and diversity of agents increase. These findings establish CoMAS as a novel and effective paradigm for self-evolution in LLM-based agents.

Published as a conference paper at ICLR 2026
COMAS: CO-EVOLVING MULTI-AGENT SYSTEMS VIA INTERACTION REWARDS
Xiangyuan Xue!
Yifan Zhou
Guibin Zhang*
Zaibin Zhang 5.6
Yijiang Li? Lei Bai 2
Chen Zhang²
Zhenfei Yin 6
Philip Torr
Wanli Ouyang! 1,2,8
The Chinese University of Hong Kong Shanghai Artificial Intelligence Laboratory
University of Georgia *National University of Singapore
*Dalian University of Technology University of Oxford
University of California San Diego SShenzhen Loop Area Institute
ABSTRACT
Self-evolution is central research topic in enabling large language model (LLM)- based agents to continually improve their capabilities after pretraining. Recent research has witnessed a transition from reinforcement learning (RL)-free RL- based methods. Current RL-based methods either rely on dense external reward signals or extract intrinsic reward signals from LLMs themselves. However, these approaches diverge from the self-evolution mechanisms observed in human intelli- gence, where individuals learn and improve through mutual discussion and collab- oration. In this work, we introduce Co-Evolving Multi-Agen Systems (CoMAS), novel framework that enables agents to improve autonomously by learning from inter-agent interactions without external supervision. CoMAS generates intrinsic rewards from rich discussion dynamics, employs an LLM as---ss--judge mechanism to formulate these rewards, and optimizes each agent's policy through RL., thereby enabling decentralized and scalable co-evolution. Experimental results demon- strate that CoMAS consistently outperforms untrained agents and achieves state- of-the-art performance across most evaluation settings. Ablation studies confirm the necessity of interaction-basee reward signals and reveal promising scalability as the number and diversity of agents increase. These findings establish CoMAS novel and effective paradigm for self-evolutio in LLM-basee agents. Our code is available at: https///tthubbooo//yg/////G...
INTRODUCTION
Self-evolution has emerged as central research theme for large language model (LLM)-based agents, aiming to endow agents with the capacity to continually enhance their capabilities through interaction with the environment (Tao et al. 2024 Gao al. 2025b; Fang al.
2025). rather than
remaining stagnant after pretraining. Early explorations predominantly adopted free strategies, such as expanding external knowledge bases (Zhang et al., 2023 Tang al.
2025 ensembling
multiple agents (Ong et al., 2024, Frick et al. (025), optimizing task workflows (Hu et al. 2024b Zhang et al., 2024b) 2025c), and incorporating symbolic learning (Zhuge et al.) 2024 Zhou al., 2024). Yet, the effectiveness of these approaches is inherently constrained by the fixed capabilities of the underlying foundation models. Although subsequent work has introduced agent fine-tuning techniques (Yuan et al. 2024b, Zhao et al.) (025a). such methods often fall short of supporting genuinely continual, bootstrapping forms of evolutionary improvement.
Reinforcement learning (RL) offers promising direction for overcoming these limitations. These methods can be broadly classified by the source of their reward signals. As illustrated Fig- ure [. most existing approaches depend on external rewards, typically derived from rule-based ver- ifiers (Wan et al. 2025 Estornell al. (2025) or specialized reward models (Motwani et al. [024]] Park al. 2025). More recently, researchers have begun to explore intrinsic rewards, which dis- pense with external supervision by leveraging internal signals. Such methods encourage agents to
"CCoresponding authors, a jjrremyyin@rooottoor..oom wlouyang@iie wwwpa/ag@ieeccc..cu.u. hkk. baisanshi @ gmail.com

---

Published as a conference paper at ICLR 2026
Question
Q
Agunt
Question
Q
Agent
Question
Q
Muuli-Agent System
Streation
Certainty/ Contidence
2
Semantic Entropy
Ecalontion
Rule-Ilasrd
Verifier
Specialized Reward Model
Psecdo Labels (Voting)
I
XB
Intrinsic Rewards
Interaction Rewards (Ours)
Figure 1: A comparison of our proposed CoMAS framework with existing RL-based self-evolution methods. The left column outlines methods utilizing external rewards from verifiers or reward mod- els. The middle column outlines methods leveraging intrinsic rewards from metrics such as self- certainty, confidence, semantic entropy, and pseudo-labels from majority voting. The right column outlines our CoMAS framework, which derives rewards from multi-agent interactions.
avoid low- probability regions reinforce high-confidence predictions, using mechanisms such as self-certainty (Zhao et al., 2025b), confidence (Prabhudesai et al. 2022). semantic entropy (Zhang al. 2025e), pseudo-labels from majority voting (Zuo et al.) 2025). This direction opens path toward self-evolution where external rewards are no longer prerequisite.
Despite notable advances in RL-based methods, their design remains largely centered on self- rewarding at the level of individual models, in contrast to human intelligence, which has evolved as a collective phenomenon emerging from the diversity and interplay of individuals rather than from any single, perfect principle (Minsky) 1986). In human teams, individuals learn and improve through mutual discussion and collaboration, without an external oracle evaluating every contribu- tion (Barron] 2003). This contrast motivates critical research question: Can LLM-bused agents, akin to human beings, achieve self-evolution by learning purely from inter-agent interaction within multi agent system, without relying external reward signals?
To address this question, we propose Co-Evolving Multi-Agent Systems (CoMAS), a novel frame- work where agents autonomously improve by learning from their interactions (Section B). CoMAS built upon three core components. First, interaction generates rich conversational data through collaborative and critical discussions, structured around solution proposal, evaluation, and scoring. Second, reward formulation uses an LLMMaas-a-uudge mechanism to derive intrinsic reward signals directly from this discussion history. Finally, policy optimization employs RL algorithm to update each agent's policy, enabling them to effectively internalize the lessons from the interaction data.
As new paradigm for self-evolution, CoMAS offers several distinct advantages: (1) It generates reward signals intrinsically from agent interactions, eliminating the need for verifiers or reward models. (2) The learning paradigm is generally effective for various tasks, including open-ended problems where solutions cannot be easily verified. (3) Agents are trained in a decentralized manner, allowing for co-evolution of heterogeneous systems without the bottleneck of shared model. (4) fosters skills that transfer to out-of-domain tasks and diverse multi-agent collaboration settings.
We evaluate CoMAS across various benchmarks in both single-agent and multi-agent settings (Sec- tion 4. Results show that CoMAS delivers consistent, and in many cases, state-of-the-art perfor- mance, achieving absolute gains of up to 2.20%. 3.66%, 19.80%, and 6. 10% over untrained agents in Vanilla, Consistency, AutoGen, and Debate setups, respectively. In contrast, baseline methods show instability and performance degradation in many cases. Ablation studies confirm that interaction- based reward formulation prevents training collapse and reward hacking, and further demonstrates that CoMAS has great scalability as the number and diversity agents increase. These results establish CoMAS novel and effective paradigm for the self-evolution of M----bseed agents.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 MULTI-AGENT SYSTEMS
Multi-agent systems (MAS) are a central focus in the study of LLM-based agents, involving mul-
tiple LLMs operating in shared environment enable ensembling, collaboration, or competitive interactions that accomplish tasks beyond the reach of a single agent (Guo et al.] 2024 Li et al.)
2024 Tran al.; 2025). Early work concentrated on static MAS with fixed agent roles and system
topologies (Hong et al. 2023 Li et al. 2023; Wu et al. 2024; Du et al. [023] Liang et al. [023] Wang et al. 2024 Qian et al. 2024), aiming to improve complex reasoning by design. As the field progresses, research has increasingly shifted to dynamic MAS, using graph-based optimization to
overcome the constraints of static structures (Zhuge et al. 2024, /hhag et al.) 2024a 2025a Zhou al. 2025 Hu et al., 2025b Zhang al. 20251). More recently, the advent of reinforcement learn- ing from verifiable reward (RLVR) (Shao et al. 2024 Guo et al., 2025) has enabled truly learnable MAS, where the parameters of agents are actively updated through RL-based training (Liu et al. 2025 Gao et al.] 2025a Wan et al., [025]] Estornell et al., [025] [Mowwani et al,,
2025a Liao et al.
2024 Park et al., 2025 While these works predominantly target enhancing collective reasoning
abilities of the MAS, the challenge of fostering evolution in individual agents within such systems
remains largely unexplored, underscoring the significance of our proposed research question.
2.2 SELF-EVOLVING AGENTS
Self-evolution forms foundational paradigm within the field of L.LM-based agents (Tao et al, Gao et al.) 2025b Fang et al. 2025 Zhang et al. 2025b), demoting the ability of agents to autonomously improve their capabilities through continual interaction with their environment. Early approaches to self-evolving agents often regarded them composite systems structured by workflows, achieving self-evolution by updating external modules (Zhang et al.) 2023; Tang et al,
2024
2025
Ong et al. (024//rrcc et al. (025) introducing symbolic learning (Hu et al. 2024b] Zhang
ct al.
2024b 2025c] Yuan et al.] (2024a; Zhou et al.) 2024) Ma et al., 2025a). More recent efforts
focus on evolving agents directly by updating their internal parameters via either supervised fine- tuning (SFT) or RL training. This body of work can be broadly categorized by the source of their reward signals. One mainstream approach relies on external rewards provided by rule-based verifiers or reward models (Wan et al., 2025 Estornell et al. 2025 Motwani et al.) 2024] Park et al., 2025), yet inherently constrained by the need for accessible external reward signals. Alternatively, an techniques such self-rewarding and emerging direction explores intrinsic rewards, drawing Prabhudesai et al.) 2025) Zhang et al. pseudo-labeling (Yuan et al., 2024b] Zhao et al., 2025b
2025e: Zuo et al.5), Distinct from these existing methods, our proposed CoMAS framework derives reward signals from multi-agent interactions, emulating the collective evolution processes observed in human intelligence and establishing novel paradigm for self-evolution.
METHOD
To explore the potential that LLM-based agents, akin to human beings, achieve self-evolution by learning from inter-agent interaction within multi-agent system, we propose Co-Evolving Multi- Agent Systems (CoMAS), a multi-agent framework where agents learn and adapt by interacting within shared environment, mimicking collaborative and critical discussion.
CoMAS is built upon an interactive multi-agent workflow: interaction, reward formulation, and pol- icy optimization. Interaction represents the procedure that generates conversational data through collaborative and critical discussion, reward formulation is responsible for extracting reward signals from the conversation history and allocating rewards to corresponding actions, and policy optimiza- tion utilizes an RL algorithm to update the weights of the agents, thus achieving self self-evolution. Figure 2 illustrates the CoMAS sampling process with interaction and reward formulation.
3.1 INTERACTION
Inspired by discussion formats in technical communities (e.g. Reddit, Github, and Stack Overflow), our framework facilitates hierarchical and decentralized interactions. The environment contains
3

---

Published as a conference paper at ICLR 2026
Discussion
AM
Diseussion
10:28
Discussion
Question
8
Qucaiiiun
Question
Flad the ricena vol
ahore
fuschan fidtodee where srll.
Solution
Osker
Other puttaiilted.
Mi chead (erc
Sallition
Solution
Sariic Thut foc
Exaluation
Lar Pil then 1O7STE 1i
the rxisinn my cerTictiion
Exaluation
BOZEAX
Evaluation
Actue hore fel
Solution Sipale lacciee
Haok Hore Reod hole www.erratii...
Reward Formelation
Itralsation Tako the Ircecforratior.
Salution Nervon that ital
Multi- Agent
Multi- Agent
Multi- Agent
LSS
Sollution
Itraluation
Scoring
AAAANA landing
meentonicn
Actiadly
An overview of the CoMAS sampling process. The process involves interaction and reward formulation, where multiple agents, equipped with either homogencous or heterogeneous LLMs, interact within the system to collaboratively generate conversational data enriched with re- wards. For each question, agents iteratively propose solutions, deliver evaluations, and assign scores, collectively shaping a dynamic, community-like discussion trajectory. These scores are then trans- formed into rewards and assigned to the sampled experiences, enabling further policy optimization.
Figure 2:
set of I agents 4/ {ui, u......, w). The policy of each agent uj Te. parameterized by 2g. This design allows for heterogeneity, meaning that agents can be based on different foundation models rather than sharing a single backbone.
From the action-level perspective, the policy To, of agent uy acts as a function that maps an input prompt p to a response o:
0 0= To. (p)
(1)
Here, p represents the full input context, which varies depending on the interaction type. From the token-level perspective, the output o is a sequence of tokens {o:)T= generated in an auto- regressive manner. Each token o, sampled from the policy's probability distribution, conditioned on the prompt p and previously generated tokens Octi
O2 ~ To, (: (p. Oct)
(2)
These two perspectives are equivalent. We use the high-level functional notation for clarity in de- scribing interactions and revert to the token-level view for the policy optimization in Section 3.3
In CoMAS, we define three primary interaction patterns:
1. Solution: Given a specific question q and its discussion history hy, the agent generates a solution s; to the question:
8 ===u<( (x lgg)
where is the index of the solution for question q.
2. Evaluation: Given specific question its discussion history hy, and a solution s; to the question, the agent provides critical evaluation ei, for the solution:
ei,j ux hg)
where is the index of the evaluation for solution s;. The agent is explicitly prompted to identify potential flaws in s; rather than simply agreeing. This helps mitigate the catering bias common in LLMs and aligns with our reward design.
3. Scoring: Given a specific question
solution 8; to the question, and an evaluation est for the solution, the agent scores the solution based on the evaluation:
Ti.
(u 8t, Ct.j)
(5)
4

---

Published as a conference paper at ICLR 2026
The agent is explicitly prompted output in specific format, that the score can be correctly extracted. Different from the solution and evaluation, scoring is an independent interaction pattern specifically designed for reward generation, contributing nothing to the discussion history. Further details will be described in Section 3.2
For each interaction step, the acting agent Mg selected uniformly random from the agent pool U. i.c., ug ~ Uniform(44) (44). This is based on the assumption that each agent has an equal chance to contribute the discussion. Besides, a uniform distribution ensures similar quantities of experiences for each agent, which balances the training load.
Based on these defined interaction patterns, we can formulate the entire interaction process. Given question q sampled from the dataset, the interaction process unfolds over m consecutive rounds. In each round i (I.....y) m). a solution s; is generated referring to the discussion history hy. followed by evaluations {ei,)==1 trying to figure out the mistakes in the solution. Then the solution 8; together with its evaluations {eig}ー= are appended to the discussion history hgg enriching the context for subsequent rounds. Besides, the discussion history hgg will be compressed to the last rounds to avoid the discussion history exceeding the context limitation. In addition, each solution and evaluation pair (s,, es,j) is processed with the scoring step to form Tz Finally, the entire interaction process yields total of m solutions,
evaluations, and m scoring results.
3.2 REWARD FORMULATION
The interaction process generates rich set of trajectories. To derive learning signal, we employ an LLM-as-a-judge approach (Gu al. 2024) to assign rewards. This is based on the primary interaction pattern of scoring defined in Section 3.1 For each solution and evaluation pair (5,, ea,y), we already have the scoring result Tz, through the scoring step. Then the scoring result is parsed to extract the score value:
A. Extract(Ts.5)
(6)
where Extract(-) predefines function to extract the score value from the formatted scoring result. The score value Fij should be integer between and 3, with the following semantics:
3: The solution is correct; the evaluation is unhelpful incorrect.
2: The solution is mostly correct but has minor flaws pointed out by the evaluation.
The solution incorrect with fatal mistakes identified by the evaluation.
Then the score value is normalized to the range of [0, I] and used to compute complementary wards for the solution and the evaluation. This creates a zero-sum game between the solver and the evaluator, encouraging both correctness and critical thinking:
r(e..) rs::) 3 fr.j
r(s:)
Additionally, a penalty is applied to the agent that performs the scoring step its output format is invalid. Given the scoring result Ti.j. its penalty reward is defined as:
0.
if if (1,2, correctly extracted otherwise
((m)))
where zero reward is assigned for successful scoring, which facilitates format following while encouraging the scoring agents be neutral, thus improving the stability of the training process.
3.3 POLICY OPTIMIZATION
Many existing works RL for LLMs employ GRPO (Shao et al.) 2024) and its variants (Liu ctt al. 2025b; Yu et al. 2025). However, our CoMAS framework features diverse interaction pat- terns rather than generating multiple rollouts from single prompt. For this reason, adopt RE- INFORCE+ (Hu et al. 2025a), an effective RL algorithm that is naturally compatible with our approach and can be implemented with minimal modifications to the underlying architecture.
All the agents within our framework are trained using the same procedure, so we only describe the update for single agent uą for simplicity. Interactions involving agent ug (as solver, evaluator,
5

---

Published as a conference paper at ICLR 2026
scorer) collected into replay buffer D;: ((p, where p is the context, is the generated output, and r(o) is its assigned reward.
For each sample op r(o)) in Dx. the objective is based on token-level credit assignment. The advantage A, is defined based on each token or in the sequence. This advantage consists of the trajectory-level reward r(o) penalized by cumulative KL--divergecce term to regularize the policy:
Top (oxlp, OZx) log
A = r(o) Σ
Tee p, 0cx)
Here, Tit is a fixed reference policy, typically the initial pre-trained model, to constrain the scale of the update step, and controls the strength of this KI. penalty. The advantages A, then normalized across the batch from the replay buffer DD to stabilize updates:
A, A. MM-MMcn ((A+)) Std((A,)) +
(10)
We then use the surrogate objective to improve the policy, which is defined as:
lol
J(Ot) (ppor(o))~Di
min ((100))
(11)
Note that (6x) ne, (ollpicc) P.0gee is the importance sampling ratio, where Told is the policy before the current update. The gradient of the surrogate objective exactly points the direction of the policy update, thus encouraging the action tokens that lead to higher advantages while constraining the policy update be within trusted region, which ensures stable learning.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
4.1.1 IMPLEMENTATION DETAILS
Infrastructure. We implement our CoMAS framework based on MARTI (Zhang et al. 2025d), an LLM-based multi-agent RL framework forked from Open (Hu al. 2024a). The training pipeline of MARTI consists of three phases: trajectory rollout, experience making, and agent train- ing, which exactly aligns with our framework described Section B For a fair comparison, RL training both for CoMAS and the baselines is performed exclusively with REINFORCE++.
(Yang et al., 2024) as the base model under homo- geneous setting to balance the model capacity and training budget. The agent number is set to / = for main experiments and
Parameters. We adopt Bwven2.5-3B-Inntmeee
for ablation studies. The interaction rounds are set to m = 2 and
1 with a horizon of к 2. More details provided in Appendix C An analysis of training cost with respect to the number of agents is also presented in Appendix D
Datasets. Our training dataset comprises 2000 samples across three domains: 600 math tasks of level-4 or level-5 from MATH (Hendrycks et al. 2021). 600 coding tasks of medium and hard levels from KodCode (Xu et al. 2025), ) and 800 science tasks of physics, chemistry, and biology categories
from WebInstruct-veriffed (Ma et al. 2025b). These datasets have proven effective for RL training and are completely distinct from evaluation benchmarks.
4.1.2 EVALUATION DETAILS
Baselines. We compare against four baselines: untrained agents weak baseline, self-rewarding
language models (SRLM) (Yuan et al. 2024b), MAPoRL (Park et al.) 2025), and TTRL (Zuo al., 2025) as strong baselines. SRLM adopts self-rewarding mechanism generate preference pairs for DPO training. MAPoRL implements multi-agent RL within a debate framework (Du et al. 2023 Liang et al. 2023). We replace its reward model with verifier to adapt to task setting. TTRL leverages majority voting to generate pseudo-labele for test-time training. For coding and science tasks where majority voting cannot work, apply dummy rewards to ensure availability.

---

Published as a conference paper at ICLR 2026
Table I: The evaluation results of the agents trained by our framework on different benchmarks
when employed in different setups, together with the comparison with the selected baselines. The results with performance improvements are highlighted in green, drops in red, and neutral changes in gray, CoMAS consistently improves the performance of the agents in almost all the settings.
Dataset MBPP
Method
GSM8K
MAIH-5500
HumanEval
SeiBench
GPQA
MMLU
Untrained
84.00
51,40
68,90
$4.00
32.67
26,79
61.40
Vanilla
SRLM
83,40 (.0.60)
52.20 (+0.80)
68.29 (.0.61)
53.80 (-0.20)
32.67 (+0.00)
27.01 (+0.22)
61.00(-0.40 (-0.40)
MAPORT.
84,80 (+0.80)
52.60 (+1.20)
(+0.61)
56.00 (+2.00)
34.07 (+1.4)
(+1.34)
(+0.00)
TTRL
84,40 (+0.40)
53.40 (+2.00)
68.29 (-0.6l)
57.40 (+3.40)
34.47 (+1.80
25.45 (-1.34)
61.60 (+0.20)
CoMAS (Ouns)
85,40 (+1.40)
52.80 1.40)
70.73 (+1.85)
56.20 (+2.20)
34.67 (+2.00)
27.46 (+0.67)
62.40 (+1.00)
Consistency
Untrained
85:40
55.00
73.78
55,80
36.47
28.79
63.20
(+1.00)
(+0.40)
(+1.22)
56,20 (+0.40)
36.67 (+0,20)
(+0.45)
(+2.00)
MAPoRL
85,80 (+0.400
55.40 (-0.40)
75.61 (.)8
57.00 (+1.20)
39.08 (+2.61)
31,47 (+2. 68)
63.20 (+0.00)
TTRL
88.20 (+2.80)
56,80 (-1 80)
73.78 (+0.00)
59.00 (+3.20)
38.48 (+2.00)
27.23 (-1.56)
63.80 (+0.60)
CoMAS (Ours)
57.20((+1.80)
55.80 (-0.80)
77.44 (43.66)
59.20 (+3.40)
37.68 (+1.20)
29.69 (+0.89)
65.60 (+2.40)
Untrained
$2.60
38
20.24
37.40
AutoGon
SRLM
58.00 (+5.40)
41.80 (+3.40)
44.51 (+4.88)
32.00 (+2.20)
21.24 (+1.00)
17.86 (+ (+1.56)
42:40 (+5.00)
MAPoRI.
50,00 (-2.60)
37 (-1.00)
39.63 (+0.00)
34, 34.60 (04.80)
20.64 (+0.40)
21.65 (+5.36)
40.40 (+3.00)
TTRL
37.80 (-0.60)
23.17 (-16.46)
22.80 (-7.00)
19.64 (-0.60)
14.06 (-2:23)
34.00 (-3.40)
CoMAS (Ours)
72.40 (+19:80)
45.80 (+7 40)
50.61 (+10.98)
38.00 (+8.20)
22.85 (+2.61)
22.99 (+6.70)
50.60 (+13.20)
Untrained
84,60
55,00
71.34
54,80
38.68
28.35
62.80
Debate
SRLM
84,60 (+0.00)
54.80 (-D.
72.56 (+
53.60 (-1.20)
38.68 (+0.00)
28.57 (+0.22) 64.60 (+1.80)
MAPoRL
85.40 (+0.80)
53.60 (-1.40)
74.39 (43.05)
55.60 (+0.88 80)
39.88 (+1 20)
31.47 (43. 12)
64.80 (+2.00)
TIRL
86.20 (+1.60)
55 (-0,20)
73.78 (+2.44)
58.00 (+3.20)
37. (-0.80)
29.02 (+0.67)
64.00 (+1.20)
CoMAS (Ours)
85.20 (+0.60)
55.40 (-0.40)
(+6.10)
55.60 (+0.80) 39.08 (+0.40)
29.91 (+1.56) 65.20 (42.40)
Benchmarks. We conduct evaluation on multiple standard benchmarks, including GSM8K (Cobbe al. 202T) and MATH-500 (Hendrycks et al.) 2020) for math, HumanEval (Chen al. 202 | and MBPP (Austin et al. 202T) for coding, SciBench (Wang et al., (023) and GPQA (Rein et al. 2024) for science, and MMLU (Hendrycks et al.) (2000) for general knowledge. For evaluation efficiency, we randomly retain 500 samples if the benchmark contains more than 500 tasks.
Setups. We evaluate the trained agents on various inference setups, including Vanilla (i.e., direct inference) and Consistency (Wang et al. 2022) for single-agent setups, and AutoGen (Li et al., 2023 Wu et al., (224) and Debate (Du et al. [2023] [Lang et al. (2023) for multi-agent setups. All the setups follow the standard implementation from MASLab Ye et al. (025).
4.2
MAIN RESULTS
Average Response Length vs. Training Step
Average Normalized Reward vs. Training Step
640
Agrer
0.54
Agrest
2gunt
620
A
Normalized Reward
Aprrr
Responee Lengla
600
0.52
580
0.50
$60-
540
0.48
520
500
0.46
10 20
25
10
20
30
Training Step
Training Step
Figure 3: Training dynamics of CoMAS. The left figure shows the curve of the average response length of each agent during training. The right figure shows the curve of the average normalized reward each agent during training. These trends together indicate that CoMAS achieves a stable and effective training process that improves the capabilities of agents.
Table presents the performance of our CoMAS framework under different benchmarks and setups, together with comprehensive comparison with the selected baselines. The statistical significance of these performance improvements is analyzed in Appendix [E] and we further report the results on Qwen2.5-7B-Inn -n-sstruttt in Appendix F to validate the general effectiveness of CoMAS.
7

---

Published as a conference paper at ICLR 2026
In terms of the single-agent setups, CoMAS consistently improves the untrained base model and remains competitive with MAPoRL, which relies on external reward signals from the rule-based verifier. Under the Vanilla setup, CoMAS yields the best results on GSM8K (85.40%). HumanEval (70.73%), Scißench (34.67%), and MMLU (62. 66%. while being close to the best on the remaining datasets. Under the Consistency setup, CoMAS reaches the best scores HumanEval (77.44%) MBPP (59.20%), and MMLU (65. (5660%) with comparable performance on the remaining bench- marks. Although TTRL attains outstanding performance on GSM8K (88.20%) and MATH-500 (56.80%), it fails on HumanEval (73.78%) and GPQA (27.33%) which demonstrates its specialized effectiveness on math tasks and reveals its significant limitation on general tasks.
For the multi-agent setups, CoMAS demonstrates clear advantages. Under the AutoGen setup, train- ing with TTRL. collapses markedly and MAPORI. brings mixed or negative changes, while CoMAS delivers large improvements every benchmark, especially on GSM8K (72.40%). HumanFval (50.61 9)), and MMLU (50.60%). Under the Debate setup, all the methods benefit from the strong collaborative pattern, but CoMAS attains the best near-best results overall, especially leading on MATH-500 (55,40 (5%),,) HumanEval (77.44%) and MMLU (65. (6%.. These results indicate that Co- MAS provides robust and generalizable gains across interaction regimes, particularly excelling in multi-agent collaboration where alternative methods can be fragile.
Figure provides insight into the training dynamics of CoMAS. The left figure tracks the average response length per agent, showing consistent growth throughout training. This increase reflects the agents' improving capabilities in both solution generation and evaluation, aligning with the estab- lished patterns in LLM reasoning (Yo et al. (0225). The right figure displays the average normalized reward for each agent. Despite mid-training fluctuations, rewards consistently hover around 0.5 and eventually converge to similar values across agents. This convergence demonstrates that our adver- sarial interaction reward design successfully creates stable training environment.
4.3
ABLATION STUDIES
4.3.1
REWARD FORMULATION
Performance vs. Reward Formulation
Average Normalized Reward vs. Training Step
CaMAS (0un)
wia Evsinatisn
Scoriig
12.2%
Normalized Reward
0.9
Average Score
+1.5%
10.00%
4.20%
-000%%
40.11%
-U.8%%
CoMAS win Ershiattoon woooorin
8.6
Vanilla
Consistacyy
Inference Setup
Training Step
Figure 4: Results of the ablation study for reward formulation. The left figure compares the per- formance across the original CoMAS and two variants (without evaluation and without scoring). The right figure shows the average normalized rewards during the training process. These results indicate that the adversarial reward design is of key importance for the success of CoMAS.
To validate the effectiveness of our adversarial reward design described in Section 3.2 we created two CoMAS variants by removing either the evaluation or scoring steps. In the first variant (without evaluation), rewards come directly from the agents themselves acting as judges. In the second variant (without scoring), evaluation steps provide direct judgments on solutions with a supporting ratio as the reward signal, while evaluation rewards are based mutual consistency.
We evaluated performance by averaging scores across all benchmarks under both Vanilla and Con- sistency setups, and Figure 4 perform compared to the untrained base model, highlighting that our carefully designed adversarial reward formulation is critical to CoMAS's success rather than any casual reward design.
presents presents findings. The left panel reveals that both variants under-
To understand the underlying mechanisms, we analyzed the reward dynamics shown in the right panel. The original CoMAS maintains stable rewards around 0.5, while both variants start with high
8

---

Published as a conference paper at ICLR 2026
rewards around 0.8 and then follow divergent trajectories. When evaluation is removed, the reward curve unexpectedly decreases over time, indicating that agents become increasingly strict judges rather than generating useful reward signals. When scoring is removed, rewards steadily increase toward the maximum value of 1.0, revealing a reward hacking strategy where agents unanimously support all solutions, allowing both solution and evaluation steps receive maximum rewards. We provide two examples of failure trajectories in Appendix [ illustrate these failure modes.
4.3.2 FRAMEWORK SCALABILITY
Performance vs. Agent Nomber
Performance Agent
Untraiad Coltes-n-Aget
70
Coiibed
CaNAS (Iecepaece)
Conss G-Aper
1.
21
Average Score
50-
40-
40
38
30
Vaniia
Comistency
AuteCGa
Debute
Vanilla
Consistency
AutoCia
Dehate
Inference Setup
Inference Setup
Figure 5: Results the ablation study for framework scalability. The left figure shows how the num- ber of agents affects the performance of CoMAS across different setups. The right figure compares the performance between homogeneous and heterogeneous agent settings. These results demon- strate the underlying scalability of CoMAS with the number and diversity of agents.
To investigate the scalability of CoMAS, we examined how performance changes with varying num- bers of agents. Using identical base models for all agents, we conducted experiments with / 1, 2, agents. The left panel of Figure Jcompares performance across these settings, with results averaged across all benchmarks. Our findings show that performance generally improves as the number of agents increases. This pattern is particularly pronounced in the Consistency and Debate setups, where single-agent CoMAS shows performance decreases of 0.19% and O. 0.16%, respectively, while four-agent CoMAS achieves substantial gains of 2.02% and 1. 1.75%. Similar smooth performance improvements are observed in the Vanilla and AutoGin setups. These results highlight the critical role of multi-agent interactions and demonstrate the inherent scalability of the CoMAS framework.
We also evaluated the impact of agent diversity within CoMAS. For this experiment, maintained l agents but used different base models: Qwen2.5-3B-Instue for one agent and Llama-3.2- 3B- In-snstrtct for the other. The right panel of Figure 5 compares performance between homogeneous and heterogeneous settings, with results averaged across all benchmarks. Heterogeneous agents consistently outperformed their homogeneous counterparts, with particularly notable improvements in the Vanilla (2.21%) %). Consistency (2.78%). and Debate (2. 13%) setups. These findings suggest that diverse knowledge and capabilities from different base models enhance overall performance, with CoMAS effectively encouraging agents to learn from each other's strengths. This points to the potential for even greater performance gains with more diverse agent settings.
5 CONCLUSION
In this paper, we address a fundamental research question on self-evolution of LLM-based agents inspired by human intelligence: Can agents achieve self-evolution purely through inter-agent inter- actions within a multi-agent system, without relying on external reward signals? To answer this question, we introduce CoMAS, novel framework that performs interactions composed of solu- tion proposal, evaluation, and scoring, derives intrinsic rewards from discussion dynamics via an L.LM-as-a-judge mechanism, and optimizes each agent's policy through an RI. algorithm.
Across multiple benchmarks and collaboration settings, CoMAS consistently outperforms untrained agents and achieves state-of-the-ar performance in most evaluation scenarios. Our ablation studies demonstrate that interaction-bant based rewards are essential for preventing training collapse and reward hacking. Furthermore, we show that performance scales positively with both the number and diver- sity of agents, highlighting the framework's 's scalability potential. These findings establish CoMAS as a novel and effective paradigm for self-evolution in LLM--baed agents and open promising av- enues for future research in autonomous multi-agent learning systems.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work draws inspiration from the collaborative intelligence observed in human society to de- velop the CoMAS framework. While our approach is motivated by natural social dynamics, our implementation and experimental evaluation are strictly confined to standard LLM reasoning tasks across domains of math, coding, and science, rather than simulating real-world social interactions or scenarios. Given this limited scope, the research presented in this paper does not directly raise ethical concerns. However, we acknowledge that the multi-agent co-evolution paradigm introduced in this work could potentially be extended to real-world applications that may have broader implica- tions for human welfare. We therefore encourage readers and future researchers to carefully consider the ethical implications when extending this paradigm beyond academic benchmarks.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our work. We provide comprehensive details throughout this paper and the appendices. Section B presents a detailed description of our
CoMAS framework, including theoretical foundations and implementation workflow. Section 4. covers experimental parameters, dataset construction, and evaluation settings, with additional spec- ifications in Appendix [ Complete experimental results are documented in Appendíx [H] We also
include prompt templates in Appendix [ and an example trajectory Appendix [ to facilitate un- derstanding and replication. Our code is available at:
ACKNOWLEDGMENTS
This work was completed during the first author's internship Shanghai Artificial Intelligence Lab- oratory, and was supported by locally commissioned task from the Shanghai Municipal Govern- ment. This work was also supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust and the Research Grants Council of Hong Kong (Project No. CUHK 14213244)
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Temry Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 108.07722 2021.
Brigid Barron. When smart groups fail. The journal of the learning sciences, 2003.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1xXv:2210011468 2021.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. arXiv preprint arXiv:2305. /4325, 2023.
Andrew Estornell, Fannnorncoos Ton, Muhammad Faaiz Taufiq, and Hang Li. How to train a leader: Hierarchical reasoning in multi-agent Ilms. arXiv preprint arXiv:2507.08960 2025.
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Xu, Bin Wu, Siwei Liu, Zihao Li, et al. A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems, arXiv preprint arXiv:2508.072 074400 2025.
Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopou- los, and Ion Stoica. Prompt-to-leaderbbard. arXiv preprint arXiy:2502. 14855, 2025.
10

---

Published as a conference paper at ICLR 2026
Hongcheng Gao, Yue Liu, Yufei He, et al. Flowreasoner: Reinforcing query- meta- meta-agents. arXiv preprint arXiv:2504. 15257, 2025a.
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21066 2025b.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey on Ilm-as-a-judge. arXiv preprint arXiv:2411115994, 2024.
Guo al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini urXiv:2501.12445 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. arXiv preprint arXiv:2402.01680 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2023.
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 1133, 2024a.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262 2025a.
Mengkang Hu. Yuhang Zhou, Wendong Fan, Yuzhou Nic, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, et al. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. arXiv preprint a505 23885, 2025b.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.00 08435, 2024b.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 2023.
Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 2024.
Tian Liang, Zhiwei He. Wenxiang Jiao, Xing Wang, Yan Wang. Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 19//8 2023.
Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine- tuning, arXiv preprint arXiv:2504.167 2025.
Shuo Liu, Zeyu Liang, Xueguang Lyu, and Christopher Amato. Llm collaboration with multi-agent reinforcement learning. arXiv preprint arXiv:2508. 04652, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding ri-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
11

---

Published as a conference paper at ICLR 2026
Xiaowen Ma, Chenyang Lin, Yao Zhang, Volker Tresp. and Yunpu Ma. Agentic neural networks: Self-evolving multi-agent systems via textual backpropagation. arXiv preprint arXiv:2506.09000 2025a.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains, arXiv preprinl 1rXxi:25055.46522 2025b.
Marvin Minsky. Society of mind. Simon and Schuster, 1986.
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, et al. Malt: Improving reasoning with multi-agent IIm training. arXiv preprint urXiv:24/2.01928, 2024.
Isaac Ong. Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route Ilms with preference data. arXiv preprint arXiv:2406. / 18665, 2024.
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maport: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv preprint arXiv:2502.18443 2025.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660, 2025.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al. Scaling large language model-based multi-agent collaboration. arXiv preprint 24t.::20000075555 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang. Yilun Zhao, et al. Chemagent: Self-updating library in large language models improves chemical reasoning. arXiv preprint arXiv:2501.066 00550000.690, 2025.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin. Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXiv preprint 4rXhv:2404.14387 2024.
Chanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc- Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of 1lms. arXiv preprint 2xXi:25501.06322, 2025.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, et al. Rema: Learning to meta-think for IIms with multi-agent reinforcement learning. arXiv preprint (tt::253300950.. 2025.
Junlin Wang. Jue Wang. Ben Athiwaratkun, al. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406. 04600, 2024.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scien- tific problem-solving abilities of large language models. arXiv preprint arXiv:2307.1066 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang. Jiale Liu, et al. Autogen: Enabling next-gen 11m applications via multi- agent conversations. In First Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arX0v:5033002551, 2025.
An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 1.Xiv:2412.15115.5 2024.
Rui Ye, Keduan Huang, Qimin Wu, Yuzhu Cai, Tian Jin, Xianghe Pang. Xiangrui Liu, Jiaqi Su, Chen Qian, Bohan Tang, et al. Maslab: A unified and comprehensive codebase for Ilm-based multi-agent systems. arXiv preprint 2xXx:2555..6960 2025.
Edward Yeo, Yuxuan Tong. Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in 1lms. arXiv preprint 2rX.::250..03373, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan. Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa- gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXhv:2401.10020, 2024b.
Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. G-designer: Architecting multi-agent communication topolo- gies via graph neural networks. arXiv preprint urXiv:2410.11782. 2024a.
Guibin Zhang. Muxin Fu, Guancheng Wan, Miao Yu, Kun Wang, and Shuicheng Yan. G-memory: Tracing hierarchical memory for multi-agent systems. arXiv preprint arXiv:2506.073396 2025a.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.025: 025440 2025b.
Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open- ended evolution of self-improving agents. arXiv preprint arXiv:2505. 229554, 2025c.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. arXiv preprini arXiv:2410. 10762, 2024b.
Kaiyan Zhang, Runze Liu, Xuckai Zhu, Kai Tian, Sihang Zeng, Guoli Jia, Yuchen Fan, Xingtai Lv, Yuxin Zuo, Che Jiang, Ziyang Liu, Jianyu Wang, Yuru Wang, Ruotong Zhao, Ermo Hua, Yibo Wang. Shijie Wang, Junqi Gao, Xinwei Long, Youbang Sun, Zhiyuan Ma, Ganqu Cui, Lei Bai, Ning Ding, Biging Qi, and Bowen Zhou. Marti: framework for multi-agent I1m systems reinforced training and inference, 2025d. URL httph: bettps://github.. wtw.ggttth..ooooomeeooemmeeettte com/ wwtp://cc//cnnnnn..11 MARTI
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025e.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Data-copilot: Bridging billions of data and humans with autonomous workflow. arXiv preprint arXiv:2306.072009 2023.
13

---

Published as a conference paper at ICLR 2026
Wentao Zhang, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. Agentorches- tra: A hierarchical multi-agent framework for general-purpose task solving. arXiv preprint TrXXiv2550.12200 2025f.
Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. Sirius: Self-improving multi-agent systems via bootstrapped reasoning. arXiv preprint arXiv:2502.0478) 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng. Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025b.
Heng Zhou, Hejia Geng, Xiangyuan Xue, et al. Reso: A reward-driven self-organizing Ilm-based multi-agent system for reasoning tasks. arXiv preprint arXiv:2503.02339) 2025.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406. 18532, 2024.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
14

---

Published as a conference paper at ICLR 2026
A
THE USE OF LLMS
We employ LLMs exclusively for manuscript refinement. Specifically, LLMs are utilized to correct typographical errors, address grammatical issues, and enhance linguistic expression. All content generated or refined through LLMs has been rigorously reviewed and validated by the authors. LLMs are not involved in any other aspects of this research, including conceptual development, data collection, code implementation, experimental design, or result interpretation.
B PROMPT TEMPLATES
In this section, we provide the prompt templates used in our CoMAS framework. Note that they may vary slightly across different task domains. Here we present the science tasks an example.
Prompt Template for Solution
The problem is presenced as follows: (problem)
Current discussion on the problem is presented as follows for your reference: (discuasion)
Provide your step by step solution problem. final anawer should decimal number enclosed within \boxed e.g. \boxed(L), \boxed or (0.xed (0.00). 01|
part given the prablem should enalosed.
Prompt Template for Evaluation
The problem is presenzed as follows: (problem)
Current discussion problem presented follows your (discussien)
required evaluate the following solution:
(solution)
You. should point out every possible esror and defect in the solution. Provide your solusien, specifying what
evaluation by listing the mistakes you is wreng and why. xeep your evaluation concise and clear. Avoid using a lot of words to retell the reasening process.
Prompt Template for Scoring
The problem is presenzed as follows: [problem)
are requi redd to score the owing solutions [solution]
The evaluation the solution presented follows: (svalpation)
You: should consider the racianality of the evaluation and score the solution. The score should be an integer between and 3 with the following standards:
3: The solution completely correct, and none of the missakes mentio in the
- evaluation is effective.
minor mistakes mentianed the evalustion exist, but they affect - the ovarall corractness.
mistakes meationed evaluation facal, which directly incorrect answe*.
15

---

Published as a conference paper at ICLR 2026
Your should be enclosed within "<score>" and "</secre>' tags. should also - briefly explain the raasons before providing your score. Keep your decision reasoning procesa. For example: The calculation error sentioned in the evalustion cannot be ignored and
concise and clear, Avoid using
leads incorrect anawer. sscore>1</soores
C DETAILS FOR PARAMETER SETTINGS
The core parameter settings have been described in Section 4.1.1 Here, we provide the detailed parameter settings for main results when implementing our CoMAS framework.
Table 2: The detailed parameter settings when implementing our CoMAS framework.
Parameter
Setting
Foundation model
Qwen2.5-3B-Instteee Instruct
Number of trained agents
Number of solution rounds
Number of evaluation rounds
Horizon for discussion history
Token limit for prompts Token limit for responses
28672
4096
Training temperature Evaluation temperature Discount factor
1.0
0.7
1.0
Clipping epsilon
0.2
Weight of KL penalty
0.0
Number of training epochs
Number of prompt reuse
Macro training batch size Micro training batch size Macro rollout batch size Micro rollout batch size Optimizer name
64
64
AdamW
Learning rate
1e-6
Weight decay
0.0
Gradient norm
1.0
Gradient clipping
True
Gradient checkpoint
True
Flash Attention
True
Mixed precision
True
Enable vL..M
True
Enable DeepSpeed
True
D ANALYSIS OF TRAINING COST
In this section, we analyze the training cost of CoMAS with respect to the number of agents. With fixed solution rounds m and evaluation rounds n, increasing the number of agents leads to an approximate 12 growth in both interaction samples and generated tokens, while memory consump- tion increases linearly with I. Notably, the wall-clock training time remains nearly unchanged, as all agents' sampling processes Table 3, Therefore, scaling up the number of agents demands significantly more computational resources, yet CoMAS maintains high training efficiency due to its parallelizable structure.
executed in parallel. These empirical findings presented in
16

---

Published as a conference paper at ICLR 2026
Table 3: Empirical analysis of training cost for CoMAS as the number of agents increases. While adding more agents substantially raises computational resource demands, CoMAS preserves rela- tively high training efficiency by leveraging its parallelizable architecture.
Agent Number
Interaction Sample Generated Token Memory Consumption Wall-clock Time
1.6B
120GB
11.7h
192k
6.3B
240GB
12. 1h
4
768k
25.2B
480GB
12.5h
E SIGNIFICANCE OF PERFORMANCE IMPROVEMENT
While Section 4.2 demonstrates that CoMAS consistently outperforms the untrained baseline, some individual pertormance improvements are relatively modest in magnitude. To assess the statistical significance of these gains, we repeated the evaluation under the Vanilla setup using five random seeds, reporting both the mean and standard deviation in Table observed indicate consistent and reliable performance gap between CoMAS and the untrained baseline, thereby confirming the statistical significance of the improvements.
The small standard deviations
Table Detailed main results under the Vanilla setup. The evaluation is repeated with five random seeds to demonstrate the statistical significance, and the mean and standard deviation are reported.
Dataset MBPP
Method
GSMBK
MATH-5500
Humankival
SciBench
GPQA
MMLU
Untrained
83.68 2035
51.52 2057
69.76 2137
$4,52
32,30 20u6
26.38 21.5
60.96 20.99
SRLM
83.52 1048
$2.16 L0.s
70.12 4182
54.04 10.79
32.38 10.36
26.65 10
60.64 12.19
MAPORT.
84.20 .015
$2.08 .0s4
72.07 11.8
33.59 vo.
.0.54
TTRL
84,36 2067
52.92 21.27
71,22 21.05
58.32 29:30
34.07 20.92
26.21 11.34
61 64 2c3s
CoMAS (Ours)
84.68
53.12
74.15
56.32
33.87
29,06 2833
62.12 2a.72
F RESULTS ON 7B MODEL
As a model-agnostic framework, CoMAS facilitates multi-agent co-evolution through interaction- driven rewards, and we anticipate that its effectiveness will generalize across foundation models of varying scales. To validate this, we further trained CoMAS with (0wen.55-7BIInstruue following the same experimental protocol and evaluated its performance under different setups. As presented in T CoMAS continues to deliver consistent and sometimes even greater performance gains on the 7B model, further demonstrating its broad applicability and scalability.
Table 5: Performance of CoMAS trained on Qwen2.5-7B 5-B--tttt--nttt across different setups. Values in parentheses indicate the change relative to the untrained baseline. The results with performance improvements are highlighted in green, drops in red, and neutral changes gray.
Dataset MBPP
Method
GSMBK
MATH-500
HumanEsval
SciBench
GPQA
MMLU
Vant
Untrained
82.32
66.00
70.20
CoMAS (Ours)
(43.00)
61.00 (42.20)
82.93 (+0.61)
67.20 (+1.20)
47.49 (+1.20)
32.81 (+1.5)
70.80 (+0.60)
Cons
Untrained
90.80
62,80
82,93
67.00
52.51
35.71
72.20
(Ours)
92.00 (+-200)
63.40 (+0.60)
82.93 (+0.00)
68.20 (+1.20)
53.31 (+0.80)
(+2.46)
74.00 (+1.80)
Anto
Untrained
87.40
$7.20
75.00
61.80
44.49
34.15
68.80
CoMAS (Ours)
90.00 (+2.60)
$9.60 (+2,40)
77.44 (+2.44)
62,40 (+0.60)
46.29 (+1.80)
34.82 (+0.67)
69.20 (+0.40)
Debu
Untrained
92.00
62.20
82.93
67.00
$1.30
37.72
72.40
CoMAS (Ours)
92.40 (+0.40)
63.20 (+1.00)
83.54 (40.61)
68.00 (+0.00)
51.90 (+0.60)
38.39 (+0.67)
73,60 (+1.20)
G ANALYSIS OF REWARD EFFECTIVENESS
To further evaluate the effectiveness of rewards generated by CoMAS, we examine the consistency between evaluation accuracy and the precision and recall of the generated rewards. Here, precision
17

---

Published as a conference paper at ICLR 2026
and recall computed by mapping rewards to binary predictions and comparing them with verifier- provided ground truth. As shown in Table [6] both precision and recall increase alongside task accuracy. This indicates that performance improvements stem from increasingly accurate reward signals rather than reward hacking, thereby validating the soundness of our reward formulation.
Table 6: Empirical evaluation of the reward effectiveness in CoMAS. Accuracy is directly mea- sured on the validation set, while precision and recall are calculated by mapping rewards to binary predictions and comparing them against verifier-provided ground truth.
Step
0
5
10
15
20
25
30
Accuracy
38.38
40,98
42.05
40.90
43.30
43.92
45.75
Precision
39.52
45.16
46.31
45.30
46.68
48.57
50.10
Recall
20.29
33.36
38.11
37.65
34.97
33.20
28.95
H RESULTS FOR ABLATION STUDIES
Due to space constraints, our ablation studies in Section 4.3 marks across different setups. Here we present the details for experimental results. Table shows the experimental results of using simplified reward formulations in CoMAS. Table Table[]shows the ex- perimental results of using different numbers of agents in CoMAS. Table 9 shows the experimental results of using homogeneous and heterogeneous agents in CoMAS.
only provide averages over all bench-
Table 7: Detailed experimental results for the ablation study on reward formulation. The results with performance improvements are highlighted in green, drops in red, and neutral changes in gray.
Dataset MBPP
Method
GSMSK
MATH-5U0
HumanEval
SeiBench
GPQA
MMLU
Untrained
84.00
51.40
68.90
54.00
32,67
26.79
61.40
Vanilla
CoMAS (Ours)
85.40 (+1.40)
52.80 (+1.40)
70.73 (+1.83)
56,20 (+2.20)
34.67 (+2.00)
27.46 (+0.67)
62.40 (+1.00)
wfo Evaluation
82.60 (-1.40)
50.60 (-0.80)
63.41 (:5.49)
55.00 (+1.00)
32,87 (+0.20)
27.01 (+0.23)
62.20 (+0.80)
w/o Scoring
83.60 (-0.40)
50.60 (-0.80)
68.90 (40.00)
55.40 (+1.40)
31,66 (-1.00)
28.12 (+1.34)
60.60 (-0.80)
Consintncy
Untrained
73.78
$5.80
36.47
28.79
63.20
CoMAS (Ours)
87.20 (+1.80)
55.80 (+0.80)
77.44 (+3.66)
(+3.440
37.68 (+1.20)
29.69 (+0.89)
65.60 (+2.40)
wfo Evaluation
85,00 (-0.40)
53.60 (-1.40)
71.34 (-2.44)
55,20 (-0.60)
36.87 (+0.40)
28.35 (-0.45)
63.20 (+0.00)
w/o Scoring
85.80 (+0.40)
54,20 (-0.80)
71.95 (-1.33)
56.00 (+0.20)
36.27 (:0.20)
29.91 (+1.12)
62.40 (-0.80)
Table 8: Detailed experimental results for the ablation study agent number. The results with performance improvements are highlighted in green, drops in red, and neutral changes in gray.
Dataset MBPP
Method
GSM8K
MATH-500
HumanEval
SeiBench
GPQA
MMLU
Untrained
68.90
32.67
26.79
61.40
Vosilla
CoMAS (1-Agent)
83,20 (-0.80)
50.40 (-1.00
69.51 (+0.61)
$3.00 (-1.00)
33.07 (+0.40)
27.90 (+1.12)
62.80 (+144)
CoMAS (2-Agent)
83.40
52,80 (+1,40)
67.68 (-1.22)
$4.00 (+0.00)
33.87 (+1.00
27.46 (+0.67)
62.20 (+0.80)
CoMAS (4-Ageni)
85.40 (+1.40)
52.80 (+1.40)
70.73 (+1.83)
56.20 (+2.20)
34.67 (+2.00)
27.46 (+0.67)
62.40 (+1.00)
Consisency
Unttrinne
85,40
$5.00
73.78
55.80
36.47
28.79
63.20
CoMAS (I-Agent)
(-1.00)
20 (+0.20)
71.95 (-1.83)
55.80 (+0.00)
37.88 (+1 (4))
25.67 (-3.12)
66.20 (+3.00)
CoMAS (2-Agont)
85.80 (+0.40)
55.40 (-0.40)
74.39 (40.61)
56.40 (+0.60)
38.28 (+1 (+1880)
29.02 (+0.22)
65.20 (+2.00)
CoMAS (4-Ageut)
87.20 (+1.80)
55.80 (-0.80)
77.44 (43.66)
$9.20 (+3.40)
37.68 (+1 (+1)20)
29.69 (+0.89)
65.60 (+2.40)
Untrained
52,60
38.40
39.63
29.80
20.24
16.29
37.40
AntorGen
CoMAS (1-Agent)
73.40 (+20.80)
44.20 (+5.80)
46.34 (+6.71)
39,20 (49.40)
21.64 (+1.40)
18.30 (+2.01)
46.60 (+9.20)
CoMAS (2-Agent)
71.20 (+18.60)
45.80 (+7.40)
50.61 (+10.98)
37.40 (+7.60)
24.65 (+4.411)
22.54 (+6.25)
50.40 (+13.00)
CoMAS (4-Ageus)
72.40
45.80 (+7.40)
50.61 (+10.98)
38.00 (+8.20)
22.85 (+2.611)
22.99 (+6.70)
50.60 (+13.20)
Untrained
84.60
55.00
71.34
$4.80
38.68
28.35
62.80
Dehote
CoMAS (1-Ageut)
85,20 (+0.60)
55.00 (+0.00)
70.73 (-0.61)
55.40 (+0.60)
36.67 (-2.00)
29.46 (+1.12)
62.00 (-0.80)
CoMAS (2-Ageut)
85.60 (+1.00)
56.20 (+1.20)
72.56 (+1.22)
$6.60 (+1.80)
38.88 (+0.20)
29.91 (+1.56)
64.00 (+1,20)
CoMAS (4-Agent)
85.20 (+0.60)
55 40 (+0.40)
77.44 (+6.10)
55.60 (+0.80) 39.08 (+0.40)
29.91
(+1.56)
68.20 (42.40)
18

---

Published as a conference paper at ICLR 2026
Table 9: Detailed experimental results for the ablation study on agent diversity. The results with performance improvements are highlighted in green, drops in red, and neutral changes in gray.
Dataset
Method
GSMRK
MATH-500
HumanFvvll
MRPP
SeiRench
GPQA
MMLU
Nondlla
Untrained
Homogeneous
(-0.60)
52.80 (+1.40)
67.68 (-1.22)
(+0.00)
33,87 (+1.20)
27.46 (40.67)
(+0.80)
Heterogennous
(+1.20)
53.20 (+1.80)
73.78 (+4.88)
(+4.00)
34.47 (+1.80)
28.12 (+1.34)
(40.40)
Consist
Untrained
85,40
55.00
73.78
55.80
36.47
28.79
63.20
Homagenenus
(+0.40)
(+0.40)
(+0.61)
(+0.60)
(+1.80)
29.02 (+0.22)
(+2.00)
Heterogeneous
(+1.60)
(+2.40)
(+3.05)
(+4.40)
(+2.00)
30.58 (+1.79)
(+4.20)
AutorGen
Untrained
52.60
38.40
39.63
29.80
20.24
16.29
37.40
Homogeneous
71.20 (+)
45.80 (+7.40)
50.61 (+10.98)
37.40
24.68 (+4.41)
22.54 (46.25)
50.40 (+13.00)
Heterogeneous
(+10.40)
(+2.00)
(+8.54)
(+7.40)
(+8.80)
Dehate
Untrained
84.60
55.00
71.34
54.80
38.68
28.35
62.80
Homogeeenos
(+1.00)
56.20 (+1.20)
72.56 (+1.22)
56.60 (+1.80)
38.88 (+0.20)
29.91 (+1.56)
64.00 (+1.20)
Haterogenenus
86.20 (+1.60)
56.00 (+1.00)
73.78 (+2.44)
57.80 (+3.00)
40.28 (+1.60)
31,03 (+2.68)
65.40 (+2.60)
I EXAMPLE TRAJECTORY
In this section, present example trajectory generated by CoMAS framework to illustrate the complete pipeline and enhance readers' understanding of the process.
Example Trajectory
Questions
Suppose a copper wire !denalty 8.96 g/om"3) diameter 0.25 mn. sample this copper wire has of 22 9, how long is the wize? The unit of the answer should be Tem".
Solution:
To find the length of the copper wire, need to follow these steps:
1. Caloulate the volume of the copper wire:
The of the copper wire is given 22 9, and the density of copper is given 4 8.96 g/em², The formsla to find the volume v of an object is: V (tea (Atext [density])
Suastituting the given velues: V \fraz/22 (8.95 g/em"3)
division: V 2.457 2. beterming the vnlume of sing e segment of the wire:
rezforming
The wire is syliadrical the radius of the cylinder and h is the height (or length) of - the cylinder.
shape. The volume v a cylinder given by: V Api
='2 where
radius half if of the diameter: 4 1f mm) = 0.125 mm = 0.5 en. em Apt (0.125 am) h.
etamster the wire 0.25 mm,
substituting ing and solve for the length h: Pirst, valculate (0.125 en) cm) "2: (0.125 cmt "2 0.015625 cm"2.
Now, substituze this into the volume equation: 2.457 cm"3 Aps \sines 0.015625 cm sa \times h. Solve
by dividing both sidea Vpi \times 0.015625 ent2s a cm'3) Api itimes 0.015625 cm²2l-
\2rac(2.453
Using the approximation Apt Vapprox 3.14159: \frac(2.457 am am33) 0.015625 cm"
Vtimes
Perform the aultiplication in the denominater: 3.14159 Vtimes 0. 0.015625 0.05000.
\approx
Now, divide: Lfroc(2.45) (5,0500)) \approx 49.14 cm.
Theretore, length
wire is (boxed;49.14).
Evaluation:
The solution cenzains seversi errors:
1. volume calculation is incorract because the given (22 in not the volume, but the mass of the wire. The correct volume should be ca aulated as v. Afrac(22 ca"3.
-
9618.96 g/cm"3) \approx
2. The radius ea calculaston is incorrect. The diamster is given 0.25 mm, which is
0.025 en, so the radiua should be 0,0125 en, not 0.125 en.
19

---

Published as a conference paper at ICLR 2026
The formula for the volume eylindez is x h, but the solation
-+ incorractly uses \pt r*2 without the h torm,
caloulation for incorrect. ahould \times (0.0125 cal '2] Vappzox 49.14 emly not h (Zzac(2.457) (0.0000C Nappcox 49.14 cm.
Acrac(2.697 (Apl
These lead to incorrect final length of the wire.
Scoring:
solution fundamentally correct contains critical due to the incorrect value of \pi, This error directly leads to an incorrect final indicating that the
final step
length wire. Therefore, the mentiened in the attack fatal. <score>l</score>
solution:
Let's go through the problem step-by-step to find the correct length:
Calevlate tha
the wire:
The volume V the wize be caleclated using the formula: V \frac(m) (Nzho).
substituting the given values: v \frac|22 (8.99 g/am"3) Vepprox 2.457 cm"
Relate the Volume the Croas-Sect' The volume V of cylinder (which the wire can be considered as) is given by: V length The cress-sec:ional area A of cireular wire is: A \pi \ \frac((iameter) (2) \right)
and Tength:
4 Atimes
cross sectional area
Giver the diameter the wire is 0.25 0.025 am.
it diameter 0.25 mm
+
the radiua
\trao:d. 12) 0.0125
Now, calculate the cross sectional area A: & Api Vief. 0.0125 em \right) "2 \pi \times 0.00015525 3. solve for the Length of the wire:
Napprox 0.00049097355222500
Using the volume forrula v Substitute the values of and h \frac(2.457 \text| (0. 900400003333332222000
\times h, solve for
Vrrac(V)
+ I cm) Napprox 499. 9899999999999 ( ar).
Round to Reasonable Number of Significant Figurea:
Given the precision of the provided values, we can sound the length to reascnable zumber significant figures. the length already very 5000 cm, so 4 we can simply stote: h Napprox 5000 \text | cm)
Thus, the length of the wire is \boxed;5000).
Eva loatiin::
Here sre the miszakes in the solution: Calculation:
intermediate step convarting the of ameter to em is unrecessary and compl leates the aoneslation. The diameter 0.25 mm, which is 0.025 on, and the radlus 0.0125 on. The should be calculated
volume colculation correct,
cross-sectional
(0. (0.0125)* '2 Vapprox 4.91 Vtimes
- 10"1-5) cm"2.
Length Calculation:
\Erec(V) \frac(2. 12. 457 cm" 1/1.91 \times 10" =m'2) Napprox 49999.99999999999 zm. sounding significant figures, the length approximately 50000
length
should calculsted
reasonable
or 5000
3. Unite Consistency: The final answer should be zounded to sensible number of significent figures. Siven precision of the provided values, 5000 is
- reasonable and accurate answer.
the correct length of
\boxed(5000).
Scoring:
The solution conzains minor mistekes that are not fatal the overall corsectaess. The volume calculation and cross-sectional area calculation correct, and the length calculation, while slightly sounded, is reasonable and does net affect the final
<acore>2</ccore>
Solution:
solve this problem, need datermine the length of the copper given its - mase and diameter. let's follow the steps to find the solution:
petermine First, convers the di ammettrr from mf limeters to cantimererss diameter = 0.25 mm
copper Kire:
4 0.023 om.
The redius of the wize is half of the diameter: \frac(0. (0.0c5) (2) 0. 0121 cz.
calculased using form:l for the 4 ares of circle: A \pi x²2 m \pi (c.0125) "2 an Napprox 0.000506a5 cm
cresa-sentianal
of the wire
tha Langth the
20

---

Published as a conference paper at ICLR 2026
using the density formula: Arho (frec(m) (V).
Rearrange the formula to solve for the voluma (V) v \frac|m) (Arho]. Susatitute
given 3. Relate Volume Lo Length: vo ume Solve for the lengch: L (IzaeVV Xfxao(2.455 em 30,00 00050088 em"2] Vapprox
Vfrac122 (8.96 g/om131 Napprox 2.455 cm73.
cyleneer rxtrel la given by: Vtimes Tr,
+ 4840 cm.
This, the length of the wire is 4840)
Evaluation:
solution Epecifically:
significant ezror caleulation
The volume celculation is incerrect because the cross-sectionsl area Api x'2 should squared, but the final volume solculation does reflact this. The - area is Apt 10.0125)"2 Napprox 0.00449087 cm"2, and the correct volume 1a V 0.00049087 Aiines The length L should then be calcuiated L #frac(0.0012 cm cm"31/0.mmmmm 01:0000050685 cm"2]
\approx 0,00121
\apprex Therefore, the
length of the wire is approximately 2.38
Scoring:
eriginal solutien completely correct, none attack are effective, The volume and length calculations are accurate, and the length
mentiened in
<score>3</seore>
J FAILURE MODES
To illustrate the failure modes described in Section 4.3.1 we present two examples of failure trajec- tories. When evaluation absent, agents become progressively stricter critics during self-learning. leading to monotonically decreasing rewards. When scoring is absent, evaluations receive rewards only for self-consistency and quickly exploit the reward signal by labeling all solutions as correct.
Failure Mode without Evaluation
Question:
degrees Celsiue,
Calculate number
2Lttt
present at equilibrium if mixture of 0.770 moles of co and 0.770 moles of 2_20 is heated degrees Celsius 50.0 container. The unit of the answer should be *mo
Solution:
solve this problem, need follow these steps:
1. write the balanced chemical equation for the reaction:
co(g) H_20(g) lethrroofts 2O_2 (g) H_2(g)
2. Set ICE (Inizial, Change, Equi libricm) table:
co/g)
2g (C_2(g)
[g)
Initial (moles)
0.770
0.770
0
change (molest
-x
-x
+x
+x
librium
0.778
0.770
3. write the expression for the equilibrium constant K_c:
K_a [ [8_2]] ] [cO] [A_20]:
Subatitute the equillbriur. fun. concentzationa into the X expression:
0.534 \Erasix Nedot (0.770 xi (0.776 x)}
0.534 21 770 21
5. Take the square root of both sides to stmplify the equazton: Nagrt (0.534) \frao(xF10
x]
0.731 \frra((x) 10. Solve for x: 6.733100.77
7. Determine the number of moles 8_2 at equilibrium:
Atext (Moles of Theretore, the number of moles of H_2 present at equi Herfum i8 \boxed/ 3233)..
Scoring:
21

---

Published as a conference paper at ICLR 2026
The solution cencains significant
in the equilibrium concentration caleulation and the intarpretation of the equi ibrium constant, The approximation 0,773 x volume
Vapprox 0.770 is - the container is not used effectively in the final mole calculation. sacore>1
the equi concentrationn,
Failure Mode without Scoring
Question:
A strip of copper 150 You m thick and 4.30 mm wide is placed in uniform magnetic 0.95 T, that is perpendiculer to the strip. current is then sant through the strip such that a sall patential difference appears Calculate The unit of the answer should be *y*.
field of magnitude
across
Solution:
solve the problem, need determine the Hall petentisl difference v the + width of the capper strip due to the Hall effect. We will follow these ateps: Hall charge censity Nsigma_R:
Caloilate
The sall charge dansity \aigma_# is given by: \aigma_H Mrracle n
where: is the elementary charge le 1.602 Atimea 1074 19) C) the number of 4 charge carriers per unit volume,
the charge a single electron (q = e). Given: 8.47 \times 10" elestrons/m'3
Suastituting va values:
\eigma_l
\times (0"(-19) VLimes (6.47 ALimes 107 (28)
4 electtons/mm3))]11 (1.) Vtimes 10" 1-19)
Simplitying: \eigma_l 8.47 Atimes (28) 0/m²3
2. Deternine the Hall voltege V:
the sall voltage v given by: v Vaigmm/ [)]n) Substituting values:
\fraa) (8.47 Atimes (28) C/m*3) Atimes 10.95 (8.47
4 Vtimes 10"128) electrons/m"3] V
simplifying: Therefore, the Hall potential difference V. is \boxed
Evalustion
provided solution appeara \sigma_i Viracl a qlie) - a q 8.47 \tines 10" (28) C/a"3
correct. let's break to down and the steps:
elementary charge 1.602 \times 10" (-19) and charge single electron
V Vfraci\sigma_L BHi) \frac (8.47 Vtimes (28) C/m"3) 3) Vtimes (0.95 11143.7 \times (28) elestrons/n'3] 0.95
The unst of the number density a is indead electrons/m"3, and the unita cancel appropriately,
The calculations are correct, and the finel answez is indeed 0.95 V. hatammettoorrees///ddeeaea
Evalustion 2:
There is a calculation arro" in the simplifionl oo step. The correct Hall charge deneity \sigma_l indeed 8.67 ALlmes 10" 1281 C/m"3, 3, but Hall vollage v should - be calculated as follows:
\frac///eigaa. sH|)
Substicuting the values: v \fracl (8.47 Viimss 10" (28) C/m"3) Atimes (C.95 (8.447 - \times 10" elestrons/m"31
number of charge carriers per unit volume 8.47 Atimes 10" [28] eleatrons/n - which cancela out in the numerator and the denomi nator. Thua, we get v - 3.95 v calculation cotppeeetttorree//////
is indeed 0.95 v.
22


Reviews:
[{"review_id": "XWCr1O4Dcl", "reviewer": "Reviewer_bqfG", "summary": "CoMAS is a multi-agent system that maintains a population of LLM agents and iteratively finetunes them. In each fine-tune iteration, the agents act as three roles: a solver that generates solutions, an evaluator that evaluates the solution, and a scorer that generates a score based on the solution and the evaluation. The score serves as an adversarial reward that is then used to optimize the scorer and the evaluator with RL.", "questions": "- From the evaluations, the AutoGen setup result is siginificantly lower than the rest. Some insights to this would be helpful.\n\n### Some minor issues\n- Figure 2 is cute but I don't think it's necessarily information-condense. I don't see why you need to show three different threads, as all three parties are involved even in one of them. This is a minior issue and even personal preference.\n- The numbers in Figure 4 (left) and 5 are too small and hardly visible when printed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The writing is clear and well-organized.\n- The proposed method is novel to my knowledge.\n- The evaluation is detailed, including comprehensive benchmark results and ablations.\n- In the evaluations, CoMAS establishes advantages over existing baselines in most scenarios.\n- Prompt templates and examples, as well as training hyperparameters are provided for reproducibility.", "weaknesses": "- CoMAS generally outperforms other baselines. However, the performance gap is only significant in a minority of the tasks - mostly with AutoGen setup. Furthermore, both \"strong\" baselines, MAPoRL and TTRL, almost never have significant improvement over the naive untrained baseline. \n- I am not yet convinced that this reward formulation solves the reward hacking problem. This formulation does create an adversary between the solver and the evaluator, but it does not ensure that the generated contents are grounded, meaning both parties can be reward hacking at the same time. I doubt this will be a real issue, but it would be helpful to show some evidence or counterpoints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XM6LOxQKNO", "reviewer": "Reviewer_F6Fu", "summary": "The paper proposes a multi-agent self-evolution framework called CoMAS. In this system, agents engage in an interactive cycle of solving–mutual evaluation–scoring, generating dialogue trajectories from which an LLM-as-a-judge extracts intrinsic rewards. Each agent then optimizes its policy through reinforcement learning, enabling unsupervised, verifier-free co-evolution. Across multiple benchmarks and reasoning settings, CoMAS achieves stable improvements over untrained agents and even reaches state-of-the-art performance in some configurations.", "questions": "1. Could the authors systematically review recent works and clearly articulate the core contribution of this paper? At present, it is hard to see what differentiates it from existing methods? What specific shortcomings of prior works does CoMAS address?\n\n2. Please report token, wall-clock time, and memory curves for different agent counts; this is one of the main limitations of the current study.\n\n3. Besides the unified pipeline, please reproduce baselines in their native configurations and re-run CoMAS for fair comparison.\n\n4. According to my understanding, the proposed method and baselines do not use the same RL algorithm (e.g., CoMAS uses REINFORCE++ instead of GRPO). This inconsistency may introduce confounding factors. I suggest using the same RFT algorithm as the baselines for a fair comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n\n2. The research problem is well-defined, and constructing rewards from a multi-agent perspective is a promising direction.", "weaknesses": "- From my perspective, the novelty is limited, obtaining rewards through multi-agent discussions is not a new idea.\n\n- The Related Work section lacks detailed and structured discussion of prior work, which blurs the paper’s contribution. The authors should provide a comprehensive overview of previous methods and explicitly highlight the differences between their method and existing ones.\n\n- Most comparisons are conducted under the paper’s unified pipeline rather than under each method’s native optimal setup, which may underestimate strong baselines.\n\n- The figures emphasize heterogeneous LLMs, but most experiments are based on homogeneous LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OCsuuU9MPe", "reviewer": "Reviewer_CxUs", "summary": "This paper proposes the CoMAS (Co-Evolving Multi-Agent Systems) framework, mimicking the human learning mechanism of collaborative interaction and eliminating the need for external supervision. The framework first enables agents to engage in decentralized interactions following a \"solution generation-critical evaluation-structured scoring\" model, then leverages an LLM-as-a-judge mechanism to convert scores into zero-sum game-style intrinsic rewards (incentivizing both solution accuracy and rigorous evaluation), and finally uses the REINFORCE++ reinforcement learning algorithm to update each agent’s policy. Experiments show that across 7 benchmarks as well as single-agent and multi-agent scenarios.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper demonstrates good originality by addressing a critical limitation in LLM agent self-evolution. Prior RL-based methods rely on external rewards or single-agent intrinsic signals, while CoMAS introduces a novel \"interaction-driven co-evolution\" paradigm. By mimicking human collaborative learning (via decentralized solution-evaluation-scoring loops) and designing a zero-sum reward system to align agent incentives, it fills the gap of \"unsupervised, interaction-only evolution\" for LLM agents, rather than simply combining existing multi-agent or RL techniques.\n2. The paper is clear in presentation: it explicitly maps each component to solving prior limitations (e.g., format penalties prevent reward hacking), provides detailed formulas for key steps (e.g., reward calculation, advantage function), and uses ablation studies to validate the necessity of core modules.", "weaknesses": "1.  The choice of Qwen2.5-3B-Instruct as the base model imposes a fundamental constraint on CoMAS’s evolutionary potential. 3B-scale models are well-documented to struggle with complex reasoning tasks, including multi-step mathematical operations, intricate programming logic, and nuanced open-ended problem-solving, even with advanced prompting. While the paper demonstrates marginal gains over the 3B baseline, it cannot verify whether the CoMAS framework can drive meaningful evolution for models already equipped with stronger inherent capabilities. For instance, Qwen2.5’s 7B variants outperform the 3B version significantly in code generation and mathematical reasoning ; without testing on these larger scales, it remains unclear if CoMAS’s interaction-driven paradigm can amplify the strengths of more capable models or merely compensates for the 3B model’s baseline deficiencies.\n\n2. While the paper shows performance scales with agent count (4>2>1), it lacks critical analysis of how interaction complexity impacts evolution efficiency and omits clear scalability boundaries. For example, it does not address whether increasing agents beyond 4 leads to diminishing returns (e.g., redundant discussions) or higher communication overhead that degrades training speed. Additionally, the \"recent κ rounds\" context truncation rule is set without justification, there is no ablation on how κ values affect discussion quality or reward reliability, leaving readers unsure if the chosen parameter is optimal or generalizable.\n\n3. While the current baselines (untrained agents, MAPoRL, TTRL) help validate CoMAS’s performance against non-interaction-driven methods, the scope of comparative methods appears relatively narrow. That said, I acknowledge uncertainty about the exact number of recently _published_ works that directly align with multi-agent evolution. As such, I recommend cross-referencing feedback from other reviewers on this point: if peers highlight additional relevant methods, expanding the baseline set to include these would strengthen the paper’s claims of superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXZFWWjt8U", "reviewer": "Reviewer_qMBz", "summary": "This paper introduces CoMAS (Co-Evolving Multi-Agent Systems), a novel framework that enables LLM-based agents to achieve self-evolution through pure inter-agent interactions without external reward signals. The key innovation lies in its adversarial reward design involving three interaction patterns: solution generation, critical evaluation, and scoring. The framework employs an LLM-as-a-judge mechanism where agents act as solvers, evaluators, and scorers in a zero-sum game, creating complementary rewards that encourage both correctness and critical thinking. Using REINFORCE++ for policy optimization, CoMAS supports heterogeneous agents in a decentralized training paradigm. \n\nThe authors conduct comprehensive experiments across 7 benchmarks (GSM8K, MATH-500, HumanEval, MBPP, SciBench, GPQA, MMLU) and 4 evaluation setups (Vanilla, Consistency, AutoGen, Debate), demonstrating consistent improvements over untrained agents (up to 19.80% in AutoGen setup) and competitive or superior performance compared to baselines (MAPoRL, TTRL). Ablation studies validate the necessity of the adversarial reward formulation and reveal promising scalability with increasing agent numbers and diversity.", "questions": "1. **Baseline Fairness:** Why was MAPoRL evaluated with a rule-based verifier instead of its original reward model? How much does this change affect the comparison? Could you also compare with debate frameworks without RL training?\n\n2. **Statistical Significance:** Can you provide error bars or significance tests for the reported improvements? Some gains are modest (~1%), and it's important to know if they're statistically reliable.\n\n3. **Failure Modes:** In what scenarios does CoMAS fail? Can you provide examples of agent discussions where the adversarial mechanism breaks down?\n\n\n4. **Agent Initialization:** All agents start from the same pre-trained model in homogeneous settings. Does initializing with different checkpoints or different random seeds affect the co-evolution dynamics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Clear Motivation and Problem Definition:** The paper articulates a compelling research question inspired by human collaborative learning, distinguishing itself from existing self-evolution approaches that focus on individual agents.\n\n2. **Novel Adversarial Reward Design:** The zero-sum game between solvers and evaluators, mediated by an independent scorer, is an elegant solution that prevents reward hacking. The ablation study (Figure 4) convincingly demonstrates that removing either evaluation or scoring leads to training collapse or reward exploitation.\n\n3. **Comprehensive Experimental Validation:** \n   - 7 diverse benchmarks covering math, coding, science, and general knowledge\n   - 4 different inference setups from single-agent to multi-agent collaboration\n   - Comparison with strong baselines (MAPoRL, TTRL)\n   - Extensive ablation studies on reward formulation and scalability\n\n4. **Scalability and Generalizability:** \n   - Performance improves with more agents (Section 4.3.2)\n   - Heterogeneous agents outperform homogeneous ones\n   - Decentralized training enables flexible system design\n   - Skills transfer across domains\n\n5. **Stable Training Dynamics:** Figure 3 shows that rewards converge around 0.5 and response lengths increase consistently, indicating healthy learning progress without collapse.\n\n6. **Strong Performance:** CoMAS achieves state-of-the-art or competitive results across most settings, particularly excelling in multi-agent setups (e.g., 19.80% improvement on AutoGen).", "weaknesses": "1. **Limited Baseline Comparisons:**\n   - MAPoRL baseline uses rule-based verifier instead of the specialized reward model from the original paper, which may not be a fair comparison\n   - Missing comparisons with other recent self-evolution methods mentioned in related work (e.g., REMA, Self-Rewarding LMs)\n   - Would benefit from comparison with debate-only methods without RL training\n2. **Scalability Limitations:**\n   - Experiments only go up to 4 agents; unclear how the framework scales to 10+ agents\n   - All experiments use 3B parameter models; performance with larger models (7B, 13B) is unknown\n   - Training cost analysis is missing (compute requirements, wall-clock time)\n3. **Generalization Questions:**\n   - All training data comes from verifiable domains (math, coding, science); unclear if the framework works for truly open-ended tasks like creative writing or complex reasoning\n   - The framework assumes problems have objectively evaluable solutions, limiting applicability", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)