CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
Authors: name: Xiangyuan Xue; name: Yifan Zhou; name: Guibin Zhang; name: Zaibin Zhang; name: Yijiang Li; name: Chen Zhang; name: Zhenfei Yin; name: Philip Torr; name: Wanli Ouyang; name: Lei Bai
Published: 2025-10-09 17:50:26

Abstract: Self-evolution is a central research topic in enabling large language model
(LLM)-based agents to continually improve their capabilities after pretraining.
Recent research has witnessed a transition from reinforcement learning
(RL)-free to RL-based methods. Current RL-based methods either rely on dense
external reward signals or extract intrinsic reward signals from LLMs
themselves. However, these approaches diverge from the self-evolution
mechanisms observed in human intelligence, where individuals learn and improve
through mutual discussion and collaboration. In this work, we introduce
Co-Evolving Multi-Agent Systems (CoMAS), a novel framework that enables agents
to improve autonomously by learning from inter-agent interactions without
external supervision. CoMAS generates intrinsic rewards from rich discussion
dynamics, employs an LLM-as-a-judge mechanism to formulate these rewards, and
optimizes each agent's policy through RL, thereby enabling decentralized and
scalable co-evolution. Experimental results demonstrate that CoMAS consistently
outperforms untrained agents and achieves state-of-the-art performance across
most evaluation settings. Ablation studies confirm the necessity of
interaction-based reward signals and reveal promising scalability as the number
and diversity of agents increase. These findings establish CoMAS as a novel and
effective paradigm for self-evolution in LLM-based agents.