MultiAgentBench: Evaluating the Collaboration and Competition of LLM
  agents
Authors: name: Kunlun Zhu; name: Hongyi Du; name: Zhaochen Hong; name: Xiaocheng Yang; name: Shuyi Guo; name: Zhe Wang; name: Zhenhailong Wang; name: Cheng Qian; name: Xiangru Tang; name: Heng Ji; name: Jiaxuan You
Published: 2025-03-03 05:18:50

Abstract: Large Language Models (LLMs) have shown remarkable capabilities as autonomous
agents, yet existing benchmarks either focus on single-agent tasks or are
confined to narrow domains, failing to capture the dynamics of multi-agent
coordination and competition. In this paper, we introduce MultiAgentBench, a
comprehensive benchmark designed to evaluate LLM-based multi-agent systems
across diverse, interactive scenarios. Our framework measures not only task
completion but also the quality of collaboration and competition using novel,
milestone-based key performance indicators. Moreover, we evaluate various
coordination protocols (including star, chain, tree, and graph topologies) and
innovative strategies such as group discussion and cognitive planning. Notably,
gpt-4o-mini reaches the average highest task score, graph structure performs
the best among coordination protocols in the research scenario, and cognitive
planning improves milestone achievement rates by 3%. Code and datasets are
public available at https://github.com/MultiagentBench/MARBLE.