MultiAgentBench: Evaluating the Collaboration and Competition of LLM
  agents
Authors: name: Kunlun Zhu, name: Hongyi Du, name: Zhaochen Hong, name: Xiaocheng Yang, name: Shuyi Guo, name: Zhe Wang, name: Zhenhailong Wang, name: Cheng Qian, name: Xiangru Tang, name: Heng Ji, name: Jiaxuan You

Large Language Models (LLMs) have shown remarkable capabilities as autonomous
agents, yet existing benchmarks either focus on single-agent tasks or are
confined to narrow domains, failing to capture the dynamics of multi-agent
coordination and competition. In this paper, we introduce MultiAgentBench, a
comprehensive benchmark designed to evaluate LLM-based multi-agent systems
across diverse, interactive scenarios. Our framework measures not only task
completion but also the quality of collaboration and competition using novel,
milestone-based key performance indicators. Moreover, we evaluate various
coordination protocols (including star, chain, tree, and graph topologies) and
innovative strategies such as group discussion and cognitive planning. Notably,
gpt-4o-mini reaches the average highest task score, graph structure performs
the best among coordination protocols in the research scenario, and cognitive
planning improves milestone achievement rates by 3%. Code and datasets are
public available at https://github.com/MultiagentBench/MARBLE.