Evaluate, Scale, and Credit: A Comprehensive Study on Multi-Agent Collaboration of Large Language Models
Authors: Hao Xiang, Yaojie Lu, Hongyu Lin, Jia Zheng, Sirui Wang, Xunliang Cai, Ben He, Xianpei Han, Le Sun


Reviewer paper summaries: This paper investigates the collaboration mechanisms among multiple Large Language Model-based agents (LLM-MAS). The study addresses three core questions: whether multi-agent systems can outperform single-agent systems, the benefits of scaling multi-agent systems, and how to credit and structure these agents effectively. Five collaboration architectures were designed and evaluated across different models and tasks, revealing insights into system performance, scalability, and cost efficiency. The research concludes that multi-agent systems generally outperform single-agent systems, scaling increases benefits, and pruning based on agent contribution using Shapley value can enhance performance while reducing communication costs. These findings aim to advance the understanding and development of efficient LLM-MAS systems.
Summary of strengths: 1. Innovative Crediting Mechanism: The introduction of a Shapley value-based approach for crediting agent contributions and pruning communication paths is a novel and effective method for optimizing MAS performance and reducing costs.

2. Insightful Findings: The study provides significant insights into the benefits of multi-agent systems, the importance of agent agreement, and potential effective structures, contributing to the broader understanding of MAS collaboration mechanisms.
Summary of weaknesses: 1. RQ1 and RQ2 have been explored in various papers about multi-agents. The novelty of this empirical study is lacking.
2. Though there are many conclusions, there is no explanation for all of them.
3. ChatGPT, the most popular and basic baseline LLM is not explored in this paper.
Comments, suggestions and typos: 1. '.' should be at the end of Line 126.
2. Line 264 and Line 408 should be subsections.
Overall assessment scores: 2.5