Why Do Multi-Agent LLM Systems Fail?
Authors: name: Mert Cemri, name: Melissa Z. Pan, name: Shuyi Yang, name: Lakshya A. Agrawal, name: Bhavya Chopra, name: Rishabh Tiwari, name: Kurt Keutzer, name: Aditya Parameswaran, name: Dan Klein, name: Kannan Ramchandran, name: Matei Zaharia, name: Joseph E. Gonzalez, name: Ion Stoica

Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
agents collaborate to accomplish tasks, their performance gains across popular
benchmarks remain minimal compared to single-agent frameworks. This gap
highlights the need to analyze the challenges hindering MAS effectiveness.
  In this paper, we present the first comprehensive study of MAS challenges. We
analyze five popular MAS frameworks across over 150 tasks, involving six expert
human annotators. We identify 14 unique failure modes and propose a
comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
emerges iteratively from agreements among three expert annotators per study,
achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
organized into 3 categories, (i) specification and system design failures, (ii)
inter-agent misalignment, and (iii) task verification and termination. To
support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
explore if identified failures could be easily prevented by proposing two
interventions: improved specification of agent roles and enhanced orchestration
strategies. Our findings reveal that identified failures require more complex
solutions, highlighting a clear roadmap for future research. We open-source our
dataset and LLM annotator.