Title: How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: multi-agent LLMs, collaborative reasoning, LLM

Abstract: Large Language Models (LLMs) have achieved strong performance on a wide range of complex reasoning tasks, yet further gains are often possible by leveraging the complementary strengths of multiple models.  While multi-agent frameworks can improve solution quality by leveraging multiple LLMs, existing methods are often computationally expensive, both at training and inference time.  In this work, we introduce a hierarchical multi-agent framework that addresses these challenges by training only a single leader LLM to coordinate a team of untrained peer agents.  To this end, we propose **M**ulti-agent guided **L**eader **P**olicy **O**ptimization (MLPO), a novel approach which trains the leader to evaluate and synthesize agent responses without auxiliary value networks or explicit agent feedback.  Leaders trained with MLPO exhibit improved performance not only when interacting with the agent team at inference time, but also enjoy improved performance when deployed in single-agent settings without the team.  Empirical results on BBH, MATH, and MMLU demonstrate that our framework achieves substantial performance improvements over both single-agent and multi-agent baselines.  Our results highlight the effectiveness and efficiency of training a single, flexible leader for collaborative reasoning in multi-agent LLM systems.

TL;DR: We propose MLPO, a hierarchical framework that trains a single leader LLM to coordinate multi-agent reasoning efficiently, achieving substantial performance gains over both single- and multi-agent baselines.