Scaling Large Language Model-based Multi-Agent Collaboration

## Abstract

Abstract Pioneering advancements in large language model-powered agents have underscored the design pattern of multi-agent collaboration, demonstrating that collective intelligence can surpass the capabilities of each individual.
Inspired by the neural scaling law, which posits that increasing neurons leads to emergent abilities, this study investigates whether a similar principle applies to increasing agents in multi-agent collaboration.
Technically, we propose m ulti-agent c ollaboration net works ( MacNet ), which utilize directed acyclic graphs to organize agents and streamline their interactive reasoning via topological ordering, with solutions derived from their dialogues.
Extensive experiments show that MacNet consistently outperforms baseline models, enabling effective agent collaboration across various network topologies and supporting cooperation among more than a thousand agents.
Notably, we observed a small-world collaboration phenomenon , where topologies resembling small-world properties achieved superior performance. Additionally, we identified a collaborative scaling law , indicating that normalized solution quality follows a logistic growth pattern as scaling agents, with collaborative emergence occurring much earlier than previously observed instances of neural emergence. The code and data will be available at https://github.com/OpenBMB/ChatDev .

## 1 Introduction

In the rapidly advancing field of artificial intelligence, large language models (LLMs) have catalyzed transformative shifts across numerous domains due to their remarkable linguistic capacity to seamlessly integrate extensive world knowledge Vaswani et al. (); Brown et al. (); Bubeck et al. ().
Central to this breakthrough is the neural scaling law that fosters emergent capabilities, where well-trained neural networks often exhibit power-law scaling relations primarily with the number of neurons, alongside factors such as dataset size and training time Kaplan et al. (); Schaeffer et al. (); Muennighoff et al. ().

Figure: Figure 1: Given a task, multi-agent collaboration networks (MacNet) utilize directed acyclic graphs to organize diverse agents for collaborative interactions, with the final solution derived from their dialogues.
Refer to caption: x1.png

Despite this, LLMs have inherent limitations in enclosed reasoning, particularly when addressing complex situations that extend beyond textual boundaries Richards ().
To this end, subsequent studies have successfully transformed foundational LLMs into versatile autonomous agents by equipping them with advanced capabilities such as tool use Schick et al. (), long-context memory Park et al. (), and procedural planning Wei et al. ().
Along this line, multi-agent collaboration has emerged as an effective paradigm to integrate the specialities of different agents Park et al. (); Li et al. (); Qian et al. ().
Through linguistic interaction, agents engage in instructive and responsive utterances to foster high-quality collaboration, leading to final solutions(^1^11Solutions can range from a multiple-choice answer to repository-level code or a coherent narrative, among numerous other possibilities.) derived from their dialogues Qian et al. (, ); Chen et al. ().

Inspired by the neural scaling law, a natural question arises: does increasing agents in multi-agent collaboration exhibit emergent capabilities?
Investigating the cooperative scaling law is essential for accurately estimating the relationship between computing resources and performance trends in multi-agent systems. This understanding enables the optimization of resource utilization and the minimization of unnecessary waste, ultimately leading to more scalable, practical, and resource-efficient agent systems Kaplan et al. ().
However, effective multi-agent collaboration transcends the mere aggregation of responses from different agents through majority voting Chen et al. (); instead, it constitutes an organically integrated system that requires task-oriented interactions and thoughtful decision-making Hopfield ().

In this paper, as illustrated in Figure 1, we envision multiple agents as a well-organized team composed of specialized agents, investigating their interdependent interactive reasoning and collective intelligence in autonomously solving complex problems. To further this goal, we design appropriate topologies and effective interaction mechanisms that align with both the static organizational structure and the dynamic reasoning process.

- $\bullet$
To ensure generalizability, we design the topology as a directed acyclic graph where each edge is managed by a supervisory instructor issuing directional commands, and each node is supported by an executive assistant providing tailored solutions. This mechanism effectively fosters a division of labor among agents through functional dichotomy, seamlessly integrating a static topology with specialized agents to form a multi-agent collaboration network (MacNet).
- $\bullet$
To facilitate agents’ interactive reasoning, the interaction sequence is orchestrated via topological ordering, ensuring orderly information transmission throughout the network. Within this arrangement, each interaction round involves two adjacent agents refining a previous solution, with only the refined solution, rather than the entire dialogue, being propagated to the next neighbors. This mechanism strategically avoids global broadcasts and significantly reduces the risk of overly extended contexts, enabling scalable collaboration across nearly any large-scale network.

We conducted a comprehensive quantitative evaluation of three prevalent topologies—chain, tree, and graph—divided into six variants, across multiple heterogeneous downstream scenarios.
Extensive experiments demonstrate that MacNet consistently outperforms all baseline , enabling effective agent collaboration even in fully-connected dense networks, supporting cooperation among more than a thousand agents.
Notably, we observed a small-world collaboration phenomenon, where topologies resembling small-world properties demonstrated superior performance.
Additionally, we identified a collaborative scaling law, revealing that normalized solution quality follows a logistic growth pattern as scaling agents.
Meanwhile, collaborative emergence can be observed occurring significantly earlier compared to previous instances of neural emergence.
We hope our findings offer valuable insights into resource prediction and optimization to enhance the efficiency and scalability of LLM systems Kaplan et al. ().

## 2 Related Work

Trained on vast datasets and capable of manipulating billions of parameters, LLMs have become pivotal in natural language processing due to their seamless integration of extensive knowledge Brown et al. (); Bubeck et al. (); Vaswani et al. (); Radford et al. (); Touvron et al. (); Wei et al. (); Shanahan et al. (); Chen et al. (); Brants et al. (); Chen et al. (); Ouyang et al. (); Yang et al. (); Qin et al. (); Kaplan et al. ().
Central to this breakthrough is the neural scaling law, which posits that loss scales as a power-law with model size, dataset size, and the amount of compute used for training Kaplan et al. (); Smith et al. (); Muennighoff et al. (). The principle underscores that scaling up language models can lead to emergent abilities—where performance experiences a sudden leap as the model scales Wei et al. (); Schaeffer et al. ().

Despite these, LLMs have inherent limitations in enclosed reasoning, motivating subsequent studies to effectively equip LLMs with advanced capabilities such as role playing Li et al. (); Chan et al. (), tool use Schick et al. (); Qin et al. (), long-context memory Park et al. (); Wang et al. (), and procedural planning Wei et al. (); Yao et al. (), thereby transforming fundamental LLMs into versatile autonomous agents Richards (); Shinn et al. (); Zhao et al. ().
Along this line, multi-agent collaboration has emerged as an effective paradigm to integrate the specialities of different agents Park et al. (); Zhou et al. (); Chen et al. (); Chan et al. (); Chen et al. (); Cohen et al. (); Li et al. (); Hua et al. ().
A straightforward collaboration strategy is majority voting Chen et al. (), where individuals remain independent; however, more effective multi-agent collaboration should form an integrated system that fosters interdependent interactions and thoughtful decision-making Li et al. (); Chen et al. (); Piatti et al. ().
Based on this, pioneering studies have dichotomized the functionality of agents into two distinct roles: instructors, who provide directional instructions, and assistants, who respond with tailored solutions; these agents engage in instructive and responsive utterances to foster an interaction chain, collaboratively arriving at final solutions derived from their dialogues Qian et al. (); Li et al. ().
This paradigm facilitates a well-orchestrated workflow for task-oriented interactions, significantly reducing the need for manual intervention while demonstrating promising quality. Chen et al. (); Chan et al. ().

Figure: Figure 2: Representative topological structures.
Refer to caption: x2.png

## 3 Multi-Agent Collaboration Network

We aim to establish a scalable framework for multi-agent collaboration, comprising two key components: the design of a multi-agent collaboration network (MacNet) and collaborative reasoning.

### 3.1 Network Construction

To establish a organational structure for multi-agent collaboration that is both efficient and scalable, drawing on the concept of graphs—a data structure that describes entities and their interrelations, we model the topology as a directed acyclic graph (DAG) Nilsson et al. () to organize interactions among collaborative agents Qian et al. ().
Concretely, a feasible topology is denoted as $\mathcal{G}=(\mathcal{V},\mathcal{E})$:

$$ $\displaystyle\mathcal{V}\!=\!\{v_{i}|i\!\in\!I\},\ \mathcal{E}\!=\!\{\langle v _{i},v_{j}\rangle|i,j\!\in\!I\wedge i\!\neq\!j\}$ (1) $$

where $\mathcal{V}$ denotes the set of nodes indexed by $I$, and $\mathcal{E}$ denotes the set of edges, with each edge directed from one node to another and no cycles exist.

Given the impracticality of enumerating all possible topologies, our study focus on three prevalent types—chain, tree, and graph—further divided into six structures, as depicted in Figure 3.
Chain topologies, resembling the waterfall model Petersen et al. (), linearly structuring interactions along agents.
Tree topologies enable agents to branch out, interacting in independent directions; further categorized into "wider" star-shaped and "deeper" tree-shaped structures.
Graph topologies support arbitrary interaction dependencies, with nodes having multiple children and parents, forming either divergent and convergent interactions; further classified into fully-connected mesh structures, MLP-shaped layered structures, and irregular random structures.
These representative topologies are extensively studied in complex network Strogatz (); Albert and Barabási () and LLM agent reasoning Liu et al. (); Besta et al. (), ensuring a comprehensive examination of the most significant and practical structures in understanding multi-agent systems.

In the ecosystem of LLM-powered agents, a functional dichotomy Li et al. ()—consisting of supervisory instructors who issue directional instructions and executive assistants who provide tailored solutions—can effectively promote division of labor, stimulate functional behaviors, and facilitate efficient task resolution Qian et al. (, ).
To integrate this strategy into the topology, as depicted in Figure 3, we strategically assign an instructor to each edge and an assistant to each node:

$$ $\displaystyle\bm{a_{i}}$ $\displaystyle=\rho(v_{i}),\ \forall v_{i}\in\mathcal{V}$ (2) $\displaystyle\bm{a_{ij}}$ $\displaystyle=\rho(\langle v_{i},v_{j}\rangle),\ \forall\langle v_{i},v_{j} \rangle\in\mathcal{E}$ $$

where $\rho(x)$ represents the agentization operation on an element $x$, achieved by equipping a foundation model with professional roles Li et al. (), external tools Schick et al. (), and context-aware memory Park et al. (); $\bm{a_{i}}$ and $\bm{a_{ij}}$ denote an assistant agent assigned to node $v_{i}$ and an instructor agent assigned to edge $v_{ij}$, respectively.

This dichotomous design allows agents to specialize in their functions, driving task-oriented language interactions and facilitating efficient information transmission throughtout the network. Additionally, the "directed" nature of the edges enables the orchestration of agent interactions, while the "acyclic" configuration prevents information propagation deadlocks.

Figure: Figure 4: Streamlining the agents’ reasoning process involves a series of dual-agent interactions. The topological order guides the interaction sequence, while the original connectivity governs the data flow.
Refer to caption: x4.png

### 3.2 Interactive Reasoning

In the process of completing complex tasks, interactive reasoning among agents within a static MacNet requires strategical traversal to establish an orderly interaction sequence.
Our graph traversal strategy adheres to the principles of topological ordering Bondy and Murty (), a fundamental algorithm in graph theory, which ensures that each node is visited only after all its dependencies have been traversed Gross et al. ().
Formally, for a MacNet $\mathcal{G}$, its topological order is a linear arrangement of agents $\bm{a_{i}}$ and $\bm{a_{ij}}$ such that for every directed edge $\langle v_{i},v_{j}\rangle\in\mathcal{E}$, the ordering satisfies:

$$ $\forall\langle v_{i},v_{j}\rangle\in\mathcal{E},\ \mathbb{I}(\bm{a_{i}})< \mathbb{I}(\bm{a_{ij}})<\mathbb{I}(\bm{a_{j}})$ (3) $$

where $\mathbb{I}(x)$ denotes the index of agent $x$ in the topological sequence. This arrangement ensures that each node-occupied agent $\bm{a_{i}}$ precedes its corresponding edge-occupied agent $\bm{a_{ij}}$, and $\bm{a_{ij}}$ precedes $\bm{a_{j}}$, thereby guaranteeing ensuring orderly information transmission throughout the network.

After establishing the global order, as illustrated in Figure 4, we enable each pair of edge-connected adjacent agents to interact and exchange information. For a topology $\mathcal{G}$, the design result in a total deployment of $|\mathcal{V}|+|\mathcal{E}|$ agents and require $2|\mathcal{E}|$ interaction rounds. Within each edge, the interaction pattern between assistants and instructors follows a multi-turn instruction-response sequence:

$$ $\displaystyle\tau(\bm{a_{i},\bm{a_{ij}}},\bm{a_{j}})=\big{(}\tau(\bm{a_{i},\bm {a_{ij}}}),\tau(\bm{a_{ij}},\bm{a_{j}})\big{)}$ (4) $\displaystyle\tau(\bm{a_{i},\bm{a_{ij}}})=(\bm{a_{i}}\rightarrow\bm{a_{ij}},\ \bm{a_{ij}}\leadsto\bm{a_{i}})_{\circlearrowleft}$ $\displaystyle\tau(\bm{a_{ij}},\bm{a_{j}})=(\bm{a_{ij}}\rightarrow\bm{a_{j}},\ \bm{a_{j}}\leadsto\bm{a_{ij}})_{\circlearrowleft}$ $$

where $\rightarrow$ symbolizes the act of instructing, $\leadsto$ indicates the corresponding responding, and $\circlearrowleft$ represents the iterative nature of the process.
Specifically, $\bm{a_{i}}$ requests feedback, $\bm{a_{ij}}$ offers optimization suggestions and requests further refinement, and $\bm{a_{j}}$ provides the refined solution. Thus, the agents associated with a single edge can effectively optimize a solution in one iteration.

Delving deeper, the topological ordering methodically unfolds agent interactions into an interaction sequence, outlining the control flow(^2^22Note that although the interaction order is unfolded as a sequence for visualization purposes only, certain substructures (e.g., star-structured topology) inherently support parallel processing, which is essential in enhancing the reasoning efficiency of practical systems.) within a multi-agent collaboration process. Concurrently, the data flow within this process is consistent with the original dependencies connected by edges, ensuring that the flow of interacted information aligns with the inherent dependencies outlined in the topology.

### 3.3 Memory Control

In a multi-agent collaboration system, unrestrained context information exchange can lead to excessively long contexts, ultimately limiting scalability by supporting only a few agents. To address this, we adopt a heuristic mechanism Qian et al. () to manage context visibility using short-term and long-term memory. Short-term memory captures the intra-interaction working memory during each dual-agent interaction, ensuring context-aware decision-making. Long-term memory maintains inter-interaction context continuity by transmitting only the final solutions derived from dialogues, not the entire conversational history.
This approach ensures that the context of ancestor agents remains Markovian, with solutions propagated only from adjacent agents rather than from all previous dialogues. Consequently, it reduces the risk of context overload while preserving context continuity, thereby enabling scalable multi-agent collaboration across nearly any large-scale network.

Furthermore, an original solution propagating through the network undergoes continuous refinement, improving its quality over time. As solutions traverse the network, they either branch off at divergent nodes or aggregate at convergent nodes. Branching is achieved through parallel propagation, while merging from multiple nodes, akin to a non-linear perceptron, requires an effective aggregation mechanism. Technically, convergent agents assess the strengths and weaknesses of each solution, synthesizing their strengths and discarding weaknesses, which results in a strength-aggregated outcome from "non-linear" decision-making, rather than a simple combination of all solutions.

## 4 Experiments

#### Baselines

We select different kinds of representative methods for quantitative comparison.

- $\bullet$
CoT Wei et al. () is a technically general and empirically powerful method that endows LLMs with the ability to generate a coherent series of intermediate reasoning steps, naturally leading to the final solution through thoughtful thinking and allowing reasoning abilities to emerge.
- $\bullet$
AutoGPT Richards () is a versatile single-agent system that employs multi-step planning and tool-augmented reasoning to autonomously decompose complex tasks into chained subtasks and iteratively leverages external tools within an environment-feedback cycle to progressively develop effective solutions.
- $\bullet$
GPTSwarm Zhuge et al. () formalizes a swarm of LLM agents as computational graphs, where nodes represent manually-customized functions and edges represent information flow, significantly surpassing the tree-of-thought method by optimizing node-level prompts and modifying graph connectivity.
- $\bullet$
AgentVerse Chen et al. () recruits and orchestrates a team of expert agents in either a horizontal or vertical topological structure, employing multi-agent linguistic interaction to autonomously refine solutions and demonstrating emergent performance compared to individual agents, serving as both general and powerful multi-agent framework.

#### Datasets and Metrics

We adopt publicly available and logically challenging benchmarks to evaluate across heterogeneous downstream scenarios.

- $\bullet$
MMLU Hendrycks et al. () provides a comprehensive set of logical reasoning assessments across diverse subjects and difficulties, utilizing multiple-option questions to measure general world knowledge and logical inference capabilities. We assess the quality of generated solutions via accuracy, reflecting the correctness of responses to multiple-choice questions.
- $\bullet$
HumanEval Chen et al. (), a widely recognized benchmark for function-level code generation, desined for measuring basic programming skills. We assess via pass@k, reflecting function correctness across multiple standard test cases.
- $\bullet$
SRDD Qian et al. () integrates complex textual software requirements from major real-world application platforms, designed for repository-level software development, including requirement comprehension, system design, and integration testing. We assess using quality, a comprehensive metric that integrates crucial factors including completeness, executability, and consistency.
- $\bullet$
CommonGen-Hard Madaan et al. () requires models to generate coherent sentences incorporating discrete concepts, designed to test systems’ advanced commonsense reasoning, contextual understanding, and creative problem-solving. We assess using a comprehensive score that integrates crucial factors including grammar, fluency, context relevance, and logic consistency Li et al. (); Chen et al. ().

#### Implementation Details

By default, our method utilizes the GPT-3.5-turbo model, chosen for its optimal balance of reasoning efficacy and efficiency. We enhance the diversity of perspectives by leveraging GPT-4 to generate a pool of 4,000 profiles for assignment. These agents are equipped to autonomously use external tools (e.g., Python compilers), and their temperatures decrease linearly from 1.0 to 0.0 according to topology depths. Topological sorting is implemented via Kahn’s algorithm Kahn (). During agent interactions, a maximum of three rounds of utterances is allowed. To ensure fairness, all baselines adhere to identical hyperparameters and settings in the evaluation. All code and data will be publicly available.

**Table 1: The overall performance of LLM-driven methods across various datasets, including both single-agent () and multi-agent () paradigms. For each dataset, the highest scores are highlighted in bold, while the second-highest scores are underlined. A dagger ($\dagger$) denotes statistically significant differences ($p\leq 0.05$) between the baseline and our chain-structured setting.**
| Method | Paradigm | MMLU | HumanEval | SRDD | CommonGen | AVG. |
| --- | --- | --- | --- | --- | --- | --- |
| CoT |  | 0.3544^† | 0.6098^† | 0.7222^† | 0.6165 | 0.5757^† |
| AutoGPT |  | 0.4485^† | 0.4809^† | 0.7353^† | 0.5972^† | 0.5655^† |
| GPTSwarm |  | 0.2368^† | 0.4969 | 0.7096^† | 0.6222^† | 0.5163^† |
| AgentVerse |  | 0.2977^† | 0.7256^† | 0.7587^† | 0.5399^† | 0.5805 |
| MacNet-Chain |  | 0.6632 | 0.3720 | 0.8056 | 0.5903 | 0.6078 |
| MacNet-Star |  | 0.4456 | 0.5549 | 0.7679 | 0.7382 | 0.6267 |
| MacNet-Tree |  | 0.3421 | 0.4878 | 0.8044 | 0.7718 | 0.6015 |
| MacNet-Mesh |  | 0.6825 | 0.5122 | 0.7792 | 0.5525 | 0.6316 |
| MacNet-Layered |  | 0.2780 | 0.4939 | 0.7623 | 0.7176 | 0.5629 |
| MacNet-Random |  | 0.6877 | 0.5244 | 0.8054 | 0.5912 | 0.6522 |

### 4.1 Does Our Method Lead to Superior Performance?

We first employ the simplest topology—chain—as the default setting for our comparative analysis. As shown in Table 1, the chain-structure method consistently outperforms all baseline methods across most metrics, demonstrating a significant margin of improvement.
The primary advantage of MacNet, compared to a single agent providing answers from a specific perspective, lies in its facilitation of a sequential process where solutions are continuously refined. This enables autonomous and incremental optimization, effectively alleviating previously imperfect solutions or false hallucinations Qian et al. (, ); Chen et al. (); Chan et al. ().
Moreover, we observe that CoT exhibits strong performance on certain datasets, even surpassing some multi-agent methods in specific cases. This is primarily because the underlying knowledge of widely-researched benchmarks is largely embedded in foundational models, giving single agents a notable capability in these relatively "simple" tasks.
Although GPTSwarm self-organizes agents through dynamic optimization of nodes and edges, it still requires extensive task-specific customization for all agents and their behaviors, making it challenging to seamlessly transfer to heterogeneous downstream tasks.
Given the increasing need for highly performant and automatic real-world systems, it is unrealistic to expect that all preparatory knowledge can be fully pre-encoded in foundation models, nor can specific adaptations be pre-made for all unforeseen complex tasks. Luckily, MacNet addresses this challenge by automatically generating various networks through simple hyperparameters (e.g., topology type and scale), without requiring additional specific adaptations, which represents a more promising paradigm for enhancing autonomy, scalability, and generalizability.

In addition, we ablate agents’ profiles and temperature, which is equivalent to graph-of-thought method—graph-guilded reasoing thoughts by a single agent who lacks a profile and has a temperature set to 0.
We find that ablating these mechanisms results in significant performance degradation across all topologies, with an average decrease of 2.69%.
This highlights the superior collective intelligence over any form of reasoning by a single agent, as the latter corresponds to a feature dimension reduction of the high-dimensional multi-agent combination space, which solidifies reasoning ability due to the lack of flexibility to explore a better configuration.

### 4.2 How Do Different Topologies Perform Against Each Other?

To understand the topological properties, we conducted extensive experiments by altering MacNet’s topologies. The results in Table 1 demonstrate that different topologies exhibit varying levels of effectiveness for distinct tasks. For instance, a chain topology is more suitable for software development, while a mesh topology excels in logical selection. No single topology consistently delivers optimal results across all tasks.
Further observation reveals that topologies approaching the small-world property Watts and Strogatz ()—characterized by a small average path length(^3^33Average path length Albert and Barabási () is the average number of steps along the shortest paths for all possible pairs of network nodes, which is a measure of the efficiency of information transport on a network.)—tend to exhibit superior performance, which we refer to as the "small-world collaboration phenomenon".
Concretely, as each edge in MacNet triggers agent interactions, the graph’s density naturally represents the agents’ interaction density.
Empirically, higher interaction density is associated with improved performance among the three coarse-grained topological types.(^4^44For example, the densely connected mesh topology outperforms the moderately dense tree topology, which in turn outperforms the sparsely connected chain topology.)
This performance discrepancy can be attributed to the fact that a higher graph density generally correlates with a higher clustering coefficient(^5^55The clustering coefficient measures how densely connected a node’s neighbors are to each other Strogatz ().). This increase in clustering coefficient results in more adjacent node pairs, decreasing the average path length; consequently, the likelihood of long-distance solution invisibility is correspondingly decreased.
Along this reason, we also discover that irregular random structures outperform regular mesh structures. This advantage can be attributed to random edge connections, which, in analogy to social networks, potentially link "unacquainted" agents via a direct shortcut, making them adjacent "acquaintances" and implicitly reducing the average path length, thus resembling small-world properties.
Meanwhile, unlike mesh topology, which exhibit the highest interaction density, random topology achieve an optimal balance between reduced arrangement depth and enhanced reasoning efficiency, making it a more suitable tradeoff in practice.

Additionally, it is observed that, given the same density, "wider" star-shaped topologies generally outperform "deeper" tree-shaped ones. This is primarily attributable to our solution propagation mechanism, which inhibits the propagation of excessively long contextual reasoning processes throughout the entire network. As a result, deeper topologies may cause agents to lose sight of farther contexts, potentially leading to version rollback—solutions revert to earlier or similar versions.
The same principle applies to graph structures, in which mesh topologies, compared to layered ones, enable direct reasoning between agents through direct edges, thereby implicitly reducing network depth and enhancing performance.

In addition to the structural point of view, the directional characteristics of some topologies, which exhibit inherent asymmetry—reversing the edges results in an entirely unequal one—motivated us to explore reverse topologies. As shown in Figure 5, merely altering the symmetry topologies’ orientation leads to significant performance degradation. Typically, divergent structures (those with more child nodes than parent nodes) significantly outperform convergent counterparts.
Intuitively, solution flow smoothly diverges, allowing each agent to propose solutions from varied perspectives concurrently; conversely, converging the solutions of multiple agents at a single point poses a greater challenge, illustrating the complexity involved in integrating diverse perspectives into a cohesive strategy.

Figure: Figure 5: The average performance of the divergent topology (default) and its convergent counterpart.
Refer to caption: x19.png

Figure: Figure 6: Scaling performance of multi-agent collaboration under different topologies.
Refer to caption: x20.png

### 4.3 Does a Collaborative Scaling Law Exist?

Recall that the neural scaling law fosters emergent capabilities Kaplan et al. (); Schaeffer et al. (); Muennighoff et al. (), where the synergy among numerous neurons enables a continuous trend of performance improvement. To investigate the collaborative scaling law—the potential predictable relationship between agent scale and performance, considering the associated time and economic costs—we scaled different topologies by exponentially increasing the number of nodes from 1 (regressing to a single-agent method) to 50 (corresponding to 1,275 agents on a mesh setting). As shown in Figure 6, our results confirm the small-world collaboration phenomenon, where optimal outcomes are achieved in high-density networks. Additionally, a reverse degradation phenomenon can be also observed, where certain configurations led to an overall quality reduction ranging from 2.27% to 6.24%.

As the topology scales, the quality of solutions produced by the multi-agent system initially rises rapidly before reaching a saturation point (or slightly declining), which can be approximated by a sigmoid-shaped function:

$$ $f(x)=\frac{\alpha}{1+e^{-\beta(x-\gamma)}}+\delta$ (5) $$

where $\alpha$, $\beta$, $\gamma$ and $\delta$ being real numbers specific to a topology. It is important to emphasize that this is only an average characterization based on scale; a more precise multi-agent system should consider additional factors (e.g., foundation models, profile, and tool spaces).
Notably, neural scaling laws typically require a million-fold increase in neurons to reveal significant trends around a scale of $10^{18}$ to $10^{24}$ Schaeffer et al. (). In contrast, most topologies in MacNet exhibit performance saturation around a scale of $2^{4}$ to $2^{5}$. This collaborative emergence occurs more rapidly compared to neural emergence and is observable at smaller scales. The underlying reason is that neuron coordination, relying on from-scratch training in latent space via matrix operations, requires a vast scale to incorporate extensive world knowledge and develop learning capabilities. In contrast, agent coordination, based on the implicit knowledge of pretrained LLMs, leverages the understanding and refinement of textual information through linguistic interactions, often circumventing the extensive scaling needed by neuronal coordination. Combining these two scaling mechanisms at different levels holds promise for producing higher-quality outcomes.

## 5 Conclusion

We have introduced MacNet, which leverages DAGs to structure the agents’ cooperative topologies and streamline their interactive reasoning through topological ordering, with solutions derived from their dialogues.
Extensive experiments demonstrate that MacNet consistently outperforms all baseline models, enabling effective agent collaboration across various topologies.
Notably, we observed a small-world collaboration phenomenon, where topologies resembling small-world properties demonstrated superior performance.
Additionally, we identified a collaborative scaling law, revealing that normalized solution quality follows a logistic growth pattern as scaling agents.
Meanwhile, collaborative emergence can be observed occurring significantly earlier compared to previous instances of neural emergence.
We hope our findings offer valuable insights into resource prediction and optimization to enhance the efficiency and scalability of LLM systems.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: I find the modeling of MAS LLM agent and the study of the scaling law in the paper quite intriguing, but I have reservations about the problems that the multi-agent is used to solve. 

In the experiments within the paper, single-agent methods like CoT and AUTOGPT are compared, but to my knowledge, for datasets like MMLU, the pure LLMs (without designing agents deliberately) can already achieve a performance of 0.8 as can be check in [paperswithcode](https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu), and for HumanEval, based on [1],[2], the performance of above 0.8 can also be achieved.  The works I mentioned above are all better than the baselines and this method MACNET.

So, if using a multi-agent collaboration work, considering ten to thousands of agents, only achieves the performance shown in Table 1, I don't think it proves the necessity of multi-agent; in fact, it could be seen as undermining their necessity. 

- Compare MACNET against these state-of-the-art single-agent methods and Explain why MACNET underperforms compared to these methods, if that is indeed the case.
- Can you discuss specific scenarios or types of tasks where multi-agent approaches might offer unique advantages over single-agent methods?

[1] Zhong, Li, Zilong Wang, and Jingbo Shang. "Ldb: A large language model debugger via verifying runtime execution step-by-step." arXiv preprint arXiv:2402.16906 (2024).

[2] Shi, Yuling, et al. "From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging." arXiv preprint arXiv:2410.01215 (2024).
soundness: 4
strengths: - The paper introduces a pioneering framework that employs directed acyclic graphs (DAGs) to organize agents within a multi-agent system. It provides insightful comparisons between various topologies.

- The research delves into the impact of scaling the number of collaborative agents on overall performance, uncovering a collaborative scaling law. This law suggests that performance follows a logistic growth pattern as agents are added, which is a significant finding as it parallels the neural scaling law but applies to the domain of agent collaboration.
summary: The paper introduces a novel multi-agent collaboration framework, MACNET, which utilizes directed acyclic graphs (DAGs) to organize agents and orchestrate their reasoning for autonomous task solving. In their extensive evaluations, the paper finds that multi-agent collaboration can achieve better performance than the individure agent and also reveals a collaborative scaling law, indicating that performance follows a logistic growth pattern as the number of agents increases. The study also demonstrates the superior performance of irregular topologies over regular ones in multi-agent collaboration, attributing this to the catalyst effect of scaling agents on multidimensional considerations during interactive reflection and refinement.
weaknesses: Check in Questions.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Beyond the major concerns I list, there are the following questions: 
1. Can the authors provide mathematical insights regarding the implications of multiple LLMs collaborating? From a mathematical theory standpoint, what is the difference between increasing the size of large models and increasing the number of large models collaborating? 
2. The abstract part mentions "with collaborative emergence occurring earlier than traditional neural emergence". How should this be understood? Can the authors provide a detailed discussion?
soundness: 3
strengths: 1. The paper is clearly articulated and well-structured, with the discussion on "why random topologies perform better" and "tail aspects being factors leading to the emergence of collaboration" being particularly enlightening.
2. The experimental section of the paper is comprehensive (although there may be areas lacking in rigor), covering a wide range of perspectives in its design. I believe the work presented in this paper can inspire the entire research community.
summary: The paper explores the scaling law of collaboration for LLM-based agents. The authors propose a paradigm, MacNet, that allows for the collaboration of up to thousands of agents, and demonstrate its superiority over baseline methods across four diverse datasets. Utilizing MacNet, the authors investigate the impact of different topological structures on multi-agent collaboration, distill a scaling law concerning the number of nodes in multi-agent graphs, and analyze the factors leading to the emergence of multi-agent collaboration through case studies.
weaknesses: As an empirical research paper, the experimental design and conclusions of the paper are particularly important and need to be sufficiently rigorous. I currently have some concerns.

(Major) 1. Regarding the emergence in multi-agent systems. The paper discusses the laws of neural scale emergence and agent collaboration emergence between lines 427 and 429. However, the claim of collaboration emergence might not be rigorous enough. For instance, in neural emergence, a neural network model can evolve from merely recognizing images to handling PhD-level problems, which is a significant leap. In contrast, the emergence described in this paper only involves an improvement in success rate from around 55% to 70% on certain datasets, a particularly small span that might not even qualify as emergence. If an appropriate case could be found where multiple agents attempt to solve a very difficult problem, and the agents manage to solve this problem once a certain scale is reached (analogous to human society solving a very complex and challenging problem), it would greatly enhance the paper.

(Major) 2. Corresponding to the above is the selection of benchmarks and the design of metrics. A fundamental question is why, even with the use of up to thousands of agents, the performance improvement is only around 20%, without a breakthrough improvement akin to advancing from grade school level to PhD level. Another phenomenon is that, as mentioned in section 3.4 of the paper, while the number of Interacted Aspects continues to increase when the number of agents ranges from 2^5 to 2^6, there is no significant improvement in performance. Does this imply that the current tasks are not challenging enough? Are the metrics for quantifying performance not rich enough?

(Minor) The paper mentions " alongside neuron collaboration, agent collaboration may serve as a shortcut to enhance intelligence levels, especially when large-scale retraining resources such as data and hardware are constrained." To arrive at this general conclusion, conducting experiments only on the same series of large models (GPT3.5 and GPT4) might be insufficient. Experiments with other types of LLMs might be necessary.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: Questions are asked in above sections.
soundness: 4
strengths: The scaling of multi-agent systems powered by LLMs is a significant research question for the community.

The idea of organizing LLM agents using a DAG is not novel; however, the authors present a comprehensive analysis of factors including types of topologies, densities, directions, and scales. 

The experiments involve at maximum of 1000 LLM agents, with the help of the proposed memory control mechanism to avoid context explosion.

The proposed method is highly flexible and generalizable, requiring little human supervision or prior knowledge as claimed by the authors.
summary: This paper presents pioneering work on scaling large language model-based multi-agent collaboration to the level of a thousand agents. The authors propose a method to automatically organize LLM agents into a directed acyclic graph (DAG) without needing specific adjustments for generalization. Comprehensive evaluations are conducted on diverse benchmarks against a variety of baselines. The influence of topologies, the number of agents, and other factors on collaboration performance is discussed in depth. The paper is well-written with a clear flow. The scaling law experiments involving over a thousand agents represent a truly rare finding in the literature I have read about LLM agents. I would highly recommend accepting this paper.
weaknesses: What is the rationale for using "instructor" for edges and "executor" for nodes? It seems that both can be treated as nodes with different profiles, with edges between them indicating interactions.

Implementation details are somewhat brief. As mentioned in lines 259-261, agent profiles and tools are generated by GPT-4 and randomly assigned to network agents. It is unclear how this process can be fully automated without interventions or adjustments at the scale of 1,000+ agents. Is there any guarantee that the randomly generated agent network can solve the given problem?

How can we ensure that the memory control mechanism does not omit context information? Additionally, how many agent profiles and tools are generated? 

How many repeated measurements are conducted for each method across different datasets? What is the variance in performance?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Doesn't the interative feedback between instructors and executors make the graph not directed? You could get stuck in a loop there.

- Have you looked into letting the agents attempt to determine what a good topology is? I.e. Agents specify to which other agent they believe they should send their result.

- If quality is the average across all tasks, does each benchmark have the same amount of tasks? Or did you average specifically task-wise, or across all 4 datasets?
soundness: 3
strengths: # originality 

* The authors study the scaling of the number of agents during inference, a relatively **novel, unexplored problem**. 

* The authors **extend the interaction DAG setting** to a functionally bipartite structure with agents as nodes and edges, and **present representative topologies** that capture prevalent DAG structures 

* The authors **address the limitations of prior work** by avoiding content explosion

# quality 

* The authors describe the problem setting with the appropriate technical language, as well as **motivating theoretical justifications** for MacNet's advantages over existing solutions.

* The authors compare against a set of **timely and relevant baselines** on ** diverse datasets**.

* The authors use **identical and reproducible settings** across methods.

* The authors provide **multiple perspectives** on their results, connecting observed phenomena to novel and documented insights.

# clarity

* The paper is **well-structured**, making it easy to read and follow along.

* The paper uses **interesting visualizations** to communicate interesting insights, such as longer tails with more agents.

* The paper employs **precise formal language and notation** to effectively communicate their points.

# significance

* The paper has a significant impact on **LLM-based applications**

* The paper is very **timely** as the research on multi-agent systems begins to regain the interest of the machine learning community.

* The paper offers **interesting insights** into collaborative inference-time structures across many agents, which is relevant to ongoing discussions around agent proliferation.

* The paper outlines **future directions of work**.
summary: The paper proposes MacNet, a multi-agent collaboration network. Inspired by neural scaling laws, the authors investigate how the number of agents and the communication topology between agents impacts performance when agents are made to collaborate on a given problem. Specifically:

- MacNet creates a Directed Acyclic Graph (DAG) of agents that attempt to solve a given task in analogy to a neural network. The **nodes and edges are agents**, where the nodes are "executors", which accomplish tasks, while the edges are "instructors", which evaluate the task and provide feedback. Agents are allowed to **interactively and iteratively refine** their subtask solutions, where an instructor asks an executor to refine their proposed subtask solution. To avoid **content explosion**, nodes in the network are expected to only pass on the proposed solution, not its entire reasoning, and they are also expected to aggregate incoming solutions. Importantly, MacNet does not have a specific topology; the method can be applied to many different types of topologies, including random topologies.

- To evaluate the performance of MacNet, the authors evaluate on MMLU, HumanEval, SRDD and CommonGen-Hard. They compare MacNet variants based on different DAG topologies against Chain of Thought, AutoGPT, GPTSwarm and AgentVerse. Their method exhibits significantly better performance on MMLU, mostly worse performance on HumanEval, slightly better on SRDD and sometimes better performance on CommonGen. 
  - They compare the performance of different topologies against each other, finding that random topologies often perform statistically better.
  - They observe the performance of different topologies across multiple network sizes, demonstrating a non-decreasing performance trend.
  - They perform an error analysis on one of the topologies at different sizes, finding longer tails when the number of agents increases which diversifies the solutions, making the proposed solutions more comprehensive.

Ultimately, they conclude that agent collaboration could bypass the need for retraining by enabling inference time performance scaling through scaling multi-agent collaboration.
weaknesses: # Problem Setting

* **collaborative emergence is undefined**, even though it seems to be a central concept of the paper.

# Results

## Major 
- **Lack of transparent methods**: The use of the closed source line of GPT models hinders the analysis. The authors take the stance that the multi-agent scaling laws are analogous to the neural scaling laws, but we do not know much about the underlying models. It would be more judicious to compare open-source models, for which we know the FLOPs and parameter counts. This could allow for comparisons such as a single 7B model vs a MacNet of 7 1B models. An additional point here is that we do not know what type of post-training the GPT models undergo, which could help in the collaboration. The main point of this criticism is the following: **the use of a closed-source method in your analysis effectively hinders the potential for your results to generalize to other LLM-based agents**. We don't know the impact of model size, architecture or training strategy for the GPT models, so we can't know how the established analysis and scaling laws transfer to other architectures.


- **Entangled sources of improved performance**: Since the actual FLOPs used are not kept track of, it becomes impossible to tell whether the improved results are due to additional computational resources spent on figuring out a solution. It would be desirable to **isolate the impact of the multi-agent collaboration** from other factors, such as how much compute is poured into a given solution. Otherwise, it could be that a given MacNet works better because it pours more resources and time into a solution. While this result would be interesting in itself, it would shift the significance of the approach mostly onto methods for scaling inference-time compute, at which point you'd have to normalize for how much time and compute is spent on a given solution. You would then also have to compare against other non-multi-agent methods for scaling inference time compute. Instead, the diversity and comprehensiveness of the solutions output by a MacNet could be the subject of a greater analysis, while being more direct about the resource requirements.
 
## Minor
- **Lack of scaling law analysis**: The scaling laws, although the sigmoid functional form is described, are not actually fit, nor analyzed. Furthermore, there is little to no discussion as to *why* the scaling laws should take a sigmoidal form in this case.


# Overall 

The presentation of the paper is solid. The writing is easy to understand and the use of technical language is judiciously sparse. The problem setting is good and the contribution would be too, but the soundness of the claims is affected by the limits to the results outlined above. This unfortunately impacts the value of the contribution itself, as the results are not wide-ranging.

Meta-review: This paper, using the MACNET framework, investigates the impact of scaling the number of collaborative agents on overall performance, revealing a collaborative scaling law. However, the experimental setup is significantly confusing, particularly in the implementation details, including agent profiles and tools, and the objective reporting of performance. The overall framework and idea of the paper are relatively simple and easy to understand, and the authors have effectively articulated their core contributions.

Decision: Accept (Poster)