Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration
Authors: Sukwon Yun
Jie Peng
Pingzhi Li
Wendong Fan
Jie Chen
James Zou
Guohao Li
Tianlong Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Collaboration
Multi-Agent LLM
Primary area: foundation or frontier models, including LLMs

Abstract: With an ever-growing zoo of LLMs and benchmarks, the need to orchestrate multiple models for improved task performance has never been more pressing. While frameworks like Mixture-of-Agents (MoA) attempt to coordinate LLMs, they often fall short in terms of (1) selecting relevant agents, (2) facilitating effective intra-agent communication, and (3) integrating responses efficiently. In this work, we propose Graph-of-Agents (GoA), a new graph-based framework for modeling multi-agent LLM communication. Our approach begins with node sampling, selecting only the most relevant agents by leveraging model cards that summarize each model’s domain, task specialization, and other characteristics. Next, we construct edges between the selected agents by evaluating their responses against one another to determine relevance ordering. Directed message passing is then performed from highly relevant agents to less relevant ones to enhance their responses, followed by reverse message passing to refine the original responses of the more relevant agents. Finally, the updated responses are aggregated via graph-based pooling (e.g., max or mean pooling) to produce a single, unified answer. We evaluate GoA on diverse multi-domain benchmarks (MMLU, MMLU-Pro, GPQA) and domain-specific benchmarks (MATH, HumanEval, MedMCQA), with an agent pool of 6 LLMs spanning multiple domains. Surprisingly, GoA achieves superior performance18 using only 3 selected agents, outperforming recent multi-agent LLM baselines that utilize all 6 agents simultaneously. By adopting a graph structure, GoA offers both scalability and effectiveness through structured message passing—positioning it as a strong candidate for navigating the challenges of the ever-growing LLM zoo.

Published as a conference paper at ICLR 2026
GRAPH-OF-AGENTS, A GRAPH-BASED FRAMEWORK FOR MULTI-AGENT LLM COLLABORATION
Sukwon Yun'
Jie Peng'
Pingzhi Li'
Wendong Fan
Jie Chen
James Zou
Guohao Li²
Tianlong Chen'
2 Eigent Al UNC Chapel Hill
MMIT-IBM Watson Al Lab
Stannoord University
ABSTRACT
With an ever-growing zoo of LLMS and benchmarks, the need to orchestrate multiple models for improved task performance has never been more pressing. While frameworks like Mixture-of-Agent. (MoA) attempt to coordinate LLMs, they often fall short in terms of (1) selecting relevant agents, (2) facilitating ef- fective intra-agent communication, and (3) integrating responses efficiently. this work, propose Graph-of-Agent (GoA), new graph-based framework for modeling multi-agent LLM communication. Our approach begins with node sampling, selecting only the most relevant agents by leveraging model cards that summarize each model's domain, task specialization. and other characteristics. Next, construct edges between the selected agents by evaluating their re- sponses against one another to determine relevance ordering. Directed message passing is then performed from highly relevant agents to less relevant ones enhance their responses, followed by reverse message passing to refine the orig- inal responses of the more relevant agents. Finally, the updated responses are aggregated via graph-based pooling (e.g., max or mean pooling) to produce a single, unified answer. We evaluate GoA on diverse multi-domain benchmarks (MMLU, MMLU-Pro, GPQA) and Nomain-specinn manEval, MedMCQA), with an agent pool of 6 L.MMs spanning multiple domains. Surprisingly, GoA achieves superior performance using only 3 selected agents, outperforming recent multi-agent LLM baselines that utilize all agents simulta- neously. By adopting a graph structure, GoA offers both scalability and effective- ness through structured message passing- poiiin---osiiioiniggg it as strong candidate for navigating the challenges of the ever growing LLM zoo.
benchmarks (MATH, Hu-
INTRODUCTION
Too many L.Mss, too many benchmarks. As the ecosystem of Large Language Models (LLMs) (Wei et al., 2022a) rapidly diversifies, researchers number of models available, but also by the complexity of evaluating and combining them effectively test time solve complex tasks. In this era of abundant LLMs, central challenge emerges:
increasingly overwhelmed not just by the sheer
(Q) Given the diversity of available LLMss, how can design an effective playground where agents interact perrgisiicall-eeeerraeee strengths, compensating for weaknesses, and im- proving decision- decision-making makinng through efficient collaboration?
As an early attempt to address this challenge. Mixture-of-Agemt (MoA) (Wang et al., 2024a; Li et al., 2025) has recently been introduced as pioneering approach that explores how leveraging mul- tiple L.MMs can enhance overall model performance through the Mixture-of-Experr (MoE) (Shazeer et al.. 2017) framework. As illustrated in Figure 1 (c), MoA aggregates responses from multiple LLM agents, appends them to the original query and feeds the enriched input to the next layer. This facilitates multi-agent synergy, enabling models to complement each other and refine predictions collaboratively. While MoA has shown that integrating multiple LLMs can be advantageous over a single model, still faces several limitations that hinder its scalability and effectiveness:
Which agents? As shown in Figure (b), following the scaling law diverse LLM agents, se- lecting relevant subset from large pool of agents handling complex and diverse queries (Figure

---

Published as a conference paper at ICLR 2026
0:A patient's glocoso (C) flactuntes due traulin irvections and nutural metebolic respoross. the glacoss dynamica fellow the oor d002
Node:
Edge
Graph:
</>
-00) in sinff) 40
17
network prodict a ed function
f (,0000 603f de
Wrise code to trai nerual network prodict G
General Math Blomedical Code
Finance Legal
Query
Pool of Agents
(c) Mixture-of Agents (MoA)
(d) of
Figure 1: Current multi-agent LLM pipeline and our proposed approach.(a) Given a query that spans diverse domains (e.g.. biomedical + math code), (b) selecting and organizing agents from a large pool of LLMSs form effective multi-agent system remains significant challenge.
(c) The current Mixture-of -Agents (MoA) approach integrates all available agents, aggregates their responses, and forwards the combined output to the next layer. However. it lacks generalizability to larger agent pools (e.g.. 10 or 100) and suffers from heavy intra-layer communication overhead. (d) In contrast, our proposed Graph-of Agents (GoA) addresses this challenge through graph-based structure. In GoA, only subset of relevant agents is selected to form the graph's nodes, with intra-layer communication facilitated via directed message bussing-flowwed relevant agents to less relevant ones. By leveraging this graph structure, GoA achieves greater scalability and enables more efficient communication between agents, resulting in a more powerful and adaptive framework.
from more
(a)) is crucial challenge. The current MoA approach lacks effective agent selection mecha- nism, instead forwarding queries to all available agents, leading to excessive computational costs. This not only risks multi-agent system explosion but also introduces noise from irrelevant agents, highlighting the need for a more efficient and practical agent selection strategy.
How do they communicate? Once agents are selected, facilitating effective communication among them becomes a pivotal challenge. MoA adopts a many-to-one aggregation scheme, but this ap- proach has notable drawbacks: requires collecting responses from all available agents and treating them as single chunk, which fails to capture fine-grained interactions such as one-to-ome commu- nication between individual agent pairs. Moreover, since different agents have varying strengths and relevance depending on the query. treating all agent messages equally can hinder consensus. Instead, adaptively weighting responses based on their relevance is crucial for improving decision-making
How to integrate? Finally, when making the final decision, MoA aggregates responses by con- catenating tokens from all agents. However, this approach incurs a huge computational cost, with complexity of O(Ldd), where L is the number of communication layers, N is the number of agents, and is the token length per agent. Given that token usage is directly tied to cost, this method becomes prohibitively expensive. Moreover, not all agents contribute equally valuable sponses-sone domain- speniin-ppeializeed agents produce significantly higher-quality outputs than others. A scalable integration mechanism that prioritizes more reliable agents while reducing the impact of less relevant ones is essential for cost-efficient and effective decision-making.
Our Approach. To address these challenges, we propose graph-based mulli-agent framework, Graph-of-Agents (GoA), as a novel remedy to enhance multi-agent communication. GoAfunda- mentally rethinks multi-agent collaboration by modeling agents as nodes and their relevance-based relationships as edges, enabling structured message passing. This graph-based design allows for the selective activation of only relevant agents (as a subgraph of entire pool) while capturing inter-agent interactions through message passing and finalizing decisions via graph-based pooling. GoA follows a structured process, as shown Figure (d):
O Which agents? - Node Sampling: GoA begins by selecting relevant agents based on available metadata (e.g., domain, task) from model cards. This information is provided to a meta-I.LMM, simply gemeral-domain LLM, which identifies the most relevant agents given the query.
Once nodes (agents) are selected, we obtain their initial responses and ask each agent to rank others, capturing the signifi- cance of each agent's 's output. Based on these rankings, we construct directed edges in two perspec- tives: (i) Source-to- TTarge- Higher-ranked agents (i.e., highly influential nodes with more relevant responses) propagate their information to lower-ranked agents, allowing them to refine their re- sponses based more confident initial answers. (ii) Toruuee After lower-ranked agents
How do they communicate? - Edge Sampling & Message Passing:
2

---

Published as a conference paper at ICLR 2026
update their responses, the refined information is passed back to the higher-ranked agents, enabling them to further adjust their outputs based on the improved responses from their neighborhood agents.
mean pool- evaluated on diverse multi-domain benchmarks (MMLU, MMLU-Pro, GPQA) and benchmarks (MATH, HumanEval, MedMCQA), demonstrating its generalizability and adaptabil- ity across wide range of tasks. These results highlight the effectiveness of viewing multi-agent collaboration through the lens of graph structures.
With refined responses, GoA applies ing, akin to graph pooling, to adaptively aggregate the outputs of multiple agents. GoA is extensively
How to integrate? Graph Pooling:
It important note that, unlike traditional multi-agent learning method that require model fine- tuning or additional training, our proposed GoA framework, as like MoA, operates purely through the prompt interface. This design ensures compatibility with black-box LLM APIs while maintain- ing high adaptability across diverse domains during test-time inference.
Our contributions are three-fold:
We identify key challenges in current multi-agent LLM systems: selecting which agents to sample, facilitating effective communication, and integrating responses efficiently.
We formulate multi aalti- collaboration as a graph-based framework and introduce GoA, which incorporates node sampling, edge sampling, message passing, and graph pooling, This enables not only the construction a scalable multi-agent LLM ecosystem, but also enhances inter-agent communication.
We demonstrate the effectiveness of GoA across diverse benchmarks, including MMLU, MMLU- Pro, GPQA, MATH, HumanEval, and MedMCQA. Notably, using only 3 agents, GoA outper- forms recent multi-agent baselines that rely pools of 6 agents, highlighting how graph-based structures improve both the scalability and effectiveness of multi-agent collaboratio..
2 RELATED WORK
LLM Reasoning and Test-Time Inference. Test-time reasoning with LLMs has seen rapid ad- vances through prompt engineering techniques such as Chain-of-Thougee (CoT) (Wei et al., 2022b), Tree-of-Thought (Yao al., 2023), and Graph-of-Thought (Besta al., 2024; Yao et al., 2024), which enable models to decompose complex problems into structured sub-tasks. While most prior work focuses on improving a single LLM's reasoning via internal prompting strategies (Zhou al., 2023; Xu et al., 2024; Feng et al., 2024), a parallel direction has emerged around collaborative rea- soning at inference time using multiple L.LMs (Du et al., 2023a; Chan et al., 2023). In these multi- agent setups, multiple LLMs interact in test-time without additional finetuning, often aiming to boost factual accuracy or reasoning diversity. However, these approaches typically rely on simplis- tic communication protocols such as symmetric debate or sequential refinement, lacking structured mechanisms for message routing or adaptive collaboration. Our work builds on this foundation by proposing graph- based test-time collaboration framework that formalizes agent interactions proposing through directional message passing, enabling richer and more scalable multi-agent reasoning.
LLM Ensembles and Multi-Agent LLM Collaboration. Another approach to leveraging mul- tiple LLMs is ensemble- based inference, where outputs from several models are aggregated or se- lected (Fang et al., 2024; Yuan et al., 2023). Recent works introduce router mechanisms (Wang et al., 2023c; Hari & Thomson, 2023; Wang et al., 2024b; Yue et al., 2025) to reduce computational cost by selectively querying a subset of models. However, many of these ensemble strategies treat LLMs as interchangeable units without modeling their relationships. Meanwhile, graph-based structures have begun to emerge coordinate multi- multi-agent systems, notably in frameworks like MacNet (Qian al., 2024) and GPTS warm (Zhuge et al., 2024), which model agents as nodes in static DAGs. Also DyLAN (Liu et al., 2024b) performs dynamic agent activation using an Agent Importance Score computed via ral feed-forward communication network. Yet, these approaches often rely on predefined topologies or assume a single agent role-playing multiple personas, limiting flexibility and expressiveness, and sometimes even requiring additional optimization.. In contrast, our proposed framework, Graph- of-Agents (GoA), constructs dynamic graphs based on task relevance, enabling one-to-one com- munication across a diverse pool of specialized LLMs. Through node sampling, directional edge construction, and graph pooling, GoA supports efficient and adaptive collaboration addressing the
backward peer-rating propagation and coordinates agents through tempo-
3

---

Published as a conference paper at ICLR 2026
[a] Overview
(b) Node Sampling
</>
[Top-k Model Cards)
Chockt MadeeCaal
Madettad
(c) Edge Sampling
Σ
(1) Source > Target
Sampling
Score Matrix
0.6
0.3 0.1
Vrital Rapporane "Ti padi
0.5
0.0
Graph-of-Agae
(c] Ecle Sampling
0.4
0.1
0.9
0.0
Threshold
(40 Massuge Passing
Burger
1.0
2.0
0.8 0.2
(2) Tanget + Source
(+) Graph Pooling
(d) Message Passing
(e) Graph Pooling
Edges
MMann
[1] Source + Target
[2] Target + Source
Max Pooling
Mran Poaling
Figure 2: Overall pipeline of GoA. (a) Overview: Given query (Ω) spanning diverse domains, GoA ap- proaches multi-agent LLMs through the lens of graph framework and produces an answer (AA). (b) Node Sampling: Each agent mapped to model card containing domain and task information. The Mcta-LLM, general-purpose LLM, takes Q and the model cards input and selects the most relevant agents, forming an adaptive multi-agent framework. (c) Edge Sampling: After collecting initial responses from selected agents, each agent evaluates the relevance of others (excluding itself) to generate a normalized score matrix. Edges are established in Source-to- Target and Target-to-Sonn Sourrr directions, while low-relevance nodes are pruned using threshold r=0.05. (d) Message Passing: GoA first passes messages from source target nodes, allowing lower-ranked agents to refine responses. then reverses the flow update source nodes. (e) Graph Pooling: With updated responses structured as graph, GoA outputs the final prediction via max or mean pooling.
underexplored challenge of coordinating heterogeneous agents for domain-diverse reasoning at test time.
METHODOLOGY
3.1 PRELIMINARIES AND NOTATIONS
Multi-agent LLMs. We define a system of N LLM agents, where each agent E 1,..., N special- izes particular domain or task. The agents collaborativlly process a given query Q generate an optimized response A.
As a graph. We represent the multi-agent system as a directed graph, ....... UN denotes the set of agent nodes and & C V x V is the directed edges. Once subset of S relevant agents are selected from the multi-agent pool of size N, the adjacency matrix SxS is constructed. The matrix is defined such that Aij > 0 if (v,, Uz) E & and A;; = otherwise.
(V,E), where
3.2 OUR APPORACH: GoA
In this section, we present GoA, a novel approach specifically designed to enhance multi-agent communication through graph-based framework, as shown in Figure 2.
Overview. GoA begins with node sampling, where a meta-LLM (a general-purpose LLM) selects the most relevant agents from a given pool based on the query and model card dictionary. Each model card summarizes information extracted from the Hugging Face README file, including the LLM's domain and specialized tasks (Sec. 3.2.1). After collecting initial responses from the selected agents, we rank each agent based scores assigned by other agents. Agents are then sorted by these scores to define two node types: source nodes (highly relevant and influential) and target nodes (less influential). This establishes bidirectional relationships: source-to-target and target-to- source (Sec. 3.2.2). In the messsgge-passingg we first propagate messages from source to target, allowing target nodes refine their responses based on input from more relevant agents. We then reverse the flow-target updated responses. This two-step process ensures that each agent's contribution weighted by its
target to source-ss so that source nodes can further refine their outputs using the
4

---

Published as a conference paper at ICLR 2026
relevance the query (Sec. 3.2.3). Finally, apply graph pooling: either pooling based the most connected source node or mean pooling guided by the meta-LLM, to generate the final answer (Sec. 3.2.4).
3.2.1 NODE SAMPLING
Given query Q. goal is to select subset of agents most relevant to the task. To achieve this, we leverage publicly available model cards from Hugging Face (Jain, 2022), which contain useful metadata such the dataset the LLM Using this information, we summarize each agent's model card into three key categories: (1) The domain of the LLM, (2) The specialized task, and (3) The model size and special features. Once the summarized model card is obtained, we prompt the Meta-LLM' to determine which agents are most likely to generate the most effective response given the query. Formally, the selected subset of agents, V. C V, is obtained as:
trained on, its specialized domain, model size.
V3 Meta-LLM(TTp-kk]e Q, Model Cards),
where Top-k selects the k most relevant agents based on their alignment with the query and model card information. For example, as illustrated in Figure (b), if the query pertains to biomedical, mathematics, and code-related domains, the selected agents would primarily belong to these special- ized areas to maximize multi-agent synergy. This approach effectively filters out unnecessary agents (e.g., law-related models), preventing agent explosion (i.e., involving unmanagcable number of agents) while maintaining relevance agents for handling the query, answering "Which agents?".
3.2.2 EDGE SAMPLING
Once the subset selected agents V. is obtained, we prompt each agent to generate its initial response to the query Q, forming response set R (v) Es [Q): ) where S = [V/l. To model inter-agent relevance, we construct a score matrix in which each agent scores the responses of all other agents (excluding its normalized such that each agent distributes total score of 1.0 across the remaining S-1 1 agents. We then compute relevance score Sj for each agent j by summing the scores it receives from others:
reduce self-bias) based alignment with Q. These scores
S₁ = Score
The relevance scores S are then used rank agents and determine their communication roles (e.g., source vs. target). To avoid including weak or noisy Socnnddrrs-pattcullarly in cases where model cards may lack detailed information-we introduce a threshold hyperparameter т. Agents with Sj < T are pruned from the communication graph, ensuring scalability and better task-fit in the constructed structure.
Using the remaining high-relevance agents, we define a weighted directed adjacency matrix A E S to govern message passing. Each entry Aji represents how much influence agent exerts on agent when passing messages. It is computed by normalizing the total relevance scores of all neighbors of agent j:
S, Afr ELEN, St
where N₁ = (i + E).
This scoring formulation ensures that more relevant agents have proportionally greater influence, and it enables fine- gre-grained 1-to-1 communication tailored to task-specific needs. Figure 2(c) illustrates how the score matrix and pruning mechanism dynamically shape the communication graph.
Any general- I.I.M. In our experiments, used D-ven.557BB-Instte (Team, 2024) from 7-8B agent pool.
5

---

Published as a conference paper at ICLR 2026
3.2.3 MESSAGE PASSING
Now, given the edge information and weighted adjacency matrix, we proceed with message passing, a key advantage of the graph structure. To incorporate the significance of each agent, GoA performs message passing in two steps: Source-to-Targer followed by Target-to-Source.
Source-to- touree----Target. For highly influential nodes, it is crucial to maintain their initial strength rather than being influenced by less sigmificant or noisy nodes. Conversely, less significant nodes benefit from receiving messages (i.e., more relevant responses to the query) from stronger nodes. Thus, we first propagate information from source nodes (higher-ranked agents) to target nodes (lower-ranked agents), allowing the latter to refine their responses based on more confident initial answers:
R U. (1)==ct1AA) A R.j
where i < S S
4)
where Rj represents the updated response for target node j after receiving messages from source node Here, (-)-) denotes the forward pass of LLM j. represents the concatenation of neighboring responses, and Re represents the sorted responses, ranked from highly relevant to less relevant based the relevance scores S obtained in Equation 2. With these updated responses, we now proceed with the Target-to-Source step.
Torgee- Since the previous step only updates target nodes, also allow source nodes to refine their responses based the improved outputs of their neighbors (R,). rather than the initial responses from target nodes. This enables source nodes to incorporate the consensus of their neighboring agents, indirectly influenced by the original source node, leading to further refinement:
R (I5 where i i
where R. represents the refined response for source nodes. With both source and target nodes refin- ing their responses collaboratively through the graph structure, we effectively address the question of "How do they communicate?". We now move on to the final step: response integration.
3.2.4 GRAPH POOLING
To address the question of "How to integrate?" while minimizing the computational cost associated with token stacking, we formalize response integration through graph pooling, a common approach in graph-based tasks that requires aggregating node representations into single graph representa- tion. Motivated by this, we propose two pooling strategies: 1 Max-Pooling, which relies on the most influential node (i.e., the agent with the highest number of incoming edges, indicating higher relevance score). @ Mean-Pooling, which balances contributions by considering responses from all selected agents but on reduced scale, unlike MoA, which involves all available agents. Formally, this can be expressed as:
Rid R'ax source if max mooxipo Metaa-LLM(AverageRRRR) if mean-pooling,
A
(6)
where Rmaxx-souree denotes the refined response of the agent with the highest number of source edges (i.e., the most relevant agent). In summary, max-pooling prioritizes the response of the most sig- nificant agent, while proaing incorporates responses from all selected agents, requiring an additional forward pass through the Meta-LLM. We introduce these two variants as GoAmax and GoAmean. which will be analyzed in the experiment sections. The averaging is performed in weighted manner using the relevance scores assigned during edge sampling.
3.3 GOA GENERALIZES MoA
Lastly, we demonstrate that GoA generalizes the existing MoA framework. Specifically, as illus- trated in Figure 1 (c), the message-passing and response-updating procedure of MoA (Wang et al., 2024a) can be formulated as:
R = xx ( + Q).
Comparing this Equation 4, we establish the following:

---

Published as a conference paper at ICLR 2026
Table I: Benchmark performance across multi-domain and domain-specific benchmarks. Single- agent baselines: General: -wee2.5--BBInntteee (Team, 2024), Code: Qwen2.5-Coder-78.
Instruct (Hui et al., 2024), Math: Mathstral-7B-vO.. (AI, 2024), Biomedical: Bio-Medical-Llama
3-8B (Con, 2024), Finance: financeLLlama--81 (Cheng et al., 2024), Legal: Saul-7B-Instrune
vl (Colombo et al., 2024). Mulli-agent baselines: Debate (Du et al., 2023b), Self-Consistnncy (SC) (Wang et al., 2023a), Refine (Madaan et al., 2023), ReConcile (Chen et al., 2023), MoA (Wang et al., 2024a), and Self-MoA (Li et al., 2025). Our proposed framework, GoA (Graph-of-Agente despite using only 3 agents (i.c., top-k=3), outperforms both multi-agent baselines with 6 agents and single-agent models, demonstrating strong collaborative synergy and capability across both multi- domain and dopain-specific tasks. All performance is measured using zero-shot CoT in test-time.
Multi-Doman
Domain-Specifie
MMLU
MMLU-Pro
GPQA
MATH
Human Eval
McdMCQA
General
77.61
53.90
32.83
69.00
81.50
55.22
Code
68.04
42.33
33.84
59.60
85.37
45.57
Math
63.47
37.19
30.81
48.60
57.93
45.25
Single-Agent
Biomedical
46.60
27.90
25.25
18,80
20.73
47.00
Finance
54.11
25.52
28.28
13.80
27.44
42.08
Legal
55.86
27.57
30.30
12.40
36.59
41.50
Debate
72.53
47.05
29.29
69.60
40.24
53.05
SC
77.97
54.12
36.36
69.80
82.57
55.70
Multi-Agent (6 Agents)
Refine
77.40
54.71
38.92
71.60
80.49
54.94
ReConcilc
69.61
44.19
34.34
45.60
50.20
54.60
MoA
75.71
53.33
32.83
65.80
76.22
54.94
Self elf-
78.14
54.19
33.84
68.20
79.27
55,56
Multi-Agent
GoAMas
79.18
54.78
39.98
69.83
84.67
60.04
Agents)
GoAMean
78.52
54.27
40.54
73.12
84.98
57.92
Proposition 1 Graph-of-Agents (GoA) reduces MoA when the node sampling parameter equals the total number of agents N, the adjacency matrix is fully connected with all edge weights set to 1, i.e., A RN = 1, and self-loop included with the initial query (Q) at each layer, ultimately aggregated via mean pooling.
Thus, GoA serves more flexible and extensible generalization of multi-agent communication frameworks. By introducing structured message passing, adaptive agent selection, and weighted aggregation, GoA can scale to large agent pools while maintaining efficiency and robustness. As our work focuses on lest-time inference, the prompts used this study are provided in Appendix B.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Benchmarks. We evaluate performance on two
(MMLU (Hendrycks et al., 2020),
MMLU-Pro (Wang et al., 2024c), GPQA (Rein et al., 2023)) and three domain-specific benchmarks (MATH (Lightmaan al., 2023), HumanEval (Chen et al., 2021). MedMCQA (Pal al., 2022))- For MMLU and MMLU-Pro, due their large sizes, used stratified sampling: 50 samples per category across 57 categories for MMLU, and 150 samples per category across 14 categories for MMLU-Pro. For the agent pool, we primarily leveraged six LLMs with 7-8B parameters, covering diverse domains such as General, Code, Math, Biomedical, Finance, and Legal. The specific LLMs listed in Table and details the benchmarks and baselines are provided in Appendix C.
4.2
MAIN RESULTS
Effectiveness. In Table present benchmark results across both multi-domain and domain- specific tasks, comparing single-agent baselines, multi-agent baselines with agents, and our pro- posed GoA (3 agents). ① Among single-agent baselines, the general-purpose model achieves the highest average performance (61.15), but falls short the best-performing multi-agent models. Specialized agents (e.g., Math, Biomedical, Finance) tend to perform well only in narrow domains

---

Published as a conference paper at ICLR 2026
and underperform in others, leading to lower overall averages. Multi-agent baselines with 6 agents-especcialle especially Refine (62.15) and Self-MoA (61.63) demonstrate improved performance over all single-agent baselines. confirming the benefits of agent collaboration. However, their effec- tiveness varies across benchmarks, with no single method dominating all tasks. @ Our proposed GoA method outperforms all baselines across most metrics. GoAMux achieves the highest aver- age score, with top performance on MMLU (79.18), 18), MMLU-Pro (54.78), and MedMCQA (60.04). GoAMean records the best scores on GPQA (40.54), MATH (73.12), and HumanEval (84.98), show- ing robust and consistent gains across both reasoning and domain-specine tasks. 2 Notably, while other agent methods rely on integrating all 6 agents, GoA achieves superior results using only 3. suggesting that selective and structured agent collaboration can be more effective than full inte- gration. For detailed performance comparison plots for each category in MMLU and MMLU-Pro, please refer to Appendix D.
Table 2: Efficiency analysis in MMLU-Pro.
Efficiency. Another crucial perspective for test- time LLM collaboration is efficiency-thhtt is, han- dling multiple LLMS in a scalable manner. Table 2 presents a comparison between MoA and GoA on
Acc.
Calls
Tokens (k)
Time (8)
MoA
53.33
19
56.05
240.26
the MMLU-Pro dataset in terms of accuracy, number of LLM calls, average token usage (input output), and latency. Unlike MoA, which employs multiple rounds of proposers (i.e., all available agents) and final aggregator, GoA reduces LLM usage and latency while improving accuracy. This improvement stems from its design philosophy-node and edge sampling followed by graph pooling-whiek selectively involves only relevant agents in the system, making it more efficient and practical approach for multi-agent LLM collaboration. Ap- pendix provides additional results on GPQA and HumanEval benchmarks across more datasets.
GoAMas
54.78
19.18
100.43
GoAMein
54.27
12
22.58
18852
Scaling Up. To evaluate generalization to pro- prietary models, we tested GoA with got-4o on GPQA, MedMCQA (100 sampled). and HumanEval. As shown in Table 3, multi- agent setups consistently outperform the single- agent baseline. Notably, while DyLAN (Liu et al., 2024a) employs eight specialized agents (e.g.. 'Python Assistant', 'Algorithm Devel- oper'), our GoA with only three agents achieves higher performance, mirroring trends seen with open-source models. This advantage stems from our graph-bases reasoning and relevance- aware sessage-passing mechanism (Section 4.3), which enables targeted and noise-resilient com- munication. These results highlight that well well-designed communication strategies can be more effec- tive than simply increasing the number of agents, underscoring both the effectiveness and scalability of GoA.
Table 3: Scaling up with gpt-4o model.
GPQA
MedMCQA
HumanEval
gpt 40
47.47
77.00
90.20
Debate (6 Agents)
$3.03
80.00
85.98
SC Agents)
$4.27
81.00
92.07
Refine (6 Agents)
54.98
82.00
91.92
Reconcile (6 Agents)
$3.03
75.00
91.46
MoA (6 Agents)
50.51
80.00
92.07
DyLAN (8 Agents)
58.89
81,00
92.07
GoAsa (3 Agents)
55.05
82.00
93.29
GoAMar (6 Agents)
56.57
83,00
93,90
4.3 WHY GRAPH?
The key aspect of GoA is its graph-bases framework for multi-agent LLM collaboration. In this sec- tion, we show how the graph structure and the message- message-passing benefit LLM collaboration.
Graph-based Reasoning. Figure 3 presents a case study illustrating how graph-based framework improves reasoning by comparing recent MoA with our proposed GoA when handling an anatomy- domain from the MMLU dataset. In MoA, all available agents are used, including those from unrelated domains such math and code. These irrelevant agents introduce noise (e.g.. 'Answer: 1'), which negatively affects the final prediction. In contrast, GoA avoids irrelevant agent usage by applying node sampling followed by edge sampling, thereby constructing query-specific graph structure. This enables targeted wressag-passsing among relevant agents, leading to more accurate discussions (e.g.. 'Answer: 0') and final predictions. Overall, this comparison shows how graph- based framework enhances reasoning in multi-agent scenarios by leveraging structured interactions and relevance-aware message passing.
8

---

Published as a conference paper at ICLR 2026
Q:
(a) MoA
Less of soratic sensation over
Linqual sensatiion the anterier tve-tnirds of the tonque, but responsible for taste, The chorda tyrpani branch of the facial nerve (1) is responsible le for providing general sensation to the anteriar tventhirds the tonque, Therefore, facial serve,
nancibular
anterior two-thirds tongue indicates danege to the
Answwr:
Answer:
prisarity
#: "Lingual branch of the mandibullär trigeninal nerve.".
Answer:
"chorda tyspani branch of facial nerve.",
Answer:
correct
"Lingual branch of the g glossopharyngeal nerve.",
of
3: lespog nerve."
Answers
(b) GoA
Node Sampling
Edge Sampling
Message Passing
Graph Pooling
(ilDorain Bionedical domain, which highly relevant to anatony questioon...
Σ
(1) Sourm----Dege
branch sandibalar trigexinal procides somatic senset ion the acterior thi-thirds of the tongue, The chords tyrgani branch facial provides taste sensation to thirds the therefore, loss of sonatic sessation ever darage to the str Final Answers
0.9
0.1
1
1.0
0.0
1
'Answer:
8'
'Answer:
8'
(21 Repeated ensure streng performance
0.3
0.7
twc-
(31 General knowledge model can handle wide range of questions, including anatomical ones.
Annwer:
AAnwer: 8'
Figure 3: Reasoning process of (a) Mixture-of- (MoA) and (b) Graph Agents (GoA).
Table 4: Effectiveness of Message-Passing
Relevance-awarg Message-Passing. We now delve into GoA's relevance-awan message- passing framework under stricter fixed-node setting, targeting the HumanEval benchmark with three code-specific models. As shown in Table 4, the baseline models exhibit vary- Qweeei--eeeee22.-5dderrre
HvmanEval
Qwen2.5 2aann.5--oodr-7---nnsiiie
85.37
Beeed-Cnder-B--nnstuee deepseck-coder- 70-rodrrr7--nsstrttt-11....
80.49
73.17
MoA MoE-based Agents) Debate Debate-basee Agents)
85.37
71.95
ing performs best, while the others fall behind. In the multi aagent setting, we compare our ap- proach against existing frameworks: MoA (MoE-based) Wang et al. (2024a), Debate Du et al. (2023b), and Reconcile (confidence-based) Chen al. (2023), with our GoA (graph-based) achiev- ing the highest performance. This gain stems from its tailored cressage-passing mechanism, where edges are formed based on node relevance to the given question: information flows from highly relevant nodes (e.g., Qwen2.5-Coder-T -one---Cooor---innttrueee to less relevant ones (e.g., deepseek-coder-7k instruct-vl.s 1,5), which update their internal state, followed by a feedback phase that produces an ag- gregated response. These results demonstrate that graph-based approaches apprachhe--especilllyy when com- bined with relevance-aware message passing-offer a promising paradigm for LLM collaboration.
Reconcile Confidence based Agents)
80.61
GoAMax Gruph-buser (3 Agents)
85.98
4.4 ABLATION STUDY
Table 5: Ablation study on modules, top-k, and
Table 5 presents an ablation analysis on MMLU-Pro and GPQA. The top row shows the original GoA setting with the best performance. Interestingly, reversing the message- message-passing direction causes the largest performance drop (-2.60 on MMLU-Pro and -5.05 on GPQA), even worse than removing only one direction "Source-to-Target or Target-to-Source). This matches our expectation: when less-influential nodes become artificially dominant, they dis- rupt the intended flow and inject noise back into the original source nodes. This underscores key design mnsight-preserving the correct bidirectional flow, (1) coure-tt-Taagee and (2) Target- to-Source, is essential for GoA's 's effectiveness. Removing Taruete message passing leads to a noticeable drop (-1.12 MMLU-Pro, -1.95 GPQA), showing that feedback from target nodes crucial. @ Removing Source-to- -arget causes even larger degradation (-2.57, -3.86), highlight-
MMLLUPrr
GPQA
GoA (Top-k=3, 7=0.05)
54.78
39.98
Reverse Message Passing
52.18
34.93
w/o Target-to-Sae to-uuree w/o Source-to- Target w/o Scoring (A = 1)
53.66
38.03
52.21
36.12
52.91
37.34
Top-k=2 Top-k=5
53.54
36.75
54.65
39.13
0.1
53.12
38.43
7=2.2
52.78
37.12
ing the importance of initial information flow. Together, these results confirm the complementary role of bidirectional message passing. @ Disabling edge scoring (Aij 1) consistently reduces

---

Published as a conference paper at ICLR 2026
performance, validating the benefit of relevance-based weighting. 4 Varying the number of agents shows & = 2 limits diversity while = introduces slight degradation (likely requiring tailored 7), with & 3 remaining scalable and performant. @ Adjusting the edge threshold 7 shows that overly sparse graphs (r 0.1, 0.2) harm performance, while = 0.05 balances focus and connectivity. Overall, these results demonstrate that bidirectional message passing, adaptive edge scoring, and selective agent activation all contribute to GoA's 's effectiveness.
CONCLUSION
In this study, introduce GoA, graph-based framework that re-designs multi-agent communi- cation to address key challenges: selecting relevant agents, enabling effective communication, and integrating responses efficiently. Extensive experiments demonstrate consistent gains across diverse benchmarks. We believe GoA can advance the development of large-scale collective intelligence in the emerging multi-agent LLM era.
ACKNOWLEDGEMENTS
This research was partially funded by the National Institutes of Health (NIH) under award IR01EB03710101. The views and conclusions contained this document are those of the au- thors and should not be interpreted representing the official policies, either expressed or implied, of the NIH. This research was also partially supported by the Amazon Research Award.
ETHICS STATEMENT
We propose graph-based based multi-agent LLM collaboration framework. GoA. However, the LLMs used in GoA may still exhibit inherent biases and undesirable traits from pretraining, Consequently, its outputs carry similar risks of misuse as other test-time methods.
REPRODUCIBILITY STATEMENT
We provide our code at the anonymous link: attps://anoree 4open. www.scceeeeeeeeeee.ee // goa-F23C. All experiments were conducted on an A6000 GPU. The datasets used are publicly available.
REFERENCES
Bio-medical:
A high-performance biomedical 2024.
language
model.
Mistral AI. MathEtral. htpp://miistallaaaewwaaaaaatttti July 2024.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Giani- nazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoe- fler. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682 7682--176677 March 2024. ISSN 2374- 3468. doi: 10.1609/aaai, wtw.a/aaa.iiiiii....... 33816..27720.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate, August 2023.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse lms. arxiv preprint, 2309.07864, 2023. URL
httpps://arxte org/ abs abs/2309 13007.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1rXiv:2107.03374 2021.
10

---

Published as a conference paper at ICLR 2026
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. In- struction pre-training: Language models are supervised multitask learners. arXiv preprint 144491 2024.
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. Saullm- 7b: A pioneering large language model for law, 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14325, 2023a. doi: (0.48550/arXiv.2335..55 14325.. URL https: ///oi. org/10. 2tg/100.4850/arxi... 2305. 14325.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint 2rxiv:2305.14325, 2023b.
Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Demoncourt, Ryan A. Rossi, and Hanieh Deilam- salehy. Multi-LLM Text Summarization, December 2024.
Li Feng. Ryan Yen, Yuzhe You, Mingming Fan, Jian Zhao, and Zhicong Lu. CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming. In Proceedings of the 2024 CHI Conference Human Factors in Computing Systems, CHI '24, pp. 1-21, New York, NY. USA, May 2024, Association for Computing Machinery. ISBN 97984000033000, doi: 141144//361304.34222
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models, August 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, October 2020.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Shashank Mohan Jain. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models solve problems, pp. 51-67. Springer, 2022.
Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. Rethinking mixture-of-agents: Is mixing dif- ferent large language models beneficial?, 2025. URL httpp:://arxiiv.org////////////e ab5/2502. 00674.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic lom-powered agent net- work for task-oriented agent collaboration, 2024a. URL 02170.
abs abs/2310. 2310.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A Dynamic LLM-PPoeredd Agent Network for Task-Oriented Agent Collaboration. In First Conference on Language Modeling, August 2024b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Ma- jumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self- feedback,, 2023.
11

---

Published as a conference paper at ICLR 2026
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248-260 260. PMLR, 2022.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language language-model model-based multi-agent collaboration. arXiv preprint wxXiv2406.07155 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https: /tt///axivv////////////a////aata org/ 0ttps/231..2222
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:/701.000 06638, 2017.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. bttthu..i//bog///ee
5/.
Junlin Wang. Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agent enhances large language model capabilities. arXiv preprint arXiv:2406.0400 2024a.
Junlin Wang. Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents Enhances
Large Language Model Capabilities, June 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models, 2023a. URL https:ttptt.. waw.vv.vvvvvvo/avaoaaaa/vvava org/ /twg/bs//////er..e/eo.//eeeeeeer 2203. 11 11177 11.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https: ftttps://opeereeee //pppepeeeeeeeeeeeeeeeeeeeeeeeee net www.:/ettoootiteeo www.mmidd1PPNNNNNNNN
Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. Tabi: An Efficient Multi-Level Inference System for Large Language Models. In Proceedings of the Eighteenth European Conference on Computer Systems, EuroSys '23, pp. 233-248, New York, NY, USA, May 2023c. Association for Computing Machinery. ISBN 978- 1-400399487-- doi: 10. 14111/5553222635744533
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2024c.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a. URL hwtps:tpp//.... arxiv, org/ abs 2206 07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed Chi, Quoe V. Le, and Denny Zhou. Chain-of- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35:24824-24837 December 2022b.
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian Guang Lou, and Shuai Ma. Re- Readea Improves Reasoning in Large Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. $5549-155755 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Ad- vances in Neural Information Processing Systems, 36:1 1809-11822, December 2023.
12

---

Published as a conference paper at ICLR 2026
Yao Yao, Zuchao Li, and Hai Zhao. GoT: Effective Sraph-of-Thougee Reasoning in Language Models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguisties: NAACL 2024, pp. 2901-2921, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 08653/v1/202// 2ang224..nnning--aacll 183..
Xingdi Yuan, Tong Wang. Yen-Hsiang Wang, Emery Fine, Rania Abdelghani, Hélène Sauzéon, and Pierre-Yves Oudeyer. Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 12952-12965, Toronto, Canada, July 2023, Association for Computational Linguisties. doi: /1116//////222..nnnn.oogss/////
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning route Ilms for multi-agent systems, 2025. URL https://arxix> org/ hbs//bb//aa 2502.2502..0 11133.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. LEAST:TO-MOST PROMPTING ENABLES COMPLEX REASONING IN LARGE LANGUAGE MODELS. 2023.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning 2024.
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
We use 2hatGPT exclusively for grammar checking and text refinement. Its role was limited to polishing author-written text and did involve research ideation.
B PROMPTS
As our proposed GoA framework operates during test-time inference, the detailed prompts for each stage are provided below:
Model Card Extraction (Example: Qwen2.5-Coder-7BB;
Model Card Extraction System Prompt
You expert in analyzing and summarizing Al model documentation.
hhups://chatgntt.com
13

---

Published as a conference paper at ICLR 2026
Model Card Extraction User Prompt
You are given the README file of language model:
language:
en
base_modell
5- 7B
pipeline_tag: text-generation
library_namer transformers
tags:
code
codeqwen
chat
qwen
qwen-coder
# roductitoo
Qwen2.5-Coder is the latest series of Code-Speci code generation, code reasoning and code fixing.
Please extract and summarize the model's key characteristics clearly and concisely in the following structured format:
1. Domain: The primary domain application area the model is designed for (e.g.,
general-purpose, biomedical, finance, coding, math, etc.).
2. Task Specialization: Describe the task types the model is designed for or excels at. Be as specific as possible, including the domain context of each task (c.g., biomedi- cal question answering, clinical decision support, financial sentiment classification, code generation). Do not include performance metrics, benchmark names, or evaluation re- sults.
3. Parameter Size: The number of parameters in the model (approximate if not explic- itly stated).
4. Special Features: Any distinguishing aspects such as fine-tuning datasets (if applica- ble).
Your summary will later be used to compare multiple models for selection purposes. Return your answer in bullet-poin format, using the exact field names shown above. Keep it concise but specific enough for model comparison.
Answer:
14

---

Published as a conference paper at ICLR 2026
Node Sampling:
Node Sampling System Prompt
You are an AT model selection expert. Your task is to select the most relevant AT models to answer a given question based on their domain, specialized capabilities, and overall performance.
Selection Criteria:
Domain Match (Primary Factor): Prefer models that are trained in the relevant domain for the question.
TTas Specialization: If the question requires specific skill (e.g.. reasoning, code genera- tion, biomedical text processing), prioritize models that explicitly specialize that area.
Generalist Models Consideration: If generalist model is known to perform well in the given domain or task, include least one such model in the selection.
Size vs. Performance Balance:
Do NOT rely solely on model size when selecting between generalist and specialized models. If generalist model is significantly larger (e.g.. 13B vs. 7B), prefer the larger model. Otherwise, choose hased task performance and known effectiveness.
If a model is highly relevant, you may select it multiple times. If a question does not specify a domain, general models should be preferred.
Node Sampling Use Prompt
Given the question: nessages [0] and the following model descriptions:
model descriptions, select the top top k models that best fit the question.
Selection Rules:
Return comma-separated list indices (e.g.. "0, 1,1,
The list must contain exactly top_k values.
You may repeat an index the model highly relevant.
Only use numbers in the range [0, max max_index]. Do not include explanations.
Example Selections:
If the question is about biomedical research: "0, 5, 5 (favoring biomedical models, but keeping generalist model if available).
the question is a reasoning- based general question: "0, 0, 1, (favoring generalist models with some specialized).
example Le dict
1: "0"
2: 30,3", "0, 5,
4: "0, 0,
5: 30,1,2,3,5"
6: "0, 3,,
Answer Format: Strictly follow the format below. Do not add any explanations or extra text. Example: example.dict
Answer:
15

---

Published as a conference paper at ICLR 2026
Edge Sampling:
Edge Sampling System Prompt
You are an expert evaluating Al model responses. You must rank and score the responses relatively, assigning numerical score to each such that the total sum of all scores is exactly 1.0.
Edge Sampling User Prompt
Given the question: messages [0], and other models' responses: otherrres
Evaluate following responses from models: other.modele
Assign a score to each response based on:
Correctness (most important)
Coherence
Relevance
Distribute a total 1.0 point across all responses. Better responses should receive higher scores. Do not provide explanations.
Response Format:
Please assign scores in the order the responses shown above, from top to bottom. The order of models othee_
Make sure:
The list has exactly len(ot her modelss) scores
The sum the scores exactly 1.0
Do not include extra text
Example Dictionary:
example lecpceee_lc_____ee
[1.0], 2: [0.7, 0.31,
3: [0.1, 0.1, 0.81,
4: [0.3, 0.4, 0.1, 0.2),
5: [0.2, 0.4, 0.1, 0.3, 0.01
example_sta ".join
hacp__ooeelssss (example dict [Len [2]1" for in range (len wother_modelee
Example: example_str
Answer:
Message-Passing (1) Target:
Source-to- Mource-to-Target- System Prompt
You refining your response by incorporating insights from other models. Each model's Is contribution is weighted by relevance score. Use their ideas carefully improve your answer, while keeping the strengths and clarity of your original response.
16

---

Published as a conference paper at ICLR 2026
Source-to-Taryee Targett User Prompt
Question: messages [0]
Your initial response: initialrrespon [target]
The following responses provided by other models. They considered more relevant and may offer valuable insights improve your answer.
descriptions
While reviewing these responses, carefully consider accuracy and reliability the information they contain, as some parts may be incorrect or influenced by bias. Update your response by integrating any useful information from these answers, while preserving own original strengths.
Message-Passingg (2) lurget-to-Source.
Target-to-Souret System Prompt
You are finalizing your response after seeing how other models refined their answers based on your initial response. Use their improvements to further refine your answer and make it as complete and accurate as possible.
Target-to-Souree User Prompt
Question: messages [0]
Your initial response:
source]
The following models updated their responses based on your original answer. Their updates incorpo- rate insights from you while maintaining their strengths:
descriptions
Now, review how your answer influenced others. While reviewing these responses, carefully consider information they contain. as some parts may incorrect or influ- enced by bias. Based on these updated responses, write your final, upgraded response, incorporating any new ideas or refinements find valuable.
accuracy and reliability
Graph-Pooling:
Graph Pooling System Prompt
You are syuthesizing multiple model responses into one. Prioritize relevance consider inputs. Your final answer should be accurate, coherent, and well-structured.
Graph Pooling User Prompt
Given the question: me messages [0].
following responses have been generated by different models:
i thppt_reeponssee
Please synthesize a final, well-reasoned answer that is cohesive, accurate, and best addresses the ques- tion by effectively integrating these responses. While reviewing these responses, carefully consider the accuracy and reliability of the information they contain, some parts may he incorrect influenced by bias.
C DETAILS ON EXPERIMENTAL SETTINGS
Benchmarks, We evaluate our framework on the following benchmarks:
MMLU (Hendrycks et al., 2020): A comprehensive multi-domain benchmark with 57 subjects (which can be grouped into 4 broader categories), designed to test general
17

---

Published as a conference paper at ICLR 2026
knowledge and reasoning across humanities, STEM, and other domains. In this study, employed stratified sampling with 50 samples per subject to ensure balanced representa- tion.
MMLU-Pro (Wang et al., 2024c): A professional-level extension of MMLU comprising 14 categories focused on advanced reasoning tasks professional fields. this study, we employed stratified sampling with 150 samples per category ensure balanced represen- tation.
GPQA (Rein et al., 2023): A challenging benchmark targeting graduate-level question answering with an emphasis on deep understanding and domain-specific reasoning.
MATH (Lightman et al., 2023): A benchmark comprising high school and competition- level mathematies problems, designed to symbolic reasoning and multi-step problem solving. We used subest of 500 problems.
HumanEval (Chen et al., 2021): A code generation benchmark consisting of programming problems that evaluate functional correctness of generated code.
MedMCQA (Pal al., 2022): A multiple-choice medical question answering benchmark based on real medical entrance exams, assessing biomedical and clinical knowledge.
Multi-Agent Baselines. We compare our method against the following multi aggti LLM baselines:
Debate (Du et al., 2023b): A multi-agent collaboration approach where agents engage in structured debates converge on final answer.
Self-Consistency (SC) (Wang al., 2023b): A single agent generates multiple reasoning paths, and the final answer is chosen by majority voting across these diverse outputs.
Refine (Madaan et al., 2023): A refinementtbased collaboration framework where agents iteratively improve another's responses.
ReConcile (Chen et al., 2023): A confidence-based collaboration strategy where agent responses are weighted and reconciled based on self-assessed confidence.
MoA (Wang et al., 2024a): A Mixture-of-Agents method where different experts contribute to decision- makinn in proposer-and-aan and-aggreeator fashion.
Self-MoA (Li et al., 2025): A variant of MoA composed of multiple instances of a single model, where the single model the top-performing model for each task.
D DETAILED COMPARISON ON EACH CATEGORY
Figure and Figure 5 show the fine-grained performance across each category in the MMLU and MMLU-Pro benchmarks, respectively. We compare domain-specific agents and a strong baseline, Self-MoA (utilizing agents), with proposed method, GoA (utilizing 3 agents). We observe that in most cases, the general-domain agent (G) performs strongly across diverse domains. However, lacks generalizability across all categories and, like Self-MoA (Li al., 2025), underperforms compared to GoA. This is due to GoA 's ability to adapt to diverse domains through relevant node selection (e.g., the law agent in speciiec question (Figure 5)) and a tailored message-pant passsin framework that effectively captures domain-relevan information in response to each question.
Ower2.5 Instruct Oheen S-lrdS-d-BBBBaae Ji-inr-----nnrrus
Mathitral 73-0.
Medical 3:58
Aqunts) (3 Agents)
Figure 4: MMLU benchmark results across 4 categories. The last color bars represent Self-MoA and GoA, respectively. Self-MoA integrates 6 agents, while GoA utilizes 3 agents.
18

---

Published as a conference paper at ICLR 2026
biology
business
chemistry
computer science
economics
CO
60
40-
40-
20-
20
engineering
hea lth
history
law
math
40
20
20
70-
philosophy
physics
psychology
other
00
Owen2.55 Coder
M: Malhiand--B-ad.
8: Sio Mudical-Jaaaee ma-3--3
firance Llama2 8B
20
Solt MoA (6 Agental GoA Agants]
Figure 5: MMLU-Pro benchmark results across 14 categories. The last two color bars represent Self-MoA and GoA, respectively. Self-MoA integrates 6 agents, while GoA utilizes 3 agents.
E LIMITATION AND FUTURE WORK
While GoA demonstrates strong performance in multi-agent collaboration, several limitations re- main. First, the current node sampling mechanism primarily relies on publicly available model card information to select relevant agents. However, model card descriptions may be incomplete, incon- sistent, or lack crucial details about model's strengths and weaknesses in specific domains. As result, the selection process may not always optimally align with the query requirements. Future work could explore more adaptive selection mechanisms, such as meta-learning-basee optimiza- tion or reinforcement learning techniques, to dynamically refine agent selection based on historical performance and contextual relevance.
Morcover, GoA primarily focuses on text-based reasoning tasks, leveraging structured communica- tion among language models. However, many real-world applications require multimodal reasoning, where agents process and integrate textual, visual, and numerical information simultaneously. Ex- tending GoA to multimodal domains by incorporating vision- aaiion-anguage models, structured numerical reasoning, or even cross- modal knowledge fusion would be promising direction.
Lastly, while the current benchmarks used our study (e.g., MMLU, GPQA, McdMCQA) are not interactive environments, our framing is consistent with the Mixture-of-Aget work-both conceptually (multiple L.Ms collaborating as an agent system, illustrated in Figure I of the main paper) and in evaluation setup, which similarly relies question-answering tasks (e.g., AlpacaEval 2.0, MT-Bench, FLASK) to study test-time agent collaboration. That said, fu- ture work, plan explore more interactive, environment environment-grounded agentic benchmarks, such as WebAgent and GUI navigation. Incorporating GoA into such settings where agents interact with tools, execute actions, access the web require adapting the agent pool and communica- tion protocols, and is indeed a promising direction for future work. In particular, GoA's graph-based construction could help integrate heterogeneous tool responses into coherent final answer, some- what analogous to GPTSwarm's 's application of Self-Consistency.
(MoA) line of
F EXTENSION TO LARGER AGENT POOL
In addition the 6 agents covering (general, code, math, biomedical, finance, legal) domains, we have added 4 additional agents from general (Meta-LLama-33 -iw-i----sntttr ruct), code (Seed-Coder- - 8B- -n--ndrrrutt--tueetteeee Instruct),, math (deppsee--math-255 ins ruct), and biomedical
19

---

Published as a conference paper at ICLR 2026
MSsL-MedLLama-3--222 .0) domains, resulting in agent pool of 10 diverse LLMs. Given this extended agent pool in the GPQA benchmark, the performance is shown below:
Agent PooL-((CCCMBBBLL-(((C2222EE
GPQA
Refine (10 agents)
35.86
MoA (10 agents)
31.82
Self-MoA (10 agents)
35.86
ReConcile (10 agents)
37.89
GGA_maa (3 agents)
38.23
GoA_mean (3 agents)
39.39
Interestingly, even after increasing the number of agents from 6 to 10, the baseline methods still fail to outperform GoA max and GoA mean, despite GoA using only 3 agents. This suggests that in heterogeneous agent-pool setting, naively scaling up the total number agents does not necessarily yield performance gains; instead, selectively leveraging only the most relevant agents can more effectively and efficiently achieve superior reasoning performance.
G ROBUSTNESS OF NODE SAMPLING
To evaluate the robustness of the node-sampling module-which based on model card information from open-source sources-we conducted an experiment where we initially fix the sampled nodes to the relevant code agent (Qwen2. 5-Coder-7B- Instruct) for solving the HumanEval bench- mark, and then gradually replace the relevant agent (Code) with an irrelevant, noisy agent (Bio). This setting simulates scenarios where node sampling mistakenly selects suboptimal agents due to misleading or incomplete model card information. The agents used in this experiment are:
Code: -wwen255-Coeer-7B--11
Bio: ContactDoctor/Bioot
3-BB
The results follows:
HumanEval Agents
MoA
GoA
Code, Code, Code
84.93
86.59
Code, Code, Bio
82.54
(A=-2.39)
84.15
(A=-2.44)
Code, Bio, Bio
79.88
(A=-5.05)
82.93
(A=-3.66)
Here, we quantify as the performance gap between the fully relevant agent pool (top row) and the noisy settings (cach subsequent row). We observe that when more noisy agents are introduced, MoA suffers larger performance degradation. This is because MoA concatenates the responses from all participating agents uniformly, without weighting or filtering, causing noise from irrelevant (Bio) agents to propagate through subsequent layers and accumulate.
GoA, the other hand, passes messages in relevance-aware manner (most-relevant least- relevant) and introduces a threshold that filters out noisy agents from actively participating in the graph. This prevents irrelevant signals from contaminating the message passing process, resulting in significantly smaller performance drops than MoA and consistent outperformance in all settings.
H COMPARISON WITH MACNET
We additionally ran experiments on the MacNet (Qian et al., 2024) baseline, another network-based multi-agent collaboration framework utilizing predefined topologies such chain and tree, to eval- uate how existing graph-base multi-agent methods compare to our proposed approach, GoA. The results shown below:
Interestingly, across diverse fixed jopologics-chantt star, tree, and random we observe that Mac- Net consistently underperforms GoA (and in some cases even underperforms MoA and Self-MoA).
20

---

Published as a conference paper at ICLR 2026
Agent Pool=(C,CMM.BEL)) MMLU-Pro
GPQA
Refine
54.71
38.92
ReConcile
44.19
34.34
MoA
53.33
32.83
Self-MoA
54.19
33.84
MacNet Chain
50.67
36.36
MacNet Star
50.52
35.35
MacNet Tree
50.52
35.35
MacNet Random
50.71
32.83
GoA_Max
54.78
39.98
GoA Mean
54.27
40.55
This indicates that in multi-agent graph construction, incorporating task relevance and leveraging relevance- welevance-weighted message passing (such GoA's bidirectional propagation) is crucial for boost- ing reasoning performance, rather than relying on static graph structures.
I DISCUSSION ON ADDING MORE AGENTS
Referring to Table (top-k 3) and GoA's ablation study in Table (top-k: 2,5), have already varied the number of agents. We additionally include the k = 6 case below:
Agent Pool=[G,C.MBEL]
MMLU-Pro
GPQA
top-k
53.54
36.75
top-k
(GoA)
54.78
39.98
top-k
54.65
39.13
top-k
54.27
38.45
As shown above- and the reviewer suggested adding agents does not necessarily lead to performance gains, especially in heterogeneous agent-pool spanning diverse domains. Our goal to create environment where agents interact synergistically, yet specific agents alone are often individually weak (c.g., B, F. L agents score 27.90, 25.52, and 27.57 on MMLU-Pro), likely due to domain mismatch, limited coverage, or insufficient instruction- -tuniingonnnitnn.... Therefore, selecting a subset of relevant agents and emphasizing more task-relevant, influential agents becomes crucial-an an ability that existing approaches lack, and that GoA uniquely enables through relevance- -uided sampling and graph-based communication. Consequently, top-k 3 yields better performance than naively increasing the number agents, since including weaker or domain-irrelevant agents introduces noise (given their inherently weak single-agent performance) rather than useful signal. This aligns with DyLAN (Liu et al., 2024b), which similarly observes that smaller set of well-chosen agents outperform larger unfiltered group.
in Table domain-
However, we would also like to highlight Table 3 in the main paper, where we evaluated a more controlled scaling aeeaaiio-aamer homogeneous agent setting- by fixing the agent pool single LLM (gpt-4o) and simply varying the number of agent copies. The results are shown below:
Agent Pool=(GPT-40)
GPQA
MedMCQA
HumanEval
gpt-44
47.47
77.00
90.20
GoA (3 Agents)
55.05
82.00
93.29
GoA (6 Agents)
56.57
83.00
93.90
In this controlled setting where all agents have uniform capability, GoA with 6 agents consistently outperforms GoA with agents across all benchmarks. This demonstrates that when agent quality consistent (single-LLLM replication), increasing agent count indeed improve performance.
21

---

Published as a conference paper at ICLR 2026
J MORE RESULTS ON EFFICIENCY ANALYSIS
We further conducted efficiency analysis on two additional bewchmarks- (Table 6) and Hu- manEval (Table 7) verify that our improvements are dataset-specifie but hold consistently across reasoning and program synthesis settings. This suggests that selective collaboration among relevant agents
realized reallse-e through node/edge sampling and graph pooling- ing quality but also enables scalable multi-agent pipeline.
not not only preserves core reason-
Table 6: Effficiency comparison on GPQA.
Acc.
Calls
Tokens (k)
Time (S)
MoA
32.83
19
56.87
245.65
GoAmax
39.98
11
17.32
88. 1.5
GoAmeen
40.54
12
20.59
105.72
Table 7: Efficiency comparison HumanEval.
Acc.
Calls
Tokens (k)
Time (s)
MoA
76.22
19
29.44
122.52
GoAmas
84.67
11
10.21
59.04
GoA
84.98
12
12.32
70.01
22


Reviews:
[{"review_id": "9Oe64YZHem", "reviewer": "Reviewer_4HpR", "summary": "This paper addresses the shortcomings of current multi-agent LLM collaboration frameworks (e.g., Mixture-of-Agents, MoA) in three key aspects: agent selection, effective communication, and response integration, proposing a graph-structured framework named Graph-of-Agents (GoA). The core idea is to model LLM agents as nodes and the inter-agent relevance as directed edges. The paper’s main contributions include: Identifying three core challenges in current multi-agent LLM systems, thus providing clear research directions; Introducing a graph-based formulation of multi-agent collaboration that integrates four essential modules (node sampling, edge sampling, message passing, and graph pooling), and theoretically showing that GoA generalizes MoA, providing conceptual unification; Demonstrating that GoA achieves better performance and efficiency with fewer agents across diverse benchmarks and both open-source and proprietary models (e.g., GPT-4o).", "questions": "1. Have the authors experimented with larger or more diverse agent pools (e.g., 10, or synthetic agent ensembles)? If so, did they observe bottlenecks in latency or accuracy?\n2. Could GoA be extended to multimodal or non-text agents? Is it possible to generalize this framework to multimodal tasks?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a graph-structured multi-agent collaboration framework, integrating node sampling, edge construction, bidirectional message passing, and graph pooling. This design is a meaningful extension of existing multi-agent methods (e.g., MoA) and introduces the idea of modeling multi-agent interaction as a dynamic graph, enabling task-adaptive communication paths—an innovative and elegant formulation.\n2. The method is prompt-based and requires no additional training, making it compatible with black-box APIs and thus generalizable to different LLM setups.", "weaknesses": "1. Although GoA improves upon MoA, bidirectional message passing still requires multiple LLM calls, which may cause latency issues in real-time or large-scale agent pools. The paper lacks a systematic comparison of token usage and inference time.\n2. The paper does not provide quantitative ablation on the robustness of the node sampling strategy. Although Table 4 includes ablations on the number of agents and message-passing variants, it does not explicitly examine robustness when model card information is incomplete or noisy.\n3. While “Proposition 1” claims that GoA is a generalization of MoA, this statement is largely informal, without rigorous theoretical proof. There are no formal results on the convergence, generalization, or optimality of the message-passing mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YrH1GNK95F", "reviewer": "Reviewer_7qjd", "summary": "This paper introduces Graph-of-Agents (GoA), a graph-based framework designed to improve multi-agent communication among LLMs. GoA addresses the challenges particularly in Mixture-of-Agents (MoA), by providing solutions for selection of agents and intra-agent communications. The proposed framework with 3 agents outperforms other multi-agent baselines with 6 agents on various benchmarks.", "questions": "Could you provide failure cases where GoA does not perform well or even fails to MoA or other baselines? Understanding the limitations of your approach would help in assessing its overall effectiveness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Clear ideation:** The paper clearly identifies the limitations of Mixture-of-Agents (MoA), and motivates the need for a selection approach for agents and agent communications.\n2. **Extensive experiments:** The authors conduct comprehensive experiments across various multi-domain and domain-specific benchmarks, demonstrating the effectiveness of the GoA framework.\n3. **Clear ablations:** The ablation study is well designed for evaluating the impact of different components of the proposed framework.", "weaknesses": "1. **Limited discussions about prior work:** The related work section is brief and lacks depth in discussing prior works on dynamic multi-agent collaboration with LLMs. [1,2,3] show different levels of dynamic agent selection and communication strategies that could undermine the novelty of the proposed method.\n2. **More agents may not bring better performance:** The authors claim that GoA with 3 agents outperforms baselines with 6 and more agents. But [1] suggests that increasing the number of agents may not always lead to better performance, especially in self-organized agent systems. Fairer comparisons with different numbers of agents are necessary.\n3. **Justification and presentation issues:** The content is redundant by repeating node sampling, edge sampling, and message passing in multiple sections, without enough justification of method design choices. Minor presentation issues exist such as tcboxes should not be labeled as tables and etc.\n\n[1] Liu et al. A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration. COLM 2024.  \n[2] Zhuge et al. Language Agents as Optimizable Graphs. ICML 2024.  \n[3] Yue et al. MasRouter: Learning to Route LLMs for Multi-Agent Systems. ACL 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzlrVQ1pSy", "reviewer": "Reviewer_47YL", "summary": "The paper introduces a novel method for coordinating multiple LLMs using a graph structure. Each LLM is treated as a node, and directed edges represent relevance-based communication, enabling structured message passing between more and less relevant LLMs. Across multi-domain and specialized benchmarks, GoA outperforms prior multi-agent systems (like Mixture-of-Agents) while using fewer models, offering both scalability and efficiency in test-time collaboration among LLMs.", "questions": "- I am wondering when each node is generating the response, are they aware that their response will be routed to others? This awareness may cause difference in model’s behavior I believe.\n\n- Will the domain specific questions always drive GoA to choose the expert model of that certain domain? Why do you think GoA can be better than using the expert model alone, empirically and theoretically?\n\n- I am wondering how you make the MoA and GoA comparable, as prompt input should be quite different, and in pure prompting framework, I think the performance is very sensitive to the instruction inputs. Therefore, how you ensure the scale up / efficiency experiment results are comparable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The writing flow is very clear, and the motivation for developing this framework is strong and convincing. The node sampling and edge sampling are relatively new methods to me and seems interesting to promote communication efficiency.\n\n- I pretty like the motivation behind the source-target information passing design, which is said to be able to avoid noise while enhance the model’s original answer through aggregate other’s opinion.\n\n- The benchmarks tested are comprehensive, even though the ablation studies can be more comprehensive and insightful. The comparison with many other related methods are also very comprehensive.", "weaknesses": "- The agent here I am afraid is kind of abused. The different nodes from my understanding is just different LLMs. For agent, their core trait is task-oriented and environment interaction, which I think current node (it does not interact with environment using tools) does not entail. The paper may clarify what they view as agent if claim this paper as graph-of-agent.\n\n- I think more ablation studies should be done on the method design motivation itself, like what if we reverse the source-target message passing sequence? Also it can be more insightful by discussing why the results are like this, such as top-k=3 is better than 2 and 5, instead of just describing the numbers which are obvious from the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZGBdxDeFA3", "reviewer": "Reviewer_Usox", "summary": "The paper proposes Graph-of-Agents (GoA), a graph-based framework for orchestrating multi-agent collaboration among Large Language Models (LLMs). GoA addresses three key limitations of existing approaches like Mixture-of-Agents (MoA): (1) inefficient agent selection, (2) unstructured communication, and (3) costly response integration. It introduces a pipeline of node sampling, edge sampling, bidirectional message passing, and graph pooling (max/mean) to produce a final answer. Evaluated on diverse benchmarks (MMLU, GPQA, MATH, etc.), GoA with only 3 agents outperforms MoA and other baselines using 6 agents, demonstrating superior efficiency and effectiveness.", "questions": "1) In the main experiment (Table 1), the authors use 3 agents—how many candidate agents were available for selection? How do baseline methods perform when restricted to the same 3 agents? What is GoA’s performance with 6 agents? Which base model is used in Table 1? The comparison appears unfair unless all methods use the same number and type of agents.\n2) The method constructs a graph based on agents’ initial responses to determine execution order, then re-executes the task using this order. Does this two-stage process significantly increase computational or implementation complexity? Can the constructed graph capture multi-layer or hierarchical dependencies among agents?\n3) Intuitively, the benefit of GoA should grow with more agents, yet Table 3 shows limited gains when increasing agent count. How do other models in Table 1 (e.g., Refine) perform under the same GPT-4o setting used in Table 3? Given that Refine already achieves strong results in Table 1, is GoA’s improvement marginal?\n4) You cite router-based ensembles (e.g., Wang et al. 2023c, Hari & Thomson 2023) but don’t compare against them. How does GoA’s node sampling + graph communication compare to learned or prompt-based routing in terms of accuracy and cost?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) GoA creatively reframes multi-agent LLM collaboration as a dynamic graph construction problem, combining agent metadata, relevance-based edge formation, and directional message passing. This is a synthesis of graph neural network concepts with test-time LLM orchestration, distinct from static ensembles or debate-based methods.\n2) The methodology is well-structured and technically sound. The ablation study (Table 5) convincingly validates each component (e.g., bidirectional message passing, relevance scoring). Experiments span 6 diverse benchmarks and compare against 6 strong multi-agent baselines, with clear metrics (accuracy, token usage, latency).\n3) The paper is clearly written and well-organized.", "weaknesses": "1) The proposed multi-agent collaborative framework is evaluated with only a small number of agents (e.g., 3 in Table 1), which fails to demonstrate the scalability or advantages of the method in more complex, large-scale multi-agent settings. Moreover, The ablation shows k=5 slightly underperforms k=3. Is this due to increased noise or insufficient message-passing depth? Would adding more rounds of message passing help with larger agent sets?\n2) The paper lacks a clear comparison with workflow-based multi-agent systems or auto-workflow generation [1]. It remains unclear whether the proposed approach—selecting agents based on task–agent relevance—outperforms hand-crafted workflows, and the selection mechanism itself shows limited novelty.\n3) Key graph-based multi-agent baselines, such as MacNet (Qian et al., 2024) and GPTSwarm (Zhuge et al., 2024), are not included in the experiments, weakening the empirical evaluation.\n4) The method is not tested on established complex multi-agent benchmarks that require coordination, such as WebAgent, GUI navigation, or WebShop tasks, raising concerns about its generalizability and practical utility.\n5) The results show that their model with 3 agents outperforms the baselines with 6 agents. However, the agents set for selection is larger than 3. Moreover, the performance of GoA_max and GoA_mean is unstable for various tasks. Particularly, both GoA_max and GoA_mean perform similarly with the baselines. The improvement of the proposed method is very limited.\n6) The manuscript contains grammatical errors (e.g., “... and graph pooling,GoA supports efficient and ...”), which affect readability and professionalism.\n\n[1] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge,\nXin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)