Mixture-of-Agents Enhances Large Language Model Capabilities
Authors: Junlin Wang
Jue WANG
Ben Athiwaratkun
Ce Zhang
James Zou
Venue: ICLR 2025 Spotlight (2025)
Keywords: Multi-Agent Inference
Large Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper presents a method that synergistically leverage multiple LLMs to significantly improve their performance.

Abstract: Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, Arena-Hard, MT-Bench, and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs achieves a score of 65.1% on AlpacaEval 2.0 compared to 57.5% by GPT-4 Omni.

Published as a conference paper at ICLR 2025
MIXTURE-OF-AGENTE ENHANCES LARGE LANGUAGE MODEL CAPABILITIES
Junlin Wang!* Jue Wang²
Ben Athiwaratkun
Ce Zhang
James Zou
Duke University Together AI Universityy of Chicago
4Stanford University
junlin inn www.wwwuue.eeeeeeee.e... (jue, ben) @together.....
cez@uchicago edu, jamesz@stanford.edd
ABSTRACT
Recent advances in large language models (LLMs) demonstrate substantial capa- bilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, propose new approach that lever- ages the collective strengths of multiple LLMs through a Mixture-of-Ageme methodology. In our approach, we construct layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves statc-of-art performance on AlpacaEval 2.0, Arena-Hard Hard, MT--Bench, and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs achieves score of 65. 1 % on AlpacaEval 2.0 compared to 57.5% by GPT-4 Omni
(MoA)
INTRODUCTION
Large language models (LLMs) Zhang al.
2022a
Chowdhery et al. 2022 Touvron al. 2023a Team
Single Model
With responses from other models
al.. Brown et al. 2020: OpenAl| (023) have significantly advanced the field of natural language understanding and generation in recent years. These models are pretrained on vast amounts of data and subsequently aligned with human preferences gen- erate helpful and coherent outputs (Ouyang et al. 2022). However, despite the plethora LLMs and their impressive achievements, they still face inherent constraints on model size and training data. Further scaling these models is exceptionally costly, of- ten requiring extensive retraining on several trillion tokens.
Owant
Owent15 10B Chat
Wizard 8x22b
Mixtri-8c222 Mittn--acd2B-/attaat--hh
Liama-3-77B--haa-uu
cbns-istruct
17.5
35
52.5
70
Figure AlpacaEEval 2.0 LC win rates im- prove when provided with responses from the six models this figure. Table presents the template.
At the same time, different LLMs possess unique strengths and specialize in various tasks aspects. For instance, some models excel at complex instruction following (Xu et al. 2023a) while others may be better suited for code generation (Roziere al. 2023 Guo et al. (2024). This diversity in skill sets among different LLMs presents an intriguing question: Can we harness the collective expertise of multiple L.MMs create a more capable and robust model?Our answer to this question is Yes. We identify an inherent phenomenon we term the collaborativeness of LLMs wherein LLM tends generate better responses when presented with outputs from other models, even if these other models are less capable by itself. Figure [ showcases the LC win rate on the AlpacaEval 2.0 benchmark (Dubois al.) (024) for popular LLMs. We first tested each model independently, and then use each of them aggregator to
Work done while interning at Together AI https: thub. com cwwm//oggethercomppteerre.

---

Published as a conference paper at ICLR 2025
[Pronn]]
mIdiamediite Output]
Briemnoiiat: Duipur]
[Iiiemediate Outgut]
[Final Outgut]
Ai
Aa.1
A2
Agent
Tolser
concalorale
Loyer
Layer
Leyer
Leyer
Figure 2: Illustration of the Mixture-of-Agent Structure. This example shows 4 MoA layers where the first layer has proposers, the second and third layer have 3 aggregators that also serve as proposers for the next layer, and the last layer has one aggregator.
combine their outputs. We found when model is provided with answers generated by other models, its LC win rate significantly improves. This indicates that the collaborativeness phenomenon is widespread among LLMs. Remarkably, this improvement occurs even when the auxiliary responses provided by the other models are of lower quality than what an individual LLM could generate independently.
Based this finding, this paper introduces Mixture-of-Agents (MoA) methodology that leverages multiple L.Ms to iteratively enhance the generation quality. The structure of MoA is illustrated in Figure 2 Initially, LLMs in the first layer, denoted as Agents A1,1, ...A. independently gen- erate responses to given prompt. These responses are then presented to agents in the next layer A2. (which may reuse model from the first layer) for further refinement. This iterative refinement process continues for several cycles until obtaining more robust and comprehensive response.
To ensure effective collaboration among models and improve overall response quality, careful selection of I.LMs for each MoA layer is crucial. This selection process guided by two primary criteria: (a) Performance Metrics: The win rate of each model plays important role in determining their inclusion in MoA. (b) Diversity Considerations: The diversity of model outputs is also crucial, and responses generated by heterogeneous models contribute significantly more than those produced by the same model, as we show later in section 3.3 By leveraging these criteria - performance and diversity MoA aims mitigate individual model deficiencies and enhance overall response quality through collaborative synthesis.
We conduct comprehensive evaluations using AlpacaEval 2.0, Arena-Hard (Li al. 2024), MT- Bench (Zheng et al. 2023). FLASK (Ye et al. (202)) benchmarks for assessing the response quality across various dimensions. The results demonstrate substantial improvements with our proposed method, achieving SOTA win rate of 65.8% on AlpacaEvall 2.0, outperforming GPT-4 Omni.
The contributions of this work are summarized follows: Novel framework: we propose Mixture-ofi offturee-f-AAgntss framework designed to leverage the strengths of multiple LLMs, thereby improving their reasoning and language generation capabilities. (2) Finding of collaborativeness of language models: we highlight the inherit collaborativeness among I.LMs, where models tend to generate better quality responses when they have access to outputs from other models, even those outputs are of lower quality. (3) State-of-the-art LLM performance: we conducted extensive experiments using multiple highly-competitive benchmarks such as AlpacaEval 2.0, MT-Bench, and FLASK; our MoA framework achieves state-of-the- art performance on these benchmarks.
2 MIXTURE-OF-AGENT METHODOLOGY
In this section, we present our proposed methodology for leveraging multiple models to achieve boosted performance. We begin by demonstrating that LLMs possess collaborativeness and thus can improve their responses based on the outputs of other models. Following this, we introduce the Mixture-of-Agent methodology and discuss its design implications.
2

---

where Published as a conference paper at ICLR 2025
2.1 COLLABORATTIVENESS OF LLMS
We begin by demonstrating the collaborativeness of LLMs, specifically their ability to generate higher quality responses when they can reference outputs from other models. As we have shown in the introduction and Figure many of today's available LLMs exhibit this collaborative capability.
An important pathway to extract maximum benefits from collaboration of multiple LLMs to characterize how different models are good at in various aspects of collaboration. During the collaboration process, we can categorize LLMs into two distinct roles:
Proposers excel generating useful reference responses for use by other models. While good proposer may not necessarily produce responses with high scores by itself, should offer more context and diverse perspectives, ultimately contributing to better final responses when used by aggregator.
Aggregators models proficient in synthesizing responses from other models into single, high- quality output. An effective aggregator should maintain or enhance output quality even when integrating inputs that of lesser quality than its own.
Section 3.3 empirically validate the roles of aggregators and proposers. Specifically, we show that many LLMS possess capabilities both as aggregators and proposers, while certain models displayed specialized proficiencies in distinct roles, GPT-4o, Qwen1.5, LLaMA-3 emerged as versatile model effective in both assisting and aggregating tasks. In contrast, WizardLM demonstrated excellent performance an proposer model but struggled to maintain its effectiveness in aggregating responses from other models.
Given that an aggregator can generate higher-quality responses by building upon outputs from other models, we propose further enhancing this collaborative potential by introducing additional aggregators. One intuitive idea is to replicate the exercise with multiple aggregators using several to aggregate better answers and then re-aggregating these aggregated answers. By incorporating more aggregators into the process, can iteratively synthesize and refine the responses. leveraging the strengths multiple models to produce superior outcomes. This leads to the design of proposed Mfxture-of-Ageme
initially
2.2 MIXTURE-OF-AGOE
The structure of MoA is illustrated in Figure 2
It has layers and each layer-i consists of n LLMs. denoted by A,, A,,, .... Ar,. It is important to note that LLMs can be reused either within the same layer or across different layers. When many LLMs in a layer are identical, this configuration leads to a special structure that corresponds to model generating multiple possibly different outputs (due to the stochasticity of temperature sampling). We refer to this setting as only a sparse subset of models are activated.
Here, each LLM Ai,j processes an input text and generates its continuation. Our method does not require any fine-tuning and only utilizes the interface of prompting and generation of LLMs. Formally, given input prompt 11, the output of i-th MoA layer y be expressed follows:
y/ = + + æ1, Fi+1 y/i
where + here means concatenation of texts; + means application of the Aggregate-ande aatteeannn-SS-ysiiiiii prompt shown in Table to these model outputs. Table [able[[repree represents the template of system prompt, and the original user query is included immediately after this system prompt.
In practice, we do not need to concatenate prompt and all model responses so only one LLM needed to be used in the last layer. Therefore, we use the output of an LLM from the 1-th layer (Ai,(1:)) as the final output and evaluate the metrics based on it.
2.3 ANALOGY TO MIXTURE-OR-EXPERT
Mfxture-of-Eaae Ex-perts (MoE) (Shazeer et al. 2017) is prominent and well-established technique in machine learning where multiple expert networks specialize in different skill sets. The MoE approach has shown significant success across various applications due to its ability to leverage
3

---

Published as a conference paper at ICLR 2025
Table 1: Serregate-and-Synnae Prompt to integrate responses from other models.
You have been provided with of responses from various open-source models the latest user query. Your task is to synthesize these responses into single, high-quality response. It is crucial critically evaluate the information provided in these responses, recognizing that some it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability.
Responses from models:
1. [Model Response from A,
2. [Model Response from A,2]
7h. [Model Response from A
diverse model capabilities for complex problem-solving tasks. Our MoA method draws inspiration from this methodology.
typical MoE design consists of a stack of layers known as MoE layers. Each layer comprises a set of expert networks alongside a gating network and includes residual connections for improved gradient flow. Formally, for layer i, this design can be expressed as follows:
y/ = Σ Ea.(s) + x4 1=1
where Gi,j represents the output from the gating network corresponding expert j, and Fi, denotes the function computed by expert network j. The leverage of multiple experts allows the model to learn different skill sets and focus on various aspects of the task at hand. The gating network G dynamically routes to the appropriate experts, enabling efficient utilization of computational resources by activating only the specialized sub-networks necessary.
From high-level perspective, our proposed MoA framework extends the MoE concept to the model level by operating at the model level rather than at the activation level. Specifically, our MoA approach leverages LLMs and operates entirely through the prompt interface rather than requiring modifications to internal activations or weights. This means that instead of having specialized sub-networks within single model like in MoE, we utilize multiple full-fledged LLMs across different layers. Note that in our approach, we consolidate the roles of the gating network and expert networks using LLM, as the intrinsic capacity of LLMs allows them effectively regularize inputs by interpreting prompts and generating coherent outputs without needing external mechanisms for coordination. For this work, we design and evaluate MoA in a dense configuration, where all "expert" LLMs the network process the inputs. However, the method can be seamlessly extended to sparse configuration. dynamically selecting which I.LMs to generate.
Moreover, since this method relies solely on prompting capabilities inherent within off-the-shell models: (1) It eliminates computational overhead associated with fine-tuning: (2) It provides flexibility and scalability: our method can be applied to the latest LLMs regardless of their size or architecture.
EVALUATION
This section presents a comprehensive evaluation of our proposed MoA. Our findings show that:
1. We achieve significant improvements on AlpacaEval 2.0, Arena-Hard, MT--Bech, and FLASK benchmarks. Notably, with open-source models only, approach outperforms GPT-4o on AlpacaEval 2.0, MT-Bench, and FLASK.
2. We conduct extensive experiments to provide better understandings of the internal mecha- nism of MoA.
3. Through a detailed budget analysis, several implementations of MoA can deliver better performance GPT-4 Turbo while being 2x more cost-effective.
4

---

Published as a conference paper at ICLR 2025
3.1 SETUP
Benchmarks We mainly evaluate models on AlpacaEval 2.0 (Dubois et al. 2024). a leading benchmark for assessing the alignment LLMs with human preferences. It contains 805 instructions representative of real use cases. Each model's 's response is directly compared against that of the GPT-4 (gpt-4-14 101-1100-pp1- - with a GPT-4-based evaluator determining the likelihood of preferring the evaluated model's 's response. To ensure fairness, the evaluation employs length-controlled (I.C) win rates, effectively neutralizing length bias ?
Additionally, we also evaluate on Arena-Hart (Li et al. 2024), MTT-Bench (Zheng et al. 2023) and FLASK (Ye et al., 2023). Arena-Hard evaluates performance on 500 challenging user queries, encompassing diverse range of topics such as coding, mathematics, and logic puzzles. MT-Bench uses GPT-4 to grade and give a score to model's answer. FLASK, on the other hand, offers a more granular evaluation with 12 skill- skill-specific scores.
Models In our study, we constructed our default MoA by using only open-source models to achieve competitive performance. The models included are: Qwen 1.5-1 10h-Chat (Bai et al. (023). Qwen 1.5-
72B-Chat, WizardI M-8x22B (Xu et al. 2023a), Mixtral-8x22B-vv (Jiang et al., 2024), dbrx-instruct The Mosaic Research Team| 2024). We construct 3 MoA layers and use the same set of models in each MoA layer. We use Qwen 1- Chat as the aggregator in the last layer. We also developed a variant called MoA w/ GPT-4o, which prioritizes high-quality outputs by using GPT-4o the aggregator in the final MoA layer. Another variant, MoA-Lite, emphasizes cost-effectiveness. It uses the same set of models as proposers but includes only 2 MoA layers and employs Qwen 1.t.1l1------hi cost-effective than GPT-4o while achieving a 1.8% improvement in quality on AlpacaEval 2.0. We ensure strict adherence to the licensing terms of all models utilized in this research. For open-source models, all inferences were ran through Together Inference Endpoint We mainly use large models to prioritize accuracy here, but we observed similar performance improvements with smaller models as well. The results can be found in Appendix A
( (Touvron et al. (023b),
the aggregator. This makes it more
3.2 BENCHMARK RESULTS
In this subsection, we present our evaluation results on three standard benchmarks: AlpacaEval 2.0, Arena-Hard, MT-Bench, and FLASK. These benchmarks were chosen to comprehensively assess the performance of our approach and compare with the state-of-the-art LLMs.
We conducted comparisons against leading models such as GPT-4 and other state-of-the-ar open-source models. The detailed results are presented in Table 2 where our MoA methodology achieved top positions on the AlpacaEval 2.0 leaderboard, demonstrating remarkable 8.2% absolute improvement over the previous top model, GPT-4o. Moreover, it is particularly noteworthy that our model outperformed GPT-4o using solely open-source models, achieving margin of 7.6% absolute improvement from 57.5% (GPT-4o) to 65.1% (MoA). Our MoA-Lite setup uses less layers and being more cost-effective. Even with this lighter approach, still outperform the best model by 1.8%, improving from 57.5% (GPT-4o) to 59.3% (MoA-Lite). This further highlights the effecttivnnss of our method in leveraging open-source models capabilities with varying compute budget to their fullest potential.
AlpacaEval 2.0
Arena-Hard In the Arena- benchmark, our MoA with GPT-4o achieved 90.3% score, significantly outperforming GPT-4o's single model score 79.2%. Even the more cost-effective MoA- Lite variant showed strong results with a 71.3% win rate. These results highlight the robustness and efficiency of our approach in challenging prompts.
MT-Bench Though improvements over individual models on the MT-Beech are relatively incre- mental, this understandable given that current models already perform exceptionally well on this benchmark, single model alone achieve scores greater than 9 out of 10. Despite the marginal
This metric tracks closely with human preferences, achieving a Spearman correlation 0.98 with actual human evaluations (Dubois et al. 2024).
httpps//aat together.t......... hptpaa/apapgggga/cccctt.
5

---

Published as a conference paper at ICLR 2025
Table Results on AlpacaEval 2.0, Arena-Hard, and MT-Bench. MoA and MoA-Lite correspond the 6 proposers with 3 layers and with layers respectively. Moλ w/ GPT-4o corresponds to using GPT-4o the final aggregator MoA. Mon-Lite-Single has the same aggregator but only contains one proposer which is Qwenl.5 1 110B Chat. We our experiments three times and reported the average scores along with the standard deviation. denotes our replication the AlpacaEval results. We ran all the MT-Bench scores ourselves to get turn-based scores. 'Agg.' denotes the aggregated score with formula: (AlpacaEval-LC-vi)) + Arena-Hard-wae + MT-Bench--core x 10)/3.
AlpacaEval 2.0
Arena Arena-Hard Hard
MT-Bench
Model
Agg.
LC win.
win.
win.
Avg.
1st turn
2nd
MoA w/ GPT-40
83.3
65.7 257
78.7
90.3 % 77.4
9,40+o00
9.49
9.31
MoA
78.3
65.1 100%
59. 59.8103%
9.25+0.10
9,44
9.07
GPT-4 Gurbo (04/09)
76.7
55.0%
46.1%
82.0
9.31
9.35
9.28
GPT-4 Ommi (05/13)
76.2
57.5%
51.3%
79.2
9.19
9.31
9.07
MoA-Lite
74.1
59.3 %
57.0107%
71.3 %
9.181009
9.38
8.99
GPT-4 Preview (11/06)
73.6
50.0%
50.0%
78.7
9.20
9.38
9.03
Wizard 8x22B
70.1
51.3%
62.3%
71.3
8.78
8.96
8.61
MoA-Lite-Singd!
64.7
47.8%
37.9%
59.5
8.69
9.19
8.19
Qwenl.5 110B Chat
63.3
43.9%
33.8%
56.4
8.96
9.23
8,63
Llama 70B Instruct
56.8
34.4%
33.2%
46.6
8.68
GPT-4 (03/14)
53.9
35.3%
22.1%
37.9
8,84
9.08
8.61
Qwen 1.5 72B Chat
52.4
36.6%
26.5%
36.1
8.44
8.55
8.34
%
36.4
8.78
9.11
8.44
rabustrues
harmiessnnsss
corretnass
Appregator
y- 0.500
Propooor
0.001
condiseress
efficiency
00
readability
factuaity
LOwnnss
metatognition
completeness
comgrehersion
25
20
40 45 50
ingghtt mess
with individual
GPT-4 Ornii (05/13)
Qwen1 Qwen1.5-1108-40 5-1108-Chat
0115
Figure The 'As Aggregator' curve represents the score when the selected model serves as aggre- gator, using the proposers from the default MoA setup; The 'As Proposer' curve depicts the score when the selected model serves as proposer, gen- erating proposed answers 6 times, with Qwenl.5- 110B as aggregator.
Figure 3: Results on FLASK where we use the default 6-proposer MoA setup and Qwen 1.5- 10B is the aggregator. We include the results of GPT- 3.5, GPT-40, and Qwen1.5- standalone models for comparison.
1-5 10B when used as
enhancements, our approach still secures the top position on the leaderboard. This demonstrates that even with already highly optimized benchmarks, our method can push the boundaries further, maintaining the leadership.
FLASK FLASK provides fine-grained evaluation of models. Among those metrics, MoA excels in several key aspects. Specifically, methodology shows improvement in robustness, correctness, insightfulness, compared to the single model score the aggregator, Qwen- 110B-Chat Additionally, MoA also outperforms GPT-4 Omni in terms of insightfulness, correctness, factuality, completeness, and metacognition. One metric where MoA did not do as well was conciseness; the model produced outputs that moderately more verbose.

---

Published as a conference paper at ICLR 2025
Aggregation 1st aggregation ard aggregation
SWin1.S-1100
QVen155722
GPT4 Omn
LC 1C:000 rata
50
Aggrogator
WizandLM
Mitrral 8x228
Uama 70B Instrcct
dbni instrct
LLM-Ranker
20
Layer1
Layer
Layer
Layer
0.00
0.05
0.10
0.15
0.20
0.9
0.30
Spearman correlation coefficient
Figure 5: (a) LC win rate on AlpacaEval 2.0 with different aggregators in the 6-model Mixture-of- Agents setup. All the curves use the same proposer agents; they only differ in the choice of the final aggregator. The LLM ranker uses Qwen 1.5-1 (OB-Cha model with prompt format in Appendix Appendix C The GPT-4o model is only used to aggregate the output for the purpose of evaluation and does not participate as a proposer towards the next layer. (b) Spearman correlation between BL.UU scores (calculated using 3-gram, 4-gram, and 5-gram metrics) and win rate of the proposed outputs.
3.3 WHAT MAKES MIXTURE-OF-AGENT WORK WELL?
In this subsection, we conduct experiments that provide us better understandings of the internal mechanism of Mixture-of-Agent. We summarize key insights below.
Mixture-of-Agents significantly outperforms LLM rankers. First, we compare Mixture-of- Agents with an LLM-based ranker which uses the aggregator model to select one of the answers that are generated by the proposers, instead of generating new output. The results are shown in Figure 5 where we can observe that the MoA approach significantly outperforms LLM-ranker baseline. The fact that MoA outperforms the ranking approach suggests that the aggregator does not simply select one of the generated answers by the proposers, but potentially performs more sophisticated aggregation over all proposed generations.
MoA tends to incorporate the best proposed answers. also compare the aggregator's response with the proposers' responses via similarity scores such BLEU (Papineni al. 2002 which reflects n- n-gram overlaps. Within each sample, given n proposed answers by the proposers, we calculate the Spearman's rank correlation coefficient between the similarity scores and the preference scores determined by the GPT-4 based evaluator. The results in (Jindeed confirms a positive correlation between the win rate and the BLEU score. We also provide results with Levenshtein similarity (RapidFuzz. 2023) or TF-IDF as opposed to BLEU scores in Appendix B where both alternative approaches for textual similarities also yield positive correlation with the preference scores. For similarity comparison between proposers and aggregator, can refer to Appendix
Relationship of single model win rates to the MoA system. Our results indicate a positive relationship between the individual LLM's performance in each role (proposer aggregator) and the final performance of MoA. Through linear regression analysis, presented in Figure 4 with the x-axis representing the model's performance setting for the proposer one), and the y-axis representing the MoA's final performance, we observed that the regression coefficient for the aggregator model (0.588) is higher than that for the proposer model (0.281). This suggests that high-quality models are useful for both the aggregator and proposer roles. The steeper slope for the aggregator fit suggests that the final MoA performance more sensitive to the quality of the aggregator model than the proposer.
proposer/aggreeattor (we adopt adopt the Songle-Proposee
Effect of model diversity and the number proposers. We analyze how the number of proposals affect the final output quality by varying the number of proposers in each layer. We show the results in B we find that scores increases monotonically with n, reflecting the benefits of having more auxiliary information. In addition, also quantify the impact of using diverse
7

---

Published as a conference paper at ICLR 2025
Table 3: Effects of the number of proposer models on AlpacaEval 2.0. We denote as either the number of models in an MoA layer or the number of proposed outputs in the single-proposer setting. use Qwen 1 .55 10B--hCh as the aggregator and use MoA layers for all settings in this table.
Table 4: Impact of different models serving as proposers vs aggregators. When evaluating aggre- gators, all six models serve as proposers; when evaluating proposers, as the aggregator. use MoA layers here.
serves
Model
As aggregator
As proposer
Setting
Single-Proposee Proposer
Qwen1.5 1er.n.5--1100 10--111-----1 Chat
61.3%
56.7%
6
61.3%
56.7%
Qwen1.5-72B 5-BB--hh
59.3%
53.3%
3
58.0%
56.1%
LLaMA-3-700 70b--nstruuet
45.0%
60.6%
2
58.8%
54.5%
WizardI M 8x22B
52.9%
63.8%
n
47.8%
47.8%
Mixtral-8x22B-nnstii
48.4%
54.8%
dbrx-instruct
41.5%
55.1%
Table 5: Comparison Multi-Agent Methods.
Method
Aggregation
AlpacaEval2 (LC)
Avg Cost ($)
Standalone
0
43.9
61.3
0.00852
MoA
2
65.7
0.03150
1
53.5
0,00819
MAD
2
50.6
0,03140
47.6
0.00818
Reconcile
2
47.7
0.02910
set LLMs as proposers. For each n, we compare two settings: responses generated by the same I.LM with temperature of 0.7; and "multtple-ppoposee" where each response is generated by a different L.M.s. Overall, using multiple different L.MSs consistently yielded better results. Both results suggest that having a larger number of diverse LLM agents each MoA layer can improve performance. Further scaling the width of MoA is a promising direction of future investigation.
where the n
Specialization of models in the MoA ecosystem. We also conducted experiments to determine which models excel in specific roles. Specifically, Table 4 shows that GPT-4o, Qwen, LLaMA-3 emerged as a versatile model effective in both assisting and aggregating tasks. In contrast, WizardLM demonstrated excellent performance as an proposer model but struggled to maintain effectiveness in aggregating responses from other models.
Comparison Existing Multi-Agent Methods We conducted comparative experiments to evalu- ate MoA against existing multi-agent methods like MAD Liang et al. (2023) and Reconcile Chen et al. (2023a), which typically focus on tasks with short, deterministic answers and are not directly applicable ended, contextually rich chat scenarios. We adapted their prompts for chatting and removed features like JSON outputs and confidence levels for Reconcile.
Our results (Table show that MoA consistently outperforms MAD and Reconcile. With round of aggregation, MoA achieves an AlpacaEval2 score of 61.3, surpassing MAD's 53.5 and Reconcile's 47.6. This gap widens with two rounds, where MoA scores 65.7 compared MAD's 50.6 and Reconcile's s 47.7. MoA scales better with more aggregation rounds, and maintain cost efficiency, achieving better performance within similar budget constraints.
3.4 BUDGET AND TOKEN ANALYSIS
To understand the relationship between budget, token usage, and LC win rates, we conducted budget and token analysis. Figure 6a and Figure 6b illustrate these relationships.
8

---

Published as a conference paper at ICLR 2025
65
65
Alpacetvvll 2.0 Lengith Controlled Sigare
Mokiine
Alpacatvell Controlled
GiT:6-turto
50
50
model type Muli Proposer
model type
My. li Proposer Single Proposer
35
Sing la Propoier
25
layer
3.000
0.005
0.010
0.015
0,020
0.025
0.030
0.355
50
100
150
200
250
300
350
Cost
trlops
(a) LC win rate cost
(b) LC win rate tflops
Figure 6: (a) Performance trade-off versus cost. The dots (layer > 2) with the same size indicate MoA with different final aggregator. (b) Performance trade-off versus the number of tera floating operations (tflops), which we use as a proxy for latency. Note that we calculate the sum over layers of the max number of tflops among proposers in each MoA layer as multiple proposers can run in parallel. Our plots illustrate a Pareto frontier that strikes an optimal balance between performance and cost. We show that the MoA approach lies on this Pareto front, as opposed to GPT-4 Turbo and GPT-4o which are not cost-optimal and are more expensive compared to MoA approaches of the same LCC win rate. Single Proposer: uses the same model to generate multiple responses in each MoA layer, Multi Proposer: uses different models in each MoA layer. The actual tflops of GPT-4 is unknown, so we use the rumored size from the community of an 8x220B architecture.
Cost Effectiveness Figure 6a we plot the LC win rate against the average inference cost for each instance in the AplacaEval 2.0 benchmark. The cost is calculated based on pricing information available from API provider websites This helps identify cost-effective models that achieve high performance without incurring excessive expenses. The chart reveals Pareto front where certain models strike an optimal balance between cost and performance. Models closer this Pareto front more desirable as they provide better monetary value by delivering high LC win rates at lower costs. Specifically, if we prioritize the quality, MoA is the best configuration. However, if we want to strike a good balance between quality and cost, MoA-Lite higher level of quality. outperforms GPT-4 Turbo by approximately 4% while being more than twice cost-effective. We have also benchmarked MoA using small models (S9B) presented in Appendix A
match GPT-4o's cost while achieving
Tftops Consumption Figure 6b| depicts the relationship between LC win rate and the number of tflops. Here we use the number of tflps as a proxy for latency since latency can vary depending on the inference systems. This analysis is understand how different models manage their budgets while maintaining or improving performance levels. Similar to the cost efficiency analysis. Pareto front can be observed here as well. Models on this front effectively utilize their computational resources to maximize their LC win rate. For models accessed via inference endpoints, latency may not directly correlate with tflops, as other computational demands (e.g., batching) and factors (e.g., server load) can influence actual response time. However, we use tflops as an approximate indicator of the relative resource intensity of different models, as it reflects the theoretical computational requirements.
'For open-source models, we calculate the price using data from wttp://///pccccccccc httpt//ttr/////poooo////////o/oa models] for OpenAl models, we use pricing details from httpps:/oppmaaa ing data was retrieved of May 22, 2024.
ai/ Pric-

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
4.1 LLM REASONING
In order improve generation quality of LLMs, recent researches have experienced great progresses in optimizing LLMs to various downstream tasks through prompt engineering. Chain of Thought (CoT) (Wei et al.] 2022 Kojima et al., 2022) prompting techniques represent linear problem- solving approach where each step builds upon the previous one. Fu et al. (2022) applied CoT to multi-step reasoning tasks. To automate CoT prompting, Auto-CoT (Zhang et al.) 2022b) constructs demonstrations by sampling diverse questions and generating reasoning chains. Active- Prompe (Diao itt al., 2023) focuses on selecting the most uncertain questions for task-specific annotations, PS Prompt ( Wang al. 2023) decomposes tasks into subtasks. Tree-of- Trreuufh (ToT) (Yao al. (023a) expands on the reasoning process by considering multiple paths of reasoning and self-evaluating choices. Effective Graph-of-Though (Yao et al., (023b) frames thoughts as graphs. Natural Program prompting (Ling et al. 2023) is proposed for better solving deductive reasoning tasks. And re-reading prompt (Xu al. 2023b) revisits question information embedded within input prompts.
4.2 MODEL ENSEMBLE
A straightforward solution to leverage the strengths of multiple models is reranking outputs from different models. For instance, Jiang et al. (2023) introduce PAIRRANKER, which performs pairwise comparisons candidate outputs select the best one, showing improvements instruction dataset. To address the substantial computational costs associated with multi-LLM, inference, other studies have explored training router that predicts the best-performing model from a fixed set of LLMS for given input (Wang et al. 2024a; [Shitzer et al.. 2024 Lu et al. 2023). Additionally, FrugalGPT (Chen (2023b) proposed reducing the cost of using L.MMs by employing different models in a cascading manner. In order to better leverage the responses of multiple models, Diang et (023) trained GENFUSER, a model that was trained generate an improved response to capitalize on the strengths of multiple candidates. Huang et al. (2024) proposed to fuse the outputs of different models by averaging their output probability distributions.
self-constructed
Another line of work is multi-agent collaboration. Several studies explore using multiple large language models as agents that collectively discuss and reason through given problems interactively. Du et al. (2023) establishes mechanism for symmetric discussions among agents. Around the same time, MAD (Liang et al. 2023) introduces an asymmetric mechanism design, with different roles, i.e. debater and judge. Other similar works include (Chan et al. 2023 Xu et al. 2023c Liu et al. 2023 He et al. 2023). Morcover, ReConcile (Chen et al. 2023a) exemplifie an asymmetric discussion involving weighted voting. To understand discussion more deeply. Zhang et al. (2023) and Chen ct al. (2023c) aim to explain such collaboration mechanism a social psychology view. Wang et al. (2024b) compared multi-agent approaches and found single agent with strong prompt including detailed demonstrations can achieve comparable quality to multi-agent approaches.
5 CONCLUSION
This paper introduces a Mixture-of-Agent approach aimed at leveraging the capabilities of multiple I.MMs via successive stages for iterative collaboration. Our method harnesses the collective strengths of agents in the Mixture-of-Agents family, and can significantly improve upon the output quality of each individual model. Empirical evaluations conducted on AlpacaEval 2.0, MT-Bench, and FLASK demonstrated substantial improvements in response quality. with our approach achieving the LC win rate up to 65%. These findings validate our hypothesis that integrating diverse perspectives from various models can lead to superior performance compared to relying on single model alone. In addition, we provide insights into improving the design of MoA; systematic optimization of MoA architecture is an interesting direction for future work.
10

---

Published as a conference paper at ICLR 2025
Limitations. Our proposed method requires iterative aggregation of model responses, which means the model cannot decide the first token until the last MoA layer is reached. This potentially results in high Time to First Token (TTFT), which can negatively impact user experience. To mitigate this issue, we can limit the number of MoA layers, as the first response aggregation has the most significant boost generation quality. Future work could explore chunk-wise aggregation instead of aggregating entire responses at once, which can reduce TTFT while maintaining response quality.
Broader Impact. This study holds the potential to enhance the effectiveness of LLM-driven chat assistants, thereby making Al more accessible. Moreover, since the intermediate outputs that are expressed in natural language, MoA presented improves the interpretability of models. This enhanced interpretability facilitates better alignment with human reasoning.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1991, 2020.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint (tt.::300.0020011 2023.
Justin Chih-Yas Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309. /3007, 2023a.
Lingjiao Chen, Matei Zaharia, and James Zou, Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv: 00.55.557770 /5676,77660 2023b.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Ya- Ting Lu, Yi-Hsin Hung, Cheng Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. International Conference on Learning Representations, 2023c
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.023111 2022.
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-of-thouge for large language models. arXiv preprint arXiv:2302. /2246, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint 13355
2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint 0444: 2024.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00 10.00720, 2022.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li. et al. Deepseck-coder: When the large language model meets programming- the rise code intelligence. arXiv preprint arXiv: 14196. 2024.
Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, and Jun Zhao. LEGO:
A multi-agent collaborative framework with role-playing and iterative feedback for causality explanation generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2033/3 findings-emnlpp URL httpt:: /eattss//acll antho ogy. wgg..orrg/223333 fid ngs emn 1p. 613

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021b.
Yichong Huang. Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. En- abling ensemble learning for heterogeneous large language models with deep parallel collaboration. arXiv preprint arXiv:2404.1271. 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril. Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401.04088, 2024. doi: 10.48550/ARXIV.24011 240.008 URL
https: //w.i. www.rrl...... 48550/ a rxi v. 2401 04088
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language mod- els with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1165-11178. Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v1/200 act.://....llo-..99999 URL
http:///acaatthhoocooooo www.aatooow...///2...........
-aoaaaclloonn..... 792
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22211 2022.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-han and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint 1.Xi:23305.19118 2023.
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. arXiv preprini urXiv:2306.03872, 2023.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An Ilm-agent collaboration framework with agent team optimization. arXiv preprini arXiv:2310.02110, 2023.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward- sward-ggidee guided ensemble of large language models, 2023.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311331111111 1-318. ACI, 2002. doi: 10.33115/107303331077333 URL /ttps/////////eaaaatooooooooooo
RapidFuzz. python-levenshtein by rapidfuzz.
https://qithub.com htt.e/uuu.oom/raaag//e
Python-1eevnshteiin 2023.
12

---

Published as a conference paper at ICLR 2025
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely mixture-of-expene layer. arXiv 2017.
preprint
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2024. URL
net httt:/oommmmmi-----aaa
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1 13003-13057 Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. (0..18633/1120000 1/2023. htw.2.....nnnnnaal...3 acl.224. URL httphttts///aaaateeee Findings-acl 824
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsensequ 2.0: Exposing the limits of ai through gamification. NeurlPS Datasets and Benchmarks, 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
The
Mosaic 11m.
Research Team.
Introducing dbrx: A new state-of-the-art
open
2024. introducing-abrre stte-eabbbee-----t-e---ee-----e- stt
URL https:/ /www www.bricks.......o www.coooool...... /
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023b.
Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. In The Twelfth International Conference on Learning Representations, 2024a. URL
net http//forrrmmmiddPMMMMMMM
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy KeiWei Lee, and Ee- Lim. Plan-and-solve prompting: Improving zero-shot fainnoff-hoough reasoning by large language models. arXiv preprint arXiv:2305.0409) 2023.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of Ilm reasoning: Are multi-agent discussions the key? arXiv preprint 27X2:2 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprini ar 2203 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint 23044:2304 12244, 2023a.
13

---

Published as a conference paper at ICLR 2025
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian-guang Lou. Re-reading improves reasoning in language models. arXiv preprint arXiv:2309.0622 2023b.
Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. Towards reasoning in large language models via multi-agent peer review collaboration. arXiv preprint wrXiv:2311.08133 2023c.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan, Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305, 10601, 2023a.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2300. 2023b.
Seonghycon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thome, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets, arXiv preprint arXiv:2307.10928, 2023.
Jintian Zhang. Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint arXiv:2310.021244 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-tre rre-trained transformer language models. arXiv e-prints, pp. arXiv-2205, 2022a.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.00668, 2023.
14

---

Published as a conference paper at ICLR 2025
Table 6: Results on small models. When benchmarking each aggregator, we use the following models as proposers: gemma-2-9b, 1lam--3.1-8b,
qwen-1.5-7b.
Aggregator
Layers
Alpaca (LC)
Alpaca (win)
Arena-Hard
MT-Bench
gemma-2-9b
1
48.54
36.26
40.6
8,49
56.54
48.20
47.5
8,44
3
56.83
49.47
47.8
8.50
1lama-3.1-8b
26.06
27.48
28.0
8.34
2
29.52
38.48
34.9
8.39
33,34
42.68
36.6
8.49
mistral-7b-instrrue--0.3
19.88
15.67
16.3
7.59
2
26.93
24.68
22.1
8.26
3
27.98
27.79
24.1
8.17
qwen-1.5-7b
16.58
13.12
12.6
7.64
2 25.54
24.36
20.7
7.98
3
28.94
29.91
23.2
7.86
Aggregation 1st aggregation
Aggregation
DWWent. 5-1100
0Wle11.5-13 1-11115--1100
1st aggregation
2nd eggregation aggregation
2nd aggregation
@Wen15:722
QWenn15-72 728
3rd aggrepuiion
Aggrepato
Aggregator
Uama-3-708
Uama-3-7
Mistral-bx228
Mixtral-xx228
obnx Instruct
obrx instruct
0.00
0.06
0.10
0.15
0.20
0.55
0.90
0.05
0.30
0.15
6.20
0.25
correlation coeffident
Spearman correlation confficient
Figure 7: (a) Spearman Correlation using TF-IDF similarity; (b) Spearman Correlation using Leven- shtein similarity.
A RESULTS ON SMALL MODELS
We have benchmarked and observed similar performance improvements with smaller models as well. As shown Table 6 for models with <9B parameters, MoA can improve performance by up to 1 12% compared to individual models. Specifically, using gemma-2-99 2-9b as the aggregator, we achieve a 56.83% L.C win rate with small models alone, outperforming GPT-4 and comparable to GPT-4o. These results demonstrate that MoA can achieve significant performance improvements even with reduced computational resources. Note that for MT-BBench sometimes layer worse than layer mostly due smaller models have less multi-turn capability. layer MoA is consistently better than the original model.
B SPEARMAN CORRELATION USING DIFFERENT SIMILARITY FUNCTIONS
We present results using TF-IDF-based similarity and Levenshtein similarity when calculating the Spearman correlation. Specifically, within each sample of n proposed answers, we calculate Spearman correlation coefficient between the similarity scores and the preference scores determined by the GPT-4-basel evaluator. As shown in Figure 7 there is indeed a positive correlation between win rate and both TF-IDF similarity and Levenshtein similarity.
15

---

Published as a conference paper at ICLR 2025
C LLM RANKER
This section introduces the setup the to evaluate and rank the best output generated by some LLMs. Below presents the template for prompting the model during these evaluations.
used in this paper. The LLM--anked designed
Prompt for ranking with LLMs
You are a highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality of their responses to given instruction. This process will be used to create leaderboard reflecting the most accurate and human-preferred answers. I require leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, and select the model that produces the best output from a human perspective.
## Instruction
"instruction":
(innstruucioon """,
## Model Outputs
Here are the unordered outputs from the models. Each output is associated with a specific model, identified by unique model identifier.
"model pondellidentifferr "[identifier_1)" "output": (output_tt)"
proodeliddentffien "[identifier_]) "output": """(output_2)"m
"model_ iinieeliedettiiee" (identiifier_]) "output": (""(ouput_33) "M*"
"model_identifierr" "[identifier_4"" "output": """(output44)" "M*
proodel_identifiie"" "(identifeer_""" "output": """ ""(ootuut_5)"
"model iindeiledentifieee "(identifier_6)" "output": """ ("utpuuut_6]
## Task
Evaluate the models based the quality and relevance of their outputs, and select the model that generated the best output. Answer by providing the model identifier of the best model. We will use your output as the name of the best model, so make sure your output only contains one of the following model identifiers and nothing else (no quotes, spaces, new lines, ...).
## Best Model Identifier
16

---

Published as a conference paper at ICLR 2025
Table 7: Case: Some models produce high quality answers.
Role
Preference
Content
User
want to talk about "Smooth", song by Rob Thomas.
0.35
It's perfect blend of rock, pop, and Latin music. showcasing
Santana's signature guitar work and Rob Thomas's 's soulful vocals
Qwenl.5-72B-CHN
0.00
Llama-3-70B-Insse
0.00
it's collaboration between Rob Thomas, the lead vocalist of Matchbox Twenty, and Santana, the legendary guitarist...
WizardLM-228x22BB
0.99
It spent 12 weeks at the top of the Billboard Hot 100 chart in the United States capturing the song's is sultry and energetic vibe
Mixtral-&x22B- (0-.--22-------t---)))
0.00
dbrx-instrue
0.00
Aggregated
0.99
"Smooth" is actually a collaboration between Rob Thomas, who the lead singer of the band Matchbox Twenty, and the renowned Latin rock guitarist Carlos Santana This iconic song blends rock, pop. and Latin music elements, showcasing Santana's 's signature guitar work Rob Thomas's distinctive vocals The song was a colossal success, spending 12 weeks at the top of the Billboard Hot 100 The music video, set in vibrant club scene, complements the song's sultry vibe
(Qwen1.5-1108 110B-Chat)
Table 8: Case: all proposed responses are not good enough.
Role
Preference
Content
User
How do you become an author?
Qwen1.5-1B 1aa1l55-11B--Chh
0.00
Becoming an author involves several steps and 3. **Choose your niche 8. **Understand the publishing industry 10.
#uSell-publishing options*s:
Qwenl.5-72B- Chat
0.00
Llama-3-70B Instruct
0.16
Becoming an author can be a rewarding and fulfilling career... 1. *eDeveoop love for reading and writing **.... 10. **Market and promote your work * .... 11. **Continoously improve**:...
WizandL.M-288222B
0.03
I... 4. Finish your work...
Mixtral-&x22B- 8x222--nsttuett-00
0.00
dbrx-instruc
0.00
Aggregated
0.33
Becoming an author involves dedicated journey.. tivate a Love for Writing and Reading 4. "Chhoose Your
* *Cll-
(.wen1.5-11g 111.5-1-10---BB 10B-Chat)
Niche 5. **Finish Your *o 10. PuSelf-ubblishng
TT. Markettng and Promotion 12. & Continuous Learning and Writin**...
17

---

Published as a conference paper at ICLR 2025
D CASE STUDY
We present case study in this section. Due to the length of the responses generated by all models, we will only show selected fragments for brevity. To illustrate how the aggregator synthesizes the response, underlined similar expressions between the proposed responses and the aggregated response in different colors. omit the content that all proposed responses have mentioned.
Table showcases the responses generated by different proposers. The aggregated response generated by Qwen w.vel.33- 1.3-1 110--Cht reflects a high preference for its own content but also incorporates key points from Alama-3-70B Instrrcctt and WizardLM 8x22B. Notably, GPT-4's preference score for WizardLM 8x22B's ' response is 0.99, and the final aggregated answer also achieves a preference score of 0.99.
where of the proposed responses achieve high GPT-4 preference score. Despite this, the aggregator successfully identifies and incorporates the strong points from these responses, achieving preference score of 0.33.
Meanwhile, Table 8 presents another
E REASONING EVALUATIONS
Here, we demonstrate that our approach is applicable to reasoning tasks including the MATH dataset Hendrycks et al. (202Tb). Big-Bench Hard (BBH) [Suzgun al. (202)), MMLU Hendrycks et al. (2021a) and CSQA Talmor et al. (202I). Specifically, BBH evaluates models on 23 multi-step complex reasoning tasks; MMLU contains 57 tasks knowledge and reasoning; CSQA includes difficult commonsense reasoning questions. We posit this covers a wide range domains including coding, math, knowledge, commonsense QA and complex reasoning.
The results are presented in Table where we show that our method consistently enhances accuracy, This indicates that approach effective for variety of reasoning tasks. Notably, method complementary to existing reasoning techniques such as Chain of Thought WWi et al. (022) and Self-consistency Wang et al. (2022).
In addition, we investigate whether more layers MoA can further increase performance on the MATH dataset. As shown in Table 10 MoA with three layers improves from having just two layers.
Table 9: Performance comparison across reasoning benchmarks.
Model
BBH
MMLU
CSQA
MATH
Average
Dwenl.5-72B-CH.
0.619
0.6931
0.8231
0.428
0.641
Owenl.5-110B--
0.6733
0.7624
0.8346
0.500
0.693
Wizard 8x22b
0.7461
0.7989
0.7871
0.544
0.719
0.6693
0.7821
0.8075
0.282
0.635
Llama-3-70b-chat--
0.7438
0.7978
0.8305
0.456
0.707
dbrx-instruct
0.3552
0.6867
0.7625
0.314
0.530
MoA-Lite
0.7667
0.8268
0.8444
0.570
0.752
Table 10: Results on the MATH task. We evaluate different aggregators, with all six models serving as proposers each MoA layer.
Aggregator
Layer
Layer
Layer
Qwenl.5 55-72B-Chaa
0,428
0.526
0.552
Qwen 5- 110B-hhh
0.500
0.570
0.576
Wizard 8x22b
0.544
0.574
0.580
Siatrall8x22B-Instntee
0.282
0.534
0.556
1lama-3-70B-Innttre
0.456
0.584
0.578
dbrx-instruct
0.314
0.456
0,522
18

---

Published as a conference paper at ICLR 2025
Table 11: Performance comparison of MoA-Lite and MoA searched using our proposed optimization method.
Model
Aggregate
AlpacaEval (LC)
Arena-Hard
MT-Bench
MoA-Lite
74.1
59.3
71.3
9.18
MoA-Lite searched
75.0
62.0
71.8
9.11
F SEARCH FOR AN OPTIMAL MOA ARCHITECTURE
We found automatic architecture optimization method convenient for practical use, especially when new models are introduced. We implemented a relatively basic optimization method to select the set of LLMs as a proof of concept.
Setup Specifically, fix the number of layers to be two and the aggregator to be Qwen-1.55 10b- Chat, and set the numher models and which model in proposers to be variables for optimization. We utilized problem.
fllecche-GGoldfabbb
algorithm (BFGS) for this unconstrained optimization
Validation Data It is important to have a good set of validation data. We randomly sampled 50 problems from AlpacaEval and 50 from wera-Hard. The combined size of 100 enables to verify architecture performances quickly. We averaged the scores AlpacaEval and ArenaHard to be our final metric.
We ran the optimization and found the best mixture to be WizardI M-2-8x22b, Qwen-1.5-11 Chat, Twen-1.5-72b-CH. and three 1omma-3-7Ob-Instrune as proposers and Qwen-1.5- 1.m0111.-110b-haa as aggregator. The resulting mixture outperforms our MoA-Lite on two out of the three benchmarks as shown in Table
G INDIVIDUAL PROPOSER'S S SIMILARITY TO AGGREGATOR
We conducted ablation study to determine whether the aggregator merely repeats the outputs of the proposers or performs additional processing. To investigate this, we calculated the similarity between the outputs of individual proposers and the aggregator using BLEU scores (considering 3 3-grams, 4-grams, and 5-grams).
The results, shown in Table 12 indicate that the aggregator tends to exhibit the highest similarity with its own generation (Qwen I. .5- 10B-Chat). However, the overall similarity scores remain relatively low, suggesting that the aggregator is not simply replicating the proposers' outputs but is instead synthesizing or refining them.
Additionally, we observed that as the layer depth increases, the similarity between the aggregator and the proposers also increases. This indicates that responses across layers grow more aligned, potentially reflecting convergence toward consensus or refinement as the sequence progresses.
H
GPT MODELS USED
Here's consolidated list of the GPT-family models used across the experiments:
GPT-3.5-turbo-0125 Referenced in Figure 3
GPT-4 Preview: Referenced in Figure
GPT-44turbo: A more cost-effective variant of GPT-4 with improved latency and efficiency, referenced Figure 6a and Figure 6b
GPT-4o: We 4p1-40-2024-0513 across the paper.
19

---

Published as a conference paper at ICLR 2025
Table 12: Similarity scores between proposers and aggregators in MoA-Lite and MoA settings.
Aggregator
Score
MoA-Lite (Qwenl.5-100--Char
Owenl.5-110B--
0.3972
Wi
0.3742
(reenl.5-72BCCh
0.3528
Blama-3-770B-nsttnn
0.3230
0.3149
dbrx-Instruct
0.3062
MoA
(Qwen1.5-110B-Chhll
Owenl.5-110B-
0,5102
0,4539
Dwenl.5-722B-6
0,4009
Llama-3--0BB-Instruae
0.3827
Mixxral-xx22BB-nnstrr-tt--
0.3730
dbrx-Instruct
0.3640
EFFECT OF PROPRIETARY MODELS
In this section, we investigate whether a bit more in-depth into the incorporation of gpt-40-2024- 05-13 in MoA. Specifically, we try incorporate it both as aggregator and as proposer. When using GPT4-o as a proposer, replacing abbx-Innsruce we can see a clear boost in performance for AlpacaEval and Arena- Hard scores while maintaining MT-Bench score demonstrated in Table [3] Using GPT-4o as an aggregator increases the performance significantly. The aggregate score jumps from 74.1 to 83.3.
Table 13: When incorporating GPT-4o into MoA, we can see clear benefits in terms of benchmark improvements. Although more improvements come from using GPT-4o the aggregator. We use 201-40-2024400-11 for this experiment.
Model
Agg.
AlpacaEval 2.0 (LC)
Arena Hard
MT-Bench
MoA-Lite
74.1
59.3
71.3
9.18
MoA-Lite w/ GPT-4o replacing dbrx
75.7
63.1
73.1
9.1
MoA-Lite w/ GPT-4o
83.3
65.7
90.3
9.4
20


Reviews:
[{"review_id": "Hnuoy2J7p9", "reviewer": "Reviewer_U7wn", "summary": "This paper proposes a mixture-of-agents (MoA) framework that leverages the collective insights of multiple large language models (LLMs).\n\nThe core process involves several iterative steps: (1) multiple LLMs generate initial draft proposals; (2) these proposals are shared with all LLMs in the following iterations to generate refined proposals; (3) this second step is repeated multiple times, with an aggregator ultimately producing the final response.\n\nExperimental results on four benchmarks (AlpacaEval 2.0, MT-Bench, Arean Hard, and FLASK) demonstrate that MoA can outperform GPT-4 Omni.", "questions": "1.  Why do you call your method as mixture of agents instead of mixture of LLMs.   I did not see any behaviours of agents. \n\n2.  Are there any lines missing in Figure 6 (a) and (b)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. To the best of my knowledge, the proposed framework is both novel and reasonable. MoA can be viewed as a specific method for combining multiple weaker models to create a stronger model.\n\n2. The model's performance is competitive, showing improvements over GPT-4 Omni on three benchmarks.", "weaknesses": "1.  The proposed model is more resource-intensive than single LLM-based models.  \n2.  Most evaluations use LC metrics, with only a limited evaluation on MATH tasks included in the appendix. Further evaluations on diverse tasks are necessary to illustrate the general advantages of the proposed method.  \n\n3. An important question is how to select the set of proposal LLMs. Currently, the paper demonstrates two setups: one with relatively large models and one with smaller models. However, there is no guideline or formal strategy for choosing LLM sets. Additionally, it would be interesting to see the performance of a mixed set containing a small number of large LLMs and a majority of small LLMs.  Combining models with different language data is also a very interesting topic. Another crucial question is determining the number of layers. Aside from treating it as a hyperparameter, is there a way to determine an optimal layer count for a given set of proposal LLMs?  \n\n4. GPT-4o is only used as the aggregator. It would be informative to evaluate its performance if it were included as one of the proposal LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WORNXjdXN3", "reviewer": "Reviewer_NT9P", "summary": "The paper proposes Mixture-of-Agents (MoA) as a possible pathway to leverage the collective strengths of multiple LLMs.  By stacking LLMs into multiple layers and employing the output of LLM in previous layers, the frameworks allows a iterative refinement and strike a balance between inference cost and the performance.", "questions": "See the weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Proposal of a new effective framework to employ the collective intelligence of multiple LLMs.\n+ Empirical evaluation on AlpacaEval 2.0, Arena-Hard, and MT-Bench verifies the effectiveness of the proposed solution.", "weaknesses": "+ Stacking LLMs into layers and revising the output obtained from previous layers seems like another form of model ensemble and I would suggest including model ensemble as one of the comparative methods.\n+ In Figure 6, the max number of tflops among proposers in each MoA layer is used as an approximation of the total tflops of the entire layers since different proposers can run in a parallel way. However, the approximation is only reasonable when considering the inference latency for a single query. Otherwise, if we have a batch of data, with the same computation resources MoA can introduce extra latency since the parallel of proposers limits the data parallel size. \n+ The evaluation of MoA is only performed on instruction-following benchmarks. More evaluation of reasoning and commonsense knowledge benchmarks such as MMLU and BBH is necessary for a comprehensive understanding of the efficacy of MoA.   \n+ Detailed ability on the specific role of each proposer is missing. How does each proposer contribute to the final output? Is it possible for a proposer to simply repeat the output from previous layers? Will the models in later layers revise the mistakes made in previous layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iz3if6bpFq", "reviewer": "Reviewer_YS2v", "summary": "This work presents a Mixture of Agents (MoA) Architecture. This multi-layer architecture harnesses the collaborativeness of LLMs, where agents (models) in later layers aggregate and combine auxiliary information from previous layers with the user task to yield the final output. They evaluate this proposed architecture on AlpacaEval 2.0, Arena-Hard, MT-bench, and Flask and show that their architecture performance surpasses GPT-4o. \n\nThe \"collaborativeness\" of LLMs is defined as the phenomenon where the LLM generates better responses when presented with outputs from other models, even if these models are less capable. \n\nThe architecture presented has two types of models: \"proposer models,\" which generate useful references for use by other models; these models offer \"perspective\" or might even act as auxiliary/additional information providers to the aggregator. The other type of models present are \"aggregator models,\" which are proficient in combining responses from other models into a single high-quality output. \n\nThey not only show an MoA architecture but also a MoA-Lite architecture that outperforms GPT-4o and is both better and more cost-effective. They conduct various empirical experiments showcasing the effectiveness of their approach to MoA outperforming LLM-Ranking, the ability to incorporate best answers, and the effects of model diversity and different proposers.", "questions": "1] What is the point of the tflops consumption analysis? And where are the numbers for these obtained from? Gpt-4o and Gpt-4-turbo are closed-source models, so it is unclear where those numbers are obtained.\n\n2] In a similar vein, can tflops really be a proxy for latency where one is used to perhaps look at real-time serving of models while the other looks at consumption? Especially when the system in question is not locally run but uses an inference endpoint (as mentioned in section 3.1)\n\n3] While you analyzed a model's propensity to be an aggregator or a proposer, did you consider combining models that might be good at different kinds of tasks (for example, math or science)?\n\n4] Was there any reason to include LLM reasoning in the related work when the work does not explicitly show gains on any reasoning dataset and the evaluation is mostly on instruction following/preference benchmarks?\n\nLast but not least, just a suggestion for the paper's readability: it would significantly enhance the analogy to MoE if there was an explicit mention that MoE uses routing/gating networks to sometimes choose between routing between single or multiple experts. They specialize in different aspects vs traditional networks where all input data is processed through every layer. There is a mention in the work that the MoA, from the specification of the architecture, consolidates the role of the gating network and expert network using a LLM. Still, it is not the case on some level because it is not choosing which experts it will route its output from, and the input does pass through all layers, the aggregator picks and chooses information from the proposer LLMs but it does not necessarily always just drop some inputs.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- While collaborativeness has been harnessed in various ways, a layered funnel architecture in which earlier layers add information for later layers to consume and interplay to summarize these outputs efficiently to yield a final output has not been explored. \n\n- The authors also thoroughly conducted their experiments to establish collaborativeness and benchmark various datasets. The usage of open-source models to showcase the results helps to make replicating these results possible. \n\n- They also analyze the cost-effectiveness of these models compared to some base models, showing the effectiveness of applying this architecture to obtain better results at a cheaper rate.", "weaknesses": "- The idea of collaborativeness or hierarchical processing in LLMs is not exactly novel [1][2]; if you think of different layers in the architecture using the same model, this reduces to some form of iterative refinement of outputs as shown in [2]. \n\n- Some of the analysis in the paper to support budget analysis is unclear.\n\n\n\n#### *References*\n1] [2308.10848] AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors \n\n2] LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation - ACL Anthology", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52qlHUyarp", "reviewer": "Reviewer_or3e", "summary": "The paper proposes a method to use multiple LLMs as agents to improve the quality of generated responses to arbitrary queries. Agents are deployed in mixture-of-agent (MoA) layers. In the first MoA layer, multiple agents generate an answer to the input query independently. In subsequent layers, each agent in layer $T$ generates an answer to the query while also seeing the answers of all agents from layer $T-1$. At the last layer, a single aggregator LLM is used to combine all answers generated by the agents in the last MoA layer into a single answer. The proposed method shows state-of-the-art performance on multiple benchmarks compared to recent models from the GPT family including GPT4-omni, GPT4-turbo.", "questions": "- Do you use humans in your evaluation in any part of your paper? It seems that all your tests use (language) models for evaluation. Did you conduct any (manual/human) analysis of the outputs generated by the models at any stage?\n- An experiment that could be more prominently contextualised is the case where there is one aggregator/proposer agent in each layer (numbers in Table 3, discussion in **Effect of model diversity and the number of proposers**. This is kind of a middle-case between the LLM-as-reranker baseline (which you clearly outperform) and your proposed idea in full generality. In the case of a single proposer, your win rate in Table 3 is 47.8% (so not better than the baseline). I think it would be very valuable to include this setting as a baseline also in all other experiments and not only those for AlpacaEval2.0, since this really shows when the proposed method starts to show improvements to the baselines.\n- The ablations are very informative and well conducted, and covers all the important points one would want to see in the main paper.\n- Across the paper, there are mentions to GPT4 Preview (Figure 5a), GPT-3.5-turbo-0125 (Figure 3), GPT4-turbo (Figure 6a), GPT4-omni (in multiple places). Could you please introduce all the models from the GPT family you used across your experiments in a single place with an unambiguous description of their model name if available? The paper is in general very well organised, this is however one of the few points where this organisation could be improved.\n- In Figures 6a and 6b, please include in your description why you have multiple circles of the same size/colour, and provide some context on what you mean by '1', '2', '3'. For someone who read the paper and reach the figures that is very clear, but someone looking at the figure without that context fresh may not understand it. Also, is it correct to assume that each of the large blue circles (MoA, multi-proposer, with 3 MoA layers) is an MoA model with different configurations (like different LLMs as final aggregators)? If that is the case, it may make sense to add some nuance to your discussion of the results. In practice, for someone with a new problem who want to use your paper's idea, do you assume by reusing your best-performing MoA, they will also get the best results on their new problem? In other words: should you not include also something like the average over the 6 large blue circles (or perhaps over the 3 best large blue circles)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of using multiple LLMs as agents to generate answers and to aggregate these answers in a sequence of mixture-of-agents (MoA) layers is interesting.\n- Very good performance compared to many relevant baselines.\n- Experiments are solid and well conducted, and the ablation studies give important insight into the most relevant aspects of the proposed method.", "weaknesses": "- The main weakness, in my opinion, is the fact the authors did not use human evaluation at any parts of their pipeline. I understand that LLMs can perform very well (according to measures of correlation to humans) on multiple benchmarks, but at least one small-scale human evaluation could have been provided as a sanity-check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)