Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Authors: Wenzhe Li
Yong Lin
Mengzhou Xia
Chi Jin
Venue: Accepted by TMLR ()

Abstract: Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple \emph{different} Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial?  
We propose Self-MoA --- an ensemble method that aggregates outputs from only the \emph{single} top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves $6.6\%$ improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of $3.8\%$ improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.

Published in Transactions on Machine Learning Research (03/2026)
Rethinking Mixture-of-Agents Is Mixing Different Large Language Models Beneficial?
Wenzhe Li
Yong Lin
Mengzhou Xia
Chi Jin
Princcton University
Reviewed on OpenReview:
https: // openreview. net/ forum? id- K6WWKBURI
Abstract
Ensembling outputs from diverse sources is a straightforward yet effective approach boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggre- gates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA - an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Sclf-MoA outper- forms standard MoA that mixes different LLMs in large number of scenarios: Self-MoA achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of 3.8% improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Melf-MoA, we systematically investigate the trade-off between diversity and quality of out- puts under varions MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces sequential version of Self-MoA, that is capable of aggregating large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.
1 Introduction
Large language models have made remarkable strides in improving performance across different domains, with notable examples such as GPT (Achiam al.] 2023). Gemini (Team et al., (003).. and Claude (An-) thropic (202)). Significant efforts have been directed toward increasing model size and training data to boost capabilities. However, scaling at training time comes with steep costs, while scaling computation during inference remains largely underexplored.
A straightforward way to utilize test-time compute is ensembling, which aims to combine outputs of multiple LLMs [Wang et al. 2024a Lin et al. 2024 Jiang et al. 2023a Wang et al., 2024a). Among various ensembling approaches, Mixture-of-Agents (MoA) (Wang et al., 2024ab has garnered significant interest, achieving superior performance in challenging tasks such as instruction following (Wang et al., 2024a
summarization, data extraction (OpenPipe 2024 and real-world code issue resolution (Zhang al. 2024b) Specifically, MoA first queries multiple LLMs (proposers) to generate responses, and then uses an LLM (aggregator) to synthesize and summarize these responses into high-quality responsc.
Previous research highlights the significance of model diversity within the proposers for optimizing the performance of MoA, primarily focusing on strategies for ensembling a diverse set of individual models. We consider cross-model diversity as the variation among different models. However, pursuing cross-model
Equal contribution. Fmails: (wenzhe.li yong.lin, mengzhou, whwhhhhosshhhhhiaaaaaeeaeeee apiriarrinnntnnee

---

Published in Transactions on Machine Learning Research (03/2026)
diversity may inadvertently include low-quality models, resulting in quality-diversity trade-off. While previous studies mainly concentrate on achieving a high cross-model diversity (Wang et al. 2024a; Zhang et al.) 2024b), we adopt a holistic perspective on model diversity by considering in-model diversity, which arises from the variability of multiple responses generated by the same model. In-model diversity enables us to aggregate multiple outputs from an individual model. Intuitively, leveraging outpuls from the best-performing individual model can more effectively navigate the quality-diversity trade-off by creating a higher-quality proposer mixture. Thus, we propose Self-MoA as depicted in Figure [b, which utilizes the same prompting template as MoA but aggregates outputs that are repeatedly sampled from the same model, rather than from a set of different models. To distinguish, we use Mixed-MoA to refer to MoA configurations that combine different individual models when necessary.
Surprisingly, we find that Mixed MoA usually sub-optimal compared with Self-MoA, especially when there exist significant quality differences among proposers. Specifically, we revisit the same experiment setting of MoA with six open-souree instruction fine-tuned models as Wang al. (2024a). Compared with Mixed-Moo Moo which aggregales all six models, Self-MoA the strongest model achieves 6.6 point improvement over its mixed counterpart
the AlpacaEval 2.0 benchmark, showing case of when intra-model diversity is more effective. Morcover, Self-MoA on two point gain and secures the top position among non-adversarial methods on the leaderboard, which further confirms the effectiveness of Self-MoA in this task.
models on AlpacaEval 2.0 consistently achieves a 2-3
To explore the limits of model diversity for MoA, we extend our experiments to a setting with three specialized models, each excelling in a specific task. Specifically, we utilize Breen2-BB-nnsruue (Bai et al.] common sense QA (MMLU-redux (Gema et al. 2024)); ), Qwen2-Mathh-7B-nnstree mathematics (MATH (Hendrycks et al.) 2000)), ), and DeepSeek- V2-Lite-Instruct 4Zhu et al. coding (CRUX (Gu et al. 2024)). We compare Self- MoA against range of Mixed-MoA strategies, evaluating 13 combinations of individual models based on their average performance across the three tasks. Our findings indicate that employing task-specific models as proposers for Self-MoA can significantly outperform the best Mixed-MoA, Furthermore, even in a constructed mixture task tailored for Mixed-MoA where each individual model excels in specific subtask, only two Mixed-MoA strategies slightly outperform Self-Mo. by 0.17% and 0.35%.
2023 for
(Bai et al.,
2023 for 2024 for
To better undersland the effectiveness of Self-MoA, we conduct a comprehensive investigation of the trade-off between quality and diversity in MoA, involving over 200 experiments. We use the Vendi Score (Dan Fried- man & Dieng 2023) to evaluate the diversity among the outputs of the proposers, while the average perfor- mance of the proposers serves as the measure of quality. In Section 4 we confirm that MoA performance has a positive correlation with both quality and diversity. Moreover, we clearly show a trade-off along the achiev- able Pareto front of quality and diversity. Interestingly, we find that MoA is quite sensitive to variations in quality, with optimal performance typically occurring in regions characterized by high quality and relatively low diversity. This finding naturally explains the effectiveness of Self-MoA, as it utilizes the strongest model as the proposer, ensuring high quality in its outputs.
Finally, we evaluate the performance of Self-MoA under increasing computational budgets. As the number of outputs grows, the scalability of Self-MoA becomes constrained by the context length of the aggregator. To address this issuc, we introduce Sclf-MoA-Seq (Figure [+), sequential version that processes samples using a sliding window, allowing it to handle an arbitrary number of model outputs. Our findings show that Self-MoA-Seg performs at least as effectively as Self-MoA, enabling scalable ensembling for LLMs with shorter context lengths without compromising final performance.
Overall, our contributions are three-fold:
We introduce Self-MoA, which leverages in-model diversity by syuthesizing multiple outputs from the same model. Surprisingly, it demonstrates superior performance compared to existing Mixed-Mo.. approaches, which emphasize cross-model diversity, across a wide range of benchmarks.
Through systematic experiments and statistical analysis, we uncover a core trade-off between di- versity and quality among the proposers, emphasizing that MoA is highly sensitive to proposer
2

---

Published in Transactions on Machine Learning Research (03/2026)
quality. This finding also explains the success of Self-MoA, which leverages outputs from the highest- performing model, ensuring superior overall quality.
We extend Self-MoA to its sequential version Self-MoA-Soq, which iteratively aggregates small amount of outputs step by step. Self-MoA-Seq unlocks LLMs that are constrained by the context length and enables computation scaling during inference.
2 Related Work
Ensembles of LLMs. Model ensembling aims to combine strengths from multiple models. Previous studies have explored various methods to leverage a diverse set. of models, including but not limited to prompting (Wang et al. 2024a), weight averaging (Lin al., 2024 Ramé et al. 2024), routing (Jiang et al.; 2024b; Lu a 2023), training a generative fusion model (Jiang al., 2023b), and so on. Zhang et al.) (2024a) argues that the fusion of specialized models with certain general abilities could be a promising direction toward Artificial General Intelligence. Mixture-of-Agents (MoA, Wang et al. (2024a)) first queries multiple LLMs to generate responses, then iteratively aggregates these samples through several rounds of synthesis. MoA shows promising results on several benchmarks, and its variants achieve superior performance on the AlpacaEval 2.0 leaderboard. Our method is inspired by the prompt pipeline proposed in MoA. However, while cxisting MoA focuses on unlcashing the strength from multiple different. models (Wang et al., 2024a Jiang et al. 2023b Zhang al., 2024b), we demonstrate the trade-off between diversity and quality within the proposers, highlighting that focusing solely on diversity may compromise overall quality and final performance.
LLM Inference with Repeated Sampling. Previous studies have shown that combining model outputs from repeated sampling can yicld better response in various domains. In tasks with automatic verifiers available, such as math (Hendrycks el al. 2021 and code (Chen et. al., 2021), simply sampling LLMs multiple times can significantly improve the passåk metric and hence boost the success rate of solving the tasks (Roziere et al. [2023] Li et al. 2022 Brown et al. 2024).. In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency. and best-of-n to choose the most promising one from candidate responses (Wang et al. 2022 Chen et al., 2023b Gui et al., 2024 Li et al. 2024). Therefore, repeated sampling is recently regarded as one approach of scaling compute during inference time (Brown) et al. 2024). In this work, we identify the surprising effectiveness of repeated sampling in the context of MoA. Unlike majority vote best-of-N, Self-MoA asks LLMs to synthesize outputs generated from repeated sampling, hence can further improve over cach individual output.
Collaborative Agents There is surge of interest in building agent systems based on verification, critique, discussion, and refinement. For example, (Stechly et al. (2023). Valmeckam et al. (2023). and Madaan et al. (2024h use self-critique to iteratively refine outputs through a chain structure. Madaan et al. (2024) Chen et al. (2024), and WWag et al.) (2024a) explore the incorporation of multiple models to create a stronger agent that outperforms each individual model. Du et al. (2023) incorporates multiple LLMs that propose and debate their individual responses over several rounds to reach a common final answer. Liang el al. (2023) proposes Multi-Agent Debate, which encourages divergent thinking during L.M debates to arrive at more informative conclusions and avoid rushing to incorrect answers. Chen et al. (2023a) introduces RECONCILE, which adopts confidence-weighte discussions. Interestingly, Wang et al. (2024b) shows that single model with carefully designed prompts can sometimes match the performance of agent discussions. Morcover, agent discussions mainly outperform a single LLM when the prompls are insufficient.
voting mechanism for better consensus among LLM
3 Is Ensembling Different LLMs Beneficial?
As introduced in Soction 1 previous rescarch primarily emphasizes cross-model diversity, which can inadvertently inchude low-quality proposers. In this work, we introduce Self-MoA (Figure [ which uses a single top-performing model to generate multiple outputs and aggregate them to produce the final result.
3

---

Published in Transactions on Machine Learning Research (03/2026)
M2
2
M
A
%
Ma
(a) MoA
(b) Self-MoA
(c) Self-MoA-Seq
Figure 1: Comparison of MoA, Self-MoA, and Self-MoA-Soq, (a) In MoA, multiple models respond to a query, followed by aggregator synthesizing their outputs. (b) Self-MoA simplifies this by repeatedly sampling from a single model. (c) Self-MoA-Seq extends Self-MoA by applying sliding window to combine the best output so far with candidate outputs. At each timestep, the synthesized output is repeated to bias the aggregator towards it, reducing the context length roquirements and expanding the method's applicability. Note that MoA can exlend to multiple rounds of aggregation (Appendix A.1 1 while Self-MoA and Self- MoA-Seq can extend more outputs, but we omit them here for clarity.
We provide the detailed description and prompt in Appendix A.I Self-MoA leverages in-model diversity repeated sampling often produces varied outputs. We propose research question follows:
Is the benefit of MoA from cross-model diversily? Can we build stronger MoA using in-model diversity?
3.1 Experiments on AlpacaEval 2.0 with General Purpose Models
Evaluation benchmarks. We adopt the same experiment setting as Wang et al. (2024)) AlpacaEval 2.0 benchmark (Dubois al., (024) and compare the performance of Mixed-MoA and Self-Mo. AlpacaEval 2.0 is widely used benchmark for assessing the instruction-folloving abilities of LLMs. It offers set of world instructions and employs CPPT-4-bsed annotator to compare the model's responses against reference answers generated by GPT-4. To address length bias inherent in model-based evaluation, Dubois ct al. 2024) introduced the length-controlled (LC) win rate as a more robust evaluation metric.
Models. Following Wang et al. 2024a), we construct MoA based on six individual models: Qwen1.5- 10BtChat (Bai et al.) 2023), 1.5e2..5-7BB-B 1---a2h (Bai al., 2023), WizardIM-8x222B (Xu et al., 2023). Touvrro al. 2003),, Mixxtral-xx222--Insttutt--- instruct (Team et al.] 2024b/. Each model is sampled with temperature of 0.7, following the default in (Wang et al. 20244). ). For Self-MoA, we aggregate six outputs sampled from W- as it
(Jiang et al. 2024a), and dbrx-
consistently outperforms the other models. In line with Wang et al. (2024a), we use @iven155-110B. Chat as the aggregator for both Mixed-MoA and Self-MoA
Results. We present the LC win rate for each model configuration in Table [ For individual models, we report the higher value between the leaderboard results and our reproduction. Notably, Self-MoA demon- strates remarkable effectiveness in this task, outperforming the Mixed-MoA baseline by 6.6 point. This suggests that, while using multiple models intuitively offers greater diversity, ensembling multiple outputs from a single model is more effective.
We note that this experiment is similar to the "singlo-proposer" setting in Wang et (2024a) however reproduced result different. We conjecture that such major difference is due different choices of the proposer model, which is not mentioned in Wang et al. 20044). As we shall later Section ensembling performance is more sensitive quality rather than diversity. Therefore, worse proposer model will lead suboptimal performance of SelEMMA.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Comparison of Self-MoA and Mixed-MoA on AlpacaEval 2.0 leaderboard. We use Liven1.5-110E Chat the aggregator.
Model Configuration
LC Win Rate
W-
53.1
Qwen1 5-1100-Chhtt
43.9
34.4
Individual
Swrnnl.5-72--hhr Chat
36.6
Saarral-8222--Tnssuutttv- dbrx-instruct
30.2
25.4
Mixed-MoA
Layee MoA (Wang et al. 2024a)
59.1
Self-MoA
Layer Self-MoA WizardLM
65.7
Table 2: Self-MoA achieves state-of-the-art performance on the AlpacaEval 2.0 leaderboard when using top- performing models as both proposers and aggrogators. We only ensemble 4 outputs due to context window constraints.
Model Configuration
LC Win Rate
gemmaa--99-ii--POO-HH
76.7
Individual
gemmm-2-9b-it-- 2-9b-it-SmmPO
72.4
Self-MoA + Den-na---bbit--P---1
78.5
Self-MoA
Self-MoA
gemmme-2-9--ii-SinnO
75.0
Applying Self-MoA on top performing models. To further validate the effectiveness of apply it to the two top-performing models on AlpacaEval 2.0: 1it.mm----b-tt-PO---- (Zhou al. 2024 and 2-0-i-----b----SimPP (Meng el al. 2024). We use each model as both the proposer and the aggrega with a temperature of 0.7 for all the generations. Due to the context length constraint of Gemma 2 TTam et al.] 2024a), the aggregator can only take four samples as the input. As shown in Table 2 Self-MoA consistently achieves a 2-3 point gain and secures the top position on the leaderboard during submission.
we
Results MT-Bench. Beyond AlpacaEval 2.0, we further evaluate Self-MoA and Mixed-MoA on MT- Bench Zheng et al. 2023), another benchmark used in Wang et all (024a)). The results align with findings from AlpacaEval 2.0, reinforcing the effectiveness of Solf-MoA. Please refer to Appendix B.1 for more details.
3.2 Experiments on Multiple Datasets with Specialized Models
In this section, compare different ensembling methods diverse benchmarks using specialized models.
Evaluation datasets. We conduct evaluations across a diverse set of benchmarks:
MMLU (endrrycks et al, 2020 is a multiple-choice dataset designed to assess a model's multitask accuracy. MMLU is widely used to evaluate both the breadth and depth of language understanding capabilities of current LLMs across a diverse array of subjects, including mathematics, history, computer science, logic, and law. We adopt MMLU-redux (Gema et al., 2022)) for evaluation, which is subset of MMLU with 3,000 samples fixing the errors in the dataset through human re-annotating.
2Qwem1. 1.5- 1-00--Chhtt used the aggregator since the two top models significantly outperform

---

Published in Transactions on Machine Learning Research (03/2026)
Table Comparison of Self-MoA and Mixed-MoA in MMLU, CRUX, and MATH. The labels 1, m, and d refer to Qwen2-7B-Instruct, DeepSonk-Coder-2212. V2-----itt--nssrrec, and Swon2-Math--7B-hstreee respectively. The average performance represents the mean accuracy across MMLU, CRUX, and MATH. TaskBest indicates that we use the strongest model for each task as both proposer and aggregator.
Aggregator
Proposer
MMLU
CRUX
MATH
i 66.16
36.25
51.91
Individual
d 60.91
49.51
57.26
54.36
27.88
69.57
1immdd
67.89
42.88
64.38
imdddd
67.42
44.50
63.90
imd
68.90
41.25
63.00
immmmd
66.63
42.75
66.02
1immmm
66.23
39.25
66.10
i immm
67.49
38.25
64.16
Mixed-Mil
i
ii imm
68.00
37.00
62.92
iidddd
68.21
45.50
62.56
iiiddd
68.21
42.88
62.38
iiiidd
68.47
40.75
mmdddd
66.34
46.75
66.48
mmmddd
65.80
47.00
67.32
mmmmdd
65.44
42.50
67.62
TaskBest 69.01
50.75
68.42
Self-MoA
TaskBest
6 TaskBest
69.01
52.62
69.80
CRUX Gu et al. 2024 consists of 800 Python code functions, each containing 3 to 13 lines along with am input-output pair. Based on this dataset, [uu et al. (2024) constructs two tasks: input prediction and output prediction. To successfully complete these tasks, the LLM must demonstrate code reasoning abilitics.
MATH (Hendrycks et al. 2021 comprises 12,500 challenging competition-level mathematics prob- lems. For our analysis, we utilize the testing subset of MATH, which consists of 5,000 samples.
Models. To ensure sufficient diversity, we select three LLMs with specialized strengths: Qwen2-7B. Instruct (Yang et al.) 2024), Segeeeel-Coodr--2-Lieeeeee Instruct. We fix the number of proposers to six and sweep various combinations of these three models. For convenience, we denote 20ve2-77B-Instrruee Bath-7B-Instrued as As shown in Table 3 Swen2-7BB-nstrruee Bron2--Math7BB-Intteee excel on MMLU, CRUX, and MATH, respectively. We use the short name for the mixture proposers. For example, iiddmm indicates the inclusion of two samples from each model respec- tively. When model is represented multiple times the proposer mixture, we ensure that two samples are generated with different random seeds. We the temperature of each model to be 0.7 for the individual model, and use temperature 0 for the aggregator. We mainly use Qwwn2-7B-Innstruee but also try different models as the aggregator. We explore various MoA configurations, including indi- vidual models, combinations of two or three models as proposers, and using single top-performing model (est, for example
(Zhu et al.) 2024), and Qwen2-Math-7BB.
as i, DeepSeek-Cooler-2220
as d, and Qwen2- and
DeepSeek-Codee Coder-
the aggregator
for CRUX) as the proposer (Self-MoA).
Results. The results are presented in Table g When using as the aggregator, Self-MoA with the Best model consistently outperforms 13 tested Mixed-MoA configurations across all lasks. Further-
3 As Qwen2-Mathe Math-77--nstrrun only supports context length of 4096, for these data points, sample the proposer with reduced token length 1024, and only aggregates three outputs from proposer.

---

Published in Transactions on Machine Learning Research (03/2026)
MNLU
CRUX
28
Vixed Mo4 Se MaA
0
Nissd-Mok Seli
SafANGA
MieadMoA
2.6
66
2.4
1.0
65
2.2
0
42
2.0
166
18
38
1.4
14.
3.2
40
10
35
65
20
30 35
40
25
40
45
30
15 Quality
60
65
75
Quality
Figure 2: The qiversity-quality trade-off: Mixed-MoA incorporates different individual models as proposers, while Self-MoA uses the same individual model for this role. Quality is assessed based the average performance of each proposer, and diversity is computed with the Vendi Score (Dan Friedman & Dieng, 2023 of outputs generaled by proposers on the samne prompls.
more, adopting a task-specific aggregator yields an additional performance boost of 1-2 points. Interestingly, increasing model diversity does not always lead to better performance. For instance, while MoA with iimmdd surpasses mmmddd on MMLU, it underperforms on CRUX and MATH. This discrepancy aligns with the rel- ative strengths of the individual models excels on MMLU but lags behind on CRUX and MATH. We postpone more discussion to Section 4.2
4 The Quality-Dvversity Trade-off
We investigate factors that contribute to the strong performance of Self-MoA through careful experiments. Previous studics have mainly focused on incrcasing model diversity within the group (Wang et al. 2024a Jiang et al. 2023a Zhang et al. 2024b/. However, searching for diverse models can sometimes lead to including poorly performed models, resulting in a trade-off between diversity and quality, where quality refers to how well each individual model performs in the group.
Therefore, we aim identify the existence of general relationship between MoA's 's performance and quality well as diversity. Following Section B evaluate MoA's 's performance on MMLU, CRUX, and MATH, which cover tasks requiring wide range of capabilities. We vary the quality and diversity with two orders of freedom: 1) combinations of individual models in proposers from Section 3.2 and 2) sampling lemperature. i.., 0.5, 0.7, 1.0, 1.1, and 1.2. This results in a total of over 70 unique MoA proposer mixtures. We measure the quality and diversity as follows:
Diversity: We utilize the Vendi Score (Dan Friedman & Dieng 2023 to assess the diversity among individual models the proposer mixture. The Vendi Score represents the effective number of unique elements within a collection of samples (Dan Friedman & Dieng 2023 with further details provided in the Appendix A.2 Specifically, for a given prompt æ, we obtain responses from each model, denoted as V1, ........66 Mo. The diversity of the proposers for prompt æ, denoted as d(z), is calculated using the Vendi Score the sct [У1 [yı,..., ......,,, yo]. We then compute the overall diversity across the dataset S as:
d
Σ d(z).
S
DES
Quality: We first determine the accuracy of each model on the datasct S, yielding values .., 46. The average accuracy, g = { 42 +... + qo), serves as our measure of the quality of the proposers. We will explore additional quality measurement strategies in later sections.
Results.
We plot MoA's performance with corresponding diversity and quality for each mixture of pro- posers in Figure 2 We summarize key observations follows:
7

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Linear regression (Equation of MoA's performance on diversity d and quality q.
Dataset
Cx
B
R2
Coefficient
P-value
Coefficient
P-value
MMLLU
2.558 + 0.176
0.001
1.841 + 0.176
0.001
0.771
CRUX
4.548 + 0.459
< 0.001
1.421 + 0.459
< 0.001
0.685
MATH
4.719 = 0.416
0.001
2.839 = 0.416
0.001
0.760
The trends among MMLU, CRUX, and MATH are consistently aligned.
When the quality is fixed, increasing diversity can enhance MoA's performance.
When the diversity fixed, improving quality can also boost MoA's 's performance.
There exists a trade-off in the achievable Pareto front between diversity and quality.
Notably, the best performance of MoA typically observed in the bottom right of each subplot, indicating a strong sensitivity to quality.
Previous work on ensembles |Wang et al. 2024a: Jiang al. 2023a: Zhang al. 2024b) primarily focuses increasing the diversity of models within the proposer mixture. However, as shown in Figure 2 compared to Self-MoA on the best-performing model, simply aiming for greater diversity in the proposer mixture can result in lower overall quality, which may negatively impact MoA's 's performance. This tradc-off between diversity and quality helps to explain why Self-MoA achieves superior performance across various benchmarks.
4.1 Statistical Analysis
To further understand the numerical correlation between MoA's performance and diversity as well as quality, we conduct linear regression for MoA's 's performance & on diversity d and quality Specifically, we fit the following equation for each dataset:
1 1= x=xxq+3xd++
(1)
where a, 3, E are real-valued coofficients to be determined. For cach datasct, we collect around 70
data points from Figure 2 to construct the set {q', d', J. The coefficients a, B, and 7 are then derived To make coefficients and 3 comparable, and d by subtracting their means and dividing by their standard deviations (detailed in Appendix A.3) respectively. The results are presented in Table E We observe that the p-values for both a and / are less than 0.001, indicating a significant correlation between MoA's 's performance and both quality and diversity (Arnold] 1990). The R² values from the linear regression across three datasets are approximately around 0.7, indicating that the lincar model based on quality and diversity explains 70% MoA's performance and hence strong correlation between inputs and outputs, according Appendix A.4 In later parts, we show that using a more fino-grained quality calculation can further increase the R value.
by solving linear regression on {g', d', J, fi=1
normalize q
Comparing the effect strength of quality and diversity. From Table 4 we observe that a is greater than / across all three datasets. In particular, for CRUX and MATH, the gap bet these two measures is even more pronounced. These results suggest that MoA's performance is particularly sensitive to variations in quality, highlighting the importance of prioritizing quality within the proposer mixture. This finding is also consistent with our observation that MoA achieves its best performance the bottom right of the plot in Figure further supporting the effectiveness of our proposed Self-MoA approach.

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: The R2 of the linear regression when we use different quality measurement methods. We find using Centered-1/K-Nel K-rm with K-2 can achieve good performance among all these three datasets.
Dataset
Method
Avg. (K=1)
K-2
K=3
K=4
K-Norm
0.771
0.809
0.832
0.845
MMLU
Centered-1/K-Norm
0.771
0.881
0.902
0.903
K-Norm
0.685
0.736
0.765
0.779
CRUX
Centered-1 -aatrreddd/K-ornn
0.685
0.753
0.758
0.753
K-Norm
0.760
0.720
0.692
0.672
MATH
Hentered-1/K-Norr
0.760
0.720
0.692
0.672
Alternative quality measurements. We use the averaged accuracy of each individual model to measure quality in the previous analysis. In this section, we explore alternative methods for assessing the quality of proposers. Recall that QI...., 96 denote the accuracy of cach individual model among proposcrs, and without loss of generality, we assume qı 2 Q2 2 ... 2 qe- It is reasonable to assume that the aggregator can select the correct answer from the proposers, particularly when the responses of individual models are inconsistent. In such cases, the aggregator would rely more heavily on models with better individual performance, mcaning the weight of would be greater than that of 96-
Therefore, we compare the following methods to calculate quality:
Average:
Ei= d6-
K-Norm: El xth) 1/K
where a larger K places more emphasis on stronger individual models.
(n K In this formulation, we first compute the difference between qi and the best model's q1- The 1/K norm emphasizes the weights of models whose performance closer to Q1-
Centered-1/K--ormm:
Φ -
All three methods are the same when K K- 1. For each quality measurement, we fit linear regression to assess the relationship between MoA's 's performance and the quality and diversity metrics, reporting the R² values in Table 5] Our analysis shows that in MMLU and CRUX, applying larger weight to better- performing individual models tends to increase the R² values. However, this trend is inconsistent for MATH. We conjecture that this inconsistency arises because the aggregator Qwen2-7BBTnstruee is relatively weak on MATH compared the strongest individual model, Qwen2-Math-7I1 -a-t-att------sstrre.. This limitation constrains the performance of MoA, leading to an inconsistent trend in the linear regression results. In contrast, on MMLU, where Qwent-7B-Instruuet is the strongest individual model, we find that the R² value can exceed 0.9 with K = 2 using the Centered-1/Ke /KK-Nomm.. This indicates very strong linear relationship between MoA performance and the quality and diversity metrics. Overall, we conclude that employing Centered-1/K-Noon with K = 2 (marked in blue) achioves strong performance across all three datasets.
Instruct.
4.2 When Mixed- Moo Outperforms Self-MoA?
According to the quality-diversity trade-off illustrated in Figure 2 we conjecture that increasing diversity can enhance MoA's 's performance when the quality is controlled.
Mixed-MoA generally exhibits greater diversity than Self-MoA, which can lead to improved performance when the model quality similar. This advantage arises when individual models achieve similar overall performance while maintaining significant cross-model diversity. To simulate such a scenario, we construct a mixture task combining MMLU, CRUX, and MATH as described in Section 3.2 In this setting, test samples are drawn uniformly from the three tasks, and models do not have prior knowledge of sample's origin.
9

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Comparison of Self-MoA and Mixed-MoA on the mixture task of MMLU, CRUX, and MATH,
measured by the average performance of three tasks from Table 3 Mixed-MoA models with top two average performances are highlighted by underline.
A Aggregator
Proposer
Average
i
51.44
Individual
d
55.89
m
50.60
i 1immdd
58.80 $8.38
imdddd
58.61
iiiimd
57.72
immmmd
58. 58.47
iimmmm
57.19
1iimmm
56.63
Mixed-MoA
i
i iiiim 1iiimm
55.97
1 1idddd
58.76
iiiddd
57.82
iiiidd
56.82
mmdddd
59.86
mmmddd
60.04
mmmmdd
58.52
i
dddddd
59.69
Self-MoA
i
TaskBest
62.73
TaskBest
6x TaskBest
63.81
For given MoA strategy, we evaluate its performance on this mixture task by averaging its performance across the three datasets. The results are reported in Table In this mixture task, each model specializes different subtasks, with performing best on MMLU, d on CRUX, and m on MATH. As TaskBest requires additional prior knowledge of the sample origin, we also report Self-MoA with as the proposer, given that it achieves the highest average performance among individual models.
observe that Mixed-MoA indeed outperforms Melf of dddddd. Specifically, Mixed- MoA of mmdddd and mmmddd achieves the average performance of 59.86% and 60.04%, improves upon Self- MoA of dddddd by 0.17% and 0.35%. Given the reported small margin, we argue that Self-MoA is still a very competitive bascline under this setting, not to mention the dominant performance of Sclf-MoA over Mixed-MoA when focusing on one single task (Self-MoA with TaskBest models achieve an average of 3.8% improvement from Table 6). In Appendix B.4 also report normalized results that account for different variances among tasks, which leads to similar conclusion.
From Table
We further consider another single-task case MMLU, involving two individual models: Llama-3. 1-BBB- Instruct and Qwen2-7B- Twwen2-BBInssrruet, with Qwen2-7B- -eent--B-Instruct serving as the aggregator. We choose Llama-3.1 8B-Instruct because it performs similarly to (wen2--B-Insttaee as an individual model. Table 7 demon- strates that even when the performance two individual models is close, Self-MoA MoA-utilizing six Llama- 3.1-8R-Instruct proposers (denoted as 111111) still outperfomms the Mixed-MoA configuration (denoted iiilll).
5 Scaling Inference Compute with Self-MoA
In previous soctions, we have provided evidence that SeIf-MoA over one strong model straightforward but effective. As the community becoming more aware of scaling inference time computing (Brown et al. 2024 Snell et al. 2024 Wu et al. 2024), one natural question to ask is:
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: MoA of Llama-3. 1----.1-------tteee and Brren2-7B-Instruee short for Llama-3. Instruct and i is short for Saven-7B-Inntrrun
Aggregator
Proposer
MMLU
66.16
Individual
1
66.40
Mixed-MoA
111111
70.73
iiiiii 111111
69.01
Sel--MoA
71.27
MMLU
CRUX
141
Accuracy
0.4
0055
MoA
Base TB-nstruct)
Modal (DeepSock. Vodar
878
Number of Samples
Number of Samples
Figure 3: The performance of Self-MoA and Self-MoA-Seg with a growing number of samples. Dashed lines indicate the performance of single forward pass with the base model.
Given strong model, does Self-MoA 's performance scale with the number of repeated samples?
Intuitively, Self-MoA cannot scale indefinitely by simply increasing the computation budget. for at least three reasons:
As more responses are sampled from a single model, the diversity among those samples tends to plateau.
Aggregating information from many samples is more challenging for LLMs compared to handling smaller number of samples.
Every LLM has context length limit (e.g., 8192 tokens for Gemma 2), which restricts the number of responses aggregator can process
While the first limitation is inherent to repeated sampling, we address the latter two by introducing Self- MoA-Soq, a sequential variant designed to manage large numbers of responses without overwhelming the aggregator. Self-MoA-Seq uses sliding window aggregate fixed number of responses time, allowing it to handle unlimited number of responses, regardless of context length constraints. A visual illustration is provided in Figure 1
We evaluate the performance of Self-MoA and Self-MoA-Seq with increasing sample sizes on the MMLU and CRUX benchmarks to study their scaling behavior. For each benchmark, we use the best- best- test-performing model as both the proposer and aggregator (Qwen2-7B-Instruct In--rrtt for MMLU and DeepSock- Coder-V2-Lieeee V2-Litt-Inntruct for CRUX), with a sampling temperature of 0.7. Self-MoA-Seq, the window size set six, with the first three slots reserved for the current synthesized output. We vary the number of samples from to 30
11

---

Published in Transactions on Machine Learning Research (03/2026)
and plot the accuracy curves from three runs with different seeds in Figure 3 Our key observations are follows:
Both Self-MoA and Self-MoA-Seg significantly improve performance over the individual base model.
Adding more samples can have both positive and negative effects, meaning there is no universal compute-optimal solution.
MeA-ee delivers performance that is comparable to, slightly better than, Melf-MoA.
These findings suggest that Self-MoA-Seq can extend the effectiveness of Self-MoA LLMs with shorter context lengths, without sacrificing performance. Following Section 4.2 we explore whether introducing a second model can enhance performance in the sequential setting. Given that Llama-3.1-8B-Inst Instruct performs similarly to 2wwen2-7B-Instrucee on the MMLU lask, we compare the impact of adding and DeepSeek-Codee Coder-V2 Lnstuee- (which underperforms Syee2-7BBInstruce by 5%) after aggregating 30 samples from Qwwe2-7BB-Insrruc in Self-MoA-Seq: We find that incorporating JJama-3.1-8B-Inttruu boosts accuracy by around 2%, whereas adding DeepSeek PrrpSee--Coder--- Coder-V2-Lite V2-Lite-Instruct reduces accuracy by more than 1.5%. This result provides another example of cross-model diversity benefiting MoA, and shows the potential of Self-MoA-Seq with increasing computation budget.
6 Conclusion and Limitations
In this paper, we introduce Self-MoA, an innovative approach that utilizes in-model diversity to enhance the performance of large language models during inference. Our experiments demonstrate that Self-MoA outperforms traditional Mixed-MoA strategics in many popular benchmarks, particularly when the proposer model quality varies. By aggregating outputs from single high-performing model, Self-MoA effectively addresses the quality-diversity trade-off. We further identify the scenarios where mixing LLMs can be potentially beneficial and extend Self-MoA to the constrained context length setting.
While these findings highlight the potential of in-model diversity in oplimizing LLM performance, our current work intentionally focuses on temperature sampling and random seeds as the primary sources of diversity. This design choice requires minimal intervention and allows to isolate the effects of in-model versus cross-model diversity under tightly controlled settings; thus, our current results should be viewed conservative baseline for Self-MoA. Future research could explore additional mechanisms such as prompt- template diversity or structured Chain-of-Thoughh (CoT) branching to further increase in-model diversity, paving the way for even greater advancements in ensemble methods.
Acknowledgments
We thank the reviewers for their valuable feedback. CJ acknowledges the support from XSF-OAC-2411225 NS2-117--239297, and Princcton AI Lab Soed Fund.
12

---

Published in Transactions on Machine Learning Research (03/2026)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ngee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arv preprint arXit:2303.08774, 2023.
Al Authropic. Introducing claude, 2023.
Harvey J. Arnold. Introduction to the practice of statistics. Technometrics, 32:347 348, 1990. URL https:
//api . wwwpattiiiiccolaaoooooorroppiiee 1222991555
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint urXi::2009. 16609, 2023.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. orXiv preprint arXiv:2407.21787, 2024.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llss. arXiv preprint arXim:2309.11 1...111000 13007, 2023a.
Mark Chen, Jerry Tworck, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (xxi:2207.03374, 2021.
Sandy Chen, Leqi Zeng, Abhinav Raghunathan, Flora Huang, and Terrence C Kim. Moa all you need: Building I1m rescarch team using mixture of agents. arXiv preprint Xii:24409004487 2024.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint 2eXi:2211111311, 2023b.
Dan Dan Friedman and Adji Bousso Dieng, The vendi score: A diversity evaluation metric for machine learning. Trunsactions machine learning research, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXin:2305.14322 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXio preprint arXiv:24040444 2024.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, al. Are we done with mmlu? arXiv preprint Xrxiv:2406.0 04127, 2024.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. anXiv preprint arXiv:2401.0000 2024.
Lin Gui, Cristina Gárbacca, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling, arXie preprinl orXiv:2406.0083 00832, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXie preprint arXiv:2009.0330 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. XrXiv preprint 0xc:2241000488, 2024a.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024b. URL https:
7/axxiv. /aaxiv.org/aie abaaabss2245 04088
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arx preprint 1ix::2306.02561 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion, 2023b. URL httpss//arxiv.. wwwpxxii.rrggbb///oo http//bb/2330..0222
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need, 2024. URL https://arzin http://axxi..og/ab/a///aaaa wwrg/bba/442.00511 2402.05120
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092 1097, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint Xiv:22305 19118, 2023.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjic Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. Mitigating the alignment tax of rlhf, 2024. URL pettps//aarxin org/abs/ httg/abs/2309 06256
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models, 2023. URL htttss///aaxt:
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Newral Information Processing Systems, 36, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14784, 2024.
OpenPipe. Openpipe mixture of agents: Outperform gpt-4 at 1/25th the cost, 2024. URL https://
openpipe. openpppe..i/blog/mmee.. wttpo///miiuuuro---aaaeee
Alexandre Ramé, Johan Forret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Codoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies, 2024. URL httt::://axxiv.orr/ab/////////
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint Xiv:22300..1 12950, 2023.
Ketut Sarjana, Laila Hayati, and Wahidaturrahmi Wahidaturrahmi. Mathematical modelling and verbal abilities: How they determine students' ability to solve mathematical word problems? Bela: Jurnal Tadris Matematika, 1372):117-129, 2020.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL wtt.aa/wwww..rr///////...../ 03314
14

---

Published in Transactions on Machine Learning Research (03/2026)
Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprinl arXiv:2310. 0...22300....7,, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrae, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1105, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giusoppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, An- ton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Tean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Ali- cia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska,, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klinezak-PPluiifske Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gor- don, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikoya Badola, Kat Black, Katic Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kil- patrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastion M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jor- dan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovie Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukeuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgcaud, Noah Fiedel, Armand Joulin, Kathleen Ko- nealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024a. URL mnttps://arx:: org/ wwg/abs/////eeeoeee/e..eeeeeeeae 22t.//00000001111
Mosaic Research Team et al. Introducing dbrx: A new state-of-the-art open 1lm, 2024. URL https://www/ databricks. com/blog/ ht///blogg///rrre /tttt/ooduiiggdbr----aa webrr-nwwwstate-rre crre--rrrr-pp--lm. Accessed on April, 26, 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finc-tuned chat models. arxie preprinl urXiv22377.0928. 2023.
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint 0rXi::2310.08118, 2023.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mfxture-of-agents enhances large language model capabilities. arXiv preprint ar Xrx:222006 04692, 2024a.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of Ilm reasoning: Are multi-agent discussions the key? aixiv preprint or Xrxi:22402 18272, 2024b.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consitenccy improves chain of thought reasoning in language models. ar arXiv preprinl 0rXiv:2208.11111, 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute- optimal inference for problem-solving with language models, 2024. URL https://arxiy wrg..ss//////...to.oo..........a aaaa//405 00724
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowwring large language models to follow complex instructions. ar preprinl arXiv:2304.1280 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: //arxiv. www.aba.//aaeo.ooo.gooooeee 2407./244....0 10671
Kaiyan Zhang, Biqing Qi, and Bowen Zhou. Towards building specialized generalist ai with system 1 and system 2 fusion. arXiv preprint arXiv:2407.08644 2024a.
Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. Xiv preprint or orXiv:240.00000 07060, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.116 118277 2024.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-codent voder- Breaking the barrier closed-source models code intelligence. arXiv preprint arxi::2406 11931, 2024.
16

---

Published in Transactions on Machine Learning Research (03/2026)
A Supplements
A.1 Preliminary on Mixture-of-Agents
Eixture-of-Agente (MoA) (Wang et al. 20244a is an ensemble method that leverages multiple large language models (LLMs) to enhance performance on various tasks. In MoA, the models are categorized into two roles: proposers and aggregators. Proposers are responsible for generating diverse responses to a given query, while the aggregator synthesizes these responses into a single response which is expected to be of higher quality, and this refinement. process can be applied iteratively across multiple layers.
Speicifcally, we copy the prompt design from Wang et al. (2024a) for aggregators in Table S which is used throughout this paper. The prompt for the proposer is simply the original user prompt without any modification.
MoA can be extended to multiple layers. For MoA with / layers and n LLMs {A each layer i, we can formulate it as follows:
① j=1
[Aig(aa)] + +1,, #i+1 Vi.
y:
where each LLM A generates a response for the query wi, which is further concatenated with the original query by the aggregator's prompt +
Table compares the performance of 3-Layer Mixed-MoA and 2-Layer Self-MoA well as the total number of forward passes required for each method. Specifically, one forward pass is counted each time a proposer model generates an output or an aggregator synthesizes result. Notably, Self-MoA outperforms the 3-Layer Mixed-MoA baseline with only half the forward passes.
Table 8: The prompt design for the aggregator MoA (Wang et al. 2024a
You have been provided with set of responses from various open-source models to the latest user query. Your task is to synthesize these responses into a single, high-quality response. It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response well-structured, coherent, and adheres to the highest standards of accuracy and reliability.
Responses from models:
1. Model Response from A,
2. Model Response from Ar
n.
Model Response from Ar.).
Table 9: Results of 3-Layer Mixed-MoA.
Model Configuration
LC Win Rate
# Forward Passes
Mixed-MoA
L-Layer MoA
Waan et al. 2024a
65.4
13
Self-MoA
2-Layer Self-MoA
M-2222LM-2-xx228
65.7
7
A.2 Vendi Score
The Vendi Score (VS) is a metric designed to evaluate diversity machine learning. It takes as input a collection of samples along with pairwise similarity function, and it outputs a single value that represents the effective number of unique elements within the sample set..
17

---

Published in Transactions on Machine Learning Research (03/2026)
The score is computed using positive semi-definite similarity matrix K E R as follows:
K VS(K) exp (-tr log
(-tr"
))) exp -
Σ A log(As) 1=1
Here, A; are the eigenvalues of the normalized matrix n and Olog a which reflects the Shannon entropy of its cigenvaluos, also referred to as the effective rank. This metric provides a quantitative measure of diversity based on the distribution of similarity scores among the samples.
Essentially, the Vendi Score is
the exponential of the von Neumann entropy of K
A.3 Normalization of Inputs
Given a sequence of inputs 21, In. Let x' denote the normalized We have
n
n 1
where F
n uis and std(x) =
Σ
Σ [z 7)2 i=1
std(x))
n
A.4 Implication of R-squre
The implications of Rº are presented in Table 10 and dependent variables. (Sarjana et al. 2020
illustrating the degree of influence between the independent
Table 10: The interpretation of R-square
R-square
Level
[0, 0.2)
Very weak
[0.2, 0.4)
Weak
[0.4, 0.6)
Modian
[0.6, 0.8)
Strong
(0.8, 1.0]
Very Strong
B Additional Results
B.1 MT-Bench Results
We also compare MoA and Self-MoA on the MT-Bench (Zheng al. 2023 benchmark under the same experiment setting as Wang et al. (2024a). We copy the numbers from Wang al. (2024a) for 3-Layer MoA settings, and report our implemented results for the other experiments ensure that 2-Layer experiments are fair comparisons. Table I shows that Self-MoA outperforms its Mixed-MoA counterpart, and using GPT-4o as the aggregator can achieve the best performance even with fewer forward passes compared to 3-Layer MoA with GPT-4o.
B.2 Mixing Open-Source Source and
Models
To further investigate the quality-diversity trade-off when mixing open-source and closed-source models, we conduct an additional experiment using 20een--4B-Insstuut2225 (q) and GPT-5-mini (g). We benchmark both Mixed-MoA and Self-MoA on AIME 2024, a challenging and unsaturated math evaluation set.
Using GPT-5-mini the aggregator for all settings, we compared three proposer configurations: six open- source models (bbbbbb); a mixture (qq9ggg), and six closed-source models (gggggg).
18

---

Published in Transactions on Machine Learning Research (03/2026)
Table 11: Comparison of Self-MoA and Mixed-Moo on MT-Bench. We use Qwen1.5- 11--1-----hiit and GPT-4o the aggregator.
Model Configuration
Avg.
1st turn
2nd turn
# Forward Passes
M-2-x21
8.99
9.05
8.93
Qwen1.5- 1--1-10
8.61
8.77
8.45
LLaMA-3-70B-Instruuc
8.84
9.14
8.54
Individual
Qwen1.5-72B-Chr
8.62
8.66
8.58
Mistral-&kx22B-sstruuuhhh
8.49
8.89
8.09
dbrx-instruct
7.82
8.21
7.43
2-Layer MoA
9.06
9.23
8.89
2-Layer MoA w/ GPT-4o
9.39
9.40
9.37
Mixed-MoA
3 Layer MoA
9.25
9.44
9.07
13
3 L-yyr MoA w/ GPT-40
9.40
9.49
9.31
13
Self-MoA +
2- Layer Self-MoA
9.13
9.36
8.89
2-xrrdLM-2-8x221
2-Layer Self-MoA w/ GPT-4o
9.52
9.56
9.47
As shown in Table 12 the results align with the trends reported in our main experiments: Self-MoA with the top-performing model (gggggg) achieves the highest performance, surpassing the Mixed-MoA configuration (qq9ggg).
Table 12: Comparison of Self-MoA and Mixed-MoA on AIME 2024 mixing open-source and closed-source models. q is short for Qwve33-4B-Instruct-2555 and g is short for GPT-5-mini, GPT-5-mini used as the aggregator for all MoA configurations.
A Aggregator
Proposer
AIME 2024
52.78% + 4.87%
Individual
76.67% + 6.94%
Mixed-MoA
qq9ggg
86.67%
2.36%
85.00%
1.67%
Self-MoA
gggegg
89.17% + 1.44%
B.3 Comparison to Universal Self-Consistency
We conduct further experiments to compare Self-Consistency WWag et al. 2022 with MoA and Self-MoA on the AlpacaEval 2.0 benchmark. As this benchmark is an instruction-followigg task without exact answers, we evaluate on Universal Self-Consistency (USC) (Chen et al. 2023b) which prompts LLMs to generate the most consistent response. We report the result Table 13 which shows that USC performs worse than MoA counterpart when proposers and aggregators are controlled. This further suggests that rather than finding the most consistent response, MoA and Self-MoA can encourage LLM synthesize the references and produce better response.
B.4 Normalizing Sub-tasks in Table
The results in Table Jindicate that the variance of models on CRUX is generally higher than that the other two tasks, which could bias the average performance towards CRUX. To ensure that each task contributes equally the overall performance metric, we assign weights to the three tasks based on the inverse of their variance.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Table 13: Comparison of Self-MoA, Mixed-MoA, and Universal Self-Consisteney (USC) on AlpacaFval 2.0
lcaderboard. We Qwen1.5-110B-Char the aggregator.
Model Configuration
LC Win Rate
# Forward Passes
Mixed-MoA
MoA
59.1
7
Self-MoA
Self-MoA + WizardI.M-2-8x22BB
65.7
Mixed-USS
53.8
7
Universal Celf-Consistency
Self-USC + W
60.2
Table 14: This table compares Self-MoA and Mixed-MoA using weighted composition of three sub-tasks. The weights are assigned to each sub-task to prevent high-variance task, such as CRUX, from dispropor- tionately influencing the overall performance metrics. This approach ensures a more balanced evaluation, allowing for a fairer comparison between the two models.
Aggregator
Proposer
MMLU
CRUX
MATH
Average
WeightedAvg
Individual
66.16
36.25
51.91
51.44
53.81
Individual
d 60.91
49.51
57.26
55.89
56.80
Individual
54.36
27.88
69.57
50.60
52.80
Mixed-MoA
i
i iimmdd
67.89
42.88
64.38
58.38
60.40
Mixed-MoA
i
imdddd
67.
44.50
63.90
58.61
60.46
Mixed-MoA
i
111imd
68.90
41.25
63.00
57.72
59.94
Mixed-MoA
i
immmmd
66.63
42.75
66.02
58.47
60.40
Mixed-MoA
i
i 1immmm
66.23
39.25
66.10
57.19
59.38
Mixed-MoA
i
i1immm
67.49
38.25
64.16
56.63
59.00
Mixed-MoA
i
11iimm
68.00
37.00
62.92
55.97
58.47
Mixed-MoA
i
iidddd
68.21
45.50
62.56
58.76
60.58
Mixed-MoA
i
iiiddd
68.21
42.88
62.38
57.82
59.86
Mixed-MoA
i
iiiidd
68.47
40.75
61.24
56.82
59.05
Mixed-MoA
i
mmdddd
66.34
46.75
66.48
59.86
61.45
Mixed-MoA
i
mmmddd
65.80
47.00
67.32
60.04
61.57
Mixed-MoA
i
mmmmdd
65.44
42.50
67.62
58.52
60.39
Self-MoA
i
dddddd
65.23
50.75
63.08
59.69
60.86
Self-MoA
i
6x TaskBest
69.01
50.75
68.42
62.73
64.21
Self-MoA
TaskBest
TaskBest 69.01
52.62
69.80
63.81
65.14
For example, considering MMLU, report 19 performance metrics (including individual models, Mixed- MoA, and Self-MoA) in Table 3 The standard deviation of performance for MMLU across these 19 settings is calculated to be 3.50. In comparison, the standard deviation for CRUX and MATH are 5.70 and 4.27, respectively. Consequently, the weight assigned to MMLU when calculating the "WeightedAvg" is given by:
1/3.50
Weight (1/3.50) + (1/5.70) + (1/4.27)
The normalized results are shown in Table
B.5 Practical Guidelines for Deploying Self-MoA
Based empirical analysis, provide the following guidelines for practitioners deploying Self-MoA:
20

---

Published in Transactions on Machine Learning Research (03/2026)
Choice of Aggregator: The choice of aggregator significantly impacts the final experimental results. We recommend users always select the strongest available model (e.g., GPT-4o or task-specific top performer) as the aggregator. As observed in our experiments, a stronger aggregator consistently boosts the performance of both Mixed-MoA and Sclf-MoA.
Optimal Sample Counts: To determine the optimal number of samples when scaling up, we suggest a three-step approach:
1. Context Constraints: First, determine the maximum number of responses the aggregator can process in a single pass based on the LLM's context window and the average proposer response length.
2. Diversity Saturation: Monitor the diversity of candidate responses (e.g., using the Vendi Score). Sampling should generally cease once the diversity score plateaus, as additional samples are likely redundant.
3. Sequential Scaling: For users requiring performance beyond the context limit, recommend Self-MoA-Seq. This allows for the iterative addition of samples; users can continue adding samples until the performance gain marginally decreases or stops improving.
21


Reviews:
[{"review_id": "VoaAluOfL3", "reviewer": "Reviewer_4AcX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Self-MoA, where the same strong model is sampled multiple times as proposers and an aggregator combines the proposals. The main claim is that Self-MoA often outperforms Mixed-MoA (mixing different proposer models), and the paper includes an analysis of a quality–diversity tradeoff plus a sequential variant to handle context-length limits.\n\nStrengths:\n1. This paper presents a simple, practical idea that is easy to implement and could be a strong baseline.\n2. The experiments in this paper cover multiple benchmarks and the results are sometimes clearly positive.\n\nMajor Concerns:\n1. This reviewer is not fully convinced the comparison to Mixed-MoA is “fair” or strong enough: outcomes in these setups can be very sensitive to prompts, temperatures, sample counts, and aggregator settings, and it’s unclear whether Mixed-MoA was tuned with the same effort.\n2. The methods are stochastic. The outputs can change with different sampling randomness and small hyperparameter changes. Some reported improvements are large, but some gaps are small enough that they could be within normal run-to-run variation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "NA", "audience_yn": "Yes", "audience_explanation": "NA", "requested_changes": "NA", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JeRbE3pFI1", "reviewer": "Reviewer_yV6p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper challenges the conventional wisdom of Mixture-of-Agents (MoA), which aggregates outputs from different Large Language Models (LLMs). The authors propose Self-MoA, an alternative approach that repeatedly samples diverse models. This paper conducts experiments on benchmarking datasets to study the trade-off between output quality and diversity in MoA settings. This paper also introduces Self-MoA-Seq, which is a sequential variant that uses a sliding window to handle context length constraints when aggregating a large number of samples.  \n\n \n\nStrengths:  \n\n \n\nS1) This paper addresses a practically relevant question about ensembling LLMs and provides a counter-intuitive finding that challenges the emphasis on cross-model diversity in previous MoA work. The proposed Self-MoA is simple to implement and reduces the complexity of model selection.  \n\nS2) The experimental analysis is comprehensive, covering multiple benchmarks across different domains including instruction-following, question-answering, coding, and solving math problems. The statistical analysis in Section 4.1 with linear regression provides quantitative evidence for the quality-diversity trade-off.  \n\nS3) In Section 4.2, this paper clearly identifies when Mixed-MoA can still be beneficial, which provides a balanced view rather than claiming universal superiority of Self-MoA. The Self-MoA-Seq extension also addresses practical concerns on context length limitations, improving the generalizability of Self-MoA given long context.  \n\n \n\nWeaknesses and Corresponding Questions:  \n\n \n\nW1) The experimental setup uses a limited set of models, and the conclusions may not apply to other LLMs with different versions and scales. This paper primarily uses Qwen and a few other open-source models.  \n\nHow does the quality-diversity trade-off change when mixing open-source and closed-source models? What if the model quality gaps are smaller?  \n\nW2) Based on Section 4.1, the choice of aggregator appears to significantly affect experimental results, but this paper does not systematically study aggregator selection.  \n\nTo deploy Self-MoA, what are the suggested guidelines for the users to follow when selecting an aggregator, and how sensitive is Self-MoA to aggregator quality relative to Mixed-MoA?  \n\nW3) In Section 5, the scaling analysis seems a bit inconsistent, where adding more samples could both help and hurt performance, and this paper does not provide clear guidance on optimal sample counts, or how the selection should work.  \n\nCan the authors provide theoretical or empirical characterization towards when additional sampling helps versus hurts? If not, are there any intuitive insights on selecting the number of samples for different tasks?", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, please refer to the strengths in the summary of contributions.", "audience_yn": "Yes", "audience_explanation": "Mixture-of-Agents (MoA) or Mixture-of-Experts (MoE) for LLMs are popular topics across both academia and industry. This paper’s claims are insightful when practically deploying LLMs with MoA or MoE strategies.", "requested_changes": "Please address those questions mentioned in the sub-section of weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yzlhuh1nY3", "reviewer": "Reviewer_nn1E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Self-MoA, which aggregates the outputs repeatedly sampled from the same model to enhance the inference performance of LLMs. Compared with the popular ensemble method MoA,  Self-MoA utilizes the in-model diversity and effectively addresses the quality-diversity trade-off. Moreover, the Self-MoA-Seq are proposed to solve the problem of limited context length, which processes samples using a sliding window, enabling scaled to an arbitrary number of model outputs.\n\nStrengths\n- The proposed method is simple and effective.\n- The experiments and analysis is detailed.\n\nWeaknesses\n- The formal description of the proposed method are missed. Although the pipeline of Self-MoA is similar to MoA and the overview is presented in Figure 1, this paper should add the detailed description in the section 3 for readers without a background in this area.\n- The exploration of in-model diversity is insufficient. It appears that the authors rely solely on varying temperature and random seeds to obtain multiple model outputs. Techniques such as using multiple prompt templates and inducing chain-of-thought branching, which can substantially enhance in-model diversity, are neither explored nor discussed.\n- To make the methodology and results more transparent, the authors should present the prompt template used to query the aggregator, as well as some cases of each proposer’s response and the aggregator’s response.\n- Because this is an ensemble method, quantifying the additional computational load and cost is crucial. The paper provides only a coarse estimate of the number of forward passes in the appendix. Different methods and models incur different computational and monetary costs; a more granular quantification in terms of FLOPs and API call expenses is needed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claims are supported by consistent empirical results across multiple benchmarks.", "audience_yn": "Yes", "audience_explanation": "These findings are directly relevant to TMLR’s audience working on LLM inference, evaluation, and multi-agent systems.", "requested_changes": "Please see the weakness.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)