Balancing Act: Diversity and Consistency in Large Language Model Ensembles
Authors: Ahmed Abdulaal
Chen Jin
Nina Montaña-Brown
Aryo Pradipta Gema
Daniel C. Castro
Daniel C. Alexander
Philip Alexander Teare
Tom Diethe
Dino Oglic
Amrutha Saseendran
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
ensembling
diversity
consistency
mixture of agents
self decoding
Primary area: foundation or frontier models, including LLMs

Abstract: Ensembling strategies for Large Language Models (LLMs) have demonstrated significant potential in improving performance across various tasks by combining the strengths of individual models. However, identifying the most effective ensembling method remains an open challenge, as neither maximizing output consistency through self-consistency decoding nor enhancing model diversity via frameworks like "Mixture of Agents" has proven universally optimal.  Motivated by this, we propose a unified framework to examine the trade-offs between task performance, model diversity, and output consistency in ensembles. More specifically, we introduce a consistency score that defines a gating mechanism for mixtures of agents and an algorithm for mixture refinement to investigate these trade-offs at the semantic and model levels, respectively. We incorporate our insights into a novel inference-time LLM ensembling strategy called the Dynamic Mixture of Agents (DMoA) and demonstrate that it achieves a new state-of-the-art result in the challenging Big Bench Hard mixed evaluations benchmark. Our analysis reveals that cross-validation bias can enhance performance, contingent on the expertise of the constituent models. We further demonstrate that distinct reasoning tasks—such as arithmetic reasoning, commonsense reasoning, and instruction following—require different model capabilities, leading to inherent task-dependent trade-offs that DMoA balances effectively.

Published as a conference paper at ICLR 2025
BALANCING ACT: DIVERSITY AND CONSISTENCY IN LARGE LANGUAGE MODEL ENSEMBLES
Ahmed Abdulaal Chen Jin'
Nina Montaña-Broww 1f Aryo Pradipta Gema
Daniel Coelho de Castro" Dino Oglic²
Daniel C. Alexander
Philip Teare'
Tom Diethe'
Amrutha Saseendran'
Centre for AI, Data Science & Artificial Intelligence, R&D, AstraZeenca, UK "Centre for Medical Image Computing, UCL, London, UK "Microsoft Research, Cambridge, UK
ABSTRACT
Ensembling strategies for Large Language Models (LLMs) have demonstrated significant potential in improving performance across various tasks by combining the strengths of individual models. However, identifying the most effective en- sembling method remains an open challenge, as neither maximizing output consis- tency through self-consistency decoding nor enhancing model diversity via frame- works like "Mixture of Agents' has proven universally optimal. Motivated by this, we propose a unified framework to examine the trade-offs between task perfor- mance, model diversity, and output consistency in ensembles. More specifically, we introduce a consistency score that defines a gating mechanism for mixtures of agents and an algorithm for mixture refinement to investigate these trade-offss at the semantic and model levels, respectively. We incorporate our insights into novel inference-time LLM ensembling strategy called the Dynamic Mixture of Agents (DMoA) and demonstrate that it achieves a new state-of-the-art result the challenging Big Bench Hard mixed evaluations benchmark. Our analysis re- veals that cross-validation bias can enhance performance, contingent on the ex- pertise of the constituent models. We further demonstrate that distinct reasoning tasks-such arithmetic reasoning, commonsense reasoning, and instruction fol- lowing-require different model capabilities, leading inherent deeendepnd trade-offss that DMoA can balance effectively.
INTRODUCTION
Frontier Large Language Models (LLMs) (Brown al., 2020; Chowdhery al., 2023; Touvron et al., 2023; OpenAl, 2023; Anthropic, 2024) continue to demonstrate ever-improving capabilities in natural language generation tasks, such as mathematical reasoning, creative writing, and interactive communication (Rafailov et al., 2024a). As foundation models, LLMs are trained on broad datasets diverse range of downstream applications (Bommasani et al., 2021).
scale and are adaptable However, further scaling model training is increasingly challenging in most practical settings due to prohibitive cost requirements (Stojkovic al., 2024) and the necessity of retraining on several trillion tokens (Wang et al., 2024).
Ensembling LLM outputs has emerged as an increasingly promising alternative method for boosting performance in variety of tasks including commonsense reasoning (Wang et al., 2022), instruction- following (Wang et al., 2024), and coding for mathematical reasoning (Project Numina, 2024). However, there is uncertainty around how L.M ensembles should be designed to achieve optimal performance across different tasks. A particularly salient issue seems to arise around the interplay between ensemble's diversity and its consistency.
Recent work on Mixtures-of-Agents (MoAs) (Wang et al., 2024) has proposed that there exists an inherent 'collaborativeness' between LLMs, whereby LLM can produce higher quality outputs when shown the outputs of other (heterogeneous) LLMs, irrespective of whether these other LLMs
"rmapabd@uc1 a....ut.uuu

---

Published as a conference paper at ICLR 2025
A. (Gated) Mixture of Agents
B. Mixture optimization
Lever
A
Intomarkce
Dynamic Mixture A gents
suboct dertifcation
aclection
Emammble consruction
A2
Figure 1: Our frameworks to evaluate the trade-offs between task performance and ensemble di- versity/consistency. A: The Gated Mixture of Agents experiment introduces a divergence metric which we to filter semantically inconsistent information from each layer. B: The Mixture Opti- mization experiment employs an algorithm (represented by the gear icon) to vary LLM mixtures to improve performance in task-dependent manner, this allows us to investigate for possible trade-offs with other tasks. C: We propose an inference-time LLM ensembling strategy we call the Dynamic Mixture of Agents (DMoA) which identifies the skills required to respond to a query, and then selects subset of models predicted to perform well given those skills. An inference- time ensemble then constructed and executed. Comprehensive evaluations are conducted on benchmarks for struction following, commonsense reasoning, and arithmetic reasoning to validate our methods.
are less capable than the main model. The authors demonstrated that using maximally diverse mix- tures consistently led to higher Longth-Controlled (LC) win rates on AlpacaEval 2.0 (Dubois al., 2024). On the other hand, fif-conniisency-basee several outputs are sampled from a single LLM for given query, continue to achieve state-of-the- art (SOTA) results in challenging tasks such as the AI Mathematics Olympiad (AIMO Prize, 2024; Project Numina, 2024). Indeed, it appears that maximising consistency, for example by combin- ing self-consistency decoding with entailment verification, can further improve performance in number tasks (Sanyal al., 2024a).
(Wang et al., 2022) decoding strategies, whereby
In this work, we systematically explore the trade-offs between task performance and ensemble di- versity/consisteneyy. and find strong empirical evidence that neither is universally optimal in gen- eral. We investigate the interaction between diversity and consistency at two levels: at the semantic level within fixed heterogeneous ture-and the model level by optimizing the LLMs in given mixture. We wish to systematize LLM ensembling strategies, then investigate these trade-off's at the semantic and model levels, and then opertaionalize our insights. To this end, our contributions are as follows: 1) Unified perspec- tive on LLM ensembling: To enable clearer conceptualization and hypothesis generation for several ensembling strategies we propose a unified framework that subsumes and systematizes current LLM ensembling approaches: 2) Novel divergence metric (' ('Gated MoA'): To investigate the effect of increasing output consistency given heterogeneous L.M mixture, we develop an 'FigenDi- vergence' metric, utilizing enhance semantic consistency across LLM outputs. (Fig. 3) Model-level ensemble optimiza- tion: We formulate simple optimization algorithm which allows to systematically investigate the diversity/consiitennyy trade-off at the level of mixture composition 1-ig.1-B); 4) Novel ensembling framework: We propose dynamic infrence-time LLM ensembling strategy call the 'Dynamic Mixture of Agents' (DMoA) (Fig. with which we achieve state-of-the- -rrte-rr performance in the Big Bench Hard mixed evaluations benchmark.
-rawning inspiration from hallucination-detectiio litera-
ceneinattioo-etecttoon based scores in the sentence embedding space to
2

---

Published as a conference paper at ICLR 2025
2002
20,2
fex
(a) Ensembling unit
(b) MoA Framework
(c) Self-Ensembling Framework
Figure 2: Illustration of single layer ensembling unit and comparison of MoA and Self-Ensembling frameworks. (a) The ensembling unit shows the optional preprocessing function Fire() and aggre- gator function F(O,) that combines outputs {O,, O2..... O₂} produced by agents A into a single output. (b) The MoA framework uses different LLMs (represented by different colors) while (c) the Self-Ensembling framework uses a single LLM (represented by uniform color) to generate multiple outputs. Both use the aggregate and synthesize (fas) processing function to produce an output.
A UNIFIED PERSPECTIVE ON LLM ENSEMBLING
In the following, our use of the word 'layer' refers specifically to a layer of LLMs (Fig. 1-Panel A). In LLM ensembling, an input query I; is used to sample a set of candidate outputs for each layer of the framework from models Af,j- where j corresponds the output sentence index and k is the index for a given LLM. The input z; may optionally be preprocessed by function F though most often this is the identity function and, as such, we henceforth refer to the input as I; for notational simplicity. Outputs for layer are then defined as O; {Or,1, Oi.,,... .., }. where Oi, A (z;). layer processing function is then applied to these outputs as F(O.). Layer processing functions can be described as layers in their own right, for instance as in Wang al. (2024), and we adopt this nomenclature here. Processing functions JF include (re-)rankers fR max, S(O,), where S(-) is scoring function assessing the quality of each response, major- ity voting fM arg maxoco, Σj= I[O,,, O] (Wang et al., 2022), and verifiers fv V(O,) (Chen et al., 2023b), where V..) is a function directly predicting response quality. Processing func- tions optionally parameterised by residual-like connections. For instance, in the 'aggregate and synthesise" (Wang et al., 2024) approach, the input query (x1) is concatenated with prompt to iteratively improve previous outputs
(As(O;;2) 21) (OO; + z1), where the symbol @ reflects concatenation of text and @ is an instruction to aggregate the previous answers and synthesise a novel response given the input query. Instantiations of this framework include self-consistency (Wang et al., 2022), which is a majority-voting self-ensemble setup (which amounts to an argmax operation on the outputs of layer)
Ti+1 = (O,) = arg max
Σ T[A (2:) =
OEO. 1=1
for fixed index & (as one LLM is used to sample multiple outputs), and cascading-styl ensembles such as FrugalGPT (Chen et al., 2023b)
-++1 = fv(O.,=1i2)) = V(O @x1) s.t. At.
1 22) 2
(2)
whereby each layer contains single LLM that attempts produce assessed for correctness by a verifier function. The process continues until the verifier attains a quality score above a pre-specified threshold 7 (Chen et al., 2023b). More recently, Mixture of Agents (MoA) (Wang et al., 2024) was proposed as multi-layer, multi-LLM ensemble which uses an aggregate and synthesise approach to produce single output for each LLM at each layer:
that is subsequently
Z++1 fAs(0( c (o [ @ (x)).
We are principally interested in multi-layer multi-LLMM ensembling with an aggregate and syn- thesise processing function (i.e. the MoA framework as in Eq. 3; Fig. 2b) and multi-layer self- ensembling with an aggregate and synthesise processing function (Fig. 2c):
Fi+1 lAs (A;(0;;)1)
(0 [A (xxa) m1).
3

---

Published as a conference paper at ICLR 2025
We believe this enables an appropriate comparison between the current sate-of-the-as art ensembling methods for instruction following benchmarks (Wang et al., 2024; Dubois et al., 2024; Zheng et al., 2024; Ye et al., 2023) and popular self-ense methods (Wang et al., 2022; Sanyal et al., 2024b). Nevertheless, a diverse L.M "ensembling zoo' is enabled by the unified perspective proposed here. For instance, one could imagine multi-layer self-ensembling framework with majority voting the processing function, which would represent multi-layer extension to the original self-consistency framework proposed by Wang et al. (2022), or multi-layer, multi-LLM ensemble with verifier function where the LLMs are increasingly more capable in each successive layer (which would rep- resent a multi-LLM extention to work by Chen et al. (2023b)). Whether every prompt completion should be cross- validaadi to improve response quality, and with which LLM ensembling architec- ture, remains an open question.
3 METHODS
Our overarching hypothesis is that task-dependent trade-off's exist between performance and ensem- ble diversity/consiitencyy informing our experimental progression. First, in the Gated Mixture of trade-off within fixed mixture
Agents (GMoA) experiment, we assess the by introducing a divergence metric based on the EigenScore for hallucination detection (Chen et al., 2024a) to enforce greuter semantic consistency among LLM outputs. Second, in the Mixture Opti- mization experiment, we examine this trade-off at the level of mixture composition, proposing an algorithm to systematically vary the included LLMs explore how different combinations impact task performance. Results from the first two experiments provide insights which guide the design of third approach: an inference- ensembling strategy, the Dynamic Mixture of Agents (DMoA), which constructs task-dependent ensembles based on the specific skills required to pro- duce high-quality outputs.
3.1 GATED MIXTURE OF AGENTS VIA DIVERGENCE FILTERING
In this section we describe a novel divergence metric which we leverage in Sec. 4.1 enforce greater semantic consistency by filtering out some LLM outputs (Fig, I-Panll A). Motivated by the observation that sampling multiple outputs for a fixed input useful for estimating sequence-level uncertainty (Manakul et al., 2023), Chen et al. (2024a) proposed an EigenScore to detect LLM hallucinations based on the consistency of semantic information in the outputs. For K sampled outputs, the covariance of the sentence embeddings can be calculated
== la 7 1a Z,
(5)
where E [gd a matrix of d-dimensional sentence embeddings, 1a is an all-one column vector, Ia is d-dimensional identity matrix, and Σ E I is a covariance matrix capturing semantic consistency in cetencc-embedddee space. The EigenScore is then the logarithmic determinant of Σ or, equivalently, the average logarithm of the eigenvalues
E E xz 0) = K
log det E +
Ik) = K Σ log(Ar),
(6)
where is the input context, 0 = [01. .2.....0x are the parameters of the LLMs in an MoA (which may be shared in the case of single LLM producing multiple outputs), and Ix is regularization term to ensure the covariance matrix is full rank. Figenscores were demonstrated to be more robust than perplexity (Ren et al., 2022), entropy (Kadavath al., 2022), length-controlled entropy (Malinin & Gales, 2020; Lin al., 2023), and lexical similarity scores (Lin et al., 2022) for consistency-ban consistency metric (Chen et al., 2024a). It should be noted that the logarithm determinant operation in Eq. has a strong relationship to information entropy (Cover, 1999). Indeed, for a multivariate Gaussian X Mi,, E), the Shannon differential entropy is defined
hallucination detection and are therefore operationalized here as a semantic
1 d
h(X)= = log det + d (log 2n + 1) = 2 2
Σ
log A + +C,
(7)
4

---

Published as a conference paper at ICLR 2025
where d is the dimension of the variable X and C' is constant term (Zhouyin & Liu, 2021). The EigenScore can therefore be interpreted as the differential entropy in sentence embedding space. Chen et al. (2024a) used the final token embeddings in the middle layer proxy for sentence em- beddings, however this limited analyses to a single open-source L.M,, as this required access to the internal model states. To enable evaluations between multiple (open- and closed-source) L.MMs, project LLM outputs into a sentence-embedding of K output sequences and e(-) is an embedding function. We demonstrate strong concordance between 'internal' and 'external' eignscores in Appendix F. We additionally introduce EigenDiver- gence as a measure of an individual output's contribution to overall semantic consistency within an MoA. For given MoA with K outputs, we calculate the EigenDivergence of the i-th output as the difference between the overall EigenScore and the EigenScore computed without that output:
space as Z = e(S), where S = {81, $1..... 8 is
D DE 0) E a: 0) E (E-1xx;; 0 0-1),
where E(> ax; 0) is the EigenScore calculated using all K outputs, and E- 0-i) is the EigenScore calculated using K outputs, excluding the i-th output. The EigenDivergence thus quantifies the extent to which each individual output contributes to or detracts from the overall semantic consistency of the MoA. As per Eq. 7, the FigenDivvrgence can be interpreted as the change in differential entropy when removing an output from the ensemble:
< 0) x h(z) h(E-.).
This formulation provides an information-thhoreee interpretation of EigenDivergence: it quantifies the amount of unique information contributed by each output to the overall ensemble, and is there- fore closely related to information gain (MacKay, 2003). further discussion of this relationship is given in Appendix G. A large positive EigenDivergence indicates that removing the i-th output significantly reduces the entropy (or uncertainty) of the ensemble, suggesting that this output con- tributes novel or diverse information. Conversely, a negative FigenDivergence suggests that the output more closely aligned with the information already present in the ensemble.
3.2 MIXTURE OPTIMIZATION
To investigate the tradeoff between diversity and consistency at the level of mixture composition, require an optimization approach with three desiderata: 1) The algorithm is cost efficient; 2) optimizes for stronger performance for a chosen downstream task; 3) It allows for interpolation between multi-LLMM and self-ensembling mixtures. To this end, propose simple algorithm to optimize a mixture composition towards higher performance for given benchmark as follows: For GMoA (Fig. 1-Panel A), let P. be the performance metric for run i. The performance difference across runs is then
AP P++1 - P..
(10)
Let Ui(m) be the usage of model m within the mixture for run é, then the usage difference across runs is
(mm) = (m) - (m)).
(11)
The delta for each model (AM) then its proportional impact on the performance change
(m Σm [A(m))
AM
x AP
(12)
where Eq. 12 assumes that each model's 's impact is directly proportional to its usage change relative to the total usage change. A high model delta indicates that either increasing model usage correlated with an increase in performance, or decreasing its usage correlated with decrease in performance. A negative model delta indicates that increased model usage correlated with a decrease in perfor- mance, or decreasing its usage correlated with an increase in performance. In each run, the model with the lowest delta is replaced by a 'clone" of the model with the highest delta, thereby adapting the mixture towards models that demonstrate the strongest positive impact on overall performance. Clones sample a different output for the same input. A full description the algorithm is given in Appendix C.
5

---

Published as a conference paper at ICLR 2025
Table 1: Performance impact of EigenDivergence filtering across benchmarks. All ensemble exper- iments are run three times and average scores are reported with standard deviations. denotes our replication of results.
Instruction Following
Arithmetic Reasoning
Commonsense Reasoning
Model
AlpacaEval 2.0 MT-Beech
GSM8K
MATH
CSQA
ARC-C
ARC-E
GMoA (Ours)
58.66 40.29
8.97
94.23
56.35
85,20
4-.22
92.32 40.38
93.75
MoA
59.50 =0.08
9.19
#0.07
93.87
=0.35
55.22
40.29
84.32
10.31
91.85
10.14
94.31
=0.32
LJama-3-70B
34.4
93.0
50.4
83.8
90.5
94.1
Qwen- 1.-
43.9
8.9
85.4
49.6
82.1
69.6
93.9
Qwen-1.5-2B
36.6
8.4
79.5
34.1
83.2
65.9
92.7
WitardLM-9K22B
51.3
8.8
81.6
22.7
69.0t
62.5
90.11
Mixtral 8x22B
30.9
8.8
83.7
41.7
81.71
70.7
91.87
DBRX-Instruct
25.4
8.4
72.8
32.5
82.21
68.9
89.71
GPT-4 Omni (05/13)
57.5
9.19
94.11
61.2%
88.61
94.6"
94.3*
3.3 DYNAMIC MIXTURES AGENTS (DMoAs)
Building on the insights from GMoA and Mixture Optimization balancing semantic consistency and diversity for spec-fpecific performance, we propose the Dynamic Mixture of Agents (DMoA) to create adaptive, skill-targeted ensembles inference time. The preceding experiments demonstrate that balancing diversity and consistency is crucial yet varies by task type: Given a set of queries Q = zs and models M m,, for each query 4 E Q, first identify the required skills S fs(; 0) (based on Sec. 4.4; Insight 3), where 0 are optionally learnable parameters. Next, we select a subset .M predicted to perform well given these skills (based on Sec. 4.4: Insight 2). This forms a preprocessing function for query routing, Fpre(q;) fm of (as introduced in Sec. 2). Each model in .Ms generates an output, which we combine using the aggregation and synthesis method (Eq. 3), leveraging validaainn bias to improve results (based on Sec. 4.4;
of models MS = fm (8; ; qi
Insight 1). Unlike traditional L.LM routing which often selects a single optimal model for query (Lu et al., 2023; Shnitzer et al., 2023; Liu et al., 2023), the DMoA constructs ensemble of models queried at inference time. We assess the DMoA framework on Big Bench Hard (BBH) (Suzgun al., 2022), a set of 23 challenging tasks including language understanding, algorithmic reasoning, and multistep arithmetic, Details of the experimental setup are provided in Appendix E.
4 EXPERIMENTS
4.1 GATED MIXTURE OF AGENTS VIA DIVERGENCE FILTERING
Experimental setup We construct an MoA (Fig. 2b) using the same open-source models proposed by Wang et al. (2024). Namely, we use Llama-3-70B: -tnsttu (Touvron et al., 2023), Qwenl.5-72B. Chat (Bai al., 2023), Qwenl.5- 10B-Chat (Bai et al., 2023), Mixtral-Sx22B-vd,, (Jiang et al., 2024), WizardLM-8x2D1 (Xu et al., 2023), and dbrx-instruct (The Mosaic Research Team, 2024). We construct the 'MoA-Lite" variant of this meta-architecture, which contains MoA layers and uses Sweel.5-72B-CC
the 'aggregate and synthesize" processing function (Eq. 3). To inves- tigate the interplay between semantic consistency and diversity and its effect on task performance, FigenDivvrrgecce to filter the two least semantically consistent outputs from given MoA layer. The idea to establish a balance between diversity of outputs and consistency of semantic information across the ensemble. By removing the outputs with the highest EigenDivergence, we aim to maintain a core set of semantically consistent responses while still preserving some level of diversity from the heterogeneous LLMs. We then compare the performance of standard MoAs variants, which we call Gated Mixtures of Agents (GMoAs). Ad-
against their divergence-filtern ditional implementation details can be found in Appendix B, and description of the relationship between GMoAs and classical Mixtures of Expert (MoE) approaches (Shazeer et al., 2017) is given in Appendix B.5.
Results Results of the EigenDivergence filtering experiment are shown in Table 1. Whilst the LLM ensembles broadly outperform their individual constituent L.MMs across all tasks, FigenDivergeence

---

Published as a conference paper at ICLR 2025
filtering demonstrates different effects across different types of reasoning tasks. Instruction follow- ing: AlpacaEval 2.0 is a popular wonth-conttoolldd reference-free free instruction following benchmark (Dubois et al., 2024). Here. the GMoA (filtered) model performs substantially worse with 0.84% absolute degradation performance, with a similar decline of 0.22% noted on MT-Bench (Zheng al., 2024). Arithmetic reasoning: Increasing semantic consistency by performing filtering does not degrade performance for arithmetic reasoning, with the GMoA demonstrating a 0.36% absolute improvement performance for the GSM8K benchmark (Cobbe et al., 2021), with a 1.12% per- formance gain in the MATH benchmark (Hendrycks et al., 2021). Commonsense reasoning: In the commonsense reasoning tasks, the impact of EigenDivergence filtering is nuanced. For exam- ple, the GMoA model shows an absolute improvement of 0.88% on the CSQA benchmark (Talmor al., 2018), indicating that filtering enhances the model's ability to manage tasks requiring broad knowledge and logical inference. However, this effect is less consistent across other benchmarks. On ARC-E (Clark et al., 2018), where most models already cxhibit high accuracy, filtering might inadvertently remove correct but diverse responses, leading to slight underperformance compared to the unfiltered MoA model. This suggests that in tasks where models have already reached a high level of proficiency, the filtering process could sometimes hinder rather than help by exclud- ing potentially valuable outputs. Conversely, in more challenging tasks like ARC-C (Clark et al., 2018), where inconsistency is more likely correlate with errors, filtering proves beneficial. The differential impact of EigenDivergence filtering across benchmarks appears to support our initial hy- pothesis of task-dependent trade-off between diversity and consistency. This is noted the level of semantic consistency within fixed (heterogeneous) LLM ensemble.
4.2
MIXTURE OPTIMIZATION
Usage Proportion Across Runs Correlation between AlpacaEval
GOMBI)
KTBench
AG22
7.n
MATH
Sm
39
1un
88
CSGAR
1
0%
23%
40%
60%
8914
1003
355
54.0
16.3
15.0
25.5
26.0
Proportion of Usage
GSMBK (16)
AhilChellonge
Gwerr/laan1 1-100--htt 0per/Owenn 5-72--hhttt
mets-llamallams 70b-chat-hf
matratal Instruct-vO.11
GPTH
Figure 3: Left: Model usage and performance comparison. The algorithm used optimize for arithmetic reasoning on the GSM8K benchmark. The left-most figure shows usage proportion across runs. In this instance -lama-3--0b-che demonstrated consistently positive model deltas across runs. The central-plot illustrates that as mixture performance increased for GSMSK, length-controlled winrates for AlpacaEval 2.0 appreciably decreased. Right: Performance trade-offs in LLM ensem- ble optimization across reasoning tasks. MoA-ARTH = Arithmetic reasoning optimized mixture, MoA-CSR = Commonsense reasoning optimized mixture, and MoA-IF Instruction following op- timized mixture. Represents benchmarks used to optimize an ensemble.
Experimental setup For each task, we start with the same LLM composition and MoA architec- ture as in Sec. 4.1 and run the optimization algorithm for each of instruction following, arithmetic reasoning, and commonsense reasoning. For arithmetic reasoning tasks, we optimize the ensemble performance on the GSM8K benchmark. For commonsense reasoning, we optimize performance on CSQA, and for instruction following we optimize on AplacaEval 2.0. We use the same sam- pling scheme as in Sec. 4. (described Appendix B.4) to generate LLM outputs, and compare the relative performance of optimized mixtures to the baseline heterogeneous MoA.
Results Optimizing mixture for a given benchmark demonstrates clear trade-off with other reasoning tasks. Fig. 3-left/center illustrates an example run of the optimization algorithm on the GSM8K benchmark and the effect of adapting the mixture for improved arithmetic reasoning on instruction following performance as measured by AlpacaEval 2.0. As the mixture becomes less diverse, arithmetic reasoning improves. Additionally, there is negative correlation between
7

---

Published as a conference paper at ICLR 2025
performance GSM8K and AlpacaEval 2.0; 59.59% on AlpacaEval 2.0, whereas the optimized mixture is a self-ensemble which scores 96.21% on GSM8K and 55.87% on AlpacaEval 2.0. Fig. 3-right illustrates the performance trade-offs more broadly; the "MoA-IF" configuration, optimized for instruction following, yielded the high- est AlpacaEval score of 61.74, but its arithmetic reasoning performance dropped significantly, with a GSM8K score of 91.36. In commonsense reasoning tasks, the "MoA-CSR" configuration, op- timized for these tasks, performed best on the CSQA benchmark with score of 87.3. However, this focus led to decreased performance in both arithmetic reasoning and instruction following, with GSM8K and AlpacaEval scores of 92.4 and 52.83, respectively. These findings demonstrate that is challenging to achieve universally optimal LLM ensemble, as improvements in one area often necessitate compromises others.
The baseline MoA scores 93,48% on GSM8K and
160
100
ARCE
-1.5%
Riand
ABC
-3.9%
C50A
-1.99
80
Score
+8.7%
Score
COMEK
10,5
Lendmuue appccatval 2.8
M7 MSBecch
4.1%
AND-C
Alpaca
Coneutercy Devrriy
50-
-00
Percantage Change
GEMEP
Appection
#raanodd
MCC
ARCa
Prompting Strategies
The aggregation and synthesis (AS) strategy (which explicitly performs critical appraisal of outputs from the previous layer in the ensemble) outperforms both simple LL.--based ranking (LR), and universal self-consistency (USC), which selects for the most consistent informa- tion. Center: Enforcing higher semantic diversity by removing more consistent outputs degrades performance across all reasoning tasks relative to both the baseline ensemble and the lower seman- tic diversity ensemble. Right: Filtering already specialized ensembles degrades performance across all reasoning tasks, and suggests that specialized knowledge retained in unfiltered ensembles con- tributes meaningfully to performance.
Figure 4: Left:
4.3 ABLATION STUDIES
We conduct a set of additional experiments which further characterize the effect of semantic diversity in LLM mixtures, the effect of filtering in specialised ensembles, and different processing functions for different reasoning tasks.
Aggregation and synthesis outperform LLM-based ranking and self-consistency. We com- pared the aggregation and synthesis (AS) function (-s(.) with LLM--bsedd ranking f(((.) and self- consistency (+) as described in Sec. 2. Specifically, we used the recently proposed 'universal' self-consistency (USC) by Chen et al. (2024b). which extends self-consistency to open-ended lan- guage generation tasks (experimental setup in Appendix D). Our results show that AS outperforms both ranking and USC, with ranking particularly detrimental for commonsense reasoning and in- struction following (Fig. 4-left). This suggests that combining multiple results through critical ap- praisal is preferable to performing an argmax operation or selecting repeating information in open- ended tasks (Chen al., 2024b). Higher semantic diversity leads to worse performance across reasoning tasks. We compared the baseline MoA (Sec. 4.1) with two GMoA variants: one with the two most semantically divergent outputs removed (maximizing consistency), and one with the two most consistent outputs removed (maximizing diversity). We found that higher semantic diversity in heterogeneous ensemble degrades performance across all tasks (Fig. 4-center), indicating that some semantic consistency is necessary for high-quality results, even in open-ended instruction- following queries. Filtering specialized ensembles degrades performance. Applying semantic filtering to ensembles already optimized for a specific task leads to slight performance decreases across arithmetic reasoning (MoA-ARTH), instruction following (MoA-IF), and commonsense rea- soning (MoA-CSR), with reductions ranging from 0.04% to 3.3% (Fig. 4-right). This suggests that the specialized knowledge retained in unfiltered, optimized ensembles contributes meaningfully to task performance and should therefore be maintained.
8

---

Published as a conference paper at ICLR 2025
4.4 DYNAMIC MIXTURES OF AGENTS (DMoAs)
In this section, we operationalize the following insights from our experiments: Insight 1 Induc- tive cross-validation bias: Reasoning traces validated by multiple hypotheses are more likely to be correct. As shown in Sec. 4.1, LLM ensembles outperform individual models regardless of divergence filtering, across both open- and close-ended tasks. Maximizing semantic diversity harms performance (Sec. 4.3), indicating that some cross-validation between outputs is necessary for high-quality results, even in open-ended instruction-folle following tasks (Wang et al., 2022: 2024). Insight 2 Task-dependent LLM expertise is crucial: Contrary to prior work (Wang et al., 2022), we found in Sec. 4.1 that removing information from an ensemble can improve task-specific per- formance. Sec. 4.2 shows that optimizing mixture for specific task enhances performance on similar tasks but degrades it on dissimilar ones, especially between instruction following and arith- metic or commonsense reasoning. Our ablations (Sec. 4.3) demonstrate that removing information from specialized ensembles diminishes their target performance. Insight - Task-specific skills reside in different subspaces: Mixtures which are optimized for type of reasoning task tend to underperform in others (Sec. 4.2). Additionally, in our third ablation (Sec. 4.3), we showed that while aggregation and synthesis outperform other processing functions, tasks differ in sensitivity to the processing method used.
Table 2: Performance on the Big Bench Hard (BBH) benchmark, evaluating accuracy, normalized accuracy, and best/worst subset results. The LLM ensembles Qwen2-72B 1 the final aggregator, with the exception of DMoA/Sonnet, which uses Claude 3.5 Sonnet.
Acc. (9X() Norm, acc. (%)()
Best subset
Worst subset
DMoA/Sonnet (ours)
93.12%
91.85%
temporal sequences
causal judgement causal judgement
Claude 3.5 sonnet
93.1 1%
90.20%
tracking 5 objs
DMoA (ours)
86.05 %
83.63%
hyperbaton
causal judgement
MoA
83.81 %
79.25%
web of lies temporal sequences
dyck languages
84.55%
78.28%
formal fallacies
82.12%
75.85%
web of lies hyperbaton web of lies
causal judgement
81.69%
75.11%
causal judgement
Bwen/Qween.5--110---
76.40%
68.45%
formal fallacies
Qwen/QQwen1.-772B-c4
75.24%
66.94%
web of lies
formal fallacies
dattbriebssddrrx-nnsrrttt
69.84%
59.29%
movie recommendation sports understanding
Table 2 shows that the MoA framework outperforms individual models, and the DMoA surpasses the MoA, achieving 83.63% normalized accuracy-922.0 of the performance of Claude 3.5 Sonnet, the current state-of the-hhh-rrtt (SoTA) model for BBH (Anthropic. 2024). When Claude 3.5 Sonnet is used as the final aggregator in the DMoA, it achieves a new SoTA normalized accuracy of 91.85%. Despite differences in architectures, data mixtures, and training regimes, many models perform poorly on tasks requiring causal judgment or evaluating formal fallacies, performing better on simpler requence-based tasks. This limitation persists even ensembles, suggesting that novel expertise cannot be acquired through simple ensembling; if the underlying models lack the necessary skills to solve task, their ensemble cannot compensate for that deficiency. The DMoA balances the trade-offs identified in prior experiments.
Results
5 RELATED WORK
LLM Reasoning Chain of Thought (CoT) reasoning was introduced as a few-shot 'multi-step' prompting technique encouraging LLMs produce intermediate reasoning traces to improve per- formance on reasoning tasks (Wei et al.. 2022), with both few-shot (Rubin et al., 2021) and zero-shot (Kojima et al., 2022) variants. Decomposed prompting (Khot et al., 2022) tackles complex tasks by breaking them into simpler sub-tasks, each delegated to dedicated LLMs. Similarly, least-to-most prompting (Zhou et al., 2022) decomposes problems into sub-tasks, but each solution depends on the answers to previous sub-tasks. Other approaches integrate multiple reasoning traces (Fu et al., 2022; Wang et al., 2022; Li et al., 2022) or select traces based on similarity (Rubin et al., 2021; Lu et al., 2022), diversity (Zhang et al., 2022), or entailment (Sanyal et al., 2024a). The Tree of Thought (ToT) framework (Yao et al., 2024) generalizes CoT by having LLMs self-evaluate multiple rea- soning paths, while Graph of Thought (GoT) frames reasoning a graphical task by producing

---

Published as a conference paper at ICLR 2025
'thought graph' (Yao et al., 2023). However, most of these frameworks restrict analysis to single LLM for each task.
LLM Ensembling Jiang et al. (2023) performed pairwise comparisons between model outputs to select the higher quality response. A number of works trained models which predicted the best performing LLM from a fixed set for a given user input (Lu et al., 2023; Shnitzer et al., 2023; Wang et al., 2023a). Other studies have investigated combining outputs from different models by syn- thesis of results (Jiang et al., 2023), or by averaging model oulput probability distributions (Huang al.. 2024). A related line of work proposed symmetric (Du et al., 2023) or asymmetric (Liang et al., 2023; Chan et al., 2023; Chen et al., 2023a) frameworks to improve reasoning through prob- lems. Wang et al. (2024) proposed the Mixture-of-Agent of-Agents (MoA) approach a multilayer framework with different LLM agents within each layer. Each agent iteratively improved the outputs from the previous layer to achieve higher quality outputs. This represented the assumed benefit of model heterogeneity on downstream tasks.
Hallucination Detection Models which produce inconsistent results (by sampling multiple out- puts given fixed query) have been thought to be 'indecisive' about how to respond to the query and more likely to produce hallucinations (Kuhn et al., 2023; Manakul et al., 2023; Raj et al., 2023; Chen et al., 2024a). Kersting et al. (2024) perturbed LLM input queries in a non-semantically mean- ingful way and sampled a single output for each perturbation, following which a consistency metric was computed to detect hallucinations. Whilst consistency metrics have included (normalized Jen- tropy (Kadavath et al., 2022; Malinin & Gales, 2020; Lin et al., 2023), perplexity (Ren et al., 2022), and lexical similarity scores (Lin et al., 2022), several works have leveraged LLMs to self-evaluate their responses directly (Manakul et al., 2023; Kadavath et al., 2022; Cohen et al., 2023). Jesson al. (2024) framed hallucination detection as a Bayesian task by defining hallucinations as low- probability predictions under a true latent parameter. Broadly. hallucination detection work does not explicitly perform hallucination mitigation (Kuhn al., 2023; Chen al., 2024a; Jesson al., 2024). For instance, whilst the INSIDE score proposed by Chen et al. (2024a) was used to detect the likelihood of hallucination, the score itself was not used for hallucination mitigation.
6 DISCUSSION, LIMITATIONS, AND CONCLUSION
In this work, we systematically explored the interplay between diversity and consistency in LLM en- sembles and proposed a unified framework that subsumes existing ensembling approaches. Through the development of the EigenDivergence metric, we assessed the impact of semantic consistency within ensembles and found that while strong consistency improves performance in tasks like arith- metic and commonsense reasoning, it can impair instruction-folloiing proficiency. Conversely, we observed that introducing an optimal degree of diversity can partially mitigate these limitations, enhancing instruction -folowiinn capabilities at the expense of reasoning and mathematical perfor- mance. Our mixture optimization experiments revealed inherent trade-offs when tailoring ensembles to specific tasks, emphasizing the challenge of balancing diversity and consistency. Our proposed Dynamic Mixture of Agents (DMoA) achieves satte-of-theait performance the challenging Big Bench Hard (BBH) benchmark by dynamically selecting models based on task-specific skills. The ability scale inference-time compute based query requirements opens new directions for opti- mizing LLM performance across a diverse range of tasks, and is a natural avenue of future research.
The trade-offs in performance across benchmarks can be explained by Goodhart's law, which states that statistical regularities collapse under pressure for control (Goodhart, 1984). For instance, the OpenAl ol lpreeviw model dramatically outperformed gpt-4o at arithmetic reasoning, yet under- performed it in 'personal writing' (OpenAI, 2024). Additionally, overoptimizing proxy reward models in RLHF settings has led to decreased utility relative to ground-truth rewards (Gao et al., 2023), a phenomenon also observed with direct alignment algorithms like DPO (Rafailov et al., 2024a;b). Our findings suggest this overoptimization extends ensembles of pretrained L.Mss. Specifically, Extremal Goodhart's 's law (Manheim & Garrabrant, 2018) indicates that optimizing for extreme proxy values-such as human--reeferece--aasee mance in unrelated tasks like arithmetic reasoning. This may result from model insufficiency, where the proxy metric oversimplifies the target skill (Zheng et al., 2024; Koo et al., 2023; Wu & Aji, 2023; Wang al., 2023b), or because optimization pushes models into regions where underlying processes differ between skills. Consequently, models highly optimized for a specific task such as instruction following may unexpectedly underperform in other reasoning tasks.
instruction following- can degrade perfor-
10

---

Published as a conference paper at ICLR 2025
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):1477-69, 1985.
AIMO Prize. Aimo prize, 2024. URL https:/ aimoprize com/. Accessed: 2024-07-12.
Anthropic. Introducing claude 3.5 sonnet, 2024. URL http::::/wwww..ahhoo.......... news/ wtwss/clauee-ee 3-5--onnet.. Accessed: 2024-07-12.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint 1XXiv2108.007553 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.0720 2023.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of ballucination detection. arXiv preprint arXivv2402.03744, 2024a.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse 1lms. arXiv preprint arXiv:2309.13007 2023a.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.055 05533055.16.... 176, 2023b.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin. Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language models. In ICML 2024 Workshop on In-Context Learning, 2024b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 11188833.05555 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs Im: Detecting factual errors via cross examination. arXiv preprint arXiv:230s.13288 2023.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B Hashimoto. pacaeval: A simple way debias automatic evaluators, arXiv preprint arX1v.2404.04475, 2024.
al-

---

Published as a conference paper at ICLR 2025
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint MrXiv:1805.04833, 2018.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. arXiv preprint 1eXiv:1707..0 02663, 2017.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Charles AE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint IrXiv:I805.c0mm $05.06087 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: ExXiv:1900 190..0994.97,, 2019.
Yichong Huang, Xiaocheng Feng. Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. Enabling ensemble learning for heterogeneous large language models with deep parallel collabo- ration. arviv preprint arXiv:2404.12715, 2024.
Hugging Face. Open Ilm leaderboard: About. https:/ htopss//huggingfcce co/ docs// leaderboards/ open 11m Iww.eeeerrbaaraaaaaaaaaaaa abbout, 2023. Accessed: 2024-09-16.
Andrew Jesson, Nicolas Beltran-Veles Quentin Chu, Sweta Karlekar, Jannik Kossen, Yarin Gal, John Cunningham, and David Blei. Estimating the hallucination rate of generative arXiv e-prints. pp. arXiv-2406, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 04088, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.022. 026661 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dode Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint ar arXiv:2207. 002221,,, 2022.
Nicholas S Kersting, Mohammad Rahman, Suchismitha Vedala, and Yang Wang. Harmonic Ilms are trustworthy. arXiv preprint arXiv:2404.199 197088 2024.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213.. 2022.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.1001 2023.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302009664 2023.
12

---

Published as a conference paper at ICLR 2025
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier. arXiv preprint arXiv:2206.02336, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 19118. 2023.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. arXiv preprint urXiv:2305.19187 2023.
Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. Findings of the Association for Computational Linguisties: ACL 2022, 2022.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.1502: 2023.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward- swwarddguided ensemble of large language models. arXiv preprint S.ii:2311.086922 2023.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.1444 2022.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university
press, 2003.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint a00iv:2002.07650, 2020.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallu- cination detection for generative large language models. arXiv preprint arXiv:2303.088% 2023.
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's 's law. arXiv preprint arXiv:1803.04585, 2018.
LLM Leaderbbarrr Team. Open Ilm leaderboard. https: htwggggggggggg.eeceeeeac htteefaaeeccceeeeee copspeaes///cccccccocccccc..cssc open-11n--Ledderborrd open 11m leaderboard, 2024. Accessed: 2024-09-16.
OpenAl. Gpt-4 technical report, 2023.
OpenAl. New embedding models and api updates, January 2024. URL httsttt:: com/ -atp::/ee--edodoo..ooeeee---aaa- and-appi-pdatess. Accessed: 2024-06-27.
OpenAl. Language model embeddings.
www.aaaa.o.pp.i.....mmmmo....... /occ//
quidess/ebedddigee wwwiingg//whwttttte awttt/aa-----bb-bea-a--eee-e-ee- 2024a. Accessed: 2024-08-15.
OpenAl. Gpt-3.5 turbo.
3pt-3-5-ture 2024b. Accessed: 2024-09-16.
Hello https: http://twws.ggaaaaaaa..a///aaaaa :ccssee 2024-09-16.
OpenAl. Learning to reason with large language models. https://openai.com learning-tor to- reason-wizd with llms 2024. Accessed: 2024-09-18.
Project Numina. Publications, 2024.
URL. ht:ps::/pprojecchummnn
publications/ Accessed: 2024-07-12.
Qwen Team. Qwen1.5-110b, The first 100b+ model of the qwen1.5 series. https: //wenlm.
github.ic/blooo /wwp///wlll...... 5-11 10/, 2024. Accessed: 2024-09-16.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikehi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900 02900, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Harsh Raj. Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models. arXiv preprint arXiv:2308.091 2023.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. ditto--distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2022.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint 12Xiv:2112.086633 2021.
Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang, and Xiang Ren. Are machines better at complex reasoning? unveiling human-machine inference gaps in entailment verification. arXiv e-prints, pp. arXiv-2402 2024a.
Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang, and Xiang Ren. Minds versus machines: Rethinking entailment verification with language models. arXiv preprint arXiv:2402.03666, 2024b.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 170 701.06538, 2017.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15780 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint MtXiv:2206.04611 2022.
Jovan Stojkovic, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas. Towards
greener 1mss: Bringing energy-efficiency to the forefront of 1lm inference. arXiv preprint arXiv:2403.20306, 2024.
Mirac Suzgun and contributors. Big-bench-hard: Challenging big- tasks and whether chain- of-thought can solve them. https: / com/ suzgunmi racc/aaccC Bench- Hard, 2023. Accessed: 2024-09-16.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether ch can solve them. arXiv preprint arXiv:2210.092611 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 181.00997.097 2018.
The Mosaic art open 11m, 2024.
Introducing dbrx: A new state-of-the- https:/ /tt///ww.... aataabicke com/blog/ introducing- dbbx- new state- art nett-open1111 11m. The Mosaic Research Team.
Research Team.
URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
14

---

Published as a conference paper at ICLR 2025
Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. arXiv preprint arXiv:2310.01542, 2023a.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.0466 0.000. 2024.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17922 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Lc, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv preprint arXiv:2307.00025 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint 2304:2304.12244 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305. 16582, 2023.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928, 2023.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 2022.
Zhanghao Zhouyin and Ding Liu. Understanding neural networks with logarithm determinant en- tropy estimator. arXiv preprint arXiv:2105.03700, 2021.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Appendix A Reproducibility Statement
17
Appendix B Gated Mixture of Agents Experiment
B.I. Tasks and Datasets
18
B.2. Language Model Prompting
18
B.3. Language and Embedding Models
19
B.4. Sampling Scheme
19
B.5. GMoAs and MoEs
19
Appendix C- Mixture optimization experiment
19
Appendix D Ablation study for post-processing function
20
Appendix E Dynamic mixture of agents experiment
F.I. Corrected BBH
22
E.2. DMoA pre-processing function
24
E.3. BBH evaluation
26
Appendix F Concordance between internal and projected embeddings
F.I. Experimental Setup
27
F.2 Results
27
Appendix Relationship between eigendivergence and information gain
27
Appendix Investigation of the EigenDivergence (ED) Score
29
Appendix I Cost analysis
32
Appendix J Test-time scaling.
32
16

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY STATEMENT
We provide details on the experimental setup, hyperparameters, and data preprocessing steps to max- imize the reproducibility of our results. The codebase can be found at: https:/ /anonmoous. 4open. wwwiecceeeeecceeeeeeeeeeeeeeeeee /twpaaaaninngatttaaaaaaaaaaa
Listing 1 ARC CoT prompt
Q: George wants to warm his hands quickly by rubbing them. Which skin surface will produce
the most heat?
Answer Choices:
(a) dry palms
(b) wet palms
(c) palms covered with
(d) palms covered with lotion
A: The dry palms will produce the most heat when rubbed together because they create the
highest friction, allowing for more efficient conversion of mechanical energy into thermal energy without the lubricating effects of moisture or other substances. The answer is (a).
Q: Michael learned that the movement of Earth in the solar system causes changes that can be seen on the planet. Which change could be seen on Earth in the time it takes Earth rotate once on its axis?
Answer Choices:
(a) day becoming night
(b) winter changing to spring
(c) January changing February
(d) a new moon becoming a full moon
A: Day becoming night is the change that can be observed Earth during one complete rotation on its axis because this rotation is what causes the cycle of daylight and darkness, with different parts of the planet facing towards away from the Sun the course of approximately hours. The answer (a).
Q: Which would scientist use in trying to model the cause of planetary years?
Answer Choices:
(a) planetary mass
(b) planetary color
(c) planet's core temperature
(d) a planet's distance from the Sun
A: A planet's distance from the Sun is the key factor scientist would use to model the cause of planetary years because the length of a planet's orbit around the Sun, which determines its year, is primarily dependent on its distance from the Sun according to Kepler's laws of planetary motion. The answer is (d).
Q: Which phrase does not describe asexual reproduction in organisms? Answer Choices:
(a) requires two parents
(b) little variation in offspring
(c) only one type of cell involved
(d) duplicates genetic material
A: 'Requires two parents' does not describe asexual reproduction because asexual reproduction
characterized by a single organism producing genetically identical offspring without the need for mate, unlike sexual reproduction which involves genetic material from two parents. The answer (a).
17

---

Published as a conference paper at ICLR 2025
Listing 2 Aggregation and Synthesis Prompt
You have been provided with a set of responses from various open-source models to the latest user query. Your task is to synthesize these responses into single, high-quality response in order to answer the final question given to you note that, in order to help you understand the task, the user may first show some example questions. the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well- structured, coherent, and adheres to the highest standards of accuracy and reliability. If you are asked provide single answer a question, do not include multiple answers in your response.
crucial to critically evaluate
Responses from models:
<Model 1 Response>>
<Model 2 Response>
Model N Response>
B GATED MIXTURE OF AGENTS EXPERIMENT
B. TASKS AND DATASETS
We considered the following tasks and their associated benchmarks for the EigenDivergence filtering experiment:
Arithmetic reasoning: We used GSM8K (Cobbe et al., 2021), a set of linguistically di- verse grade school math word problems, and MATH (Hendrycks et al., 2021), a set of challenging competition mathematics problems which require step-by-step reasoning to solve.
Commonsense reasoning: For these tasks, we used CommonsenseQQ (CSQA) (Talmor et al., 2018), popular commonsense question answering dataset constructed by extracting target concepts from text that have the same semantic relation to a single source concept; questions then aim to discriminate between the target concepts. Correctly solving the ques- tions often requires prior knowledge. We also used the A12 Reasoning Challenge (Clark et al., 2018), a dataset of natural grade-school science questions authored for human tests.
Instruction following: We considered AlpacaEval 2.0 (Dubois et al., 2024), which uses reference- free method (GPT-4) to evaluate the quality of outputs by how aligned they are with human preferences. The benchmark calculates conth-controlled win-rate which ex- plicitly accounts for the confounding whereby models prefer longer answers. This bench- mark aligns strongly with human preference and has a Spearman correlation of 0.98 with LMSYS' Chatbot Arena. We also considered MT-Bench (Zheng et al., 2024), multi-turn question set which uses LLM judges to evaluate answers, and which demonstrates strong agreement with human preferences.
B.2 LANGUAGE MODEL PROMPTING
We perform all arithmetic and commonsense reasoning tasks in the few-shot setting. For arithmetic reasoning, generate 5-shot CoT prompt by randomly selecting samples from the training set of each benchmark for each question. For the CSQA task, we used the same prompt as in (Wang et al., 2022: Wei et al., 2022). For the A12 Reasoning Challenge tasks, we manually construct 4-ohot CoT prompt which is shown in Listing 1.
We similar prompt in Wang et al. (2024) for the aggregation and synthesis function with two notable exceptions: 1) The addition of an instruction to focus on the 'latest' user query to account for tasks in the few-shot setting, and 2) An instruction to provide single answer to query if this
18

---

Published as a conference paper at ICLR 2025
is explicitly requested by the user, to account for tasks where there is single correct answer. The prompt reproduced in Listing 2.
B.3 LANGUAGE AND EMBEDDING MODELS
We used the same LLMs in the baseline Mixture of Agents ensemble from Wang et al. (2024). Namely, we use Aaama---7BB-nntruue (Touvron et al., 2023), Qwenl.5-72B-CCT (Bai et al., 2023), Wwenl.5-110B-Cha. (Bai et al., 2023), Mixtral-8x225 8x222-v0.. (Jiang al., 2024), FizardLM-8x22D (Xu et al., 2023), and dbrx-instruct (The Mosaic Research Team, 2024). We used OpenAl's text- embedding-3-sminn model (OpenAI, 2024a) the sentence-embedding function e(-) (described in more detail in Sec. 4.1).
B.4 SAMPLING SCHEME
Wang et al. (2022) had demonstrated that self-consistency decoding is robust to both sampling strat- egy and model scale. In view of this, to generate LLM outputs we used temperature sampling (T) (Ackley et al., 1985; Ficler & Goldberg, 2017), with T = 0.7 for all LLMs across all experiments. We do not consider top-k truncation (Radford et al., 2019; Fan et al., 2018; Holtzman et al., 2018) nucleus sampling (Holtzman al., 2019). This aligns sampling scheme with prior work LLM ensembling (Wang et al., 2024; 2022).
B.5 GMoAs AND MOEs
The mixture of experts (MoE) approach leverages the relative expertise of specialised modules inference time to improve performance over monolithic decoder-only autoregressive transformers (Shazeer et al., 2017). An MoE is usually stack of layers where each layer contains set of N experts (E()) alongside a gating network G(-) and a residual connection. An MoE produces an output for layer
N F++1 Σ Gi.n( + x4. 1=1
(13)
As noted by Wang et al. (2024), the MoA extends the MoE approach to the model level, operating the prompt interface for multiple 'expert' LLMs without modifying internal activations. However, the MoA approach consolidates the roles of the gating and expert functions for each LLM. In the Gated Mixture of Agents experiment (Sec. 4.1), we explicitly factorize these functions once more as
U1++1 Σ (., + Ji,
(14)
where =, (Z;), and thus we explicitly gate outputs depending on how divergent they are relative to other LLM outputs. In a later ablation experiment (Sec. 4.3) we reverse the gating mechanism and remove the most consistent outputs, rather than the most divergent ones.
C MIXTURE OPTIMIZATION EXPERIMENT
In the mixture optimization experiment, the setup is the same for the Gated Mixture of Agents experiment (Appendix B); namely the same tasks and datasets, language model prompting strategies, language and embedding models, and sampling scheme are used. Each baseline heterogeneous mixture agents optimized against one of: 1) Arithmetic reasoning; 2) Commonsense reasoning: 3) Instruction following. Optimizations are done in accordance with Alg. 1, which has the following stopping criteria: 1) If no improvement in target benchmark performance (past
camuulll-selectte threshold) in three iterations, halt; 2) If both adding and removing model to the mixture degrades performance, halt; 3) the mixture contains a ensembe,, and this represents an improvement in performance heterogeneous ensemble, halt.
19

---

Published as a conference paper at ICLR 2025
Data: Initial mixture of models {m,, with usage proportions Uo(m))
Result:
Optimized mixture composition
Initialize mixture composition with models {m²} and usage proportions Up( (m=);
Run the ensemble with current mixture composition and obtain performance Po:
Measure usage proportions Um(m)
Run the ensemble again with the same mixture composition and obtain performance Pi;
Measure usage proportions U1 (m²);
Set iteration counter i <- 1;
repeat
Compute performance difference: AP = P:-1; foreach model my do Compute usage difference: AU (mm) Uz(ma)) U2-1 (m-)(
end
Compute total usage change: T Σx [m((m))] foreach model my do
Compute model delta: (M(m)) = (m(mm)) end
x AP:
T
Identify model mmin with the lowest AM (m,);
Identify model mmax with the highest A AM (mm): Replace model Main with a clone of model M
Run the ensemble with updated mixture composition and obtain performance P,+1: Measure usage proportions U++1 (m&);
Set i + i +
until stopping criterion is met;
Algorithm 1: Mixture optimization experiment algorithm.
D ABLATION STUDY FOR POST-PROCESSING FUNCTIONS IN ENSEMBLES
In this section we describe the experimental setup for ablation study whereby we assess three different post-processing functions for LLM ensembles. We assess the 'aggregation and synthesis' function proposed by Wang et al. (2024), simple LLM-rrnking (Wang al., 2022; 2024), and 'uni- versal self-consistency' recently proposed by Chen et al. (2024b). Descriptions of these functions follows:
Aggregate and synthesize: This function allows the final LLM in the ensemble to look through all information in the previous layer's outputs, perform a critical appraisal, in- cluding assessing for bias or incorrect results, and produce novel output, based on the previous outputs.
LLM-ranking: Here, the final LLM is instructed to look through all information the previ- ous layer's outputs, and to simply select the answer it believes to be the best. The function therefore outputs single, previously generated response that the final LLM believes rep- resents the highest quality response.
Universal self-consistency (USC): A recently proposed (Chen et al., 2024b) generaliza- tion of self-consistency decoding (Wang et al., 2022). Here, the final LLM looks through all information in the previous layer's outputs, and selects the most highly repeating facts of information (thus selects the most consistent response). This is distinguished from aggrega- tion and synthesis by the fact that USC does not perform any sort of critical appraisal/error assessment.
Listings 3, 4, and 5 illustrate the aggregate and synthesize, LLM-ranking, and USC functions, re- spectively. We assess already 'optimized" LLM ensembles tasks for which they already performant. For each run, we only replace the head function. Qwen1. 5- 72B used as the final L.LM in the ensemble across all models (Qwen Team, 2024).
20

---

Published as a conference paper at ICLR 2025
Listing 3 Aggregation and synthesis function
'You have been provided with set of responses from various open-source models to the latest user query. Your task is synthesize these responses into single, high-quality response in order to answer the final question given to you note that, in order to help you understand the task, the user may first show some example question and answer pairs. It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well- structured, coherent, and adheres to the highest standards of accuracy and reliability. If you asked provide single response.
question, do not include multiple answers in your
Responses from the models:
Listing 4 LLM ranking function
'You have been provided with set of responses from various open-source models to the latest user query. Your task evaluate the responses and select the best answer the final question given by the user note that, in order to help you understand the nature of the task, the user may first show you some example question and answer pairs. Your response should simply replicate the answer which you believe represents the best response to the final user query, with no further additions of any kind. This should be the only output you produce. If you are asked to provide single answer to question, do not include multiple answers in your response. Responses from the models:
Listing Universal self-consistency
'You have been provided with set of responses from various open-source models to the lat- est user query. Your task is to evaluate the responses and select the most consistent response based majority consensus the final question given by the user. If there reponses, respond by saying 'I cannot answer this query as there is no consistent information in the outputs have received" note that, order help you understand the nature of the task, the user may first show you some example question and answer pairs, but you should focus on selecting the most consistent answer based on majority consensus for the final user query. Responses from the models:
consistent
21

---

Published as a conference paper at ICLR 2025
E DYNAMIC MIXTURE OF AGENTS EXPERIMENT
In this section we describe the experimental setup for the Dynamic Mixture of Agents (DMoA) experiment on the Big Bench Hard (BBH) benchmark. BBH is a subset of 23 challenging tasks from the BigBench dataset (Srivastava et al., 2022). The tasks include algorithmic reasoning, multi- step arithmetic, and world knowledge (Suzgun al., 2022; Hugging Face, 2023).
E.I CORRECTED BBH
The BBH repository on github (Suzgun & contributors, 2023) contains number of formatting errors across some tasks. We concatenate all questions across the 23 subsets, and correct for any errors in the expected output format. Listing shows the expected output format for each subset. Listing demonstrates manual corrections of previously incorrectly formatted questions.
Listing 6 BBH expected output format
The expected output format for each subset is shown below:
boolean boolean.exprestteee expressions: Binary (True/False)
causal judgemend::mm Binary (Yes/No)
date understanding: Multiple-choice (e.g., (A), etc.)
disambiguation.cm qa: Multiple-choice (e-g.. (A), (B), etc.)
languagess Bracket sequences (e.g., [.]
formal fallacies: Binary (valid/invalid)
geometric shapes: Multiple-choice (e.g.. (A), (B), etc.)
hyperbaton: Multiple-choice (e.g.. (B), etc.)
logical.deduction dwwuctioducnnnnco ditte Multiple-choice (e.g., (A), (B), etc.)
logical deduction seven objects: Multiple-choice (e.g., (A), (B), etc.)
oegical.eeduction three objects: Multiple-choice (e.g., (A), (B), etc.) www.ttttrreeeeeeee
movie movie.recommendatie Multiple-choice (e.g., (A), (B), etc.)
multistep arithmetic two: Numeric (integers or real numbers)
navigate: Binary (Yes/No)
object.counting: Numeric (integers or real numbers)
penguins in table: Multiple-choice (e.g.. (A). (B), etc.)
reasoning about colored objects: Multiple-choice (e.g., (A), (B), etc.)
ruin_names Multiple-choice (e.g.. (A), (B), etc.)
wwwtttttttalattneennrer error detection: Multiple-choice (e.g., (A), (B), etc.)
snarks: Multiple-choice (e.g., (A), (B),
pprtts.nderstandiig Binary (Yes/No)
cemporal.sequencee.
sequences: Multiple-choice (e.g., (A), (B), etc.)
tracking shuffled objects five objects: Multiple-choice (e.g., (A), (B), etc.)
Multiple-choice (e.g.. (A), (B), etc.)
Brackingshhufleee cowuufleddobjecs..nee thajjecssttrre.obeectt. Multiple-choice (c.g., (A), (B), etc.)
web of lies: Binary (Yes/No)
wordino Free-text (open-ended answers)
22

---

Published as a conference paper at ICLR 2025
Listing 7 Corrections of previously incorrectly formatted questions in BBH
The corrections are shown in the following format: question id: question, corrected answers, target answer:
4182: Which of the following humorous edit of this artist or movie name: 'earth, wind, & fire"?
Options:
(A) eareth, wind, & fire
(B) earth, bind, fire
(C) earthm, wind, & fire (D) dearth, wind, & fire
Answer: (D)
2850: Find movie similar to Minority Report, Shrek, Catch Me If You Can, Aladdin:
Options:
(A) Monsters, Inc
(B) Children the Night
(C) The Incredible Shrinking Man
(D) Town Country
Answer: (A)
4227: Which of the following is humorous edit of this artist or movie name: 'rita,
sue and bob too"?
Options:
(A) rita, sue and bob too
(B) rita, sue and bob poo
(C) rita, sue and box too
(D) ritay sue and bob too
Answer: (B)
23

---

Published as a conference paper at ICLR 2025
E.2 DMOA PRE-PROCESIING FUNCTION
As described in Sec. 4.4, we require a pre-processing function which takes user query its argu- ment and identifies both a set of skills required to produce a high-quality answer to the query, and prediction of which model(s) are likely to perform well given the prerequisite skills required. An en- semble is then constructed and executed at test-time to answer the query. We use openai/ /gpttttt- 40 (OpenAI, 2024c) as the pre-processing function. Listing shows the prompt for this task. Listing shows example output for a randomly-selecter prompt.
Listing 8 Pre-processing function for the Dynamic Mixture of Agents
<general information>
You will be shown performance information on a number of benchrmarks for the following reference models: Srinsof/WWizrdd..M M-2.8x22B, QweniQwen 1.511 Qwen/Qwen2-773 2e--2-22--Inseeeee Instrnet, meta-Ilama/t 1ama-3-70b-chat-- hf, MataalaiMMittl--k22----nnneee I, datibricks@box-iistuuct
</general Information>
<benchmark desoription lons>
IFEval: IFEval is a dataset designed to test model's ability to follow explicit instructions, such as "include keyword on mandll's adherence to formatting instructions allowing for the use of strict and rigorous metrics.
x" or format y
content generated.
MATH: MATH compilation of high-school level competition problems gathered from several sources, formatted consistently using Latex for equations and Asymptote for figures. Generations must fit very specitic output format. We keep only level 5 MATH questions and call it MAT Lvl
GPQA (Graduate-Levv Gongle-Proof Q&& Benchmark): GPQA a highly challenging knowledge dataset with questions crafted by PhD-level domain experts fields like biology. physics, and chemistry. These questions designed to be difficult for laypersons but relatively easy for experts. The dataset has undergone multiple rounds of validation to ensure difficully
accuracy.
MuSR (Multistep Soft Reasoning): MuSR is dataset consisting algorithmically generated complex problems, each around 1,000 words in length. The problems include murder mysteries, object placement questions, and team optimizations. Solving these problems requires models integrate reasoning with long-range context parsing.
MMLU-PRO (Masive Multitusk Language Understanding Professional) MMLU-Pr Pro is refined version of the MML.I J/ dataset, which has heen a standard for multiple-choiie knowledge nssessment. Recent research identified issues with the original MMLU, such noisy data (some unanswerable questions) and decreasing difficulty due advances model capabilities and increased data contamination. MMLU-Pro addresses those issucs by presenting models with 10 choices instead requiring reasoning on more questions, undergoing expert review to reduce noise. < /bbecchack descriptions
<model benchmark performance>
IFEval
GPQA
MUSK
MNLU-PRO
Qwen//Qwenl..a www.e//w--e..-----.... 1-5--10B-hhat
59.39
0
12.19
16.29
42.50
Owen/Qwen2-722) -2ntstttttt
38.24
29.15
19.24
19.73
NA
Microsoot/Wizard.M333 M-2-8x2BB
52.72
22.28
17.56
14.54
39.96
meta-llama//lanne Llama-3-77m---a
80.99
23.34
4.92
10.92
46.74
mistralai/Mixtrall 8x22B Instruct-vO.1
datahricks/dbrs. innssrrtcc
54.16
6.87
12.19
12.20
29.81
</model benchnark performance>
kyour taak>
I'm going to show you a prompt (user query). Note that to help you better understand the nature of the task, may be given a chain thought (3-shot CoT) prompt, and you'll be expected to focus on the final query. I'd like you to perform the Tollowing functions:
First, identify list of skills you think are relevant for producing a high-quality response to this query. Second. predict which models given <general in.formati ion> and <model benchmerk performance are best suited to answer the required query. Finally, produce
choices create an ensemble models. This list include repenting model choices.
list should be of the format:
"model, name Lmodel_namae 2.., -.., moudel_name_s"
punctuation or oulput musi produced. The names must match the entire string given in the general information section exactly. For instance, given the MATH Lvl 5 scores, supposing a task was mathematical in nature, then you may wish to create list sach as:
should no spuces between names,
"Qwen/Qwen2-72" 72B Instruct Instruct.Qeen/Qweerr Qwen/Qwen2 72B Instruct, Owen//onQeee//ee222222 72B Instruct, QwerncttQeee/Qeee222... 72B.
Instruct.meta-liee Ilam/ Mlama//lama-- 1-mma----7b--ae ciab-chtt-he faatthffmm Hama/ Jama. 7.0h--hthhht
This models, and it contains two models which and the other twice. This is just an illustrative example. I'm going to show the actual promptiquery, and would like you to carry out task. </your task>
highest MATHI performance. repeated times,
24

---

Published as a conference paper at ICLR 2025
Listing 9 Example outputs for LLM-based pre-processing function
Prompt:
How would a typical person answer each of the following questions about causation? Eugene and Tina were a young married couple who lived in the country. Both were partially paralyzed and confined to wheelchairs. They had met four years before when Tina counsellor with the Canadian Paraplegic Association, had fallen in love, and were married one year later. On this particular evening, Eugene had phoned to request cab to take them downtown. When the taxi driver arrived, Eugene and Tina were waiting by the street. On seeing that they were both in wheelchairs, the taxi driver refused their fare because he thought would be too crowded in the taxi with both of them and the wheelchairs. So the taxi driver headed back downtown without them. Because there was no time to call another cab, Eugene and Tina took Tina's car, which was equipped with special hand controls. In order to get downtown from their house, they had travel across bridge over Rupert River. severe storm the night before had weakened the structure of the bridge. About 5 minutes before Eugene and Tina reached it, section of the bridge collapsed. The taxi driver had reached the bridge shortly before them, and had driven off the collapsed bridge. He barely managed to escape from his taxi before sank in the river. In the dark, Eugene and Tina drove off the collapsed bridge and their plummeted into the river below. They both drowned. Their bodies were retrieved from the car the next morning. Did the taxi driver's s refusal take Fugene and Tina cause their death? Options:
Yes
No
Skills relevant to the query:
Causal reasoning: The query involves determining the chain of events and their contribution to the final outcome (the deaths Eugene and Tina).
Ethical and moral reasoning: The scenario requires consideration of responsi- bility and potential negligence.
Legal reasoning: The question is somewhat legalistic, as it asks about causation and fault, which are common in legal contexts.
Lonn-context understanding: The model must handle lengthy narrative and parse multiple factors the scenario.
Commonsense reasoning: Determining how typical person would respond re- quires an understanding everyday logic and reasoning.
Best-suited models based the benchmarks:
Multistep soft reasoning (MuSR) and Instruction following (IFEval) are key benchmarks for this type of task. These scores measure reasoning through com- plex scenarios and adherence to following instructions.
Strong IFEval score (80.99) and moderate scores in reasoning based tasks (MuSR: 10.92).
Strong IFEval (71.84) and reasonable
MuSR performance (13.49).
Micrrsoft/WizzrrdLMM-3 M-2-8x22B and Qwen/Qwen2-722B: - also have de- cent MuSR and IFEval scores, making them good candidates for causal reasoning and custructiionfollowiin tasks.
Ensemble of models:
htt. law.llllaa-------a---taa hit.aaaaiiaamiiiaaaaaaaiiittaa 8x22B-Instruct-vO. Matiraaaaaiiiitaaa----------eee oato://snnrmttttmmmmmmoooooaaaaa mmicrosoft/Wizrrdl.MM.
'meta-llama/Lllmaa www.aaaa------aaaa-a-aaaaaaa meta-
8x22B, (et.e//eee/ee2222--mmme
25

---

Published as a conference paper at ICLR 2025
Table 3: Performance of the DMoA across instruction following, arithmetic reasoning, and com- monsense reasoning benchmarks. The DMoA outperforms other models and ensembling strategies across the majority of the benchmarks. denotes our replication of results.
Instruction Following
Arithmetic Reasoning Commonsense Reasoning
Model
AlpacaEval 2.0 MT-Bench GSM8K
MATH
CSQA ARC-C
ARC-E
DMoA
63.21
9.19
96.67
71.23
87.51
92.50
94.47
GMoA
58.66
8.97
94.23
56.35
85.20
92.32
93.75
59.50
9.19
93.87
55.22
84.32
91.85
94.31
Llama-3-708
34.4
8.8
93.0
50.4
83.8
90.51
94.1
Qwen-1_5-110R
43.9
8.9
85.4
49.6
82.1
69.6
93.91
Qwen-1_5-725
36.6
8.4
79.5
34.1
83.2
65.9
92.71
WizardL
8.8
81.6
22.7
69.01
62.5
90.11
Mixtral 8x22B
30,9
8.8
83.7
41.7
81.71
70.7
91.81
DBRX-Instruc
25.4
72.8
32.5
82.21
68.9
89.71
GPT-4 Omni (05/13)
57.5
9.19
94.1
61.2
88.6'
94.6'
94.31
E.3 BBH EVALUATION
Whilst the tasks in BBH all use objective metrics, we found that regex patterns can be brittle to conrectly extracting the answer produced by model. As result, we found that BBH performance can underestimated by using regex-bases answer-extraction approaches in a pilot study. Based the observation by Chen al. (2024b) that LLMs more easily compute consistency between responses that judging the correct answer to question directly, we therefore use an LLM-as-a-judge approach evaluate BBH performance. Namely, 2024b) to evaluate the model responses. We first extract the answer from the proposed output and then perform entailment verification (Sanyal et al., 2024b) between the extracted answer and the ground-true -tuutt answer. Listing 10 illustrates the prompt for this.
openai /gpt 5-turbo (OpenAl,
Listing 10 Answer extraction and entailment approach for BBH
Query: (example('inpur")
Prediction: {examplel ]
Ground Truth:
"arget'"
Does the prediction match the provided ground truth? First extract the final answer from the prediction, then, matches the provided ground truth, and then answer with 'yes' or 'no".
We additionally note the large discordance between quoted BBH scores. For instance, the Qwen1 101115-11100 5-110B model is cited as having a BBH score of 74.8 (Qwen Team, 2024). However on the HuggingFace (HF) open LLM leaderboard, the same model has score of 44.28 (Open LLM Leaderboard Team, 2024). We believe this partially due to HF quoting normalized accuracy, whilst several labs quote raw accuracy. To ensure fair comparison, and ensembles using our BBH evaluation pipeline, and quote raw accuracy as well as normalized accuracy. For all models/mixtures, also note the best and worst performing subsets.
therefore assess all models
E.4 ADDITIONAL RESULTS
We additionally report the perform of DMoA the instruction following, arithmetic reasoning, and commonsense reasoning benchmarks used for the gated mixture of agents and mixture optimization experiments. As can be in Table. 3, the DMoA framework outperforms other models and ensembling strategies across the majority of the benchmarks, further supporting our results in the BBH experiment.
26

---

Published as a conference paper at ICLR 2025
F CONCORDANCE BETWEEN INTERNAL AND PROJECTED EMBEDDINGS
This section describes an experiment to demonstrate the relationship between eignscores calculated from the internal token embedding space of LLM,, and eigenscores calculated from projecting natural language outputs into shared sentence embedding space.
F.1 EXPERIMENTAL SETUP
For the internal Eigenscore we use the final token embedding of the middle layer of 1 1lama 1omm--3--a 3-8B (Touvron et al., 2023) as per Chen et al. (2024a). We sample 100 datapoints from AlpacaEval 2.0 and for each question sample generations with a temperature length and repitition penalty parameters are both set to 1. For the 'external' eigenscore, project natural language outputs from each generation into a semantic embedding space using one of three models: OpenAl's -ext-embeddiin--3--
0.3 and top-k of 50. The
and text wet---mmeediingg -dda aada-aaa00 (OpenAI, 2024) - we then perform the Eigenscore calculation on these embeddings and calculate correlations for scores derived from each set of "external" embed- dings with Llama's internal embeddings.
-ext-embedding--3--
F.2 RESULTS
Results are illustrated in Fig. 5. As can be seen, there is strong positive relation- ship between Eignscores calculated from the internal token embedding space of 1lama- 3-88 and those calculated from first projecting the natural language output to a sentence embedding space using -oxt--embeddin-3--- text wwweeeeeeeeeaaaeeeeeeeeeeee aaap-a-a--0222 The Pearson's 's correlation coefficients of were 0.731, 0.779, and 0.781 (3 s.f.), respectively, demonstrating minimal variance and robust concordance for different text embeddings models.
text-embeddinng- 3 or
G ON THE RELATIONSHIP BETWEEN EIGENDIVERGENCE AND INFORMATION GAIN
In machine learning, information gain is often used to refer to the mutual information I(.) between two random variables (X, Y) as
I(X;Y) Dri. (Px,y @ Pv);
(15)
which can equivalently be expressed difference of entropies as follows:
((: / (x,((z.)) log
Px,y(2,))
dxdy
(x(x)
[ 1.1/ (x.
(x,
dxdy + I / (x, (2, log Pr dady
Px(x)
-fo (x.y(z,V)
(xx((x)))
dxdy / / Px, v) log Py (y)
Px(x)
|P((x)) (sg Prix-=i log Prix=x(y)dy dr
-S (X.xxlo. log (r
(x( x)dr /, Pr( log Pr(v)dy =-H(Y + H(Y)
= -H(Y) - H(Y))),
Where H(Y) is the marginal entropy of random variable Y and H({ is the conditional entropy of given X. Unless (X, Y) are independent, then by learning or assuming the information in X. measure the change in uncertainty about Y, and therefore information gain measures
27

---

Published as a conference paper at ICLR 2025
External
Casa Paints
Regresiee Line
5.0
5.0
7.5
(a) Correlation text -nnng- 11.
Correiation betwear intoral and Extemal Bienscores
Cats Forts
Rog exaton Lin
-12
(b) Correlation for text -ottte--bbeddddg------ee
External
Casa Regresiin Line
-4-
Eitoral
-25
-25
3.0
2.5
5.0
7.5
200
(c) Correlation text--mbbeddng---a----a
Figure Correlation between internal token embeddings from 1lama-3---- 8B and external se- mantic embeddings from text text-eebbedding-2 3-smal,, Lext-enbeediing-3-L1
and
The Pearson's correlation coefficients of are 0.730, 0.779, and 0.781, respectively. These coefficients indicate strong positive relationship. demonstrating the consistency between Eigenscore calculations from internal and projected embedding spaces.
text sett-embeddinggaaa..
the reduction in entropy when feature is known. As be in Eq. 9, the EigenDivergence is proportional to the difference in differential entropies in sentence embedding space when comparing all L.LM outputs subset where output has been removed. Interpreting F., -ili::: (-=iZiilze
28

---

Published as a conference paper at ICLR 2025
analogously to XX) the entropy of the ensemble given that exclude the i-th output demon- strates that the EigenDivergence effectively captures the change in uncertainty about the ensemble's s prediction space when excluding specific model output.
H INVESTIGATION OF THE EIGENDIVERGENCE (ED) SCORE
In this Appendix we further investigate the potential impact of using the FigenDivergence score as a filtering mechanism for GMoA. Namely, we want to assess the correlation between EigenDiver- gence and accuracy of individual queries prior to the filtering mechanism in order to establish the effectiveness at removing incorrect queries from mixtures.
Our experimental design is as follows; firstly, construct a single layer GMoA using the Llama-3- 70B- Instruct,, Qwen2-72B-Innn Instruct,, Nextral-8222-vvee els, using Qwen2-72B-Inntrruct and ARC-C benchmarks respectively, and for each question, we capture the individual responses for a given layer prior aggregation and calculate and store the EigenDivergence scores for each response.
BerardLM-8x2222 and dbrx-instruct mod- the head model. We sub-sample 200 questions from the GSM8K
H.1 RESULTS
Difference in distributions of FigenDivergences categorised by correct and incorrect answers across benchmarks, as well as considering correct answers for a given layer with at least one incorrect answer, are shown in Fig.
2400 individual layer answers were assessed over the GSM8K and ARC-C benchmarks. For the GSM8K sub-sample, 1119 (93%) answers were correct, whilst 81 (6.7%) answers were incorrect. Over the 200 questions, 148 (74%) had 6 correct answers, 39 had (19.5%) 5 correct answers, (2.0%) had 4 correct answers. (2.5%) had 3 correct answers, (1.0%) had 2 correct answers, (0.05%) had correct answer and 1 (0.05%) had correct answers. We find that correct answers had a mean (median) ED score of -0.269 (-0.307) and standard deviation of 0.263. The inter- quartile range for correct answers was found to be [-0.439, -0.166]. For incorrect answers, the mean (median) ED score was found to be -0.065 (-0.168) with a standard deviation of 0.317. The inter-quartile range for incorrect answers was found be [-0.288, 0.082]. Considering only correct questions with at least one incorrect answer in given GMoA layer, we find that correct answers had mean (median) ED score of -0.314 (-0.332) and standard deviation of 0.210. The inter-quartile range for correct answers was found to be [-0.447, -0.209].
For the ARC-C sub-sample, 1069 (89.15 %) individual layer answers were correct, whilst 131 (10.9%) were incorrect. Over the 200 questions, 134 (7.0%) had 6 correct answers, 39 (19.5%) had 5 correct answers, 12 (6.0%) had 4 correct answers, (2.5%) had correct answers, 2 (1.0%) had correct answers, 3 (1.5%) had 1 correct answers, and 5 (2.5%) had no correct answers. We find that correct answers had a mean (median) ED score of -0.238 (-0.267) and a standard deviation of 0.285. The inter-quartile range for correct answers was found to be [-0.422, -0.101]. For incorrect answers, the mean (median) ED score was found to be -0.152 (-0.185) with a standard deviation of 0.263. The inter-quartile range for incorrect answers was found to be [-0.315, -0.009]. Considering only correct questions with at least one incorrect answer in given GMoA layer, we find that correct answers had mean (median) ED score of -0.251 (-0.274) and standard deviation of 0.266. The inter-quartile range for correct answers was found to be [-0.414, -0.1
Comparatively, these results show that ED scores of individually correct queries are more negative on average than individually incorrect queries. Additionally. in both datasets, we find that correct answers in layers with at least one incorrect query are more negative on average. This suggests that filtering based on ED scores is more likely to lead to removing diverse and potentially incorrect answers for close-ended tasks such as arithmetic and commonsense reasoning.
H.2 QUALITATIVE ANALYSIS
Monthly cell phone costs We demonstrate failure mode of the ED score with the follow- ing question from the GSM8K dataset: 'Shiela bought five cell phones for $150 each for a 3-month installment. A 2% interest will be charged for each unit. How much will Shiela
29

---

Published as a conference paper at ICLR 2025
GSM8K Full (p<0.001)
GSM8K Partial
(p<0.001)
EigenDivergence Score
2
**
***
0
-1
ARCC Full (p<0.001)
ARCC Partial (p<0.001)
EigenDivergence Score
2
***
1
0
-1
False
True
False
True Correct Answer
Correct Answer
Figure 6: Distribution of EigenDivergence scores with answer correctness across GSMSK and ARCC subsets. EigenDivergence scores are more positive (i.e. increase semantic diversity if con- sidered in the mixture) for incorrect answers, and conversely more negative (i.e increase semantic consistency in the mixture) when correct. Across all comparisons, Wilcoxon rank-sum tests revealed statistically significant (p<0.001) distributions when comparing correct versus incorrect answers, marked with (***). In the second column, compare correct answers from questiosn with least one incorrect answer in the layer; in these cases correct answers are more negative.
pay each month for 3 months?'. As can be in Table. abrx-instrun and Baeal--x22B--nntrre .1 show a remarkably similar semantic template in their rea- soning:
1. 'First, let's calculate the total cost.
2. 'Next, let's calculate the [total] interest..."
3. 'Finally, let's calculate the [monthly payment/ total amount] Sheila...'.
However, both make errors in their calculations and reach different wrong conclu- sions (of $755 and $265, respectively). Swen1.5-110B-C0
Meanwhile, take slightly different approaches from another deduce the correct answer. 1aama-3-70b-che -hf opts to calculate the total interest in one step before calculat- ing the total cost and then per-monthly cost, whilst Qwen1 5-1 OB-Chat correctly calculates
and
30

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of model responses for calculating cell phone installment payments.
ED Score
Model
Response
-0.255
metaallama/LLammm
Let's break it down step by step!
Shiela bought 5 cell phones for $150 each, the total cost is: $150 $750
2% interest will be charged for each unit, which means the total interest is: $750 2% S750 x 0.02 = $15
The total amount Shiela needs pay is: $750 + $15 $765 Since it's 3-month installment, Shiela will pay: S765 3 $255 per month for months.
## 255
-0.125
Qwen/Qwenl.5tt 1.5veen1..511 15-1110B-hhat
cell phones $1.50 each = 55<<<=555007050>>2>33333
Next, calculate the total interest for all cell phones. The interest rate 2% per unit, for all five units: 2% of $150 0.02 $150 $3 per cell phone
Total interest for cell phones 5 $< <5<3=1>>>>>>>>>
Total amount to be paid including interest = Cost of cell phones
Total interest = $750 + $15 $<<<70++5=765>>
This total amount is spread over months, the monthly payment will be:
S765 3 months $< <765/322555>>>
-0.376
databricks/dbrx instruct
First, let's calculate the total cost the phones without inter- est: cell phones $150 per cell phone = $770
Next, let's calculate the interest rate per month for one cell phone: 2% per months is approximately 0.67% per month (2/3). 0.67% of $150 is $1 (0.0067 150).
Now, let's calculate the monthly interest per cell phone
cell phones:
cell phones = $5
Finally, let's calculate the total amount Shicla will pay cach month: S750 (the cost of the cell phones) + $5 (the monthly interest) S755
So, Shiela will pay $755 each month
months.
-0.319
mistralai/Mixtral -x220
First, let's calculate the total cost of the five cell phones: cell phones $150 per cell phone = $$<<5$15000 1 Next, let's calculate the total interest for the 3-month installment period:
2% interest per month * $7.0 months 0.02 * $7.50
$<<00.0*750+==<5>>>>>>> >45
Now, let's add the total interest to the total cost to find out the total amount Shiela has to pay:
Total cost
Total
interest
$750
$45
S<<<55<45=795>>>>
Finally, let's calculate the monthly payment Shiela has to make for
3 months:
Total amount/ / months S795 S< 7953326665>>>>>>2 per month
Shiela will pay $265 each month months.
the interest per unit first, then the total additional costs accrued from the interest, and then the total overall cost and per-monthly cost. By filtering using semantic consistency score, we remove both correct answers (as the other models use syntactically near-identical reasoning flow), leading the final aggregator to also answer incorrectly by giving a final answer of 1251.11. This failure mode is likely limitation of all consistency scores which do not explicitly take account of the meaning of the answer being provided (Kadavath et al., 2022; Malinin & Gales, 2020; Lin et al., 2023; Ren al., 2022; Lin et al., 2022; Chen al., 2024a; Jesson et al., 2024; Kuhn et al., 2023).
31

---

Published as a conference paper at ICLR 2025
I COST ANALYSIS
We aim investigate the trade-offs between cost efficiency and performance in the challenging Big Bench Hard benchmark (Suzgun & contributors, 2023) based on our analysis in Sec. 4.4. Fig. illustrates performance against cost. Costs
calculated from the pricing information found the API providers' websites'. The pareto optimal front illustrates models/frameworks which balance cost the performance most effectively. Most individual models offer relatively low-cost options with moderate performance. Ensembles exhibit a clear boost in performance but at increased costs. Frontier models outperform open open-source models, however this also comes a cost premium. be seen, the DMoA offers well balanced option on the pareto front. In particular, it achieves similar performance to gpt-40-2025-05-13 whilst remaining cheaper to inference. Indeed, the pareto front produced by DMoA- Claude-3.5-Somme
can
DMoA/Sonnet strictly dominates gpt-4o-2025-05-13. Notably DMoA/Sonnet achieves the highest normalized accuracy but is also the most expensive, whilst the fully open-source DMoA provides more balanced trade-off with high performance at moderate cost.
Individual Models
DMoA/Sonnet
DMoAs
Claude 3.5.Sommet
90
gpt-4o & Claude 3.5 Sonnet
Pareto Optimal Front
85
gpt-4o-2024-05-13
B8H Normalize aized Accuracy (96
80
metalllammaalaaa-33//hhhhahaaaa
Micrrssft M:2-
mistralal wttpa//iiiaaaa22222lnssurrtttta
75
70
Qwenn/Owen15-110B:CC
Owwn//Owen1.5722-co2
65
datebricks/doxxinsttuutt
0
2
8
10
Cost per Million Takens (USD)
Figure 7: Normalized accuracy versus cost per million tokens for various Al models. The plot illus- trates the trade-off's between model performance (normalized accuracy) and operational cost, with individual models, ensemble models (DMoAs), and high-performance standalone models (GPT-4o and Claude 3.5 Sonnet) represented by distinct colors. The dashed line marks the Pareto optimal front, highlighting models that achieve the best balance between cost and accuracy without being strictly dominated by others.
TEST-TIME SCALING
We investigate whether the DMoA framework aligns with test-time scaling laws. Namely, we in- vestigate whether increasing the number of models per layer leads to consistent improvements in
For Together AI: http://www.oogehheeriiniiingggg For anthropic: http://www.nthppiccoommrrnggggg For
OpenAl: https//opeemaiccomm/iippmnngg
32

---

Published as a conference paper at ICLR 2025
Accuracy
Accuracy
90
Normalized Accuracy
Normalized Accuracy
BBH performance (%)
BBH performance (%)
88
90
86
88
84
82
2
4
10
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Number of Models/Layer
Number of layers
(a) Scaling the number of models per layer.
(b) Scaling the number of layers in the framework.
Effect of the number LLMs per layer in the Dynamic Mixture of Agents framework. As more models are introduced in a layer, Big Bench Hard benchmark per- formance improves. Right: The effect on the number of layers in the framework.
Figure 8: Test-time scaling. Left:
more layers are added, the DMoA demonstrates higher performance on the Big Bench Hard benchmark.
performance, and additionally whether scaling the number of layers in the framework is associated with improved performance.
The baseline framework contains 6 models per layer. To investigate fewer models in a given layer, keep the pre-processing function for the DMoA fixed (as per Appendix E.2), and randomly sub- select fewer models from the pool of models predicted to perform well for given query. Conversely, increase the number of models in layer, we sample additional models (with replacement) from the pool of models which are predicted to perform well for the current query. Fig. 8a illustrates that as layer dimension increased. BBH performance consistently improves.
We investigate scaling the number of layers in the framework whilst fixing the layer dimension (the number of models/layer) to As can be shown in Fig. 8b, adding more layers can also improve BBH performance. In our analysis, increasing the number of layers was associated with higher final performance than increasing the number of models for single layer.
33


Reviews:
[{"review_id": "UpmjFKXXmu", "reviewer": "Reviewer_bHjk", "summary": "The paper introduces a unified approach to explore the balance between diversity and consistency in LLM ensembles. It addresses the challenge of finding optimal ensembling methods, where traditional strategies like self-consistency decoding (focusing on output consistency) and Mixture of Agents (MoA, focusing on model diversity) each have limitations. The authors propose the **Dynamic Mixture of Agents (DMoA)**, a novel inference-time ensembling strategy that integrates insights on the interplay between diversity and consistency. \n\n1. The paper establishes a framework for examining trade-offs between task performance, diversity, and consistency within LLM ensembles.\n2. Introducing a consistency score enables selective filtering within model outputs, enhancing ensemble consistency.\n3. This method refines mixtures of agents by considering semantic and model-level adjustments, optimizing task performance.\n4. DMoA dynamically selects models based on task-specific needs, achieving state-of-the-art results on the Big Bench Hard benchmark, demonstrating effective balance across tasks.", "questions": "1. I am curious if the proposed methods, particularly DMoA and GMoA, align with test-time scaling laws. Specifically, does performance consistently improve as the number of models in the ensemble or the length of inference chains increase?\n2. In the experimental setup, the paper mentions constructing a \"MoA-Lite\" variant with a limited number of layers. What would happen if additional layers were added to MoA?\n3. In what specific scenarios are GMoA and DMoA each most effective? Can these methods be combined within a single framework to further improve performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses an intriguing and nuanced problem—how to balance consistency and diversity within LLM ensembles. This issue is fundamental, as different tasks often require prioritizing one over the other. \n2. The paper is well-structured and logically rigorous. The authors provide clear, precise definitions of both DMoA and GMoA frameworks, enabling readers to fully understand the distinctions and specific innovations in each method.\n3. The authors show that DMoA achieves state-of-the-art results on the Big Bench Hard (BBH) benchmark, indicating the framework's efficacy across various challenging tasks.", "weaknesses": "While the paper demonstrates the effectiveness of GMoA and DMoA in achieving high performance, it does not analyze the computational costs associated with these methods. Cost considerations are essential in ensemble approaches, particularly with LLMs, where scaling and inference-time model selection can be computationally intensive. For a comprehensive comparison, the authors should evaluate the total cost associated with GMoA and DMoA, including resource utilization during inference and model selection. This cost analysis would offer a more balanced view of the trade-offs between performance gains and computational expense, especially compared to other ensemble strategies and Chain-of-Thought (CoT) approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZUyCNKCIJ", "reviewer": "Reviewer_jGzM", "summary": "The paper proposes a new model ensemble strategy that builds on top of a Mixture of Agents called Dynamic Mixture of Agents (DMoA). First, the authors attempt to unify model ensemble methods with a formal definition. The overall hypothesis for all the experiments are that different tasks require different ensemble diversity/consistency. With that, they first do an experiment where they propose a divergent metrics EigenScore and EigenDivergence that measure how semantically consistent/inconsistent LLM outputs are to the overall semantics. The effect of this Gated MoA is nuanced. For the next experiment, they try to optimize MoA structure by proposing a model delta that correlates with task performances. They found that there are trade-offs between each task. If they optimize for one task, it would decrease for another task. Lastly they tried DMoA, which chose mixtures based on some criteria. They showed that DMoA outperforms MoA on BBH.", "questions": "- I wonder about the performance of DMoA on seven tasks used in the first two experiments. This should be a natural progression. It feels weird that for the first two, we are using the same set of seven and for third, we are using BBH.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors design and execute a pretty comprehensive analysis and extensions to the model ensembling method Mixture-of-Agents. The analysis and conclusions are useful because they give intuitions on what may work and may not work in a mixture of model frameworks. \n They first investigate whether reducing semantic inconsistency would help (it didn't, I wonder if the reverse would help though -- increase the diversity). Then a mixture optimization method is proposed and they found some useful insights, e.g., aggregate and synthesis works better than ranking and self-consistency. They then tested DMoA which dynamically uses different expertise models for the mixture.\n- It is quite important to dive deeper into what makes model collaboration work as model ensemble methods are a promising way to inference-time scaling.\n- The tasks are pretty comprehensive.", "weaknesses": "- The purpose of the first experiment is a bit questionable. The result from divergent filtering is nuanced, as it only slightly improves four out of five reasoning tasks, and decreases performances on three tasks. I understand you use this experiment's result as insights to build DMoA in section 4.4, but I question whether those insights are actually correlated. \n  - For insight 1 in section 4.4, divergent filtering is used as a supporting evidence as \"As shown in Sec. 4.1, LLM ensembles outperform individual models regardless of divergence filtering, across both open- and close-ended tasks.\" If the conclusion arrived \"regardless of divergence filtering\" then it doesn't really add much value to the argument.\n  - For insight 2, it says \"We found in Sec. 4.1 that removing information from an ensemble can improve task-specific performance.\" But it improves four and hurts three.\n  - For insight 3, it says \"Performance varies when semantic diversity is altered within a fixed ensemble (Sec. 4.1).\" This claim is not that clear and doesn't provide much insight since performance can vary if you change anything about the mixture. It would be more useful if more patterns could be discovered. \n- Similarly, I am concerned about the claim in section 4.3: \"two GMoA variants...one with the two most semantically divergent outputs removed (maximizing consistency), and one with the two most consistent outputs removed (maximizing diversity)...indicating that some semantic consistency is necessary for high-quality results, even in open-ended instruction-following queries.\" I don't think removing two most consistent outputs would maximize diversity. You could very well be removing two very similar outputs that are very distinctive from other outputs. This also has the unwanted effect of removing output that's more correct (since more models output it).\n  - This leads to insight 1 in section 4.4 that \"Maximizing semantic diversity harms performance (Sec. 4.3), indicating that some cross-validation between outputs is necessary for high-quality results, even in open-ended instruction-following tasks.\" This can be problematic since the premise is wrong.\n- The structure of the paper gives a strong vibe that the first two experiments are very disconnected from the third as neither divergent filtering (GMoA) nor the mixture optimization methods are used in the third. And the insights they provide are also very limited which I elaborated above. I would encourage maybe downplaying the portion of the first two experiments and focusing more on the third.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQ6rJv3osT", "reviewer": "Reviewer_EPHj", "summary": "The authors in this work look at Mixture of Agents (MoA) and propose:\n1) A framework that captures different variations of possible MoAs\n2) Divergence metric called `EigenDivergence` based on the hallucination detected in K sampled outputs with the additional proposition of using an external embedding instead of the model's internal embedding\n3) Propose an optimization algorithm based on incremental performance gains and usage\n4) Propose DMoAs that dynamically select the models\n\nResults of this work are shown as follows: 1) Gated MoA against standard MoA and other openly-available models, with GMoA only providing marginal improvement on settings with some models performing reasonably better while underperforming in settings where all models perform close to each other; 2) Show that mixtures do not translate the same across various tasks and 3) DMoAs perform better on the BBH", "questions": "1. Can the details of 3.3 and 4.4 be clarified? (see weaknesses for comments)\n2. Can Appendix F.2 be expanded to understand the behaviour of multiple sentence embeddings ? The 0.78 doesn't make much sense as an individual number - it is unclear if it a consequence of choice of embedding models / how much variance can exist, etc.\n3. Is there a reason why closed-source models weren't used since the eigendivergence doesn't require access to the weights ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Methodology, presentation, and results are good for the first 3 / 4 proposed contributions - I quite liked the idea of EigenDivergence and the analysis around it\n2. The takeaways on diversity and consistency are clear and well presented", "weaknesses": "1. Sections 3.3 and 4.4 were both unclear and unnecessary, in my opinion - I couldn't quite understand how these sections tie to the main point of this paper. The presentation and motivation around this subset of contribution requires significant re-write \n2. No critical focus on looking at the intermediate reasoning - while I get that semantic diversity was a focus, I would have liked to see a deeper look at how diverse semantic reasoning looked at with a few examples - were there instances were correct reasonings by multiple models were still judged to be semantically diverse, etc. I liked Appendix H - it was a good start, but a detailed, controlled experiment could've provided readers with much more about EigenDivergence\n\nNitpicks that can be easily fixed and does not affect the review/score:\n1. Typos in some parts - line 182 for example\n2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqEaIncgVk", "reviewer": "Reviewer_7TjC", "summary": "This paper proposes a unified framework to examine the trade-off between diversity and consistency to the final performance of model ensemble. The author proposes a dynamic mixture of agent approach to optimize the balance between task-specific capabilities, ultimately enhancing overall performance.", "questions": "1. In Section 4.4, Table 2 presents a comparison between DMoA/Sonnet and the Claude-3.5-Sonnet baseline. While DMoA/Sonnet demonstrates a marginal performance improvement (91.85 vs. 90.20 normalized accuracy on BBH), it is important to consider the associated computational costs. DMoA/Sonnet necessitates multiple inferences across diverse models and subsequent aggregations using Claude-3.5-Sonnet. This process incurs significantly higher expenses compared to the baseline due to the additional model inferences and the substantially longer input required for aggregation. Moreover, the efficacy of the MoA approach is heavily contingent upon the final aggregation model employed. When Claude-3.5-Sonnet is not utilized as the aggregation model in the DMoA approach, a substantial performance degradation is observed (90.20 vs. 83.63 normalized accuracy on BBH).\n2. What if testing DMoA on the seven benchmarks (AlpacaEval 2.0, MT-Bench, GSM8K, MATH, CSQA, ARC-C, ARC-E) in accordance with the experimental setups in Sections 4.1 and 4.2? \n3. Based on the experimental findings presented in Section 4.3, several key conclusions can be drawn regarding the impact of diversity and consistency on various cognitive abilities. Firstly, high levels of diversity appear to have a detrimental effect across all measured abilities. Secondly, strong consistency enhances reasoning and mathematical capabilities, but impairs the instruction-following proficiency. Lastly, when strong consistency is coupled with an appropriate degree of supplemental diversity, there is an observed improvement in instruction-following abilities, though this comes at the cost of diminished mathematical and reasoning skills. Compared to the discussion in current version, the above summary appears to more accurately reflect the core idea of this paper: balancing diversity and consistency for model ensemble across various tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of balancing the model diversity and output consistency in model ensemble is compelling and well-founded.\n2. The unified dynamic mixture of agent framework effectively addresses the trade-off between diversity and consistency across various tasks.", "weaknesses": "1. The proposed dynamic mixture of agent framework necessitates the individual search and optimization of MoA structures for each distinct task. The process relies on divergence filtering and mixture optimization, which is costly and requires additional task-specific datasets for evaluation.\n2. The application of EignScore originally proposed for hallucination detection in a single model presents inherent limitations when extended to an ensemble of multiple models. This is primarily due to the fact that the sentence-embedding spaces of various models are not aligned during their pre-training or fine-tuning phases. Consequently, these embeddings do not inhabit the same representational space, which poses challenges for direct comparison and aggregation across different models.\n3. The design of mixture optimization leads to a scenario where the final MoA model is absolutely dominated by a single model, as shown in Fig. 3, left. Since the search process for each run is determined by a greedy algorithm that replaces the model with the lowest delta to the one with highest delta.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)