Smoothie: Label Free Language Model Routing
Authors: Neel Guha
Mayee F Chen
Trevor Chow
Ishan S. Khare
Christopher Re
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
weak supervision
graphical models
routing
Primary area: natural_language_processing
TL;DR: We propose an algorithm for learning LLM routers without labeled data.

Abstract: Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. _routing_). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform _unsupervised_ routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown “true” outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.

SMOOTHIE: Label Free Language Model Routing
Neel Guha
Mayee F. Chen" Trevor Chow Ishan S. Khare Christopher Ré Stanford University, Department Computer Science
{nguha, mf chen, tmychow, iskhare, chrismre @attaford.ddeee
Abstract
Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models
-amann-- data, work explores whether it is possible to perform unsupervised routing. We propose SMOOTHIE, a weak supervision- inspired routing approach that requires labeled data. Given a set of outputs from different L.LMs, SMOOTHIE constructs latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample- deneeddd quality scores for each LLM, and route each sample the LLM with the highest corresponding score. We find that SMOOTHIE'S LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that SMOOTHIE outperforms baselines for routing by up to 10 points accuracy.
Introduction
Large language models (LL.Ms) are increasingly being deployed in multi-capability regimes where data inputs may span diverse range of tasks, each of which requires different capabilities [8]. For instance, an LLM-powered chatbot may be asked to write code, answer questions about different domains, summarize documents, perform extraction, and more [3, 8, 14, 30j. One challenge is that while engineers often have access to numerous pre-trained LLMs (i.e., through Huggingface or various APIs), they do not know which LLM is optimal for each possible user input [86]. Because the quality of generations can significantly vary across LLMs, choosing the right LLM for each input sample is important for ensuring high task performance [41].
Recent work has explored various ways utilize ensembles of pretrained L.MMs in multi-capability settings, by (1) collecting diverse pool of L.LMs and (2) identifying which LLM to route each sample to [55, 86]. However, the majority of existing approaches require labeled data; engineers typically either (1) train an auxiliary model using labeled data to rank or predict which LLM each sample should be routed to [41, 79], or (2) directly use labeled data to determine which LLM the best on average [86]. As result, engineers designing routing protocols face the practical difficulty of constructing labeled datasets.
Given a candidate pool of LLMs and an unlabeled test dataset, this paper explores how to best select LLM outputs for each sample an entirely unsupervised manner -withuut labeled data, or models trained on labeled data. To make progress in addressing this question, we face two technical challenges:
Unknown LLM quality: The first challenge is estimating the quality of each LLM. Access labeled data allows engineers to identify higher performing LLMs by measuring the accu-
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1) Embed test inputs, select x's nearest
Use subset estimate each LLM's score on x, (,)x), latent variable graphical model
Route to LLM with highest score (i.e., use its response as final output)
Owen
Unlabeled test input x
neighbors as subset
resporiow embedding
LLM (Gamma 78). Theodore Rocsevelt
Router: pick
"Who was the youngest person to become
highest of (x(x), 8:(x), 8s(x)
LLM (Vicuna 7B). "The bocome presidemt of the USA was Theodore Reasevelt, who wes iasugurated age of
Unkncwn
President of the United States?'
resporoe ald arteedngg
embedding
LLM (Owen 7B%: "Jocnn F. Kennady
embeddling
Figure 1: For a given input æ. SMOOTHIE estimates the quality of every LLM ensemble's generation. and uses this quality weight to route to single LLM.
racy/quality of LLM outputs. In this paper, study the question of how estimate quality without labeled validation data.
Sample-conditional generator performance: The second challenge is determining how select the best LLM for each individual test sample. LLM outputs can vary in quality over different samples, which could render population-Iever estimates of LLM quality misleading.
this work, we propose SMOOTILE, a method for routing samples to LLMs a label-free manner (Figure 1). Below, we describe how SMOOTHIE addresses the two challenges described above.
Quality estimation: Using the LLM outputs for each test sample as "voters," SMOOTHIE estimates the quality of each generator using methods from Weak Supervision (WS). Concretely, SMOOTHIE constructs a latent variable graphical model over observable LLM outputs and an unknown true output. By modeling the embedding vector difference between each LLM output and the true output as a multivariate Gaussian, we can derive a closed-fore estimator adapted from [85] for learning LLM quality scores efficiently.
We condition these quality estimates to be particular to a given test sample by only using the nearest neighbors of test sample in the training data as inputs to the estimator (i.e., kernel smoothing). We then route each test sample to the LLM with the highest quality score estimate on that sample. We call the version of SMOOTHIF that produces quality estimates using all available test data SMOOTHIF- BOLTTHI--GGLBBAL....... and we call the version that uses sample's nearest neighbors SMOOTHIE-LOCAL.
Conditioning:
We empirically evaluate SMOOTHE in three stages.
LLM selection: First, we assess SMOOTHHE- SMOOTIL--GOOAL''S ability to identify-from from an ensemble of mixed quality LLMs the optimal LLM for given task overall. On traditional generation tasks such as summarization, reading comprehension, and data-to-text generation, we find that SMOOTHIE- GLOBAL'S 's learned LLM quality-weightt correlate with actual LLM performance (p 0.72)), and on the AlpacaEval benchmark, SMOOTHIF--C GI.BAAL identifies the hest-performing instruction model 70% of the time [50]. The highest quality LLMM identified by SMOOTHIE- GL.OBAL-all computed without labeled data-can can beat random-selection by up to 15 points win-rate on AlpacaEval, and by up to points SQuAD.
Routing: Second, we study whether SIGOTTHI--LCAAE 's sample-conditions scoring mechanism allows roule samples in mixed-task datasets
performinn LLMs (i.e., the multi- capability regime). We find that SMOOTHEE-LOCAM can improve the quality of produced generations by up to 7 points accuracy over AOOOTTHEE-GLOBA.. and that SMOOTHIE-LOCAL outperforms baseline unsupervised routing methods by to 10 points accuracy and supervised routing methods by up to 5.0 points accuracy.
Prompt selection: Finally, we assess whether SMOOTHH''s quality-estimation mechanism can be applied select the optimal prompt template in a candidate pool while using fixed LLM. We find that SMOOTHE--LLOR points, allowing a 410M parameter model to match the performance of 6.9B parameter model.
can outperform other prompt selection approaches by up to 18

---

2 Related Work
We provide an abbreviated related work, with full treatment Appendix C.
Routing Routing has been classically utilized in Mfxture-of- models [25, 37, 42, 82), which involve jointly training
of models as well router. Recently, routing mechanisms have been used at inference time to decide which pre-trained LLM to use for a given sample [79]. Some approaches involve training an auxiliary model using labeled training data to either score or rank the performance of each LLM on each test sample [38, 74], Others do not involve training a model but instead use nearest neighbor methods, selecting the LLM that does the best test sample's labeled neighbors [48, 86). In contrast, SMOOTHIE does not require any labels.
Ensembling Ensembling is another way of utilizing pool of LLMs. Existing work has primarily focused on ensembling outputs for classification tasks [2. 68, 98]. Ensembling generative outputs typically requires training an auxiliary model [41], combining or switching among outputs when decoding [36, 83], or averaging in weight space [95].
Prompt selection In addition to selecting the best LLM for sample, prior works have studied how to select the best prompt or in-context examples. While the simplest approach is to use a held-out labeled approaches to selecting the best in-context examples [90], as well as approaches based on mutual information [89] and probability- although the latter two are limited to classification.
dataset [67], there are also retrieval-
measures [103],
Weak supervision SMOOTHIE utilizes statistical techniques inspired by weak supervision, which programmatically generate labels for an unlabeled dataset by aggregating the predictions of several weak "voters" via a latent variable graphical model [71, 73]. Weak supervision has mostly been studied in classification settings [26, 72] but more recently has been extended to tasks such as learning rankings and manifolds [85, 94]. We derive our estimation procedure from the Gaussian model in [85]. applying it to LLM embeddings and the routing setting.
3 Preliminaries
3.1 Problem setup
Let V be the token vocabulary space, and let D = Vx... ... x....Vbbe Vbe the space of all vocabulary sequences.
We consider a generative task with input text x E X' C V and reference output text y ) C V. We have candidate pool of m LLMs, G {9),-... .9mm ). where each yi E G X - y produces generative output sequence (a)))) for given input text sequence x. We are given an unlabeled test dataset Diest
{z4 ) where the ground-trun truth reference outputs are unknown.
Diest to one of the LLMs in G. Specifically, we wish to construct a router route 9Th x X + G that selects the LLM that yields the highest quality generation on x for each test sample æ, without any labeled data.
Our goal is to route each sample
3.2 Graphical model
We present a probabilistic graphical model (see Figure 1 (center)) that describes how the L.MM outputs, gi (a), ... Im (zr), are related to output y in terms of each LLM's quality on a given input ær, which we call (z)), corresponding to each (z).. Let zgo V + Rd map from sequence of tokens to a d-dimensional embedding using a common model go such as SentenceBERT [76]. Define X,(x) :=
Zgo ([z, g: (x:) )) to be the observable embedding of and the LLM output, and define 2 2²(x) =
go ([x, v/) to be the latent ground -tunt embedding of x and reference output y. Similar to the approach in [85], we model the distribution over embedding vectors, Pr(=* A (z), ..)....... Am (z))a)
as
cxp
-((=)) (x))1)
(I)
where Z is the log partition function and the (z(2:)s- the LLM quality scores- are canonical param-
eters of the graphical model. Intuitively, our model captures I.M quality by supposing that if g; is of high quality and (z) is very large, then it should be unlikely for the LLM output to be very
3

---

Algorithm EESTIMATE SCORES
1: Input:
unlabeled test dataset Dies, LLMs G, no nearest neighbors parameter, 9o embedding model with dimension d.
2: For all x E Dest and E G, obtain the gemerator output (x) and embed the input and generator output using model go get embedding (( == 2go ( (z)))
3: for E Piett 91 E G do
4:
for j. k # i [m]] do
5:
Compute (1)(T) = 1 E''ENNo [1((x') > (x)) and similarly 8 (x) and 85x(x) no
6: Set 0g (x) (z( + )u(x)) - (z)(
7: Compute averaged estimate ((()) (7 Ej,kys (x) (z).
8:
return (z(z) for all x Dreat 9s G.
different from the true output in terms of Euclidean distance in embedding space. Conversely, if (x)) is small, we assign larger probability to the setting where (2 and 2" (z) differ significantly. Finally, note that this graphical model corresponds to multivariate Gaussian. That is, the vector (z)) =² (rr)..... Am (2) = (1r) Rdm is Gaussian with mean l Ō and diagonal covariance matrix D E gdmxdd with زز (0) Intuitively, this means that the average difference
vector between each A and 2 =*(x) is centered, with its magnitude inversely proportional to the LLM score 0 (z) and independent of other LLMs. Given this probabilistic graphical model, our goal is to learn each quality score (z) from the unlabeled test dataset and use these for improved routing.
Method
Given an unlabeled test dataset Disa and a pool of LLMs G, SMOOTHIE consists of two steps:
1. Estimation: The LLM quality scores 0 (I)),... Om (ar) are learned for each x E Diesa (Section 4.1, Algorithm 1).
2. Routing: The LLM with the highest scores is selected, and its output is used final prediction for (Section 4.2).
We describe each step in the following sections.
4.1 LLM score estimation
We describe how to estimate each 0, (z)s in the graphical model in (1) using only unlabeled data from Diea- Then, we describe how the LLM score estimate can be instantiated to be sample-conditional
Computing (z()) Below, we state simple property arising from the fact that (1) corresponds to multivariate Gaussian with diagonal covariance matrix.
Proposition 1 [85] For any و E [m). it follows from the graphical model in (1) that
E [[A(()) - 11] -y(=))
[[Ax(0) (x)] + [1Ag(t)) --")x))="(x)1111) (x))]]]
(2)
The proof is in Appendix D and relies on the fact that off off-diagonal entries of Σ are O. Note that the left hand side of the equation observable while the two expectations on the right are unknown. We can apply this equation to pairs of LLM embeddings over triplet of X; Aj, d to form system of three equations with three unknown expectations. Solving, we have
E [1)(()) - (x) = 2
()))()) + (z( (x) jk E [m],
(3)

---

where (z) = E [[]A(()) >,(2))19 Since (1) is multivariate Gaussian with E5 201=/m]
we can write (z) as the following function of E [11A(x)) - 2 (x)113):
E [1A+(x)) - (z)] Σ B [())(())) ((Nx)(=))23(=){) (}(2)) = Σ Var -2)(()) = =2,(zz)
d
(4)
where As.5 (z) and (z) are the jth indices of the embeddings (x) and 2 =x)x) respectively. There- fore, we write 6.* (x) (z) (z)))) where each 5.g (2) can be estimated using the LLM
outputs on Diest and in practice in Algorithm we estimate (x))) by averaging 0.hh over all (">z)) pairs of j. k # i.
Sample-conditional estimation of (x))) Note that the expectation in 51z(2) E (1]))()) - X=(r)] is over the randomness in Az(x)))((z) conditioned on fixed point I. However, we only have one sample per æ. One simple approach is use the entire dataset to estimate 1
n E [)x(2')) {1(2)) We denote this as SIRDOTHIE-GLOBA.A However, in SMOOTHIE-GLOBAL each (z(x) for [m]] is constant over the entire Diest Therefore, we use nearest neighbor kernel smoothing to estimate each 515(x) in sample-dependent manner, an approach we call AMOOTHIE-LOCAS Concretely, for Dusa define NN Drest as the no < nearest neighbors of z (excluding x itself) in fu embedding space. Then, we construct 8ig (z) Bo
(zx), i.., (x)=))
EN' (a) [x)) (zz))l and do the same for (zx(2), (z), to get
sample-conditional estimate of 0z(x). The procedure for estimating (x)) in SMOOTHEE-LOCAL outlined in Algorithm 1.
4.2 Routing
Once we have estimates of (z) for each of the m generators by using Algorithm 1, we can construct our route() function. We define route(G, = gi where arg max (z)..... ......... 6m (z)), which selects the highest scoring LLM for input r based on (z)). We apply this on Diest to determine the best LLM for each input sample.
5 Results
We empirically analyze SMOOTHIE-GI.OBAL and SMOOTHIF-LOCAL, focusing on four questions:
1. How well does SMOOTIIB-GLOBAL recover ground-truth LLM rankings over samples belonging the same task (Section 5.1)?
2. In multi-task datasets, how well can SMOOTHIF-LLOAAL perform unsupervised-routing, by identifying the best LLM for each sample (Section 5.2)?
3. Can SMOOTIHE-GLOBA and SMOоTиHB-LLAAL be applied to select from or route between different prompts (Section 5.3y?
4. How does SMOOTHIE-GI.OOm and SMOOTHIF-LOOCAL'S performance change as a function of different algorithmic choices (Section 5.4)?
5.1 Single- Task LLM Scoring
Setup We begin by evaluating whether SMOOTHIE-GLOORE can accurately learn the relative performance of different L.MSs on single task-dataset. We study three categories of tasks. First, we consider datasets corresponding to commonly-studied natural language generation (NI.) tasks [51]: CNN/DailyMail and XSum (summarization), SQuAD (reading comprehension), TriviaQA (factual recall), E2E and WebNLG (data-to-text generation), and LegalBench's 's Definition Extraction (text extraction) [1, 27, 30, 31, 43, 61, 62, 70, 81, 84]. We report Rouge2 for summarization and data-to- text generation tasks and accuracy for all others. For all tasks other than Definition Extraction we evaluate SMOOTHIE GLOBAL 1000 sample subset.? For these tasks, consider two ensembles
Deeinittion Extraction has fewer than 1000 samples.
5

---

of LLMs at different size points. At the 3B size point, our ensemble consists of Pythia-2.8B. [7]. Gemma-22 [91]. M3 [17], and Dolly-3B [18]. At the 7B size point, our ensemble consists of L [92], Mistral [40]. Vicuna [107]. Gemma-7B [,1], and Nous Capybara [19]. We manually write single prompt template for each task, and all model generations rely on this template.
benchmarks: AlpacaEval and MixInstruct [23, 24, 41, 50J. For AlpacaEval. we rely on responses accessible via the online leaderboard." 3 We identify 10 LLMs (each from a different base family), and download these models' responses to the AlpacaEival instructions. We conduct 10 different simulations, where in each simulation we randomly select 5 LLMs from our pool to function as an ensemble. Reported win-rates use the standard GPT-4 references. For MixInstruct, we use generations from an ensemble of 1 different LLMs originally studied in [41]. Following [41], we measure generation quality using ChatGPT- rank.
Second, we consider two cestruction-followiii
Finally, we consider a more "reasoning-intensive"" task, GSM8K [16]. We consider ensemble of three models: Gemma-7B, Phi-2 [39], and Llema-7b [4]. We prompt each model to provide a chain-of-though reasoning [100], and apply SMOOTHEE to these generations.
For all datasets, we apply SMOOTIIE-LLLBL using SentenceBERT pall-mppet-base-ee em- beddings of generations [76].
Results We first measure how frequently the highest-weighted LLM according to SMOOTHIE- GLOBAL corresponds the best-performing LLM in the ensemble. We observe that SMOOTHIE- GLOBAL selects the best- pert-performing LLM for 4/7 tasks on the 3B ensemble, and for 5/7 tasks on the 7B ensemble (Figure 8). On AlpacaEvall, SMOOTIHE-GLOBA selects the best-performing LLM by win-rate for 8/10 ensembles, and the best performing LLM by length-controlled win-rate for 7/10 ensembles. On MixInstruct and GSM8K, SMOOTHHE-GLLOBB. again identifies the best-performing LLM in the ensemble.
Second, we measure how well SMOOTHIE-GLOBAL captures quality differences between LLMs in the ensemble, by computing the Spearman's 's rank correlation coefficient between 8₂ and ground truth quality scores ensemble models. Overall, we find that SMOOTHIE- -ITHIII--GLOBAL''' learned weights approximate the relative ordering of model quality well. On the NLG tasks SOOOTHIE--GLOBA measure average correlation coefficient (across both ensembles and seven tasks) of 0.72. Figure 2(a) visually depicts the distribution of task coefficients-on only one ensemble/dataset pair is there correlation coefficient O. On MixInstruct, we observe correlation coefficient of 0.94, and on AlpacaEval, we observe a correlation coefficient of 0.46.
we
Finally, we measure how the performance of the LLM selected by SMOOTHIE compares other selection algorithms. We first compare MOOOTHHE-GLOBAA to an unsupervised random baseline (RANdOM), which would select a random model from the ensemble. We reported the expected performance of this method, which is equivalent to taking the average performance of the ensemble. We also compare AMOOTHIE-GLORRE the basis of a small amount of validation data [67] (BEST-ON- VAL). We sample a small labeled validation set (50 samples) and select the LLM that performs the best on this set. To account for sampling variation, we repeat this with 10 random draws and report the average performance. Because AlpacaEval has no training split and MixInstruct has no labeled data, we only compare MROOTHIE-GLOBA. to RANDOM on those datasets.
labeled baseline which simulates selecting an LLM on
Table 1 provides results for the seven NLG tasks. We find that SMOOTHIE- GLOBAL outperforms the unsupervised RANDOM baseline on 6/7 tasks for the 3B ensemble and on 7/7 tasks for the 7B ensemble. SMOOTHIE- GLOBALL outperforms RANDOM by up to 7pts (on tasks measured by rouge2), and by up to 12pts (on tasks measured by accuracy). We also observe that SMOOTHIE-GiO BOOTHII--GLOBLLL.. is frequently competitive with and even outperforms the BEST-ON: VAL baseline, which uses labeled data. SOROTTIEE-LLOR 5/7 tasks for the 7B ensemble. On GSM8K, SMOOTHIB-GL.BAA achieves solve-rate of 37.5% (matching BEST-ON- VAL, while RANDOM achieves a solve-rate of 28.3% (Table 11).
outperforms BEST-ON- VAL on 4/7 tasks for the 3B ensemble, and
also outperforms the RANDOM bascline on the instruction-follont followiwn datasets. On MixInstruct, SMOOTHIE- GLOBAL achieves a ChatGPT-rank (L) of 3.91, while RANDOM achieves ChatGPT-rank of 5.95 (Table 10). On AlpacaEval, SMOOTHIE-GLONE outperforms RANDOM on all but one trial (across both win-rate and length-comtrolled win-rate). SMOOTHIE-GI.OBAL
Responses available the AlpacaEval website: https://tatau- w.laaa.ggub..........aa......... taabb.io//a 1t/ wt//apacc_eevalll/.l

---

CNN
Def. Ext.
E2E
SQuAD
TriviaQA
WebNLG
XSum
RANDOM
12.9
52.4
27.3
59.6
32.7
23.4
4.5
3B
MMOOTHIE-GLOBAN
14.3
61.5
31.8
60.7
32.1
30.7
4.5
BEST-ON-VAL
13.0
60.5
31,1
66.4
38.7
30.3
5.3
RANDOM
13.7
58.5
35.3
67.9
59.3
44.1
6.9
7B
SUBOTHIE-GLOBA.
14.5
70.9
36.9
76.2
68.3
45.9
8.4
BEST-ON VAL
14.5
69.4
36.7
74.0
65.8
48.3
8.3
Table I: Comparing SMOOTHIE GLOBAL to baseline methods on different ensembles across NLG datasets. Underlined values are the best performing unsupervised methods, Bold values are the best performing overall methods, We report rouge2 scores for CNN, XSum, WebNLG, and E2E, and accuracy for the rest, All metrics scaled to 0-100.
outperforms RANDOM by an average of 15pt win-rate, and up to 27pts. Figure 2(b) and Figure 2(c) visualize this distribution.
Correlations Distribution
NLG Smoothie
Smoothie Random Alpaca
Smoothie Random on Alpaca
1.00
30
30
Correlation Coaft.
0.75
win mate Improvement
LC Win Nate Improvement
20
0.50
0.25
0.00
-10
-10
(a)
(b)
(c)
Figure (a) Spearman's 's rank correlation coefficient between SMOOTHIE- GLOBAL weights and ground- LLM performance 3B and ensembles across NLG tasks. (b) SMOOTHIE GLLOAA'' improvement
over RANDOM by win-rate on AlpacaEval. (c)SMOOTHEE-GLa) GLOBIA'' improvement over RANDOM by length- controlled win-rate on AlpacaEval.
5.2 Multi-task Routing
Setup We next assess whether AMOOTHEE-LCCA. sample-conditinal scoring mechanism allows to route samples to LLMs the multi-capability regime. construct two mixed-task distributions by combining existing datasets. The first distribution corresponds to tasks measured by accuracy, and contains SQuAD, TriviaQA, and Definition Extraction. We refer to this as AISTR-ACC The second distribution corresponds to tasks measured by Rouge2, and contains CNN/Daily XSum, For each mixed-task dataset, we report the
Web NL.G, and E2R. We refer to this as metric averaged across all tasks. We compare to three baselines.
RANDOM: A cundom-selection baseline which returns generation from a random LLM in the ensemble. Though naive, prior work has found this be a strong method in practice [56]. We run 10 trials and report the mean of this approach to account for variance.
labeled data-based KNN baseline. For this, we sample 50 labeled samples from a separate hold-out set (Dval), and measure the performance of each candidate LLM on this set. For a given test sample x, we identify the 20 most semantically similar instances in D (using SentenceBERT embeddings [76]), and route to the highest performing LLM over this subset. We note that the LABELED- BENEEEE--NNN baseline is derived from routing methods in [48, 86).
LABELED-KNN.
PAIRRM: A reward model from [41] which accepts an instruction and multiple generations input, scores each generations suitability for the instruction, and returns the predicted best generation. PAIRRM is a labeled-data method which [41] trained on collected preference data.
In addition, we also compare the best individual model the ensemble (BEST-MODEL), and SHOOTHIE-GLOBA.. For both mixed-task datasets, run SMOOTHIE-LOCAL with SentenceBERT

---

3B
7B
Method
DISTR-ACC
DISTR-ROUGE2
DISTR-ACC
DISTR-ROUGE2
RANDOM
48.7
17.0
65.4
25.0
PAIRRM
53.9
19.0
25.5
LABELED-KNN
51.0
16.8
71.7
26.2
BEST-MODEL
52.3
18.1
73.2
26.4
SMOOTHIE-GLOBAL 51.3
18.1
66.5
26.1
SMOOTHIE-LOCAL
58.7
20.2
75.0
26.9
Table 2: Comparing SMOOTHIF-I.OO LOCAL. to baseline methods on the 3B and 7B ensembles for multi-task distributions. DISTR-AcC and DISTR-ROUGE2 are measured with accuracy and rouge2 respectively. Bold values indicate the best performing method for each dataset and model size. Metrics scaled to 0-100.
embeddings, and the sample-conditional version of SMOOTHIE-LOCAL estimates (((x)) using neighborhood size ng
Results for the 3B and 7B ensembles over DISTR-ACC and DISTR-ROUGE2 are provided in Table 2. We find that -MOOTHIE-LOCK. outperforms all baselines across both data distributions, for both ensembles. Though SMOOTHIE-LOCAL requires no labels, it still outperforms labeled data baselines like LABELED-KNN and PAIRRM. We observe a substantial gap between SMOотIHE- and SMOOTHE-GLOBAL which indicates that SMOOTUL--LCALL sample-specific scoring mechanism provides performance improvements.
Distr-Acc (3B)
Distr-Acc (7B)
Relative Frequency
1.00
Relative Frequency
1.00
0.75
0.75
0.25
0.25
Rank
Rank
(a)
(b)
Distr-Rouge2 (3B)
Distr-Rouge2 (7B)
Relative Fraquency
1.00
Relative Frequency
1.00
0.50
0.50
0.25
0.25
0.00
0.00
4
2
5
Rank
Rank
(c)
(d)
Figure 3: On DISTR-AC and HISTR-ROUGEL measure how frequently SMOOTHIE-LOCAA selects the i-th best generation across the ensemble, for both the 3B and 7B ensembles.
Notably, we see that SMOOTHIE-LOCAL substantially betters BEST-MODEL, indicating that SMOOTHIF-LOCAL13 routing mechanism is offering performance improvement over strategy which merely selects the best LLM on average. We study this in greater detail by examining the relative rank of the LLM selected by SMOOTHE--OCAAL for each sample. For each sample in

---

CNN
Def. Ext.
E2E
SQuAD
TriviaQA
WebNLG
XSum
RANDOM
7.1
60.3
27.8
47.3
22.0
29.2
4.7
Falcon
SMOOTHIE-GLLOBA
7.9
62.2
31.6
53.3
31.4
28.3
6.4
SMOOTHIF-LLOCA.
8.0
69.2
31.5
53.3
27.4
30.8
6.0
BEST-ON-VAL
64.2
31.0
52.7
31.4
32.5
RANDOM
47.8
31.6
45.9
45.5
11.2
Llama-2
SMOOTHIE-GLOPE
6,9
64.6
37.6
61.4
68.7
48.5
12.8
LMOALHIE-LOCAL
9.5
59.3
33.6
63.1
61.3
48.0
12.7
BEST-ON-VAL
11.8
64.6
35.0
66.1
68.7
48.7
13.0
Table 3: Comparing SMOOTHIE- SMOOTHIE-GL.OBAL GLOBAL and SMOOTHIE-LOCAL baseline methods in the prompt-sclection setting. Underlined values the best performing unsupervised methods. Bold values are the best performing overall methods. We report rouge2 scores CNN, XSum, WehNLG, and E2E, accuracy for the rest. metrics are scaled to 0-100.
DISTR-ACC and DISTR-ROUGERS ROUGE2, rank the quality of each LLM's generation according to standard-competition ranking (i.e., "1-2-2-4" ranking). We then count how frequently SMOOTHIE- LocAL selects the rank-i generation across each distribution for each ensemble. We visualize results in Figure 3. As the visualizations demonstrate, SMOOTHIE-LOCAL consistently selects the best or second-best generation from within the ensemble.
5.3 Prompt Selection
Third, we study whether SMOOTHEE-LOCA, and SMOOTHIE- GLOBAL can be generalized to other settings where engineers have a candidate pool text generators of unknown quality, and must select one of them to use for some application. In particular, we focus the setting where an engineer has access to multiple prompt templates for given generation task, and must select which prompt-templates generation to use the final output [29]. Unlike above, assume the engineer only has access to one LLM. We study SMOOTHIE-LOCAL and SMOOTHIE-GLOBA. in this regime using the NL. tasks from Section 5.1. For each task, manually write between 3 and 5 prompt templates, varying the wording of instructions and the choice of in-context samples. We analyze SMOOTHIE applied two models different size points: Falcon (1B) [65] and Llama-2 (7B) [92].
Table 3 provides the results. Overall, we find that SMOOTHIE-GLOBA. selects the optimal prompt 2/7 times for Falcon-1B, and 3/7 times for Llama-2. SMOOTHHB-LCAAT and SMOOTHIE-GLOBA. consistently outperform RANDOM-On 6/7 tasks for Falcon-Ib and 6/7 tasks for Llama-2. On task/model combinations, one of either SMOOTHIE- GLOBAL SMOOTHIE-LOCAL matches outperforms a labeled baseline. To better contextualize performance improvements from SMOOTHIE- GLOBAL, also compare to the improvement that accompanies increasing model size. Following common practice in recent work, we can quantify the extent to which SMOOTHIE- GLOOAAL allows smaller models match exceed the performance of larger models [2, 29]. In Figure (Appendix E), we compare RANDOM and SMOOTHIE-GiL GLLOBAL on models from the Pythia suite at four sizes: 410M, IB, 2.8B, and 6.9B parameters [7]. We observe that SMOOTHIE- GLOBAL substantially improves performance-on on E2E, SMOOTIIE-GLOBAL enables a 410M parameter model to outperform a 6.9B parameter model.
5.4 Ablations
Finally, conduct ablations to examine different aspects of SMOOTHIE-GLOBAL and SMOOTHIE- LOCAL: improving its efficiency, adjusting the neighborhood size, varying the choice of embedding model, and using different LLM ensembles.
Improving efficiency First, we explain SMOOTHIE's current efficiency properties. estimate Smoothie weights for routing, we use simple closed-form procedure that does not require any SGD or training, as described in Algorithm 1. As result, SMOOTHIE weights on the entire dataset can be computed in seconds-fft for the 7B ensemble, SHOOTHIE-LOCAS on the multi-task datasets takes

---

2. 14 seconds per 1000 samples, and MHOOTHIE-GLOBAA on the single-task datasets takes under 0.03 seconds per 1000 samples. Moreover, SMOOTHIE does not require any ground ground-truth annotations; however, all model generations per test sample are needed as input to the algorithm. That is, we need n X m generations for a Dirain of size n samples.
Fortunately, the need for computing all model generations per test sample can be removed with small algorithm tweak, making Smoothie even more efficient and its runtime independent of Suppose we have a held-out set of Prrain train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm KNN is now over held-out training dataset). This approach, which we call SMOOTHIE- TRAIN, selects the model for test sample without needing model generations for that sample. Only Turain x m generations are needed, regardless of how large the test dataset is.
We study the NLG tasks, using Rerain 250 samples. In Table 7 (Appendix E), we evaluate version of model-dataset pairs, and performs worse on the remaining 2/14 pairs. We also evaluate SMOOTHEE- LOCAL-TRAIN, on DISTR-ACE and DISTR-ROUGER (Table 8) using a neighborhood of size no = 20. We find here that while SMOOTHIE LOCAL-LAALLLLLLL underperforms SMOOTHEE-LOCAL on both the 3B and 7B ensemble for both DISTR-ACU and HISTR-ROUGE2, it still outperforms RANDOM and remains competitive with supervised baselines.
and observe that it matches SMOOTиHE-GLBBA on 12/14
Neighborhood size We study the impact of ng. and consider SMOOTHIF-LOCAL'S performance for no E (1 5, 10, 20, 50, 100). Figure 5 provides performance over DISTR-AcC and Figure 6 provides Overall, we find that SMOOTUE--LOALL' performance steadily
performance over degrades as no increases, and is highest when no 1.
Choice of embeddings We study how the choice of embeddings affects SMOOTHEE-GLORE perfor- mance (Table 9). Specifically, we compare the performance of -INOTHIE-LocAA using Sentence- Bert embeddings (all-mpnet-baase base- [76] to BGE embeddings (bge seteemmmml-en----- We observe that SMOOTHIE-LOCAM appears robust different embeddings SaASSS--MOOOOO-OOOSOSOSSSS LOCAL with BGE embeddings still outperforms other labeled and unlabeled baselines. Interestingly, we observe that certain embedding models appear to yield better performance over certain distribu- tion/ensemble combinations. For instance, SMOOTHIE-LOCAA with SentenceBERT embeddings outperforms SMOOTHIE-LOCAL with BGE embeddings on DISTR-ACC for the 3B ensemble and DISTR-ROUGE2 for the 7B ensemble, while performing worse on ensemble and DISTR-ACC for the 7B ensemble.
[102].
for the 3B
Different ensembles Finally, we consider whether SMOOTHIE- GLOBAL can generalize to a wider array of ensembles (Figure 7). We combine the LLMs contained in the 3B and 7B ensembles into a single pool, and sample 50 distinct ensembles ranging in size from 4-7 LLMs. For each of the 7 NLG tasks, we evaluate SMOOTHIE GLOHHLL- ability identify the best model from within each ensemble. Across these 350 settings, we find that SMOOTHIE-GI.OO 211 of them (60.2% of the time), and one of the two best models in 292 of them (83% of the time).
identifies the best model in
6 Conclusion
In this paper we study and propose an algorithm for learning label-free routers for generative tasks. We validate approach across a variety of evaluation regimes, finding it consistently beats other unsupervised approaches and often matches/exceeds supervised approaches.
Limitations We discuss several of SMOOTHIE's limitations. First, its multivariate Gaussian graphical model currently uses diagonal covariance matrix. This assumes independent error vectors for each generation, though SMOOTHIE could be extended to account for dependencies [72, 93]. Additionally, SMOOTHIE optimizes only for performance without considering cost tradeoffs between large and small models. Finally, its reliance on embeddings may capture only certain aspects of semantic similarity. Other embedding models and additional heuristics could be used to create richer input features for SMOOTHIE.
10

---

7 Acknowledgements
We thank the Microsoft Accelerate Foundation Models Research Program for providing portions of the compute used for the results in this paper. We thank Gautam Machiraju, Jon Saad-Falcon, Krista Opsahl-Ong. Sabri Eyuboglu, Jordan Juravsky, Vishnu Sarukkai, Ben Spector, Eric Nguyen, Jerry Liu, Chris Fifty, Avanika Narayan, and Michael Zhang for their helpful feedback and discussion.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 (Hadwwar-AAware) CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARI. under Nos. W911NF-23-2-01 84 (Long-context) and W91 (Interactive Human-AI Teaming); ONR under Nos. N0001 100014231200 42313633 (Deep Signal Processing): Stanford HAI under No. 247183; NXP. Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VMWare. NG is suported by the Stanford Interdisciplinary Graduate Fellowship (SIGF). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed implied, of NIH, ONR, or the U.S. Government.

---

References
[1] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2024.
[2] Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models, 2022.
[3] Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes, 2023.
[4] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
[5] Regina Barzilay and Kathleen R. McKeown. Sentence fusion for multidocument news summa- rization. Computational Linguistics, 31(3):297-328, 2005.
[6] Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew R. Gormley. It's mbr all the way down: Modern generation techniques through the lens of minimum bayes risk, 2023.
[7] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[8] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint
2021.
[9] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoscini. Large language monkeys: Scaling inference compute with repeated sampling, 2024.
[10] Lei Chai, Hailong Sun, and Zizhe Wang. An error consistency based approach to answer aggregation in open open-ended crowdsoureing. Information Sciences, 608: 108:1029-104 2022.
[11] Ting- Yun Chang and Robin Jia. Data curation alone can stabilize in-context leaming. arXiv preprint arXiv:22 12. 10378, 2022.
[12] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more Ilm calls you need? towards scaling laws of compound inference systems, 2024.
[13] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How use large language models while reducing cost and improving performance, 2023.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1xXXi::207700332 2021.
[15] Mayee F. Chen, Daniel Y. Fu, Dyah Adila, Michael Zhang, Frederic Sala, Kayvon Fatahalian, and Christopher Ré. Shoring up the foundations: fusing model embeddings and weak supervi- sion. In James Cussens and Kun Zhang, editors, Proceedings of the Thirty- Thirty-Eighth Conference Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 357-367. PMLR, 01-05 Aug 2022.
[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1410 2021.
12

---

[17] Together Computer. Redpajama: an open dataset for training large language models, 2023,
[18] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tune tuttondon-teedddde 1lm, 2023.
[19] Luigi Daniele and Suphavadeeprasit. Amplify-insee innitti-sstret::: Synthetically generated diverse multi- turn conversations for effecient IIm training. arXiv preprint arXiv:(comming soon), 2023.
[20] Yihe Deng, Weitong Zhang. Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves, 2024.
[21] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing, 2024.
[22] Viet-Tung Do, Bha-K Hoang, Duy-Hung Nguyen, Shahab Sabahi. Jeff Yang, Hajime Hotta, Minh- -ieh Nguyen, and Hung Le. Automatic prompt selection for large language models. arXiv preprint arXiv:2404.0211 02717, 2024.
[23] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length- congth-onttoolled alpacaeval: simple way debias automatic evaluators. arXiv preprint arXiv:2404.04475. 2024.
[24] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
[25] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.
[26] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper. Kayvon Fatahalian, and Christopher Re. Fast and three-rious: Speeding up weak supervision with triplet methods. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3280-322 3291. PMLR, 13-18 Jul 2020.
[27] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachin! Creating training corpora for NLG micro-planners. In Regina Barzilay and Min-Yer Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Pupers, pages 179-188. Association for Computational Linguistics, 2017.
[28] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04033 2022.
[29] Neel Guha, Mayee Chen, Kush Bhatia, Azalia Mirhoseini, Frederic Sala, and Christopher Ré. Embroid: Unsupervised prediction smoothing can improve few-shot classification. Advances in Neural Information Processing Systems, 36, 2024.
[30] Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas- Choolas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36, 2024.
[31] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693 1701, 2015.
[32] Bairu Hou, Joe O' O'connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. Promptboosting: Black-box text classification with ten forward passes. In International Conference on Machine Learning, pages 13309-13322 PMLR, 2023.
13

---

[33] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system, 2024.
[34] Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. Enabling ensemble learning for heterogeneous large language models with deep parallel collaboration, 2024.
[35] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023,
[36] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 2020.
[37] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79 87, 1991.
[38] Joel Jang, Seungone Kim, Sconghyeon Ye. Doyoung Kim, Lajanugen Logeswaran, Moontac Lee. Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning, 2023.
[39] Mojan Javaheripi and Sébastien Bubeck. small language models. phi- 22tthesuuppriing -oa.ga---oppooo-----------aaaaa-
Phi-2: The surprising power of
com/ en- us wus/rseaarch/blog 2023.
of mal--mmamnn-agggeeeeee
[40] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock. Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[41] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023.
[42] M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the em algorithm. In Pro- ceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan), volume 2, pages 1339-1344 vol.2, 1993.
[43] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, page arXiv:1705.033: 00555551 2017.
[44] Hayato Kobayashi. Frustratingly easy model ensemble for abstractive summarization. In Ellen Riloff, David Chiang, Julia Hockemmaier, and Jun icchi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4165-4176. Brussels, Belgium, OctoberrNovember 2018. Association for Computational Linguistics.
[45] Sawan Kumar and Partha Talukdar. Reordering examples helps during priming-based few-shot learning. arXiv preprint arXiv:2006.0751, 2021.
[46] Logan Lebanoff, Franck Dernoncourt, Doo Soon Kim, Walter Chang, and Fei Liu. cascade approach to neural abstractive summarization with content selection and fusion, 2020.
[47] Logan Lebanoff, Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Walter Chang, and Fei Liu. Learning to fuse sentences with transformers for summarization, 2020.
[48] Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Orchestrallm: Efficient orchestration of language models for dialogue state tracking. 2023.
[49] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
[50] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction- following models. https: /tte::///tttii h//gggtttbb..t// com www.//tttullab///ooooaaaoo alppaa_ppcaaaaae/eeeaeaa _eall, 2023.
14

---

[51] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga. Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning. Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2023.
[52] Chonghua Liao, Yanan Zheng, and Zhilin Yang. Zero-label prompt selection. arXiv preprint Sikiv:2211.04664 2022.
[53] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2 102.000.00000 101.06804, 2021.
[54] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[55] Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guice -eiied ensemble of large language models, 2023.
[56] Xiaoding Lu, Zongyi Liu, Adian Liusie, Vyas Raina, Vineet Mudupalli, Yuwen Zhang, and William Beauchamp. Blending all you need: Cheaper, better alternative to trillion- cillion-parammeen parameters 1lm, 2024.
[57] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
[58] Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. Shyam Upadhyay, Mausam, and Manaal Faruqui. Automix: Automatically mixing language models, 2024.
[59] Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Imms: Model fusion at test-time via perplexity optimization, 2024.
[60] Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization, 2024.
[61] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018.
[62] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation, 2017.
[63] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data, 2024.
[64] OpenAl. Learning to reason with large language models. tereiing-to-reesono weston--iiitt wheeenwwwhhhhlmmmmeee tatthllmm/.,....mm 2024. Accessed: 2024-10-29.
cow/index//
[65] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeh dataset for Falcon I.M:: outperforming curated corpora with web data, and web data only, arXiv preprint ar arXiv:2306.01 01 016, 2023.
15

---

[66] Xiangyu Peng. Chen Xing, Prafulla Kumar Choubey, Chien- Chien-Sheng Wu, and Caiming Xiong. Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning. arXiv preprint arXiv:22 10. /2587, 2022.
[67] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1051-11540000 2021.
[68] Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304. M.XX::2304000970,, 2023.
[69]
Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang. Tianren Gao, Angela Wang, Karen Li, Yongning Sheng, Joshua Brot, Denis Sokolov, Apurv Vivek, Calvin Leung, Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jackson, Mark Luttrell, Manish K. Shah, Edison Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J. Brown, and Kunle Olukotun. Sambanova sn40I: Scaling the ai memory wall with dataflow and composition of experts, 2024.
[70] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016.
[71] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 166):269-282, November 2017.
[72] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision, 2018.
[73] Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly, 2017.
[74] Mathieu Ravaut, Shafiq Joty, and Nancy F. Chen. Summareranker: A multi-task mixture-of- experts re-ranking framework for abstractive summarization, 2023.
[75] Mathieu Ravaut, Shafiq Joty, and Nancy Chen. Towards summary candidates fusion, 2023.
[76] Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties. 11 2019.
[77] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:21 2021.
[78] Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha Christopher Ré, and Azalia Mirhoseini. Archon: An architecture search framework for inference-time techniques, 2024.
[79] Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24. ACM, March 2024.
[80] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. 2020.
[81] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[82] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gate mixture-of-experts layer, 2017
[83] Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models, 2024.
16

---

[84] Anastasia Shimorina and Claire Gardent. Handling rare items in data-to-text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 360-370. Association for Computational Linguistics, 2018.
[85] Changho Shin, Winfrred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederie Sala. Universalizing weak supervision. In International Conference on Learning Representations, 2022.
[86] Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2023.
[87] Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. Getting more out of mixture of language model reasoning experts, 2023.
[88] Aditi Singla, Aditya Singh, and Kanishk Kukreja. A bi-objective e-constrained framework for quality-cost optimization in language model ensembles, 2023.
[89] Taylor Sorensen, Joshua Robinson, Christopher Michael Rytring, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. An teformatiion-thoortiie approach to prompt engineering without ground truth labels. arXiv preprint 2rXiv:2203.113335 2022.
[90] Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah Smith, et al. Selective annotation makes language models better few-shot learners. arXiv preprint 22Xiv:2209.11975 2022.
[91] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Chrisse Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean- Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Matco Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiee, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Bli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. 2024.
[92] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Fsiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mi- haylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
17

---

[93] Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. Learning dependency structures for weak supervision models, 2019.
[94] Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Lifting weak supervision to struc- tured prediction, 2022.
[95] Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi Fusechat: Knowledge fusion of chat models, 2024.
[96] Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. Fusing models with complementary expertise, 2024.
[97] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou, Mfxture-of-agents enhances large language model capabilities, 2024.
[98] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
[99] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[100] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[101] Zhiyong Wu, Yaoxiang Wang. Jiacheng Ye, and Lingpeng Kong. Self Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. arXiv preprint arXiv:2212. 10375, 2022.
[102] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[103) Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, and Minjoon Seo. Improving probability-based prompt selection through unified evaluation and analysis. arXiv preprint arXiv:2305. 14877, 2023.
[104] Chenrui Zhang, Lin Liu, Chuyuan Wang, Xiao Sun, Hongyu Wang. Jinpeng Wang, and Mingehen Cai. Prefer: Prompt ensemble learning via feedback-reflect-rrfee In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19525-19532, 2024.
[105] Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. arXiv preprint arXiv:221 20.:22211.44480 2022.
[106] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03449, 2022.
[107] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao WU, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
18

---

A Appendix
In Appendix we provide glossary of notation used in the paper. Appendix C, we provide an extended related work, and in Appendix D we provide a proof of Proposition 1, which is used in deriving the SMOOTHIE algorithm. Finally, in Appendix E we provide additional experimental results and details.
Code for reproducing our results and using SMOOTHIE is available httpa://github. com/ http///searrc/mmoothhht
B
Notation
The glossary is given in Table below.
Symbol
Used for
V
The space of all vocabulary sequences. Input text X V.
y
Reference output text y E y C V. Candidate pool of m LLMs, produces generation gi (1r) input
G
(.).....mm.) where each ys E E x y
Desa
Unlabeled test datasct Dest {x,
route
Routing function route G X + that selects the best LLM from G each sample.
(x)
Quality score of the ith LLM on test sample æ, also used in the graphical model in (1).
Embedding mapping Zgo V + Rd for any text sequence, where gu is embedding model as BeRt [76].
(x)
The observable embedding of x concatenated with the ith LLM'' generated output,
A; (x) := Zau ([x,, (=)]).
=*(x)
The latent embedding of concatenated with unknown reference output, 2 2xxx) == zg ([x, v))- Partition function for normalization of (1).
Z
no
Number nearest neighbors used learn (x(x)) x. no n (i.e., the entire test dataset) corresponds to SMOOTHIE- SMOOTHIE-GLOBAL GLOBAL and ny < n corresponds to SMOOTHIE- LOCAL.
(x)
The average squared Euclidean distance between the ith and jth LLM embeddings over neighborhood around 8.g (z) mu ENN. (x) (²)) (z
This is the primary expression used in computing (x)).
Table 4: Glossary of variables and symbols used in this paper.
C Extended Related Work
LLM Routing The problem determining how to route samples to various models has been long studied in statistics [37, 42] well as Mixture of Experts deep neural networks [25, 82]. These works focus on how to jointly train the models and router in a stable and efficient manner.
Since many LLMs are now available off-the-shelf, recent works study how routing mechanisms can be applied inference time to trained models. Some works involve training task or domain-spectiee expert models and then learning router. The router can be nearest neighbors algorithm [38], neural network [96] that classifies among the different domains corresponding to the experts, or an extra gate learned when training the expert models [60]. These approaches do not explicitly require labels, but they require knowledge of what domain is used to train each expert and assume that each expert is the best model for its corresponding domain, therefore effectively using this mapping form of labels. contrast, our setting focuses on routing among pre- pre-trained LLMs where we do not know what models are optimal what tasks and their samples.
second category of inference-time routing works studies how choose among collection of pre- trained LLMs, which is the setting that SMOOTHIE focuses on. Several approaches involve training meta-model that either scores or ranks how a LLM will perform on a sample [41, 74, 79], all of which required labeled data to train. MoRE [87] involves training simpler random forest classifier, using the rate of agreement among L.IMS as one of the features, which is similar to how Smoothie estimates scores; however, it also requires labeled data train the classifier. Some approaches [48, 86] do not
19

---

require training routers and simply use nearest neighbor methods. However, these nearest neighbor methods still use labeled data to determine what training samples each LLM performs the best on. [55] invokes a trained reward model for the routing mechanism. [21] trains a classification- -ased router using the BARTScore metric on LLM generations as pseudolabels; this avoids using manually labeled data, demonstrating that while majority of routing methods require labeled data, there exist some alternatives that do not. We leave it to future work to compare and integrate SMOOTHIE with other unsupervised approaches.
Finally, complementary to our setting are works that jointly focus on cost minimization as well as quality of generations. RouterBench [33] creates a benchmark for studying the cost- cost-quality tradeoffss in routing systems. Optimizing for cost can be done algorithmically. such as in FrugalGPT [13], AutoMix [58]. RouteLLM [63]. and [88], as well as via hardware enhancements such as SambaNova Systems' Composition of Experts [69].
LLM Ensembling A rich literature has observed that ensembling LLM outputs- across different prompts or base models- improve the accuracy of generated predictions. Prior work has proposed and studied a number of different ensembling algorithms for classification tasks, including majority-voting [54, 98], weak-supervision [2, 29], boosting [32, 68, 104], and others [49, 66, 80].
More relevant to our work here is a literature on ensembling for generative tasks. One category of models to "fuse" generations from different prompts or base LLMs [41]. Though recently applied in the context of modern LLMs, the concept of fusion traces back older work on summarization [5, 46, 47, 75]. Some techniques combine or switch among multiple outputs at inference time [34, 36, 59, 83, 97], while others involve averaging in weight space [35, 38, 95]. Lastly, ensembling can also be approximated by randomly selecting model to be used in multi-turn settings [56].
methods rely on auxilliary siguence-to-sequee
Other LLM Selection Algorithms Beyond the setting of selecting among multiple LLMs, other works have explored how to select the optimal prompt template from a collection of candidate prompts. These works can be grouped into two categories. The first category assumes that engineers have access to labeled data. In the naive case, this labeled data can simply be used to select the best performing prompt [45, 67, 77]. Another subset of this category focuses the setting where new prompts can be generated by selecting in-context demonstrations from a set of labeled samples (typically a small training set) [22, 53). Prior work has proposed different methods for identifying the optimal in-context demonstrations to use, depending on the sample for which the LLM is being used to produce a prediction for [11, 77, 90, 101, 105, 106]. The second category focuses on zero-label prompt selection methods, but solely for classification tasks [52, 89, 103J. Prior work here selects prompts on the basis of mutual information [89], agreement rates between predictions produced by different prompts [52], and various probability based measures [28, 57, 103].
Weak supervision SMOOTHIE utilizes techniques inspired by weak supervision literature. Weak supervision aims to programmatically generate labels on an unlabeled dataset by aggregating the predictions of several weak "voters". such as heuristics, pre-trained models, and knowledge graphs [71, 73]. It assumes a particular latent variable graphical model and uses its structure to estimate latent quantities, such as the accuracy of each voter (in setting, the quality score of each LLM). Typically, this graphical model is a binary Ising model, as weak supervision has generally been studied in classification settings [26. 721, where embeddings have been utilized as auxiliary signal but not modeled explicitly [15, 29]. Weak supervision has been applied to broader settings, such as for learning rankings, graphs, and manifolds [85, 94]. We derive our estimation procedure from the Gaussian model in [85], applying it to LLM embeddings. While both SMOOTHIE and [85] use a multivariate Gaussian model. in SMOOTHIE we apply model routing with SBERT embeddings on natural language datasets, whereas [85] conducts synthetic experiments in hyperbolic spaces and metric spaces induced by synthetic graphs. Moreover, SMOOTHIE uses nearest neighbor kernel smoothing to allow for sample-dependen weights-critice critical for nouting-wiile [85] calculates one global set of weights over the dataset.
Consistency- selection Consistency is central to unsupervised selection and aggregation methods, the simplest being majority vote. While weak supervision methods [26] and SMOOTHE heavily rely on notions of voter agreement as depicted in a graphical model, there are several other consistency-basee methods. Minimum Bayes Risk methods [6, 44] selects the generation that has the highest average similarity (i.c.. cosine) with other generations. This is similar to SMOOTHIE, which routes the lowest value of (3). If we ignore the subtraction of 65)(z) in (3) and average
20

---

over more than just (1) and d (z)), then SMOOTHIE with = 1 is equivalent to [44] Therefore, SMOOTHIE can be considered as slightly modified and more general version of this approach. Another approach [10] relies on consistency between "global" and "local" embedding for each generation. They solve an optimization problem that estimates each generation's quality score by constructing a loss that enforces that the similarity between the estimated true generation (produced by a weighted average candidates) and the candidate generation should be the same according to both global and local embeddings. In contrast, SMOOTHIE uses one embedding space, relies multivariate Gaussian structure among embeddings, and does not require gradient descent to learn the quality of each generation.
Temt-Time Compute Approaches like model routing, ensembling, and selection can all be seen as ways of utilizing time compute to produce higher-quality generations from a system of LLMs. Test-time compute can also be utilized over a single L.LM via techniques such as those used in OpenAl's o1, Chain of Thought, and Rephrase and Respond [20, 64, 99]. Other works have recently studied how test-time compute scales [9, 12] finddinigg that producing more generations can often yield the correct response-ane and how to combine multiple test-time methods, such as Archon [78]. It interesting future work to consider how SMOOTHHE can be integrated with other test-time compute techniques.
D Proof of Proposition 1
We provide a proof of proposition 1, which a direct property of multivariate Gaussians that is also presented in [85]. We first expand E [[];(x)= {=))]
- 2y
[1)((()) - -
(5)
E [1A[(z) =-)))) + (x) (z) 2E [(A(<*) - =x) (x)))) =x)==))((x)))
Let (xx) be the kth element of the (z(x) embedding, and similarly define = (2). Note that since is diagonal, we can write
Cov (A,x(z) - (x), (y,(()) [x)]
E [(Ai,(T)) xx)) (Ag,(()) (x()) -E [Mc((r)) (x) E (xa(()) =z((x)
=
for all l E [d]. Since = Õ, we thus have that E [(A,,(()) ==((x)) ()), = = 0 for all
1 E [d], which implies that E [(>((z) =- (x)) (Az(z) - 2x(r)] = 0. Plugging this into (5), we have
E [[A(2)) {+)0) [1A)(*) - (1))]] +E [1A)(()) =x))]]
(6)
E Additional Experiments and Details
This section contains additional details on experiments discussed in Section 5.
E.1 Datasets and Models
Table 5 provides links to the Huggingface datasets used for each task. For E2E, CNN//ailyMail, XSum, and Web NLG we measure performance using rouge2. For SQuAD, TriviaQA, and Definition Extraction we measure using "aceuracy." model generation treated as "correct" if contains the answer, and incorrect otherwise [1].
21

---

Dataset name
Huggingface URL
E2E
htcp:://huggingacccc.ccttttttsss
Maill
SQuAD
XSum TriviaQA Web NLG
Definition Extraction
Table 5: Datascts used.
Table 6 contains links for all models used.
Model name
Huggingface URL
Pythia-410M,
Pythia-IB
Pythia-2.8B
Pythia-6.9B
Gemma-2B
http:///nnggnnggeeeooooo//////b2
Incite-3B
Dolly-3B
Llama-2-7B
Mistral-7B
Vicuna-7B
Gemma-7B
Nous Capybara
Phi-2
https///unggingkeeecccohhoooooop
Llema-7B 7B
Table 6: Huggingface model URLs.
For the Alpaca leaderboard experiments, we run each trial by sampling models from the following
1, claude-2, --11..5t11BB-Chat, yi-large-prent set of 10: Meta-Llama- 2-w.-mmmma--------sstuttttt Instruct. Ein-70B-vo. mistral-large -arreel-1 2402, gemini-pre Storm-7B, Fsfairr- Fwfairx--ephyr--onen Pasfairr-Zzphhy--caat---...
E.2
Additional Results
CNN
Def. Ext.
E2E
SQuAD
TriviaQA
WetNLG
XSum
RANDOM
12.9
52.4
27.3
59.6
32.7
23.4
4.5
3B
SMOOTHIE SHOOTHIE-GGOBBL
14.3
61.5
31.8
60.7
32.1
30.7
4.5
MMMOTTIEE-G.OBBA..
14.3
61.5
24.7
60.7
32.1
30.7
4.5
BEST-ON- VAL
13.0
60.5
31.1
66.4
38.7
30.3
5.3
RANDOM
13.7
58.5
35.3
67.9
59.3
44.1
6.9
7B
SMOOTHIE- GLOBAL
14.5
70.9
36.9
76.2
68.3
45.9
8.4
SMOOTHIE-GLOBAL- -RAIN
14.5
70.9
36.5
76.2
68.3
45,9
8.4
BEST-ON-VAL VAL
14.5
69.4
36.7
74.0
65.8
48.3
8.3
Table 7: We compare SEOOTTIE--GLOBA. to
for which weights are learned on hold-out set. We provide results from baseline methods for referenee. Underlined values the best performing unsupervised methods. Bold values the best performing overall methods. We report rouge2 scores for CNN, XSum, WebNLG, and E2E, and accuracy for the rest. All metrics are scaled 0-100.
22

---

E2E
XSum
36
6.5
32
30
5.5
26
24
22
4.0
0.418
2.88
6.98
0,418
18
2,88
6.98
CNN
SQUAD
12
60
55
10
50
45
35
6.98
Figure 4: We compare RANDOM (blue) and SMOOTHI--GLOBB. (orange) for prompt-selection different sized models the Pythia suite. The x-axis denotes model size, and y-axis denotes performance (either rouge2 or accuracy).
3B
7B
Method
DISTR-ACC
DISTR- ROUGE2
DISTR-ACC
RANDOM
48.7
17.0
65.4
25.0
PAIRRM
53.9
19.0
71.8
25.5
LABELED-KNN
51.0
16.8
71.7
26.2
BEST-MODEL
52.3
18.1
73.2
26.4
GMOOTHIE-GLOBAL
51.3
18.1
66.5
26.1
LMOOTHIE-LLOCAL
58.7
20.2
75.0
26.9
SMOOTHIE-GLLOBAL-TRRAA
51.3
18.1
66.5
26.1
SMOOTHIE-LOCAL-TROOE
50.7
18.8
70.9
26.0
Table 8: We compare SMOOTHIE-LOCAL to SMOOTHIE-LOCAL-ttrin for which weights are learned on hold-out set, the 3B and 7B ensembles for multi-task distributions. DISTR-ACO and DISTR- ROSTR-ROUGE2 measured with accuracy and rouge2 respectively. Bold values indicate the best performing method for each dataset model size. Metrics are scaled 0-100. Other baseline methods are provided comparison.
23

---

3B
7B
Method
DISTR-ACC
DISTR-ROUGER
DISTR-ACC
DISTR-ROUGE2
RANDOM
48.7
17.0
65.4
25.0
PAIRRM
53.9
19.0
71.8
25.5
LABELED-KNE -
51.0
16.8
71.7
26.2
BEST-MODEL
52.3
18.1
73.2
26.4
SMOOTHIE-LOCAL (BGE-small [102])
59.3
19.7
74.6
27.1
SMOOTHIE-LOCAL (SBERT (76))
58.7
20.2
75.0
26.9
Table 9: Comparing SMOOTHE-LLCAL with different embeddings the 3B and 7B ensembles for multi-task
distributions. DISTR-ACE Ace and HISTR-ROUGES measured with accuracy and rouge2 respectively. Bold values indicate the best performing method for each dataset and model size. Metrics are scaled to 0-100.
Effect of Neighborhood Size
(Distr-Acc)
75
70
Score
65
60
55
1 5
10
20
50
100
no
Figure 5: We measure how SMOOTHIE-LOCAL performance DISTR-ACE ACC changes nu changes.
24

---

Effect of Neighborhood Size (Distr-Rouge2)
27
26
25
24
Score
23
22
21
20
20
50
100
Figure 6: We measure how SMOOTHIB-LOOCA. performance on changes as changes.
Rank of Model Selected by Smoothie-Global
200
175
150
Frequency
125
100
75
50
1
2
4
5
Rank
Figure 7: We evaluate SMOOTHIE- GLOBAL'S ability to identify the best model by randomly sampling 50 ensembles of size 4-7 LLMs from pool of the LLMs contained in the 3B and 7B ensembles. We apply SMOOTHIE-GLOBAR. select the best LLM within each these ensembles across the NLG tasks, and measure the rank (relative to the ensemble) of the LLM selected by
Method
CharGPT-Rank (1)
RANDOM
5.96
SMOOTIIE-GLOBAL
3.91
Table 10: Results for SMOOTHIE-LOBBA and baselines MixInstruct.
25

---

Method
Accuracy
RANDOM 28.4
VAL
37.5
SMOOTHIE-GLOBAN
37.5
Table 11: Results for SMOOTHIE-GLLOBLL and baselines on GSM8K. We report accuracy, with scores scaled to 0-100.
Rank of Model Selected by Smoothie-Global
Frequency
4
2
4
5
Rank
Figure 8: We construct a histogram over the rank the LLM selected by SMOOTHIE-GLOBAL across both the 3B and 7B ensembles, for 7 NLG tasks.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the conclusion.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide link a Github repository.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [NA]
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https::/neurrt http:/::///rrrseeeeeeeees
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
11. Safeguards
27

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
28


Reviews:
[{"review_id": "8bdvzWOHkZ", "reviewer": "Reviewer_RLiV", "summary": "This paper presents a model for routing an input to an LLM from a pool of LLMs. The aim is to estimate which LLM will produce highest quality generation without using any labeled training data in estimating the routing model. Instead, the approach relies on weak supervision to learn the parameters of a Gaussian graphical model. \n\nThe technique is evaluated by examining the model's ability to pick the oracle best LLM from the pool on various tasks; then by evaluating it's end-to-end routing performance compared to baselines (some of which have access to labeled data); and finally by using the same technique to pick amongst a set of LLM+prompt pairs for various tasks in order to effectively do prompt selection.", "questions": "- How would address the efficiency concerns mentioned in the weaknesses section?\n- Did you observe anything about the behavior of the routing across different tasks?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper makes creative use of the Gaussian graphical model from Shin et al. (2022) by using a separate embedding model (SentenceBERT) to produce an embedding representation of each {input, generation} pair from an LLM. The graphical model then provides a joint distribution over those embeddings plus the latent embedding of the {input, true output}. Inference in the model is straightforward and efficient and allows for both (1) training from unlabeled data and (2) estimation of the LLM scores conditioned on the observed input. Overall, the technique provides a simple, elegant application of Shin et al. (2022)'s model and could be easily replicated. The approach is clearly described.\n- The results on correlation of the graphical model quality estimates with the oracle best LLM are fairly high.\n- The end-to-end routing evaluation considers a breadth of tasks including both classification and generation problems. Overall the results tend to be on par or better than strong baselines, including those that use labeled training data for model selection.\n- Finally, the application to prompt selection is a nice addition demonstrating the applicability of the approach beyond the obvious routing application.\n- Each experiment considers a variety of models (as well as tasks) and most experiments include multiple pools (e.g. a 3B and 7B pool) to showcase that effects across different familys of LLMs.", "weaknesses": "- The paper has no discussion of efficiency, but this seems like an important point to address. This has important implications for the motivation: if one has the capacity to run a pool of small models, why wouldn't one instead run a single larger model. In this way, the paper should really address the tradeoff of accurracy/ROUGE with # of parameters (or even more realistically runtime). \n- The paper does very little to discuss its limitations. There are a few comments peppered throughout.\n- The paper does not mention anything about which LLMs are being picked. For example, is the job of the router trivialized by certain pools of models? For example, one could imagine a setting in which the router simply identifies and then always picks the best model. The paper does no analysis of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and for engaging with our work. We are glad to hear they appreciated our approach and the comprehensiveness of our experiments. We refer the reviewer to our general response for more information on (1) Smoothie’s routing behavior, and (2) Smoothie’s limitations. Our individual response to reviewer RLiV focuses primarily on the concerns regarding the efficiency of routers.\n\n**Why run a pool of small models as opposed to one large model?**\n\nReviewer RLiV asked about the tradeoff between spending resources on a single large model, as opposed to multiple small models. We believe there are several reasons why multiple smaller models might be preferred to a large model. \n\nFirst, an ensemble of small models may exceed the performance of a large model while incurring the same computational cost. For instance, we observe that applying Smoothie to an ensemble of three 1-2B parameter models (Qwen, Gemma, and Phi-3) outperforms each of the following 7B models by up to 20pts accuracy on SQuAD: Llama-2, Storm, Snorkel, Vicuna, Nous-Capybara, and Mistral. \n\nSecond, engineers increasingly access models through APIs, and API calls for larger models can often be 4-5x the cost of API calls to smaller models. On Together for instance, Llama-3 8B costs 10c per million tokens and Llama-3 70B costs 54c per million tokens. Running an ensemble of five 8B models thus costs less than a single 70B model.\n \n\n**Can Smoothie be made more efficient?**\n\nReviewer RLiV also asked whether it was possible to incorporate more discussion of Smoothie’s efficiency. To estimate the Smoothie weights for routing, we use a simple closed-form procedure **that does not require any SGD or training**, as described in Algorithm 1. As a result, Smoothie weights on the entire dataset can be computed in seconds—for the 7B ensemble, SmoothieLocal on the multi-task datasets takes 2.14 seconds per 1000 samples, and SmoothieGlobal on the single-task datasets takes under 0.03 seconds per 1000 samples. Moreover, Smoothie **does not require any ground-truth annotations**; however, all m model generations per test sample are needed as input to the algorithm. That is, we need $n \\times m$ generations for a test dataset of n samples.\n\nFortunately, the need for computing all model generations per test sample can be removed with a small algorithm tweak, making Smoothie even more efficient and its runtime independent of $n$. Suppose we have a held-out set of $n_{train}$ train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm 1, KNN is now over a held-out training dataset). The benefit here is that Smoothie selects the model for a test sample without needing model generations for that sample. Only $n_{train} \\times m$ generations are needed, regardless of how large the test dataset $n$ is.\n\nSmoothie is still effective with this modification, which we refer to as Smoothie-Train. For instance on the 3B and 7B ensemble (using a train set of size 250, compared to test sets of size 1000), Smoothie-Train identifies the best performing model on 8/14 single-task datasets, outperforming a random selection baseline by up to 7.1 points rouge2 and 12.5 points accuracy. On the multi-task datasets across both the 3B and 7B ensemble, Smoothie-Train outperforms random sampling by an average of 3.7pts accuracy on the accuracy tasks, and by an average of 1.6pts rouge2 on the rouge2 tasks.\n\nWe will update the paper to include this discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ua8zVRHWtg", "reviewer": "Reviewer_r3vp", "summary": "- This work proposes a method called SMOOTHIE, which can route label-free test examples to LLMs. Specifically, \n  - it employs a latent variable model and Gaussian distribution for efficient quality score estimation and uses LLM outputs to estimate generator quality. \n  - it estimates specific to each test sample using nearest neighbors and routes samples to LLMs with highest quality scores. \n\n- Empirical results show that:\n1. SMOOTHIE's learned quality weights correlated with actual LLM performance.\n2. In mixed-tasks datasets SMOOTHIE is able to route different samples to different LLMs which boosts the performance.\n3. SMOOTHIE can be used for prompt-selection.", "questions": "- In the algorithm, it is not quite clear how to select indices j and k different from i. Is it just random selection?\n- How do you compare SMOOTHIE with Minimum Bayes method, where you choose the output from an LLM that aligns the most with all others?", "limitations": "- The authors did not address the limitations \n- There is no negative societal impact", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- With the recent advances in LLM research, how to choose the best LLM/ how to select the best prompt for different tasks is an interesting topic and this work proposes an approach to deal with this problem. \n- The empirical results show that SMOOTHIE does help increase the performance for a dataset with mixed-tasks for different embedding models and different neighborhood sizes chosen.", "weaknesses": "- In figure 3. there is no label on x-axis\n- typo: in Section 5.2, line 263 and 264, the \"SMOOTHIE-independent\" be SMOOTHIEGLOBAL instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are happy that they found the topic interesting. In our global response, we discuss the limitations of Smoothie, which the reviewer pointed out. Below, we address the reviewer’s comments on writing clarifications as well as the Minimum Bayes method.\n\n**Writing clarifications**: We will update the draft to address the writing errors mentioned in the weaknesses section. For selecting j and k in the Smoothie algorithm, they can be randomly selected, although to reduce variance we average over all $C(m-1, 2)$ pairs of (j, k) to get $i$’s Smoothie weight. We will clarify this in our paper.\n\n\n**Minimum Bayes Risk (MBR)**: One way of minimizing Bayes risk is to select an output that has the highest similarity to all other outputs (such as [1], mentioned in [2]). That is, the model to route to for sample $x$ is $\\arg \\max_i \\sum_j sim(g_i(x), g_j(x))$, where $sim$ is cosine similarity, for instance.\n\nWe demonstrate theoretically and empirically that Smoothie-Local with k=1 (no nearest neighbor smoothing) is conceptually similar to MBR. Note that Smoothie routes to the model with the lowest value of equation 3 in the paper. If we ignore the subtraction of $\\delta_{jk}(x)$, Smoothie selects the model $\\arg \\min_i \\sum_j \\delta_{ij}(x)$; for k=1, this is the generation with the lowest embedding distance from all other generations for x. Since L2 distance is inversely correlated with cosine similarity, Smoothie-Local (k=1) is hence similar to MBR. Most importantly, we found that MBR and Smoothie-Local (k=1) matched performance on both multi-task datasets and both ensembles (a 0.0003 point average difference).\n\nTherefore, Smoothie is a more general version of the MBR rule above that can almost exactly recover MBR’s behavior when k=1. Moreover, we find that Smoothie performs better than MBR for 4 out of 7 single-task datasets when using a larger k. We will update our draft to compare to MBR.\n\n\n[1] https://aclanthology.org/D18-1449/\n\n[2] https://arxiv.org/pdf/2310.01387", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sRcatKH6TX", "reviewer": "Reviewer_Cr26", "summary": "This paper proposes a method for selecting LLMs' responses for generative tasks.\n\nIt can essentially be viewed as a \"truth inference\" problem in the research community of \"weak supervision\" and \"crowdsourcing\"; unlike ordinary truth inference methods, this paper takes into account unstructured textual information and, for each sample, the method in this paper needs to pick one out of all candidate answers.", "questions": "Please refer to the \"main concerns\" above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) First, the problem investigated in this paper---selecting LLMs' responses for generative tasks---is very practical and interesting.\n\n2) The proposed method is generally very intuitive and I think it will be effective.", "weaknesses": "Main concerns:\n\n1) First, authors should summarize the most relevant works in the main text of the paper. For this current version, I need to scrutinize the most relevant works in the appendix.\n\n2) Related to the above point, there are already works that focus on the task that this paper addresses, e.g. \"An error consistency based approach to answer aggregation in open-ended crowdsourcing\". \n(Although this paper focuses on crowdsourcing workers rather than LLMs.)\n Therefore, it would be advisable for the authors to research the relevant literature in more depth and to consider them in experiments as comparison methods.\n\n3) The core theoretical part of this paper is applied to the content in the existing work [72]. In the main text, it is necessary to state the technical and theoretical differences with the existing work [72] more clearly and in detail.\n\nSome other concerns:\n1) The meaning of some symbols is not explained, e.g., line 109.\n2) In this paper, a graphical model is presented, then a graphical representation of this graphical model should be shown.\n3) Typos. E.g., \"x(Section 4.2)\" in line 134, \"$\\theta_i(x)$s\" in line 137.\n4) The quality of the images (e.g. sharpness) can be further improved.\nAlso, in Figure 2, the name of the proposed method should be shown in capital letters.\n5) The tables are not self-explanatory enough, e.g. the metric of interest is not shown in Table 1.\n6) Some of the presentations are confusing, such as lines 609 and 610.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and are glad that they found the problem interesting and practical. Below, we address the reviewer’s concerns around related work and baselines, differences with Shin et. al., and writing clarifications.\n\n**Related work**: Thank you for your suggestion on moving more related works up into the body. We will add relevant works on routing and ensembling to the body in our updated draft. \n\nWe compare and evaluate the AEC method presented in the paper mentioned in W2. In this paper, each generation has two embeddings, a global embedding (Universal Sentence Encoder) and a local one (GLEU). They solve an optimization problem that estimates each generation’s error by constructing a loss that enforces that the local and global embedding similarity between the true generation (produced by a weighted average) and the candidate generation should be the same. In contrast, Smoothie uses one embedding space, relies on a multivariate Gaussian structure among embeddings, and does not require gradient descent to learn the weight parameters. \n\nTo evaluate AEC, we implemented it ourselves since we were unable to find a codebase online. We used 100 epochs and a learning rate of 0.001 for all datasets. We find that Smoothie outperforms AEC on multi-task datasets by 0.9 points on average, and on single-task datasets by 2.3 points on average across 3B and 7B ensembles. We will update our draft with the discussion and empirical results on AEC.\n\nFinally, in the spirit of the reviewer’s requests for additional baselines, we also report a comparison to PairRM. We refer to the global response for a description of this baseline’s performance.\n\n**Comparison with Shin et. al.**: Both Smoothie and Shin et. al. use a multivariate Gaussian model. However, in Smoothie we apply it to model routing with SBERT embeddings on natural language datasets, whereas Shin only conducts synthetic experiments in hyperbolic spaces and metric spaces induced by synthetic graphs. Moreover, Smoothie uses nearest neighbor kernel smoothing to allow for sample-dependent weights—critical for routing—while Shin only calculates one global set of weights over the dataset. We will add this comparison to the related work in the body of our paper.\n\n\n**Writing clarifications**: We thank you for pointing these writing errors out and apologize for them. We will address them in our updated draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPCK0VU81K", "reviewer": "Reviewer_gxgv", "summary": "This paper proposes a label-free routing method, Smoothie, to route an ensemble of LLMs without annotated data. Smoothie constructs a latent variable graphical model over semantic embedding representations of observable LLM outputs and the unknown ground truth, estimates the sample-independent quality scores of each LLM, and routes to the LLM with the highest quality score. Experimental results indicate that Smoothie competently performs several generation tasks over the 3B and 7B ensembles.", "questions": "1. (Refer to W1 and W2) Can you provide the performance statistics of the models used in the experiments?\n2. Does Smoothie require more computation than the supervised methods?", "limitations": "The limitations and broader impacts are not discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Smoothie utilizes a latent graphical model and an embedding language model to estimate the quality of each separate LLM without labels, thus is more applicable than the supervised routing metods.\n2. The experimental results indicate that Smoothie can outperform other supervised routing methods on several generation tasks and can also conduct prompt selection.", "weaknesses": "1. The method relies on the embedding method to measure the semantic similarity, which limits the scope of Smoothie to semantic-related tasks. The capability of LLMs should vary more on tasks such as mathematical reasoning, but these tasks depend less on semantic similarities.\n2. The experimental comparisons are unclear. The chosen metrics are highly abstractive. Given the performances of each LLM are unknown, it's hard to identify whether the method successfully routes to the best or slightly better model than the baselines. The presentation of the experiment section is also hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and for engaging with our work. We are grateful they appreciated (1) the generality of Smoothie’s algorithm beyond supervised methods, and (2) the breadth of experimental results. \n\nWe refer the reviewer to our general response for more information on the performance statistics for individual models in the experiments (W2), as well as a discussion of the limitations. We apologize for the lack of presentation clarity in the current version of the experimental section. We hope that revisions in response to this review and others will improve clarity. \n\nOur response here addresses concerns regarding (1) the types of tasks Smoothie can be used for (W1), and (2) whether Smoothie requires more computation than supervised methods (Q2).\n\n**Performance on non-semantic tasks**\n\nThe reviewer noted that because Smoothie relies on embeddings of model generations to learn quality scores for each model, Smoothie is inherently limited by the types of relationships those embeddings can capture. We agree with this observation, and have mentioned it in the limitations in the global response. However, we find that current embedding models do allow Smoothie to perform well on tasks like mathematical reasoning, as shown by our evaluation on GSM8K in the global response.\n\n**Does Smoothie require more computation than supervised methods?**\n\nSupervised methods require engineers to expend computation in two ways. First, engineers must produce generations for the ensemble over a training set. Second, engineers must train a router–typically using SGD–to map queries to models in the ensemble. At test time, supervised routers only expend computation performing inference for the selected model, i.e., one model out of the ensemble. \n\nSmoothie does not require the creation of a large training dataset, and operates solely on test-time generations without annotations. Fitting Smoothie’s weights is significantly cheaper than training a router via SGD, with our method taking only seconds in practice due to their closed form in Algorithm 1. However, Smoothie requires a generation from each model in the ensemble for the test sample, which supervised methods do not. \n\nFortunately, the need for computing all model generations per test sample can be removed with a small algorithmic tweak, making Smoothie even more efficient and its runtime independent of $n$. Suppose we have a held-out set of $n_{train}$ train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm 1, KNN is now over a held-out training dataset). The benefit here is that Smoothie selects the model for a test sample without needing model generations for that sample. Only $n_{train} \\times m$ generations are needed, regardless of how large the test dataset $n$ is.\n\nSmoothie is still effective with this modification, which we refer to as Smoothie-Train. For instance on the 3B and 7B ensemble (using a train set of size 250, compared to test sets of size 1000), Smoothie-Train identifies the best performing model on 8/14 single-task datasets, outperforming a random selection baseline by up to 7.1 points rouge2 and 12.5 points accuracy. On the multi-task datasets across both the 3B and 7B ensemble, Smoothie-Train outperforms random sampling by an average of 3.7pts accuracy on the accuracy tasks, and by an average of 1.6pts rouge2 on the rouge2 tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We are glad that reviewers found the Smoothie algorithm to be elegant (RLiV, Cr26, gxgv), recognized the practical applications of this work (r3vp, Cr26), and appreciated the breadth of our evaluation (RLiV).

Our global response (1) discusses results on new datasets and baselines showcasing Smoothie’s performance, (2) provides more details on Smoothie’s routing behavior, and (3) describes Smoothie’s limitations. 

## Results on new datasets and baselines

__Comparison to PairRM__: We benchmark Smoothie against PairRM [1], a popular pre-trained reward model used in prior work for both generation selection and routing. Given one or more responses to an instruction, the PairRM scores are used to rank the responses by relative quality. 

On the multi-task accuracy dataset, Smoothie-Local outperforms PairRM for both the 3B and 7B ensemble, by an average of 1.6pts accuracy. On the multi-task rouge2 dataset, Smoothie-Local slightly outperforms PairRM by an average of 0.2pts rouge2. That Smoothie-Local can match and even outperform PairRM is notable because PairRM is a supervised method, since it requires annotated pairwise data to train. In contrast, Smoothie requires no annotations.

__Performance on GSM8K__: Reviewer gxgv asked whether Smoothie could be applied to mathematical reasoning tasks. We run Smoothie-Global on GSM8K, a benchmark consisting of grade-school level mathematical word problems. In order to apply Smoothie-Global, we prompt models to produce a chain-of-thought style generation culminating in the final numeric answer. We evaluate Smoothie-Global on the 7B ensemble, and find it  successfully identifies the best performing model in this ensemble, producing an accuracy/solve-rate of 37.5%. In contrast, a random-selection baseline over the ensemble produces a solve-rate of 20.1%, and using PairRM to select from within the ensemble produces a solve-rate of 27%.


__Performance on MixInstruct__: We evaluate Smoothie-Global on MixInstruct [1], a dataset consisting of instructions and corresponding responses from 12 models, along with relative rankings of response quality. Smoothie-Global successfully identifies the best average model from the 12 ensemble models.


We will update our paper to include these results.

## Routing behavior

Reviewers RLiV and gxgv asked for more details regarding Smoothie’s routing behavior. Specific details requested include: 
- The performance of individual models.
- How often Smoothie picks different models across samples.
- The relative quality of the model selected by Smoothie for each sample.

We will update our draft to include the individual performance of each model for all datasets (single task and multi-task). We briefly summarize the important findings below, and include corresponding visualizations in the attached PDF. 

**(1) Smoothie-Local picks different models across multitask datasets**

For both multi-task datasets for both the 3B and 7B model groups, we observe that Smoothie-Local selects every model in the ensemble for at least one sample. The least selected model was selected 8.7% of the time, and the most selected model was selected 43% of the time. More information is in Figure 1 of the PDF.

**(2) Smoothie-Local improves upon the best ensemble model**

Smoothie-Local’s selection of different models improves performance. Smoothie-Local outperformed the best model in the ensemble for both multi-task datasets across both the 3B and 7B groups, by as much as 7pts accuracy and 1.6pts rouge2.   

**(3) Per-sample, Smoothie-Local frequently selects the best model available**

For each of the multi-task datasets, we study whether the model generation selected by Smoothie-Local was the best generation that Smoothie-Local could have selected. In Figure 2 of the PDF, we provide the distribution of the per-sample rank of the generation that Smoothie selects. On the rouge2 dataset, we find that Smoothie selected the best generation possible for 36% of samples for the 3B ensemble, and for 27% of samples for the 7B ensemble. Smoothie’s selected generation was better than the median for 72% of samples for the 3B ensemble and for 79% of samples for the 7B ensemble. On the accuracy multi-task dataset, Smoothie selected the best possible generation for 85% of samples for the 3B ensemble, and 89% of samples for the 7B ensemble. Smoothie’s selected generation was better than the median for 99% of samples for both the 3B and 7B ensemble.

In short: we find that Smoothie selects a diverse array of models, and that selection of multiple models improves performance.


## Limitations

Reviewers RLiV, r3vp, and gxgv noted that our submission did not mention Smoothie’s limitations. We apologize for this omission. We discuss limitations of Smoothie below and include them in the Discussion section of our updated draft.

1. The multivariate Gaussian model uses a diagonal covariance matrix. Smoothie thus assumes that the error vector for each generation is independent, although Smoothie can be extended to learn and account for dependencies [2, 3]. 
2. Smoothie only optimizes for performance, not cost. Recent works focus on the cost-performance tradeoff when routing among large costly models and small cheaper models. We consider cost optimization as future work.
3. Smoothie relies on embeddings, which may only capture particular aspects of semantic similarity among generations, to determine quality. Other embedding models and additional heuristics could be used to create richer input features for Smoothie.  

[1] https://arxiv.org/abs/2306.02561

[2] https://arxiv.org/abs/1810.02840

[3] https://arxiv.org/abs/1903.05844

Meta-review: This paper introduces a method for routing inputs to the most suitable LLM from a pool of models. The goal is to predict which LLM will produce the highest-quality output without relying on labeled training data for the routing mechanism. Instead, the approach utilizes weak supervision to learn the parameters of a Gaussian graphical model with latent variables.

The method is interesting, and the results appear promising. During the rebuttal, the authors adequately addressed most of the concerns raised by the reviewers. Therefore, I vote for acceptance.
Decision: Accept (poster)