Smoothie: Label Free Language Model Routing

Authors: neel  guha
mayee f chen
trevor  chow
ishan s khare
christopher  ré
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:13:39

Abstract: Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. _routing_). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform _unsupervised_ routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown “true” outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.

Main paper: 
 introduction: 
 Large language models (LLMs) are increasingly being deployed in multi-capability regimes where data inputs may span a diverse range of tasks, each of which requires different capabilities [8] . For instance, an LLM-powered chatbot may be asked to write code, answer questions about different domains, summarize documents, perform extraction, and more [3, 8, 14, 30] . One challenge is that while engineers often have access to numerous pre-trained LLMs (i.e., through Huggingface or various APIs), they do not know which LLM is optimal for each possible user input [86] . Because the quality of generations can significantly vary across LLMs, choosing the right LLM for each input sample is important for ensuring high task performance [41] .Recent work has explored various ways to utilize ensembles of pretrained LLMs in multi-capability settings, by (1) collecting a diverse pool of LLMs and (2) identifying which LLM to route each sample to [55, 86] . However, the majority of existing approaches require labeled data; engineers typically either (1) train an auxiliary model using labeled data to rank or predict which LLM each sample should be routed to [41, 79] , or (2) directly use labeled data to determine which LLM is the best on average [86] . As a result, engineers designing routing protocols face the practical difficulty of constructing labeled datasets.Given a candidate pool of LLMs and an unlabeled test dataset, this paper explores how to best select LLM outputs for each sample in an entirely unsupervised manner-without labeled data, or models trained on labeled data. To make progress in addressing this question, we face two technical challenges:• Unknown LLM quality: The first challenge is estimating the quality of each LLM. Access to labeled data allows engineers to identify higher performing LLMs by measuring the accu- racy/quality of LLM outputs. In this paper, we study the question of how to estimate quality without labeled validation data.• Sample-conditional generator performance: The second challenge is determining how to select the best LLM for each individual test sample. LLM outputs can vary in quality over different samples, which could render population-level estimates of LLM quality misleading.In this work, we propose SMOOTHIE, a method for routing samples to LLMs in a label-free manner (Figure 1 ). Below, we describe how SMOOTHIE addresses the two challenges described above.• Quality estimation: Using the LLM outputs for each test sample as "voters," SMOOTHIE estimates the quality of each generator using methods from Weak Supervision (WS). Concretely, SMOOTHIE constructs a latent variable graphical model over observable LLM outputs and an unknown true output. By modeling the embedding vector difference between each LLM output and the true output as a multivariate Gaussian, we can derive a closed-form estimator adapted from [85] for learning LLM quality scores efficiently.• Conditioning: We condition these quality estimates to be particular to a given test sample by only using the nearest neighbors of a test sample in the training data as inputs to the estimator (i.e., kernel smoothing). We then route each test sample to the LLM with the highest quality score estimate on that sample. We call the version of SMOOTHIE that produces quality estimates using all available test data SMOOTHIE-GLOBAL, and we call the version that uses a sample's nearest neighbors SMOOTHIE-LOCAL.We empirically evaluate SMOOTHIE in three stages.• LLM selection: First, we assess SMOOTHIE-GLOBAL's ability to identify-from an ensemble of mixed quality LLMs-the optimal LLM for a given task overall. On traditional generation tasks such as summarization, reading comprehension, and data-to-text generation, we find that SMOOTHIE-GLOBAL's learned LLM quality-weights correlate with actual LLM performance (ρ = 0.72)), and on the AlpacaEval benchmark, SMOOTHIE-GLOBAL identifies the best-performing instruction model 70% of the time [50] . The highest quality LLM identified by SMOOTHIE-GLOBAL-all computed without labeled data-can beat random-selection by up to 15 points win-rate on AlpacaEval, and by up to 8 points on SQuAD.• Routing: Second, we study whether SMOOTHIE-LOCAL's sample-conditional scoring mechanism allows it to route samples in mixed-task datasets to higher-performing LLMs (i.e., the multicapability regime). We find that SMOOTHIE-LOCAL can improve the quality of produced generations by up to 7 points accuracy over SMOOTHIE-GLOBAL, and that SMOOTHIE-LOCAL outperforms baseline unsupervised routing methods by up to 10 points accuracy and supervised routing methods by up to 5.0 points accuracy.• Prompt selection: Finally, we assess whether SMOOTHIE's quality-estimation mechanism can be applied to select the optimal prompt template in a candidate pool while using a fixed LLM. We find that SMOOTHIE-GLOBAL can outperform other prompt selection approaches by up to 18 points, allowing a 410M parameter model to match the performance of 6.9B parameter model. 
related work: 
 We provide an abbreviated related work, with a full treatment in Appendix C.Routing Routing has been classically utilized in Mixture-of-Experts models [25, 37, 42, 82] , which involve jointly training a set of models as well as a router. Recently, routing mechanisms have been used at inference time to decide which pre-trained LLM to use for a given sample [79] . Some approaches involve training an auxiliary model using labeled training data to either score or rank the performance of each LLM on each test sample [38, 74] . Others do not involve training a model but instead use nearest neighbor methods, selecting the LLM that does the best on a test sample's labeled neighbors [48, 86] . In contrast, SMOOTHIE does not require any labels.Ensembling Ensembling is another way of utilizing a pool of LLMs. Existing work has primarily focused on ensembling outputs for classification tasks [2, 68, 98] . Ensembling generative outputs typically requires training an auxiliary model [41] , combining or switching among outputs when decoding [36, 83] , or averaging in weight space [95] .Prompt selection In addition to selecting the best LLM for a sample, prior works have studied how to select the best prompt or in-context examples. While the simplest approach is to use a held-out labeled dataset [67] , there are also retrieval-based approaches to selecting the best in-context examples [90] , as well as approaches based on mutual information [89] and probability-based measures [103] , although the latter two are limited to classification.Weak supervision SMOOTHIE utilizes statistical techniques inspired by weak supervision, which programmatically generate labels for an unlabeled dataset by aggregating the predictions of several weak "voters" via a latent variable graphical model [71, 73] . Weak supervision has mostly been studied in classification settings [26, 72] but more recently has been extended to tasks such as learning rankings and manifolds [85, 94] . We derive our estimation procedure from the Gaussian model in [85] , applying it to LLM embeddings and the routing setting.The LLM with the highest scores is selected, and its output is used as our final prediction for x (Section 4.2).We describe each step in the following sections. 
problem setup: 
 Let V be the token vocabulary space, and let V = V ×• • •×V be the space of all vocabulary sequences. We consider a generative task with input text x ∈ X ⊂ V and reference output text y ∈ Y ⊂ V. We have a candidate pool of m LLMs, G = {g 1 , . . . , g m }, where each g i ∈ G : X → Y produces a generative output sequence g i (x) for a given input text sequence x. We are given an unlabeled test datasetD test = {x i } n i=1, where the ground-truth reference outputs are unknown. Our goal is to route each sample x ∈ D test to one of the LLMs in G. Specifically, we wish to construct a router route : G m × X → G that selects the LLM that yields the highest quality generation on x for each test sample x, without any labeled data.We present a probabilistic graphical model (see Figure 1 (center)) that describes how the LLM outputs, g 1 (x), . . . , g m (x), are related to a true output y in terms of each LLM's quality on a given input x, which we call θ i (x), corresponding to each g i (x). Let z g0 : V → R d map from a sequence of tokens to a d-dimensional embedding using a common model g 0 such as SentenceBERT [76] . Define λ i (x) := z g0 ([x, g i (x)]) to be the observable embedding of x and the LLM output, and define z ⋆ (x) := z g0 ([x, y]) to be the latent ground-truth embedding of x and reference output y. Similar to the approach in [85] , we model the distribution over embedding vectors, Pr(z ⋆ (x), λ 1 (x), . . . , λ m (x)|x) asPr(z ⋆ (x), λ 1 (x), . . . , λ m (x)|x) = 1 Z exp m i=1 -θ i (x)∥λ i (x) -z ⋆ (x)∥ 2 (1)where Z is the log partition function and the θ i (x)s-the LLM quality scores-are canonical parameters of the graphical model. Intuitively, our model captures LLM quality by supposing that if g i is of high quality and θ i (x) is very large, then it should be unlikely for the LLM output to be very 2: For all x ∈ D test and g i ∈ G, obtain the generator output ⃗ g i (x) and embed the input and generator output using model g 0 to get embedding λ i (x) := z g0 ([x, g i (x)]).3: for x ∈ D test , g i ∈ G do 4: for j, k ̸ = i ∈ [m] do 5: Compute δij (x) = 1 n0x ′ ∈NNn 0 (x) ∥λ i (x ′ ) -λ j (x ′ )∥ 2 , and similarly δik (x) and δjk (x).Set θjk i (x) = d/( δij (x) + δik (x) -δjk (x)).Compute averaged estimate θi (x) = 1 ( m-1 2 ) j,k̸ =i θjk i (x).8: return θi (x) for all x ∈ D test , g i ∈ G.different from the true output in terms of Euclidean distance in embedding space. Conversely, if θ i (x) is small, we assign larger probability to the setting where λ i (x) and z ⋆ (x) differ significantly. Finally, note that this graphical model corresponds to a multivariate Gaussian. That is, the vector (x) . Intuitively, this means that the average difference vector between each λ i and z ⋆ (x) is centered, with its magnitude inversely proportional to the LLM score θ i (x) and independent of other LLMs. Given this probabilistic graphical model, our goal is to learn each quality score θ i (x) from the unlabeled test dataset and use these for improved routing.[λ 1 (x) -z ⋆ (x), . . . , λ m (x) -z ⋆ (x)] ∈ R dm is Gaussian with mean µ = ⃗ 0 and a diagonal covariance matrix Σ ∈ R dm×dm with Σ jj = 1 2θ ⌈j/m⌉ 
method: 
 Given an unlabeled test dataset D test and a pool of LLMs G, SMOOTHIE consists of two steps:1. Estimation: The LLM quality scores θ 1 (x), . . . , θ m (x) are learned for each x ∈ D test (Section 4.1, Algorithm 1).We describe how to estimate each θ i (x)s in the graphical model in (1) using only unlabeled data from D test . Then, we describe how the LLM score estimate can be instantiated to be sample-conditional.Computing θ i (x) Below, we state a simple property arising from the fact that (1) corresponds to a multivariate Gaussian with a diagonal covariance matrix.Proposition 1 [85] For any i, j ∈ [m], it follows from the graphical model in (1) thatEQUATION)The proof is in Appendix D and relies on the fact that off-diagonal entries of Σ are 0. Note that the left hand side of the equation is observable while the two expectations on the right are unknown. We can apply this equation to pairs of LLM embeddings over a triplet of λ i , λ j , λ k to form a system of three equations with three unknown expectations. Solving, we haveEQUATIONwhere δ ij (x) = E ∥λ i (x) -λ j (x)∥2 . Since (1) is a multivariate Gaussian with Σ jj = 1 2θ ⌈j/m⌉ (x) , we can write θ i (x) as the following function of E ∥λ i (x) -z ⋆ (x)∥ 2 :EQUATIONwhere λ i,j (x) and z ⋆ j (x) are the jth indices of the embeddings λ i (x) and z ⋆ (x) respectively. Therefore, we can writeθ jk i (x) = d δij (x)+δ ik (x)-δ jk (x), where each δ ij (x) can be estimated using the LLM outputs on D test , and in practice in Algorithm 1 we estimate θ i (x) by averagingθ jk i (x) over all m-1 2 pairs of j, k ̸ = i. Sample-conditional estimation of θ i (x) Note that the expectation in δ ij (x) = E [∥λ i (x) -λ j (x)∥] is over the randomness in λ i (x), λ j (x) conditioned on a fixed point x.However, we only have one sample per x. One simple approach is to use the entire dataset to estimateθ i (x), i.e., δij (x) = 1 n x ′ ∈Dtest ∥λ i (x ′ ) -λ j (x ′ )∥ 2 .We denote this as SMOOTHIE-GLOBAL. However, in SMOOTHIE-GLOBAL each θ i (x) for i ∈ [m] is a constant over the entire D test . Therefore, we use nearest neighbor kernel smoothing to estimate each δ ij (x) in a sample-dependent manner, an approach we call SMOOTHIE-LOCAL. Concretely, for x ∈ D test , define NN n0 (x) ⊂ D test as the n 0 < n nearest neighbors of x (excluding x itself) in f 0 's embedding space. Then, we construct δij(x) = 1 n0 x ′ ∈NNn 0 (x) ∥λ i (x ′ ) -λ j (x ′ )∥ 2 ,and do the same for δik (x), δjk (x) to get a sample-conditional estimate of θ i (x). The procedure for estimating θ i (x) in SMOOTHIE-LOCAL is outlined in Algorithm 1.Once we have estimates of θi (x) for each of the m generators by using Algorithm 1, we can construct our route() function. We define route(G, x) = g i where i = arg max{θ 1 (x), . . . , θ m (x)}, which selects the highest scoring LLM for input x based on θi (x). We apply this on D test to determine the best LLM for each input sample. 
results: 
 We empirically analyze SMOOTHIE-GLOBAL and SMOOTHIE-LOCAL, focusing on four questions:1. How well does SMOOTHIE-GLOBAL recover ground-truth LLM rankings over samples belonging to the same task (Section 5.1)? 2. In multi-task datasets, how well can SMOOTHIE-LOCAL perform unsupervised-routing, by identifying the best LLM for each sample (Section 5.2)? 3. Can SMOOTHIE-GLOBAL and SMOOTHIE-LOCAL be applied to select from or route between different prompts (Section 5.3)? 4. How does SMOOTHIE-GLOBAL and SMOOTHIE-LOCAL's performance change as a function of different algorithmic choices (Section 5.4)?Setup We begin by evaluating whether SMOOTHIE-GLOBAL can accurately learn the relative performance of different LLMs on a single task-dataset. We study three categories of tasks. First, we consider 7 datasets corresponding to commonly-studied natural language generation (NLG) tasks [51] : CNN/DailyMail and XSum (summarization), SQuAD (reading comprehension), TriviaQA (factual recall), E2E and WebNLG (data-to-text generation), and LegalBench's Definition Extraction (text extraction) [1, 27, 30, 31, 43, 61, 62, 70, 81, 84] . We report Rouge2 for summarization and data-totext generation tasks and accuracy for all others. For all tasks other than Definition Extraction we evaluate SMOOTHIE-GLOBAL on a 1000 sample subset. 2 For these tasks, we consider two ensembles of LLMs at different size points. At the 3B size point, our ensemble consists of Pythia-2.8B [7] , Gemma-2B [91] , Incite-3B [17] , and Dolly-3B [18] . At the 7B size point, our ensemble consists of Llama-2 [92] , Mistral [40] , Vicuna [107] , Gemma-7B [91] , and Nous Capybara [19] . We manually write a single prompt template for each task, and all model generations rely on this template.Second, we consider two instruction-following benchmarks: AlpacaEval and MixInstruct [23, 24, 41, 50] . For AlpacaEval, we rely on responses accessible via the online leaderboard. 3 We identify 10 LLMs (each from a different base family), and download these models' responses to the AlpacaEval instructions. We conduct 10 different simulations, where in each simulation we randomly select 5 LLMs from our pool to function as an ensemble. Reported win-rates use the standard GPT-4 references. For MixInstruct, we use generations from an ensemble of 11 different LLMs originally studied in [41] . Following [41] , we measure generation quality using a ChatGPT-based rank.Finally, we consider a more "reasoning-intensive" task, GSM8K [16] . We consider an ensemble of three models: Gemma-7B, Phi-2 [39] , and Llema-7b [4] . We prompt each model to provide a chain-of-though reasoning [100] , and apply SMOOTHIE to these generations.For all datasets, we apply SMOOTHIE-GLOBAL using SentenceBERT (all-mpnet-base-v2) embeddings of generations [76] .We first measure how frequently the highest-weighted LLM according to SMOOTHIE-GLOBAL corresponds to the best-performing LLM in the ensemble. We observe that SMOOTHIE-GLOBAL selects the best-performing LLM for 4/7 tasks on the 3B ensemble, and for 5/7 tasks on the 7B ensemble (Figure 8 ). On AlpacaEval, SMOOTHIE-GLOBAL selects the best-performing LLM by win-rate for 8/10 ensembles, and the best performing LLM by length-controlled win-rate for 7/10 ensembles. On MixInstruct and GSM8K, SMOOTHIE-GLOBAL again identifies the best-performing LLM in the ensemble.Second, we measure how well SMOOTHIE-GLOBAL captures quality differences between LLMs in the ensemble, by computing the Spearman's rank correlation coefficient between θ i and ground truth quality scores ensemble models. Overall, we find that SMOOTHIE-GLOBAL's learned weights approximate the relative ordering of model quality well. On the NLG tasks SMOOTHIE-GLOBAL we measure an average correlation coefficient (across both ensembles and seven tasks) of 0.72. Figure 2 (a) visually depicts the distribution of task coefficients-on only one ensemble/dataset pair is there a correlation coefficient ≤ 0. On MixInstruct, we observe a correlation coefficient of 0.94, and on AlpacaEval, we observe a correlation coefficient of 0.46.Finally, we measure how the performance of the LLM selected by SMOOTHIE compares to other selection algorithms. We first compare SMOOTHIE-GLOBAL to an unsupervised random baseline (RANDOM), which would select a random model from the ensemble. We reported the expected performance of this method, which is equivalent to taking the average performance of the ensemble. We also compare SMOOTHIE-GLOBAL to a labeled baseline which simulates selecting an LLM on the basis of a small amount of validation data [67] (BEST-ON-VAL). We sample a small labeled validation set (50 samples) and select the LLM that performs the best on this set. To account for sampling variation, we repeat this with 10 random draws and report the average performance. Because AlpacaEval has no training split and MixInstruct has no labeled data, we only compare SMOOTHIE-GLOBAL to RANDOM on those datasets.Table 1 provides results for the seven NLG tasks. We find that SMOOTHIE-GLOBAL outperforms the unsupervised RANDOM baseline on 6/7 tasks for the 3B ensemble and on 7/7 tasks for the 7B ensemble. SMOOTHIE-GLOBAL outperforms RANDOM by up to 7pts (on tasks measured by rouge2), and by up to 12pts (on tasks measured by accuracy). We also observe that SMOOTHIE-GLOBAL is frequently competitive with and even outperforms the BEST-ON-VAL baseline, which uses labeled data. SMOOTHIE-GLOBAL outperforms BEST-ON-VAL on 4/7 tasks for the 3B ensemble, and 5/7 tasks for the 7B ensemble. On GSM8K, SMOOTHIE-GLOBAL achieves a solve-rate of 37.5% (matching BEST-ON-VAL, while RANDOM achieves a solve-rate of 28.3% (Table 11 ).SMOOTHIE-GLOBAL also outperforms the RANDOM baseline on the instruction-following datasets. On MixInstruct, SMOOTHIE-GLOBAL achieves a ChatGPT-rank (↓) of 3.91, while RANDOM achieves a ChatGPT-rank of 5.95 (Table 10 ). On AlpacaEval, SMOOTHIE-GLOBAL outperforms RANDOM on all but one trial (across both win-rate and length-controlled win-rate outperforms RANDOM by an average of 15pt win-rate, and up to 27pts. Setup We next assess whether SMOOTHIE-LOCAL's sample-conditional scoring mechanism allows it to route samples to LLMs in the multi-capability regime. We construct two mixed-task distributions by combining existing datasets. The first distribution corresponds to tasks measured by accuracy, and contains SQuAD, TriviaQA, and Definition Extraction. We refer to this as DISTR-ACC. The second distribution corresponds to tasks measured by Rouge2, and contains CNN/DailyMail, XSum, Web NLG, and E2E. We refer to this as DISTR-ROUGE2. For each mixed-task dataset, we report the metric averaged across all tasks. We compare to three baselines.• RANDOM: A random-selection baseline which returns a generation from a random LLM in the ensemble. Though naive, prior work has found this to be a strong method in practice [56] . We run 10 trials and report the mean of this approach to account for variance. • LABELED-KNN: A labeled data-based KNN baseline. For this, we sample 50 labeled samples from a separate hold-out set (D val ), and measure the performance of each candidate LLM on this set. For a given test sample x, we identify the 20 most semantically similar instances in D val (using SentenceBERT embeddings [76] ), and route x to the highest performing LLM over this subset. We note that the LABELED-KNN baseline is derived from routing methods in [48, 86] . • PAIRRM: A reward model from [41] which accepts an instruction and multiple generations as input, scores each generations suitability for the instruction, and returns the predicted best generation. PAIRRM is a labeled-data method which [41] trained on collected preference data.In addition, we also compare the best individual model in the ensemble (BEST-MODEL), and SMOOTHIE-GLOBAL. For both mixed-task datasets, we run SMOOTHIE-LOCAL with SentenceBERT embeddings, and the sample-conditional version of SMOOTHIE-LOCAL estimates θ i (x) using a neighborhood size n 0 = 1.Results for the 3B and 7B ensembles over DISTR-ACC and DISTR-ROUGE2 are provided in Table 2 .We find that SMOOTHIE-LOCAL outperforms all baselines across both data distributions, for both ensembles. Though SMOOTHIE-LOCAL requires no labels, it still outperforms labeled data baselines like LABELED-KNN and PAIRRM. We observe a substantial gap between SMOOTHIE-LOCAL and SMOOTHIE-GLOBAL, which indicates that SMOOTHIE-LOCAL's sample-specific scoring mechanism provides performance improvements. Notably, we see that SMOOTHIE-LOCAL substantially betters BEST-MODEL, indicating that SMOOTHIE-LOCAL's routing mechanism is offering a performance improvement over a strategy which merely selects the best LLM on average. We study this in greater detail by examining the relative rank of the LLM selected by SMOOTHIE-LOCAL for each sample. For each sample in DISTR-ACC and DISTR-ROUGE2, we rank the quality of each LLM's generation according to standard-competition ranking (i.e., "1-2-2-4" ranking). We then count how frequently SMOOTHIE-LOCAL selects the rank-i generation across each distribution for each ensemble. We visualize results in Figure 3 . As the visualizations demonstrate, SMOOTHIE-LOCAL consistently selects the best or second-best generation from within the ensemble.Third, we study whether SMOOTHIE-LOCAL and SMOOTHIE-GLOBAL can be generalized to other settings where engineers have a candidate pool of text generators of unknown quality, and must select one of them to use for some application. In particular, we focus on the setting where an engineer has access to multiple prompt templates for a given generation task, and must select which prompt-templates' generation to use as the final output [29] . Unlike above, we assume the engineer only has access to one LLM. We study SMOOTHIE-LOCAL and SMOOTHIE-GLOBAL in this regime using the NLG tasks from Section 5.1. For each task, we manually write between 3 and 5 prompt templates, varying the wording of instructions and the choice of in-context samples. We analyze SMOOTHIE applied to two models at different size points: Falcon (1B) [65] and Llama-2 (7B) [92] .Table 3 provides the results. Overall, we find that SMOOTHIE-GLOBAL selects the optimal prompt 2/7 times for Falcon-1B, and 3/7 times for Llama-2. SMOOTHIE-LOCAL and SMOOTHIE-GLOBAL consistently outperform RANDOM-on 6/7 tasks for Falcon-1b and 6/7 tasks for Llama-2. On 7 task/model combinations, one of either SMOOTHIE-GLOBAL or SMOOTHIE-LOCAL matches or outperforms a labeled baseline. To better contextualize performance improvements from SMOOTHIE-GLOBAL, we also compare to the improvement that accompanies increasing model size. Following a common practice in recent work, we can quantify the extent to which SMOOTHIE-GLOBAL allows smaller models to match or exceed the performance of larger models [2, 29] . In Figure 4 (Appendix E), we compare RANDOM and SMOOTHIE-GLOBAL on models from the Pythia suite at four sizes: 410M, 1B, 2.8B, and 6.9B parameters [7] . We observe that SMOOTHIE-GLOBAL substantially improves performance-on E2E, SMOOTHIE-GLOBAL enables a 410M parameter model to outperform a 6.9B parameter model.Finally, we conduct ablations to examine different aspects of SMOOTHIE-GLOBAL and SMOOTHIE-LOCAL: improving its efficiency, adjusting the neighborhood size, varying the choice of embedding model, and using different LLM ensembles.Improving efficiency First, we explain SMOOTHIE's current efficiency properties. To estimate the Smoothie weights for routing, we use a simple closed-form procedure that does not require any SGD or training, as described in Algorithm 1. As a result, SMOOTHIE weights on the entire dataset can be computed in seconds-for the 7B ensemble, SMOOTHIE-LOCAL on the multi-task datasets takes 2.14 seconds per 1000 samples, and SMOOTHIE-GLOBAL on the single-task datasets takes under 0.03 seconds per 1000 samples. Moreover, SMOOTHIE does not require any ground-truth annotations; however, all m model generations per test sample are needed as input to the algorithm. That is, we need n × m generations for a D train of size n samples.Fortunately, the need for computing all model generations per test sample can be removed with a small algorithm tweak, making Smoothie even more efficient and its runtime independent of n.Suppose we have a held-out set of n train train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm 1, KNN is now over a held-out training dataset). This approach, which we call SMOOTHIE-TRAIN, selects the model for a test sample without needing model generations for that sample. Only n train × m generations are needed, regardless of how large the test dataset n is.We study the NLG tasks, using n train = 250 samples. In Table 7 (Appendix E), we evaluate a version of SMOOTHIE-GLOBAL-TRAIN) and observe that it matches SMOOTHIE-GLOBAL on 12/14 model-dataset pairs, and performs worse on the remaining 2/14 pairs. We also evaluate SMOOTHIE-LOCAL-TRAIN, on DISTR-ACC and DISTR-ROUGE2 (Table 8 ) using a neighborhood of size n 0 = 20.We find here that while SMOOTHIE-LOCAL-TRAIN underperforms SMOOTHIE-LOCAL on both the 3B and 7B ensemble for both DISTR-ACC and DISTR-ROUGE2, it still outperforms RANDOM and remains competitive with supervised baselines.Neighborhood size We study the impact of n 0 , and consider SMOOTHIE-LOCAL's performance for n 0 ∈ [1, 5, 10, 20, 50, 100] . Figure 5 provides performance over DISTR-ACC and Figure 6 provides performance over DISTR-ROUGE2. Overall, we find that SMOOTHIE-LOCAL's performance steadily degrades as n 0 increases, and is highest when n 0 = 1.We study how the choice of embeddings affects SMOOTHIE-GLOBAL performance (Table 9 ). Specifically, we compare the performance of SMOOTHIE-LOCAL using Sentence-Bert embeddings (all-mpnet-base-v2) [76] to BGE embeddings (bge-small-en-v1.5) [102] .We observe that SMOOTHIE-LOCAL appears robust to different embeddings-SMOOTHIE-LOCAL with BGE embeddings still outperforms other labeled and unlabeled baselines. Interestingly, we observe that certain embedding models appear to yield better performance over certain distribution/ensemble combinations. For instance, SMOOTHIE-LOCAL with SentenceBERT embeddings outperforms SMOOTHIE-LOCAL with BGE embeddings on DISTR-ACC for the 3B ensemble and DISTR-ROUGE2 for the 7B ensemble, while performing worse on DISTR-ROUGE2 for the 3B ensemble and DISTR-ACC for the 7B ensemble.Different ensembles Finally, we consider whether SMOOTHIE-GLOBAL can generalize to a wider array of ensembles (Figure 7 ). We combine the LLMs contained in the 3B and 7B ensembles into a single pool, and sample 50 distinct ensembles ranging in size from 4-7 LLMs. For each of the 7 NLG tasks, we evaluate SMOOTHIE-GLOBAL's ability to identify the best model from within each ensemble. Across these 350 settings, we find that SMOOTHIE-GLOBAL identifies the best model in 211 of them (60.2% of the time), and one of the two best models in 292 of them (83% of the time). 
conclusion: 
 In this paper we study and propose an algorithm for learning label-free routers for generative tasks. We validate our approach across a variety of evaluation regimes, finding it consistently beats other unsupervised approaches and often matches/exceeds supervised approaches.Limitations We discuss several of SMOOTHIE's limitations. First, its multivariate Gaussian graphical model currently uses a diagonal covariance matrix. This assumes independent error vectors for each generation, though SMOOTHIE could be extended to account for dependencies [72, 93] . Additionally, SMOOTHIE optimizes only for performance without considering cost tradeoffs between large and small models. Finally, its reliance on embeddings may capture only certain aspects of semantic similarity. Other embedding models and additional heuristics could be used to create richer input features for SMOOTHIE. 
 
 Appendix: In Appendix B, we provide a glossary of notation used in the paper. In Appendix C, we provide an extended related work, and in Appendix D we provide a proof of Proposition 1, which is used in deriving the SMOOTHIE algorithm. Finally, in Appendix E we provide additional experimental results and details.Code for reproducing our results and using SMOOTHIE is available at https://github.com/ HazyResearch/smoothie.The glossary is given in Table 4 below.Symbol Used for V The space of all vocabulary sequences. Embedding mapping zg 0 : V → R d for any text sequence, where g0 is an embedding model such as SentenceBERT [76] .x Input text x ∈ X ⊂ V. y Reference output text y ∈ Y ⊂ V. G Candidate pool of m LLMs, G = {g1, . . . ,The observable embedding of x concatenated with the ith LLM's generated output, λi(x):= zg 0 ([x, gi(x)]). z ⋆ (x)The latent embedding of x concatenated with unknown reference output, z ⋆ (x) := zg 0 ([x, y]). Z Partition function for normalization of (1). n0Number of nearest neighbors used to learn θi(x) for x. n0 = n (i.e., the entire test dataset) corresponds to SMOOTHIE-GLOBAL and n0 < n corresponds to SMOOTHIE-LOCAL.The average squared Euclidean distance between the ith and jth LLM embeddings over a neighborhood around x, δij(x) = 1 n 0x ′ ∈NNn 0 (x) ∥λi(x ′ ) -λj(x ′ )∥ 2 . This is the primary expression used in computing θi(x).Table 4 : Glossary of variables and symbols used in this paper.LLM Routing The problem of determining how to route samples to various models has been long studied in statistics [37, 42] as well as Mixture of Experts deep neural networks [25, 82] . These works focus on how to jointly train the models and router in a stable and efficient manner.Since many LLMs are now available off-the-shelf, recent works study how routing mechanisms can be applied at inference time to trained models. Some works involve training task or domain-specific expert models and then learning a router. The router can be a nearest neighbors algorithm [38] , a neural network [96] that classifies among the different domains corresponding to the experts, or an extra gate learned when training the expert models [60] . These approaches do not explicitly require labels, but they require knowledge of what domain is used to train each expert and assume that each expert is the best model for its corresponding domain, therefore effectively using this mapping as a form of labels. In contrast, our setting focuses on routing among pre-trained LLMs where we do not know what models are optimal on what tasks and their samples.A second category of inference-time routing works studies how to choose among a collection of pretrained LLMs, which is the setting that SMOOTHIE focuses on. Several approaches involve training a meta-model that either scores or ranks how a LLM will perform on a sample [41, 74, 79] , all of which required labeled data to train. MoRE [87] involves training a simpler random forest classifier, using the rate of agreement among LLMs as one of the features, which is similar to how Smoothie estimates scores; however, it also requires labeled data to train the classifier. Some approaches [48, 86] do not require training routers and simply use nearest neighbor methods. However, these nearest neighbor methods still use labeled data to determine what training samples each LLM performs the best on. [55] invokes a trained reward model for the routing mechanism. [21] trains a classification-based router using the BARTScore metric on LLM generations as pseudolabels; this avoids using manually labeled data, demonstrating that while a majority of routing methods require labeled data, there exist some alternatives that do not. We leave it to future work to compare and integrate SMOOTHIE with other unsupervised approaches.Finally, complementary to our setting are works that jointly focus on cost minimization as well as quality of generations. RouterBench [33] creates a benchmark for studying the cost-quality tradeoffs in routing systems. Optimizing for cost can be done algorithmically, such as in FrugalGPT [13] , AutoMix [58] , RouteLLM [63] , and [88] , as well as via hardware enhancements such as SambaNova Systems' Composition of Experts [69] .LLM Ensembling A rich literature has observed that ensembling LLM outputs-across different prompts or base models-can improve the accuracy of generated predictions. Prior work has proposed and studied a number of different ensembling algorithms for classification tasks, including majority-voting [54, 98] , weak-supervision [2, 29] , boosting [32, 68, 104] , and others [49, 66, 80] .More relevant to our work here is a literature on ensembling for generative tasks. One category of methods rely on an auxilliary sequence-to-sequence models to "fuse" generations from different prompts or base LLMs [41] . Though recently applied in the context of modern LLMs, the concept of fusion traces back to older work on summarization [5, 46, 47, 75] . Some techniques combine or switch among multiple outputs at inference time [34, 36, 59, 83, 97] , while others involve averaging in weight space [35, 38, 95] . Lastly, ensembling can also be approximated by randomly selecting a model to be used in multi-turn settings [56] .Other LLM Selection Algorithms Beyond the setting of selecting among multiple LLMs, other works have explored how to select the optimal prompt template from a collection of candidate prompts. These works can be grouped into two categories. The first category assumes that engineers have access to labeled data. In the naive case, this labeled data can simply be used to select the best performing prompt [45, 67, 77] . Another subset of this category focuses on the setting where new prompts can be generated by selecting in-context demonstrations from a set of labeled samples (typically a small training set) [22, 53] . Prior work has proposed different methods for identifying the optimal in-context demonstrations to use, depending on the sample for which the LLM is being used to produce a prediction for [11, 77, 90, 101, 105, 106] . The second category focuses on zero-label prompt selection methods, but solely for classification tasks [52, 89, 103] . Prior work here selects prompts on the basis of mutual information [89] , agreement rates between predictions produced by different prompts [52] , and various probability based measures [28, 57, 103] .Weak supervision SMOOTHIE utilizes techniques inspired by weak supervision literature. Weak supervision aims to programmatically generate labels on an unlabeled dataset by aggregating the predictions of several weak "voters", such as heuristics, pre-trained models, and knowledge graphs [71, 73] . It assumes a particular latent variable graphical model and uses its structure to estimate latent quantities, such as the accuracy of each voter (in our setting, the quality score of each LLM). Typically, this graphical model is a binary Ising model, as weak supervision has generally been studied in classification settings [26, 72] , where embeddings have been utilized as auxiliary signal but not modeled explicitly [15, 29] . Weak supervision has been applied to broader settings, such as for learning rankings, graphs, and manifolds [85, 94] . We derive our estimation procedure from the Gaussian model in [85] , applying it to LLM embeddings. While both SMOOTHIE and [85] use a multivariate Gaussian model, in SMOOTHIE we apply it to model routing with SBERT embeddings on natural language datasets, whereas [85] conducts synthetic experiments in hyperbolic spaces and metric spaces induced by synthetic graphs. Moreover, SMOOTHIE uses nearest neighbor kernel smoothing to allow for sample-dependent weights-critical for routing-while [85] calculates one global set of weights over the dataset.Consistency-based selection Consistency is central to unsupervised selection and aggregation methods, the simplest being majority vote. While weak supervision methods [26] and SMOOTHIE heavily rely on notions of voter agreement as depicted in a graphical model, there are several other consistency-based methods. Minimum Bayes Risk methods [6, 44] selects the generation that has the highest average similarity (i.e., cosine) with other generations. This is similar to SMOOTHIE, which routes to the lowest value of (3). If we ignore the subtraction of δ jk (x) in (3) and average over more than just δ ij (x) and δ ik (x), then SMOOTHIE with n 0 = 1 is equivalent to [44] Therefore, SMOOTHIE can be considered as a slightly modified and more general version of this approach. Another approach [10] relies on consistency between a "global" and "local" embedding for each generation. They solve an optimization problem that estimates each generation's quality score by constructing a loss that enforces that the similarity between the estimated true generation (produced by a weighted average of candidates) and the candidate generation should be the same according to both global and local embeddings. In contrast, SMOOTHIE uses one embedding space, relies on a multivariate Gaussian structure among embeddings, and does not require gradient descent to learn the quality of each generation.Test-Time Compute Approaches like model routing, ensembling, and selection can all be seen as ways of utilizing test-time compute to produce higher-quality generations from a system of LLMs. Test-time compute can also be utilized over a single LLM via techniques such as those used in OpenAI's o1, Chain of Thought, and Rephrase and Respond [20, 64, 99] . Other works have recently studied how test-time compute scales [9, 12] -finding that producing more generations can often yield the correct response-and how to combine multiple test-time methods, such as Archon [78] . It is interesting future work to consider how SMOOTHIE can be integrated with other test-time compute techniques.We provide a proof of proposition 1, which is a direct property of multivariate Gaussians that is also presented in [85] . We first expand E ∥λ i (x) -λ j (x)∥ 2 :E ∥λ i (x) -λ j (x)∥ 2 = E ∥(λ i (x) -z ⋆ (x)) -(λ j (x) -z ⋆ (x))∥ 2 (5) = E ∥λ i (x) -z ⋆ (x)∥ 2 + E ∥λ j (x) -z ⋆ (x)∥ 2 -2E (λ i (x) -z ⋆ (x)) ⊤ (λ j (x) -z ⋆ (x))Let λ i,k (x) be the kth element of the λ i (x) embedding, and similarly define z ⋆ k (x). Note that since Σ is diagonal, we can writeCov [λ i,k (x) -z ⋆ k (x), λ j,k (x) -z ⋆ k (x)] = E [(λ i,k (x) -z ⋆ k (x)) • (λ j,k (x) -z ⋆ k (x))] -E [λ i,k (x) -z ⋆ k (x)] E [λ j,k (x) -z ⋆ k (x)] = 0 for all k ∈ [d]. Since µ = ⃗ 0, we thus have that E [(λ i,k (x) -z ⋆ k (x)) • (λ j,k (x) -z ⋆ k (x))] = 0 for all k ∈ [d], which implies that E (λ i (x) -z ⋆ (x)) ⊤ (λ j (x) -z ⋆ (x)) = 0.Plugging this into (5), we haveEQUATIONThis section contains additional details on experiments discussed in Section 5.Table 5 provides links to the Huggingface datasets used for each task. For E2E, CNN/DailyMail, XSum, and Web NLG we measure performance using rouge2. For SQuAD, TriviaQA, and Definition Extraction we measure using "accuracy." A model generation is treated as "correct" if if contains the answer, and incorrect otherwise [1] . For the Alpaca leaderboard experiments, we run each trial by sampling 5 models from the following set of 10: Nanbeige-Plus-Chat-v0.1, claude-2, Qwen1.5-110B-Chat, yi-large-preview, gemini-pro, Meta-Llama-3-70B-Instruct, Ein-70B-v0. BEST-ON-VAL 37.5 SMOOTHIE-GLOBAL 37.5Table 11 : Results for SMOOTHIE-GLOBAL and baselines on GSM8K. We report accuracy, with scores scaled to 0-100. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer Definition Extraction has fewer than 1000 samples.Responses are available on the AlpacaEval website: https://tatsu-lab.github.io/alpaca_eval/. 


Decision: This paper introduces a method for routing inputs to the most suitable LLM from a pool of models. The goal is to predict which LLM will produce the highest-quality output without relying on labeled training data for the routing mechanism. Instead, the approach utilizes weak supervision to learn the parameters of a Gaussian graphical model with latent variables.

The method is interesting, and the results appear promising. During the rebuttal, the authors adequately addressed most of the concerns raised by the reviewers. Therefore, I vote for acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents a model for routing an input to an LLM from a pool of LLMs. The aim is to estimate which LLM will produce highest quality generation without using any labeled training data in estimating the routing model. Instead, the approach relies on weak supervision to learn the parameters of a Gaussian graphical model. 

The technique is evaluated by examining the model's ability to pick the oracle best LLM from the pool on various tasks; then by evaluating it's end-to-end routing performance compared to baselines (some of which have access to labeled data); and finally by using the same technique to pick amongst a set of LLM+prompt pairs for various tasks in order to effectively do prompt selection.
strength_weakness: strength_weakness: - The paper has no discussion of efficiency, but this seems like an important point to address. This has important implications for the motivation: if one has the capacity to run a pool of small models, why wouldn't one instead run a single larger model. In this way, the paper should really address the tradeoff of accurracy/ROUGE with # of parameters (or even more realistically runtime). 
- The paper does very little to discuss its limitations. There are a few comments peppered throughout.
- The paper does not mention anything about which LLMs are being picked. For example, is the job of the router trivialized by certain pools of models? For example, one could imagine a setting in which the router simply identifies and then always picks the best model. The paper does no analysis of this.
questions: questions: - How would address the efficiency concerns mentioned in the weaknesses section?
- Did you observe anything about the behavior of the routing across different tasks?
limitations: limitations: Yes
paper_summary: paper_summary: - This work proposes a method called SMOOTHIE, which can route label-free test examples to LLMs. Specifically, 
  - it employs a latent variable model and Gaussian distribution for efficient quality score estimation and uses LLM outputs to estimate generator quality. 
  - it estimates specific to each test sample using nearest neighbors and routes samples to LLMs with highest quality scores. 

- Empirical results show that:
1. SMOOTHIE's learned quality weights correlated with actual LLM performance.
2. In mixed-tasks datasets SMOOTHIE is able to route different samples to different LLMs which boosts the performance.
3. SMOOTHIE can be used for prompt-selection.
strength_weakness: strength_weakness: - In figure 3. there is no label on x-axis
- typo: in Section 5.2, line 263 and 264, the "SMOOTHIE-independent" be SMOOTHIEGLOBAL instead.
questions: questions: - In the algorithm, it is not quite clear how to select indices j and k different from i. Is it just random selection?
- How do you compare SMOOTHIE with Minimum Bayes method, where you choose the output from an LLM that aligns the most with all others?
limitations: limitations: - The authors did not address the limitations 
- There is no negative societal impact
paper_summary: paper_summary: This paper proposes a method for selecting LLMs' responses for generative tasks.

It can essentially be viewed as a "truth inference" problem in the research community of "weak supervision" and "crowdsourcing"; unlike ordinary truth inference methods, this paper takes into account unstructured textual information and, for each sample, the method in this paper needs to pick one out of all candidate answers.
strength_weakness: strength_weakness: Main concerns:

1) First, authors should summarize the most relevant works in the main text of the paper. For this current version, I need to scrutinize the most relevant works in the appendix.

2) Related to the above point, there are already works that focus on the task that this paper addresses, e.g. "An error consistency based approach to answer aggregation in open-ended crowdsourcing". 
(Although this paper focuses on crowdsourcing workers rather than LLMs.)
 Therefore, it would be advisable for the authors to research the relevant literature in more depth and to consider them in experiments as comparison methods.

3) The core theoretical part of this paper is applied to the content in the existing work [72]. In the main text, it is necessary to state the technical and theoretical differences with the existing work [72] more clearly and in detail.

Some other concerns:
1) The meaning of some symbols is not explained, e.g., line 109.
2) In this paper, a graphical model is presented, then a graphical representation of this graphical model should be shown.
3) Typos. E.g., "x(Section 4.2)" in line 134, "$\theta_i(x)$s" in line 137.
4) The quality of the images (e.g. sharpness) can be further improved.
Also, in Figure 2, the name of the proposed method should be shown in capital letters.
5) The tables are not self-explanatory enough, e.g. the metric of interest is not shown in Table 1.
6) Some of the presentations are confusing, such as lines 609 and 610.
questions: questions: Please refer to the "main concerns" above.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a label-free routing method, Smoothie, to route an ensemble of LLMs without annotated data. Smoothie constructs a latent variable graphical model over semantic embedding representations of observable LLM outputs and the unknown ground truth, estimates the sample-independent quality scores of each LLM, and routes to the LLM with the highest quality score. Experimental results indicate that Smoothie competently performs several generation tasks over the 3B and 7B ensembles.
strength_weakness: strength_weakness: 1. The method relies on the embedding method to measure the semantic similarity, which limits the scope of Smoothie to semantic-related tasks. The capability of LLMs should vary more on tasks such as mathematical reasoning, but these tasks depend less on semantic similarities.
2. The experimental comparisons are unclear. The chosen metrics are highly abstractive. Given the performances of each LLM are unknown, it's hard to identify whether the method successfully routes to the best or slightly better model than the baselines. The presentation of the experiment section is also hard to follow.
questions: questions: 1. (Refer to W1 and W2) Can you provide the performance statistics of the models used in the experiments?
2. Does Smoothie require more computation than the supervised methods?
limitations: limitations: The limitations and broader impacts are not discussed in the paper.

Comments: title: Reply to rebuttal
Response
comment: Thanks for the response and clarification to the questions I had. I think it will be good to have the additional details in an updated version.
Thanks for the thorough response! It'll be great to have these in the next version of the paper.
Dear reviewer Cr26,

Thank you for your response! We are glad to hear that our additional experiments and comments addressed most of your main concerns. We apologize for those that were unaddressed. 

First, we intend to update the draft to remedy and include the suggestions you made. We will add a visual depiction of the PGM to the paper. Smoothie's PGM is similar to the "label model" in [this image](https://cdn.snorkel.ai/wp-content/uploads/2022/03/Weak-supervision-modeling-pipeline-Snorkel-Blog-1024x576.png) where the vertices in our case are $V = \{\lambda_1(x), \dots, \lambda_m(x), z^\star(x)\}$ and the edges are $E = \{(\lambda_i(x), z^\star(x))\}_{i=1}^m$. We will also update our draft with higher quality images. 

Second, we apologize for the lack of clarity regarding PairRM. PairRM is a pretrained reward model introduced and described by Jiang et al here: https://arxiv.org/abs/2306.02561. It is built from the DeBERTa model (400M parameters), and accepts  an instruction $i$, and a set of candidate responses to the instruction $r_1, ..., r_k$. PairRM then produces a ranking over these candidate responses, according to the predicted quality of the response (i.e., the best response is ranked first). We choose to evaluate PairRM because of its strong performance; it has been used to rank responses (in conjunction with KTO/DPO methods) to allow for 7B models, such as Contextual AI's model, to do well on the [AlpacaEval leaderboard](https://tatsu-lab.github.io/alpaca_eval/), where the top models are typically much larger. PairRM can be downloaded from huggingface here: https://huggingface.co/llm-blender/PairRM
Thanks to the authors for making the response, including adding some valuable experiments.  
Most of my main concerns have been addressed. 


Though there are still a few details: In the authors' response, there is no explanation of which method "PairRM" refers to, and there is no explanation of whether it is possible to add a graphical representation of the proposed graphical model to the text.
Dear reviewer RLiV,

We appreciate your valuable feedback and suggestions. We hope that our response has addressed your questions about efficiency, analysis of routing behavior, and limitations of Smoothie. Since the discussion period is ending in 2 days, please let us know if there are any additional questions or comments you have. Thank you so much!
Dear reviewer r3vp,

Thank you so much for your helpful feedback in your review. We hope that our response has addressed your concerns about comparison to the Minimum Bayes method, writing clarifications, and limitations. In particular, we ran extensive experiments comparing Minimum Bayes to Smoothie, with our results summarized in our rebuttal to you. Since the discussion period is ending in 2 days, please let us know if there are any additional questions or comments you have. Thank you so much!
Dear reviewer Cr26,

We greatly appreciate your detailed comments on our work. We hope that our response has addressed your concerns about comparisons to other works and writing clarifications. In particular, we implemented the algorithm in the error consistency paper you mentioned as well as an additional reward model baseline. We compare these methods to Smoothie in our rebuttal to you and in the global rebuttal, respectively. Since the discussion period is ending in 2 days, please let us know if there are any additional questions or comments you have. Thank you so much!
Dear reviewer gxgv,

Thank you so much for raising your score. We agree that the performance on GSM8K is interesting and will update our final draft with these results!
I appreciate the authors for providing (1) details about the routing, (2) performance on the non-semantic task, and (3) inference time analysis.
I'm particularly interested in why Smoothie can perform well on GSM8K, and whether it can perform well on reasoning datasets with questions generated from templates (similar questions with different values). 
Still, the newly added limitations discuss the problem of semantic-based embedding.
My other concerns are addressed and I will raise the rating.