Large Language Model Routing with Benchmark Datasets
Authors: Tal Shnitzer
Anthony Ou
Mírian Silva
Kate Soule
Yuekai Sun
Justin Solomon
Neil Thompson
Mikhail Yurochkin
Venue: COLM (2024)
Keywords: Model selection, LLM routing, OOD Generalization
TL;DR: We reuse benchmark evaluations to learn router models for selection of LLMs on unseen tasks.

Abstract: The number of open-source Large Language Models (LLMs) grows daily, as does the number of available benchmark datasets used to evaluate LLMs. While some models dominate these benchmarks, no single model achieves the best accuracy in all tasks and use cases. In light of this observation, we address the challenge of selecting the best LLM from a collection of pre-trained models, given a new task.  While related work relies on evaluating each candidate model on a set of labeled examples, our new formulation does not assume any labeled data from the new task is available.  Instead, we repurpose a collection of benchmark datasets---which may focus on different tasks than the one at hand---to learn a ''router'' model for  LLM selection from inputs only; this problem reduces to a collection of binary classification tasks. Empirically, our strategy consistently improves performance over using any single model for all tasks.

Published as a conference paper at COLM 2024
Large Language Model Routing with Benchmark Datasets
Tal Shnitzer* 1/ A'nthooyy Ou" Mírian Silva?, Kate Soule', Yuekai Sun*, Justin Solomon', Neil Thompson' & Mikhail Yurochkin +3
Broad Institute, 2MIT, MIT-IBM Watson Al Lab, UUniversity of Michigan
Abstract
The number of open-source Large Language Models (LLMs) grows daily, as does the number of available benchmark datasets used to evaluate LLMs. While some models dominate these benchmarks, no single model achieves the best accuracy in all tasks and use cases. In light of this ob- servation, we address the challenge selecting the best LLM from a col- lection of pre-trained models, given new task. While related work relies on evaluating each candidate model on a set of labeled examples, our new formulation does not assume any labeled data from the new task is avail- able. Instead, we repurpose a collection of benchmark datasets-whit may focus on different tasks than the one at hand-to learn "router" model for LLM selection from inputs only; this problem reduces to a col- lection of binary classification tasks. Empirically, our strategy consistently improves performance over using any single model for all tasks.
1 Introduction
T5
Large Language Models (LLMs) demonstrate groundbreaking abilities to solve tasks across a variety of NLP domains (Devlin et al., 2018; Brown et al., 2020). Today, researchers in academia and industry release new LLMS daily; as of January 30th, 2024, Hugging Face hosts nearly 50k models for text generation. These models perform tasks ranging from text classi- fication to question-answering, summarization, and dialogue.
?
2
? ? P2
This influx of open-source LLMs and the di- versity of their potential use cases has inspired numerous benchmarks, collections of datasets to compare LLMs on different tasks and domains. For example, HELM (Liang et al., 2022) con- sists of 42 scenarios covering a variety of uses, MMLU (Hendrycks et al., 2020) is multiple- choice question answering benchmark with 57 tasks, Open LLM Leaderboard (Beeching et al., 2023) combines MMLU with other question- answering datasets, and LM Evaluation Har- ness (Gao et al., 2021) tests over 200 tasks. While one LLM will be best on average across benchmarks, no single model is likely to be the best on on each individual dataset. In contrast, a practitioner typically wants to know what is the
Figure We evaluate candidate LLMs (T5, Falcon, Llama) on various tasks (emojis inside boxes: QA, reasoning, summarization, sentiment analysis, dia- log) and domains (4 sections within each box: finance, legal, general knowledge, math) from benchmark datasets by train- ing a binary classifier per LLM (decision boundaries marked with colors in the upper part of the figure). For a new task (paper stack), we score each LLM with these binary classifiers and recommend an LLM (here Falcon) to the user.
'Equal contribution
"Work done while at MIT
I *Correspondence mi miaahlll www.cckiieiebaaaeoo www.hkiiiiibm..mmmm com

---

Published as a conference paper at COLM 2024
best model for their specific use case rather than average performance across datasets.
To address this gap, we study the problem of identifying the best LLM for a given task. We seek strategy that can identify an effective LLM given example inputs for the task, with- out relying on pre-labeled input/ output pairs. We use benchmark datasets to understand the performance of LLMs across tasks and domains: For example, to evaluate whether LLM will be able to answer math questions, we might find LLMs that do well on other STEM-related tasks rather than, e.g., sociology or toxicity detection.
We cast the learning of model strengths as binary supervised learning. Our binary task is to efficiently predict whether an LLM will perform correctly on given input, given an embedding of just the input; our data for this prediction problem comes from running the LLMs on the benchmark datasets. See Figure 1 for an illustration. This information is collected during benchmark evaluations anyway and is simply reused for training model routers without additional expensive LLM inference. Our router is efficient test time, as it does not need to test every possible LLM on the new task.
Our contributions are summarized below:
We formalize the problem of learning the strengths and weaknesses of LLMs for down- stream routing, i.e,, selecting the best model, as a collection of binary classification prob- lems that predict whether given LLM will be "correct" on input.
We propose three scores for selecting LLMs for a new task using these correctness pre- dictors. Our third score accounts for mistakes correctness predictor makes on out-of- distribution data from a new task, which is likely to differ from benchmark data used for training the correctness predictors. We connect to meta-learning to obtain theoretical
insights into the scores' efficacy.
We verify our routing scores' effectiveness on 29 datasets from HELM (Liang et al., 2022) representing scenarios like QA, text classification, knowledge, and reasoning, and Mix- Instruct (Jiang et al., 2023), which evaluates Instruction-following abilities of LLMss. We discuss and empirically investigate generalization of correctness predictors to new tasks, the importance of a larger pool of benchmarks, and the potential of routing smaller LLMs to reduce cost.
2 Related work
Benchmarking. Comparing models or algorithms across various tasks is a standard prac- tice in ML and AI literature. Prior to Foundation Models (Bommasani et al., 2021), it was typical to apply the same learning algorithm to train a model on each of the datasets and compare the performance against other learning algorithms. The UCI Machine Learning Repository (Kelly et al., 2023) is one prominent example of such a collection of datasets often used to compare learning algorithms. With the emergence of Foundation Models, ie., models with billions of parameters trained on massive datasets using large compute clusters, the paradigm changed to evaluating the same model (or few-shot tuned version of it) on a variety of tasks (Bojar et al., 2014; Goyal et al., 2019; Li et al., 2022). In the context of Large Language Models, many benchmarks (Wang et al., 2018; 2019; Hendrycks et al., 2020; Gao et al., 2021; Srivastava et al., 2022; Liang et al., 2022; Beeching et al., 2023; Jiang et al., 2023) were proposed to help determine the most capable LLM. Benchmarks typically average the performance of models across tasks and provide a final ranking, discarding the rest of the information. In this work, we use the byproducts of benchmark evaluations, i.e, the per-sample performance of various LLMs across tasks, to learn about their individual strengths and identify the best LLM for a new task.
Model selection. Selecting the best model, or model selection, is a classical topic in statis- tics and ML (Bishop & Nasrabadi, 2006; Hastie et al., 2009; Raschka, 2018). However, the typical problem setting is quite different: classical methods like cross-validation aim to estimate the population error a model trained on samples from the population distri- bution. In other words, the goal is to find the best model for in-distribution test data, iee, data sampled from the same distribution as the train data. The notion of "train" data is quite elusive for LLMs, as they are usually trained massive datasets with trillions of
2

---

Published as a conference paper at COLM 2024
tokens with simple task of next token prediction (Radford et al., 2019; Brown et al., 2020).
However, the tasks we evaluate them on are often more structured, egg., classification and question-answering, and are specific to domains that may may not be sufficiently repre- sented in the train data. In addition, techniques like k-fold cross-validation require training the model multiple times, which is infeasible for LLMs.
Out-of-distribution model selection. Recognizing the limitations of the model selection methods for in-distribution test data (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021), recent work has proposed a variety of methods to select models when deployed on data that may differ from the train data. These methods rely on ideas such as bootstrapping (Xu & Tibshirani, 2022), reweighing (Chen et al., 2021b; Maity et al., 2023), agreement of models or ensembles (Jiang et al., 2021; Chen et al., 2021a; Ng et al., 2023), or aligning model accuracy in-distribution with a confidence threshold (Guillory et al., 2021; Garg et al., 2022; Yu et al., 2022). Most of these methods are nontrivial to extend to generation use-cases of LLMs; some require training multiple models, and some need well-defined in-distribution data related to the new task.
Routing LLMs. Prior work on selecting LLMs primarily considers choosing one that pro- duces the best generation for a given input. Liu & Liu (2021); Ravaut et al. (2022); Jiang et al. (2023) train dedicated scoring ranking models that can be applied to model gener- ations. Unlike our work, these approaches require generating outputs with every candidate LLM to make decision, which can be computationally prohibitive with a large pool of candidate LLMs. FrugalGPT (Chen et al., 2023) calls LLMs sequentially until a dedicated scoring model deems the generation acceptable. Prior works in this group require train- ing data sufficiently representative of each of the tasks and domains of interest to train the corresponding ranking and scoring models. Automatic evaluation using other LLMs (Fu et al., 2023; Wang et al., 2023; Liu et al., 2023) may be used instead of such scoring models, but such an approach would require producing generations with every candidate LLM and evaluating each of these generations with a typically larger/ commerccil L.M. In this pa- per, instead, we use data from benchmarks to learn the strengths and weaknesses of LLMs across tasks and domains. The resulting model router requires generating outputs only with the chosen LLM at test time, making the proposed method a lot more cost-efficient.
Mixture of Experts. We highlight two interpretations of the Mixture of Experts (MoE) in the context of our work. First is the classical MoE (Masoudnia & Ebrahimpour, 2014) where model (or a subset) is selected for each inpul from pool of experts. Performing model selection efficiently for each input individually might be challenging on new, potentially out-of-distribution, tasks as demonstrated in our experiments in Section 5.1. Second is the MoE LLM architecture (Fedus et al., 2022; Jiang et al., 2024), where "experts" are a part of the LLM and are selected for each token at each layer. In this work, our goal is to select an LLM for a task, where an MoE LLM could simply be among the pool of candidate LLMs.
3 Learning from Benchmarks
We start by introducing notation to describe the majority of NLP benchmarks (see Table for a summary of the main notations used in the paper). Let (xf,..., 1 be a collection of inputs across D tasks. Each input text x corresponds to a reference answer rf, i.., an ideal generation for the corresponding input. There is a metric (x, that can be task- dependent and measures how well response o for an input x corresponds to the reference r. To test an LLMp, m E [1,...,M, on the benchmark, for each task d = 1,..., ......,, D, its responses are generated {o = LLM (x ) and compared to the corresponding refer- ences to obtain performance metrics [fd im = (x od " Most benchmark studies take a (weighted) average of the performance metrics and report single score for every LLM rank them in performance. Instead, we reuse these evaluation results to formulate supervised learning problem to better understand the strengths and weaknesses of various LLMs based on their performance on data points and tasks.
We omit dependency on the prompt when generating with an LLM and, w.l.ogg, consider the same I.I. with a different prompting strategy as a different L.M.M
3

---

Published as a conference paper at COLM 2024
Table 1: Summary of notations and definitions.
Notation
Definition
Notation
Definition
Input text
m d d'
LLM index (I...M)
Output of LLM (x/)
Training task index (1... .) New task index Reference answer (ideal)
in
Metric evaluating response o
fim
Performance metric Ex ( od ri)
rf
yxx, m)
LLM correctness E {0, 1}, Jim based
no
No. of new task samples Binary performance predic- tor
8m (x)
Performance predictor for LLM
8m (x)
p(d", m)
Probability that 3m is correct on xf"
u(d)
Task descriptor (for estimat- ing p(d', m))
d',,d')
na Σ y(2 m)
"Oracle" correctness score
S1 (m,,d)
= 1
NN² Σ 2 8m (xx) xt()
Simple correctness score
d2m,
= E 8m (
Binary predictor correctness score
ND
(x,x)) I(gm (x7") )("") >> f)
(3m, S2p(d', m) m) (1- (1- p(d',
OOD correctness score
Supervised learning from benchmarks. Our goal is to learn a simple routing function 8m (x) for each LLM, m = 1,...,M, M, that can predict (f i-1 i.e, the performance of the corresponding LLM on a new task d'. Then it is trivial to select the best LLM for this task. For efficiency at test time, we restrict the routers {gm }
[== to only depend on the input x. This is in contrast the majority of prior works on LLM routing that first obtain generations with every candidate LLM and then use them to choose the best model (Liu & Liu, 2021; Ravaut et al., 2022; Jiang et al., 2023). With thousands of open-source LLMs, is simply infeasible to obtain generations with every LLM for every input at test time.
To complete the problem formulation, we denote the "correctness" of model m on an in- put x by y(x, m) E {0,1}. Correctness evaluated as follows: generate a response od with LLM on input xx, compare it to the corresponding reference r/ and output if the model's response is good enough, i.e., fil > J),, and otherwise, where fa some threshold that can be task and/ or metric specific. For tasks like classification multiple- choice QA, y(x+, m) fir while for various evaluation metrics used in summarization and instruction following tasks (Zhang et al., 2020; Sellam et al., 2020; Yuan et al., 2021), the notion of correctness can help to account for the heterogeneity of popular metrics and task difficulty levels. In Section 5.2, we also consider raw metrics instead of correctness.
im
To train a predictor of an LLM correctness, for each LLM, 1, optimization problem:
M, we solve the
arg min
Σ E ll (x)()
(1)
8a
where we choose é to be binary cross-entropy loss and is any standard probabilistic classifier, i.e., 8m (xx estimates Ply(x, m) 1|x). In other words, training &m is equivalent to simply training (probabilistic) binary predictor.
An important consideration when training correctness predictors is their ability general- ize out-of-distribution (OOD) since our goal is to estimate LLM performance on new task d' that has not been seen during training. Training predictors given data from multiple do- mains that need to generalize to unseen domains is indeed an active area of research in ML literature. For example, Sun & Saenko (2016); Arjovsky et al. (2019) proposed methods for improving OOD generalization when training on data from multiple domains, while Koh et al. (2021) proposed a benchmark for OOD generalization demonstrating the challenging nature of the problem in various applications.
4

---

Published as a conference paper at COLM 2024
In this work, we use simple model for the correctness predictor: embed all inputs with a sentence transformer (Reimers & Gurevych, 2019) and use a k-nearest neighbors classifier (Cover & Hart, 1967) as {gm] 1M allows us to fit potentially complicated decision boundary of an LLM's correctness across multiple tasks without extensive hyperparameter tuning. We choose this approach for learning correctness predictors to emphasize the utility of learning from benchmarks even with a basic method and instead focus on the question specific to our problem that has not been studied in prior works on OOD generalization: Can we improve the quality of LLM routing with an imperfect correctness predictor?
m==1 kNN is a simple non-maraetrric classifier that
4 LLM routing with (imperfect) correctness predictors
The goal of LLM routing is to identify an LLM that will have the frequeecy of being correct on a new task d', given the inputs {x D from this task:
arg max, 5(m, d"), where $(m, 1 n - v(x m).
Here, (m,, d') is the "oracle" score that we want estimate. The most intuitive estimator is simply using the correctness predictor
(m,, d') = nt" Em
8m (x
(2)
but prior work has shown that accurately estimating P(y(x, m)
1|x), i.e., training cali- brated correctness predictors gms, is challenging on OOD data (Ovadia et al., 2019). Mean- while, 8m may still produce accurate predictions after thresholding the predicted probabil- ity even if the class probabilities are not estimated well (Guo et al., 2017). This motivates another score:
S2 d') = 1 (m(()²))
(3)
where Eo (xx") = (g i (x >> f), (0,1) is some threshold, e.g, f 0.5, is an indicator function, and 8m (x) {0,1} can be interpreted as the prediction of 8m on x. This score, however, does not take into account the potential "imperfection" of gm, i.e.,gm, gm,
binary classifier, likely have lower accuracy on OOD data from task d'. To address this issue, we model the dusto--distribution confidence of the predictions gm.
A simple OOD confidence model
We model LLM correctness as follows:
(x) ((x,)))z (1 (x)
with probability p(d',m) with probability 1 1- 1-p(d', m),
ie., p(d', m) E [0, 1] is the probability that 8m is the correct prediction on data point from task d'. The above model can be condensed as follows:
y(x, m1) |x, d' d (x)))( m) + (1 gm(x)) (01)0(11dmmmm))))) m)))
(4)
In this simplistic (and approximate) model, we assume that p(d', m) does not depend on the input x after conditioning on the task d'. The assumption is analogous the homoscedastic error term assumption in linear regression models and allows us to interpret p(d', m) as the marginal/overal accuracy of gm on data from the task d'.
Prior work has studied the problem of estimating OOD accuracy given the inputs from new task, but existing methods are challenging to combine with our approach. For exam- ple, Garg et al. (2022) learn threshold on model confidence, which is hard to apply when using kNN classifiers, and Ng et al. (2023) require data augmentations that can be chal- lenging to identify given the diversity of tasks in benchmarks. Prior methods also do not take into account the partition of the train data into tasks inherent in our problem setting.
We treat the estimation of p(d', m) as a supervised learning problem, taking advantage of the task partition. Specifically, we assign a task descriptor u(d) E R2, to every task that measures the distance of the data from task d to the other available tasks combined. Then
5

---

Published as a conference paper at COLM 2024
collect the values of p(d, m), i.e. the accuracy of &n d, and fit non-parametric gression model to predict p(d, m) from u(d). At test time, we compute u(d") for a new task d' based on the inputs { :=1 and predict p(d", m) using the fitted regression model. In general, one can consider higher-dimensional task descriptors u(d), but here, for sim- plicity, we keep it I-dimensional and use a Gaussian kernel smoother (also known as the Nadaraya-Watson estimator) as the non-parametric regressor. See details in Appendix A.
Finally, given the model of LLM correctness 4, S(m, d') is random variable (corresponding (scaled) sum of two Bernoulli random variables. To arrive at
to 5m, d')) distributed our final score for LLM routing, we take its expected value:
Sz d') S2(m, d p(d'' m)
520m-22mmmm (1(m,, p(d', m)).
When selecting an LLM with S3, we consider an alternative to the arg max criterion based on our correctness model 4, which defaults to the best model on average across benchmark datasets when we are not sufficiently confident that candidate model will be better:
{ M3 if P(S( d') > ((m",d')
(6)
m otherwise,
where Mg = arg max Sm d'), i.e. the best LLM for the new task according to S3, and m" arg max, _P (m,,d), i.e., the best LLM across the benchmark datasets. In the experiments, we set 17 = 0.6. We summarize LLM routing procedures in Appendix A.
Connection to meta-learning. The OOD confidence model in equation 4 can be viewed as a form of meta-learning where Em and p(.,m) are meta--arameters and the adaptation step is 8m + {m( (mn((((,m We exploit this connection to theoretically demonstrate the potential advantages of routing LLMs using S3 over S2 in Appendix B.
Experiments
5.1 Model routing on HELM
Data. We select 29 datasets from the HELM benchmark (Liang et al., 2022) representing scenarios such as question answering, text classification, language, knowledge, and rea- soning, among others. We present additional information about the datasets in Table 5.
Models. We evaluate 18 open-source models ranging in size from 3B to 70B, including base and chat variations of Ilaaa 2. All models are summarized in Table 6.
Model routing. The best model on average (BMA) across the 29 considered HELM datasets is 12ama-2-70b (followed by 2-ama-2-70b-chate Our goal is to show that learning model routers from benchmark data can simultaneously outperform BMA and reduce inference costs by recommending smaller LLMs for tasks where they can perform well. We compare models selected with the three scores, S,, S2, and S3, presented in Section to the perfor- mance of 12ama-2-70b, i.e. the BMA. All correctness predictors gms are kNN classifiers with k = 5. We also report the performance of the best model according to the "oracle" score 5, which is the upper bound on what can be achieved with model routing, and 5z, which corresponds to S3 with the true p(d', m), i.e., the accuracy of (an imperfect) 8m on d'. Results for neural networks as correctness predictors are in Appendix C.I.
Baselines. We consider three baselines addition to BMA:
Mixture of Experts (MoE) (Masoudnia & Ebrahimpour, 2014) selects an LLM for each input individually based on the correctness frequency of candidate LLMs on neighbors of this input,
average ligelioelioood (LL) (or negative perplexity) selects an LLM most confident in its responses over the test inputs, and
PairRanker (Jiang et al., 2023) uses a model pre-trained to compare LLM generations to select model per input.

---

Published as a conference paper at COLM 2024
Comparison of various model scores for LLM routing with the Oracle model selection and performance of the best model on average (BMA). Best results are highlighted with bold and second best with an underline (excluding Oracle).
Table 2:
LLM routing on HELM:
Acc.
Ratio to Best
Pearson
Spearman
% BMA
# Params
Rank
S1 eq.
0.662
0.855
0.685
0.465
0.17
40.3B
6.172
S2 eq. 3
0.676
0.868
0.636
0.468
0.10
44.3B
5.897
S3 eq. 5, 6
0.694
0.898
0.727
0,492
0.48
49,8B
5.310
S3 true p
0.735
0.944
0.799
0.596
0.22
33.8B
3.800
LL MoE
0,684
0.869
0.714
0.459
0.10
6.517
0.635
0.825
0.08
34.0B
Pair Ranker
0,455
0,605
0.04
BMA
0.688
0.884
1.00
70.0B
6.069
Oracle 0.773
1.000
0.21
29.1B
1.000
The latter two baselines require producing generations with every LLM at test time make selection, while all of our scores only require generating with the chosen LLM.
Results. We conduct 29 sets of experiments, each time selecting 28 of the datasets the benchmark data for training the LLM routers and using the remaining task as the new task d' for evaluating the quality of the LLM selection for this task. In Table we report averages across experiments for the performance of the selected model (Acc.), ratio of this performance to the performance of the best model for the corresponding new task (Ratio to Best), Pearson and Spearman rank correlations between model accuracies and model scores, number of parameters of the selected model (# Params), and rank of the selected model out of 18 considered (Rank). We also report the fraction of times the BMA is selected by a method (% BMA). Best results are highlighted with bold and second best with an underline (excluding Oracle).
First, we notice that accounting for imperfections of the correctness predictors (their aver- age accuracy is 0.59) has clear benefits: when we have access to the true accuracy of correct- ness predictors, the corresponding score, S3 true noticeably outperforms all other scores. Our simple kernel smoothing estimator of this accuracy (MAE= 0.116) allows to obtain practical model routing score S3 that outperforms BMA (1lama-2-70b while choosing smaller models for some of the tasks (as evident by the average number of parameters of the chosen models). S2 sacrifices some accuracy but chooses even smaller performant models. Overall, learning from benchmarks allows us to obtain LLM routers that can im- prove overall performance while utilizing smaller models where appropriate. We note that log-likelihood (LL) also performs well. However, routing with requires passing each test input through each candidate LLM, which has 347B parameters in total.
Pair Ranker and MoE baselines attempt to select an LLM for each input individually and underperform in this experiment. Selecting a model per input has the potential to outper- form the oracle and achieve perfect accuracy (provided there is at least single LLM that is correct each input), but it is very challenging to conduct such model routing accurately on a new, i.e. OOD, task.
Reducing the OOD gap. The average accuracy of correctness predictors across tasks and models for the experiments in Table 2 is 0.59. It is a fairly low accuracy for binary classi- fication, which we attribute to the diversity of tasks in HELM leading to substantial dis- tribution shifts when predicting the correctness of LLMs on held-out tasks. We investi- gate the quality of model routing when we reduce this OOD gap. A simple strategy to reduce this gap is to collect a small number of labeled in-distribution samples. This can be accomplished by asking practitioner to provide reference answers (1 s) for small number of inputs from their task to evaluate the correctness of candidate LLMs on these in-distribution inputs and use it to improve correctness predictors.
We simulate this scenario by moving min(ar" ,50) samples from the data from a new task d' to the data for training the correctness predictors. The upper limit of 50 samples is to
7

---

Published as a conference paper at COLM 2024
maintain practical utility while accounting for varying dataset sizes (see Table 5). We also compare to Few-Shot, which selects an LLM for the new task based on the performance only on these labeled samples from the new task. We conduct 29 sets of experiments, repeating each one 10 times to obtaín standard deviations (randomness is due to random selection of data points from new task for reducing the OOD gap). We summarize the average accuracy of models selected with various routing scores for varying a in Figure 2 (a corresponds to Table 2). Results for Pearson correlation are in Figure 6(a).
We see that even a small number of in- distribution samples (a 0.05) can reduce the OOD gap (corresponding average accuracy of correciness predictors 0.65; see Figure 6(b))
0.78
Average Accuracy
and noticeably improves the model routing per- formance of all three of our scores. When the
0.74
0.72
180
number of in-distribution samples further in-
creases, S starts to outperform S3. We attribute this observation to kNN being well-calibrated in-distribution, i.e, the correctness predictors provide reliable estimates of their own confi- dence P(y(x, m) = 1[x), which are used by S in equation Finally, we note fairly large variance in the results due to random selection of the in-distribution training samples from d', suggesting that active learning (Settles, 2009) can help to improve LLM routing further.
0.70
5) 11000
02.00.1
0.68
MoF
SMA
Grcce
0.66
TwerShat
0.00
Figure 2: Using min min 50) training samples from d' to reduce OOD gap.
5.2 Model Routing Mix-Instruct
We further demonstrate approach in a different setting and task type, on the MixInstruct benchmark dataset (Jiang et al., 2023). The dataset is composed of instruction-following tasks, divided into train/validation/ test sets of 100K/5K/55 samples, and includes evaluations of N 11 open-source LLMs using common met- rics, e.g. BERTScore (Zhang et al.,
BLEURT BARTScore BERTScore
75
Ours
70
Oracle
0.1
0.2
0.3
0.4
0.5
0.6
-300
51%
845
97%
-3.5
Percentaga used
-4.0
0.1
0.2
0.3
0.4
0.5
0.6
2020), BARTScore (Yuan et al., 2021), and
BLEURT (Sellam et al., 2020). In Jiang
-0.25
0.50
et al. (2023), this benchmark was used
-0.75
to compare different LLM ranking meth- ods in per-instance model selection. We follow the same setting and apply our score S₁ (m, d') to the test set, per-instance, where we use the 100K-sample train set as the benchmark data for training our LLM router. See Appendices A and D for details on the score computation and the experi- ment parameters, respectively. Due to the per-instance setting and the test set constructed from in-distribution data, we focus on our simplest router model S₁, equation
0.1
0.2
0.3
0.4
0.6
Maximal distance
data
Figure 3: Average metrics on subsets of the MixInstruct test set, defined by limiting the maximal average distance between test in- stances and their closest neighbors in the ref- erence (train) set.
We compare our approach with the scoring methods examined by Jiang et al. (2023), as well as scoring based on the average log-likelihodd (L...) of the model responses to the inputs. Additionally, we present the metrics for the best models on average (BMA), Open-Assistant (LAION-AI, 2023) and Vicuna (Chiang et al., 2023).
We report the results of BERTScore, BARTScore and BLEURT in Table 3, along with the number of model calls per instance (MCPI), for N LLMs, performed during inference time. All compared methods require model generations for every point in the test set, by each of the examined LLMs, whereas our approach requires only one model generation and one call to some general embedding function. In addition, all methods, except for LL, require
8

---

Published as a conference paper at COLM 2024
Table 3: Average metrics for per-instance LLM selection on the MixInstruct test set.
BERTScore f
BARTScore
BLEURT 1 MCPI
Random
66.36
-3.76
-0.77
LL
65.83
-4.12
-0.96
BMA: Open-Assisant
74.68
-3.45
-0.39
BMA: Vicuna
69.60
-3.44
-0.61
MLM-Scoring (Salazar et al., 2020)
64.77
-4.03
-0.88
N
SimCLS (Liu Liu, 2021)
73.14
-3,22
-0.38
SummaReranker (Ravaut et al., 2022)
71.60
-3.25
-0.41
N
PairRanker (Jiang al., 2023)
72.97
-3.14
-0.37
Ours
74.75
-3.40
-0.38
2
Oracle
77.67
-2.87
-0.15
N
training auxiliary language models, whereas our approach is simple kNN classifier on the embedded inputs. While our approach does not consistently outperform the compared methods, these results demonstrate the potential of using benchmark datasets for model routing with significantly better inference-time efficiency.
Effect of benchmark dataset sparsity. To highlight the potential of our approach in this set- ting, we examine the effect of the reference benchmark data sparsity. We apply our method to different subsets of the test set, X where the subsets are defined by limiting the max- imal average distance of each test set point to the closest points from the reference (train)
set, denoted by NNuain ie. X = fx' E Xtest [x") ExeNNeam(1) dist(x',x) < c)
where is the maximal average distance and X the resulting subset of the test set. Figure presents the metric scores for the different subsets using our method, the oracle (best possible choices), and LL scoring, We also report the percentage of the test set that is used in each subset. This figure depicts that our predictor approaches the oracle metrics as the average distance to the reference points decreases. This suggests that adding more benchmark datasets reduce the sparsity of the reference space may lead to better LLM selections with our approach.
Discussion and Conclusion
How useful are smaller LLMs? While given LLM may work best average, these mod- els tend to be the biggest and therefore most expensive to run. Practitioners can achieve gains in cost, compute, and latency if we successfully predict whether smaller LLM can be adequate for a given task. Identifying good smaller models for tasks interest will also redefine the cost/bo benefit tradeoff behind automating certain tasks, potentially incen- tivizing the automation of new tasks that were previously cost- prohibitive to automate with larger LLMs.
To evaluate the potential of smaller LLMs we revisit our HELM experiment in Figure 2. In Figure 4, we perform LLM routing using only models with 13B parameters and compare it to the performance of Llama 2 70B. Oracle's per- formance demonstrates that is conceptually possible to outperform large model by rout- ing smaller LLMs. Results with our scores S and S2 (see Figure 7 for breakdown by scores) demonstrate that is also practically feasible to match the performance of the 70B model by combining learning from benchmarks with small number (a 0.04, i.e., 2-40 samples) of
0.76
Average Accuracy
0.72
0.70
0.68
0.66
0.64
(wmm 73R
0.62
Oraci
0.00
0.25
Figure 4: Routing with < 13B models.

---

Published as a conference paper at COLM 2024
labeled samples from new task that practitioner can provide to save on the inference costs in their LLM application.
Learning from more benchmarks. We antic- ipate learning LLM routers from benchmarks to be the most effective when new tasks similar to the benchmark tasks, thus reduc- ing the OOD gap without any labeling burden for a practitioner. To empirically investigate this hypothesis, in Figure 5 we visualize the relation between the quality of model routing with S3, measured with Pearson correlation be-
1.0
0.9
0.8
Ceris, eg
0.7
0.6
tween model scores and accuracies of candidate LLMs, and the distance u (dd)) from new task d' to the available benchmark data for training the routers. In this experiment, we aggregate results across different values from Figure 2. For smaller distance values the correlation is approaching 1, while for large distances it sometimes deteriorates. Results for other scores demonstrate similar trend and are pre- sented in Appendix C.3 along with additional details. This experiment and the benchmark dataset sparsity analysis presented in Figure 3 for MixInstruct illustrate that learning with more benchmarks can improve the efficacy and reliability of the LLM routers as new tasks are more likely to be closer to a larger collection of datasets.
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Dataset Distance uld')
Figure 5: Correlation(Ss, Accs.) vs u(d)..
Future work. Our work demonstrates the potential of learning from benchmarks for LLM routing and investigates 3 model scores in the context of OOD generalization when routing LLMs for new tasks. We summarize the potential next steps to advance LLM routing.
The major challenge of LLM routing is OOD generalization of correctness predictors. Thus, using more benchmarks and modern methods for improving OOD generalization to learn correctness predictors is a promising next step. A practitioner can also provide labels for few samples from their task, possibly guided by active learning techniques, to adapt or fine-tune correctness predictors. Even when reducing the OOD gap is too challenging, our score accounting for the (potentially low) accuracy of correctness predictors demonstrated strong results when this accuracy, p(d', m), is known for a new task, thus encouraging the development of methods for estimating it better.
We anticipate that routing "expert" LLMs fine-tuned for a specific domain can improve the results. Regions of the sample space where such models are "correct" should mostly align with the domains of their expertise (recall Figure 1), making it easier to learn the correctness predictors and simplifying LLM routing when a new task is from specific domain.
Our experiments in Figure demonstrate the utility of LLM routing with smaller models, which can reduce costs and facilitate using LLMs in broader set of domains. Thus, we want to explore modifications to our scores that will encourage the selection of smaller LLMs when their anticipated performance is comparable to the larger, more reliable mod- els. Prior work on frugal API selection (Chen et al., 2020; 2023) provides a good starting point to explore this direction.
Acknowledgments
The MIT Future research group recognizes the generous support of the MIT-IBM Wat- son Al Lab, Boston Scientific, and Good Ventures.
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants W911NF2010168 and W911NF2110293, of Air Force Office of Scien- tific Research award FA9550-19-11031, of National Science Foundation grant CHS-1955697, from the CSAIL Systems that Learn program, from the MIT-IBM Watson AI Laboratory, from the Toyota-CSAIL Joint Research Center, from a gift from Adobe Systems, and from Google Research Scholar award.
10

---

Published as a conference paper at COLM 2024
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk min- imization. arXiv preprint Stte:1907002893, 2019.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leader- board.
co/ http://gggggggaaaaa///opp.eeen//
2023.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Ondfej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Jo- hannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pp. 12-58, 2014.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint ar 1rXic:2108.07258 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34:14980-14999 2021a.
Lingjiao Chen, Matei Zaharia, and James Y Zou. Frugalml: How to use ml prediction apis more accurately and cheaply. Advances in neural information processing systems, 33: 10685-10696, 2020.
Lingjiao Chen, Matei Zaharia, and James Zou. FrugaIGPT: How Use Large Lan- guage Models While Reducing Cost and Improving Performance. arXio preprint ar arXit:2305.05176 2023.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In Infernational conference on machine learning, pp. 1617-1629, PMLR, 2021b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //msss//mmss..../ wttoolog/////ooo/oooooooooeooego 2001//22333--0 0000022--33000 -0-3000---0000 30-vviuna/a
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480s 2018.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. Xixiv preprint ar 2426:2300..04166 2023.
11

---

Published as a conference paper at COLM 2024
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL htttps://oi..... 5281/zenodo.533 5371666
Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of -fitt--diittiittioon performance. In International Conference on Learning Representations, 2022.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking supervueee visual representation learning. In Proceedings of the ieee/cof International Conference on computer vision, pp. 6391-6400, 2019.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 1134-1144, 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Inter- national Conference on Learning Representations, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, arXiv preprint arXi::2009 2020.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arviv preprint Xix::2001044444 .00000, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14165- 14178, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10. 18653 18653/v1/20235 wllo020.acl-long.. 792. 792. URL https: // www.nthologg..oooo/ www.th/llggy.oor222...eoo attl:/2223.ccclong acl- 792.
Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing General- ization of SGD via Disagreement. In International Conference on Learning Representations, 2021.
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Repository, 2023. URL https: ics. uus.u.ci.edu
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xix::1412.660 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Ak- shay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp- 5637-5664. PMLR, 2021.
LAION-AI. Open assistant, 2023. URL httpp:///gtt //tp::/b/ghtbt..../ ctt////AONAAAA/ope/AAsittttstet
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating aanguage-ugmmented visual models. Advances in Neural Information Processing Systems, 35:9287-9301, 2022.
12

---

Published as a conference paper at COLM 2024
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint ar Xix:2211.091100 2022.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXio preprint ar arXiv:2303.16633 2023.
Yixin Liu and Pengfei Liu. Simcls: A simple framework for contrastive learning of ab- stractive summarization. In Proceedings of the 59th Anmual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume Short Papers), pp. 1065-1072, 2021.
Subha Maity, Mikhail Yurochkin, Moulinath Banerjee, and Yuekai Sun. Understanding new tasks through the lens of training data via exponential tilting. In International Conference on Learning Representations, 2023.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275-299 293, 2014.
Nathan Hoyen Ng, Neha Hulkund, Kyunghyun Cho, and Marzyeh Ghassemi. Predicting out-of-domain generalization with neighborhood invariance. Transactions on Machine Learning Research, 2023.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arxiv preprint ar drXic18811112666 2018.
Mathieu Ravaut, Shafiq Joty, and Nancy Chen. Summareranker: A multi-task mixture-of- experts re-ranking framework for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 4504-4524, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics, 2019.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 2699-2712, 2020.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for D text generation. In Proceedings of the 58th Annual Meeting of the Association for Computa- fional Linguistics, pp. 7881-7892, 2020.
Burr Settles. Active learning literature survey. 2009.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xiv preprint ar arXii:2200.044 04615, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adap- tation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443 450. Springer, 2016.
13

---

Published as a conference paper at COLM 2024
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bow- man. Glue: A multi-task benchmark and analysis platform for natural language under- standing. ar preprint 0rXii:1804.07466 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Advances in neural information processing sys- fems, 32, 2019.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? preliminary study. arXiv preprint ar (rXiv:2303.04048 2023.
Hui Xu and Robert libshirani. Estimation of prediction error with known covariate shift. ar Xiv preprint ar arXiv:200500184 2022.
Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out- of-distribution error with the projection norm. In International Conference on Machine Learning, pp. 25721-25746, PMLR, 2022.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Represen- tations, 2020.
A Correctness predictors and confidence estimation
We provide additional details on the correctness predictors used in our experiments, along with more details on the dataset distance and the Gaussian kernel smoother for estimating the accuracy of the correctness predictors on new tasks, p(d', m)s.
Correctness predictors in our experiments While any probabilistic classifier may fit our setting, in the experiments, we mainly used simple kNN classifier applied in an embed- ded space. Recall that we have D benchmark datasets with inputs {x4) for d = 1,..,, D. To compute our correctness predictor based on the benchmark datasets, we first embed all their inputs. We denote the combined set of embedded inputs from the benchmark datasets as D {*(x))..... o (xx,,) 1 where Φ sentence transformer (Reimers Gurevych, 2019). We use all -aaonett-bee-e from Hugging Face in all experiments. Given sample x from new task d', we embed it using the same & and define the classifier, gm, for each model m by:
Σ
m)),
8m
K
CENN(*(X," JADD)
where y(e,m) E (0,1) is the correctness of model m on the (embedded) input and NN(Φ(x)) ), D,) the set of closest embedded neighbors from D to the new embed- ded sample ((x(') according to the cosine distance. Then, &m, as defined in equation 3, is binary kNN classifier. Finally, we compute the per-model correctness predictors, Sj (m, d') and 52(m,d'), for the new task d', according to equation 2 and equation 3, respectively.
Next, we describe method for estimating the probability p(d', m) in our confidence model and the Sy(m, d')) score, equation 5. This method comprises a task descriptor based on dataset distance and a kernel smoother, defined follows.
14

---

Published as a conference paper at COLM 2024
Task descriptor dataset distance To compare tasks need to assign them some nu- merical representations, i.e, task descriptors. Here we utilize dataset distance as task de- scriptors as follows. Our dataset distance u(d) is one-sided variant of the Chamfer dis- tance with extended neighborhood size. We define it formally below:
1 nd nn ,
(d)
L Rd
where D-a J is the set of (embedded) inputs from the datasets excluding inputs from
(for new task d', D_d'" = DD since d' is not part of the D benchmark datasets we use for training LLM routers), and nn(x,, D a) is the average distance from the input x to its closest x neighbors D-d:
1
nn(x, D)
Σ
cosine((((x),e),
(9)
cENN(((())x,))
where (N))(x), D) is the set of x closest embedded neighbors of (xx) in according to cosine distance. We set 19 for the dataset distance in all experiments.
Kernel smoother For each LLM m = 1,..,,M, to obtain the corresponding kernel smoother estimate iterate over the available benchmark datasets, each time holding one out and computing pairs (u(d), p(d, m)) for held out dataset d, where pdd, m) is the accuracy of 8m on data from d after training on D-4- We repeat this process 10 times for 15 values of in-distribution mixing parameter a (similar to the experimental setup in Figure but using benchmark datasets d I,..., D instead of d') to obtain the training of distance-accuracy pairs {uz, Pz p=(m) -
In the HELM experiments in Section 5.1, the new task for evaluating the performance).
Z = 28 * 10 # 15 4200 (28 is the number of datasets from HELM after holding one out as
For a new task d', we compute u(d") using the inputs from this task and our benchmark datasets and estimate p(d', m) for each m with simple Gaussian kernel smoothing:
p(d', IZ (m
(10)
EZ_1 ((u(dd), uz)
where (u((d',, exp( (- (u(d)))))))) 20t2
We set 0.09 in all experiments, which is the
value we found to perform well through some preliminary experimentation.
Finally, we note that the proposed confidence model, including the definitions of the dataset distance and kernel smoother, can be combined with any classifier gm, and is not restricted to the kNN classifier used for the correctness predictor in our experiments.
Additional notes regarding S3 Recall that when selecting model with S3 d') use an additional step described in equation 6 that facilitates the selection of the best model on average when we are not sufficiently confident in the model with the highest S3 .') score. Probability expression, P(S(m,,,d)) > S(m*, ,d))), required for this step is not avail- able in closed form, as Š is distributed as a (scaled) sum of two Bernoulli random variables, but it is straightforward to estimate via Monte Carlo sampling from the corresponding Bernoulli distributions.
When reporting correlations for 53 (e.g., Pearson and Spearman correlations in Table 2), we use S3 (m,d') as is, i.e., as defined equation
B Connection to meta-learning
The OOD confidence model in equation is meta-model of routing across multiple tasks, and fitting it entails a form of meta-learning. Consider the meta-learning problem
min (.)P()
m) + (1 - Zm (1 p(d,
(11)
15

---

Published as a conference paper at COLM 2024
where 8m and (()) are meta-parameters and adaptation step &m - gm (-)(,( ()) adap- tively shrinks the router output towards ambiguity. We exploit this connection to theoreti- cally demonstrate the potential advantages of routing LLMs using S3 over S2.
In expectation/ in the population, equation 11 fits a larger model class than equation 1, so the risk of the adaptively shrunken router is at most that of the non-adaptive router:
E E (l (X*) p(d, m) + (1 - 8m (x - p(d, (0),m)),(1 y( m))]
(12)
S (x)) yx mm)]]-
This suggests (subject to standard assumptions the loss function) that adaptive shrink- age routing leads to better approximations of the oracle router. Lemma B.1 confirms this intuition.
Lemma B.1. Let E(y1,y2) p(y1 1/2 for some subadditive p. R R (e.g. p(x) 2 for the square loss). We have
5( - [xe(gm(X*),, y(x", m))]),
{(S3, 5) < E[E(p(d, mm + (1 - (d), 8m(( .m))
Proof. We start by showing the upper bound of E(S2, S):
S) p(E (xx) E(y(X", m)])
(def of E)
= pmE[gm x4) - y(x*, m)])
E ( (X4) y(x", m))l)
(convexity of p)
[[((Em (X*), ), yxx,, m))]),
(def of E)
where we recalled that subadditive functions are convex in the third step. The upper bound of (S3,, 5) follows similar argument:
S) = p(p(d, m) 3m [x]] + (1 p(d, [[ (x) - E [m((),m)]
[V[[p(d,m) pm X + (1 ([d,m))(1 - 8m (X2) )) - y(x,, mi])
I Elp(p(d,m) x (1 m)) 8m (X y(x", m))])
M() x²) + (1 - p(d, m)) - 8m (x (xx)), )),y()), (m)))
S (x m))]). (x))))
Combining equation 12 and Lemma B.1, we expect the adaptive router based on S3 to outperform its non-adaptive counterpart based on S2. That said, it unclear whether adaptive shrinkage will improve the performance of the adaptive router in finite samples: the expected performance of the adaptive router may be offset by the inflation in variance from fitting the larger (adaptive) model class. Fortunately, task-specific adaption, i.e., using S3 as a score for routing, generally improves performance. The two-step method for fitting &m and p in Section approximately minimizes equation 11 with single Gauss- -eidell pass through the decision variables.
empirical results show that
C Additional results for model routing on HELM
C.1 Results with MLP as the correctness predictor
MLP Classifier We present LLM routing with more sophisticated correctness predictors. In this section, use small Multi-Layer Perceptron (MLP) as 8m to classify the embed- ded inputs. Each MLP comprises three fully connected layers (two hidden layers, 1500 units each) with ReLU activation and sigmoid output. These MLPs were trained with the
16

---

Published as a conference paper at COLM 2024
Table 4: Average metrics for the experiments using MLP. Best results are highlighted with
bold, and second best with an underline (excluding Oracle).
Acc.
Ratio to Best
Pearson
Spearman
% BMA
Params
Rank
S1 eq. 2
0.693
0.893
0.750
0.559
0.19
57.6
4.345
S2 eq.
0.683
0.878
0.675
0.547
0.45
57.9
5.041
S3 eq. 5, 6
0.690
0.884
0.755
0.544
0.81
66.3
5.566
S3 ATC
0.646
0.827
0.651
0.365
0.56
48.7
7.283
S3 true p
0.740
0.943
0.840
0.686
0.43
50.4
3,455
LL
0.684
0.869
0.714
0.459
0.10
6.517
BMA
0.688
0.884
1.00
70.0
6.069
Oracle
0.773
1.000
0.21
29.1
1.000
0.139
0.89
0.134
Reston Corrooore
Auerop ACC.
0.35
0.4
0.12
0.38
0.50
01:04
0.35
0.35
(a) Pearson correlation
(b) Accuracy of 8ms
(c) Estimatiio of p(d', mms
Figure 6: Additional results for Reducing the OOD gap experiment in Figure 2.
Adam optimizer (Kingma & Ba, 2014) with learning rate 0.01 to minimize the binary cross entropy loss described in equation 1. Training occurred over 100 epochs with early stop- ping should the validation accuracy not increase for 10 consecutive epochs. We also explore the combination of S3 with ATC (instead of the kernel smoother), prior OOD accuracy estimator proposed by Garg et al. (2022).
Results We conducted the model routing experiment analogous to the experiment in Ta- ble of the main paper. Results are reported in Table 4. We note several differences com- pared to results with the kNN classifier in Table 2. First is the clear improvement in the S score; the MLP classifier performs better on OOD data. As before S3 true p outperforms the other scores indicating the potential value of using the proposed confidence model 4. How- ever, in this experiment, S3 with the kernel smoother no longer improves upon S,. We also report results with ATC as the estimator of the accuracy of correctness predictors, which performs noticeably worse than our kernel smoothing estimator. The corresponding MAE 0.118 for the kernel smoother and 0.177 for the ATC, demonstrating the advantage our estimator in this application. Finally, across all scores the selected model sizes have con- siderably increased; this suggests that the MLP classifier underestimates the performance of smaller models compared to kNN.
C.2 Reducing the OOD gap
We present additional results for this experiment in Figure (a) shows Pearson correlation improvement as we increase similar to the trends in accuracy improvement in Figure 2; (b) demonstrates that the accuracy the number of samples from d' used for training them, thus reducing the OOD gap; (c) shows the mean absolute error (MAE) of our kernel smoothing estimator of the accuracy of correctness predictors pdd', m) the estimator does not improve much with increased a, thus S3 eventually becomes worse than S₁ in terms of correlation and accuracy the selected models.
correctness predictors gms improves as we increase
17

---

Published as a conference paper at COLM 2024
Diergge Accurny
Nveeage Accurecy
0.2
(a) S1 equation
(b) equation
(c) S3 equation 6
Figure 7: LLM routing with v 13B parameter models compared to Llama 2 70B.
Pespaon Cards, M4, A003
Pearson Conts ttoee Acos)
233
244 05
03 24
Darsset Distance MGI7)
Dataset Distance 6CP1
Datsset Distance uif1
(a) S1 equation 2
(b) S2 equation
(c) S3 equation 5; true p(d', m)
Figure 8: Correlation of scores and LLM accuracies on new tasks and corresponding data
distances.
C.3 Dataset distance and Pearson correlation
The dataset distance udd)) computed as in equation 8. As evident from equation 9, dataset distance will usually decrease for larger values of inputs from d' are moved into (assuming that inputs from d' are on average closer to each other than they are to inputs from other tasks). In this experiment, this serves mechanism to study the per- formance of LLM routing on closer datasets, providing insights into the benefits learning LLM routers on more benchmarks where it is more likely that dataset distance for new task is small.
In Figure 8 we present relations between dataset distance u(d') and Pearson correlation between various model scores and accuracies of candidate LLMs. For results with S3 see Figure
D Additional details for model routing on MixInstruct
Correctness predictor and metrics. the experiments on the MixInstruct dataset (Jiang et al., 2023), we construct S₁ S1 (m,d') following the scoring approach described in Appendix A, where the MixInstruct train set was defined as the benchmark dataset. Then, instead of computing per-dataset score for the entire test set, compute the score for each test point, i.e., S₁ (m, x²" = (m(xx") ), and select model m per-point based on this score. The reported metrics in Table and Figure 3 are averaged over the output evaluations of these per-point model selections.
our experiments, to compute 8m 3m(2f) we use the BERTScore metric on the closest train set points (as (xx m) 7). This was motivated by the conceptual relation between the implementation of our approach and the BERTScore, which relies on embedding space distances, and was validated empirically.
kNN parameter. We set k 10 for the kNN classifier, slightly higher than in the HELM experiments. This choice was motivated by the in-distribution properties of the test set in MixInstruct, which is constructed from different parts of the same datasets that comprise the train set. We note that the metrics did not significantly vary for different choices of k E (5, 100).
18

---

Published as a conference paper at COLM 2024
Table 5: HELM dataset details.
Dataset
Size (instances)
Type
RAFT-ADE Corpus V2
40
Binary Classification
RAFT-Banking 77
40
77 Class Classification
RAFT-NeurIP'S Impact Statement Risks
40
Binary Classification
RAFT-One Stop English
40
3 Class Classification
RAFT-Overruling
40
Binary Classification
MAFT-Semiconductor Org Types
40
3 Class Classification
RAFT-Systematic Review Inclusion
40
Binary Classification
RAFT-TAT Safety Research
40
Binary Classification
RAFT-Terms of Service
40
Binary Classification
RAFT-Tweet Eval Hate
40
Binary Classification
RAFT-Twitter Complaints
40
Binary Classification
IMDB
1000
Binary Classification
Civil Commeets-denoggapiicaal
1000
Binary Classification
bAbl-QA-task-all
1000
Q& one word answers
BoolQ
1000
Binary Classification
Entity
182
Binary Classification
Entity Ditts iTunes Amazon
218
Binary Classification
Entity Matching-Dattsst--Abl Buy
1000
Binary Classification
Entity Data Imputation-Dataset-RResaaa
242
Q&A: one word answers
Entity Data Imputation-Dataset-Buuy
182
Q&A: one word answers
BBQ-subject=all
1000
Multiple Choice Questions
Legal Support
1000
Multiple Choice Questions
LSAT ta-
461
Multiple Choice Questions
AMtsstaubjecttDDSrraat Algebra
111
Multiple Choice Questions
MMLU--Subjectt=College Chemistry
108
Multiple Choice Questions
MMLUUSubjectt-Cmmpueee Security
111
Multiple Choice Questions
MMLLU-Subjctt--ooommetcc
126
Multiple Choice Questions
UM foreign policy
111
Multiple Choice Questions
Truthful QA-task-me single
654
Multiple Choice Questions
Total: 29 datasets
9946
19

---

Published as a conference paper at COLM 2024
Table 6: Candidate LLMs.
Name
Model Size,
Average Accuracy on the 29 HELM tasks
codegen-16b-mono
0.451
dial-flant5-xl
0.454
falcon-40b
0.641
flan-t5-xl
0.650
flan-t5-xxl
0.658
flan-ul2
0.668
8pt-jt-6b-vl
0.576
gpt-neox-20b
20
0.492
mpt-7b-instruct
0.514
mt0-xxl
13
0.543
1lama-2-13b
13
0.624
Ilamm-2-13b--hat
13
0,623
Ilama-2-13bccattbbamm
13
0.603
1lama-2-70b
70
0.688
llama-2-70b-chat
70
0.687
llama-2-7b
0.610
1la-h-2-7b-chat
0.605
starcoder
15
0.587
Total: 18 LLMs
347
20


Reviews:
[{"review_id": "JB87cAccpr", "reviewer": "Reviewer_Gn6h", "summary": "This paper aims to selecting the best LLM from a collection of pretrained models for an unseen task. It is useful and motivates well. It also quantifies a rooting function to predict performance. The authors propose repurposing a collection of benchmark datasets to learn a \"router\" model for LLM selection from inputs only, reducing the problem to a collection of binary classification tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and encouraging feedback.\n\n**Clarity.** We are glad that the idea and motivation are clear and will revise the technical part to improve clarity. We acknowledge that the notations could appear a bit dense and will use the extra page allowed for the camera-ready version to add a table summarizing the notations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Addresses a pertinent challenge in the field of natural language processing, i.e., selecting the best LLM for a new task without labeled data.\n\nIntroduces an innovative approach by repurposing benchmark datasets for LLM selection, potentially offering a cost-effective solution compared to traditional methods.\n\nAn empirical evaluation is conducted to validate the proposed approach, providing practical insights into its effectiveness and feasibility.", "reasons_to_reject": "I am clear about the idea and motivation. However, it is really confusing with the detailed methods, for Sections 3 and 4, especially with so many new notions.\n\nLanguage and terminology should be refined for clarity and precision, especially in technical descriptions and explanations.\n It would be beneficial to include visual aids such as figures or tables to illustrate key concepts or experimental results for better comprehension.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZq90UZvqv", "reviewer": "Reviewer_f2Pv", "summary": "This paper propose a method to select the best LLM from a collection of pre-trained models, given a new task. The proposed method does not assume any labeled data from the new task. It learns a “Router” model for LLM selection from inputs only. Experimental results show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, it appears that the posted review is for a different paper. We do not consider domain-specific human evaluation of LLMs and do not propose any method named \"LalaEval\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe proposed method is well-established while the paper is overall well-written.\n2.\tSelecting language models on a new unseen task with low cost is a really interesting and beneficial topic.", "reasons_to_reject": "1.\tThe improvements are limited according to Table 2.\n\n2.\tNo human evaluation, I think there might at least be one to generally prove that the proposed method could predict the quality of a LLM’s output as expected.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "pKjaC4UUAp", "reviewer": "Reviewer_RooH", "summary": "This paper focuses on learning to route different LLMs for downstream tasks without using task-related labeled data. The authors propose to utilize the performance of these LLMs on benchmark datasets and train a router model to do binary classification on the LLMs' performance given the task inputs. The prediction scores are used to conduct the LLM selection. The approach leverages embedded inputs and task descriptions from benchmark datasets and employs k-nearest neighbors classifiers to predict LLM correctness, aiming to generalize to unseen tasks. Experimental results on HELM and MixInstruct datasets demonstrate the potential of selecting suitable LLMs to reduce the cost and improve overall performance. However, the authors mentioned that out-of-distribution generalization is the main challenge of LLM routers.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your review. Please find our responses below.\n\n> Lack of explanation of the usage of KNN models.\n\nWe used kNN models as they are easy to implement and perform well. Our goal is to illustrate the potential of LLM Routing using benchmark data evaluations and demonstrate the efficacy of the proposed scores. That said, we also present results with MLPs as correctness predictors in Appendix C.\n\n> How does the proposed method deal with the out-of-distribution challenge.\n\nAny correctness predictor will face challenges when generalizing OOD (kNN, MLP, or anything else). To clarify, OOD challenge is the problem of deteriorating accuracy of the correctness predictors on unseen tasks. For example, the average accuracy of correctness predictors for results reported in Table 1 is only 59\\% (quite low for a binary classification task - see \"Reducing the OOD gap\" paragraph on page 7). Our key insight is that our method can still select an appropriate LLM for a new task by accounting for errors of the correctness predictors. To achieve this we designed S3, a score that models confidence of the correctness predictors OOD (see \"A simple OOD confidence model\" at the bottom of page 4 and eq. (4)).\n\n> Lack of evaluation on more realistic tasks or instructions.\n\nWe respectfully disagree. We conducted experiments on 29 tasks from HELM, one of the prominent LLM benchmarks designed to cover scenarios representative of realistic LLM use cases. Among the 29 tasks we considered, there are 11 RAFT tasks, which stands for \"**R**eal-world **A**nnotated **F**ew-shot **T**asks.\" [1]\n\nWe also conducted experiments on MixInstruct, which aggregates multiple datasets designed to test instruction following abilities of LLMs in various contexts (potentially including inputs requiring compositional abilities).\n\n[1] Alex et al. RAFT: A Real-World Few-Shot Text Classification Benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This motivation is clear.\n- Extensive experimental results demonstrate the effectiveness of the proposed methods.", "reasons_to_reject": "- Lack of explanation of the usage of KNN models. How does the proposed method deal with the out-of-distribution challenge.\n- Lack of evaluation on more realistic tasks or instructions. Some instructions require the LLMs to use multiple or a combination of capabilities to solve, which is hard to reflect on the benchmarks. This composition problem needs to be discussed for the proposed method.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsNg9UeoG0", "reviewer": "Reviewer_jebY", "summary": "This paper propose a new approach to LLM selection without relying on labeled data from the new task. Instead, they develop a \"router\" model trained solely on inputs from various benchmark datasets, which reduces the selection problem to a collection of binary classification tasks. Empirical results show that this method consistently outperforms using a single model for all tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Please find our responses below.\n\n> Routing instances vs routing datasets.\n\nWe considered both. In our HELM experiments, we chose a single LLM per dataset. We also compared to choosing an LLM per instance (MoE baseline in Table 1 and Figure 2 - note that here MoE is meant as a \"classical\" mixture of experts, not an MoE LLM like Mixtral), which underperformed due to the difficulty of accurately predicting correctness of LLMs on new tasks. In the MixInstruct experiment (Table 2), we routed per instance. Here our method outperforms or matches recent baselines while being a lot more efficient as it requires generating *only* with the selected LLM (and a much smaller sentence transformer for obtaining the input embedding).\n\n> How do you effectively control the efficiency cost of maintaining multiple LLMs like MoE?\n\nOur method selects from a pool of LLMs using the inputs and calls only the selected LLM for generation. All other LLMs are unused and do not consume any resources. This is more efficient than an MoE LLM. For example, consider our HELM experiment when choosing a single LLM per dataset. At first, the router chooses a single LLM. This could be a 7B parameter LLM that is then loaded onto the GPU and used for inference. An MoE LLM like Mixtral 8x7B requires storing all of its ~47B parameters on the GPU, even though only 7B are active during inference. Larger GPU memory requirement implies that it needs a more advanced GPU and leaves less memory for KV-cache, which slows down the inference when compared to routing to a 7B model with our method.\n\n> There lacks a deep discussion how this approach is compared with MoE, in both advantages and disadvantages.\n\nIn addition to the discussion above, we note that the goal of our approach is somewhat orthogonal to how MoE LLMs work. Our goal is to learn the strengths of candidate LLMs from benchmark evaluations to choose an LLM for a new task using *inputs*. MoE LLMs, despite the name, do not typically consist of \"experts\" and route to different FFN networks for each *token* within each layer. Quoting Mixtral authors (see Section 5 in [1]): \"Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.\" An MoE LLM should simply be considered among the pool of candidate LLMs when using our method as it doesn't have a well-defined set of strengths of experts apriori. We will add a discussion on MoE LLMs to the paper.\n\n[1] Jiang et al. Mixtral of Experts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) An interesting task setting;\n(2) A formal approach with extensive experiments", "reasons_to_reject": "(1) If I understand it correctly, you still assign a unique LLM to solve some specific task, but not different LLMs to solve a task? Basically, it would be intuitive, even if a capable model can still underperform a weak model on some instances of a specific task. Thus, it seems more reasonable if you route instances but not datasets to LLMs.\n\n(2) Efficiency can be potentially an issue: since you use lots of LLMs, how do you effectively control the effiency cost of maintaining multiple LLMs like MoE?\n\n(3) There lacks a deep discussion how this approach is compared with MoE, in both advantages and disvantages.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on addressing the following problem: given a number of LLMs, pick the best one per input to optimize for quality. The proposed approach relies on training a binary classifier per LLM on existing benchmarks. At inference time, it does not rely on labels and only operates on inputs and the classifiers -- a collection of binary classification jobs are run to predict which LLM will be the most accurate. The idea of using sentence transformer for embedding the data and using KNNs as correctness predictors for new tasks is interesting. The approach is computationally efficient to implement, fast at inference Extensive results (e.g. on HELM) shows the effectiveness of the approach considering that it only relies on the input, although for truly out of distribution tasks the generalization results are understandably less promising.
Decision: Accept