On Speeding Up Language Model Evaluation
Authors: Jin Peng Zhou
Christian K Belardi
Ruihan Wu
Travis Zhang
Carla P Gomes
Wen Sun
Kilian Q Weinberger
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
evaluation
matrix factorization
Primary area: foundation or frontier models, including LLMs
TL;DR: We show that the best language model and/or prompt can be identified with just 5%-15% of the usual computation using our adaptive evaluation algorithms.

Abstract: Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an \textit{adaptive} approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources---resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval.

Published as a conference paper at ICLR 2025
ON SPEEDING UP LANGUAGE MODEL EVALUATION
Jin Peng Zhou" R Cornell University
Christian K. Belardi"
Ruihan Wu"
Cornell University
University of California, San Diego
Travis Zhang Cornell University
Carla P. Gomes Cornell University
Wen Sun Comell University
Kilian Q. Weinberger Cornell University
ABSTRACT
Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over syper-parameter. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an adaptive approach to explore this space. We are exploiting the fact that often only few samples needed to identify clearly superior inferior settings, and that many evaluation tests are highly correlated. We lean on multi- armed bandits sequentially identify the next (method, validation sample) cumple)-patt -pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top perforrrfmig method using only 5-15% of the typical resources resulting 85 95% LLM cost savings. Our code is available htttts: /ttgtggtththhoeegg com.tttbooommee.... k i an gww.ppbanditeeaaaaa........aaaaa
INTRODUCTION
AlpacoEval Monetary Savings
GSM8K Prompts Time Savings
$794.35
80
700
70
500
50
400
GPU
40
300
79.1%
hr 4.859%
$46.07
$47.9 1 99.0%
100+
J 1.94%%
13+
0-
Full Evaluction
UCB-E
U
Full Evaluation
UCB-E
UCB UCB-E-LRF
Figure 1: Cost comparison for finding the best model or prompt between our proposed algorithms (UCB-E, UCB-E-LRF) and Full Evaluation on two datasets. The overhead computation time for UCB-E and UCB-E-LRF is 2.4 and 142.6 seconds, respectively. See text for details.
Large language models (LLMs) have demonstrated remarkable proficiency diverse tasks such as question answering, machine translation, and mathematical reasoning (Brown al. 2020 Chowdhery et al. 2023 Lewkowycz et al., 2022). They are employed across numerous applications, ranging from automated customer support to content generation (Liang al., 2024). As the development of new LLMs continues, practitioners find themselves with a plethora of options, selecting the optimal model, prompt (Kojima al. 2022). and hyperparameters for their specific needs.
Querying an LLM is resource-intensive, and extensive evaluation requires significant investment in capital, time, and compute. Figure (left) shows the estimated cost of Full Evaluation of the 153 models officially included in AlpacaEval (Li et al. 2023) as of May 20, 2024 to be almost 800 USD. GPT4-turbo is the LLM judge (Zheng al. 2024 and the monetary cost calculated based on the
(left)
"gqual contribution. Correspondence (12563, ckb73)@corne1 edu and ruw0760ucsd. edu.

---

Published as a conference paper at ICLR 2025
Table 1: Selected benchmarks where L.LMs are state-of-the-ar methods. We group them based on the task type and whether additional LLM-based / humann -base scoring is needed for evaluation.
Task Type
LLM Inference + Rule-based Scoring
LLM Inference LLM-based Scoring
B Dhamala et al. 2021 GLUE Wanr al. 2019
BOLD
CNN/Dailymall Nallapaii all 2016
Natural Language Understanding
HellaSwag Zetters si. 2019 SQuAD Kajpurkar eral 2016 Trivia
Newsroon Grusky al
Jushi frassi
an. 2017 Winobmande atz 2019
XSUM XSUM[Ninyan Narayan el al
2018
Google NO Kwintkowski et all 2019 HetpaiQA Yang al. QASPER Dassgg el 202
AlpacaEval st A025
Open-ended QA
Chatbot Arena Chinl 1
2024
MT. Zheng
APPS APPS| [endrycks et al. [021a] Ard Clark et al
2018
CPQ Rem 2025 USNNKK Cobbe at
STEM und Reasoning
MATH Headiyeks et at 2021e MML Uiwnnnyycs a.
2021b
PIQANIIS
inference cost of only proprietary models and GPT4-turbo judging cost. Similarly, 78 Nvidia A6000 GPU hours are needed for evaluating 205 zero-shot prompts on 784 GSM8K (Cobbe et al. 2021 questions using Mistral-7B (Jiang et al. 2023). Figure[[rright
In this paper we investigate the familiar setting where given a set of methods we aim to identify the best performing across a set of validation examples - given a fixed evaluation budget. For example, the different methods could be LLM prompts or hyperparameter settings and the validation examples could be translation tasks or K12 math problems. The budget could be a fixed monetary sum or GPU time. We argue that this setting is extremely common among LLM applications, where the practitioner is primarily interested identifying the single best method to deploy.
A simple baseline is to evenly split the budget for each method. However, this can be very inefficient as it would spend a lot of the budget on obvious non-performers and too little effort on identifying the very best method. Intuitively, methods that consistently underperform other methods to end up as the best. Therefore, a more efficient budget allocation strategy is to spend less on low-performing methods and more on promising ones. Multi-armed bandit algorithms enable this dynamic allocation by actively selecting the next method method-example example pair to evaluate based on the results of previous evaluations.
unlikely
We propose two active selection algorithms UCB-E and UCB-E-LRF. Our first algorithm is an extension of the classical UCB-E (Audibert and Bubeck 2010) solve the multi-armed bandit problem. It estimates the upper confidence bound (UCB) to guide the selection of the method which paired with randomly chosen example for the next evaluation in order to efficiently estimate the best method. This algorithm has a strong theoretical guarantee that the probability of selecting the best arm approaches 100% with an exponential convergence in the number of evaluations. Our second algorithm, UCB-E-LRE (for Low-Rank Factorization), leverages the fact that in practice many validation samples are similar, and their results are highly correlated. In other words, if a method performs well on one sample, it will predictably also perform well similar samples. We can therefore speculatively fill in those results and evaluate the method on a sample for which the performance is more uncertain. In our method, we capture this insight through low-rank approximation of the evaluation matrix. By deploying this approach in addition to UCB-E, UCE-E- LRF actively selects both the method and example to evaluate, which leads to even bigger budget savings in settings with correlated validation samples.
We evaluate the efficacy of these two algorithms, in addition to common non-active baselines, in number of practical settings. Our empirical analysis shows that our two active selection algorithms much better than all baselines. Not surprisingly, the benefits of active selection larger if the top methods are separated by a bigger performance gap. Notably, we observe that UCB-E works best for such easier settings; and UCB-E-LRF shines in harder settings, when the performance differences between methods are more subtle.
NOTATION AND PROBLEM FORMULATION
A typical evaluation workflow in large language model (LLM) applications involves three steps: inference, scoring, and performance aggregation.
Inference. Given a dataset of examples X = {ZI,.... ②n), outputs are generated by specific L.-M-based methods F = (J1,..., Here, methods can differ based their L.M architecture
2

---

Published as a conference paper at ICLR 2025
(architecture search), prompt (prompt engineering (Kojima al.) 2022 or specific decoding parameters (e.g. temperature, decoding strategies), or other arper-parameters..
[0. 1]. Without loss of generality, we assume s (x() > (z)) indicates f has better performance than f an example The scoring function can either be computational (exact string match, BLEU PPapineni et al. (002), ROUGE (Lin 2004)). ) LLM-based (BERTScore (Zang et al. 2019), LLM judge (Zheng et al. 2024)), or human annotators. Depending on the task and dataset format, researchers have employed different types of scoring functions. In Table L we classify a selected group of commonly used datasets according to the task type and scoring function applied to them. We group LLM-based and human- based scoring together since they are usually considered alternatives to each other and have been shown to have relatively high correlation eL 2023).
Scoring, The outputs from different methods are scored with scoring function
Performance aggregation. Once the underlying scoring matrix 5 E [0, 1jmxn is computed as Si := (ff(r))). the aggregate performance of method fi is its average score across all validation samples, 14 n Σー= Sty. Alternative aggregations are possible but not considered in this work.
Both inference and scoring often rely heavily LLM-based scoring through black-box APIs (Lmbbert et al. 2024 inducingg large costs and resource consumption. However, in many practical scenarios, are only interested identifying the best method among all methods. For example, for prompt engineering and hyperparameter tuning, knowing which method (prompt / configuration) the best usually sufficient for the next round of iteration. Identifying exactly which method is ranked 84 out of 100 is irrelevant and waste of resources. We therefore claim that evaluating all methods on all examples of a dataset is excessive for this purpose, which motivates the research question of how to identify the best method i" arg max, #4 with limited evaluations.
Formally, given a finite evaluation budget T, we want an algorithm A(T, F, (2) that maximizes (A(A,, X) i*), the probability of returning the best method i* within budget T. We introduce the following additional notation to allow for a clear description of the algorithms. Let Sobs E (Io, 1] U {?}) " denote the partially observed scoring matrix, and O E {0. 10 denote the observation matrix where Oij indicates whether the method-example pair (fi, 22y) has been observed. Finally, let us denote our estimate of S by S. For ease of reference, We also list the notations used throughout the paper in Appendix Table 4
3 ALGORIHTM
One simple baseline for designing the evaluation algorithm A is: uniformly sample and evaluate T examples from X,, for each method f: F, estimate the performance fu; = Σラ= Ozg Σ}= Oig: Syyj and pick the method fi- with the highest estimated mean Íú-- as the prediction for the best method. This baseline ignores any information gathered in the process of sequentially evaluating examples, and, as expected, can be very inefficient. We will see the experiments that for some datasets, this baseline needs to evaluate at least 90% of all method-example pairs to predict the best method correctly with high probability. In the following section, we will introduce two adaptive algorithms that actively select the next method-exampl pair to evaluate based on previous results; Figure 2 illustrates this idea on high level for prompt tuning. The methods in the figure are different prompts, and the examples are simple math problems.
3.1 ALGORITHM 1 UCB-E
The main limitation of the simple baseline is that expectation it samples [T/n]] examples per method; evenly distributing its total budget across all different methods f.. In order to distinguish the best method f. from other strong methods f. (i.c. pa- - μ4 is small), we may need more than [T/n]] examples, while < [T/n] examples are sufficient to distinguish fs- from obviously bad methods (i.e.
(45- - 14 is large).
We address this limitation by adaptively selecting example based our previous t - observations until our budget is expended. Our first algorithm UCB-E (Anne: Algorithm I is a simple extension of the classic multi-arm bandit algorithm UCB-E (Audibert and Bubeck 2010). For every method fi,
3

---

Published as a conference paper at ICLR 2025
Observed Scoringg Matric S
Algorithm
Select
f1 = Let's think step by step.
0
0
A
fa I'll tip $10 you solve.
?
?
Methods F
Update
Evaluate
1
0
0
Explain
?
0
1
s(f((zz)) a(S((va)) =?
#1 sm Notolia sold clips to 48 of. tme Weng earns $12 on hour.
te Goil has two fish tonkks..
LLM Evaluation
?
0
$
Repeat Until Budget T Expended
0
Exomples
N
Figure 2: Active method pair selection: After LLM evaluated method-example pairs, then call Algorithm A to select the next method-example pair. Then we query LLM for evaluating this pair and fill the scoring received from LLM into the scoring matrix. Algorithm A then updates its internal status prepared for the next meehoodeexmmple pair selection. This process is repeated T times and, in the end, the algorithm A predicts the best method fi..
Algorithm UCB-B (T,
Input: The evaluation budget T, a set of methods F, set of examples X, exploration parameter a. Output: The prediction 8 for best method i*.
1: The upper confidence bounds B {+o0) m the observation matrix O {0) mxn the observed scoring matrix 5obs := [?)mxn.
2: for =1=1, 1,... ,T do
3: Select: Draw i E arg max: Ozy) BRi Draw uniformly at random j E {k E
[m]]Oxx= = 0}.
4: Evaluate: Run inference for the methle pair (fi, 2.)) score the result, and receive
s(f((z))) Sije < (x;() Σ Ogg
ST)
5: Update: Oig 1; B,
Σj= O2s
Σy= O23
6: end for
_ Sabo
Return: = arg max;
Oiy
at every step I, we estimate the upper confidence bound,
D Oij
Stnbs
a Eド= Oz,j
B;
Σ Ois
Here, a is hyperparameter that controls the tradcoff between exploration and exploitation, where larger value encourages more exploration. In the subsequent iteration, pick the method fi with the largest upper confidence bound (ties are resolved by uniform random sampling), and then sample an example v, uniformly random from the set of examples yet observed, that is the set {j E Onj = 0j. We run the inference procedure for fi(z;) and score the result, obtaining 5 = s(f((r)))
Comparison with the simple baseline. The UCB-E algorithm addresses the limitations of the simple baseline by minimizing evaluations on suboptimal methods. For a bad method (i.e.
is large), only few evaluations (typically proportional to [a:=-p.)) p..7 ) are needed to ensure that its upper confidence bound will never exceed f4.- Consequently, the UCB-E algorithm will not select this method again, thus saving the evaluation budget and focusing on more promising methods.
Theoretical results. We state the theoretical guarantee of UCB-E our context, which is corollary of the theorem for UCB-E in the original multi-arm bandit setting (Audibert and Bubeck 2010).
4

---

Published as a conference paper at ICLR 2025
Corollary (Lower bound on success probability of UCB-E) Define H1 - Li ((--1)))
and suppose
25 T-m 36 H1
P PA
(A, = 2 27 exp
T-m 1881
Comment. H1 in the corollary indicates the hardness of the problem specified by (F, ,K, Intuitively, the more methods that have a smaller performance gap with the best method, the harder it is to identify which method is the best. We observe that datasets with smaller Hz tend to have higher chance of predicting i" correctly, given the same evaluation budget T'
3.2 ALGORITHM 2 UCB-E WITH LOW-RANK FACTORIZATION (UCB-E-LRF)
Algorithm T the UCB-E algorithm, treats each method independently and samples examples for a method uniformly at random. In practice, however, methods and examples are correlated with each other and can predict the outcome of particular method-example pair from previous observations. Being able to accurately predict the outcome gives a better estimate of the score of a method. Additionally, instead of sampling examples uniformly at random, prioritizing examples with large prediction uncertainties reduces the overall uncertainty in the scores of each method, which provides more accurate score estimates the subsequent steps. In this section, we propose a low-rank factorization based algorithm
to capture the above intuition.
Low-rank factorization. A natural approach modeling these method exammlee interactions is through low-rank matrix factorization. Intuitively, if the method-examples are very correlated, there should exist U E Rmxr and V E Raar and rank r < min(m, n) such that S Ri UVT (Candes and Recht, 2012 Chen et al., 2020] Cai et al., 2019). Obtaining U and V from the observed scoring matrix S allows us to estimate the scores of all method-example pairs from just a few scores. For instance, if the matrix S were an exact rank-1 I matrix, only + n 1 scores would be needed to recover the full scoring matrix S, which is far more efficient than exhaustively evaluating all m x n combinations. To find U and V, we alternating least squares (Hastie al. 2015) to optimize the following problem:
min |0 の
(UVT- - Sotb )
Score estimation. We can then combine the observed matrix gobs and estimated matrix = UVT using as gating function, arrive at a prediction of the score μa as follows:
ii
Σ ==1
Spie + Ozg))
n
Uncertainty estimation. The score estimation gives us better way to estimate the score of every method by exploiting the correlation between methods and examples at every step. We now discuss how we estimate prediction uncertainties with ensembles.
By fitting many factorization models, we obtain an empirical distribution of predictions for each method-example pair. Consequently, the standard deviation of the predictions can be used to quantify uncertainty. If the predictions are very similar to each other, this suggests the ensembles tend to agree with each other, and hence there is little uncertainty. In order achieve sufficient model variance to approximate uncertainty, we uniformly at random drop some values from Sobs when solving the factorization problem in Equation for each model (Efron and Tibshirani 1994 Zoubir and
Boashash 1998). For ensemble of size C' the estimate for S becomes the following:
1 C
š = C
- Σ (V(OT..
(3)
c=1
Then the uncertainty matrix Rmxxn given by:
C
(1- 0) (UOOVV()) -s]
R=
Σ
(4)
5

---

Published as a conference paper at ICLR 2025
Algorithm 2 UCB-U- (A MT, CTo.n)
Input: The evaluation budget T, a set of methods F, a set examples X. the low-rank factorization model M, the rank r, the ensemble size C, the warm-up budget To, the uncertainty scaling 17.
Output: The prediction i" for best method i*.
I: Uniformly draw To method-example pairs from [m] x [n]] and get the observation matrix O
{0, [11)mxx and observed scoring matrix gobs E ((0, [ (?)) m w.r.t. these To evaluations.
2: Š, R - (M O;r: C): Ay, FF, < 1 2)=1 (O Sobs + (1 01y) + RRa))
3: for = To..... ,T do
4: Select: Draw i arg max (Ei=, 0z,) BRi Draw E arg maxę Ow.=0 Rik-
5: Evaluate: Run inference for the eethle pair Ry), score the result, and receive
<(((r)))); 5 <- s (.J (at)).
6: Update: Oig + 1: S, R t M(Subs Crr,C).V. E F, + (0 + +(1
Ozy)S.g + R.y))
7: end for
Return: = arg max; 1 n EE= (Oi) Sobs + (1-
Adaptive selection. Bringing score estimation and uncertainty estimation together, we propose our second algorithm UCB-E-LRE (Auel; Algorithm ( an extension of Algorithm [ leveraging the low-rank factorization model M. The high level structure of the algorithms are very similar to each other. Specifically, we estimate the upper confidence bound (UCB) for each method based on a linear combination of score and uncertainty estimation, select the method with the greatest UCB, and evaluate the method on an example that has the highest uncertainty. Lastly, we introduce a warm-up budget To. This is because M requires minimum number of initial observations in order to estimate the low-rank representation accurately. Therefore, before the active selection phase (line 3-9 in Algorithm 2). we have warm-up phase (line 1-2 in Algorithm (). where To method-example pairs are sample uniformly at random to evaluate before moving on to the "active" phase.
EXPERIMENTS
4.1 DATASETS
Table 2: Statistics of each dataset used in our experiments. The examples X' in each dataset correspond to the questions from their respective evaluation benchmarks. H1 as defined in Section 3.1 reflects the difficulty of identifying the best method, with larger values indicating more challenging settings.
More dataset details can be found in Appendix C
DA stands for Drop Annotator.
Method Set F
Dataset Name
Size m m
Scoring Function
H
Various LLMs
AlpacaEval
154 x 805
GPT4-turbo annotator
966
Various LLMs excluding GPT4-turbo
AlpacaEval (DA)
153 x 805
GPT4-turbo annotator
4462
Mistral-7B with different prompts
GSM8K Prompts
205 784
regex match with correct answer
107445
Various LLMs and sampling configurations
GSM8K Models
122 1000
regex match with correct answer
20562
Tulu-7B with different prompts
PIQA Prompts
177 x 1546
regex match with correct choice
66284
Various LLMs and sampling configurations
PIQA Models
103 x 1000
regex match with correct choice
10273
To assess the performance of our algorithms under variety of cases, we with three datasets
AlpacaEvaal (Li et al. 2023), Grade School Math 8K (GSM8K) (Cobbe et al. 202T) and Physical Interaction: Question Answering (PIQA) (Bisk et al.) 2020), together with different settings of the method set F and the scoring function S; Table 2 summarizes each dataset. For AlpacaEval, we design two sets of F. The first set contains all LLMss reported by Li et al. 2023). The second F contains the same LLMs except for the annotator model GPT-4 Turbo. The latter F that does not contain GPT-4 Turbo makes the learning more challenging and interesting since the annotator, GPT-4 Turbo, is a clear winner among all LLMs. For GSM8K and PIQA, we set JF set of prompts to simulate prompt engineering, or a set of LLMs with various sampling configurations, to mimic model selection and hyperparameter tuning. We also observe that the datasets indeed exhibit low-rank properties as seen from ratios of singular values (Table and explained variance of each principal component (Figure for the data matrices. More information about these datasets is in Appendix C

---

Published as a conference paper at ICLR 2025
USAON PISITRS
TIJA POUTER
ROR Madess
Alpocabeal (Droa Arrocatt)
Apocaceal
B.D.
0.60
0.90-
4.4
C43
25%
2285
2.0-
100 Am
0.50-
100%
1.0%
225
1010
1010
1000
1000
Sipel
400
330%
33%
1000
1.23
L00
1.00
0U90-
ND:0330
0.90
0.00-
0.0
0.95
0.41
000
of
Parcentage
Fotcer sage
Parcsfaged of
Picueriago of
Figure 3: Comparison of algorithms six datasets evaluated with various metrics. The vertical axes represents performance of a metric and horizontal axes represents the percentage of method- -eammoe pairs evaluated. All results aggregated based on 50 trials with different random seeds. The datasets are ordered by decreasing H1 (see Section 3.I.. Larger H₃ values indicate settings where finding the best method is diffieult. Our proposed algorithms: UCB-E and UCB-E-LAR consistently require much less evaluation to achieve the same performance as baselines.
4.2 METRICS
Top Precision. To determine if an algorithm finds the best method, we can directly check the algorithm's prediction matches with the best method from empirical data. However, because every method is empirically evaluated on limited number of examples, is possible that one method has slightly lower average performance than another method whereas if we were evaluate on more examples, the former would achieve a higher performance. To this end, we calculate top precision by first determining set of methods we consider equally good and checking the predicted best method from an algorithm is member of that set. We propose two ways to determine a set of methods are equally good: performance gap and statistical significance. For performance gap top I precision, we consider all methods whose performance is within of the best empirical method to be equally good. For the statistical significance p top precision, we perform McNemar's statistical test (McNemar) 1947) for each method against the best empirical method. If we cannot reject the null hypothesis that the performance of one method the same as the best empirical method up to a significance level then that method is considered equally good as the best method. We evaluate all baselines and our algorithms on two e values (0.001, 0.01}. and two p values (0.01, 0.1} simulating diverse need of precision.
NDCG@ @ K. Although our focus to identify the best method quickly, it is sometimes also desirable to generally rank the top K promising methods high. We therefore evaluate with normalized discounted
7

---

Published as a conference paper at ICLR 2025
Score Only
Batch Size
Ensemble Size
Frompts
Appacatiol (Dop Annelator)
CSMIK Prompox
Alpecatvel [Drop Annutator)
GSHIK Prompix
Absccafvo [Drop Anmokater)
1.0
1.00
Cae 0.03
0.0
6.3
0.6
6.6
0.4
0.23
0.4
100%
20%
130%
10%
100%
0.60
10%
100%
30%
100%
Parcarlage of
Percestage of
Pusceniage of
Percaniage of
Perce-tagg of
Percentiage of
IKS-F
una UM
JES-E-R iiem Dnl
12
124
124
256
Uncertainty Scaling Factor
Warm-Up Budget
Rank
hrones
Appacabol (Drop Annetwior)
CSMMK
Atpecatvel (0rop Annolator)
Promgoss
AlpaciEval (Drop Arnotatert
1.0
6.4-
12%6
100%
20%
130%
10%
100%
100%
10%
100%
10%
100%
Perceriage
Percentage of
Percentsig of
Pelcentage of
Perce-tagg of
Percertage of Evaluradd
14
3.5%
1.3%
5.2%
16.0%
Figure 4: Ablations of the proposed algorithms and hyperparameters. UCB-E and UCB-E-LRF are denoted by blue and red lines, respectively, consistent with Figure
cumulative gain (NDCG) at K. NDCG @ K takes as input the top K prediction by an algorithm and the higher their true ranks are, the higher the NDCG is. We choose K 10 to evaluate all algorithms.
4.3 SET-UP OF OUR ALGORITHMS AND BASELINES
We use a = since this consistently yields the best performance across all datasets. This parameter is ablated in Figure of Appendix D UCB-E-LRF: For all datasets, we rank r = 1 for low rank factorization with an ensemble size of C = 64. We use 5% of data for warm up i.e. To = 0.05 x m n and 7 = 5. Row Mean Imputation: We select method-example pairs uniformly at random from all pairs. The score for each method is calculated as the average of all the scores evaluated far for that method. Filled Subset: Instead of randomly selecting nethoddexamppl pairs, filled subset first selects an example index j uniformly at random. Then all methods are evaluated on example j. If all methods have been evaluated, the algorithm selects a new example index from remaining ones. The score for each method is calculated as the average of all the scores evaluated so far for that method. Although row mean imputation and filled subset do not have learning component, they both produce an unbiased estimate of the real score for each method at all times. LRF: Similar to row mean imputation, LRF (low rank factorization) randomly selects method method-example pairs to evaluate. For estimating the score of each method, LRF calculates the average of all scores for that method. That is, for evaluated examples, the score is the actual observed score, and for examples yet to be evaluated, the score is the estimated score from LRF, The details for three baselines Row Mean Imputation, Filled Subset, and LRF are found in Appendix D A Algorithm 3 and 5 respectively.
UCB-E:
Finally, to take advantaged of the parallelism available in modern computing hardware, all algorithms and baselines are implemented with batch size b. For algorithms that randomly draw an example, they instead randomly draw b examples without replacement. For algorithms that draw an example with the arg max operator, they instead draw with the Top-b operator. We use b = 32 for all experiments unless explicitly stated otherwise.
4.4 MAIN RESULTS
In Figure 3
we plot the performance of baselines and our algorithms on all six datasets (columns) as the budget I' increases from 5% to 100% of the total number of method-example pairs. In each row, we evaluate the algorithms on a different metric (either top precision with different e, p or NDCG@ 10). Each line in the figure is the average result over 50 independent trials with different seeds. For example, an average top precision 0.9 indicates that 45 out of 50 trials predict the best method correctly at the budget level that its x-coordinate represents. In addition, we calculate H/, as defined in Corollary that quantifies the difficulty of finding the best method on a dataset. Intuitively, higher H1 value suggests that the method set F is large there are many methods that have similar performance with the best one and distinguishing them can be challenging. In Appendix Figure 5] also plot the histogram show the distribution of performance among F on these datasets.
8

---

Published as a conference paper at ICLR 2025
How do our algorithms compare with the baselines? As seen from Figure both UCB-E and UCB-E-LRF consistently achieve high precision and NDCG with much less budget compared to the baselines. For example, on AlpacaEval (Drop Annotator), our proposed algorithms can reach a precision of 1 with just 8% budget whereas the baselines require 80-90%, an order of magnitude more budget needed. These results suggest that it is entirely possible to identify the best method without exhaustively evaluating all method-example pairs. They further demonstrate that the active selection algorithms (our two algorithms) are more efficient than the non-active algorithms (three baselines). Additionally, the better NDCG performance from our proposed algorithms shows that our methods can more correctly rank top- pe-pefformance methods.
How is the comparison between our two algorithms UCB-E and UCB-E-LRF? The datasets from left to right are ranked by the hardness indicated by H1. Interestingly, we find that on easier datasets such as AlpacaEval, UCB-E performs better and saves 2-3% more on budget compared to UCB-E- LRF, while on harder datasets, such as GSM8K Prompts and PIQA Prompts, UCB-E-LRF achieves higher precision faster than UCB-E, saving about 10% in absolute budget. These observations give us a hint on what algorithm to apply in practice.
4.5 MORE EMPIRICAL ANALYSIS
We provide more empirical analysis and the ablation results can be found in Figure
Does H1 correctly reflect the hardness of a dataset in the empirical experiments? Yes, going through Figure from left to right, as H, decreases, the percentage of matrix evaluation needed to reach a precision of also decreases from more than 20% to just under 5%. Moreover, the H1 values seem to be related the tasks. Prompt engincering datasets typically have higher H1 possibly due to the homogeneity of prompt performance with the same LLM. In contrast, datasets that benchmark different LLMs such as AlpacaEval make it much easier to find the best performing model.
Score Only Ablation. To study the effect of actively selecting both the next method and next example to evaluate using uncertainty matrix R. we consider an ablation of UCB-E-LRE, In this variant, called UCB-E-LRF (Score Only), we modify the UCB-E algorithm by replacing its mean calculation with the mean calculated via low-rank factorization (as described in Equation B}. The upper confidence bound calculation and next example selection are the same as the original UCB-E algorithm.
The detailed algorithmic description can be found in Algorithm 6 We see that on the hard dataset GSM8K Prompts where UCB-E-LRF has a certain benefit over UCB-E, there is significant gap between UCB-E-LRF and UCB-E-LRF (Score Only). This means that the component of uncertainty matrix R in UCB-E -CB-E-LRF is crucial to contributing the benefit over UBC-E.
Batch Size b Ablation. Intuitively, a smaller batch size is more flcxible and can have more fine grained selection. We experiment with & E {1, 128), and as shown in the plot, the smallest batch size gives the best performance, especially on harder datasets whereas a batch size of 128 significantly degrades the performance. However, it incurs more overall computational cost due to fitting low rank factorization more often than a smaller batch size; for example when b b= 2, it takes almost 16 times as much time as b 32 to achieve 1.0 precision, which ineffective.
Ensemble Size C' Ablation. We vary the ensemble size C' E {4, 16, 256} and find that very small ensemble size gives a suboptimal performance on hard datasets. There is almost no performance difference on easy datasets like AlpacaEval (Drop Annotator). The performance is robust to different C' as long as it is larger than 64.
Uncertainty Scaling Factor Ablation. We experiment with different uncertainty scaling values (0.1, 5, 10). In Figure 4d, it can be seen that a certain range of 17. from 0.1 to 5, gives similar performance on both two datasets. This demonstrates the robustness of selecting 1
Warm-up Budget To Ablation. Our algorithm UCB-E-LRF by default randomly evaluates 5% of the method-example pairs in the data matrix before selecting actively. We analyze the effect of varying the budget To among (0.5, 1, 5, 10)% on the algorithm performance on the two datasets. very small warm-up budget with 0.5% of data can achieve decent precision initially, but fall behind compared to larger warm-up budget as more data are evaluated. In contrast, a very large warm-up budget of 10% delays the active selection algorithm too much and also achieves suboptimal performance. We therefore use 5% as a generally strong starting point. On AlpacaEval, it is possible to achieve the same performance with an even smaller warm-up budget, suggesting more savings is possible.

---

Published as a conference paper at ICLR 2025
Rank Ablation. Hyperparameter r adjusts the bias- trade-off. Empirically experiment with r E (1,2,5) and find that r = 1 is consistently better than larger value counterparts. The results can be explained by the fact that larger requires more evaluated data in order to prevent overfitting, which might not have an advantage in the limited budget setting. Therefore, we recommend using for all datasets.
5 RELATED WORK
Best-arm identification in multi-arm bandits. The goal of best-arm identification (Bubeck et al. 2009 Audibert and Bubeck 2010) is to find the arm with the highest reward by pulling these arms and getting feedback. By making an analogy, in our problem method f; is the arm and the score Mi of Si is the reward. There are two ways to define the best-arm identification problem: fixed budget and fixed confidence. In the fixed budger setting, the budget for the arm pulls is fixed and the algorithm is designed for better chances identify the correct best arm our problem defined in this paper has a similar evaluation budget. UCB-E (Audibert and Bubeck 2010) and Successive Elimination (Audibert and Bubeck] 2010) are two pioneering algorithms proposed for this setting, followed by a line of improvement (Honda and Takemura, 2011 Kaufmann et al.) 2012 Cappé et al. 2013 Kaufmann et al. 2016). Qin (2022) states the optimal fixed budget best-arm identification as an open problem. In the setting of fixed confidence, the algorithms (Kalyanakrishnan et al.) 2011) Kaufmann and Kalyanakrishnan to guarantee the given confidence of getting the best arm. Garivier and Kaufmann (2016) gives an optimal algorithm in terms of the minimum of arms to pull. Another extension beyond the setting of fixed budget confidence is the PAC learning framework, where the target is maximize the chance of getting an mostly best arm, with a tolerance of gap to the highest reward (Karnin et al.) 2013 Jamieson al. 2014 Chaudhuri and Kalyanakrishnan 2019).
2013 Jamieson et al. 2014) work towards fewer number of arms
LLM evaluation with multi-arm bandits. Concurrently, Shi et al. (2024) explore using multi-arm bandit for prompt optimization in limited budget setting. Although there are similarities in the approach, it is fair to say that their work differs substantially in focus. In contrast, our main goal is to speed up model evaluation rather than prompt optimization. Further, we also propose an algorithm that leverages low-rank factorization for better best-arm identification performance. We discuss additional related work on LLLM performance evaluation functions and benchmarks in Appendix A which are relevant but orthogonal to our problem.
6 DISCUSSION AND CONCLUSION
In this work we focus on identifying the top-performer with limited budget as it is a very common problem; covering a wide range of applications particularly for practitioners who simply want to find the best prompt model for deployment purposes. One interesting direction for future work is recovering a full or top K ranking of performers, as it provides more fine-grained information that is necessary for specific use cases such as iterating time. Of course the opportunity for resource savings is less than the top- rop-performer setting. In Figure 3. we evaluate our algorithms NDCG@N @ K as preliminary evidence supporting the possibility of resource saving recovering top K ranking. We leave a more thorough study as future work.
LLM design, and tracking LLM ranking over
A limitation of proposed algorithms is that assume there is two-dimensional fixed-size scoring matrix for active method-exxmple selection. In some real-world applications, new rows or columns can be gradually incorporated, making the matrix size dynamic. In other scenarios such as Chatbot Arena, instead of being able to decide what examples to select, we can only select pair of models to compare for a user-specified example. We leave studying how to efficiently select the best method in these settings as another direction for future work.
In conclusion, we tackle the challenge of essurce-constrainee evaluation, where assessing method- example pairs is costly in terms of money, compute, and time. Our goal is to identify the best method with high probability while staying within budget. We propose two sequential decision-makine algorithms: UCB-E, inspired by the classic multi-arm bandit approach, which provides theoretical guarantee on success probability, and UCB-E-LRE which extends UCB-E by leveraging low-rank factorization to approximate the scoring matrix. Experiments show both outperform random sampling, and we identify conditions where UCB-E or UCB-E-LRE excels.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
Our code is available
ACKNOWLEDGEMENT
We thank Justin Lovelace, Varsha Kishore, and Katie Luo for their helpful discussion and feedback on the paper draft. JPZ is supported by a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) (567916). CKB and CPG are supported by Schmidt Sciences programs, an AI2050 Senior Fellowship and two Eric and Wendy Schmidt AI in Science Postdoctoral Fellow- ships; the National Science Foundation (NSF) including the NSF Research Traineeship program; the National Institute of Food and Agriculture; and the Air Force Office of Scientific Research. RW NNS-1804829), SaTC-2241 100, CCF-2217058, ARO-MURI 11911NF21N (7)1 and ONR under 400014-24-1-2304 WS is supported by NSF IIS-2154711, NSF CAREER 2339395 and DARPA LANCER: LeArning Network CybERagents, This research is also supported by grants from the NSF (1S-2107711, and IIS- HDR- 2118310), the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875), DARPA, arXiv, LinkedIn, and the New York Presbyterian Hospital.
is supported by grants from NSF (CIF-2402817,
REFERENCES
Winogrande: An adversarial winograd schema challenge scale. 2019.
J.-Y. Audibert and S. Bubeck. Best arm identification in multi-armed bandits. In COLT-23th Conference on learning theory-2010, pages 13-p. 2010.
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
O. r. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, M. Huck, A. Jimeno Yepes, P. Koehn, V. Logacheva, C. Monz, M. Negri, A. Neveol, M. Neves, M. Popel, M. Post, R. Rubino, C. Scarton, L. Specia, M. Turchi, K. Verspoor, and M. Zampieri. Findings the 2016 conference machine translation. In Proceedings of the First Conference
Machine Translation, pages 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL
http:/ www. ac web. antho ogy /00//////0 /w/Wl//// 6/W16 6 /W001331
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901 2020.
S. Bubeck, R. Munos, and G. Stoltz. Pure exploration in multi-armed bandits problems. In Algorithmic Learning Theory: 20th International Conference, ALT 2009, Porto, Portugal, October 3-5, 2009. Proceedings 20, pages 23-37. Springer, 2009.
C. Cai. G. Li, H. V. Poor, and Y. Chen. Nonconvex low-rank tensor completion from noisy data. Advances in neural information processing systems, 32, 2019.
E. Candes and B. Recht. Exact matrix completion via convex optimization. Communications the ACM, 55(6): 1116):11-1113 119,, 2012.
E. J. Candès and T. Tao. The power convex relaxation: Near-optimal matrix completion. IEEE transactions on information theory, 56(5):2053-2080 2010.
O. Cappé, A. Garivier, O.-A. Maillard, R. Munos, and G. Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, pages 1516-1541, 2013.
Y. Chang, X. Wang, J. Wang, Y. Wu, L.. Yang, K. Zhu, H. Chen, X. Yi, C. Wang. Y. Wang, al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
A. R. Chaudhuri and S. Kalyanakrishnan. Pac identification of many good arms in stochastic multi-armed bandits. In International Conference on Machine Learning, pages 991-1000. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Y. Chen, Y. Chi, J. Fan, C. Ma, and Y. Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization, 30 11:3098-31211 2020.
Y. Chi, Y. M. Lu, and Y. Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239-5269, 2019.
W.-L. Chiang, L. Zheng, Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint XX1::2403.04132, 2024.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 183.:1803300557, 2018.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, Tworek, Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1101.440 14168, 2021.
P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner. A dataset of information-seekinng questions and answers anchored in research papers. 2021.
M. Dell, J. Carlson, T. Bryan, E. Silcock, A. Arora, Z. Shen, L. D' D'Amioo D'Amico-Wong, Q. Le, P. Queru- bin, and L. Heldring. American stories: A large-scale structured text dataset of historical newspapers, 2023.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018
J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, K.-W. Chang, and R. Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability and Transparency, FAceT '21, page 872-872, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 978 450383097, doi: 0445/3342188.34450005
UtLpttpss:// //doi ottp/o..rrg/1...0 http:/.0..111..... 0145/3442188 R1T.11888444499 3445924
B. Efron and R. Tibshirani. An introduction the bootstrap. Chapman and Hall/CRC, 1994.
A. Garivier and E. Kaufmann. Optimal best arm identification with fixed confidence. Conference Learning Theory, pages
PMLR, 2016.
M. Grusky, M. Naaman, and Artzi. Newsroom: dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXi v: /188004//22. 2018.
T. Hastie, R. Mazumder, J. D. Lee, and R. Zadeh. Matrix completion and low-rank svd via fast. alternating least squares. The Journal of Machine Learning Research, 16(1):3367-3402, 2015.
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021a.
D. Hendryeks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and Steinhardt. Measuring mathematical problem solving with the math dataset, arXiv preprint arXiv:2103.03874 2021c.
W. Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409-426, 1994.
12

---

Published as a conference paper at ICLR 2025
J. Honda and A. Takemura. An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning, 85:361-391, 2011.
K. Jamieson, M. Malloy, R. Nowak, and S. Bubeck. lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Conference Learning Theory, pages 423-439. PMLR, 2014.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, L.. R. Lavaud, M.-A. Lachaux, Stock, T. L. Scao, T. Lavril,
T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023.
Q Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.061444 2019.
M. Joshi, E. Choi, D. Weld, and L. Zeutlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. Satv:1705.005511, 2017.
Kalyanakrishnan, Tewari, P. Auer, and Stone. Pac subset selection in stochastic multi-armed bandits. In ICML, volume 12, pages 655-662, 2012.
Z. Karnin, T. Koren, and O. Somekh. Almost optimal exploration in multi-armed bandits. In International conference on machine learning, pages 1238-1246, PMLR, 2013.
E. Kaufmann and S. Kalyanakrishnan. Information complexity in bandit subset selection. In Conference on Learning Theory, pages 228-251. PMLR, 2013.
E. Kaufmann, O. Cappé, and A. Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics, pages 592-600. PMLR, 2012.
E. Kaufmann, O. Cappé, and A. Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal Machine Learning Research, 17(1):1-42, 2016.
T. Kojima. S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models zero-shot reasoners. Advances in neural information processing systems, 35:22199-223 22292222 2022.
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
N. Lambert, V. Pyatkin, Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
A. Lewkowyez, A. Andreassen, D. Dohan, Dyer, H. Michalewski, Ramasesh, Slone, C. Anil, I. Schlag, T. Gutman-Solo et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843 3857, 2022.
X. Li, T. Zhang, Y. Dubois, Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of insllwction-oollowinn models. https: /ttp:bub///ttu.... com/
tatsu lab/ saab/aapaca_evve eval 2023.
W. Liang, Y. Zhang, Z. Wu, Lepp, W. Ji, Zhao, Cao, Liu, S. He, Huang, et al. Mapping the increasing use of 1lms in scientific papers. arXiv preprint arXiv:2404.01268, 2024.
C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157, 1947.
R. Nallapati, B. Zhou, C. Gulcchre, B. Xiang, et al. Abstractive text summarization using sequence- to-sequence mns and beyond. arXiv preprint (00iv:6602.6600 2016.
S. Narayan, S. B. Cohen, and M. Lapata. Don't give me the details, just the summary! topic-awar convolutional neural networks for extreme summarization. arXiv preprint arXiv: /008 8800::8458. 2018.
13

---

Published as a conference paper at ICLR 2025
K. Papineni, S. Roukos, T. Ward, and W.-J Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
C. Qin. Open problem: Optimal best arm identification with fixed-budget. In Conference Learning Theory, pages 5650-5654, PMLR, 2022.
Rajpurkar, J. Zhang, K. Lopyrev, and Liang. SQuAD: 100,000+ questions for machine com- prehension of text. In J. Su, K. Duh, and X. Carreras, editors, Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, 1264. URL
Nov. 2016. Association for Computational Linguistics. doi:
D. Rein, Hou, Stickland, Petty, R. Y. Pang. Dirani, J. Michael, and R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
V. Sahayak, V. Shete, and A. Pathan. Sentiment analysis on twitter data. International Journal of Innovative Research in Advanced Engineering (IJIREE), 201):18--83, 2015.
T. Sellam, D. Das, and P. Parikh. Bleurt: Learning robust metries for text generation. arXiv preprint arXiv:2004. 04696, 2020.
C. Shi, K. Yang, Yang, and C. Shen. Best arm identification for prompt learning under a limited budget. arXiv preprint arXiv:2402.09723. 2024.
S. Singh, F. Vargus, D. Dsouza, B. F. Karlsson, A. Mahendiran, W.-Y. Ko, H. Shandilya, J. Patel,
D. Mataciunas, L. OMahony, M. Zhang. R. Hettiarachchi, J. Wilson, M. Machado, L. S. Moura, D. Krzeminski, H. Fadaei, I. Ergün, I. Okoh, A. Alaagib, O. Mudannayake, Z. Alyafeai, V. M. Chien, S. Ruder, S. Guthikonda, F. A. Alghamdi, S. Gehrmann, N. Muennighoff, M. Bartolo, Kreutzer, A. Üstün, M. Fadace, and Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUB: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, and H. Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.
C. Yang. X. Wang, Y. Lu, H. Liu, Q. V. D. Zhou, and X. Chen. Large language models as optimizers. arXiv preprint arXiv:2309.08. 0300993399 2023.
Y. Yang. W.t. Yih, and C. Meek. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference empirical methods in natural language processing, pages 2013-2018, 2015.
Z. Yang, P. Qi. S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
W. Yuan, G. Neubig, and P. Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.0783000 2019.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 2019.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang. Z. Wu, Y. Zhuang, Z. Lin, 2. Li, Li, R. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
A. M. Zoubir and B. Boashash. The bootstrap and its application in signal processing. IEEE signal processing magazine, 15(1):56-76, 1998.
14

---

Published as a conference paper at ICLR 2025
A FURTHER DISCUSSION ON RELATED WORK
Low-rank factorization for (noisy) matrix completion. As shown in the objective function of Equation T low-rank factorization is non-convex optimization problem. Therefore line of work focus on how to solve this non-convex optimization (Candès and Tao 2010: Candes and Recht 2012: Chi al. 2019), while another line of work (Chen al.) 2020 Cai al. 2019) study the approximation error between the estimated low-rank matrix and the target matrix in terms of p when assuming the observations are i..dd sampled with a pre-assumed chance p and the additive noise to each observation is i.i.d. Gaussian noise.
LLM performance evaluation functions. Tremendous effort has been devoted developing effective evaluation functions to assess the quality of open-ended generations from language models. Early work in this direction such as BLEU (Papineni et al.] (2002) and ROUGE (Lin) 2004) are rule-based that use lexical overlaps to quantify similarity between generated response and reference. However, lexical overlaps may not align well with the underlying semantics of the text. The shortcomings of these rule-based evaluation functions motivated a line of work studying using language models (Zhang et al. 2019 Sellam et al. 2020 Yuan et al., (021) I) to evaluate generations. seminal work in this area is BERTSscore which uses embeddings from BERT model (Devlin et al.) 2018) to compute similarity. More recently, L.L.M-s--- Judee-- (Zheng et al. instruction-tunel large language models evaluate generations. "Zheng al. proprietary models such as GPT4 have high agreement rate with human.
2024) proposes using
2024 finds that costly
LLM performance evaluation benchmarks. Diverse benchmarks have been developed to evaluate LLM performance across various domains including natural language understanding on translations and sentiment analysis (Bojar al., 2016 Sahayak al., 2015 Singh al. 20224, mathematical and common sense reasoning (Cobbe et al. | Hendryckks et al. |002c| Bisk et al., (020), text retrieval and question answering (Yang et al.] 2015 Jin et al. 2019 Dell et al.) 2023). We refer readers to Chang et al. (2024) for a more comprehensive discussion on existing benchmarks. The commonality of these benchmarks that a ground truth answer is typically given and once the responses from LLMs are generated, they can be evaluated fairly efficiently without incurring large amount of overhead such as money and compute. Recently, a new type of benchmark or more precisely leaderboard has been created attempting to compare various LLMs to each other. The typical setup involves providing the same prompt to two LLMs and then having their responses compared side by side with another L.L,,, which determines which response is considered superior and thus wins. Two notable leaderboards are AlpacaEval (Li et al. 2023) and Chatbot Arena (Chiang al. 2024). AlpacaEval has static prompt dataset and for each prompt and LLM, the response is compared with that of a baseline model (GPT3 or GPT4). For Chatbot Arena, the prompts are submitted by users and two random LLMs are selected respond to the user wrrtten prompt. For both benchmarks, to aggregate the overall performance of LLMs, average win rate or ELO score can be calculated from the pairwise comparison statisties. Since the evaluation requires using another LLM to decide the better response, the evaluation incurs additional compute and / or monetary cost than the benchmarks that evaluate models based on ground truth answers.
B DERIVATION OF COROLLARY
By mostly following the proof of the original theorem of UCB-E (Section 6.2 in Audibert and Bubeck (2010). we only need to show that
Oi
Sobs S
Σi=1
Sig
a
2a
P
2 1 exp
1= Oi
n
5
Σラ Oig
25
This is equivalent to prove if there are p random variables {Xi,... .xp sampling without replace- ment from the finite set {S11, S.n).
Xi +
X,
ΣT= Sig
2a
IP
exp
P
n
25
This can be implied by the analogous Hoeffding inequality (Hoeftding) 1994) that works for the sum of random variables sampling without replacement.
15

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL DETAILS DATASETS
GSMBK Prampts [H1=107445]
SC0R
9.20
PIDA Prompts [H] =66844
GSMBK Models [H] 200522
PIQA Models [H] [10733
AlpacaEval (Drop Annolator) [H1=4462]
AlpacaEval [H1=966]
Method Rank
Figure 5: Bar plot of model performance all examples in all six datasets. Examples are ordered by their rank. Dataset H₁ values are displayed next to dataset name. It can be seen that datasets, such as AlpacaEval, that have large gap between the best and second best method have small H₃ values. The H1 value indicates the difficulty of identifying the best method with smaller values indicating easier identification.
AlpacaEval and AlpacaEval (Drop Annotator). AlpacaEval benchmarks various models fixed set of 805 questions. First, for each model, the responses to these questions are collected. Then, each response compared against that of baseline model (GPT4-turbo) by using an LLM judge (in this case, GPT4-turbo well). We collected the AlpacaEval dataset on May 20, 2024 from the official repository I
At that time, there were total 154 models benchmarked, and hence our dataset size 154 x 805. To remove the bias of favoring its own responses, also create derived dataset called AlpacaEval (Drop Annotator) where we drop the responses the annotator model, which leads to a size of 153 805.
GSM8K Prompts and PIQA Prompts. To simulate prompt engineering use case, we create two datasets: GSM8K Prompts and PIQA Prompts. For these datasets, ask GPT4 to generate 205 and 177 prompts following the prompt engineering work from Yang et al. (003). The LLM used to perform inference on the datasets Mistra-7B Diang al. (2023) and 7u Wang et al. (0233 respectively. We evaluate the prompts on 784 and 1546 questions on the training set (about 10%
of the size of the two training sets). For scoring, we extract the final answers from the responses generated by the LLMs and compare them with the ground truth answers (for GSM8K) or ground truth choice (for PIQA).
GSM8K Models and PIQA Models. Lastly, for the model selection and hyperparameter tuning use case, we similarly create two more datasets on GSM8K and PIQA. Instead of each method being prompt, each method is now combination of LLM with different sampling configurations. Specifically, we use public available models: GPT2 P GPT2-Largel CodeL Tulu-7B
hutps///ttha..oo//sssllaaaaagg com/tatsu-llaaallapaaa eval
hntps://uuginngface
http:://uuuuuiiiiiecccoppriinnnn colopenii-commmniiyggg.
'httss//hugginnfacc.. http:///wugginngaaaacooo/amammma cwvppemi-commmiityyyz.aaaggg
16

---

Published as a conference paper at ICLR 2025
5
Tulu-2-7B Gemma-7B Phi2 Llema- 7B LLaMA-2-7B Mistral-7B and StarCoder- 7B We also have three temperature choices {0. 0.5, 1}, two maximum decoding length choices {128, 512} } and two zero-shot prompt choices (directly asking for the answer and Let's 's think step by step). The Cartesian product of all these choices give total 132 different combinations as our methods. Depending
the dataset, some of these configurations experienced out-of-memory on a Nvidia 3090 when we collected our data which we drop to simulate real-worid scenarios. For the examples, we randomly select 1000 questions from each dataset.
Table 3: Ratio of ith singular value to the largest singular value of S in each setting. represents the 2t largest singular value of S. The ratio Cri1 can be interpreted as the relative reconstruction error with the best rank-r approximation of S. For many datasets, the small numerical differences
01
between the first row and subsequent rows suggest that using rank-1 approximation for UCB-E-LRE appropriate.
Singular Value Ratio
GSMKK Prompts
PIQA Prompts
GSMBK Models
PIQA Models
Alpucalival (Drup Annotator)
Alpacalival
02/d3
0. 0.1647
0.0889
0.3328
0.1972
0,3500
0.3661
1588
0.0763
0,2667
0.1758
0.2153
0.2120
74.97 (
0.1538
0,0739
0.2611
0.1715
0.1746
0.1754
Exclained
Prompts
Exclained
Warlance
PIOA Prompts
Exclained
Warlence
GSMBK
Exclained
PIOA Models
020
Exclained
Warlence
AlpacaEval (Drop Annotatori
AlpacaEval
Explained
25
75
125
175
Principal Component
Figure 6: Bar plot of the explained variance of each principal component for all six datasets. The first principal component is much more pronounced than other principal components, suggesting the dataset matrices generally low-rank.
D ADDITIONAL DETAILS ALGORITHM
The algorithms for Row Mean Imputation, Filled Subset, LRF and UCB-E-LRF (Score Only) shown in Algorithm and respectively.
hhtpss/hugginggface..o hhh.
"hups://uuggingffcc.c//////
http://hhgggggfaacc.ooollllla//a covallenaa/uulu-2-7b
http:////gginnggaaacc.c//gggggge
1 http::/hwgggnnggic..ccc/mmmm/sis
12 http::///wgiinnnhccccccoc/c/nnnn
17

---

Published as a conference paper at ICLR 2025
Exploration Parameter a
GSM8K Prompts
AlpacaEval (Drop Annotator)
1.0
1.00
0.95
0.8
0.90
Performance
Gap 0.01
0.6
0.85
0.80
0.4
0.75
0.70
0.2
0.65
10%
100%
10%
100%
Percentage
Percentage of Matrix Evaluated
Matrix Evaluated
0.1
0.5
10
Figure 7: Ablation of the exploration parameter a of UCB-E, Algorithm
!
Algorithm 3 Row Mean Imputation
FF, 2))
Input: The evaluation budget T, of methods F, of examples X.
Output: The prediction in for best method i*
1: The observation matrix O = {0) {0)mxn, the observed scoring matrix Sobs = m?)mxn
2: for t= 1,....,............ T do
3: Select: Draw uniformly at random (i.j) e {[m] x [n][O = 0)-
4: Evaluate: Run inference for the method-exampll pair (fr, zy), score the result, and receive
s(f((z))); Subs +- (ffi(z)))
5: Update: Oi - 1;
6: end for
Return: 3" arg max;
Σ Ozp.Sth
2, Ozy
Algorithm 4 Filled Subset (A,,(T,F,
Input: The evaluation budget T,
methods F, set of examples X.
Output: The prediction ?* for best method i*.
The observation matrix O = (0) mxn, the observed scoring matrix gobe = (?) mx), initial
random example index j.
2: for = 1,... .T' do
3: Select: Draw uniformly at random i {[m][O;; = 0}.
4: Evaluate: Run inference for the method-example pair 2y), score the result, and receive
s(f((z))); Sybs <- (x)()
5: Update: Oig + 1; If 2m Og m, draw uniformly at random j E [n Oi = 0}.
6: end for
Return: 1- arg max;
Σn_ Sth
Σx-1 Un
18

---

Published as a conference paper at ICLR 2025
Algorithm 5 LRF (Air M, C,ToC
Input: The evaluation budget T, set of methods F, a set examples X, the low-rank factorization model M, the rank r, the ensemble size C, the warm-up budget To-
Output: The prediction i" for best method i*.
1: Uniformly draw To method-example pairs from [m]] [n]] and get the observation matrix O
(0, (0,1)mxr and observed scoring matrix gobs E ([0, 1] (?)) w.r.t. these To evaluations.
2: for f. To. .... T do
3: Select: Draw uniformly at random (i.j) E {mm] x [m]]O.] 00)-
4: Evaluate: Run inference for the method-example pair (z), score the result, and receive
a(fi(zz)); Si s (z;))
5: Update: Oiÿ + 1; S + M(S O; C).
6: end for
Return: 8- = arg max; 1 n
(0 Soba + (1 - 01₂)
Algorithm 6 UCB-E-LRF (Score Only) (o 2:; Mr,, C,M, n)))
Input: The evaluation budget T, set of methods F. a set examples X, the low-rank factorization model M, the rank r, the ensemble size C, the warm-up budget To, the uncertainty scaling r).
Output: The prediction i" for best method i*
method-example Uniformly draw To method-ee example pairs from [m]] [n] and get the observation matrix O
(0, 10 mxx and observed scoring matrix gobs E ((0, 1] (?]) m w.r.t. these To evaluations.
2: S, R M(Sohs O;r, C); Vf. E F, 1 EE= SO Spbs + + - 01y) + nRas)
3: for =rt=-ip To..... ..... T do
4: Select: Draw i arg max;] (E Oz n BR: Draw uniformly random j E
[m][0ux 0
5: Evaluate: Run inference for the method pair (fr, 2z), score the result, and receive
s(f((z))); Sybs &- (x)()
6: Update: Oij + 1; S, R +- M(Sobs Crr,C):V E B + E0 + (1
S,,)),, + R)y))
7: end for
Return: 1- arg max; 1 n E)= (0 Sobs (1 0,1)
19

---

Published as a conference paper at ICLR 2025
E TABLE OF NOTATIONS
The notations used in the paper are described below.
Table 4: Notations used in the paper.
Symbol
Description
F (Ji...,, fm}
list of LLM-based methods
X fIi,..., Lnd
A dataset of examples
A scoring function
Se (10, (?))
A scoring matrix computed as Siş := s(f((z)))
The average score of method fi on all examples 1 Σÿ= Sij
The average score of the best method
T
An evaluation budget with T method-example queries
A(T,F,A
An algorithm that predicts the best method among F on X' with T query budget
gobs
E ((0, 1] (?))
A partially observed scoring matrix
OE E {0, 13 1
An observation matrix where means the corresponding entry in S has been observed
B.
An upper confidence bound for a method ft
A hyperparameter for UCB-E algorithm that balances exploration and exploitation
H1 = -i Lipe (MI-PE
The dataset hardness quantity
E Rmxr, V E gexr
The low-rank factorization of Sobs such that Subs UVT
S E Ruxn
An estimated scoring matrix Å UVV for UCB-E-LRF
is
An estimated average score of method f: for UCB-E-LRF
C
The ensemble size for UCB-E-LRF
R E m2X
The uncertainty matrix for UCB-E-LRE
To
The warmup budget for UCB-E-LRF
The rank of low-rank factorization
M(Sobs, Orr,C)
The low-rank factorization model that outputs and R for UCB-E-LRF
The uncertainty scaling factor for UCB-E-LRF
20


Reviews:
[{"review_id": "VkOVqj6YUa", "reviewer": "Reviewer_1wFE", "summary": "The paper proposes two active selection algorithms for evaluation based on the classical approach of estimation of the upper confidence bound. The main aim of the proposed algorithms is to identify the best performing method across a set of validation examples, given a fixed evaluation budget. The budget can be monetary cost or GPU time. The methods to evaluate can be different prompts or hyperparameter settings.", "questions": "-\tLine 186: by adaptive selecting --> by adaptively selecting\n-\tProof reading is needed to fix typos. \n-\tWhat does a stand for in equation in line 190?\n-\tI suggest some table/mapping to keep track of the different notions used and their meaning.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-\tThe proposed algorithms can be used for a variety of evaluation use cases, not limited to LLMs.\n-\tThe paper provides enough and clear description of relevant concepts on which the proposed solution is built.\n-\tThe paper is very well-written.\n-\tThe proposed approaches show great money and time reduction on large evaluation datasets. \n-\tThe approach was evaluated on variety of tasks, setups, methods, and was evaluated using a thoughtful evaluation approach.", "weaknesses": "-\tNothing major to report here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PyWuoEw7r", "reviewer": "Reviewer_MJvF", "summary": "This paper investigates the evaluation problem of large language models (LLMs) and proposes a UCB-based evaluation method that can identify the optimal LLM strategy for a specific task with a lower budget.", "questions": "In the experimental setup, the authors set the rank r = 1or UCB-E-LRF. Although the ablation study shows that r=1 achieves the best performance, this choice appears overly low compared to related research. The authors should provide more evidence to demonstrate that this is not due to sampling error or an inadequately small dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces UCB-E and its variant UCB-E-LRF. \n\nThe authors conducted extensive experiments across multiple datasets and performed repeated random seeds, which enhance the stability of the results.", "weaknesses": "- Some descriptions in the paper are unclear. For instance, Figure 3, which presents key experimental results, lacks a legend, making it difficult to interpret. \n\n- Additionally, the paper does not clearly define the baseline methods used in the experiments. \n\n- Some results also lack in-depth discussion. For example, Figure 3 shows that UCB-E and UCB-E-LRF perform inconsistently across different datasets. The authors attribute this to varying dataset difficulty; however, when comparing dataset pairs like (GSM8K Models, PIQA Models) and (GSM8K Prompts, PIQA Prompts), the conclusions are contradictory. More detailed explanation and discussion from the authors are needed here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQpd2eAFKw", "reviewer": "Reviewer_LTnJ", "summary": "This paper proposes and extends the well-known multi-armed bandit (UBC) for selection of a best method/setup given a set of method (for example: an LLM, prompt, decoding parameters), a scoring function (for example: exact string matching, BLEU, ROGUE, LLM based annotator) and dataset of examples to evaluate the methods on. This extended multi-arm bandit is referred to as UBC-E. Furthermore, the paper proposes to incorporate a low-rank factorization of the observed scores to enable it to reliably interpolate missing evaluation scores. The low-rank factorization leverages the fact that method-examples are correlated with each other. The whole UBC-E-LRF conserves resources while still guaranteeing confidence that the best method/setup will be chosen.\n\n\nAll this is supported by theoretical proof and discussions, and furthermore shown by empirical experiments on three datasets, and various methods and setups. The UBC-E and UBC-E-LRF are compared with baselines, the top-1 precision and NDCG@10 are used as metrics.", "questions": "In the whole paper, only one baseline is described: uniformly sample and evaluate T examples, but three baselines are mentioned later on, and shown in the figures. What are the two other baselines? Can they be given some attention in the paper?\n\nI have multiple small suggestions to improve the clarity and readability of the paper:\n- In Table 2, the H1 value for each dataset is stated. But there is no explanation of what a higher or lower value means in the caption, or anywhere near where the table is cited. I had to refer to the Corollary 1 where it is mentioned, re-read to figure out what a higher or lower value means, to later find an explanation in section 4.4. \n- In Table 2, the columns are ordered: “Dataset Name”, “Size m x n”, “Method Set”. The size is m x n, m stands for methods, n for data samples. I would either swap the “Dataset Name” and “Method Set” columns, or transform the “Size m x n” column to “Size n x m” to have a natural ordering of the columns and the order of the sizes.\n- In Figure 3, there is no legend for the curve colors. In the caption of Figure 4 it is stated that the UCB-E and UBC-E-LRF are blue and red (at least for Figure 4), but there is no mention of the other curves anywhere.\n- Figure 3 has the datasets ordered from highest H to lowest, and it is mentioned in 4.4 (2 pages forward) that they are ordered by hardness. There is no mention that they are ordered from hardest to easiest, and that higher H means harder and lower H means easier. It can be deducted from the whole text, but it is not immediately obvious.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- To the best of my knowledge, this is the first paper to be using the multi-armed bandit for LLM model/setup evaluation.\n- The idea is solid, and useful. Especially, with the ever growing number of models, size of models and knobs that you can tweak to improve the performance for a specific/custom task. This framework can substantially reduce resources when practitioners have to choose a best model for their use-case.\n- The algorithms are clearly outlined, making the understanding and reproduction easy.\n- A big strength is that the experiments are done on multiple datasets, with varying H1. This paints a clear picture of how this framework works in different setups, and which one (UCB-E or UCB-E-LRF) to choose for which setup.\nThe ablations are extensive: ensemble size, uncertainty scaling factor, warm-up budget, rank, etc.", "weaknesses": "- In section 3.2, low-rank factorization: “Intuitively, if the method-examples are vert correlated, there should exist….” while I do agree with the intuition, it would be nice to have a citation here. At least the citation from the appendix: “Chen et al., 2020; Cai et al., 2019”.\n- Even though the information is in the paper, it requires going back and forth to find it. For example, the figure captions are lacing information that is present elsewhere in the text, or not present at all. Some redundancy in the text for the sake of clarity is always welcome. I added suggestions to improve this in the Question section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5guKKR0NzU", "reviewer": "Reviewer_9sKN", "summary": "This paper proposes an adaptive approach that exploits the fact that few samples can identify superior or inferior settings and many evaluations are correlated. It uses multi-armed bandits to identify the next (method, validation sample)-pair and low-rank matrix factorization to fill in missing evaluations.", "questions": "See the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The studied task is interesting. \n(2) The proposed method seems to be effective in acclerating the evaluation.", "weaknesses": "(1) The paper seems to be recycled and revised from a longer sumbisison, and many parts (especially figures or tables) are with tiny fonts, which are difficult to read.\n\n(2) Some experimental results are difficult to understand, e.g., Table 3.\n\n(3) Overall, I believe evaluation is quite important, and it often involves a number of influencing factors. What if there exists biases in the test datasets? How the comparison results are consistent with human evaluation, since automatic evaluation may not reflect the real capacities of LLMs? \n\n(4) The related work part is weak, which needs more discussions on evaluation of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)