BenTo: Benchmark Reduction with In-Context Transferability
Authors: Hongyu Zhao
Ming Li
Lichao Sun
Tianyi Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: transfer learning
language model
benchmark reduction
Primary area: datasets and benchmarks

Abstract: Evaluating large language models (LLMs) is costly: it requires the generation and examination of LLM outputs on a large-scale benchmark of various tasks. This paper investigates how to efficiently reduce the tasks used to benchmark LLMs without affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information to identify the most representative subset of tasks via optimizing a facility location function. We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or FLAN) to 5\% while inducing only a $<4$\% difference to the evaluation on the original benchmark. Compared to prior works, our method is training-free, gradient-free, and highly efficient requiring ICL only.

Published as a conference paper at ICLR 2025
BENTo: BENCHMARK TASK REDUCTION WITH IN-CONTEXT TRANSFERABILITY
Hongyu Zhao', Ming Li", Lichao Sun", Tianyi Zhou' University of Maryland, College Park
"Lehigh University
[hongyuz, mi minglii, tianyi]eumd. edu Project: https: com/ /tm/tianyiii lab habb/bes o
ABSTRACT
Evaluating large language models (LLMs) is costly: requires the generation and examination LLM outputs on a large-scale benchmark of various tasks. This paper investigates how efficiently reduce the tasks used benchmark LLMs with- out affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information identify the most representative subset of tasks via optimizing facility location function. We propose practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in modern LLM benchmark (e.g., MMLU or FLAN) to 5% while inducing only a < 4% difference to the evaluation the original benchmark. Compared prior works, our method is training- free, gradient-free and highly efficient requiring ICL only.
Compueer Sclence
Anthopplingy
lamr2277b
Gioball
llama3-Bb
Ilama2-13b
Biology troviddgs coukee and
-b-003--7--003
retred cemoenee bokkey accoooooooi liddey
gemma-7o
Pistscaphy
ph22
wot.enttoonaaoco Security
Unyllama
medbol paratcs
Phyyical Sclence
stabielm2-1
ca aagaaa piyakcs
Matherratics aavoroo mechhmatti
GPT4
bandhmerlgg))
BM25(-aim)
random
high school mesoecsesrics
public
Figure 1: LEFT: In-context Transferability (ICT) reveals the clusters of benchmark tasks. We apply spectral clustering to ICT (ares)) between MMLU tasks (nodes), whose color denotes the cluster it belongs to. The discovered clusters associated with explainable themes. The theme and tasks of each cluster are listed around the chord graph. Only the top-7% arcs with the highest ICT values are shown in the graph, among which intra-cluster ares much a "sparse" topology captured by ICT. RIGHT: Evaluation accuracy of task reduction methods. Each method selects 3 out of the 57 tasks in MMLU to evaluate L.MMs (axes). The plot reports and mallmbenchmark respectively. Our method (BENTO-le) achieves 97% evaluation accuracy on average. The grey band reports the random selection baseline's 's meantstandard variation. All baselines are defined in Section
than inter-cluster arcs, implying
1 1- 1-lo-o'l1 'Io"" - in log-scale where a and 0² are the evaluation metrics on the
Table T reports the result when selecting different number of tasks.
Each connects task with target task and has the color the source task.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Evaluation of large language models (LLMM)) is critical to examining the versatile capability and identifying possible weaknesses/risks of LLMs before deploying them to downstream tasks in practice. However, the development of the LLM benchmark is still an open challenge (Chang et al. 2023] McIntosh et al. 2024): it is usually expensive and heavily relies on human involvement, yet it is still unclear how large the benchmark should be to deliver reliable and consistent evaluation results. In practice, to cover various different application scenarios and diverse skill sets of LLMs, current LLM benchmarks usually attempt to include sufficient test cases drawn from as many tasks as possible (Hendryeks et al. 2021a b. Wei al., 202T), e.g., tens to hundreds. Due the expensive sequential decoding of autoregressive LLMs, larger benchmarks greatly increase the evaluation cost and lead to severe overhead in the development process of LLMs.
The substantial costs associated with LLMs drive the need to explore the feasibility of reducing the number of tasks in LLM benchmarks without compromising their evaluative capabilities. Our study in this paper investigates the transferability (Vu et al. 2020; Jiang et al., 2022) between benchmark tasks to discern their relevance and potential overlap. Transferability indicates that skills or knowledge acquired in one task (task-i) can significantly enhance performance in another (task-j). Therefore, model demonstrating strong performance on task-i is likely to perform well on task-j. leveraging the inherent generalization capabilities of LLMs. Given accurate estimation of the task transferability, we may reduce the tasks required in LLM benchmarking and thus optimize the evaluation efficiency.
Existing transferability estimation N gguyen al. 2020: Bao et al. 2019 Tan et al. 2021 mainly rely on model finetuning or Fisher information, which is computational y prohibitive for LLMs considering the total number of benchmark tasks and large model size. Hence, this paper aims to design a cost-efficient and training-free free approach to measure the transferability between different tasks. Motivated by the current progress of in-context learning (ICI.) (Brown al., 2020 Dong ct al. 2023), we propose in-context transferability (ICT) as a training-free approach tailored for benchmark reduction. Specifically, when applying task-i's exemplars as the context for task-j's queries, it provides an effective low-cost estimation of the transferability from task-i to task-j. The resulting improvement over task-j's j'ss ss zero-shot (non context) performance reflects the merits of task-i's knowledge of task-j
Thorough analysis is conducted towards the transferability matrix computed on all the pairs of tasks from MMLU (Hendrycks et al. 20 a widely adopted LLM benchmark. In the visual representation provided by Figure (LFFT), we observe a 'sparse' clustering pattern. This pattern characterized by a concentration of dense interconnections among tasks around the periphery of the circle, with noticeably fewer connections traversing the central area, indicating that the intra-cluster transferability is larger than the inter-cluster transferability. This observation leads to a Laplacian Eigenmaps (LE) (Belkin & Niyogi 2003) embedding of tasks, which is also known as the first step of spectral clustering.
To effectively extract a representative subset of tasks that mirrors the full scope of the original benchmark, we propose Benchmark Task reductiOn (BENTO) that formulates the task selection into a facility location (FL. problem (Cornuejols et af. 1977). In BENTO, the task similarities are derived either directly from the similarity matrix computed via Laplacian Eigenmaps (LE) or are recalculated within the LE-embedded space. The FL objective was to maximize the similarity between each task in the benchmark and the closest task in the reduced subset. This objective is submodular, allowing us to employ greedy algorithm (Nemhauser et al. 1978) that efficiently achieves a high high-quality approximate solution. Extensive experiments are conducted to evaluate the effectiveness of BENTo-reduuc
benchmark by comparing the performance of several widely used I.LMs on both the reduced and original benchmarks. Remarkably, as is shown in Figure (RIGHT) and Table []] the results are highly consistent, even though the reduced benchmark comprises only 5% of the original tasks. This finding underscores the efficiency of our approach. When compared to existing benchmark reduction methods, BENTO not only yields more accurate evaluation results but also significantly lowers the costs associated with transferability estimation. Furthermore, ICT offers substantial potential benefits across wide array of LLM research problems and applications, making it a topic of independent interest.
Our main contributions be summarized as:
2

---

Published as conference paper at ICLR 2025
Table Comparison different benchmark reduction methods on MMLU. The metric is the differ- ence between performance on the full/original benchmark and the reduced benchmark. Previously reported results are available in Appendix C
Model
Full Benchmark(11085)
BENTo(5%)
CHATGPT(5%)
BM25
Llama2-13b
54.5
-0.6
+5,4
-6.8
Llama2-7b
46.0
+3,8
+6.8
-4.6
Llama2-70b
66.6
+1.3
+6.2
-9.0
Llama3-8b
61.7
-1.5
+4.2
-11.6
Mbstral-7b-v0.3
62.1
+0.1
+4.9
-10.6
Phi-2
56.5
+0.2
+5.1
-8.2
Phi-3-mini-4k
69.5
+0.5
+2.1
-12.2
StablcLMM-21166
34,6
+0.1
-1.6
-3.5
Tiaya
24.9
+1.0
+1.4
-0.4
Gemma-7b
65.2
-1.8
+9.0
-10.8
QWEN2.5-7b
74.2
+1.6
+4.2
-14.7
QWEN2.5-14b
79.8
+1.8
+3.5
-14.0
GPT-4o-mini
74.3
+0.8
+5.0
-11.6
GPT-4o
68.9
+0.9
-22
-8.8
In-Context Transferability (ICT): We harness in-context learning to estimate task transferabiliiy and discover graph and clustering structures of benchmark tasks. ICT does not require any finetuning and provides the first scalable transferability metric L.MSs.
Benchmark Task Reduction (BENTO): We develop an efficient benchmark reduction approach, which selects representative subset of tasks according ICT. It can reduce tasks in LLM bench- mark to only 5%, which substantially reduces LLM evaluation costs without hurting the quality.
RELATED WORK
Task Transferability. Efficient estimation of task transfcrability has been long-studied problem. Past works mainly use Bayesian optimization (Weiss al. (016)) and information theory (Bao et al.) 2019: Tan et al., 202T). LEEP (Nguyen et al. 2020) proposes estimating via approximately training the model by linear probing on the data of the source tasks and evaluating the target tasks, which resembles our in-context learning approach. Xia et al. (2024) leverages similarity between gradient features obtained by training with LoRA (Hu et al. 2022) as a transferability measure, which inspires us to transform the performance feature into similarity matrices.
Benchmark Reduction. Dataset reduction for L.MM training [i et al. al. (2024b a) has been a heated area while the reduction for benchmarks is till under-explored. Current benchmark reduction methods can be categorized into two major approaches: selecting tasks from benchmark and selecting examples from single task. Our work falls into the first one. It may seem that the second approach is more robust at least on non non-few-shor benchmarks (Perlitz et al. 2024), but we can prove that combining the approaches always yields better result (see Section 5.2 so these two approaches parallel and we can apply them one by one. Ye et al. (2023) also goes in the first direction and analyzes Big-bench (bench authors 2023). However, it's consuming collect the training data they need, as their methods require performances from different models with various parameters as guidance. Polo al. (2024) goes in the second direction. making use of item response theory (IRT) models, but share the same drawback. On the contrary, our method only requires the performance of single model as guidance, making it very efficient in data collection. Vivek al.) (2024) relatively efficient example-selection method, which, similar their approach can only predict the ranking instead of the exact performance of models on the whole benchmark.
also draws insight from clustering. Yet,
They may slightly different prompts and distinct random seeds, which released.
3

---

Published as a conference paper at ICLR 2025
3 TASK TRANSFERABILITY ANALYSIS BY IN-CONTEXT LEARNING
Aiming at proposing a cost-efficient benchmark reduction approach, we first analyze the structure of benchmarking tasks by studying the transferability between each pair of tasks. Given source task-i and target task-j, the transferability from task-i to task-j is often measured by training a model on task-i and then evaluating on task-j. However, this approach requires training per task and thus is not computationally scalable to modern LLMs and many tasks. We propose to harness in-context learning to provide training- frre estimation of the transferability between tasks.
Compute in-context transferability (ICT) embedding matrix A. To estimate the transferability (8) from source task-i target task-j, we randomly sample L exemplars e1.1. from task-i, where each (0) A ( yk
(8) is an input-output pair for task-i. They are combined with the instruction of task-i pl0) to constitute the context, which is used to query an LLM's answer to each question zz({) of task-j, i.e., LLM( ..... I. The performance of such transfer ICL reflects the transferability from task-i to task-j: If task-j shares more similar format, theme/topics, or can benefit more from the knowledge of task-i, then it's more likely that the transfer ICL can improve the performance on task-j To reduce the variance, we can repeat the transfer ICL. M times with different random seeds and estimate the transferability by their average.
(6)
To study the structure of multi-task benchmark, we estimate matrix of task transferability for all the pairs of tasks by applying the above operation to each pair. Assuming that we have N tasks in total, then we will get an N x N transferability matrix A, where Aig is an estimation of the in-context transferabilityy (ICT) from task-i to task-j, i.e.,
Au
T4J Σ (L (
.I 11)
D. (1)
(I)
where n2 is the total number of input-output pairs we sampled from task-j. s(..-) is an evaluation metric such as an exact match or similarity score. A larger s(-.-) indicates a better transfer-ICL) performance. To further reduce variance, we resample the nj questions multiple times using different random seeds and average the achieved A.j. Since target tasks may differ in difficulty, we normalize A by zero-centering each column of A, i.e.
N
Ag + A.j
Ag
(2)
In the normalized A, each row can be viewed embedding vector for the corresponding task. Spectral clustering. We investigate the graph structure among tasks by applying clustering based the tasks' feature matrix A, which can reveal whether the transferability between intra- cluster tasks is high (and thus they can be further reduced). Since A defines the pairwise transfcrability on a graph of tasks, we choose to use spectral clustering, widely used algorithm for graph cut problems. Given A, spectral clustering computes a similarity matrix that is symmetric and non-negative, by applying a Euclidean similarity kernel to A:
(a) S. induced by ICI. emhedding A.
N Σ (Aa A2x)²
Sy =c Eij. Eij-
(3)
where is constant to ensure the non-negativeness of S. We discuss the choice of in Section 5.2 With a similarity matrix S given, spectral clustering can be performed shown in Figure [ where each cluster is defined by a very clear theme. For example, the red cluster contains all three tasks about history in the MMLU benchmark. The clustering result aligns well with human intu- itions, which demonstrates the effectiveness of ICT capturing the inter-task transfcrability and redundancy between benchmark- ing tasks. Note that there few tasks with counter-intuitive
25-
(b) S" induced by LE embedding A'.
Figure 2: Similarity matrices.
4

---

Published as a conference paper at ICLR 2025
clustering assignments, e.g., "professional law" is in the biology cluster. This may indicate that our ICT representation captures information inherent to the task structures, which cannot be inferred solely from their names. We will come back to this in Section
The ares in the chord graph indicate which tasks are more frequently good source tasks. As we' 've mentioned in Section 1 this clustering has interesting structure: The ares within each cluster more than those between clusters. This might seem trivial at first by definition of spectral clustering. but note that the arcs come from the ICT feature matrix A, which is not the feature matrix that directly perform clustering algorithms on.
LE embedding. Let's take closer look at the process of spectral clustering. We view the similarity matrix S as the adjacency matrix of a complete graph. We first compute
A' = Eigenvector x (L), = [ D- 1D- 2, D = Diag(ST).
(4)
where K is hyperparameter. We then perform the K-Means clustering to the rows of A As expected, Figure 2 shows that the similarity matrix S' induced by A' (Figure (2b) indeed has the same clustering structure as S but exhibits stronger block-diagonal pattern than S induced by A (Figure 2ap. This inspires us to view A' as an alternative task embedding of A: It only preserves and strengthens neighborhood information of A, thus less noisy compared to the original features (Belkin & Niyogi 2003). The process to transform S into A' is called Laplacian Eigenmaps (LE), so we call A' LE embedding in this paper.
The clustering results motivate us to select tasks that are more important or representative: these tasks exhibit higher transferability more tasks than others, and each of them may transfer to different subset of tasks. If we can select a subset of representative tasks that can cover most of the tasks in a benchmark, then we use the performance this subset approximate the performance on the whole benchmark! That is the tenchmark-tass reduction problem we'll discuss in the next section.
4 TENCHMARK-TASS REDUCTION (BENTO) BY FACILITY LOCATION
Benchmark taakk reduction (BENTO) aims to select fixed-size subset of tasks, such that the per- formance of any model on this subset serves as a proxy of the model's performance on the whole benchmark. Intuitively, we want to choose the most "representative" subsets of tasks. In the context of transferability, the representativity
a subset of tasks can be measured by their transferability to other tasks in the benchmark. If subset can transfer to the whole benchmark well, then the performance on this subset will be highly correlated to the performance on the whole benchmark. Under the assumption that the task difficulty in a benchmark on similar levels, the model performance this subset can directly serve as a prediction of the performance on the whole benchmark, and this holds for any model.
How do we choose such a subset? For every target task in the benchmark, if we can always find source task in our subset with sufficiently high transferability the target, then the subset can transfer to all the tasks in the benchmark. On the other hand, if there exist two more tasks with high transferability the same group of tasks, then retaining only one of them suffices keep the transfcrability of the representative subset. Inspired by these intuitions, we aim to find subset such that the similarity between each task the benchmark and its "nearest" task in the subset maximized. This formally reduces to optimizing the facility location (FL) function, i.e.,
N max Sij- JEX
X E arg max SXISK
S(X)
larger f(X corresponds to a more representative subset of tasks X. Since this function is submodular, the optimization problem can be solved explicitly and efficiently with a greedy algorithm.
As an alternate to S in Equation (3) (which might be affected by the long-range noises in A), we derive a cosine similarity matrix S from the LE embedding A':
AA,
(6)
||ill ||4
5

---

Published as a conference paper at ICLR 2025
As shown in Figure 2 S' shares lot of common properties with S, so we propose variant that replaces in Equation 5 with S". The LE embedding removes some long-range noise and is expected to be more robust in realistic scenarios. We call the original version using S "BENTo-sim" TO-sim" and the LE variant using 5" "BENTO-le", where BENTO is an abbreviation of Benchmark Task reductiOn.
A detailed algorithm is presented in Appendix A To summarize, our pipeline is: first compute a transferabilityy matrix via in-context learning, then compute a similarity matrix based on the transferability matrix, and maximize the facility location function defined by the similarity matrix to select a subset of representative tasks.
EXPERIMENT
Benchmarks. We assess our method mainly on two benchmarks: MMLU (Hendrycks ct al. 202 la by and FLAN (Wei et al., 202T). MMLU is a grestion-answering dataset containing 57 tasks with diverse subjects and levels of difficulty, mainly focusing on the knowledge of the language models. All the questions MMLU are multiple-choice questions with 4 options for each answer. On MMLU, we use accuracy (ACC) the evaluation metric s(.,-) -) in Equation ① FLAN is dataset with more diverse forms of tasks, including free-form generation tasks like translation and summarization. Since FLAN is large dataset, sampled 100 questions from each of its 66 tasks our "whole benchmark". : On FLAN, we use response perplexity as the evaluation metric -): We follow widely used prompts for these benchmarks without any re-engineering, more details in Appendix D
Evaluation Metric. In both datasets, we apply all methods to select & representative tasks with from to K (We pick K to be approximately 18% of the tasks, i.e., 10 MMLU and 12 FLAN). For each value of k, we calculate the root mean square error (RMSE) of the predicted performance (i.e. performance on the selected tasks) across all the models. To ensure comparability, the RMSE is normalized by the root mean square (RMS) of the ground truth performance, yielding the following normalized RMSE:
ΣT (o,
NRMSE
Σ of
where T' denotes the total number of evaluated models, and of denote the evaluation metrics and the original mull-benchmark, respectively. The NRMSE
of model on the measures the relative error (error rate) of the reduced benchmark in terms of L2 error. For a more F
fine-grained evaluation of each model, please refer to Table and Appendix
Models. ICL is performed on Llama-2-13B (Touvron et al., 2023) and Llama-2-7B to estimate ICT for MMLU tasks and FLAN tasks, respectively. Since the goal
find a reduced benchmark that can replace the original benchmark to evaluate any models, we compare the benchmark evaluation results on 14 popular LLMs, including Llama-2-70B, Llama-2-70B.LLaaa222133 Llama-2-13B, Llama-2-7B, Llama-3-8B (Dubey et al. 2024), Gemma-7B (Team al. 2024), Phi-2 (Javaheripi al. 2023). ) Phi-3 (Abdin et al.) 2024).
StableL (Bellagente et al. 2024), Mistral-7b-vel (Jiang et al. 2023), TinyLLama Zhang
et al. 2024), QWEN2.5-7B, QWEN 2.5-1BBB (Yang et al. 2024), 4PT-4o-mi and GPT-4o (Hurst
et al.
2024
Baselines. We compare BENTO with the following baselines:
random: A simple baseline involves randomly selecting tasks. To reduce variance, we sample 1000 sets of k random tasks for each k and compute the average NRMSE across these samples.
GPT4 (OpenAl )023): We prompt GPT4 to suggest representative tasks and rank them based solely on the names of the tasks. Given that our evaluation is based on well-established benchmarks, GPT-4 likely has prior knowledge of these tasks and may have encountered them during training.
BM25-sim: BM25 (Robertson et al. 2009) is classic measure of text similarity. Here, calculate the BM25 score between each task's corpses (including instructions, solutions, etc.) and use it replace the ICL transferability matrix A. The remaining steps the same BENTo-sim.
BM25-le: A variant of BM25-sim, which uses S' for the FL problem, just in BENTo-le.

---

Published as a conference paper at ICLR 2025
Table 2: NRMSE on MMLU (lower the better) when selecting & tasks for evaluation. Each number is averaged over different models. The standard deviation can be found in Appendix
Method
Best
k=1
k=2
k=3
k=4
k=5
k=6
k=7
k=8
k=9
k=10
random
0.091
0.228
0.163
0.144
0.130
0.118
0.110
0.104
0.100
0.095
0.091
GPT4
0.034
0.213
0.078
0.093
0.077
0.056
0.034
0.160
0.180
0.168
0.135
BM25-le
0.074
0.074
0.218
0.217
0.216
0.205
0.193
0.199
0.213
0.209
0.176
BM25-sim
0.093
0.351
0.188
0.177
0.119
0.101
0.093
0.162
0.137
0.134
0.117
BENTo-le 0.029
0.051
0.098
0.029
0.049
0.045
0.031
0.060
0.168
0.147
0.113
BENTO-sim
0.026
0.123
0.061
0.177
0.119
0.101
0.039
0.079
0.039
0.029
0.026
5.1
MAIN RESULTS
Results the MMLU benchmark presented in Table 2 As shown, the best of our two methods consistently outperforms other approaches. Notably, Our method can achieve an error rate of 3% with only 3 tasks out of 57, which constitutes approximately 5% of the total number of tasks (5.8% of test samples), significantly surpassing the baseline methods. This demonstrates that the information embedded in the ICL transferability matrix can be effectively utilized for benchmark reduction.
deeper examination of the performance of the random and GPT-4 baselines reveals some intriguing patterns. Expectedly, the NRMSE of the random baseline always decreases k increases. In contrast, while the GPT-4 baseline exhibits a general downward trend in NRMSE as k increases, an anomalous spike occurs law" as the seventh task, justifying its choice with the reasoning that this task is "relevant for understanding societal structures." " This decision seems intuitively sound from human perspective, as professional law often deals with governance and social order. However, our clustering result in Figure |
7. Upon closer inspection, GPT-4 selects the task "professional
suggests otherwise: the task "professional law" is actually placed in the biology cluster, revealing unexpected underlying connection. This discrepancy highlights a key advantage of our approach. The fact that our methods consistently outperform the GPT-4 baseline indicates that our task representations capture more nuanced and accurate relationships between tasks beyond what their names or surface-level associations suggest.
Table 3: NRMSE on FLAN (lower the better). k is the number of selected tasks. Each number is averaged over different models. The standard deviation can be found in Appendix
Method
Best
k=1
k=2
k=3
k=4
k=5
k=
k=7
k=8
k=9
k=10
k=11
k= 12
random
0.49
1.27
1.06
0.92
0.83
0.77
0.70
0.64
0.60
0.56
0.53
0.51
0.49
GPT4
0.09
7.48
3.29
1,86
1.36
0.89
0.58
0.61
0.41
0.26
0.13
0.09
0.44
BM25-le
0.51
0.99
0.76
0.61
0.57
0.58
0.65
0.51
1.61
1.33
1.10
1.36
1.36
BM25-sim
0.24
7.93
4.36
2.57
1.69
1.30
0.93
0.65
0.68
0.49
0.36
0.31
0.24
BENTO-le
0.07
0.55
0.47
0.22
0.07
1.44
1.03
0.86
0.63
0.58
0.50
0.40
0.29
BENTO-sim
0.04
0.93
0.76
0.04
0.10
0.17
0.30
0.26
0.25
0.33
0.35
0.18
0.21
Our main results on FLAN are shown in Table 3 On FLAN, our method achieves an error rate of 4% using approximately 5% of the total number of tasks. Note that here the GPT4 baseline performs relatively better compared to its performance on MMLU, This improvement is primarily because the task names in FLAN are more informative such SST-2. GPT-4 has likely encountered these tasks during its training, enabling infer the content of each task without needing to analyze individual examples. Despite this, our methods still outperform for most values k. Our method's consistent performance across different benchmarks indicates its potential applicability to wide range of tasks and datasets.
they are names of ynol-, established datasets
The results of the similarity- based methods on both the MMLU and FLAN datasets exhibit consistent pattern: BENTO-le performs well when the value of & is sufficiently small, whereas BENTO-sim demonstrates better performance for larger values of k. This pattern also applies to the BM25 baseline; initially, BM25-sim underperforms compared to BM25-le but surpasses it as Is increases. This trend is clearly illustrated Figure 3 where we plot the difference (A) of performance of
"sim" methods and "Ie" methods. The underlying reason for this behavior lies in the properties of
7

---

Published as a conference paper at ICLR 2025
A of normalized RMSE (sim-le)
A of normalized RMSE (sim-le)
0.3
BenTe BM25
Ben BM25
2.0
0.1
1.0
0.5
0.0
0.0
-0.5
-1.0
7 11 12 Number of selected tasks
Number selected tasks k
(a) A MMLU.
(b) A FLAN.
Figure 3: Difference (A) in NRMSE between ("sim") and S' ("Ic") when used to select different numbers of tasks (x-axis). Larger A indicates the "lc" variant produces a better reduced benchmark than "sim". " For both BENTO and BM25, "Ie" is better (A 2 0) for smaller lo while "sim" is better (A 0) for larger k.
Laplacian Eigenmaps. As previously discussed, Laplacian Eigenmaps designed preserve local neighborhood relationships while discarding long-range information. This characteristic makes them highly effective when selecting small number of tasks, where local similarities However, as the number selected tasks increases, the importance of capturing global structure and long-range relationships becomes similarities become more effective for larger values.
paramount.
significant. Consequently, methods that consider long-range
Experiments other benchmarks (BBH (Suzgun al. 2022) and AgilEval (Zhong al. 2023) can be found in Appendix
5.2
ABLATION STUDY
Table 4: Ablation study of similarity metrics: we compare the best NRMSE on different datasets achieved by different metrics: "cos"" cosine similarity, and "cheby"- Chebyshev similarity.
Method
MMLU
AGIEval Eng
Big Bench Hard
cheby-le
0.05
0.07
0.09
cheby-sim
0.11
0.06
0.22
cos-le
0.05
0.03
0.20
cos-sim
0.03
0.09
0.10
BENTO-le
0.03
0.03
0.05
BENTO-sim 0.03
0.03
0.15
Choice of c. When calculating the similarity matrix from the ICL transferability A using Equa- tion (3). we need choose hyperparameter c. Note that by definition does influence the results of BENTO-sim; however, it does impact the performance of BENTo-le. In our main experiments, we set 1.5 max,,, (E ). ensuring that S 0.5 max, (E,,)
This specific choice was empirically validated on the MML.U dataset, where it produced reasonable clustering results. But how optimal this choice? Could other values yield better performance? Would this selection generalize effectively to FLAN?
To address these questions, we parameterized as c = maxi, [Ey), where t is sampled from uniform distribution ('(1, and generated 1000 random values of t. We then evaluated the performance BENTo-le under these different values of c. On MMLU, out the 1000 samples, 191 led to better average performance over compared to original choice, while 340 achieved better performance on the best contrast, on the FLAN dataset, 506 samples improved the average performance, and 872 samples enhanced the best performance compared to initial selection of
8

---

Published as a conference paper at ICLR 2025
These findings suggest that while choice of is reasonably effective MMLU, it is less optimal on FLAN. This variability in performance across datasets indicates that a more adaptive approach to selecting could be beneficial. In future work, could explore dynamic strategies for setting potentially based on specific dataset characteristics or performance metrics. This approach could lead to more consistent improvements across different benchmarks.
Choice of similarity metrics. In our study, we opted for Euclidean similarity when computing the similarity matrix S. An important question arises: how would other similarity metrics, such as cosine similarity or Chebyshev similarity, affect the results? As shown in Table while other metrics like cosine and Chebyshev similarity produced results that were slightly worse than Euclidean similarity, the performance gap was not large, especially between cosine similarity and Euclidean similarity. This suggests that Fuclidean similarity may offer slight advantage on the specific benchmarks we evaluate on, but other similarity measures could still be viable alternatives on different benchmarks.
Task selection and example selection. While one might argue that selecting representative examples within each task could yield better results while keeping inference costs low, it is important to note that our method can be combined with existing example-selection techniques to further enhance the reduction rate. To evaluate this, we compare two approaches: randomly selecting examples from each task with and without incorporating BENTO. The results, reported in Table 5] demonstrates that BENTO can further reduce the NRMSE of the 5.0% example-selection baseline ("Random") by reducing the examples to 2.0% using task selection.
When the number of examples per task becomes extremely limited, continuing reduce examples per task leads to substantial increase in NRMSE, suggesting that task selection offers a more robust alternative in such scenarios. Therefore, task selection and example selection are complementary strategies that can be effectively combined to achieve higher reduction rates.
Table 5: Example selection with and without BENTO (-sim) on MMLU. "Random" refers to random selection of examples. "Random+ BENTO" applies "Random" at first reduce the examples per task to 5% and then selects subset tasks by BENTO shows that BENTO can further improve example selection and outperforms example selection only. For example, "Random+ than "Random" with 5.0% remaining data; Baandom=BeNToo with 0.7% remaining data achieves the same NRMSF as "Random" with 2.0% remaining data.
with 2.0% remaining data achieves a lower NRMSE
Selection strategy
Selected Tasks
Remaining Examples (%)
NRMSE
Random
57
5.0%
0,029
Random
57
0.7%
0.109
Random + BENTO
11
0.7%
0.051
Random
57
2.0%
0.051
Random + BENTO
21
2.0%
0.026
Facility location K-medoids clustering. We select FL for its efficiency and precise formation of the task reduction problem. To evaluate this choice, we compare FL with more computationally inten- sive methods like K-medoid clustering, which can be viewed as K-means where real data points serve as centroids. As illustrated in Figure the choice of ICL embedding plays far more critical role in de- termining performance than the choice of clustering algorithm. When using the same embedding, both FI. and K-medoids produce comparable results; how- ever, FL offers clear advantage in computational efficiency. This makes FL the more practical option for lcrge-scale scenarios without sacrificing perfor- mance.
0.00
2.04
Normatzzed RMSE
8.00
0.022
3.005
0.126
3.0
Bents Ben
KM
km
Figure 4: Ablation study facility location (FL) vs. K K-medoids: we report the best NRMSE (lower is better) achieved by each method on MMLU, KM denotes K-medoids, KM-raw, KM-sim and KM-le denote K-medoids on the raw feature matrix A, similarity matrix S and S' respectively.
Choice of LLM. Our primary experiments employ Llama-2-133 to estimate in-context transferability(CC)) on MMLU, To evaluate whether model scale critically impacts this estimation, conduct parallel analysis using the 2.7B -Ba-paraeeter Phi-2

---

Published as a conference paper at ICLR 2025
model (Javaheripi et al. 2023). Results show remarkably high alignment between the two models: Spearman correlation reaches 0.9304 + 0.0004 and Pearson correlation 0.9282 0.0004. This strong agreement suggests that substantially smaller LLMs can still reliably approximate ICT while offering computational efficiency.
CONCLUSION
In this study, we demonstrated that large language model (LLM) evaluations can be efficiently conducted with significantly reduced benchmarks, without substantially compromising evaluative accuracy. Utilizing in-context learning to estimate task transferability, our method allows for reduction of up 95% in the number of tasks, maintaining less than 4% deviation from full benchmark results. This approach not only reduces computational and operational costs but also presents a scalable model for rapid LLM assessment. Future work may explore expanding this methodology across different model types and broader task sets to enhance its robustness and applicability real-world scenarios.
LIMITATIONS
In this paper, we have focused on achieving cost-efficient benchmark reduction for evaluating large language models (LLMs), which we have demonstrated to be effective through extensive experimentation. However, a notable limitation of this approach is that smaller benchmark may inherently be less diverse and potentially more vulnerable to adversarial attacks. We recognize that this limitation represents a fundamental trade-off between the efficiency of the evaluation process and the comprehensiveness of the metries employed.
ACKNOWLEDGEMENT
The authors gratefully acknowledge the generous gift fund provided by Adobe, which supported this research endeavor. The authors would like to express their sincere appreciation to the reviewers and area chairs for their valuable feedback and insightful comments that significantly contributed the improvement of this paper.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint, 2024. URL
https://arxe http://axxxxv...aaaa//////
www.oo////aooo / 2404. 14219
Yajie Bao, Yang Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas.
An information-thhorreiccc approach to transferability in task transfer learning. In ICIP, 2019, URL
https::/ieeexploreee www.:e/uunt///ottttttt.... 8803726
org/ document/
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data
representation. Neural computation, 2003. URL httpss:/ieeexploree ieee..ereeee....... org/
document/6689755
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, al. Stable Im 2 1.6 technical report. arXiv preprint, 2024. URL
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. TMLR, 2023. URI. htttp:://penreviiet www.ww.wweeeeeeeeeeeee.eeeeeee net/fooumm?
LyTL5Bvos]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosse,t-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel
10

---

Published as a conference paper at ICLR 2025
Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, 2020. URL https::/ ings reuriss CC/ wwwpeppppppppppppppppppeeepe.eee files/paper Papeee pdf
2020/file 0120/E1le1445c0dee 41114444000000000000000021121110
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023. URL
org/pdf/22 2307 03109
Gerard Cornuejols, Marshall Fisher, and George L. Nemhauser. On the uncapacitated location problem. In Studies in Integer Programming, volume of Annals of Discrete Mathematics, pp- 163-177. Elsevier, 1977.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023. URL httpss://axxv..or
pdf/2301. 00234
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178.. 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. ICLR, 2021a. URL httttttpa///axiia//
abs/2008 02275
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2021b. URLhttps
03300
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- system card. arXiv preprint arXiv:2410.21276, 2024.
Mojan Javaheripi, Sébastien Bubeck, et al. Phi-2: The surprising power of small language mod- els, 2023. URL http:://www: microsoft. small-languagg-mmddel. 2023.
wwwpe//ooooo.o.ooooooooooooeoeee us/ research/ hhtp:p/a........ee.........aaaaa
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. htth:tpp: //arxiv/ brg/pdff/3110.c 06825
Junguang Jiang, Yang Shu. Jianmin Wang, and Mingsheng Long. Transferability in deep learningg survey. arXiv preprint, 2022. URL
0/56//0p00//20000567 05867
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander M. Rush. Opennmt: Neural machine translation toolkit, 2018. (tRlhttss:// http://acccaaaoool.o..ooooooo org/ P P17-4012/
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning, In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers), pp. 14255-14273,
Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https:
7/aclanthie ogy. ogy. 200./////222..... acl - 769
11

---

Published as a conference paper at ICLR 2025
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.),
Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 7595- 7628, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. URL
https: httpps//aaaaaaahoooo...o.ooo org/ wtw.:///2200.. 2024. -aa.l satacl-long, 421
Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. Inadequacies of large language model benchmarks in the of generative artificial intelligence, 2024. URL
https://arxiv> wttp:///.wrgg/a///22....o 09880
G. L.. Nemhauser, L. A. Wolsey, and M. L.. Fisher. An analysis of approximations for maximizing submodular set functions-i. Math. Program., 14(1):265-299 1978.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In ICML, 2020. URL httph:tt/sssat //axivv
org/abs/ 2002. 12462
OpenAl. Gpt-4 technical report, 2023.
08774
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheueee and Leshem Choshen. Efficient benchmarking (of language models). NAACL, 2024.
11696
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuckai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint, 2024. URL https
220..//0rrr/////22242229023 14992
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ Information Retrieval, 2009. URL https:/ www
nowpublishers com/ aatii.e//etalll///ee....
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le. Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether oh of-huuht- can solve them. arXiv preprint, 2022. URL http::: //axxiv org/ abs/ 2210. 09261
Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021. holthttss///rae wwwpa/wwwww.r//////.......
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint, 2024. URL /ttp:v///axxv..ooo
abs/2403.0833 08295
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023, URL
orq/ alsgg//b//// 2237
09288
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela, Anchor points: Benchmarking models with much fewer examples. EACL, 2024. URL htttt://aaxxiiii /ttttp:://arxii.oor//abbasa
hbtg//b///// ataaaab///33000 2309.0863
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nip tasks. arXiv preprini, 2020. URL https: http://wwwwvvoroo///////
00770
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2021.
URL https:://// / org/ abs /2099 01 652
12

---

Published as a conference paper at ICLR 2025
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Jour- nal of Big data, 2016. URL battps://link.ett springer http://rccccoattieee/110.111
54C537-016-0043-8
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint. 2024. URL https:
wazxiv.org/ab//2422.... 04333
An Yang, Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rX11:2412.15115, 2024.
Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? case study on big-bench, EMNLP Findings, 2023.
abs/2305 14947
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint, 2024. URL hotttttss////xxii
/ //wpvvaavvvvv/vv//////eeeeeaaavv wwtp.///.gg/////1......// 02385
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint wwxXix23304006666 2023, URL https:://////. httlhtttps:/aaaeeee //atpps://axii...o http:/awwwii.....aa.aaoa..aooeat abs/ 00bb//23440630
13

---

Published as a conference paper at ICLR 2025
A
ALGORITHM
Our method BENTO is described in Algorithm 1
Algorithm 1 Benchmark Task Reduction (BENTO)
1: procedure
Model, L, M) >> L and M are hyperparameters = -eengh((Taskk,, p(+) instruction of Task[i]
2:
3:
for j.j = N do
Estimate ICT from Task[i] Task[j] > M random seeds
4:
for m = M do
5:
Set random seed m Sample L exemplars 11.7
(4) from source Task[i]
6:
7:
Sample ny input-output pairs <im; from target Task[j]
8: Estimate ICT As.j using Equation
9: end for
10: Average A,, over the M random seeds
11: end for
12: Normalize the columms of using Equation
13:
return A
14:
end procedure
procedure SIMILARITYMATRIXAAA K)
Transferability matrix A, hyperparameter K
16:
Compute the similarity matrix using Equation (
Compute the Laplacian embedding A' using Equation
18:
Compute the cosine similarity matrix S' from A' using Equation
return S, S
20:
end procedure procedure SENCHMARRKAASSRRIIIITTOOOG
can be replaced by S"
22:
Maximize Equation (5) by the greedy algorithm
23:
Return X*
24: end procedure
B EXPERIMENT DETAILS
For our main experiments, we use 4 A100 40G for about days. We use L = exemplars and M 10 random seeds. We set n, to be large value so that we always evaluate on the whole test set. For our main experiments on FLAN, when we normalize E, we also divide it by the standard deviation since the metric we use makes the original one too distorted.
Our implementation is based on Klein et al. (0018..
C STANDARD DEVIATIONS OF MAIN RESULTS
The ICL. accuracy on MMLU are averaged over 10 random seeds. Since the result is x matrix, it's impossible to list the standard deviation for all entries. The average standard deviation over the 57 x 57 matrix is 0.017, the standard deviation of the standard deviation 0.0065.
For our main results presented in Table 2 and Table 3 compute the error bar via bootstrapping. randomly sample M models with replacement and compute NRMSE on the sampled models. This process is repeated 1000 times and compute the mean and standard deviation of the NRMSE. Results are shown in Table and Table
D ICL PROMPTS EXAMPLE
On MMLU, the following prompts:
14

---

Published as a conference paper at ICLR 2025
Table 6: Error bar of the NRMSE on MMLU. Computed by bootstrapping.
k Method
GPT4
BM25-le
BM25-sim
BENTo-le
BENTo-sim
0.193+0.017
0.06240.015
0.349
0.059+0.010
0.130=0.013
2
0.07340.006
0.224+0.025
0.177+0.020
0.090#0.014
0.066=0.010
3
0.100+0.014
0.220+0.025
0.14740.017
0.031+0.000
0.168=0.017
4
0.082+0.012
0.217=0.023
0.122#0.016
0.050#0.007
0.113+0.012
5
0.060#0.010
0.206#0.022
0.125#0.015
0.045#0.006
0.097=0.010
6
0.041=0.005
0.193#0.021
0.08940.009
0.031#0.006
0.042+0.006
7
0.152+0.015
0.198+0.021
0.15740.014
0.058+0.008
0.077=0.007
8
0.168+0.015
0.209=0.020
0.132+0.011
0.159#0.015
0.043+0.007
9
0.157+0.014
0.205+0.019
0.129#0.011
0.142+0.013
0.033+0.004
10
0.126+0.013
0.17440.016
0.113+0.010
0.111#0.010
0.029#0.005
Table Error bar of the NRMSE on FLAN. Computed by bootstrapping
k Method
GPT4
BM25-le
BM25-sim
BENTo-le
BEXTo-sim
6.867+3.020
0.90940.400
7.276+3.199
0.502+0.220
0.851=0.374
2
3.021
0.69410.305
3.99911.758
0.427#0.188
0.702=0.308
3
1.71240.752
0.55940.246
2.360+1.036
0.202+0.08
0.033+0.015
4
1.247+0.548
0.526+0.232
1.55240.682
0.067+00030
0.091
5
0.814+0.358
0.53140.234
1.193+0.524
1.321+0.581
0.151=0.067
6
0.532++.223
0.594+0.260
0.850#0.374
0.948+0.417
0.272+0.122
7
0.559+0.246
0.473+0.208
0.598+0.263
0.79140.348
0.243+0.107
8
0.377 166
1.479#0.650
0.623+0.274
0.582+0.255
0.231
9
0.238 105
1.22340.538
0.454+0.200
0.53740.236
0.307#0.135
10
0.123+0.054
1.009:0.444
0.329 144
0.455+0.200
0.323+0.144
11
0.087+0.038
1.24840.548
0.282+0. 124
0.363+0. 160
0.161
12
0.40
1.24940.549
0.21740.096
0.26540.117
0.191#0.084
The following are multiple choice questions (with answers)
about [Task A's subject], (pln sub A' exemplars] [Task B's question] |nAnswrr:
An exemplar has the format: [Question] [enestioon((Ansseerer |nAnswrr: [Answer][p(p
On FLAN, we use the following prompts:
You are helpful AI assistant. Here are some example input-output pairs that you should follow. \n)n [Task A's exemplars] Input: [Task B's question] question][noutpuut. |nOutpu::
exemplar has the format: Input: \n [Question] |noutput: [Answer][ppp (pln
E DETAILED PERFORMANCE OF EACH MODEL
The detailed performance of each model is shown in Table 8 consistently well across different models.
Our reduced benchmark works
F RESULTS ON ADDITIONAL BENCHMARKS
AGIEval [Zhong et al., 2023) is a question-answering dataset where the questions mostly come from real human exams. The questions have diverse sources and forms, but are all formatted as multiple-choice questions. We use the English tasks in this dataset and use ACC as the initial transferability measure. The result is shown in Table 9 This benchmark is not ideal for setting since the number of tasks is too small, but our method still works relatively well comparing to the baselines. Big-Bench Hard (Suzgun al. 2022) is benchmark with 27 subtasks, including
15

---

Published as a conference paper at ICLR 2025
Table 8: Performance of different models on all tasks and selected tasks of MMLU.
Model
Selected Tasks All Tasks
Gemma-7b
65.240.2
63.420.5
1lama-2-130)
54.5+0.2
53.940.2
Llama-2-7b
46.0+0.1
49.8+0.3
Llama-3-8b
61.7+0.2
60.2
Mistral-7b-v0.3
62.1+0.2
62.2+0.4
Phi-2
56.5+0.3
56.7=0.3
Phi-3-mini-4K
69.5+0.1
70.0+0.6
Stablel.M-2-1.633
34.6+0.2
34.7+0.7
TinyLlama
24.9+0.4
25.9#1.6
Table 9: NRMSE on AGlEval English (lower the better). k is the number of selected tasks. Each number is averaged over different models.
Method
Best
k=1
k=2
k=3
k=4
k=5
k=6
Random
0.34
0.34
1.07
2.08
3.07
4.10
5.12
GPT4
0.07
0.48
0.32
0.20
0.16
0.15
0.07
BM25-le
0.06
0.17
0.24
0.06
0.08
0.11
0.15
BM25-sim
0.15
0.38
0.24
0.28
0.38
0.21
0.15
BENTo-le
0.03
0.38
0.08
0.07
0.05
0.06
0.03
BENTO-sim
0.03
0.53
0.23
0.20
0.16
0.03
0.08
filling-in -nteeiin-tttbieee tasks that require certain natural language response. We require strict match when computing ACC on this dataset. The result is shown in Table 10 On this dataset, we only use the 3 to examples
training set as the exemplars. Under this extreme few-shot setting, BM25 seems to work slightly better than our methods.
Table 10: NRMSE Big Bench Hard (lower the better). k is the number of selected tasks. Each number is averaged over different models.
Method
Best
k=
k=2
k=3
k=4
k=5
k=6
k=7
k=8
Random
0.389
0.389
1.063
2.029
3.010
4.012
5.002
6.004
7.020
GPT4
0.072
0.540
0.174
0.086
0.099
0.110
0.212
0.138
0.072
BM25-le
0.092
0.092
0.353
0.133
0.114
0.171
0.129
0.116
0.135
BM25-sim
0.032
0.748
0.325
0.313
0.087
0.186
0.119
0.032
0.049
BENTo-le 0.045
0.248
0.080
0.090
0.148
0.057
0.045
0.070
0.080
BENTO-sim
0.154
0.780
0.392
0.333
0.388
0.179
0.154
0.196
0.219
G PUBLICLY REPORTED RESULTS ON MMLU
In Table II
annotate the publicly reported results for models listed in Table We do not report our assessment of Gemma-7b on BBH because fails to generate answer in the given format.
16

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of full benchmark performance measured by us and that reported in previous works. The difference may come from different prompts / quantization. BENTo-estimann perfor- mance is also attached for comparison.
Model
Measured Acc(100%)
Reported Acc(100%)
BENTo-estimated Acc(5%)
Llama-2-70b
66.6
68.9
67.9
Llama-2-13b
54.5
54.8
53.9
Llama-2-7b
46.0
45.3
49.8
Llama-3-8b
61.7
69.4
60.2
Mistral-7b-v0.3
62.1
61.1
62.2
Phi-2
56.5
56.7
56.7
Phi-3-mini-4k
69.5
70.9
70.0
StableLMM-21166
34.6
38.9
34.7
TinyLlama
24.9
26.6
25.9
Gemma- 7b
65.2
64.3
63.4
QWEN2.5-7b
74,2
74.2
75.8
QWEN2.5-14b
79.8
79.7
81.6
GPT--4-mini
74.3
82.0
75.1
GPT-4o
68.9
88.7
69.8
17


Reviews:
[{"review_id": "iS0Ucj8sol", "reviewer": "Reviewer_3D6d", "summary": "This paper addresses the challenge of evaluating large language models (LLMs) efficiently by proposing a method to reduce the size of LLM benchmarks without compromising the quality of evaluation. Traditional benchmarking approaches are costly and resource-intensive due to the large number of tasks included to cover various LLM capabilities. The authors introduce In-Context Transferability (ICT) as a training-free method to estimate task transferability by leveraging in-context learning, allowing them to assess which tasks are likely to improve performance on others. Using ICT, they propose Benchmark Task Reduction (BENTO), which formulates the task selection as a facility location problem to select a representative subset of tasks. Experiments show that BENTO can reduce benchmark tasks by up to 95% while maintaining consistent evaluation results, offering a cost-effective solution for LLM benchmarking.", "questions": "Please refer to weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.By reducing the benchmark to just 5% of its original size without sacrificing evaluation quality, BENTO significantly lowers the computational and financial burden of LLM evaluation.\n\n2.The paper’s novel use of in-context learning for transferability estimation provides a scalable, training-free solution for task reduction, making it more practical than traditional methods that rely on finetuning.\n\n3.The ICT and BENTO methods can be applied to various LLM research and evaluation scenarios, potentially benefiting a wide range of applications beyond just benchmarking.", "weaknesses": "1.The reliance on in-context learning may limit the generalizability of the approach, as it assumes that in-context performance accurately reflects task transferability, which may not hold true across all tasks or models, especially for weak LLMs. More experiments for weak LLMs should be added.\n\n2.Although the paper demonstrates consistency with reduced benchmarks on certain datasets, additional testing across more diverse tasks and benchmarks could strengthen the validity of the results. Reducing a benchmark to a small subset of tasks may overlook nuanced skills or specific task requirements that LLMs need to perform well in specialized applications, potentially leading to incomplete evaluations in some cases. \n\n3.It is necessary to analyze if the reduced benchmark would cause the hurt of robustness for benchmark leakage issue or overfitting to few dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XigKRv33a1", "reviewer": "Reviewer_TgKk", "summary": "The paper presents a novel approach for improving the efficiency of benchmark evaluation in machine learning. The authors propose a method for reducing the number of tasks needed for comprehensive benchmarking by leveraging in-context transferability. This involves identifying and selecting a subset of tasks that can effectively represent a broader set of benchmarks while minimizing redundancy.\n(1)\tAnalyzing the transferability between tasks from the perspective of in-context learning\n(2)\tThe experiments on MMLU and FLAN show that the method proposed by the author can approximate the full performance with 5% of the data.", "questions": "Q1: Will the difference in ICL capabilities of different models lead to bias in the evaluation?\nThe authors utilize two models (LLaMA 7 b and 13b). However, it is unclear whether their ICL performance is consistent across these models. Additionally, the methodology for integrating data from these models is not specified—do you simply average the results?\n\nQ2: In the author's experimental setup for FLAN's 66 tasks, 100 questions were selected for each task to be considered as \"whole benchmark\"(Line 274-275). Does this simplification ensure the validity of the experimental results since FLAN contains 700k+ samples?\n\nQ3: The NRMSE is used to compare the performance differences among methods. If the values were to be expressed in terms of absolute accuracy, what would the corresponding values for the other methods be? (Similar to the data presented in Table 1)\n\nTypo: The letter \"Q\" in Appendix B experiment details has not been defined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tAnalyzing task transferability from the perspective of ICL presents a novel approach.\n-\tThe method does not require fine-tuning using existing evaluation data; it only needs to conduct ICL transferability tests on some models, which enhances the scalability of the approach.\n-\tThe experimental results demonstrate greater effectiveness compared to selection methods such as random sampling or BM25.", "weaknesses": "-\tThe method raises concerns about its computational efficiency, particularly regarding the actual sizes of the in-context learning (ICL) test samples. For the ICL transfer tests from task i to task j, what are the sizes of the test samples n_j? \n\nGiven that the transfer tests from i to j involve N^2 tests (where N is the total number of tasks), the total number of test samples conducted amounts to N^2 \\times n_j \\times M. Taking MMLU as an example, with N=57, if n_j \\times M is only 50 (considering M=10 in appendix B), the total number of tests already reaches 160k, which extremely exceeds the complete MMLU dataset of 12k. Extensive ICL testing suggests that the model has been evaluated on a full dataset, which contradicts the authors' motivation for efficiently reducing the test set size.\n\n-\tThe method heavily relies on the careful selection of the parameter k. In real scenarios without prior knowledge, determining the appropriate value for k is a significant challenge that could limit the method's usability in practical applications.\n\n-\tThe approach is currently limited to task-level improvements and does not address example-level enhancements. This limitation may restrict the method's applicability across existing benchmarks, particularly in cases where there are a small number of categories but a large number of examples, such as MATH. Also, the subtasks of dataset FLAN used by the authors, such as ReCoRD and SQuADv2, where the number of examples exceeds 10k, does this indicate a lack of usability for the method?\n\n-\tThe authors only used 9 models for evaluation, raising concerns about whether the method would still perform consistently across more models. For example, LLaMA2-72B, Qwen2-7B, Qwen2-13B, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0yy1NfXRXz", "reviewer": "Reviewer_CaUX", "summary": "This paper introduces BENTO for reducing LLMs evaluation benchmarks by selecting representative tasks based on a new metric, In-Context Transferability (ICT), which estimates task similarity without additional training, reducing the computational cost.", "questions": "N.A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.  The BENTO method effectively reduces the number of benchmark tasks to approximately 5% of the original set with minimal impact (<4%) on evaluation accuracy, making LLM evaluation more cost-effective.\n2. The proposed ICT metric enables efficient estimation of task similarity without requiring model training or fine-tuning, which lowers resource demands.\n3. BENTO demonstrates reliable performance across various LLM benchmarks, indicating its potential applicability to diverse datasets and tasks.", "weaknesses": "1. Currently, LLM benchmarks are highly vulnerable to contamination, which diminishes their reliability and credibility. The proposed BENTO method could exacerbate this issue, potentially causing LLMs to focus on a narrower range of tasks, thereby increasing the risk of contamination.\n2. It would be valuable to see results on more advanced models, such as Gemini or GPT-4o, to assess the generalizability and scalability of the approach.\n3. ICL results can be quite variable. It would be informative to evaluate ICL performance across different models and model sizes.\n4. Since ICL requires additional sampling iterations, it would be helpful to provide the associated computation costs for reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaHa0JfNtA", "reviewer": "Reviewer_pNg5", "summary": "This paper introduces BENTO, a method designed to efficiently minimize the number of tasks in large language model (LLM) benchmarks while preserving evaluation quality. The core ideas include:\n1. In-Context Transferability (ICT): A training-free approach to estimate transferability between task pairs in an LLM benchmark using in-context learning.\n2. BENTO: This method formulates task selection as a facility location problem, utilizing task similarities derived from ICT. BENTO identifies the most representative subset of tasks to approximate the evaluation of the complete benchmark.\n3. Experiments: Compared to the random selection baseline, BENTO achieves higher evaluation accuracy. It can reduce MMLU and FLAN benchmarks to  5% of the original task, enhancing the LLM evaluation efficiency.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an intriguing research question concerning the redundancy of data in benchmarks.\n2. The use of in-context learning to estimate task transferability is effective.\n3. The paper is well-written and easy to comprehend.", "weaknesses": "1. Potential Overfitting: Evaluating with a smaller sample size poses a risk of overfitting. It is challenging to ensure that the evaluation data in the benchmark is not used in supervised fine-tuning. Reducing the scale of evaluation data likely increases the chances of data contamination bias. The paper does not simulate or analyze this scenario.\n2. Greater Significance for SFT data: This method may be more beneficial for deduplicating training data than for selecting benchmark data. Training typically involves multiple iterations and back-propagation, which entail higher costs and thus require deduplication.\n3. Practicality of BENTO for Evaluation Data Reduction: I do not think using BENTO to remove evaluation data as a practical approach. Instead, I view BENTO as a metric for assessing benchmark quality rather than eliminating 95% of existing benchmark data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)