Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation
Authors: Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
Accepted at: acl25

Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation

Peiwen Yuan1*, Yueqi Zhang1*, Shaoxiong Feng2, Yiwei Li1, Xinglin Wang1, Jiayi Shi1
Chuyi Tan1, Boyuan Pan2, Yao Hu2, Kan Li1†
1School of Computer Science and Technology, Beijing Institute of Technology
2Xiaohongshu Inc
{peiwenyuan,zhangyq,liyiwei,wangxinglin,shi jiayi,tanchuyi,likan}@bit.edu.cn
{shaoxiongfeng2023,whd.thu}@gmail.com {panboyuan,xiahou}@xiaohongshu.com

Abstract

Evaluating models on large benchmarks is very resource-intensive, especially during the period of rapid model evolution. Existing efficient evaluation methods estimate the performance of target models by testing them only on a small and static coreset of the benchmark, which is derived from the publicly available evaluation results of source models. These methods rely on the assumption that target models have high prediction consistency with source models. However, we demonstrate that it doesn’t generalize well in practice. To alleviate the inconsistency issue, we present TAILOREDBENCH, a method that conducts customized evaluation tailored to each target model. Specifically, a Global-coreset is first constructed as a probe to identify the most consistent source models for each target model with an adaptive source model selection strategy. Afterwards, a scalable K-Medoids clustering algorithm is proposed to extend the Global-coreset to a tailored Native-coreset for each target model. According to the predictions on Native-coresets, we obtain the performance of target models on the whole benchmark with a calibrated estimation strategy. Comprehensive experiments on 5 benchmarks across over 300 models demonstrate that compared to best-performing baselines, TAILOREDBENCH achieves an average reduction of 31.4% in MAE of accuracy estimates under the same inference budgets, showcasing strong effectiveness and generalizability1.

1 Introduction

Scaling up models in multiple dimensions has led to remarkable advancements in their capabilities (Touvron et al., 2023; Ouyang et al., 2022), which also presents significant challenges for efficiently assessing them. For instance, Liang et al. (2022) reports that evaluating a model with approximately 10 billion parameters on the HELM leaderboard costs over $1,700 via APIs or more than 1,200 GPU hours. Moreover, these costs scale by a factor of \( X \) when exploring and comparing \( X \) different training or inference configurations during the development or deployment phase.

To achieve efficient evaluation, some studies (Vivek et al., 2024; Polo et al., 2024) have explored the following paradigm: step 1. constructing example embeddings according to the predictions from a set of source models (which are freely available for popular leaderboards\(^{‡,§,8}\)); step 2. clustering the benchmark and selecting the cluster centroids to form a coreset (typically less than 100 examples); step 3. approximating the performance of target models under evaluation based on their predictions on the coreset. Underlying this approach is the assumption that performance patterns generalize: if source models respond similarly to two examples \( a \) and \( b \), then a target model’s performance on \( a \) can be used to estimate its performance on \( b \).

Nevertheless, we find that such generalizability between source and target models does not necessarily hold. Following ANCHORPOINT (Vivek et al., 2024), we construct an embedding based on the correctness (e.g., the probability of the correct option) of all source models for each example and visualize them using t-SNE algorithm (Van der Maaten and Hinton, 2008). In these embeddings (Figure 1a), nearby examples elicit similar predictions from the source models, allowing cluster centroids (marked by stars) to serve as representative points. Yet, when we adopt embeddings derived from the correctness of target models instead (Figure 1b), the average distance between the example and its centroid increases from 10.09 to 12.48, in-

1Equal contribution.
*Corresponding author.
1Our code is available at https://github.com/marvelcell/TailoredBench
‡https://huggingface.co/open-llm-leaderboard
§https://rank.opencompass.org.cn
8https://crfm.stanford.edu/helm
(a) Hellaswag benchmark represented by source-model embeddings \( \mathcal{D}^S \).

(b) Hellaswag benchmark represented by target-model embeddings \( \mathcal{D}^T \).

Figure 1: The t-SNE visualization of the Hellaswag benchmark using embeddings derived from source (above) and target (below) models’ predictions. The increased average distance between examples and their cluster centroids in the target-based embedding indicates that the coreset (centroids) obtained from source-based embeddings no longer effectively represents the entire benchmark for target models.

indicating that the previously chosen centroids fail to represent their respective clusters effectively. This reveals a discrepancy in prediction behaviors between source and target models, which we term prediction consistency—the extent to which their predictions align on the same examples. When prediction consistency is low, source-model-derived coresets fail to generalize, resulting in inaccurate performance estimates for target models.

To address the aforementioned issue, we propose the TAILOREDBENCH method, which adaptively constructs model-specific evaluation coreset in a global to native manner for accurate and efficient evaluation. Specifically, we first construct a static G-set (Global-coreset) based on the prediction results of all the source models. By applying an adaptive source model selection strategy, the predictions of target models on the G-set are used as a probe to select a native source model set for each target model that has stronger prediction consistency with them. Based on this posterior, we design a scalable K-Medoids clustering technique to expand the G-set into an N-set (Native-coreset) for each target model, according to the benchmark embeddings under the metric of corresponding native source models. Finally, we approximate the overall performance of target models by employing a calibrated estimation strategy based on their predictions on the N-set.

We conduct extensive experiments on five benchmarks across more than 300 models, involving tasks in the fields of natural language and multi-modality. Compared to non-customized efficient evaluation baselines, TAILOREDBENCH can more accurately estimate the performance of models (attaining an average of 31.4% MAE degradation improvement on accuracy) under the same small-size inference budgets (generally 20–40 examples). Our contributions are summarized as follows:
• We analyze that the existing efficient evaluation methods overestimate the prediction consistency across models, thus the source-model-based static coreset may fail to assess the target models accurately.
• We propose the TAILOREDBENCH method to conduct tailored evaluation on adaptively constructed N-set for each target model to attain more accurate evaluation results.
• We conduct comprehensive experiments and analyses on multiple settings to validate the excellent effectiveness and strong generalizability of TAILOREDBENCH.
2 Related Works

As LLMs proliferate and version updates accelerate, the cost of thoroughly evaluating each model across all benchmarks has become prohibitive, leading to methods that subsample the most representative subsets from each benchmark for more efficient evaluation. (Vivek et al., 2024) clusters examples directly using the confidence scores provided by source models, leveraging these scores to select an optimal subset. Similarly, (Polo et al., 2024) employs an Item Response Theory (IRT) model, trained on the success matrix of each source model across various examples, to derive the latent representations of examples for clustering. (Pacchiardi et al., 2024) introduces a generic assessor framework that predicts the performance of a new LLM on unseen instances using its results on a small reference set, achieving comparable accuracy to full-scale evaluations. (Perlitz et al., 2023) proposes Flash-HELM, which dynamically adjusts the sizes of randomly selected subsets based on model ranking, where higher-ranked models are evaluated with greater precision. (Prabhu et al., 2024) proposes the Sort & Search (S&S) strategy, which leverages the difficulties of examples and dynamic programming to select the coreset. (See more related works in Appendix A.)

3 TailoredBench Approach

The TAILOREDBENCH approach centers on dynamically selecting prediction-consistent source models and crafting an N-set that faithfully represents the entire benchmark for each target model. Its formulation proceeds through four tightly integrated steps: constructing a globally representative G-set (\S 3.2), identifying native source models (\S 3.3) and developing N-set for each target model (\S 3.4), and finally estimating the target models’ overall performance (\S 3.5).

3.1 Task Set-Up

Let \( \mathcal{D} = \{(x_k, y_k)\}_{k=1}^{|\mathcal{D}|} \) denotes a benchmark, where \( x_k \) is the input and \( y_k \) is the corresponding ground truth output. We define the set of target models under evaluation as \( \mathcal{T} = \{t_m\}_{m=1}^{|\mathcal{T}|} \). Additionally, we denote the source model set as \( \mathcal{S} = \{s_n\}_{n=1}^{|\mathcal{S}|} \), for which we have access to their predictions across all examples in \( \mathcal{D} \). Following previous works, we ensure that \( \mathcal{T} \cap \mathcal{S} = \varnothing \). Our objective is to accurately estimate the performance \( P_{t_m} \) of each target model \( t_m \in \mathcal{T} \) and determine the ranking relationships within \( \mathcal{T} \), while minimizing the model inference cost.

3.2 Constructing G-set

We first construct the G-set \( \mathcal{G} \), which is designed as a probe for each target model to identify a set of source models with the highest prediction consistency. Consequently, it is intended to be a small yet relatively representative subset of the benchmark, ensuring its generalizability across target models.

Following prior works (Vivek et al., 2024), we employ clustering based on the correctness of source models to construct the G-set. Here, correctness can be either the predictive probability of the correct option (continuous value [0, 1]) or whether the model answers the example correctly (discrete binary value {0, 1}).

For each example \( x_k \) in the benchmark \( \mathcal{D} \), we compute an embedding using the correctness scores \( c_{s_n, x_k} \) from each source model \( s_n \):

\[
\hat{x}_k^{\mathcal{S}} = \begin{pmatrix}
c_{s_1, x_k} \\
c_{s_2, x_k} \\
\vdots \\
c_{s_{|\mathcal{S}|}, x_k}
\end{pmatrix}
\]

(1)

The superscript \( \mathcal{S} \) indicates that the embedding is derived from source models’ correctness, and each embedding is \( |\mathcal{S}| \)-dimensional. The collection of these embeddings constitutes the benchmark’s representation \( \mathcal{D}^{\mathcal{S}} = \{\hat{x}_k^{\mathcal{S}}\}_{k=1}^{|\mathcal{D}|} \).

Based on \( \mathcal{D}^{\mathcal{S}} \), we apply K-Medoids clustering (Kaufman and Rousseeuw, 2009) to select the G-set with the objective function below:

\[
\min_{\{\mathcal{G}, C_g\}} \sum_{x_g \in \mathcal{G}} \sum_{x_k \in C_g \setminus \{x_g\}} \mathrm{Dis}(\hat{x}_g^{\mathcal{S}}, \hat{x}_k^{\mathcal{S}})
\]

(2)

where \( x_g \) is an example in the G-set \( \mathcal{G} = \{x_g\}_{g=1}^{|\mathcal{G}|} \), and \( C_g \) is the cluster for which \( x_g \) is the centroid. \( \mathrm{Dis} \) denotes the distance metric in clustering.

To maximize the generalization capability of our method, the choice of distance metric is critical. Previous approaches (Vivek et al., 2024; Miller et al., 2021; Baek et al., 2022; Mehra et al., 2024) using correlation distance (Rodgers and Nicewander, 1988) to measure example consistency often assume linear relationships in scoring patterns among models or examples. However, this assumption may not hold for discrete numerical embeddings, leading to significant performance degradation. In contrast, element-wise distance (e.g.,
Manhattan distance) can effectively capture individual discrepancies in correctness vectors, thereby accommodating various correctness formats. By default, we adopt manhattan distance as \( \mathrm{Dis} \) for our TAILOREDBENCH method.

3.3 Adaptive Native Source Model Selection

After constructing G-set \( \mathcal{G} \), we attain the prediction results of target models \( \mathcal{T} \) on it, which we use as a probe to construct a Native Source Model Set \( \mathcal{S}_{t_m} \) that exhibits the highest prediction consistency for each \( t_m \in \mathcal{T} \).

Specifically, we first embed all the source models \( s_n \in \mathcal{S} \) and target models \( t_m \in \mathcal{T} \) based on their prediction correctness on \( \mathcal{G} \) as follows:

\[
s_n^{\mathcal{G}} = \begin{pmatrix}
c_{s_n,x_1} \\
c_{s_n,x_2} \\
\vdots \\
c_{s_n,x_{|\mathcal{G}|}}
\end{pmatrix}, \quad t_m^{\mathcal{G}} = \begin{pmatrix}
c_{t_m,x_1} \\
c_{t_m,x_2} \\
\vdots \\
c_{t_m,x_{|\mathcal{G}|}}
\end{pmatrix}
\]

Here, the superscript \( \mathcal{G} \) denotes that each embedding dimension is derived from the model’s prediction correctness on the G-set. Leveraging these embeddings, we compute the average prediction consistency \( \bar{d} \) among all the models (both source and target) on the G-set as follows:

\[
\bar{d} = \frac{2}{(|\mathcal{S}| + |\mathcal{T}|)(|\mathcal{S}| + |\mathcal{T}| - 1)} \sum_{i < j} d_{ij},
\]
where \( d_{ij} = \mathrm{Dis}\left(s_i^{\mathcal{G}}, t_j^{\mathcal{G}}\right) \)

In this context, \( i, j \in [1, |\mathcal{S}|+|\mathcal{T}|] \) and \( \phi \) represents any model from \( \mathcal{S} \cup \mathcal{T} \). By computing \( \bar{d} \) across all models, we establish a robust threshold that reflects the model set’s similarity landscape, enabling a consistent and effective selection of native source models for each target model.

On this basis, we determine \( \bar{n} \), the size of the native source model set for target models, by calculating the average number of source models whose prediction consistency with each target model exceeds the threshold \( \bar{d} \) as follows:

\[
\bar{n} = \left\lfloor \frac{1}{|\mathcal{T}|} \sum_{m=1}^{|\mathcal{T}|} |\mathcal{S}_{t_m}| \right\rfloor,
\]
where \( \mathcal{S}_{t_m} = \left\{ s_n \in \mathcal{S} \mid \mathrm{Dis}\left(s_n^{\mathcal{G}}, t_m^{\mathcal{G}}\right) < \bar{d} \right\} \)

For a target model \( t_m \), the top \( \bar{n} \) source models exhibiting the highest prediction consistency are selected to form its dynamic source model set \( \mathcal{S}_{t_m} \). By standardizing the number of native source models across all target models, we ensure that each target model’s feature representation maintains con-
sistent dimensionality and informational richness during subsequent clustering.

3.4 Developing N-set

Leveraging the selected native source models \( S_{t_m} \), we construct the most representative N-set \( \mathcal{N}_{t_m} \) for each target model \( t_m \). To maximize the utilization of the observed prediction results of target models on \( \mathcal{G} \), we propose a Scalable K-Medoids Clustering algorithm to extend \( \mathcal{G} \) into the N-set.

Initially, each example \( x_k \in \mathcal{D} \) is represented by a feature vector \( x_k^{S_{t_m}} \), which is based on the correctness of its native source models \( S_{t_m} \). Then, our Scalable K-Medoids Clustering algorithm operates as follows:

Anchored Medoid Initialization: We fix the examples in G-set \( |\mathcal{G}| \) as initial medoids. To reach an N-set size of \( |\mathcal{N}_{t_m}| \), we randomly add \( |\mathcal{N}_{t_m}| - |\mathcal{G}| \) additional examples from \( \mathcal{D} \setminus \mathcal{G} \) to form the initial medoid set.

Cluster Assignment: Assign each example \( x_k \in \mathcal{D} \) to the nearest medoid \( x_\mu \) to form the cluster \( C_\mu \):

\[
x_k \in C_\mu,
\]
where \( \mu = \arg\min_\mu \mathrm{Dis}\left(x_\mu^{S_{t_m}}, x_k^{S_{t_m}}\right) \) (6)

Dynamic Medoid Refinement: For each cluster \( C_\mu \) with a non-G-set medoid, update the medoid \( x_\mu \) by selecting the example within \( C_\mu \) that minimizes the total distance to all other examples in \( C_\mu \):

\[
x_\mu = \arg\min_{x_i \in C_\mu} \sum_{x_j \in C_\mu \setminus \{x_i\}} \mathrm{Dis}\left(x_i^{S_{t_m}}, x_j^{S_{t_m}}\right)
\] (7)

Medoids corresponding to G-set remain fixed during this process.

Convergence Verification: Repeat the Cluster Assignment and Dynamic Medoid Refinement steps until convergence is achieved, i.e., when medoids no longer change or a maximum number of iterations is reached.

By incorporating the G-set examples as fixed medoids, the clustering process ensures that these pivotal examples guide the formation of clusters and the selection of additional N-set examples.

3.5 Calibrated Performance Estimation

After establishing the N-set \( \mathcal{N}_{t_m} \) for a target model \( t_m \), for previous methods (Vivek et al., 2024; Polo et al., 2024), they may estimate the model’s overall performance by first evaluating it on these centroid examples and then weighting the results according to each centroid’s coverage of the benchmark. However, simply relying on medoids overlooks subtle variations in how individual examples within each cluster are predicted, potentially leading to less accurate global estimates.

To address this, we leverage the prediction consistency between the target model \( t_m \) and its native source models \( S_{t_m} \) to obtain the calibrated correctness estimates for the target model. For a given cluster with medoid \( x \), consider any non-medoid example \( x' \) in the same cluster. We compute a scaling factor based on the native source models’ average correctness, which reflects how the prediction patterns at \( x' \) relate to those at \( x \):

\[
\mathrm{Scale}(x') = \frac{\bar{c}_{S_{t_m},x'} + 0.5}{\bar{c}_{S_{t_m},x} + 0.5}
\] (8)

Here, \( \bar{c}_{S_{t_m},x} \) and \( \bar{c}_{S_{t_m},x'} \) denote the average correctness of \( S_{t_m} \) on the medoid \( x \) and the non-medoid \( x' \), respectively. The addition of 0.5 ensures numerical stability by preventing the denominator from becoming zero. Given that \( t_m \) and \( S_{t_m} \) exhibit similar prediction consistencies, we assume this scaling factor can be applied to estimate the target model’s correctness on \( x' \):

\[
c_{t_m,x'} = (c_{t_m,x} + 0.5) \cdot \mathrm{Scale}(x') - 0.5
\] (9)

By integrating these inferred correctness values across all examples in the benchmark \( \mathcal{D} \), we obtain a more faithful global performance estimation without re-evaluating the entire dataset:

\[
P_{t_m} = \frac{1}{|\mathcal{D}|} \sum_{x' \in \mathcal{D}} c_{t_m,x'}
\] (10)

4 Experiments

4.1 Experimental Setup

Benchmarks and Models We validate TAILOREDBENCH on five diverse benchmarks spanning natural language and multimodal tasks. ARC Challenge (Clark et al., 2018) consists of 1,172 scientific reasoning questions, with predictions from 153 models. Hellaswag (Zellers et al., 2019) provides 6,000 commonsense inference examples (a subset of its validation set) and outputs from 139 models. GSM8K (Cobbe et al., 2021) includes 1,319 math reasoning problems tested on 150 models. Winogrande (Sakaguchi et al., 2021) has 1,267
<table>
  <tr>
    <th rowspan="2">Benchmarks</th>
    <th rowspan="2">Inference counts</th>
    <th colspan="3">20</th>
    <th colspan="3">25</th>
    <th colspan="3">30</th>
    <th colspan="3">35</th>
    <th colspan="3">40</th>
  </tr>
  <tr>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>ARC Challenge</td>
    <td>Best Baseline</td>
    <td>0.662</td>
    <td>0.046</td>
    <td>0.663</td>
    <td>0.046</td>
    <td>0.676</td>
    <td>0.036</td>
    <td>0.713</td>
    <td>0.036</td>
    <td>0.714</td>
    <td>0.029</td>
  </tr>
  <tr>
    <td></td>
    <td>TailoredBench</td>
    <td><b>0.711</b></td>
    <td><b>0.031</b></td>
    <td><b>0.737</b></td>
    <td><b>0.029</b></td>
    <td><b>0.756</b></td>
    <td><b>0.028</b></td>
    <td><b>0.766</b></td>
    <td><b>0.027</b></td>
    <td><b>0.773</b></td>
    <td><b>0.027</b></td>
  </tr>
  <tr>
    <td>Hellaswag</td>
    <td>Best Baseline</td>
    <td>0.860</td>
    <td>0.060</td>
    <td>0.880</td>
    <td>0.053</td>
    <td>0.877</td>
    <td>0.043</td>
    <td>0.897</td>
    <td>0.038</td>
    <td>0.898</td>
    <td>0.032</td>
  </tr>
  <tr>
    <td></td>
    <td>TailoredBench</td>
    <td><b>0.900</b></td>
    <td><b>0.020</b></td>
    <td><b>0.909</b></td>
    <td><b>0.018</b></td>
    <td><b>0.913</b></td>
    <td><b>0.018</b></td>
    <td><b>0.914</b></td>
    <td><b>0.017</b></td>
    <td><b>0.918</b></td>
    <td><b>0.017</b></td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>Best Baseline</td>
    <td>0.811</td>
    <td>0.055</td>
    <td>0.828</td>
    <td>0.047</td>
    <td>0.839</td>
    <td>0.041</td>
    <td>0.847</td>
    <td>0.038</td>
    <td>0.858</td>
    <td>0.034</td>
  </tr>
  <tr>
    <td></td>
    <td>TailoredBench</td>
    <td><b>0.852</b></td>
    <td><b>0.035</b></td>
    <td><b>0.858</b></td>
    <td><b>0.034</b></td>
    <td><b>0.863</b></td>
    <td><b>0.033</b></td>
    <td><b>0.869</b></td>
    <td><b>0.031</b></td>
    <td><b>0.878</b></td>
    <td><b>0.029</b></td>
  </tr>
  <tr>
    <td>Winogrande</td>
    <td>Best Baseline</td>
    <td>0.472</td>
    <td>0.041</td>
    <td>0.487</td>
    <td>0.038</td>
    <td>0.514</td>
    <td>0.038</td>
    <td>0.521</td>
    <td>0.036</td>
    <td>0.518</td>
    <td>0.034</td>
  </tr>
  <tr>
    <td></td>
    <td>TailoredBench</td>
    <td><b>0.565</b></td>
    <td><b>0.028</b></td>
    <td><b>0.568</b></td>
    <td><b>0.026</b></td>
    <td><b>0.604</b></td>
    <td><b>0.024</b></td>
    <td><b>0.608</b></td>
    <td><b>0.023</b></td>
    <td><b>0.618</b></td>
    <td><b>0.022</b></td>
  </tr>
  <tr>
    <td>POPE</td>
    <td>Best Baseline</td>
    <td>0.488</td>
    <td>0.038</td>
    <td>0.510</td>
    <td>0.037</td>
    <td>0.518</td>
    <td>0.034</td>
    <td>0.547</td>
    <td>0.033</td>
    <td>0.556</td>
    <td>0.031</td>
  </tr>
  <tr>
    <td></td>
    <td>TailoredBench</td>
    <td><b>0.521</b></td>
    <td><b>0.036</b></td>
    <td><b>0.547</b></td>
    <td><b>0.035</b></td>
    <td><b>0.562</b></td>
    <td><b>0.031</b></td>
    <td><b>0.563</b></td>
    <td><b>0.031</b></td>
    <td><b>0.574</b></td>
    <td><b>0.032</b></td>
  </tr>
</table>

Table 1: Results on all benchmarks. For each setting, we take the best result from multiple baselines to compare with TailoredBench. The detailed performance of each baseline under each setting is presented in Table 6. Values in bold represent the best results.

pronoun resolution examples with 150 models evaluated. POPE (Li et al., 2023) features 5,127 instances for assessing multimodal hallucination, accompanied by results from 99 models. A complete list of models used for each benchmark is provided in Appendix C. We randomly split models into source and target sets for each benchmark, ensuring that their intersection is empty.

For ARC Challenge and Hellaswag, model correctness is represented by continuous probabilities, while GSM8K, Winogrande, and POPE use binary correctness {0, 1}. Predictions for ARC Challenge, Hellaswag, GSM8K, and Winogrande come from the Open LLM Leaderboard (Beeching et al., 2023), and those for POPE are from the OpenCompass Leaderboard (Contributors, 2023).

Baseline and Evaluation Metrics We compare TailoredBench against three baselines: a Random Sampling strategy that randomly selects a subset of examples from the benchmark to estimate model performance, serving as a basic reference point; the Anchor Points method (Vivek et al., 2024), which uses K-Medoids clustering on source-model predictions to identify a fixed representative coreset; and gp-IRT (Polo et al., 2024), which employs an Item Response Theory model trained on the predictions of the source models to estimate target models’ performance on the full benchmark. In all cases, we use the same source models and target models to ensure a fair comparison.

We employ two metrics to assess these methods. Kendall’s τ Correlation Coefficient evaluates the ordinal agreement between estimated and true model rankings, indicating how well the relative performance order is preserved. Mean Absolute Error (MAE) measures the average absolute deviation between estimated and true performance scores, thereby capturing the precision of performance estimation for individual target models.

4.2 Main Results

TailoredBench: Effective Ranking and Estimation of Model Performances Table 1 presents a comprehensive comparison between our TailoredBench method and the best baseline approaches for each metric across all benchmarks. Full results are available in Appendix B.1. In our experiments, we allocated 10 examples to the G-set and averaged the outcomes over 100 randomized trials to ensure statistical reliability. The inference count—defined as the number of examples in the N-set for our method—varied from 20 to 40.

As demonstrated in the table, our method consistently outperforms baseline approaches in both Kendall’s τ and MAE metrics across all inference counts and benchmarks featuring different correctness types. When the inference count increases, the performance of our method continues to improve, evidenced by a steady increase in Kendall’s τ and a continuous decrease in MAE. Notably, compared to best performing baselines, our approach achieves nearly a 31.4% reduction in MAE. These results indicate that our method effectively estimates the relative performance among target models and provides more accurate estimations of their performance on the entire benchmark. Furthermore, compared to the static AnchorPoints method, our approach significantly improves both Kendall’s τ and MAE metrics, highlighting its effectiveness in adaptively selecting a more representative N-set for each target model and thereby
(a) The impact of the quantity of Native Source Models (with prediction consistency kept the same).

(b) The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same).

Figure 3: Investigating the impact of native source model quantity and prediction consistency with target model on GSM8K using the controlled variable method.

improving estimation accuracy. We also calculate the accuracy of our method in ranking the performance between every pair of target models. The results show that the accuracy reached 96.0% on the Hellaswag benchmark and 93.6% on the GSM8K benchmark. In terms of robustness, Appendix B.8 demonstrates that our method exhibits significantly lower variance compared to the baselines.

Moreover, across all benchmarks and inference counts, we conduct a one-sided Z-test over 100 repeated experiments. Whenever our method outperformed the baselines, the p-values remained below 0.05, confirming a statistical advantage.

4.3 Ablation Studies

Element-Wise Distance Effectively Facilitates Handling Various Data Forms Our method uses element-wise Distance (specifically Manhattan distance) to effectively handle both continuous and discrete values. As shown in Table 2, with 30 inference counts, element-wise Distances outperform the correlation distance used by AnchorPoints. This confirms its effectiveness in improving our method’s performance. Detailed per-dataset results are provided in Appendix B.2.

<table>
  <tr>
    <th>Distance</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>CORRELATION</td>
    <td>0.720</td>
    <td>0.032</td>
  </tr>
  <tr>
    <td>COSINE</td>
    <td>0.736</td>
    <td>0.028</td>
  </tr>
  <tr>
    <td>MANHATTAN</td>
    <td><b>0.740</b></td>
    <td><b>0.027</b></td>
  </tr>
</table>

Table 2: Average performance with different types of distance across benchmarks.

Calibrated Estimation Strategy Improves Performance Estimation We compare TAILORED-BENCH with and without calibration. As shown in Table 3, with 30 inference counts, the calibrated variant achieves higher Kendall’s \( \tau \) and lower MAE, confirming that calibration enhances the accuracy of performance estimation. Detailed per-dataset results are provided in Appendix B.3.

More Ablation Studies We conduct additional ablation studies on our method in Appendices B.4 and B.5. The results show that (1) using a fixed number of native source models for each target model stabilizes performance; (2) fixing the G-set as part of the N-set strikes a balance between effectiveness and the inference budget.

4.4 Analyses

Impact of Native Source Model Selection on Method Performance Here, we isolate the effects of both the number of native source models and their prediction consistency with the target model by independently varying these factors.

When native source models share a fixed level of prediction consistency with the target model, increasing their number enhances performance. To investigate this, we randomly select models designated as native source models, from 20% to 100%. As shown in Figure 3a, performance improves as more native source models are included, since a larger set of models offers a greater chance of obtaining a more robust embedding. See Appendix B.9 for results on more benchmarks.

When the number of native source models is fixed, higher prediction consistency with the target model enhances performance. To examine this, we select a fixed number of native source models at various consistency levels relative to the target model (top 20%, 20-40%, up to 80-100%). As
<table>
  <tr>
    <th>Method Variants</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>NON-CALIBRATED</td>
    <td>0.724</td>
    <td>0.030</td>
  </tr>
  <tr>
    <td>CALIBRATED</td>
    <td><b>0.740</b></td>
    <td><b>0.027</b></td>
  </tr>
</table>

Table 3: Average performance with and without calibration across benchmarks.

shown in Figure 3b (with the horizontal axis representing the Consistency Percentile Range for these intervals), Kendall’s \( \tau \) decreases sharply as the consistency percentile range expands. See Appendix B.10 for additional benchmark results.

TailoredBench Method Adaptively Selects Optimal Native Source Model Sets Here, we analyze the ability of our method to select the optimal native source model sets. Figure 4 shows the performance of our method on the GSM8K benchmark, where source models with the top-k prediction consistency to the target model are selected as Native source models. The results reveal that Kendall’s \( \tau \) coefficient initially increases and then decreases as the number of native source models grows, while the MAE first decreases and then increases. This trend aligns with our observations in Figure 3. Specifically, when only a few native source models are selected, their high consistency with the target model is offset by the noise introduced due to the small sample size, which reduces clustering performance. Increasing the number of native source models helps mitigate this issue and improves performance until an optimal point is reached. However, selecting too many native source models incorporates models with lower prediction consistency to the target model, which diminishes effectiveness. Our method addresses this by adaptively selecting the near-optimal number of native source models across all benchmarks. For example, as shown in Figure 4, our approach selects 40 native source models for each target model on the GSM8K benchmark, achieving near-optimal performance. Further experiments pertaining to this section are detailed in Appendix B.11.

Additionally, we observe that target models preferentially select native source models from their own family, which can better capture the nuances and prediction patterns distinctive to their respective model lineages and contribute to more accurate performance estimations. This intra-family selection bias and the performance of our method when target models significantly differ from source models is further explored in detail in Appendix B.6.

10 Examples are Sufficient for the Probe We investigate how G-set size affects our method’s performance by fixing the N-set at 30 examples and varying the G-set from 5 to 25 examples across all benchmarks. As shown in Table 4, Kendall’s \( \tau \) peaks and MAE reaches a minimum at a G-set size of 10. Smaller G-set fail to capture the prediction consistency between source and target models, limiting effective N-set selection. Conversely, a larger G-set reduces N-set representativeness by being dominated by G-set points, leading to diminished performance. Detailed per-dataset results are provided in Appendix B.7.

<table>
  <tr>
    <th>|G-set|</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>5</td>
    <td>0.734</td>
    <td>0.030</td>
  </tr>
  <tr>
    <td>10</td>
    <td><b>0.740</b></td>
    <td><b>0.027</b></td>
  </tr>
  <tr>
    <td>15</td>
    <td>0.736</td>
    <td>0.028</td>
  </tr>
  <tr>
    <td>20</td>
    <td>0.735</td>
    <td>0.028</td>
  </tr>
  <tr>
    <td>25</td>
    <td>0.731</td>
    <td>0.029</td>
  </tr>
</table>

Table 4: Average performance with different G-set size across benchmarks.

Performance with Larger Inference Count On the Hellaswag Benchmark. We further evaluate our method with larger inference counts on the Hellaswag benchmark. Table 5 shows that as inference counts increase from 50 to 150, TAILOREDBENCH consistently improves model performance prediction and ranking, maintaining a clear advantage over baseline methods, demonstrating its effectiveness with larger inference budgets.

<table>
  <tr>
    <th rowspan="2">Inference counts</th>
    <th colspan="2">50</th>
    <th colspan="2">100</th>
    <th colspan="2">150</th>
  </tr>
  <tr>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>RANDOM</td>
    <td>0.887</td>
    <td>0.053</td>
    <td>0.920</td>
    <td>0.038</td>
    <td>0.935</td>
    <td>0.030</td>
  </tr>
  <tr>
    <td>ANCHOR POINTS</td>
    <td>0.915</td>
    <td>0.046</td>
    <td>0.931</td>
    <td>0.040</td>
    <td>0.940</td>
    <td>0.040</td>
  </tr>
  <tr>
    <td>GP-IRT</td>
    <td>0.869</td>
    <td>0.026</td>
    <td>0.915</td>
    <td>0.015</td>
    <td>0.936</td>
    <td>0.012</td>
  </tr>
  <tr>
    <td>TAILORED BENCH</td>
    <td><b>0.923</b></td>
    <td><b>0.016</b></td>
    <td><b>0.934</b></td>
    <td><b>0.014</b></td>
    <td><b>0.943</b></td>
    <td><b>0.012</b></td>
  </tr>
</table>

Table 5: Performance of compared methods on the Hellaswag benchmark with larger inference counts.

5 Conclusions

In this paper, we propose the TAILOREDBENCH method, which mainly includes an adaptive source model set selection strategy, a scalable K-Medoids clustering algorithm and a calibrated performance estimation strategy. Abandoning the one-size-fits-all approach, we have customized the evaluation on
Figure 4: Performance of TAILOREDBENCH with varying numbers of Native Source Models on GSM8K benchmark. The shaded area indicates the adaptively selected number of native source models and the corresponding performance of our method.

the constructed native coreset for each target model. This approach enables a more accurate reconstruction and ranking of the model’s performance on the entire benchmark with a small-size inference budget. Comprehensive experiments show that TAILOREDBENCH can achieve more accurate model evaluation (an average of 31.4% estimation MAE loss degradation) with few inference costs.

Limitations

A primary limitation of mainstream approaches in benchmark compression, including (Vivek et al., 2024; Polo et al., 2024), and our method, is their dependence on comprehensive evaluation results from existing models across all examples within a benchmark. As described above, these results are typically readily accessible through public leaderboards. However, obtaining initial model performance results is necessary for new or certain private benchmarks, which introduces additional inference overhead. Nonetheless, we maintain that this initial cost is justified, as it is offset by the significant resource savings achieved through numerous subsequent rapid evaluations facilitated by our method.

Ethics Statement

All of the datasets used in this study were publicly available, and no annotators were employed for our data collection. We confirm that the datasets we used did not contain any harmful content and was consistent with their intended use (research). We have cited the datasets and relevant works used in this study.

Acknowledgments

This work is supported by Beijing Natural Science Foundation (No.4222037, L181010).

References

Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, and Ludwig Schmidt. 2022. Exploring the landscape of distributional robustness for question answering models. arXiv preprint arXiv:2210.12517.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. 2022. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274–19289.

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sansevieri, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. 2023. Predicting emergent abilities with infinite resolution evaluation. In
The Twelfth International Conference on Learning Representations.

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177.

Leonard Kaufman and Peter J Rousseeuw. 2009. Finding groups in data: an introduction to cluster analysis. John Wiley & Sons.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.

Aman Mehra, Rahul Saxena, Taeyoun Kim, Christina Baek, Zico Kolter, and Aditi Raghunathan. 2024. Predicting the performance of foundation models via agreement-on-the-line. arXiv preprint arXiv:2404.01542.

John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. 2021. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning, pages 7721–7735. PMLR.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katrina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Lorenzo Pacchiardi, Lucy G Cheke, and José Hernández-Orallo. 2024. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563.

Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. 2023. Efficient benchmarking (of language models). arXiv preprint arXiv:2308.11696.

Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Ameya Prabhu, Vishaal Udandara, Philip Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. 2024. Lifelong benchmarks: Efficient model evaluation in an era of rapid progress. arXiv preprint arXiv:2402.19472.

Joseph Lee Rodgers and W. Alan Nicewander. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59–66.

Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Rohan Taori, Achal Dave, Vaishal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583–18599.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yunjing Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungra, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).

Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2024. Anchor points: Benchmarking models with much fewer examples. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long Papers, St. Julian’s, Malta,
Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya. 2024. Data efficient evaluation of large language models and text-to-image models via adaptive sampling. arXiv preprint arXiv:2406.15527.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Qiyuan Zhang, Fuyuan Lyu, Xue Liu, and Chen Ma. 2024. Collaborative performance prediction for large language models. arXiv preprint arXiv:2407.01300.
A  Related Works

Models Correlation in Predictive Consistency:
Prior works (Taori et al., 2020; Miller et al., 2021; Awadalla et al., 2022) have demonstrated a certain level of correlation between in-distribution (ID) and out-of-distribution (OOD) performances across diverse models and tasks. Building on this foundation, (Back et al., 2022) and (Mehra et al., 2024) advance this relationship by showing the phenomenon that the agreement between two models on ID data is linearly correlated with their agreement on OOD data, where the accuracy holds the similar linear relationship, enabling accurate estimation of model’s OOD accuracy based solely on ID data. Our work extends this phenomenon to address the challenge of benchmark compression, enabling the selection of more representative subsets for benchmarks.

(Xu et al., 2024) synthesizes several methods and dynamically chooses the optimal subset selection method for each benchmark but requires many examples to determine the best approach. Despite these advancements, these methods often struggle with substantial distribution shifts between the source and target models, caused by the discrepancy between their predictive consistency, potentially causing significant distortion in estimating the target model’s performance. Extending the approach of (Vivek et al., 2024), our work alleviates this issue by dynamically selecting a native source model set with the highest prediction consistency to the target model, ensuring the selection of a tailored corset for each target model that best represents the benchmark.

Scaling Approaches for Model Performance Estimations: Scaling law describes the relationship between model properties (e.g., FLOPs used during training, model parameter size) and model capabilities. Recent works (Hu et al., 2023; Ruan et al., 2024; Isik et al., 2024) have leveraged scaling laws to predict model performance on various downstream tasks, reducing the computational cost of evaluating models on complex downstream tasks. (Zhang et al., 2024) simplifies those approaches by utilizing the relationships between model families and their collaborative overall performance across tasks rather than fitting scaling laws. The aforementioned methods typically rely on overall model performance across several benchmarks and specific design factors (e.g., model size or training data properties) to either fit scaling curves or investigate correlations between models on various tasks. In contrast, our approach addresses a more general case by reducing the evaluation cost for multiple models on a single benchmark, offering a more efficient performance estimation framework.

B  More Experimental Results

B.1  Comprehensive Experimental Results Across All Datasets

In Table 6, we present a comprehensive comparison of our approach against all baseline methods across the full range of benchmark datasets. The results indicate that our method consistently outperforms every baseline under all considered inference counts, thereby demonstrating the overall effectiveness of our proposed approach.

B.2  Comprehensive Distance Measures Ablation Across Benchmarks

Here, we provide comprehensive results of our ablation study evaluating the impact of different distance measures on our method’s performance with 30 inference counts across various benchmarks. Table 7 presents detailed Kendall’s \( \tau \) and MAE metrics for cosine similarity, Manhattan distance, and correlation distance across all datasets. These results offer deeper insights into the effectiveness of Element-Wise Distance measures in enhancing benchmark compression.

B.3  Detailed Calibration Ablation Results

Table 8 presents the results of our ablation study, comparing our TAILOREDBENCH method with and without the calibrated performance estimation process under 30 inference counts. The calibrated version of our method generally achieves higher Kendall’s \( \tau \) scores and lower mean absolute errors (MAE) across various benchmarks and inference counts, demonstrating that the calibrated performance estimation process effectively enhances the performance estimation ability of our method.

B.4  Impact of Dynamic Native Source Model Quantity on Performance

In this ablation study, we investigate the effect of dynamically selecting varying numbers of native source models for each target model, as opposed to using a standardized quantity across all target models. Specifically, instead of treating \( \bar{n} \) (as computed
<table>
  <tr>
    <th rowspan="2">Benchmarks</th>
    <th rowspan="2">Inference counts</th>
    <th colspan="2">20</th>
    <th colspan="2">25</th>
    <th colspan="2">30</th>
    <th colspan="2">35</th>
    <th colspan="2">40</th>
  </tr>
  <tr>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td rowspan="4">ARC Challenge</td>
    <td>RANDOM</td><td>0.626</td><td>0.078</td><td>0.659</td><td>0.065</td><td>0.676</td><td>0.067</td><td>0.694</td><td>0.062</td><td>0.712</td><td>0.057</td>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td><td>0.662</td><td>0.064</td><td>0.663</td><td>0.058</td><td>0.676</td><td>0.053</td><td>0.713</td><td>0.048</td><td>0.714</td><td>0.043</td>
  </tr>
  <tr>
    <td>gp-IRT</td><td>0.589</td><td>0.046</td><td>0.620</td><td>0.046</td><td>0.662</td><td>0.036</td><td>0.681</td><td>0.036</td><td>0.695</td><td>0.029</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td><td><b>0.711</b></td><td><b>0.031</b></td><td><b>0.737</b></td><td><b>0.029</b></td><td><b>0.756</b></td><td><b>0.028</b></td><td><b>0.766</b></td><td><u>0.027</u></td><td><u>0.773</u></td><td><u>0.027</u></td>
  </tr>
  <tr>
    <td rowspan="4">Hellaswag</td>
    <td>RANDOM</td><td>0.811</td><td>0.083</td><td>0.836</td><td>0.077</td><td>0.850</td><td>0.066</td><td>0.863</td><td>0.060</td><td>0.871</td><td>0.058</td>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td><td>0.860</td><td>0.060</td><td>0.880</td><td>0.061</td><td>0.877</td><td>0.067</td><td>0.897</td><td>0.059</td><td>0.898</td><td>0.057</td>
  </tr>
  <tr>
    <td>gp-IRT</td><td>0.724</td><td>0.062</td><td>0.776</td><td>0.053</td><td>0.810</td><td>0.043</td><td>0.827</td><td>0.038</td><td>0.849</td><td>0.032</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td><td><b>0.900</b></td><td><b>0.020</b></td><td><b>0.909</b></td><td><b>0.018</b></td><td><b>0.913</b></td><td><b>0.018</b></td><td><b>0.914</b></td><td><u>0.017</u></td><td><u>0.918</u></td><td><u>0.017</u></td>
  </tr>
  <tr>
    <td rowspan="4">GSM8K</td>
    <td>RANDOM</td><td>0.811</td><td>0.062</td><td>0.828</td><td>0.055</td><td>0.839</td><td>0.052</td><td>0.847</td><td>0.049</td><td>0.858</td><td>0.044</td>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td><td>0.786</td><td>0.087</td><td>0.791</td><td>0.079</td><td>0.796</td><td>0.073</td><td>0.800</td><td>0.071</td><td>0.799</td><td>0.071</td>
  </tr>
  <tr>
    <td>gp-IRT</td><td>0.787</td><td>0.055</td><td>0.807</td><td>0.047</td><td>0.829</td><td>0.041</td><td>0.842</td><td>0.038</td><td>0.858</td><td>0.034</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td><td><b>0.852</b></td><td><b>0.035</b></td><td><b>0.858</b></td><td><b>0.034</b></td><td><b>0.863</b></td><td><b>0.033</b></td><td><b>0.869</b></td><td><u>0.031</u></td><td><u>0.878</u></td><td><u>0.029</u></td>
  </tr>
  <tr>
    <td rowspan="4">Winogrande</td>
    <td>RANDOM</td><td>0.373</td><td>0.078</td><td>0.408</td><td>0.067</td><td>0.446</td><td>0.062</td><td>0.470</td><td>0.055</td><td>0.492</td><td>0.052</td>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td><td>0.472</td><td>0.086</td><td>0.487</td><td>0.085</td><td>0.514</td><td>0.075</td><td>0.521</td><td>0.087</td><td>0.518</td><td>0.073</td>
  </tr>
  <tr>
    <td>gp-IRT</td><td>0.263</td><td>0.041</td><td>0.313</td><td>0.038</td><td>0.353</td><td>0.038</td><td>0.392</td><td>0.036</td><td>0.419</td><td>0.034</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td><td><b>0.565</b></td><td><b>0.028</b></td><td><b>0.568</b></td><td><b>0.026</b></td><td><b>0.604</b></td><td><b>0.024</b></td><td><b>0.608</b></td><td><u>0.023</u></td><td><u>0.618</u></td><td><u>0.022</u></td>
  </tr>
  <tr>
    <td rowspan="4">POPE</td>
    <td>RANDOM</td><td>0.488</td><td>0.058</td><td>0.510</td><td>0.054</td><td>0.507</td><td>0.048</td><td>0.515</td><td>0.044</td><td>0.547</td><td>0.040</td>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td><td>0.474</td><td>0.040</td><td>0.483</td><td>0.038</td><td>0.518</td><td>0.034</td><td>0.547</td><td>0.033</td><td>0.556</td><td><u>0.031</u></td>
  </tr>
  <tr>
    <td>gp-IRT</td><td>0.481</td><td>0.038</td><td>0.470</td><td>0.037</td><td>0.462</td><td>0.036</td><td>0.482</td><td>0.034</td><td>0.477</td><td>0.033</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td><td><b>0.521</b></td><td><b>0.036</b></td><td><b>0.547</b></td><td><b>0.035</b></td><td><b>0.562</b></td><td><b>0.031</b></td><td><b>0.563</b></td><td><u>0.031</u></td><td><u>0.574</u></td><td><u>0.032</u></td>
  </tr>
</table>

Table 6: Results on all benchmarks. Values in bold represent the best results, while values that are underlined represent the second-best results.

<table>
  <tr>
    <th rowspan="2">Distances</th>
    <th colspan="2">ARC Challenge</th>
    <th colspan="2">Hellaswag</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">Winogrande</th>
    <th colspan="2">POPE</th>
  </tr>
  <tr>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th><th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>CORRELATION</td><td><b>0.766</b></td><td>0.033</td><td>0.903</td><td>0.019</td><td>0.828</td><td>0.041</td><td>0.557</td><td>0.029</td><td>0.547</td><td>0.038</td>
  </tr>
  <tr>
    <td>COSINE</td><td>0.746</td><td><u>0.031</u></td><td><u>0.914</u></td><td><u>0.019</u></td><td>0.827</td><td><u>0.040</u></td><td><b>0.616</b></td><td><b>0.024</b></td><td><b>0.577</b></td><td><b>0.024</b></td>
  </tr>
  <tr>
    <td>MANHATTAN</td><td>0.756</td><td><b>0.028</b></td><td>0.913</td><td><b>0.018</b></td><td><b>0.863</b></td><td><b>0.033</b></td><td>0.604</td><td>0.024</td><td>0.562</td><td>0.031</td>
  </tr>
</table>

Table 7: Detailed ablation results for distance selection across all benchmarks.

in Eq. 5) as the fixed number of native source models, we now interpret it as a lower bound—thereby including all source models whose prediction consistency exceeds the threshold.

Table 9 summarizes the results for all benchmarks under an inference count of 30. Notably, we observe improvements on the GSM8K and POPE datasets, while a slight decrease in performance is seen on other datasets.

These findings underscore that the core strength of our method lies in maximizing the consistency between the source and target models. When exactly \( n \) native source models are selected for each target model, performance appears to be near optimal. In contrast, adding additional native source models for certain target models may introduce performance variability. Consequently, we adopt a standardized number of native source models to ensure stability.

B.5 Impact of Fixed Medoids in N-set Construction on Performance

In the Developing N-set module, our Scalable K-Medoids Clustering algorithm employs the G-set examples as fixed (anchored) medoids. To assess the effectiveness of this design choice, we conducted an ablation study comparing our standard approach (with fixed G-set medoids) against a variant where the G-set points are allowed to change during medoid refinement. The results are summarized in Table 10.

When the N-set size is fixed at 30, our method with fixed medoids shows slightly inferior performance compared to the variant without fixed medoids (see rows 1 and 3 in Table 10). When the inference budget is fixed at 30, our method outperforms the variant without fixed medoids (see rows 1 and 2 in Table 10). These findings suggest that
<table>
  <tr>
    <th>Method Variants</th>
    <th colspan="2">ARC Challenge</th>
    <th colspan="2">Hellaswag</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">Winogrande</th>
    <th colspan="2">POPE</th>
  </tr>
  <tr>
    <th></th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>NON-CALIBRATED</td>
    <td>0.748</td>
    <td><b>0.026</b></td>
    <td>0.910</td>
    <td><b>0.017</b></td>
    <td>0.862</td>
    <td>0.036</td>
    <td>0.588</td>
    <td>0.028</td>
    <td>0.531</td>
    <td>0.043</td>
  </tr>
  <tr>
    <td>CALIBRATED</td>
    <td><b>0.756</b></td>
    <td>0.028</td>
    <td><b>0.913</b></td>
    <td>0.018</td>
    <td><b>0.863</b></td>
    <td><b>0.033</b></td>
    <td><b>0.604</b></td>
    <td><b>0.024</b></td>
    <td><b>0.562</b></td>
    <td><b>0.031</b></td>
  </tr>
</table>

Table 8: Detailed ablation results for calibrated performance estimation process across all benchmarks.

<table>
  <tr>
    <th>Method Variants</th>
    <th colspan="2">ARC Challenge</th>
    <th colspan="2">Hellaswag</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">Winogrande</th>
    <th colspan="2">POPE</th>
  </tr>
  <tr>
    <th></th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>DYNAMIC NATIVE SOURCE MODELS NUMBER</td>
    <td>0.741</td>
    <td>0.029</td>
    <td>0.909</td>
    <td><b>0.018</b></td>
    <td><b>0.875</b></td>
    <td><b>0.031</b></td>
    <td>0.592</td>
    <td>0.025</td>
    <td><b>0.604</b></td>
    <td><b>0.031</b></td>
  </tr>
  <tr>
    <td>STANDARDIZED NATIVE SOURCE MODELS NUMBER</td>
    <td><b>0.756</b></td>
    <td><b>0.028</b></td>
    <td><b>0.913</b></td>
    <td><b>0.018</b></td>
    <td>0.863</td>
    <td>0.033</td>
    <td><b>0.604</b></td>
    <td><b>0.024</b></td>
    <td>0.562</td>
    <td><b>0.031</b></td>
  </tr>
</table>

Table 9: Ablation Study on Dynamic vs. Standardized Native Source Model Selection.

anchoring the G-set as fixed medoids helps achieve a balanced trade-off between the size of the N-set and the available inference budget.

B.6 Intra-Family Affinity and Its Impact on Performance under Different Source-Target Model Similarity

We conducted an additional analysis on the GSM8K dataset to investigate whether models within the same family (e.g., Llama, Mistral) tend to select their own family members as native source models. As shown in Table 11, with a similar number of models from each family within the source and target model set, the results indicate a significant intra-family preference. On average, each llama-series model selected approximately 5.5 Mistral models and 6.4 Llama models as their native source models. Similarly, each Mistral-series model chose about 7.6 Mistral models and 3.0 Llama models on average. These findings suggest that models exhibit a bias toward source models with similar architectures, potentially due to shared representation spaces or analogous decision boundaries. This intra-family affinity may facilitate more accurate performance estimation, as the selected native source models can better capture the nuances and prediction patterns distinctive to their respective model lineages.

Furthermore, we conduct experiments on the GSM8K dataset to evaluate the performance of our method when the target models differ significantly from the source models. Specifically, by selecting only Llama series models as the target and using an inference count of 30, we compare the performance of all methods across two sets of source models: one that includes Llama series models and one that does not, with each set comprising an equal number of models. As shown in Table 12, the performance of all methods is closely correlated with the similarity between the target and source models; when these models differ, performance declines across all methods. Nonetheless, our method consistently outperforms the baselines regardless of the target–source model similarity, underscoring the generalizability of our approach.

B.7 Comprehensive G-set Size Evaluation Across Benchmarks

In this section, we present a comprehensive evaluation of how varying G-set sizes affect our method’s performance across multiple benchmarks. Table 13 reports Kendall’s \( \tau \) and MAE metrics for G-set sizes ranging from 5 to 25 for each benchmark while fixing the N-set size as 30. These results provide deeper insights into selecting the optimal G-set size and support the conclusions drawn in the main text.

B.8 Demonstration of Method Effectiveness with Variance

In this section, we present visual comparisons of our method and other approaches, including their respective variances, as illustrated in Figures 5 to 9. The results demonstrate that our method outperforms the baseline methods on all datasets and exhibits greater robustness (with smaller variance).
<table>
  <tr>
    <th>Method Variants</th>
    <th>|G-set|</th>
    <th>|N-set|</th>
    <th>Inference counts</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>FIXED G-SET</td>
    <td>10</td>
    <td>30</td>
    <td>30</td>
    <td>0.740</td>
    <td>0.027</td>
  </tr>
  <tr>
    <td>NOT-FIXED G-SET</td>
    <td>10</td>
    <td>20</td>
    <td>30</td>
    <td>0.719</td>
    <td>0.031</td>
  </tr>
  <tr>
    <td>NOT-FIXED G-SET</td>
    <td>10</td>
    <td>30</td>
    <td>40</td>
    <td><b>0.745</b></td>
    <td><b>0.027</b></td>
  </tr>
</table>

Table 10: Performance Comparison of N-set Construction Methods with and without Fixed G-set Medoids.

<table>
  <tr>
    <th>Model Family</th>
    <th>Avg. Selected Mistral Models</th>
    <th>Avg. Selected Llama Models</th>
  </tr>
  <tr>
    <td>LLAMA</td>
    <td>5.5</td>
    <td>6.4</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>7.6</td>
    <td>3.0</td>
  </tr>
</table>

Table 11: Statistics of native source model selection within model families on GSM8K benchmark.

<table>
  <tr>
    <th>Source Models Composition</th>
    <th>w/o Llama</th>
    <th>with Llama</th>
  </tr>
  <tr>
    <th></th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>ANCHORPOINTS</td>
    <td>0.388</td>
    <td>0.048</td>
    <td>0.525</td>
    <td>0.050</td>
  </tr>
  <tr>
    <td>GP-IRT</td>
    <td>0.505</td>
    <td>0.064</td>
    <td>0.526</td>
    <td>0.038</td>
  </tr>
  <tr>
    <td>TAILOREDBENCH</td>
    <td><b>0.634</b></td>
    <td><b>0.031</b></td>
    <td><b>0.704</b></td>
    <td><b>0.022</b></td>
  </tr>
</table>

Table 12: Methods’ Performance under Different Source-Target Model Similarity.

B.9 More Analyses On the Impact of Native Source Model Quantity on Our Method

In this section, we maintain the overall prediction consistency between the native source models and the target models constant, while varying the proportion of the source models designated as native source models from 20% to 100% for the target models across various benchmarks. The results are illustrated in Figures 10 to 14, indicating that, under the condition of maintaining the prediction consistency between the native source models and the target model, the number of native source models significantly influences the method’s performance.

B.10 More Analyses On the Impact of Native Source Models’ Prediction Consistency on Our Method

We conduct ablation studies by selecting native source models based on their prediction consistency with the target model across various benchmarks, ranging from the top 20% to the 80%–100% range. The results, presented in Figures 15 to 19, indicate that the performance of the method significantly declines as the prediction consistency between the native source models and the target model decreases, under the condition of keeping the number of native source models constant.

B.11 Extended Results on Optimal Native Source Model Selection

This section presents the results of our method as the number of native source models is incrementally increased based on their prediction consistency with the target model. The results in Figures 20 to 23 show that, overall, Kendall’s \( \tau \) initially increases and then decreases as the number of native source models increases, while the MAE initially decreases and then increases with the increase in the number of native source models.

Moreover, Our method adaptively selects 45 native source models for the ARC Challenge benchmark, 40 for the Hellaswag benchmark, 33 for the Winogrande benchmark, and 35 for the POPE benchmark. These selections represent near-optimal numbers of native source models, as demonstrated in Figures 20 to 23.

C Models Used in Our Experiments

Tables 14, 15, 16, 17 provide comprehensive lists of models corresponding to each benchmark.
<table>
  <tr>
    <th rowspan="2">|G-set|</th>
    <th colspan="2">ARC Challenge</th>
    <th colspan="2">Hellaswag</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">Winogrande</th>
    <th colspan="2">POPE</th>
  </tr>
  <tr>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
    <th>\( \tau \uparrow \)</th>
    <th>MAE \( \downarrow \)</th>
  </tr>
  <tr>
    <td>5</td>
    <td>0.719</td>
    <td>0.031</td>
    <td>0.912</td>
    <td>0.019</td>
    <td><b>0.865</b></td>
    <td>0.035</td>
    <td>0.624</td>
    <td>0.026</td>
    <td>0.549</td>
    <td>0.037</td>
  </tr>
  <tr>
    <td>10</td>
    <td><b>0.756</b></td>
    <td><b>0.028</b></td>
    <td><b>0.913</b></td>
    <td><b>0.018</b></td>
    <td><b>0.863</b></td>
    <td><b>0.033</b></td>
    <td>0.604</td>
    <td><b>0.024</b></td>
    <td><b>0.562</b></td>
    <td><b>0.031</b></td>
  </tr>
  <tr>
    <td>15</td>
    <td>0.751</td>
    <td>0.029</td>
    <td>0.911</td>
    <td>0.018</td>
    <td>0.854</td>
    <td>0.034</td>
    <td>0.608</td>
    <td>0.025</td>
    <td>0.556</td>
    <td>0.033</td>
  </tr>
  <tr>
    <td>20</td>
    <td>0.740</td>
    <td>0.029</td>
    <td>0.910</td>
    <td>0.018</td>
    <td>0.862</td>
    <td>0.034</td>
    <td>0.621</td>
    <td>0.026</td>
    <td>0.541</td>
    <td>0.034</td>
  </tr>
  <tr>
    <td>25</td>
    <td>0.725</td>
    <td>0.030</td>
    <td>0.909</td>
    <td>0.019</td>
    <td>0.851</td>
    <td>0.036</td>
    <td><b>0.638</b></td>
    <td>0.026</td>
    <td>0.533</td>
    <td>0.036</td>
  </tr>
</table>

Table 13: Detailed results for G-set size across all benchmarks.

Benchmark | Model Names

ARC Challenge | Qwen2-72B-Instruct, Meta-Llama-3-70B-Instruct, Qwen2-72B, zephyr-orpo-141b-A35b-v0.1, Phi-3-medium-4k-instruct, Yi-1.5-34B-Chat, c4ai-command-r-plus, Qwen1.5-110B, Smaug-72B-v0.1, Qwen1.5-110B-Chat, Yi-1.5-9B-Chat, Qwen1.5-32B-Chat, Nous-Hermes-2-Mixtral-8x7B-DPO, deepseek-llm-67b-chat, Qwen1.5-32B, Yi-1.5-34B-32K, Meta-Llama-3-70B, Phi-3-mini-4k-instruct, mixtral-8x22B-v0.3, Mixtral-8x22B-v0.1, Phi-3-mini-128k-instruct, Yi-1.5-34B, c4ai-command-r-v01, Qwen2-7B-Instruct, Hermes-2-Theta-Llama-3-8B, yaia-23-35B, Mixtral-8x7B-Instruct-v0.1, notux-8x7b-v1, Meta-Llama-3-8B-Instruct, Yi-34B-Chat, Smaug-34B-v0.1, Qwen2-7B, Nous-Hermes-2-SOLAR-10.7B, K2-Chat, Yi-1.5-9B-Chat-16K, Llama-3-Refueled, WizardLM-70B-V1.0, Yi-34B, Yi-1.5-6B-Chat, NeuralDaredevil-8B-obliterated, Yi-1.5-9B, Nous-Hermes-2-Mixtral-8x7B-SFT, Hermes-2-Pro-Mistral-7B, Hermes-2-Pro-Llama-3-8B, openchat_3.5, neural-chat-7b-v3-2, OpenHermes-2-Mistral-7B, OpenHermes-2.5-Mistral-7B, Qwen1.5-14B-Chat, Nous-Hermes-2-Mistral-7B-DPO, neural-chat-7b-v3-1, Starling-LM-7B-alpha, Qwen1.5-14B, neural-chat-7b-v3-3, Yi-34B-200K, SOLAR-10.7B-Instruct-v1.0, Yi-1.5-9B-32K, Mixtral-8x7B-v0.1, Mistral-7B-Instruct-v0.3, zephyr-7b-alpha, Mistral-7B-Instruct-v0.2, dolphin-2.9-llama-38B, Llama-2-70b-hf, Orca-2-13b, Llama-3-8B-Instruct-Gradient-104B, neural-chat-7b-v3, zephyr-7b-beta, Mistral-7B-OpenOrca, Yi-9B, Yi-9B-200K, DeciLM-7B-instruct, gemma-1.1-7b-it, SOLAR-10.7B-v1.0, merlinite-7b, Qwen1.5-7B-Chat, 14B, Yi-1.5-6B, stablelm-2-12b-chat, aya-23-8B, zephyr-7b-gemma-v0.1, Yarn-Solar-10b-32k, phi-2, phixtral-2x2_8, gemma-7b, Qwen1.5-7B, WizardLM-13B-V1.2, LLaMA-Pro-8B-Instruct, Yarn-Solar-10b-64k, DeciLM-7B, OrpoLlama-3-8B, Qwen1.5-MoE-A2.7B-Chat, deepseek-llm-7b-chat, Mistral-7B-v0.1, CollectiveCognition-v1.1-Mistral-7B, Mistral_Pro_8B_v0.1, Mistral-7B-v0.3, Orca-2-7b, Mistral-7B-v0.2, Yi-6B-Chat, Qwen2-1.5B-Instruct, stablelm-2_12b, openchat_v3.2, falcon-11B, Yi-6B, Mistral-7B-Instruct-v0.1, Yarn-Mistral-7b-64k, Meta-Llama-3-8B, Yarn-Mistral-7b-128k, gemma-7b-it, openchat_v3.2_super, Llama-2-70b-chat-hf, Qwen1.5-MoE-A2.7B, stablelm-zephyr-3b, Qwen1.5-4B-Chat, starcoders-12b, OpenHermes-13B, MetaMath-Mistral-Pro, Yi-6B-200K, falcon-40b, Qwen1.5-4B, Llama-2-13b-chat-hf, Llama-2-13b-hf, vicuna-7b-v1.5, OLMo-7B-Instruct-hf, internlm2-chat-1_8b, falcon-40b-instruct, Qwen2-1.5B, deepseek-moe-16b-chat, OpenHermes-7B, Llama-2-7b-chat-hf, Nous-Hermes-llama-2-7b, stablelm-2-zephyr-1_6b, Qwen1.5-1.8B, Qwen1.5-1.8B-Chat, LLaMA-Pro-8B, Llama-2-7b-hf, stablelm-2-1_6b-chat, internlm2-1_8b, Yarn-Llama-2-13b-128k, NexusRaven-V2-13B, starcoders-2-7b, Llama-2-7B-32K-Instruct, deepseek-llm-7b-base, recurrentgemma-2b-it, gemma-1.1-2b-it, granite-7b-base, deepseek-moe-16b-base, gemma-2b, stablelm-3b-4e1t, gemma-2b-it, Yarn-Llama-2-7b-64k, Qwen2-0.5B, phi-1_5

Table 14: Models used for ARC Challenge benchmark.
<table>
  <tr>
    <th>Benchmark</th>
    <th>Model Names</th>
  </tr>
  <tr>
    <td>HellaSwag</td>
    <td>LLaMAntino-3-ANITA-8B-Inst-DPO-ITA, luxia-21.4b-alignment-v1.0, UNA-ThePitbull-21.4-v1, T3Q-ko-solar-dpo-v6.0, MultiVerse_70B, RoleBeagle-11B, Capricorn-7B-DPO, Tess-2.0-Llama-3-70B, Truthful_DPO_MOE_19B, multimaster-7b-v5, guanaco-65B-HF, FusionNet_34Bx2_MoE_v0.1, Mixtral-8x7B-v0.1, Evangelion-7B, Lumina-5.5-Instruct, Mistral-Hermes-2x7b, Bagel-Hermes-2x34b, shqiponja-15b-v1, CollectiveCognition-v1.1-Mistral-7B-dare-0.85, etri-ones-solar, mpt-30b-instruct, openbuddy-mixtral-7bx8-v18.1-32k, bagel-dpo-7b-v0.4, OpenHermes-2.5-Mistral-7B, NeuralHermes-2.5-Mistral-7B, dolphin-2.1-mistral-7b-snr-math-laser, NeuralHermes-2.5-Mistral-7B, openbuddy-qwen1.5-32b-v21.1-32k, intenlm2-20b-llama, Matter-0.2-7B-DPO, airoboras-13b-gpt4-1.2, L3-SnowStorm-v1.15-4x8B-B, Pallas-0.5-LASER-0.6, BgGPT-7B-Instruct-v0.1, Seagull-llama-3-8B-orpo-v0.5, vigogne-7b-instruct, Llama-2-7b-chat-hf-activity-fine-tuned-v4, Llama-2-7b-chat-hf-activity-fine-tuned-v3, vicuna-class-tutor-7b-ep3, Llama-2-7b-chat-hf-afr-200step-flan-v2, llama3-8b-instruct-align-test1-kto, MFANN3bv0.7, openbuddy-yii1.5-9b-v21.1-32k, openbuddy-mixtral-7bx8-v17.1-32k, odia_llama2_7B_base, MT7Bi-alpha-dpo-v0.2, llama-shishya-7b-ep3-v2, Instruct_Yi-6B Dolly15K, Gaja-v2.00-dpo, phi-2.0OpenHermes-2.5, lion-gemma-7b-cn-v2, ToRoLaMa-7b-v1.0, gogpt-7b, Amber, open_llama_3b_v2, openllama_3b_EvoInstruct_lora_merged, gemma-7B-it-firefly, Qwen1.5-4B, google-gemma-7b-it-dpo-v1, openhermes-2b-gemma-sf-qrlora, RedPajama-INCITE-Chat-3B-v1, mistral_v1, gpt-j-6b, GPT-J-Pyg_PPO-6B, ScarletPajama-3B-HF, LLama2 7B-Structural-Prune-1.5x, illuni-llama-2-ko-7b-test, RedPajama-INCITE-Chat-3B-Share-GPT-11K, RedPajama-INCITE-Base-3B-v1, Guanaco-3B-Unlicensed-v2-GPTQ, glave-coder-7b, xgflm-7.5B, gpt-sw3-6.7b, cisco-lNAM-1.1B, pythia-2.7B, qd-phi-1_5, pythia-2.8b-deduped, LLmRa-2.7B, TinyLlama-1.3B-Cinder-Reason-Test-2, TinyPoliticalLlama-1.1B, Galpaca-30b-MiniOrca, finetune_test_qwen15-1-8b-sft-lora, TinyLlama-1.1B-Chat-v0.3, TinyLlama-1.1B-Chat-v0.1, CroissantLLMBase, pygmalion-2.7b, blossom-v2-3b, falcon_1b_stage3, MiniMerlin-3b-v0.1, DPO-miniguacno-1.5T, CodeQwen1.5-7B-Chat, yayi2-30b-llama, rho-math-1b-v0.1, LLmRa-1.3B V2, TinyLlama-1.1B-intermediate-step-480k-1T, gemma-2b-ko-dev-pbmt192, gpt2-chatbot, CodeLlama-7b-Python-hf, Deita-500m, TinyWand-SFT, tinyllama-coder-py-v13, d-Qwen1.5-1.8B, TinyLlama-1.1B-intermediate-step-240k-503b, dlite-v1-1.5b, pythia-1b-deduped, gpt2-large, WizardCoder-Guanaco-15B-V1.0, Qwen1.5-0.5B-vortex-v2, Sailor-0.5B-Chat, WizardCoder-Guanaco-15B-V1.1, Alpaca_refine_gpt2_e1_se0, deepseek-coder-1.3b-chat, speechless-coder-ds-1.3b, Instruct_GPT, deepseek-coder-1.3b-chat-and-function-calling, megatron-gpt2-345m, starcodersbase-3b, dlite-v1-355m, gov-qna-ko-merged, SSH_355M, CodeLlama-34b-Instruct-hf, CodeLlama-34b-Instruct-fp16, mptk-1b, KoAlpaca-Polyglot-5.8B, Llama-160M-Chat-v1, llama-160m, CodeLlama-34b-hf, KoAlpaca-KoRWKV-6B, Quokka_590m, pruned-yi-3b-prerlease-ckpt01, gpt2_test, finetuned-gpt2-tiny, Kaori-34b-v2, kaori-34b-v4, tiny_starcoder_py, GPT-2-Large-51k-steps, DialoGPT-small, test_mistral2, pythia-31m-KL_v1-2048-scratch</td>
  </tr>
</table>

Table 15: Models used for Hellaswag benchmark.
Benchmark & Model Names

GSM8K & Winogrande
ExtremeDolphin-MoE, Mistral-7B-Instruct-v0.2-sparsity-20, PiVoT-SUS-RP, polyglot-math-4x7b, SOLAR-10B-Necter-DPO-Javade, Starling-LM-1B-alpha, NeuralPipe-7B-slerp, oswald-7b, MistralTrixTest, Sensualize-Mistral-bf16, Sensualize-Solar-10.7B, FusionNet-passthrough, finance-chat, Kunoichi-7B, dolphin-2.2.1-mistral-7b, CarbonVillain-en-10.7B-v3, xDAN-SlimOrca, Mistral-11B-v0.1, dm7b_sft_gpt88w_merge, Loyal-Macaroni-Maid-7B, Yi-34B-200K-DARE-merge-v5, WinterGoddess-1.4x-70B-L2, vicuna-class-shishya-ac-hal-13b-ep3, Kaori-34B-v1, mistral-megamegare-dare-7b, Chupacabra-8x7B-MoE, bagel-7b-v0.1, Mixtral-8x7B-v0.1, openbuddy-deepseek-67b-v15-base, Falkor-7b, synapseMllm-7b-mistral-v0.4-preview3, llama2-13b-ft-openmln-leaderboard-v1, synapseMllm-7b-mistral-v0.3-preview, Tess-M-v1.3, monika-ddlc-7b-v1, speechless-mistral-7b-dare-0.85, mistral-7b-v0.1-layla-v1, Mistral-v0.1-PeanutButter-v0.0.2-7B, chronos-70b-v2, L2-7b-Beluga-WVG-Test, llama-2-13b-FINETUNE_3.3w-r8_gate_up_down, airoboras-c34b-2.2.1, llama-2-13b-FINETUNE4_3.8w-r8-q_k_v_o, llama-2-13b-FINETUNE3_3.3w-r16-gate_up_down, MLewd-Chat-v2-13B, Mistral-7B-v0.1-Open-Platypus, llama-2-13b-FINETUNE1_17w-r4, EverythingLM-13B-V3-peft, Llama2-7B-guanaco-1k, llama-2-13b-FINETUNE4_3.8w-r8-q_k_v_o_gate_up_down, Koss-7B-chat, ReMM-v2.2-L2-13B, WizardLM-1.0-Uncensored-CodeLlama-34b, airoboras-13b, airoboras-7b-gpt4-1.4-1.qlora, Wizard-Vicuna-7B-Uncensored-HF, Luban-Platypus2-13B-QLora-0.80-epoch, CodeLlama-34b-hf, airoboras-33b-gpt4-m2.0, llama-2-22b-blocktriangular, GPT-JT-6B-v0., llama2-70b-qaest-sft-v10, vigogne-7b-instruct, based-30b, mpt-30b-chat, qCammel-70x, GiftedConv13bLoraNoEcons4, llama-2-13b-platypus-vicuna-wizard, GOAT-7B-Community, genz-13b-v2, chronolima-airo-grad-12-13B, Vicuna-13B-CoT, Llama-2-7b-ft-instruct-es, OpenOrca-Preview-1-13B, Tulpar-7b-v0., zephyr-7b-sft-full, Mixtral-Orca-v0.1, Marcoroni-7b-DPO-Merge, Aquila2-34B, SOLAR-10.7B-Instruct-v1.0-128k, dolphin-2.6-mistral-7b-dpo-orca-v3, flux-7b-v0.1, Turdus, A0110, yayi2-30b-llama, NeuralMarcoroi14-7B, Deacon-34b-Adapter, test0, Pallas-0.5-LASER-0.4, Marcoroi14-7B-ties, Antares-11b-v1, CodegebraGPT-10b, Mistral-Syndicate-7B, Nous-Hermes-2-Yi-34B, Half-NSFW_Normmaid-7b, neural-chat-7b-v3-3-wizardmath-dare-me, apricot-wildflower-20, SauerkrautLM-UNA-SOLAR-Instruct, kalomaze-stuff, Walter-Mistral-7B, Starling-LM-alpha-8x7B-MoE, una-neural-chat-v3-3-P2-OMA, Dans-07YahooAnswers-7b, Chupacabra-7b-v2.03, PlatYi-34B-200K-Q, chinese-alpaca-2-13b-16k, ALMA-7B-Ja-V2, speechless-code-mistral-7b-v2.0, Mistral7B_adaptor_v1, notus-7b-v1, Chupacabra-7B-v2, SciPhi-Self-RAG-Mistral-7B-32k, Ferret-7B, llama-2-13B-instructed, glaive-coder-7b, Mistralic-7B-1, kuchiki-12-7b, Ilama_7b_Jora, Slerpeno, Llama2-7b-opencora-mc-v2-7b, CAMEL-13B-Role-Playing-Data, starchat-beta, testmodel2, Huginn-13b-v1.2, Dans_AdventurousWinds-7b, Wizard-Vicuna-13B_Uncensored-HF, Llama-2-13b-hf-ds_wiki_1024_full_r_64_alpha_16_merged, Emerald-13B, koala-13B-HF, tulu7B-fp16, airoboras-c34b-2.1, airoboras-7b-gpt4-1.1, 13B-Chimera, Nous-Hermes-Platypus2-13B-QLORA-0.80-epoch, airoboras-12-7b-gpt4-m2.0, llama-7b, llama-65b-instruct, Flash-Llama-7B, StableBeluga-13B, huginn1v1, llama_13b_sharegpt94k_fastchat, CAMEL-13B-Combined-Data, MelangeC-70b, chronos-13b-v2, stack-llama-2, CodeLlama-34b-Python-hf, UltraLM-65b, Platypus-30B, bimoGPT-llama2-13b, test-llama2-7b

Table 16: Models used for GSM8K and Winogrande benchmark.

Benchmark & Model Names

POPE
InternVL2-76B, paligemma-3b-mix-448, InternVL-Chat-V1-5, cambrian_13b, cogvlm-chat, CloudWalk, Ovis1.5-Gemma2-9B, cambrian_8b, InternVL2-26B, Ovis1.5-Llama3-8B, llava_next_vicuna_13b, glm-4v-9b, emu2_chat, llava_next_mistral_7b, llava_next_vicuna_7b, WeMM, cambrian_34b, llava_next_llama3, 360VL-70B, Bunny-llama3-8B, GLM4V, MiniCPM-V-2, llava_next_qwen_32b, Yi_Vision, InternVL2-2B, GeminiPro1-5, InternVL2-8B, llava_next_interleave_7b_dpo, XComposer2d5, MiniCPM-V-2_6, Mini-InternVL-Chat-2B-V1-5, cogvlm2-llama3-chat-19B, llava_next_yi_34b, Step1V, InternVL2-1B, InternVL2-4B, Phi-3-Vision, llava_next_interleave_7b, monkey-chat, OmniLMM_12B, InternVL2-40B, idefics_2_8b, deepseek_v1_7b, GPT4o_20240806, sharecaption, monkey, llava-v1.5-7b-xtuner, GPT4o_HIGH, RekaEdge, GPT4o, Mantis-8B-Idefics2, MiniCPM-Llama3-V-2_5, llava-llama-3-8b, sharegpt4v_/7b, Mini-InternVL-Chat-4B-V1-5, llava-interlim-7b, llava-v1.5-13b-xtuner, sharegpt4v_/13b, llava_v1.5_7b, GPT4o_MINI, deepseek_v1_13b, RekaFlash, llava_v1.5_13b, Mantis-8B-siglip-llama3, MiniCPM-V_QwenVLLPlus, Mantis-8B-clip-llama3, Yi_VL_6B, llava-interlim2-20b, XComposer2_1.8b, mPLUG-Owl2, GPT4v_Yi_VL_34B, llava-interlim2-7b, Claude3-5V_Sonnet, MMAlaya, instructblg_7b, XComposer2, XComposer2_POPE, TEST, TransCore_M, Claude3V_Haiku, Claude3V_Sonnet, Claude3V_Opus, idefics_9b_instruct, chameleon_30b, QwenVLMx, qwen_chat, llava_v1_7b, PandaGPT_13B, qwen_base, XComposer, MiniGPT-4-v1-7B, VisualGLM_6b, flamingov2, MiniGPT-4-v2, VX-VERSE, idefics_80b_instruct, chameleon_7b, XComposer2_4KHD

Table 17: Models used for POPE benchmark.
Figure 5: Demonstration of method effectiveness with variance on ARC Challenge benchmark.

Figure 6: Demonstration of method effectiveness with variance on Hellaswag benchmark.

Figure 7: Demonstration of method effectiveness with variance on GSM8K benchmark.
Figure 8: Demonstration of method effectiveness with variance on winogrande benchmark.

Figure 9: Demonstration of method effectiveness with variance on POPE benchmark.

Figure 10: The impact of the quantity of Native Source Models (with prediction consistency kept the same) on ARC Challenge benchmark.
Figure 11: The impact of the quantity of Native Source Models (with prediction consistency kept the same) on Hellaswag benchmark.

Figure 12: The impact of the quantity of Native Source Models (with prediction consistency kept the same) on GSM8K benchmark.

Figure 13: The impact of the quantity of Native Source Models (with prediction consistency kept the same) on Winogrande benchmark.
Figure 14: The impact of the quantity of Native Source Models (with prediction consistency kept the same) on POPE benchmark.

Figure 15: The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same) on ARC Challenge benchmark.

Figure 16: The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same) on Hellaswag benchmark.
Figure 17: The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same) on GSM8K benchmark.

Figure 18: The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same) on Winogrande benchmark.

Figure 19: The impact of prediction consistency between the Native Source Model and Target Model (with quantity kept the same) on POPE benchmark.
Figure 20: Performance of TAILOREDBENCH with varying numbers of Native Source Models on ARC Challenge benchmark.

Figure 21: Performance of TAILOREDBENCH with varying numbers of Native Source Models on Hellaswag benchmark.

Figure 22: Performance of TAILOREDBENCH with varying numbers of Native Source Models on Winogrande benchmark.
Figure 23: Performance of TAILOREDBENCH with varying numbers of Native Source Models on POPE benchmark.

Reviewer paper summaries: - The authors present a paper that proposes TailoredBench - a methodology that constructs a model-specific evaluation coreset for accurate and efficient evaluation.
- The idea behind the paper: NLP/Multimodal benchmarks are getting very expensive and time-consuming to run and new techniques have emerged that allow researchers to iterate quickly by building a coreset of points that best approximate the benchmark. TailoredBench is one such approach that outperforms the previous techniques.
- Methodology:
  - Given: Benchmark D, Source models set S, Target models set T
  - Source models are models for which we have predictions for all the data points in the benchmark.
  - Step 1: Create a global coreset (G-set) using K-Medoids clustering over the embeddings of the benchmark. Each data point is embedded, and the embedding vector is the correctness value predicted by each of the source models.
  - Step 2: For each target model, select a set of source models (native source models) from the global set of source models that best aligns with the predictions of the given target model.
  - Step 3: Create a native coreset (N-set) using a scalable K-Medoids clustering algorithm (proposed in this paper) where part of the N-set is G-set which are fixed medoids, and the rest of the medoids are chosen during the clustering.
  - Step 4: For each data point in the N-set cluster, calibrate a scale factor with respect to its medoid. Use this scale factor to calculate the estimated performance of the target model.
- TailoredBench outperforms other techniques over five benchmarks and 300+ models.
Summary of strengths: - The paper is well-motivated. The issue of benchmarks becoming increasingly expensive is likely to intensify in the future.
- The TailoredBench methodology is very well thought out and at the same time easy to implement. The results show that this methodology is superior to previous techniques in almost all the benchmarks and settings tried by the authors.
- The proposed approach is fairly generic and is something that can be applied to any kind of benchmark, i.e. not limited to certain modalities or types of datasets.
- The ablation studies and analysis were very comprehensive.
Summary of weaknesses: - The technique works great if there are source models that are compatible with the given target model. However, in real-world scenarios, users might want to know if no such models exist so that they can run the full benchmark for such target models. It would be great if the framework had scores, thresholds, error margins, and guidance around when not to use this estimation technique.
- Section 2.3 (Adaptive Native Source Model Selection): The authors claim that "standardizing the number of native source models across all target models ensures information richness". However, wouldn’t a dynamic number of models be better? If there are more source models that are compatible with a given target model, then why shouldn’t they be included in the estimation? We could have a lower bound on the prediction consistency and choose all models above it. If very few models are above it, then the framework warns the user that the estimates might not be reliable (see my point above). There should also be guardrails around how the number of models is currently picked - with the approach detailed in the paper, a user could end up with a very low number of N.
- Section 2.4 (Developing N-set): I would have liked to see some experiments where there are no fixed medoids and compare them against baselines based on an appropriate inference budget. It would have been nice to see an ablation on how much performance is lost by using the G-set as fixed medoids.
Comments, suggestions and typos: - Given that the paper is an extension of AnchorPoints, I would have liked a section describing it. The authors briefly cover it in the appendix, but I think AnchorPoints deserves some space in the main paper.
- I would have liked to see some more citations/intuition around the equations described in Section 2.3 and Section 2.5.
- Typos:
  - Figure 2 - (a) and (c) - x should be iterated from 1 to |D|, not 1 to |S|
  - L192 - extra space near “e.g.”
  - L362 - “Tables 1 present” should be “Table 1 presents”
Overall assessment scores: 3.5
Metareview score: 4
Metareview: This paper deals with the problem of efficiently running benchmarks with many LLMs. Specifically, the problem setup is one where full benchmark results are available for a set of source models, and the goal is to estimate the performance of a set of target models by running on a subset of the benchmark examples.

The authors propose a method that is tailored to each target model, such that the performance of each target model is estimated based on a slightly different example set. This method is then tested across various existing LLM benchmarks. The method includes construction of a global set of representative benchmark examples, using that set to understand which source models most resemble the prediction behavior each target model, and using this set of source models to expand the set of target examples used to estimate target model performance.
Summary of reasons to publish: * The paper deals with the broad and timely challenge of efficiently running benchmarks over multiple LLMs.
* Reviewers agree that the overall methodology is sound.
* Results demonstrate the effectiveness of the approach relative to prior works in generating more accurate performance estimates, over multiple benchmarks with large sets of models.
* The approach is novel with respect to prior works, in that it adds an adaptive model-specific component to existing coreset-based efficient evaluation approaches.