Active Evaluation Acquisition for Efficient LLM Benchmarking

## Abstract

Abstract As large language models (LLMs) become increasingly versatile, numerous large scale benchmarks have been developed to thoroughly assess their capabilities. These benchmarks typically consist of diverse datasets and prompts to evaluate different aspects of LLM performance. However, comprehensive evaluations on hundreds or thousands of prompts incur tremendous costs in terms of computation, money, and time. In this work, we investigate strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy. Our approach models the dependencies across test examples, allowing accurate prediction of the evaluation outcomes for the remaining examples based on the outcomes of the selected ones. Consequently, we only need to acquire the actual evaluation outcomes for the selected subset. We rigorously explore various subset selection policies and introduce a novel RL-based policy that leverages the captured dependencies. Empirical results demonstrate that our approach significantly reduces the number of evaluation prompts required while maintaining accurate performance estimates compared to previous methods.

## 1 Introduction

By scaling up parameters and pretraining data, large language models (LLMs) have demonstrated remarkable abilities to solve various tasks. To reliably evaluate these capabilities and compare different models, modern LLM benchmarks typically employ a comprehensive set of examples that focus on different aspects of performance. For instance, HELM (Liang et al., ), a widely used LLM benchmark covering diverse task families, includes 42 scenarios and approximately 600,000 queries. However, the comprehensiveness of these benchmarks inevitably incurs significant evaluation costs. For example, evaluating on the HELM benchmark requires 4,200 GPU hours for a 176B BLOOM model and $9,337 for text-davinci-002 API calls (Liang et al., ). Furthermore, these substantial costs hamper development at both the modeling and inference stages, preventing frequent evaluation during the former and extensive hyperparameter tuning – such as decoding and prompting strategies – during the latter.

In this work, we aim to improve evaluation efficiency by reducing the number of evaluation prompts. We first observe that evaluation prompts are highly correlated, meaning that a model’s (in)correct prediction on a certain prompt is likely to correspond with (in)correct predictions on related prompts. To leverage this, we build a model to formally capture the dependencies across prompts. This model can predict evaluation scores based on observed scores from a subset of prompts. Given these dependencies, our goal becomes identifying the minimal subset of prompts that can accurately recover the evaluation scores for the remaining prompts.

Instead of using a fixed subset of prompts across all models, we propose selecting a unique subset of prompts for each model to evaluate its performance more efficiently. The goal of our active evaluation acquisition (AEA) approach is to find the most informative subset of prompts for each model, allowing us to predict performance on the remaining prompts. The key insight is that models may have varying strengths; for example, one model may excel in arithmetic reasoning while another shows stronger commonsense reasoning. Tailoring the subset of prompts for each model ensures a more accurate and targeted evaluation of its capabilities. Furthermore, our dynamic acquisition process allows us to adapt in real time as evaluation scores are gathered. As the model’s performance on initial prompts is observed, the system adjusts subsequent prompt selections to better explore areas of uncertainty or confirm early findings. This iterative approach not only enhances the accuracy of performance estimation but also reduces redundancy by avoiding prompts that are likely to yield predictable results, thereby saving computational resources and time. Importantly, the final evaluation score is derived from both the acquired scores on selected prompts and predicted scores on the remaining prompts, ensuring comparability across models is maintained.

Our contributions are as follows: 1) We tackle LLM evaluation efficiency through dependency modeling and subset selection, connecting LLM evaluation with the extensive literature on subset selection. 2) We design a generative model that captures dependencies across evaluation prompts and handles mixed-type evaluation scores, including both discrete and real-valued scores. 3) We thoroughly test existing subset selection algorithms on several popular LLM evaluation benchmarks, including MMLU (Hendrycks et al., ), HELM (Liang et al., ), HuggingFace Open LLM Leaderboard (Beeching et al., ), AlpaceEval (Li et al., ), and Chatbot Arena (Zheng et al., ). 4) We develop several new subset selection policies based on the dependency model and demonstrate their superiority over existing methods, with our RL-based acquisition policy achieving the best performance using the lowest acquisition budget. 5) We propose and investigate the cold-start problem, where new prompts are added to a benchmark without prior evaluation scores for any model, and extend our RL-based policy to deal with the situation effectively.

## 2 Method

### 2.1 Problem Formulation

Consider a benchmark $X$ with $N$ prompts, $X=\{x_{n}\}_{n=1}^{N}$. Note that a benchmark may contain multiple datasets. Evaluating a model $m$ on this benchmark generates evaluation scores $Y_{m}=\{y_{mn}\}_{n=1}^{N}$. A leaderboard for this benchmark contains evaluation scores for $M$ models, denoted as $\{Y_{m}\}_{m=1}^{M}$. For a new model $m^{\prime}$ to be evaluated, our AEA framework will acquire a subset of the evaluation scores $Y_{m^{\prime}}^{(o)}=\{y_{m^{\prime}o}:o\subseteq\{1,\ldots,N\}\}$, and the evaluation scores for the remaining prompts $Y_{m^{\prime}}^{(u)}=\{y_{m^{\prime}u};u=\{1,\ldots,N\}\setminus o\}$ will be predicted. The key of our AEA framework is to capture the dependencies over prompts so that the predicted evaluation scores are accurate. We explicitly model the dependencies by learning the conditional distribution $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)$. Since the set of prompts to acquire their scores is not predefined, we must estimate $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)$ for all possible subsets $u$ and $o$.

Given the generative model between subsets of evaluation scores and a fixed budget $K$, our goal for AEA is to find an optimal subset $o^{*}\subseteq\{1,\ldots,N\}$, where $|o^{*}|=K$, such that the predicted scores on the remaining prompts are accurate, i.e.,

$$ $\textstyle o^{*}=\operatorname*{arg\,max}_{o\in\mathbb{P}([N],K)}p(Y_{m^{ \prime}}^{(u)}\mid Y_{m^{\prime}}^{(o)},X),$ (1) $$

where $\mathbb{P}([N],K)$ represents all subsets of $\{1,\ldots,N\}$ with cardinality $K$. Note that the optimal subset $o^{*}$ could be different for each model; however, for notation simplicity, we omit the subscript $m^{\prime}$. It is worth noting that the objective in equation 1 cannot be directly optimized since the values $Y_{m^{\prime}}^{(u)}$ for a test model $m^{\prime}$ are unknown before we actually acquire the evaluation scores on those prompts. Additionally, equation 1 bares similarity to the objective of active learning (Ren et al., ), but our goal differs in that we aim to achieve more accurate evaluations rather than training a better model. We will later assess several acquisition policies inspired by active learning algorithms.

### 2.2 Modeling Dependencies via Neural Processes

In this section, we aim to capture the dependencies across evaluation prompts by modeling the conditional distribution $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)$. We represent the relationship between the prompts and their evaluation scores as a stochastic process $F:\mathcal{X}\rightarrow\mathcal{Y}$, where $\mathcal{X}$ and $\mathcal{Y}$ denote the spaces of prompts and their corresponding evaluation scores, respectively. Evaluating on the benchmark $X$ can be interpreted as observing finite-dimensional marginal distributions of this stochastic process. Specifically, the evaluation scores $Y_{m}$ represent the function values $\{f_{m}(x_{n})\}_{n=1}^{N}$ for a particular function instantiation, $f_{m}$, sampled from the distribution of functions.

Neural Processes (NPs) (Garnelo et al., ; ; Kim et al., ) provide a flexible and scalable approach to modeling such stochastic processes. They combine the strengths of neural networks and Gaussian Processes to predict outputs for new inputs by conditioning on a set of context points. Specifically, the function $f_{m}$ is implicitly parameterized by a latent vector $z_{m}$, and the generative model then follows

$$ $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)=\int p(z_{m}\mid Y_{m}^{(o)},X)p(Y_{m}^{(u)} \mid z_{m},Y_{m}^{(o)},X)dz_{m}.$ (2) $$

Since the integration is over a high dimensional latent space, we instead optimize the evidence lower bound (ELBO) following variational autoencoder (VAE) (Kingma & Welling, )

$$ $\log p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)\geq\mathbb{E}_{q(z_{m}\mid Y_{m}^{(u)},Y _{m}^{(o)},X)}\left[\log\frac{p(Y_{m}^{(u)}\mid z_{m},Y_{m}^{(o)},X)p(z_{m} \mid Y_{m}^{(o)},X)}{q(z_{m}\mid Y_{m}^{(u)},Y_{m}^{(o)},X)}\right],$ (3) $$

where $q(z_{m}\mid Y_{m}^{(u)},Y_{m}^{(o)},X)$ and $p(z_{m}\mid Y_{m}^{(o)},X)$ represent the posterior and prior distributions over the latent variable, respectively.

To ensure that our model represents a valid stochastic process, we adhere to the conditions stated by the Kolmogorov Extension Theorem (Oksendal, ): (finite) exchangeability and consistency. The exchangeability condition requires that the joint distribution $p(Y_{m})$ remain unchanged under permutations of its elements. In practice, we can satisfy this condition by using permutation invariant networks to parameterize both the prior and posterior distributions. The consistency demands that if we marginalize out part of $Y_{m}$, the resulting marginal distribution is the same as that defined on the original prompt $x_{n}$. This condition is met when the approximate posterior equals the true posterior. In practice, we achieve this by training the model with a sufficient amount of data from diverse model evaluations so that the lower bound approaches the actual likelihood.

Implementation In order to handle textual prompts, we utilize a pretrained embedding model to represent each prompt as a $\mathbb{R}^{d}$ vector. During training, since the entire set $Y_{m}$ might be too large to fit into memory, we randomly sample two non-overlapping subsets from each model as $Y_{m}^{(o)}$ and $Y_{m}^{(u)}$, respectively. The prior and posterior distributions share the same network, but take different inputs. The prior takes in a set of x-y pairs from $Y_{m}^{(o)}$, i.e., $\{(x_{o},y_{mo}):o\subseteq\{1,\ldots,N\}\}$, while the posterior takes in a set of x-y pairs from both $Y_{m}^{(o)}$ and $Y_{m}^{(u)}$. Following the Attentive Neural Process (Kim et al., ), we implement the prior/posterior network using self-attention blocks to better capture the dependencies across set elements. To reduce memory usage, we use Set Transformer architecture (Lee et al., ), where each set element attends to a small set of learnable induced points instead of attending to all other elements directly. The decoder network $p(Y_{m}^{(u)}\mid z_{m},Y_{m}^{(o)},X)$ employs cross-attention, allowing each unobserved prompt to attend to the relevant observed prompts. According to De Finetti’s Theorem (De Finetti, ), the likelihood over set elements $Y_{m}^{(u)}$ can be conditionally independent conditioned on the latent variable $z_{m}$. However, we still use a Set Transformer (Lee et al., ) to better capture the dependencies. Please refer to Appendix A for details of the model architecture.

### 2.3 Evaluation Acquisition Policy

Given the generative model across subsets of evaluation prompts, we now develop acquisition policies to select an optimal subset of prompts for acquiring their true evaluation scores, while the remaining scores will be predicted by the conditional $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)$.

#### 2.3.1 Random Policy

A random acquisition policy selects a subset of size $K$ at random to acquire the evaluation scores. In this work, we consider two variants: Uniform Sampling and Stratified Random Sampling (Perlitz et al., ). Uniform Sampling selects $K$ prompts uniformly from $X$, while Stratified Random Sampling considers the size of different datasets and ensures each dataset is equally represented. The stratified sampling has been verified effective on HELM benchmark (Perlitz et al., ).

#### 2.3.2 Static Policy

A static acquisition policy determines the set of prompts to be evaluated beforehand, and each model to be evaluated acquires the evaluation scores on the same set of prompts. We assess the following two types of static policies.

Clustering Given the embedding for each prompt, we group them into $K$ cluster, then we select one prompt in each cluster that is closest to the cluster centroid. We denote this approach as Clustering-Embed. Instead of using the pretrained sentence embedding, we can use the learned embedding from an Item Response Theory (IRT) model (Hambleton & Swaminathan, ; Embretson & Reise, ), which represents the difficulty and discriminability of each prompt. The Clustering-IRT method, proposed in (Polo et al., ), has been successfully applied on several public LLM benchmarks. Inspired by (Vivek et al., ), which selects representative examples by clustering based on prediction confidence, the Clustering-Score method groups the prompts based on their evaluation scores on the training set. That is, each prompt $x_{n}$ is represented by a vector of evaluation scores, with the size of the vector corresponding to the number of evaluated models in the training set. The Clustering-Score method has been used as a baseline in (Polo et al., ).

Combinatorial Optimization Given the model $p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X)$, a static acquisition policy can be derived by searching over the training set to find the optimal subset of prompts that gives the most accurate prediction of the remaining prompts. This is a typical combinatorial optimization problem, which is NP-Hard. Here, we employ a sequential approach that selects one prompt at a time until $K$ prompts are selected. Starting from an empty set $o=\emptyset$, the next prompt $i\in u:=\{1,\ldots,N\}\setminus o$ is chosen to minimize the prediction error over the training set, i.e.,

$$ $i=\operatorname*{arg\,min}_{i^{\prime}\in u}\mathbb{E}_{Y_{m}\sim p_{\mathcal{ D}}}\mathbb{E}_{\hat{Y}_{m}^{(u^{\prime})}\sim p(Y_{m}^{(u^{\prime})}\mid Y_{m }^{(o^{\prime})},X)}\|\hat{Y}_{m}^{(u^{\prime})}-Y_{m}^{(u^{\prime})}\|^{2},$ (4) $$

where $o^{\prime}=o\cup\{i^{\prime}\}$ and $u^{\prime}=u\setminus\{i^{\prime}\}$. We estimate the expectation by Monte Carlo sampling. For notation simplicity, the above equation computes the mean squared error on prompts $u^{\prime}$; however, in practice, different datasets may use different metrics. Additionally, these differences may be weighted depending on the dataset size. Please refer to Algorithm 1 in Appendix for pseudo-code of the selection process. Note that this approach has a complexity of $O(KMN)$, which could be prohibitive when the benchmark is large.

Figure: Algorithm 1 Static Evaluation Acquisition via Combinatorial Optimization

#### 2.3.3 Dynamic Policy

Instead of acquiring the same set of evaluation scores for each model, we propose dynamically acquiring adaptive subsets for different models, a method we term Active Evaluation Acquisition (AEA). This approach tailors the selection of prompts to each model’s specific strengths and weaknesses, providing a more accurate and efficient evaluation. Dynamic acquisition sequentially acquires evaluation scores and simultaneously refines the uncertainty of predictions, enabling real-time adaptation based on observed performance. AEA reduces redundancy by avoiding predictable evaluations and focusing resources on the most informative prompts. Please refer to Algorithm 2 for pseudo-code of the active acquisition process.

Figure: Algorithm 2 Active Evaluation Acquisition

Uncertainty Sampling Inspired by uncertainty sampling method widely used in active learning literature(Ren et al., ; Yang et al., ; Raj & Bach, ), where the most uncertainty data point under the current predictor is chosen to query its label, we select the next prompt to be evaluated based on the uncertainty of $p(Y_{m}^{(i)}\mid Y_{m}^{(o)},X)$. Here, $o$ contains the evaluated prompts so far, and $i\in u$ is one of the candidate prompts to be selected. We choose the prompt with the highest entropy:

$$ $\textstyle i=\operatorname*{arg\,max}_{i\in u}H(Y_{m}^{(i)}\mid Y_{m}^{(o)},X).$ (5) $$

In practice, we estimate the entropy by sampling multiple times and computing the sample variance.

Information Gain Given the latent variable based neural process model (equation 2), where the latent variable essentially parameterizes the stochastic process, a straight-forward acquisition policy is to select the prompt that provides the most information about the latent variable $z_{m}$. We use the conditional mutual information to measure the amount of information:

$$ $\displaystyle i$ $\displaystyle=\operatorname*{arg\,max}_{i\in u}I(Y_{m}^{(i)};z_{m}\mid Y_{m}^{ (o)},X)$ (6) $\displaystyle=\operatorname*{arg\,max}_{i\in u}\left[H(z_{m}\mid Y_{m}^{(o)},X )-\mathbb{E}_{\hat{Y}_{m}^{(i)}\sim p(Y_{m}^{(i)}\mid Y_{m}^{(o)},X)}H(z_{m} \mid\hat{Y}_{m}^{(i)},Y_{m}^{(o)},X)\right]$ $\displaystyle=\operatorname*{arg\,min}_{i\in u}\mathbb{E}_{\hat{Y}_{m}^{(i)} \sim p(Y_{m}^{(i)}\mid Y_{m}^{(o)},X)}H(z_{m}\mid\hat{Y}_{m}^{(i)},Y_{m}^{(o)} ,X).$ $$

The third equation follows because the observed set $o$ is the same for any candidate $i\in u$. The expectation is again estimated by Monte Carlo sampling. Note that the entropy is estimated based on predicted $Y_{m}^{(i)}$ rather than the true evaluation score as the true score is unknown before acquisition. At each acquisition step, the entropy must be estimated for each candidate prompt $i\in u$. Therefore, the total complexity is $O(KN)$, which could be prohibitive for large benchmarks.

Reinforcement Learning The active acquisition process can be formulated as a Markov decision process (MDP), where the state consists of the currently evaluated prompts and their scores, and the action space contains the remaining prompts to be evaluated. To solve the MDP, a reinforcement learning agent sequentially acquires new evaluation scores based on the current state. After acquiring evaluation score for prompt $i$, the current state transitions to a new state as follows: $o\xrightarrow{i}o\cup\{i\},Y_{m}^{(o)}\xrightarrow{i}Y_{m}^{(o)}\cup\{Y_{m}^{(
i)}\}$. When the agent acquires evaluation scores for $K$ prompts, the acquisition process terminates, and the agent receives a reward based on the prediction accuracy for the remaining prompts. Note that the reward is only required during training when we have access to all the evaluation scores, allowing us to compute the actual prediction accuracy. During testing, the next prompt to be evaluated will be directly selected by the policy:

$$ $\textstyle i=\operatorname*{arg\,max}_{i\in u}P(i\mid Y_{m}^{(o)},X).$ (7) $$

Since the policy network has constant computational cost, the total complexity of the acquisition process remains $O(K)$ regardless of the benchmark size.

In the above MDP definition, the reward is received only at the end of the acquisition process by predicting the unobserved evaluation scores. This setup poses a typical temporal credit assignment problem, which complicates the learning of an effective agent, especially when the trajectory is long (Minsky, ; Sutton, ). To address this issue, we propose providing intermediate rewards for each acquisition action $i$. Specifically, after acquiring the evaluation score for prompt $i$, the improvement in prediction accuracy per unobserved prompt is used as the intermediate reward, i.e.,

$$ $r(o,i)=\frac{\mathbb{E}_{\hat{Y}_{m}^{(u)}\sim p(Y_{m}^{(u)}\mid Y_{m}^{(o)},X )}\|\hat{Y}_{m}^{(u)}-Y_{m}^{(u)}\|^{2}}{|u|}-\frac{\mathbb{E}_{\hat{Y}_{m}^{( u^{\prime})}\sim p(Y_{m}^{(u^{\prime})}\mid Y_{m}^{(o^{\prime})},X)}\|\hat{Y}_ {m}^{(u^{\prime})}-Y_{m}^{(u^{\prime})}\|^{2}}{|u^{\prime}|},$ (8) $$

where $o^{\prime}=o\cup\{i\}$ and $u^{\prime}=u\setminus\{i\}$. The intermediate reward provides immediate feedback for each acquisition action during the acquisition process, facilitating more effective learning. Note that the intermediate reward follows the potential function structure (Ng et al., ), therefore, it will not change the optimal policy.

In addition to providing intermediate rewards, we propose using the neural process to assist the agent with auxiliary information. Specifically, the neural process can predict the evaluation scores for unobserved prompts based on the observed scores in the current state. By sampling multiple times, the neural process can inform the agent about the uncertainties of these unobserved scores. The predicted scores and their uncertainties on the unobserved prompts allow the agent to anticipate future states and guide its exploration. For instance, if the neural process is very confident about the score of a currently unobserved prompt, then acquiring its real score would be redundant. The auxiliary information helps the agent make more informed decisions about which prompts to evaluate next, improving the efficiency and accuracy of the active acquisition process.

### 2.4 Cold Start Problem

So far, we have considered scenarios where the set of prompts is fixed for each benchmark. However, as language models advance, new capabilities may emerge that need to be assessed. Therefore, it is crucial to address the cold start problem, where the benchmark must be expanded with new prompts for which no evaluation scores are initially available for any model.

Expanding the benchmark with new prompts introduces several challenges. Firstly, predicting evaluation scores on these new prompts is difficult for a neural process trained on previously observed scores. Secondly, determining which new prompts to select for the expanded benchmark is challenging, as these new prompts may introduce capabilities or areas not well-represented in the original set, making it hard to gauge their relevance and difficulty relative to the existing prompts.

To help the neural process model generalize to new prompts, we introduce a semi-supervised training procedure where the new prompts are treated as unlabeled data. We found that simple pseudo-labeling approaches (Lee et al., ; Xie et al., ; Du et al., ) work well. Specifically, we add the new prompts and their predicted evaluation scores into the training process if the uncertainties of the predicted evaluation scores are below a predefined threshold. In our preliminary experiments, we also tested several regularization approaches, such as entropy minimization (Grandvalet & Bengio, ) and consistency regularization (Tarvainen & Valpola, ), which are commonly used for semi-supervised learning, but they consistently underperformed compared to pseudo-labeling. A systematic exploration of semi-supervised techniques for neural process training is beyond the scope of this paper and will be left for future work.

Static acquisition policies are suboptimal for the cold start problem because they rely on the available evaluation scores to determine the set of prompts to be evaluated, meaning the evaluation scores on new prompts will never be acquired. One exception is Clustering-Embed method, where the clustering is based solely on the embedding of the new prompts rather than their evaluation scores. In contrast, dynamic acquisition policies are better suited to handle the cold start problem due to their adaptive nature. However, since the RL-based acquisition policy is trained to acquire evaluation scores on the existing prompts, it requires the acquisition policy to generalize to new prompts.

To enable the policy network to generalize to new actions, we design it to incorporate the action representations into its inputs. Specifically, at each acquisition step, in addition to the acquired scores $Y_{m}^{(o)}$, the policy network $h$ also takes in the representations of the available actions, where the representations are shared with the neural process model. The output of the policy network is a vector with the same dimensionality as the action representations. The probability of selecting a particular prompt $i$ is proportional to the inner product of the output vector and action representations, i.e,

$$ $P(i\mid Y_{m}^{(o)},X)=\frac{e^{a_{i}\cdot h(Y_{m}^{(o)},X^{(o)},\{a_{i}\}_{i \in u})}}{\sum_{i\in u}e^{a_{i}\cdot h(Y_{m}^{(o)},X^{(o)},\{a_{i}\}_{i\in u}) }},$ (9) $$

where $\{a_{i}\}_{i\in u}$ denote the representations for the set of candidate prompts. A similar policy architecture design has been proposed before for sequential decision-making (Jain et al., ). Please refer to Appendix C for further details.

## 3 Related Works

Active Learning Active learning (Fu et al., ; Konyushkova et al., ; Yoo & Kweon, ) addresses the problem of having a learner select specific examples to query an oracle for their labels, with the goal of learning a better model using as few labeled examples as possible. In contrast, our proposed AEA framework focuses on evaluating a model with fewer examples to accurately predict the evaluation scores for the remaining examples.

Active Testing Active testing (Kossen et al., ) reduces the labeling cost by selectively choosing test points to label, ensuring sample-efficient model evaluation. While this aligns with the goal of efficient evaluation, our work specifically targets reducing the cost of running evaluations on a large number of prompts, rather than minimizing labeling costs.

Active Feature Acquisition Active feature acquisition involves acquiring features at a cost to predict a target variable. Prior works (Zubek & Dietterich, ; Rückstieß et al., ; Shim et al., ) formulate AFA as an MDP and develop various RL approaches to optimize the acquisition plan. Li & Oliva () further propose a model-based solution by leveraging a generative model (Li et al., ) to capture the AFA dynamics. Unlike previous AFA works, our approach focuses on acquiring evaluation scores from a benchmark, aiming to recover the full set of evaluation scores rather than predict a specific target variable.

Efficient LLM Benchmarking As LLMs continue to develop and scale, ongoing efforts aim to create benchmarks that comprehensively assess their capabilities. A notable trend in these benchmarks is their evolution from single-task assessments (Bowman et al., ; Rajpurkar et al., ) to multi-task benchmarks (Wang et al., ; ), and ultimately to massively multi-task evaluations (Srivastava et al., ; Liang et al., ; Hendrycks et al., ). The ever-increasing evaluation cost has encouraged researchers to develop efficient evaluation approaches. BIG-bench Lite (Srivastava et al., ) and BIG-bench Hard (Suzgun et al., ) evaluate on a subset of BIG-bench tasks, and Ye et al. () propose clustering BIG-bench tasks and selecting the examples that are closest to cluster centers. Perlitz et al. () found that the model rankings on HELM can be accurately obtained by evaluating only a fraction of the examples. Vivek et al. () propose clustering the evaluation examples based on the uncertainty of model predictions, while Polo et al. () suggest clustering examples based on learned features from an IRT model. In this work, we comprehensively assess these methods and further propose actively selecting evaluation examples.

## 4 Experiments

In this section, we assess various evaluation acquisition policies on several popular LLM benchmarks. We divide the available leaderboard scores into training and test splits. The training split is used to fit the neural processes model, capturing the dependencies across prompts. The acquisition policies are executed for each model in the test split to acquire the evaluation scores for a subset of prompts. The evaluation scores on the remaining prompts are predicted based on the corresponding neural process model. The final score for each benchmark is computed as a weighted average across datasets, and we report the absolute differences between the predicted scores and the actual scores. Please see Appendix D for details.

We conduct experiments on 5 popular LLM benchmarks:

- •
HuggingFace Open LLM Leaderboard (Beeching et al., ) consists of 6 datasets with a total of 28,659 prompts. Evaluation scores include both binary accuracy and real-valued probabilities. We collect evaluation scores for 2,084 models and select 1,000 models for training based on their evaluation date. The most recently evaluated models are used for testing, simulating the real-world scenario.
- •
MMLU (Hendrycks et al., ) contains 57 datasets with a total of 14,042 multiple choice QA problems on different subjects. Evaluation scores are all binary accuracy. We collect evaluation scores for the same models from the Open LLM Leaderboard.
- •
HELM-Lite (Liang et al., ) include 10 datasets (each possibly containing several sub-datasets) with a total of 13,021 prompts. Evaluation scores include both binary exact match scores and real-values metrics such as F1 and BLEU. We collect evaluation scores for 33 models and randomly select 23 models for training since the evaluation does not have dates.
- •
AlpacaEval 2.0 (Li et al., ) contains 805 prompts. For each model, the generations are compared to those of GPT-4 to compute the win rate. Although this benchmark is relatively small, it requires an expensive GPT-4 based judge, so reducing the number of API calls can significantly reduce the total evaluation cost. We collect evaluation scores for 130 models and randomly select 70% for training.
- •
Chatbot Arena (Zheng et al., ) is a popular human-annotated benchmark, where annotators interact with two anonymous models using the same prompts and declare a winner. We use the pairwise comparisons evaluated on the 80 MTBench prompts (Zheng et al., ). Although this benchmark is relatively small, human evaluation is expensive, so further reducing the evaluation prompts could lower costs. The annotations include comparisons over multiple turns, but we only use the annotations for the first turn here. Unlike other benchmarks where each model directly receives an evaluation score, this benchmark evaluates each pair from a set of 6 models. To create the train-test splits, we randomly select one of the six models, and all pairs that involve the selected model are included in the test split. Note that not all 80 prompts are annotated for each model pair. While our neural process model can handle missing data, the acquisition process must acquire the true score for any prompt the policy selects. To address the missing data during acquisition process, we use the trained neural process to predict the missing evaluation scores. We report win rate for this benchmark.

Figure: (a) AlpacaEval
Refer to caption: extracted/5902739/figures/alpaca_eval.png

Results Figure 1 presents the main experimental results on 5 LLM benchmarks. We conduct experiments with 3 random seeds for each benchmark and plot the average performance and standard deviation throughout the acquisition process. Prompt embeddings are obtained using the SFR embedding model (Rui Meng, ). For the static clustering based policies, since the selected prompts do not have an inherent order, the acquisition process shuffles the selected prompts at random. For the AlpacaEval and Chatbot Arena benchmarks, stratified random sampling is equivalent to uniform sampling since there are only one dataset in each benchmark. Combinatorial optimization is too expensive to run for HELM-Lite, HuggingFace Open LLM Leaderboard, and MMLU due to the large number of prompts. We found that uncertainty and information gain based policies consistently fail to explore the action space, leading to worse overall performance. To avoid cluttering the plots, results for uncertainty sampling and information gain based policies are moved to the appendix. Please refer to Appendix D for more analysis.

For all benchmarks, our proposed RL-based acquisition policy achieves the best performance with the lowest acquisition budget, demonstrating its superior ability to select informative prompts and accurately estimate benchmark performance. The stratified random sampling policy performs similarly to uniform sampling. Interestingly, the Clustering-Embed policy does not outperform the random selection, indicating that the similarity in prompt embedding does not always translate to the similarity in evaluation scores. Among the three clustering-based policies, none consistently outperforms the others. On AlpacaEval, HELM-Lite, and MMLU, the policies that utilize the evaluation scores (i.e., Claustering-Score and Clustering-IRT) perform better, while on the Open LLM Leaderboard and Chatbot Arena, Clustering-Embed perform better. The combinatorial optimization based policy does not perform well, even on the two small benchmarks where it is computationally feasible. We attribute this to a potential distribution shift between the models used for training and those used for testing, suggesting that the static policy optimized on training models does not generalize well to new models during testing.

**Table 1: Comparison of our RL-based acquisition policy with TinyBenchmarks (TB) (Polo et al., ), using selected prompts to predict evaluation scores with either the IRT model from TB or our NP model. The metric is the absolute error in benchmark score estimation.**
|  |  | IRT | NP |
| --- | --- | --- | --- |
| AlpacaEval | TB | $0.027\pm 0.002$ | $0.003\pm 0.001$ |
| (K=100) | RL | $\mathbf{0.014\pm 0.005}$ | $\mathbf{0.001\pm 0.000}$ |
| MMLU | TB | $\mathbf{0.022\pm 0.000}$ | $0.016\pm 0.000$ |
| (K=100) | RL | $0.028\pm 0.002$ | $\mathbf{0.013\pm 0.000}$ |
| Open LLM | TB | $0.023\pm 0.002$ | $0.022\pm 0.004$ |
| (K=200) | RL | $\mathbf{0.019\pm 0.001}$ | $\mathbf{0.018\pm 0.001}$ |

Additionally, we compare our methods with tinybenchmarks (Polo et al., ). Given the selected subsets from tinybenchmarks, we predict the final benchmark performance using both the IRT models provided by tinybenchmark (^1^11https://github.com/felipemaiapolo/tinyBenchmarks/tree/main) and our neural process models. Conversely, we also evaluate our prompt selections using both IRT and NP models. Table 1 compares the prompt selections from our proposed RL policy with those from tinybenchmark. The original tinybenchmark select 600 prompts for Huggingface Open LLM Leaderboard, but in our comparison, we select 200 prompts to ensure a fair comparison with our RL policy. The results show that for both prompt selections, using NP produces better benchmark performance estimates, indicating that our neural process model better captures the dependencies and predicts the missing evaluation scores. Given a fixed prediction model (either IRT or NP), our RL-based acquisition policy achieves lower error compared to the prompt selections from tinybenchmark, demonstrating that our RL-based policy is more effective at selecting the informative prompts.

Figure: (a) AlpacaEval
Refer to caption: extracted/5902739/figures/alpaca_eval_biased.png

Model Bias An important aspect of efficient benchmarking strategies is robustness to model bias. To accurately evaluate future models, which may differ significantly from previously seen models, the strategy must accurately measure model capabilities based on the selected prompts. Our train-test splits based on date for MMLU and Open LLM Leaderboard potentially evaluate this situation since model performance tends to improve over time. To further evaluate the performance in the presence of model bias, we divide the models on the AlpacaEval and HELM-Lite leaderboards based on their organizations. For HELM-Lite, we use proprietary models, such as GPT-4 (Achiam et al., ) and Claude (Anthropic, ), for training and test on open-source models, such as LLaMA (Touvron et al., ) and Mistral (Jiang et al., ). For AlpacaEval, we do the opposite, using open-source models for training and proprietary models for testing.

Figure 3 presents the evaluation results on these two benchmarks with model bias. Firstly, static policies, especially Clustering-Score and Clustering-IRT that depend on evaluation scores from the training models, do not perform well. Secondly, although random policies do not suffer from model bias, they cannot leverage dependencies across prompts, leading to lower overall performance. In contrast, our RL-based dynamic acquisition policy can effectively exploit the dependencies across prompts even for models that is significantly different from the models it has seen before. However, we do notice that the existence of model bias makes the problem harder to solve. Compared to Fig. 1 on the same benchmark, even for our RL-based policy, it takes more acquisitions to achieve the same level of errors as in situations where no model bias exists. In practice, a continual learning framework, where the neural process model and the acquisition policies are jointly adapted to the newly added models, might be necessary. We leave this for future works.

Cold Start Problem To evaluate the cold start scenario, we create a synthetic benchmark using MMLU by designating 15 subsets as cold start prompts. During the training of the neural process model and the acquisition policies, the evaluation scores on these 15 subsets are not available. Although the evaluation scores are missing, we assume the prompts themselves are given, allowing random policies and Clustering-Embed static policy to be evaluated without any modifications. However, the Clustering-Score and Clustering-IRT policies will never acquire evaluation scores for these 15 subsets since these policies require access to the evaluation scores to determine whether a prompt will be acquired or not. On the other hand, dynamic acquisition policies can easily adapt to the cold start setting, as they acquire evaluation scores sequentially and actively.

Figure 3 presents the results on the synthetic cold start MMLU benchmark. The performance is evaluated over all 57 subsets during testing. As expected, the Clustering-Score and Clustering-IRT policies do not perform well in the cold start setting because the evaluation scores on the 15 left-out subsets are never acquired. The Clustering-Embed policy performs better than the other clustering based policies as it can select the cold start prompts by clustering based on their embeddings. The RL-based acquisition policy again achieves the best performance estimation. However, it is worth noting that the final estimated benchmark performance is not as accurate as in the fully observed setting (Fig. 1), indicating potential areas for future improvement to narrow the gap.

**Table 2: Comparson of the final benchmark performance estimation methods. w/ pred indicate the proposed method where the neural process is used to predict the missing evaluation scores. w/o pred indicates the baseline where final performance is an aggregation of the acquired evaluation scores.**
|  |  | AlpacaEval | HELM-Lite | Open LLM | MMLU | Chatbot Arena |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | (K=100) | (K=200) | (K=200) | (K=100) | (K=40) |
| Uniform | w/ pred | $0.005\pm 0.000$ | $0.038\pm 0.005$ | $0.022\pm 0.002$ | $0.018\pm 0.001$ | $0.052\pm 0.010$ |
| w/o pred | $0.012\pm 0.001$ | $0.079\pm 0.008$ | $0.043\pm 0.003$ | $0.042\pm 0.003$ | $0.036\pm 0.009$ |  |
| S-Rand | w/ pred | - | $0.035\pm 0.012$ | $0.030\pm 0.003$ | $0.017\pm 0.002$ | - |
| w/o pred | - | $0.072\pm 0.012$ | $0.023\pm 0.001$ | $0.038\pm 0.001$ | - |  |
| C-Embed | w/ pred | $0.006\pm 0.001$ | $0.051\pm 0.010$ | $0.024\pm 0.002$ | $0.020\pm 0.004$ | $0.052\pm 0.014$ |
| w/o pred | $0.023\pm 0.008$ | $0.116\pm 0.003$ | $0.029\pm 0.000$ | $0.029\pm 0.000$ | $0.032\pm 0.003$ |  |
| C-Score | w/ pred | $0.004\pm 0.001$ | $0.054\pm 0.010$ | $0.031\pm 0.003$ | $0.014\pm 0.002$ | $0.054\pm 0.004$ |
| w/o pred | $0.141\pm 0.011$ | $0.051\pm 0.017$ | $0.086\pm 0.002$ | $0.048\pm 0.002$ | $0.037\pm 0.011$ |  |
| C-IRT | w/ pred | $0.003\pm 0.001$ | $0.044\pm 0.013$ | $0.026\pm 0.001$ | $0.015\pm 0.001$ | $0.057\pm 0.002$ |
| w/o pred | $0.069\pm 0.003$ | $0.060\pm 0.014$ | $0.037\pm 0.003$ | $0.041\pm 0.006$ | $0.042\pm 0.003$ |  |
| RL | w/ pred | $0.001\pm 0.000$ | $0.030\pm 0.005$ | $0.018\pm 0.001$ | $0.013\pm 0.000$ | $0.034\pm 0.006$ |
| w/o pred | $0.064\pm 0.006$ | $0.081\pm 0.019$ | $0.063\pm 0.018$ | $0.050\pm 0.006$ | $0.045\pm 0.007$ |  |

### 4.1 Ablation Studies

Prediction Model In the main experimental results, we run the acquisition policy to select a subset of prompts for acquiring their actual evaluation scores and then use a neural process model to predict the scores for the remaining prompts. However, an alternative method to estimate benchmark performance is to directly aggregate the acquired evaluation scores without relying on another model for prediction. The aggregation computes performance per dataset first and then averages across datasets. Table 2 compares these two estimation methods. The results show that the prediction model generally provides better benchmark performance estimation.

Prompt Embedding Our approach utilizes a sentence embedding model to extract representations for the prompts. These representations are used both to train the neural process model and to build the acquisition policies. For the main results, we use the SFR embedding model (Salesforce/SFR-Embedding-Mistral) (Rui Meng, ) to extract prompt representations. In Table 3, we present results using several other embedding models: E5, BGE-large, and BGE-small, corresponding to intfloat/e5-mistral-7b-instruct (Wang et al., ), BAAI/bge-large-en-v1.5 (Xiao et al., ), and BAAI/bge-small-en-v1.5 (Xiao et al., ), respectively. The results indicate that performance generally improves with more powerful embedding models that better distinguish text inputs (^2^22At the time of writing this paper, the average scores from the MTEB English leaderboard. (https://huggingface.co/spaces/mteb/leaderboard) for these four models are: SFR (67.56), E5 (66.63), BGE-large (64.23), and BGE-small (62.17).) Thus, utilizing more powerful embedding models is an important future direction.

**Table 3: Comparison of different prompt embeddings.**
|  |  | Uniform | C-Embed | C-Score | C-IRT | RL |
| --- | --- | --- | --- | --- | --- | --- |
| AlpacaEval | SFR (4096) | $0.005\pm 0.000$ | $0.006\pm 0.001$ | $0.004\pm 0.001$ | $0.003\pm 0.001$ | $0.001\pm 0.000$ |
| E5 (4096) | $0.005\pm 0.000$ | $0.006\pm 0.001$ | $0.007\pm 0.005$ | $0.004\pm 0.001$ | $0.001\pm 0.000$ |  |
| BGE-large (1024) | $0.006\pm 0.001$ | $0.006\pm 0.002$ | $0.009\pm 0.002$ | $0.007\pm 0.001$ | $0.002\pm 0.000$ |  |
| BGE-small (384) | $0.009\pm 0.002$ | $0.009\pm 0.002$ | $0.038\pm 0.031$ | $0.015\pm 0.010$ | $0.005\pm 0.002$ |  |
| MMLU | SFR (4096) | $0.018\pm 0.001$ | $0.020\pm 0.004$ | $0.014\pm 0.002$ | $0.015\pm 0.001$ | $0.013\pm 0.000$ |
| E5 (4096) | $0.018\pm 0.002$ | $0.018\pm 0.002$ | $0.014\pm 0.003$ | $0.016\pm 0.002$ | $0.014\pm 0.003$ |  |
| BGE-large (1024) | $0.029\pm 0.003$ | $0.028\pm 0.005$ | $0.027\pm 0.005$ | $0.023\pm 0.006$ | $0.023\pm 0.003$ |  |
| BGE-small (384) | $0.028\pm 0.003$ | $0.023\pm 0.006$ | $0.022\pm 0.005$ | $0.022\pm 0.005$ | $0.022\pm 0.002$ |  |

Auxiliary Information Our RL-based acquisition policy builds on PPO (Schulman et al., ) and leverages the neural process model to provide auxiliary information and intermediate rewards. Table 4 illustrates the contributions of these components. The results clearly show that each component — both auxiliary information and the intermediate rewards — significantly enhances the acquisition policy, leading to better selection of informative prompts and more accurate benchmark performance estimation.

**Table 4: Contributions of auxiliary information and intermediate reward for our RL-based policy.**
|  | AlpacaEval | MMLU | Open LLM |
| --- | --- | --- | --- |
| PPO | $0.004\pm 0.002$ | $0.017\pm 0.004$ | $0.033\pm 0.010$ |
| +auxiliary_info | $0.003\pm 0.001$ | $0.016\pm 0.003$ | $0.029\pm 0.005$ |
| +interm_reward | $\mathbf{0.001\pm 0.000}$ | $\mathbf{0.013\pm 0.000}$ | $\mathbf{0.018\pm 0.001}$ |

## 5 Conclusion

In this work, we present a novel approach for efficient LLM evaluation by leveraging dependency modeling and subset selection. Our key contributions include developing a generative model that captures dependencies across evaluation prompts and handles mixed-type evaluation scores, as well as proposing new subset selection policies based on these dependencies. Extensive experiments on multiple LLM evaluation benchmarks demonstrate the superiority of our RL-based acquisition policy in providing accurate benchmark performance estimation with a minimal acquisition budget. Our results also emphasize the importance of robustness to model bias and the effectiveness of our approach in cold start scenarios. Future research could explore integrating continual learning frameworks to enhance performance in the presence of model bias and cold starts. Additionally, expanding our methods to other benchmarks and refining the neural process model with improved uncertainty estimation are promising areas for further investigation.

## Appendix A Neural Process

For a benchmark $X$ with $N$ prompts, $X=\{x_{n}\}_{n=1}^{N}$, we first use a pretrained embedding model to extract the representations for each prompt. During training, given a model $m$ with evaluation scores $Y_{m}$, we randomly select a subset of scores $Y_{m}^{(o)}$ as observed and maximize the log-likelihood for the remaining scores $Y_{m}^{(u)}$ based on the equation equation 3. When $Y_{m}^{(u)}$ is too large to fit into memory, we further sample a smaller subset from $Y_{m}^{(u)}$. Due to the inherent permutation invariance of the neural process model, random sampling will not affect the learning of dependencies across prompts.

### A.1 Architecture

The neural process model consists of a prior network $p(z_{m}\mid Y_{m}^{(o)},X)$, a posterior network $q(z_{m}\mid Y_{m}^{(u)},Y_{m}^{(o)},X)$, and a decoder $p(Y_{m}^{(u)}\mid z_{m},Y_{m}^{(o)},X)$. We generally follow the architecture of Attentive Neural Process (Kim et al., ), but replace the self-attention layer with a more memory-efficient Set Transformer layer (Lee et al., ). We also share the same network for both the prior and posterior. Before feeding the prompt embeddings into the prior/posterior network, we use an additional linear layer to reduce the dimensionality of the extracted representations. Similarly, the evaluation scores are passed through a linear layer to increase their dimensionality. We then concatenate the prompt representation with the score representation along the feature dimension and pass the concatenated set of vectors through a series of permutation equivariant Set Transformer layers. The outputs are then aggregated across the set elements to obtain a feature representation for the entire set. Following Set Transformer approach, we use learned pooling by multihead attention. The set representation is then passed through a linear linear to obtain the parameters for the latent distribution, which we assume to be Gaussian here. Please see Fig. 1(a) for an illustration of the prior/posterior network.

The decoder network $p(Y_{m}^{(u)}\mid z_{m},Y_{m}^{(o)},X)$ uses a cross-attention layer to produce a permutation equivariant representation for each prompt. In this layer, the query is the representation for $X$, the key is the representation for $X^{(o)}$, and the value is the permutation equivariant representation corresponding to $Y_{m}^{(o)}$ from the prior network. The permutation equivariant representation for each prompt is then concatenated with the prompt representation and the latent vector. These concatenated inputs are processed through a series of Set Transformer layers. The final outputs are then passed through a linear layer to predict the evaluation scores. Please see Fig. 1(b) for an illustration of the decoder network.

##### Mixed-type Evaluation Scores

The above architecture uses a linear layer to obtain the representation for the evaluation scores. However, the linear layer is not suitable for discrete scores. Instead, we use an Embedding layer to represent the categorical evaluation scores. When a benchmark contains mixed-type scores, meaning some datasets report real-valued metrics while others report discrete scores, we additionally include an embedding vector to indicate the metric types.

Figure: (a) Prior/Posterior
Refer to caption: extracted/5902739/figures/inference.png

### A.2 Hyperparameters

Table A.1 summarizes the hyperparameters used for the neural process model for each dataset. For the HELM-Lite and Chatbot Arena benchmarks, due to their relatively small number of models with evaluation scores, a neural process model with set transformer layers can easily overfit the data. Therefore, we use linear layers instead of the set transformer layers. Note that we did not conduct a thorough hyperparameter search. It is possible to further improve the results with optimized hyperparameters.

**Table A.1: Hyperparameters for the nueral process model.**
|  | AlpacaEval | MMLU | Open LLM | HELM-Lite | Chatbot Arena |
| --- | --- | --- | --- | --- | --- |
| representation dimension for $x$ | 16 | 16 | 16 | 16 | 16 |
| representation dimension for $y$ | 16 | 16 | 16 | 16 | 16 |
| feature dimension for permutation equivariant layer | 32 | 32 | 32 | 32 | 16 |
| number of permutation equivariant layers for encoder | 1 | 2 | 2 | 1 | 1 |
| number of permutation equivariant layers for decoder | 1 | 2 | 2 | 1 | 1 |
| number of attention heads | 8 | 8 | 8 | N/A | N/A |
| number of induced points | 16 | 16 | 8 | N/A | N/A |
| latent dimension | 16 | 32 | 32 | 16 | 8 |

## Appendix B Reinforcement Learning based Acquisition Policy

The acquisition policy determines the next prompt to acquire its evaluation score based on the current state, which includes the prompts $X^{(o)}$ and their scores $Y_{m}^{(o)}$ that have already been acquired. We further incorporate the candidate prompts $X^{(u)}$ into the policy inputs, i.e., $P(i\mid Y_{m}^{(o)},X)$, so the policy has access to the action space. Including the candidate prompts in the inputs is crucial in the cold start setting since the action space differs between training and testing. Similar to the neural process model, the policy network employs two linear layers to obtain representations for both the prompts and the evaluation scores, which are then concatenated along the feature dimension. For the candidate prompts without available evaluation scores, we use a special embedding vector. Then, a permutation-invariant network processes the set of concatenated representations and outputs a aggregated representation for the entire set. We utilize the Set Transformer architecture for the permutation invariant network. Two branches of linear layers are added on top of the set representation for actor and critic, respectively. The actor branch outputs a vector with the same dimensionality as the prompt representations. The probability of selecting a prompt is proportional to the inner product of the output vector and the prompt representations. To prevent the policy from selecting duplicate prompts, the probability of the already selected prompts is manually set to zero. The critic branch outputs a scalar indicating the value estimation for the current state. Table B.1 summarizes the hyperparameters used for the policy network and PPO training process. We did not conduct hyperparameter optimization and used the same set of hyperparameters for all datasets. Further improvements are likely possible with hyperparameter optimization tailored to each dataset.

**Table B.1: Hyperparameters for RL-based acquisition policy.**
| Policy Network | representation dimension for $x$ | 16 |
| --- | --- | --- |
| representation dimension for $y$ | 16 |  |
| feature dimension for permutation equivariant layer | 32 |  |
| number of permutation equivariant layers | 1 |  |
| number of linear layers for actor | 1 |  |
| number of linear layers for critic | 1 |  |
| PPO | advantage $\lambda$ | 0.95 |
| discount factor $\gamma$ | 0.99 |  |
| PPO clip range | [0.8, 1.2] |  |
| entropy coefficient | 0.0 |  |

## Appendix C Cold Start Problem

In the cold start setting, the benchmark is expanded with new prompts for which no evaluation scores are initially available for any model. That is, the original benchmark $X=\{x_{n}\}_{n=1}^{N}$ have evaluation scores $Y_{m}=\{y_{mn}\}_{n=1}^{N}$ for $M$ models, while a set of new prompts $X^{\prime}=\{x_{n}\}_{n=N+1}^{N^{\prime}}$ do not have any evaluation scores.

To enable the neural process model generalize to the newly added prompts, we propose a semi-supervised training procedure, where the new prompts are treated as unlabeled data. During training, we optimize the log-likelihood equation 3 for $X$ and $Y_{m}$. Simultaneously, we predict the evaluation scores for the new prompts $X^{\prime}$ based on the current trained model. When the prediction is sufficiently accurate, meaning the uncertainty is lower than a predefined threshold, we add the predicted scores as synthetic training data to optimize the ELBO equation 3.

The RL policy in the cold start setting follows a similar architecture to Sec. B. To help the policy generalize to unseen prompts, we use the learned prompt representations from the neural process model and keep them fixed throughout the training process. Additionally, we found that entropy regularization over the actor distribution aids generalization, which is set to 0.001 in our experiments.

## Appendix D Experiments

### D.1 Evaluation Procedure

For a model $m^{\prime}$ to be evaluated, the acquisition policy determines a subset of prompts $X^{(o)}$ to acquire the true evaluation scores. The neural process model $p(Y_{m^{\prime}}^{(u)}\mid Y_{m^{\prime}}^{(o)},X)$ predicts the evaluation scores for the remaining prompts. The benchmark performance is then estimated based on these predicted scores. For benchmarks with only one dataset, the benchmark performance is the average over all examples. For benchmarks with multiple datasets, the benchmark performance is averaged over the performance of each dataset. For example, the HuggingFace Open LLM Leaderboard consists of 6 datasets, so the benchmark performance is the average of the performance on these 6 datasets. The MMLU dataset further contains 57 subsets, so its performance is the average over these 57 subsets. For Chatbot Arena, we report win rate as the benchmark performance. For final evaluation results, we compute the absolute difference between the predicted benchmark performance and the real benchmark performance for each model in the test split and report the average absolute error over all models in the test split.

**Table D.1: Benchmark performance estimation error on each LLM benchmark. Lower is better.**
|  | AlpacaEval | HELM-Lite | Open LLM | MMLU | Chatbot Arena |
| --- | --- | --- | --- | --- | --- |
|  | (K=100) | (K=200) | (K=200) | (K=100) | (K=40) |
| Uniform | $0.005\pm 0.000$ | $0.038\pm 0.005$ | $0.022\pm 0.002$ | $0.018\pm 0.001$ | $0.052\pm 0.010$ |
| S-Rand | - | $0.035\pm 0.012$ | $0.030\pm 0.003$ | $0.017\pm 0.002$ | - |
| C-Embed | $0.006\pm 0.001$ | $0.051\pm 0.010$ | $0.024\pm 0.002$ | $0.020\pm 0.004$ | $0.052\pm 0.014$ |
| C-Score | $0.004\pm 0.001$ | $0.054\pm 0.010$ | $0.031\pm 0.003$ | $0.014\pm 0.002$ | $0.054\pm 0.004$ |
| C-IRT | $0.003\pm 0.001$ | $0.044\pm 0.013$ | $0.026\pm 0.001$ | $0.015\pm 0.001$ | $0.057\pm 0.002$ |
| Comb-Optim | $0.006\pm 0.003$ | - | - | - | $0.065\pm 0.012$ |
| Uncertainty | $0.011\pm 0.001$ | $0.055\pm 0.013$ | $0.063\pm 0.015$ | $0.050\pm 0.003$ | $0.035\pm 0.003$ |
| LatentInfoGain | $0.010\pm 0.003$ | - | - | - | $0.066\pm 0.025$ |
| RL | $\mathbf{0.001\pm 0.000}$ | $\mathbf{0.030\pm 0.005}$ | $\mathbf{0.018\pm 0.001}$ | $\mathbf{0.013\pm 0.000}$ | $\mathbf{0.034\pm 0.006}$ |

### D.2 Additional Results

Table D.1 presents the benchmark performance estimation errors for various acquisition policies across different LLM benchmarks. We conduct experiments with 3 random seeds for each benchmark and report the average estimation error and standard deviation under the specified acquisition budget. Prompt embeddings are obtained using the SFR embedding model. For the AlpacaEval and Chatbot Arena benchmarks, stratified random sampling is equivalent to uniform sampling since they only contain one dataset. Combinatorial optimization and information gain based policies are too expensive to run for HELM-Lite, HuggingFace Open LLM Leaderboard, and MMLU due to the large number of prompts in each benchmark.

The RL-based acquisition policy consistently achieves the lowest error across all benchmarks, indicating its superior ability to select informative prompts and accurately estimate benchmark performance.
The stratified random sampling performs similarly to the uniform sampling, and these random acquisition policies generally are competitive, particularly because they are efficient and do not rely on any other models to determine the prompt selection.

The Cluster-Embed policy does not perform any better than the random selection, suggesting that the similarity in prompt embedding does not always correlate with the similarity in the evaluation scores.
Utilizing evaluation scores for clustering shows mixed results. The Clustering-Score policy outperforms Clustering-Embed on AlpacaEval and MMLU but underperforms on HELM-Lite, Open LLM and Chatbot Arena benchmarks.
Clustering based on IRT features generally provides better performance estimation since these features are learned to reflect the evaluation scores.

The combinatorial optimization based policy does not perform well, even on the two small benchmarks where it is computationally feasible. We attribute this to a potential distribution shift between the models used for training and those used for testing, suggesting that the static policy optimized on training models does not generalize well to new models during testing.

The uncertainty sampling based acquisition policy does not perform well across all benchmarks. Theoretically, the uncertainty sampling method requires a good estimation of the aleatoric uncertainty to perform well. However, in practice, the uncertainty from the neural process model combines the aleatoric and epistemic uncertainties. Quantifying and decomposing the aleatoric and epistemic uncertainties is an active research ares in machine learning (Gawlikowski et al., ; Wimmer et al., ; Hüllermeier & Waegeman, ), which we leave for future work to explore for our AEA application.
Similarly, the information gain based acquisition policy also requires accurate uncertainty estimation, which is challenging, especially with scarce training data on AlpacaEval and Chatbot Arena benchmarks.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. I understand the paper is doing subset selection of prompts for evaluation so that it finds the most informative prompts so that if the model performs well in these prompts, then the model can be said to perform well on all prompts from the test set. This is related to the few-shot learning with LLMs [a, b, c, d] and the relevant papers that cite them. Note that these works use subset selection during training time whereas this paper does subset selection of prompts during evaluation time. However, under the condition that training and testing prompts come from the same underlying distribution, how can the two approaches be different (or can be reconciled)?
2. The significance of this setting is not clear to me. Most importantly, a more detailed discussion on related work will bring this out. What are the works that are closest to your work that do some sort of active testing? The only reference I can find is the Kossen et al. 2021 (line 318). 
3. I understand how the different acquisition functions capture prompt diversity and informativeness using embeddings. This is standard in AL/RL/bandit literature. However, an additional component in natural language generation/text generation is the difficulty of the task (such as chain-of-thought promoting or mathematical reasoning). How does your acquisition function take these into account?
4. Again, my final question is more in relation to Active learning. The approaches that you mention such as clustering, uncertainty sampling, and information gain are already well known. The combinatorial approach can be mapped to combinatorial semi-bandits (see [e]). In light of this can the authors shed more light on the key technical novelty/approach of this paper? Can I simply say that this work is an extension of active training (read active learning where you may or may not train the model and simply do few shot learning) to active testing for LLMs?

[a] Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning
[b] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? 
[c] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. 
[d] Subhojyoti Mukherjee, Ge Liu, Aniket Deshmukh, Anusha Lalitha, Yifei Ma, Branislav Kveton Experimental Design for Active Transductive Inference in Large Language Models
[e] Branislav Kveton, Zheng Wen, Azin Ashkan, Csaba Szepesvari, Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits
soundness: 2
strengths: 1. The paper is well written as it clearly lays down the motivation of the paper, prompt subset selection for evaluation. However, how significant is the motivation is not clear to me.
2. The key idea is to capture the dependencies between the prompts by modeling the conditional distribution between unobserved label and observed labels using a stochastic process makes sense to me. This is standard VAE stuff.
3. They clearly explain the acquisition function used to select the prompts. However, these acquisition functions are well-known in active learning. So I cannot consider them a novel contribution of this paper.
summary: This paper proposes the Adaptive Evaluation Acquisition (AEA) approach which is basically an adaptive subset selection of prompts for a model so that the generalization error on remaining prompts is low. Note that this is different than Active learning as the goal here is to perform better at evaluation time than training an accurate model. Here, I also want to point out that even though the authors have noted their setting is different than Active learning, their prompt selection strategy is basically an uncertainty sampling (max-entropy) approach (see eq (5)). Their key idea is to capture the dependencies between the prompts by modeling the conditional distribution between unobserved label $Y^u$ and observed label $Y^o$ as a stochastic process as they outline in eq(2). Since the integration is over a high dimensional latent space, following a standard procedure they optimize the evidence lower bound (ELBO) following variational autoencoder (VAE). Once the underlying modeling as a stochastic random process is fixed, they then use an active (or static) acquisition policy to choose the $Y^o$ in algorithm 1. They propose several acquisition strategies for selecting the prompts, ranging from static combinatorial optimization approach to max entropy. They evaluate this approach on several LLM dataset benchmarks.
weaknesses: 1) The writing needs more improvement. For example in lines 86-87, the subset $o^*$ has cardinality $K$ but is indexed from $\{1, \ldots, N\}$. Is K < N Do you mean to say that there are some repetitive prompts?
2) The approach seems very similar to few shot learning with LLMs, however they do not do any comparison with these works. See my question 1.
3) The significance of the motivation is not clear to me. Also, it is not clear to me how this subset of prompts on which the LLM is evaluated captures the hardness of the task. See my question 2, 3.
4) The difference with Active learning is not very clear. This needs more explanation See my question 4.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: It's not clear to me how to apply this method:

(a) At the design stage when not benchmark data is known.

(b) When new models are introduced to an existing benchmarks.

Could you please explain how this method could be incorporate into the design of benchmarks from the get go? What do you do right at the beginning?

Moreover, as the set of models in a benchmark grows, how does the method keep up with the new data? Do you recommend retraining after each model is introduced? 

Finally, how does the computational complexity of implementing your method (possibly with retraining after reach new model) compare with simply evaluating on more prompts?
soundness: 2
strengths: (1)

The general research direction is well motivated. Benchmark evaluation for LLMs is indeed a costly enterprise. Methods to compare models from fewer examples could potentially be very useful.

(2)

The authors tried out several different model fitting methods. The RL method required some clever reward engineering. The amount of work displayed here is adequate for an ICLR paper.
summary: The authors propose a method to speed up the benchmark evaluation for large language models. The method is to fit a neural process to existing benchmarks results and to predict a larger set of evaluation results from a smaller set of results. In doing so, the authors hope to leverage the correlations between the results of a language model on different prompts in a benchmark. What works best in the empirical evaluation is a method that chooses prompts based on a reinforcement learning heuristic.
weaknesses: (1)

There is a major methodological weakness in the evaluation.

The authors only test how well their method can fit existing benchmark results. To do so they split the the benchmark results into training and test, fit the method on train, and evaluate the approximation errors on test. You might call this the missing value interpolation error on the benchmark. This quantity, however, does *not* shed light on what would happen if a new model entered the benchmark. The new model might be out-of-distribution with respect to existing benchmark results. This can easily happen if the model answers prompts in a manner that's significantly different from prior models, as is the case with major model improvements. In this case, the interpolation error on existing benchmarks does not and cannot give an indication of how well the proposed method would do on the new model. 

To strengthen the evaluation part of this paper, it would be helpful to train the method on a subset of older models and then test what would happen if newer models entered the benchmark. Does the method correctly identify the relative odering of the new models? This is a more meaningful criterion than the missing value interpolation error on the entire leaderboard.

(2)

The method is highly complex with a lot of moving pieces and knobs to turn. In particular, there are several different confounding sources of error: the prompt embedding, the selection method, the neural approximation. This leads to several problems of which I'll mention two:

(a) It's somewhat unclear how a benchmark designer is supposed to implement this method without already having a large amount of benchmark results to draw on. In particular, it's not clear how to get this method off the ground for a new benchmark. 

(b) As the set of models in the benchmark grows, it would seem that this method requires retraining on the growing set of data points. This is an issue as it would possibly reorder the existing model ranking.

(c) The primary purpose of a benchmark is to standardize an evaluation protocol. With so many moving pieces and design choices, a benchmark based on this method will likely be harder to standardize. Moreover, the many design choices will make the benchmark appear arbitrary and reduce trust.

(3)

Given that the method necessarily introduces bias, it would be good to combine it with existing bias mitigation methods such as prediction powered inference. See, e.g., this reference and prior work: https://arxiv.org/abs/2403.07008

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Does your model have any components trained using a training set? From the experiments section, it seems so, but this is not clear from the methods section.
2) In your equation 3 RHS, shouldn't we take the expectation wrt the unobserved portion of $Y$ (given that we do not observe it)? 
3) Also from eq 3, it is not clear what are the assumptions on p(Y^u|...) and how it can be modeled because without observing Y^u;
4) As far as I understand, you need to refit your Neural Process model (or at least the posterior of Z_m) for every LLM $m$ you want to evaluate. How expensive is this step? One of the main applications of efficient evaluation of LLMs is quick evaluation during pre-training and it seems that your approach would not be friendly in this case;
5) Why do you compare against the Cluster-IRT approach and not IRT++ (from tinyBenchmarks)? IRT++ seems a better method;
6) It seems that some curves are missing in Fig 1, for example. Why? If they are not included on purpose, the legend should not include them.
soundness: 3
strengths: 1) The Neural process model is novel and interesting in this context;
2) The application of RL is also novel in the context of efficient evaluation of LLMs. The idea seems promising;
3) The authors are able to get strong empirical results.
summary: The paper proposes investigating strategies for the efficient evaluation of LLMs. Their approach models dependencies across test examples using Neural stochastic processes and then uses a novel RL-based sampling policy that leverages the captured dependencies between test examples.
weaknesses: 1) Section 2.2 can be greatly improved; it is not even clear how fitting is possible without observing labels Y^u. The authors cite VAE, but in VAEs the outputs are known, which makes the decoder training possible. This suggests that some assumptions about the model are not included (or clear) in the text. A more detailed explanation in the text would be appreciated (possibly including an algorithm box for model training);
2) I am not sure including Chatbot Arena makes sense here because it is a pairwise comparison dataset and $Y_m$ will depend on other models you're comparing against. Your NP model does not seem to accommodate that.
3) The paper does not comment on IRT adaptive testing on related work (see https://arxiv.org/pdf/2404.00712, for example). This class of methods is directly related to the paper and could be considered a competitor to the RL approach. It is important to mention why you do not compare against it;
4) The paper does not comment on limitations. One major limitation is the need to fit a neural net for every LLM we want to make inferences about. Moreover, as noted by tinyBenchmarks, adaptive testing can reduce the scope of applications as it can make efficient evaluation more costly;

Minor:
1) typo "Comparson";

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I have two questions:
1. Are there any recent "Active Testing" works with language models? Kossen et al is from 2021, and is tested on Fashion-MNIST and CIFAR-100.
2. Could you motivate the use of an RL agent to estimate the value of a data point? Why are you using an RL agent to calculate the model uncertainty? While the formulation makes sense, a few more baselines would strengthen the motivation. Baselines can include: (1) self-consistency (sample the model multiple times for a response, if the response variety is large, classify the data sample as "not confident") or (2) model feedback (use the distribution over the tokens or perplexity to determine confidence).
soundness: 2
strengths: The paper is well written, and well structured. Their selected baselines are appropriate and quite thorough, including metrics from information theory to classic clustering-based metrics. They also show great thought into the methodology by addressing cold start. The appendix also outlines helpful information, such as Algorithm 2. Finally, the small detail of including standard deviations for each of their results showcases the stability of their algorithm.
summary: Active Evaluation Acquisition (AEA) is a method that selects a subset of samples from various benchmarks to efficiently evaluate a model's performance on the benchmarks. They test various confidence- and density-based metrics for selecting subsets of benchmark data, as well as create an RL-based dynamic selection policy. Given the current selected subset and a data point $i$, the policy is used to estimate the performance of the model on $i$, which is then sampled multiple times to determine the confidence of the estimation. They test their method on popular datasets (including AlpacaEval, MMLU, and Chatbot Arena) with popular models (including GPT-4, LLaMa, and Mistral).
weaknesses: This is an interesting idea and innovative approach - I am having some trouble understanding why choosing samples with less confidence in evaluation will result in a rigorous enough evaluation. The baselines have a clear objective: choose representative samples. Your paper can be stronger if you (1) clarify the objective of your RL-based method, and (2) include more analysis on why RL achieves near 0% absolute error on AlpacaEval. Also, only AlpacaEval achieves such results - the other datasets have an error of ~2%. Including a comparative analysis on the performance of AppacaEval versus the other datasets can help readers understand the discrepancy in performance. Finally (this is more of a nitpicky thing), consider calling your RL-based method RL-AEA to make clear your contribution. Or at least append "RL" with "(ours)" in Figures 1-2 and Tables 2-3, E.1.

Meta-review: The paper studies strategies to improve evaluation efficiency by actively selecting a subset of prompts to evaluate and then predicting the performance of the rest of the prompts. The paper empirically demonstrates that the proposed approach can reduce the number of evaluation prompts. 

The weaknesses pointed out by reviewers include its comparison to related work on few-shot learning in LLM and existing techniques in active learning and active testing, the complexity of the proposed approach, and its advantages over the very simple baseline -- random sampling.

Decision: Reject