Active Evaluation Acquisition for Efficient LLM Benchmarking

Authors: Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, Graham Horwood
Primary area: foundation or frontier models, including LLMs
Keywords: Efficient LLM Evaluation, Active Learning, Subset Selection

Abstract:
As large language models (LLMs) become increasingly versatile, numerous large scale benchmarks have been developed to thoroughly assess their capabilities. These benchmarks typically consist of diverse datasets and prompts to evaluate different aspects of LLM performance. However, comprehensive evaluations on hundreds or thousands of prompts incur tremendous costs in terms of computation, money, and time. In this work, we investigate strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy. Our approach models the dependencies across test examples, allowing accurate prediction of the evaluation outcomes for the remaining examples based on the outcomes of the selected ones. Consequently, we only need to acquire the actual evaluation outcomes for the selected subset. We rigorously explore various subset selection policies and introduce a novel RL-based policy that leverages the captured dependencies. Empirical results demonstrate that our approach significantly reduces the number of evaluation prompts required while maintaining accurate performance estimates compared to previous methods.

Decision: reject
Reviewer scores: 3, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730226304461

thread_type:
official_review

<thread_object>
creator: reviewer_zhcw

<content>
summary:
Active Evaluation Acquisition (AEA) is a method that selects a subset of samples from various benchmarks to efficiently evaluate a model's performance on the benchmarks. They test various confidence- and density-based metrics for selecting subsets of benchmark data, as well as create an RL-based dynamic selection policy. Given the current selected subset and a data point $i$, the policy is used to estimate the performance of the model on $i$, which is then sampled multiple times to determine the confidence of the estimation. They test their method on popular datasets (including AlpacaEval, MMLU, and Chatbot Arena) with popular models (including GPT-4, LLaMa, and Mistral).

strengths:
The paper is well written, and well structured. Their selected baselines are appropriate and quite thorough, including metrics from information theory to classic clustering-based metrics. They also show great thought into the methodology by addressing cold start. The appendix also outlines helpful information, such as Algorithm 2. Finally, the small detail of including standard deviations for each of their results showcases the stability of their algorithm.

weaknesses:
This is an interesting idea and innovative approach - I am having some trouble understanding why choosing samples with less confidence in evaluation will result in a rigorous enough evaluation. The baselines have a clear objective: choose representative samples. Your paper can be stronger if you (1) clarify the objective of your RL-based method, and (2) include more analysis on why RL achieves near 0% absolute error on AlpacaEval. Also, only AlpacaEval achieves such results - the other datasets have an error of ~2%. Including a comparative analysis on the performance of AppacaEval versus the other datasets can help readers understand the discrepancy in performance. Finally (this is more of a nitpicky thing), consider calling your RL-based method RL-AEA to make clear your contribution. Or at least append "RL" with "(ours)" in Figures 1-2 and Tables 2-3, E.1.

questions:
I have two questions:
1. Are there any recent "Active Testing" works with language models? Kossen et al is from 2021, and is tested on Fashion-MNIST and CIFAR-100.
2. Could you motivate the use of an RL agent to estimate the value of a data point? Why are you using an RL agent to calculate the model uncertainty? While the formulation makes sense, a few more baselines would strengthen the motivation. Baselines can include: (1) self-consistency (sample the model multiple times for a response, if the response variety is large, classify the data sample as "not confident") or (2) model feedback (use the distribution over the tokens or perplexity to determine confidence).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705481516

thread_type:
official_review

<thread_object>
creator: reviewer_p8cj

<content>
summary:
The paper proposes investigating strategies for the efficient evaluation of LLMs. Their approach models dependencies across test examples using Neural stochastic processes and then uses a novel RL-based sampling policy that leverages the captured dependencies between test examples.

strengths:
1) The Neural process model is novel and interesting in this context;
2) The application of RL is also novel in the context of efficient evaluation of LLMs. The idea seems promising;
3) The authors are able to get strong empirical results.

weaknesses:
1) Section 2.2 can be greatly improved; it is not even clear how fitting is possible without observing labels Y^u. The authors cite VAE, but in VAEs the outputs are known, which makes the decoder training possible. This suggests that some assumptions about the model are not included (or clear) in the text. A more detailed explanation in the text would be appreciated (possibly including an algorithm box for model training);
2) I am not sure including Chatbot Arena makes sense here because it is a pairwise comparison dataset and $Y_m$ will depend on other models you're comparing against. Your NP model does not seem to accommodate that.
3) The paper does not comment on IRT adaptive testing on related work (see https://arxiv.org/pdf/2404.00712, for example). This class of methods is directly related to the paper and could be considered a competitor to the RL approach. It is important to mention why you do not compare against it;
4) The paper does not comment on limitations. One major limitation is the need to fit a neural net for every LLM we want to make inferences about. Moreover, as noted by tinyBenchmarks, adaptive testing can reduce the scope of applications as it can make efficient evaluation more costly;

Minor:
1) typo "Comparson";

questions:
1) Does your model have any components trained using a training set? From the experiments section, it seems so, but this is not clear from the methods section.
2) In your equation 3 RHS, shouldn't we take the expectation wrt the unobserved portion of $Y$ (given that we do not observe it)? 
3) Also from eq 3, it is not clear what are the assumptions on p(Y^u|...) and how it can be modeled because without observing Y^u;
4) As far as I understand, you need to refit your Neural Process model (or at least the posterior of Z_m) for every LLM $m$ you want to evaluate. How expensive is this step? One of the main applications of efficient evaluation of LLMs is quick evaluation during pre-training and it seems that your approach would not be friendly in this case;
5) Why do you compare against the Cluster-IRT approach and not IRT++ (from tinyBenchmarks)? IRT++ seems a better method;
6) It seems that some curves are missing in Fig 1, for example. Why? If they are not included on purpose, the legend should not include them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709239853

thread_type:
official_review

<thread_object>
creator: reviewer_vhba

<content>
summary:
The authors propose a method to speed up the benchmark evaluation for large language models. The method is to fit a neural process to existing benchmarks results and to predict a larger set of evaluation results from a smaller set of results. In doing so, the authors hope to leverage the correlations between the results of a language model on different prompts in a benchmark. What works best in the empirical evaluation is a method that chooses prompts based on a reinforcement learning heuristic.

strengths:
(1)

The general research direction is well motivated. Benchmark evaluation for LLMs is indeed a costly enterprise. Methods to compare models from fewer examples could potentially be very useful.

(2)

The authors tried out several different model fitting methods. The RL method required some clever reward engineering. The amount of work displayed here is adequate for an ICLR paper.

weaknesses:
(1)

There is a major methodological weakness in the evaluation.

The authors only test how well their method can fit existing benchmark results. To do so they split the the benchmark results into training and test, fit the method on train, and evaluate the approximation errors on test. You might call this the missing value interpolation error on the benchmark. This quantity, however, does *not* shed light on what would happen if a new model entered the benchmark. The new model might be out-of-distribution with respect to existing benchmark results. This can easily happen if the model answers prompts in a manner that's significantly different from prior models, as is the case with major model improvements. In this case, the interpolation error on existing benchmarks does not and cannot give an indication of how well the proposed method would do on the new model. 

To strengthen the evaluation part of this paper, it would be helpful to train the method on a subset of older models and then test what would happen if newer models entered the benchmark. Does the method correctly identify the relative odering of the new models? This is a more meaningful criterion than the missing value interpolation error on the entire leaderboard.

(2)

The method is highly complex with a lot of moving pieces and knobs to turn. In particular, there are several different confounding sources of error: the prompt embedding, the selection method, the neural approximation. This leads to several problems of which I'll mention two:

(a) It's somewhat unclear how a benchmark designer is supposed to implement this method without already having a large amount of benchmark results to draw on. In particular, it's not clear how to get this method off the ground for a new benchmark. 

(b) As the set of models in the benchmark grows, it would seem that this method requires retraining on the growing set of data points. This is an issue as it would possibly reorder the existing model ranking.

(c) The primary purpose of a benchmark is to standardize an evaluation protocol. With so many moving pieces and design choices, a benchmark based on this method will likely be harder to standardize. Moreover, the many design choices will make the benchmark appear arbitrary and reduce trust.

(3)

Given that the method necessarily introduces bias, it would be good to combine it with existing bias mitigation methods such as prediction powered inference. See, e.g., this reference and prior work: https://arxiv.org/abs/2403.07008

questions:
It's not clear to me how to apply this method:

(a) At the design stage when not benchmark data is known.

(b) When new models are introduced to an existing benchmarks.

Could you please explain how this method could be incorporate into the design of benchmarks from the get go? What do you do right at the beginning?

Moreover, as the set of models in a benchmark grows, how does the method keep up with the new data? Do you recommend retraining after each model is introduced? 

Finally, how does the computational complexity of implementing your method (possibly with retraining after reach new model) compare with simply evaluating on more prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730777454222

thread_type:
official_review

<thread_object>
creator: reviewer_pgqh

<content>
summary:
This paper proposes the Adaptive Evaluation Acquisition (AEA) approach which is basically an adaptive subset selection of prompts for a model so that the generalization error on remaining prompts is low. Note that this is different than Active learning as the goal here is to perform better at evaluation time than training an accurate model. Here, I also want to point out that even though the authors have noted their setting is different than Active learning, their prompt selection strategy is basically an uncertainty sampling (max-entropy) approach (see eq (5)). Their key idea is to capture the dependencies between the prompts by modeling the conditional distribution between unobserved label $Y^u$ and observed label $Y^o$ as a stochastic process as they outline in eq(2). Since the integration is over a high dimensional latent space, following a standard procedure they optimize the evidence lower bound (ELBO) following variational autoencoder (VAE). Once the underlying modeling as a stochastic random process is fixed, they then use an active (or static) acquisition policy to choose the $Y^o$ in algorithm 1. They propose several acquisition strategies for selecting the prompts, ranging from static combinatorial optimization approach to max entropy. They evaluate this approach on several LLM dataset benchmarks.

strengths:
1. The paper is well written as it clearly lays down the motivation of the paper, prompt subset selection for evaluation. However, how significant is the motivation is not clear to me.
2. The key idea is to capture the dependencies between the prompts by modeling the conditional distribution between unobserved label and observed labels using a stochastic process makes sense to me. This is standard VAE stuff.
3. They clearly explain the acquisition function used to select the prompts. However, these acquisition functions are well-known in active learning. So I cannot consider them a novel contribution of this paper.

weaknesses:
1) The writing needs more improvement. For example in lines 86-87, the subset $o^*$ has cardinality $K$ but is indexed from $\{1, \ldots, N\}$. Is K < N Do you mean to say that there are some repetitive prompts?
2) The approach seems very similar to few shot learning with LLMs, however they do not do any comparison with these works. See my question 1.
3) The significance of the motivation is not clear to me. Also, it is not clear to me how this subset of prompts on which the LLM is evaluated captures the hardness of the task. See my question 2, 3.
4) The difference with Active learning is not very clear. This needs more explanation See my question 4.

questions:
1. I understand the paper is doing subset selection of prompts for evaluation so that it finds the most informative prompts so that if the model performs well in these prompts, then the model can be said to perform well on all prompts from the test set. This is related to the few-shot learning with LLMs [a, b, c, d] and the relevant papers that cite them. Note that these works use subset selection during training time whereas this paper does subset selection of prompts during evaluation time. However, under the condition that training and testing prompts come from the same underlying distribution, how can the two approaches be different (or can be reconciled)?
2. The significance of this setting is not clear to me. Most importantly, a more detailed discussion on related work will bring this out. What are the works that are closest to your work that do some sort of active testing? The only reference I can find is the Kossen et al. 2021 (line 318). 
3. I understand how the different acquisition functions capture prompt diversity and informativeness using embeddings. This is standard in AL/RL/bandit literature. However, an additional component in natural language generation/text generation is the difficulty of the task (such as chain-of-thought promoting or mathematical reasoning). How does your acquisition function take these into account?
4. Again, my final question is more in relation to Active learning. The approaches that you mention such as clustering, uncertainty sampling, and information gain are already well known. The combinatorial approach can be mapped to combinatorial semi-bandits (see [e]). In light of this can the authors shed more light on the key technical novelty/approach of this paper? Can I simply say that this work is an extension of active training (read active learning where you may or may not train the model and simply do few shot learning) to active testing for LLMs?

[a] Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning
[b] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? 
[c] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. 
[d] Subhojyoti Mukherjee, Ge Liu, Aniket Deshmukh, Anusha Lalitha, Yifei Ma, Branislav Kveton Experimental Design for Active Transductive Inference in Large Language Models
[e] Branislav Kveton, Zheng Wen, Azin Ashkan, Csaba Szepesvari, Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies strategies to improve evaluation efficiency by actively selecting a subset of prompts to evaluate and then predicting the performance of the rest of the prompts. The paper empirically demonstrates that the proposed approach can reduce the number of evaluation prompts. 

The weaknesses pointed out by reviewers include its comparison to related work on few-shot learning in LLM and existing techniques in active learning and active testing, the complexity of the proposed approach, and its advantages over the very simple baseline -- random sampling.