Scaling Up Active Testing to Large Language Models
Authors: Gabrielle Berrada
Jannik Kossen
Freddie Bickford Smith
Muhammed Razzak
Yarin Gal
Tom Rainforth
Venue: NeurIPS 2025 poster ()
Keywords: active testing
active evaluation
model evaluation
data curation
large language models
Primary area: deep_learning

Abstract: Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.

Scaling Up Active Testing to Large Language Models
Gabrielle Berrada +
Jannik Kossen 1+
Freddie Bickford Smith
Muhammed Razzak
Yarin Gal
Tom Rainforth
OATML, Department of Computer Science, University of Oxford
Department of Statistics, University of Oxford
Abstract
Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up large language models (LL.s).. In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which show to be a useful indicator of how well active testing is working within a single run.
Introduction
Evaluating frontier models is becoming more expensive as they become more sophisticated (Burden, 2024; OpenAI, 2023). At the same time, with new models arriving in quick succession, and ever- present scope for data to leak from evaluations to training (Ganguli et al, 2023), the evaluation problem is dynamic one: requires ongoing, adaptive gathering of new evaluation data.
Active testing (Kossen et al, 2021, 2022) is an attractive solution to this problem. Motivated by the observation that some labels are more informative than others about the behaviour of target model, it involves carefully deciding which test inputs to acquire labels for. The foundation for data acquisition a surrogate model of the test-time label distribution, which typically updated on the labels acquired during testing. Making a data- aata-aqquis decision requires making predictions with the surrogate model and often also the target model. The computational cost of this process is often justifiable in light of high labelling costs, but it limits the scope of active testing's 's applicability.
In this work show how active testing can be scaled up large language models (LLMss) allowing us to more effectively evaluate them with limited labelling budgets. We begin by highlighting three computational bottlenecks in the predominant approach active testing: updating the surrogate model as labels are acquired, making predictions with the surrogate model, and making predictions with the target model. The first of these particularly important: has traditionally involved repeatedly running gradient-bass training on the acquired test data within an active-testing loop.
We then identify simple and surprisingly effective ways address each of these key bottlenecks. First, we strip back the training of the surrogate model to a single step of co-context learning (Brown al, 2020) on small amount of initial test data, removing the need for repeated gradient-hai based training. Second, we show it is possible to use a surrogate model that smaller than the target model,
'Equal contribution
Correspondence to rainforthdstats.cc ox. ac. uk.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Unitorm random
LURE-7B
LURE-708
S5T-2
FP8
H5
Sub]
Risk-estimaten arror
10-2
10
10-2
10-7
10
10-3
10
10-1
10
10-4
10-3
10-4
10-1
10.3
10-4
10
200
400
200
400
200
400
200
400
Number acquired labels
Number of acquired labels
Number of acquired labels
Number of acquired labes
Our proposed active-testing approach enables low-error estimates of the risk (expected predictive loss) of large language model (here the 70B version of Llama 2) four text-classification problems (SST-2, FPB, HS and Subj) while only using small label budget. We compare uniform- random testing against active testing (LURE), using either a 7B or 70B surrogate model with in co-context learning to guide data acquisition.
Figure 1
reducing the cost of surrogate- model predictions. Third, show we even forgo making any predictions with the target model, relying solely on the surrogate model for data acquisition. Together these changes make active testing dramatically cheaper, enabling it to be scaled up L.MMs.
Empirically we find our approach can substantially improve over the common practice of acquiring data uniformly random (Figure 1). More specifically it can estimate the risk (expected predictive loss over test examples) of LLMs with an estimation error typically 25% to 50%-and sometimes up to 80%-lower than that obtained through miiform-raddon data acquisition. Our work therefore represents significant progress in accurately and dynamically evaluating LLMs, as well as in more generally increasing the scope for active testing to be used in practical settings.
To further improve the real-world applicability of active testing, we address the challenge of judging how well it is working on a given problem. In research we can assess active testing by running multiple times and comparing its risk estimates to a known true risk. But practical deployment involves running active testing only once, without knowledge of the true risk, making it hard to gauge active-testing performance. To help address this we derive a bootstrap estimator (Efron, 1979) of the risk-estimation error. In experiments true risk-estimation error 88% of the time, suggesting our estimator can useful diagnostic tool.
find our approximate 95% confidence intervals contain the
2 Background
Our aim is to evaluate fixed target model, f. mapping inputs v E X' to outputs y Y. We formalise evaluation as estimating a form of frequentist risk (Berger, 1985), namely an expected predictive loss of the form R = E (2.) [<(((x),, w)]. where Peval represents a reference system used source of ground truth, and & denotes a loss function that represents the consequences of predictive errors. As in past work (Kossen et al, 2021, 2022), we consider pool-based setting (Lewis & Gale, 1994) where we have access to pool of N unlabelled test inputs, D Pooo = 1 but acquiring a label, y Peval (y) x) for any given is costly, so we can only afford acquire M N labels.
2.1 Uniform-random sampling
simple estimator of R uses iniform-randont samples from the pool:
M
Rami
M
m=1
where i1:M ~ Uniform( 21, N). M) are indices sampled without replacement. If the pool was constructed by sampling I; ~ Pewal (x) then Rmmit is known as the subsample empirical risk and an unbiased estimator of Rt However, it will typically have high variance for small values of M.
2.2 Sampling-based active testing
Active testing deals with how to more carefully select the M inputs for labelling to produce a more accurate estimate of R. This can be achieved through either sampling-bai based approach or an

---

If f(x) = (yla) and E(p,v) = log p(y) then the acquisition function becomes am (z) = H H[mm(y)a) py (v))))) the cross entropy between the surrogate and target models. Intuitively can understand this acquisition function as measuring the disagreement between the surrogate and target Algorithm/ Sampling- active testing
Algorithm 2 Interpolation- active testing
input Target model, f: loss function, E; acqui- sition function, a; training set, Drrain' pool
input Target model, f; loss function, E acqui- sition function, a; training set, Draiini pool
set, pool: label budget, .M
set, D Dooot label budget, M
1: Compute (xr,) for all xy
D pool
1: Compute f(x,,) for all x. Pool
2: Set Duest = 0
2: Set Drest = 0.
3: for m 21. M) do
3: for m (1,2,...; :).....Me. do
4: Train Tm (e.g., on Dtrain U rrest)
4: Train Tm (e.g., on Dtrain U (Doss)
5: Compute am (2y) for Ij E D pool
5: Compute (xy) for all Ij E D Dpool
6: Sample im (2)
6: Select im = arg max, Am am())
7: Sample V/m Peval (y/
7: Sample Yin Peval (im)
8: Set Drest + Drest U (Fim Vim H
8: Set Dust f Drest U (TTm Him
9: end for
9: end for
10: Compute RLURE
10: Compute RASE
output Risk estimate, BEURE
output Risk estimate, RASE
interpolation-based approach. In nampling-based active testing we use Monte Carlo estimators of R, similar Ruiit but with pool indices sampled from non-uniform distribution. At acquisition step m we sample pool input 2; with probability proportional to am (2:)) E R+ where am is an acquisition function that measures a notion of how useful we expect the label for given input to be (the m subscript denotes that it can depend on all the data available at step m, including the target model's training data). That is, we sample im qm (i) where qu (i) = am Ex am (I²).
Farquhar et al (2021) showed that naive Monte Carlo with im ~ qin(i) biased and, to address this, introduced the levelled unbiased risk estimator (LURE):
M
N - M
N m (N m 1) -1).
PLURE-
Um Umllf (f Dim)), Um 1
M
Given both Rmni and ÍLURE are unbiased, any advantage that RLURE brings comes from variance reduction through a well-designed acquisition function, 4m- The optimal acquisition function is on (x) = E (vl=) (x)=))/( [f(f(x)), v)], the expected loss under (y)a) (z). Since this is unknown, Kossen et al (2021) proposed approximating Peval (3/ (r) with a surrogate model, n. (z)z), which is typically trained on the acquired test labels along with the target model's training data. This surrogate model then allows us to define a practical acquisition function:
am (x) = EE,.. [(((()), ( v)]
models, leading to acquire labels for the inputs where the two models' predictions differ the most.
2.3 Interpolation-basee active testing
Kossen et al (2022) introduced an alternative approach to active testing that, like the sampling- bassdd approach, uses surrogate model to guide data acquisition, but also uses it to estimate the risk. Their active surrogate estimator (ASE) is
RASE N Σ R:ED
Br,,, (r,,, v)).
(3)
Here the risk is not directly estimated using labels acquired from y ~ Peval (z) but instead using labels simulated from the surrogate model. The goal of data acquisition is then to improve the surrogate model in a way that leads to a more accurate estimate of the risk. Kossen et al (2022) approached this using an acquisition function called the expected weighted disagreement (XWED).
3

---

3 Scaling up active testing
With an understanding of standard approaches to active testing, we can now identify and address barriers to scaling up to LLMs. We focus on three key computational bottlenecks: training the surrogate model, making predictions with the surrogate model, and making predictions with the target model. Our aim is to reduce the cost of these steps while maintaining the efficacy of active testing.
3.1 carroggate-odee training
We argue that the lop priority for reducing the cost of data acquisition is to rethink the training of the surrogate model. Existing approaches involve repeatedly running gradient-based -based training on the data acquired during testing, typically combined with the target model's training data. This can be very expensive, especially when working with large datasets and surrogates, as we are focusing on here.
To avoid this expense we suggest stripped- trippedd-back back approach: construct the surrogate model using in-context learning (Brown et al, 2020) on a small amount of randomly acquired test data, then keep it fixed. This reduces the cost of surrogate-moder training essentially the minimum possible.
This design decision has important implications for the relative merits of sampling-hasee -based and interpolation-basee active testing. We are using relatively crude surrogate model to reduce compu- tational cost. This affects the sampling-basee based estimator solely through the data-acquisition step (and related corrective weights, Vm)). whereas it affects the interpolation-batt basedd estimator not only through data acquisition but also much more directly through the expectation over Tm (y(2z) in Equation 3. Due to this difference in sensitivity between the approaches, we default the sampling-base -aaend estimator for this choice of surrogate model, and demonstrate in Section 5 that this well-justified empirically.
3.2 Surrogate-model predictions
Another cost is that of making predictions with the surrogate model. In sampling- based active testing these predictions arise when computing the acquisition distribution, 9m gm), and in interpolation- active testing they additionally arise when computing the risk estimate.
Our choice to use a fixed surrogate model, To, automatically leads to a significant reduction in how many of these predictions are required: we only need to compute To ro(y/ar)) once for each x, E 'D pool On top of this we can make each individual forward pass cheaper by using a smaller surrogate model.
3.3 Target-model predictions
The third cost we aim to reduce is that of making predictions with the target model. The standard versions of both sampling-based and interpolation-base active testing require computing f f(xz for each zy E Ppoot which is especially expensive for large target models and pool sets.
This is unavoidable in the interpolation- based approach due to the construction of RASE- But in the sampling- based approach there is scope for an approximation that can reduce the number of target-model predictions from N to M, where often M < N. In particular, when computing the acquisition distribution, qm (i), we can use the surrogate model to approximate the target model's predictions on the pool set. This leads to new acquisition function,
im (x) = E (w/n) [[ [x)].)))
where yi denotes an operation that accounts for the fact that the first argument of l might not a probability distribution (e.g., if E v) = |/ Elli, then we require something like //|T.. (-1ær) = Ej (1=>) [8)). This acquisition function can be seen as an approximation to am (z) in Equation 2. E(p. y) logg(v) ((v) then å, (x) H H[m (g/ (.z)). the predictive entropy of the surrogate model.
3.4
Active testing for dataset curation
The use of the surrogate model to approximate the target model during data acquisition also makes possible to use active testing for dataset curation, where all the inputs in the pool set are already labelled and the goal is to reduce the number of target-model predictions required for evaluation (Maynez al, 2023; Polo al, 2024; Saranathan al, 2024; Vivek al, 2024). This can be viewed

---

as a variation on standard active testing where here we want to use the surrogate model to select M N input-label pairs with which to evaluate the target model. The acquisition function this case can depend on the labels as well as the inputs:
am (z. y) E( [y]=))] y)-
If we use the logarithmic loss before, this recovers the negative log likelihood of the surrogate.
4 Estimating risk-estimation error
Aside from its scalability, active testing poses a practical challenge in assessing the quality of the risk estimates it produces. In research we can construct repeatable simulations of active testing and record ground-truth risk-estimation errors across repeat runs. But practical deployment of active testing involves only single run, without knowledge of the true risk. This leaves practitioners unaware of how well their implementation is working, making it harder to justify using active testing.
We propose addressing this using novel estimator of the mean squared enror (over possible sequences of acquired indices, iıKK ((iix), where K < M) of NR.REE. For generic risk estimator, R((..)), we can decompose the mean squared error into squared bias and variance:
MSE(R)- Egginne [(R((i.))) - Rt)" = (E x) [[((ixk)] - R)² + V xx [((ix)) (5)
Bias(
Var(R)
Since Bias [.URE) O, our task is to estimate Var Var(R). MSE( (LRRE) from single se- quence acquired indices, i1.k. Letting -= (vmf(f (xx,,, Vi,,. (F...)) )== denote the first K reweighted losses used in Equation 1, bootstrap estimator (Efron, 1979) of the risk, R, is
K
1 K
Rbooo
in ~ 2 K). Lim:
VTA=1
If we have B bootstrap estimates, (Aooot b=11 we can form bootstrap estimator of the variance:
B Roooot Rboot) ) Rboot
Varboot (RLURE)=
Σ
Σ Rb
B
(6)
B
b=1
It is important note that Rboot is not standard bootstrap estimator because the reweighted losses, Lj. are not independent and identically distributed (they are the result of active data acquisition), and the bootstrap resampling process does not capture the dependencies between the losses. As result, the estimator lacks theoretical convergence guarantees. Nevertheless we find it to be reliably accurate in our experiments (Section 5.9), suggesting it could be useful in practice.
5 Experiments
We now seek to empirically assess our proposed approach for active testing of LLMs (Section 3), as well as our proposed risk-erron-estimation method (Section 4). We provide implementation details in Appendix B and code at github. com/
5.1
Setup
We use five cext-classification datasets: Stanford Sentiment Treebank 2 (SST-2; Socher al, 2013), Subjectivity (Subj: Pang Lee, 2004), Financial Phrase Bank (FPB; Malo et al, 2013), Hate Speech (HS; De Gibert et al, 2018) and Massive Multitask Language Understanding (MMLU; Hendrycks et al, 2021). We mainly focus on the first four, which were previously used by Kossen et al (2024), then MMLU to explore problem that is more challenging for the models used in our experiments.
The models we use span three model families and have parameter counts ranging from to billion. In our main experiments we use the 7B and 70B versions of Llama 2 (Touvron et al, 2023) along with Gemma3-4B (Kamath et al, 2025); in additional experiments we use Phi-2 (Abdin et al, 2023) and Gemma3- IB. We use these models with either zero- few-shot prompting, with both including
5

---

Sarmggtee U.AE-Ulm----7 .... LURE-lam2--e0
Unilerm random
Detasel
FPB
537-2
Suth
15
700-2ero
7B-zero
700-few
7B-few
Relative Error
Relative Error
Error
Relative
1.0
1.0
1.0
0.5
0.5
0.5
Number acquired labels
Number of acquired labels
Number of acquired labels
Number of acquired labels
(a) Ilama 2 target models and Ilama 2 surrogate models
Surrogatc
Gomma3
Unform random
Donasck
S
Sub)
70B-zaro
7B-2ero
708-fow
Relative Error
1.0
Relative Emro
Relative Eiror
1.0
Relative Errr
1.0
0.0
0.0
0.0
0.0
100
200
300
400
100
200
200
400
100
200
300
400
100
200
300
400
Number of acquired labets
Number of acquired labels
Number of acquired labels
Number aequired labels
(b) Llama 2 target models and Gemma3-4B surrogate model
Cheap surrogate models support effective data acquisition for active testing. We compare uniform- random testing to active testing (LURE) across four datasets. To guide active data acquisition we use surrogate model that we train using a single step of in-context learning and then keep fixed. This stripped- back surrogate- model training, along with the use of small surrogate models relative to the target models, allows us to drastically reduce the computational cost of active testing while achieving strong performance.
Figure 2
appropriate task prompting such "Classify the sentiment of the following sentence positive or negative". We denote our four Llama 2 model configurations as 7B-zero, 70B-zero, 7B-few and 70B-few, and our two Gemma 3 model configurations as Gemma3-4B-zen and Gemma3-4B-fer
We compare active testing to uniform-random testing, standard approach to model evaluation. Acquisition methods from the active- literature (Settles, 2012) are not tailored for risk estimation (Kossen et al, 2021) and so would have limited utility as baseline methods here.
We use a logarithmic loss, as discussed in Sections and 3. Like in past work (Kossen et al, 2021, 2022), measure the performance of testing method using the median squared error of its risk estimate across random- numbbrrumerrreeeeee---- seeds, where the ground -ound-truthhao risk is computed using a held-out test set. Alongside plots of the absolute value of this risk-estimation error, we present plots of relative error, namely the active risk risk-estimation error divided by the uniform-random lisk-estimation error. Visualising performance on this relative scale lets us see the benefits of active testing more clearly. When we use a few-shot surrogate model with n labelled in-context examples to evaluate a zero-shot target model, we ensure fair comparison to acquired labels to infform-randon results for + & acquired labels.
random testing by comparing our results for
5.2 Fixed surrogate models trained by in-context learning support effective data acquisition
First we investigate whether our switch from gradient-based training of the surrogate model single step of in-context learning still allows accurate risk estimation. We do this by running data acquisition across four datasets, four target models and three testing methods. In particular we compare cinioomm-nnnom testing against two versions of active testing, LURE-7B and LURE-70B, differing only in the surrogate model used (7B-few vs 70B-few).
We find that active testing performs very well relative to the standard uniform-random testing in the vast majority of cases (Figure 2a), with risk-estimation error lowered by 32% average (median over all the experimental variables listed above, as well as over possible label budgets between and 400). This is an exciting result: practically straightforward technique allows us improve on standard practice for LLM evaluation with minimal computational overhead.

---

Sammast
.... 1amt--lamm2-00
WREE 78
Unifom random
Gemma] --b--zero
Gemma3- feww-wwwwwwwww
1.00
Reletive Lrror
Relalive Exror
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
Number acquired labels
Number acquired labels
Figure 3
Older Llama 2 models useful surrogates for active testing of newer Gemma target models.
Uniform random
LURE 708
LURE-78 78
ASE-78
ASE 70B
Rsk-estimation error
708-few on
70B-few on SST-2
708-few on Subj
70B-few on H5
10 -1
10
10
10
10
10-1
10-3
10-1
10-3
10
10
200
400
200
400
200
200
400
Number of acquired labels
Number acquired labels
Number of acquired labels
Number acquired labels
Risk-estlmation error
78
Subj
7B
10-2
10-2
10
10
10-1
10-3
10-1
10-s
10-4
10
400
400
Number of acquired Number acquired labels
Number of acquired Number acquired labels
Our sampling- campling-based active testing (LURE) approach works more reliably than interpolation serpolation-bbaed active testing (ASE) when using cheaply constructed surrogate models. In ASE the surrogate model not only guides data acquisition but also much more directly affects the risk estimate: the labels used to compute the expected loss of the target model are simulated from the surrogate model.
Figure 4
Contrasting with the overall positive result in Figure 2a, LURE-70B performs poorly in evaluating the 70B-few model on the SST-2 dataset. We explore this failure case in Section 5.8 and show it is due to a large number of incorrect labels in the dataset. In Appendix C.5 we present an additional experiment that compares dynamic surrogate models to fixed ones: for the scenarios we study, dynamic surrogate models provide no accuracy benefits but incur prohibitive computational costs.
5.3 Small surrogate models can be used to evaluate larger target models
key additional result to highlight in Figure 2a is that LURE-7B works well not just for evaluating target model of the same size, but also for evaluating target model ten times larger. In fact, show that we can reduce surrogate-model costs even further by using Gemma3-4B (Figure 2b) and Phi-2 (Figure 10a) evaluate Llama models, reducing computational costs by using even smaller surrogates. Meanwhile, older Llama 2 models effectively evaluate newer Gemma 3 models (Figure 3), which perform strongly on these datasets (Tables 2 and 3). As discussed in Section 3.2, this ability to use small surrogate model-combing combined with the fact that fixing the surrogate model allows us to only compute one forward pass on the pool set-is useful way of reducing costs.
5.4 Using stripped-bas -back surrogate models favours sampling-base -baee active testing
Next we evaluate our claim in Section 3.1 that sampling sampling-based based active testing is preferable over interpolation-based active testing given that the surrogate model fixed. We do this by comparing our approach with ASE in evaluating the 7B-few and 70B-few models. Our results show ASE produces strong results on the HS dataset with the 70B-few model, but otherwise significantly underperforms (Figure 4). This is consistent with our explanation that ASE is more sensitive to the quality of the
7

---

Dataset
FPB
SST2
Sub)
Dataset
FPS
SST-2
Subj
H5
Surrogate
UURE Usma2 78
Unitom random
Surrogate
random
708-few
708-few
1.00
Relecive Error
Relative Error
0.5
0.25
0.00
0.0
50
200
150
200
250
300
350
400
Number acquired la babels
Number of acquired labels
Figure 5 Using the surrogate model to approximate the target model during data acquisition (here this means acquiring data based on the predictive entropy of the surrogate model) reduces the need for target- model predictions while still performing well.
Figure 6 Active testing with a label-aware acquisi- tion function (here the negative log likelihood of the surrogate model) enables dataset curation (selecting a subset of already- labelled test data) tailored target model of interest, helping reduce computational costs.
surrogate model, and that computational constraints on the surrogate model make it less practically useful. Note that the ASE risk estimator is constant the surrogate model fixed, as it here.
5.5 Good data acquisition is possible without making predictions with the target model
Now we explore our suggestion Section 3.3 that the surrogate model can be used to approximate the target model, such that we do not require target-model predictions during data acquisition. For the logarithmic loss we are using, this equates to using the predictive entropy of the surrogate model as the acquisition function. We assess this approach as applied to evaluating the 70B-few model with LURE-7B, and find it is surprisingly effective (Figure 5). This is promising: it suggests all the cost-reduction measures proposed in Section are compatible with strong active-testing performance.
5.6 Active testing for dataset curation works, suggesting scope to reduce computational costs
Here we consider the alternative problem setting discussed in Section 3.4, where the inputs in the pool set already labelled but is too expensive to compute target-model predictions all of them, with a possible solution being to curate a subset using a cheap surrogate model. More concretely we run active testing using the negative log likelihood our label-aware acquisition function (Equation 4).
We find our approach improves over mniform-randon subsampling on three datasets out of four (SST- 2, Subj and HS) and performs slightly worse on FPB (Figure 6). These everall-positee results- along with those for the predictive-entropy function we considered Section .ug-st that active testing could be useful tool for dataset curation, reducing computational costs when using existing evaluation datasets. The practical benefits of this could be significant: Liang et al (2023) required almost 20,000 hours of (Nvidia A100) GPU hours to evaluate 30 models on their HELM benchmark.
5.7
The benefit of active testing tends to hold in more challenging scenarios
Next we use the MMLU dataset to assess the robustness of active testing to an increase in task difficulty, which corresponds to worse surrogate-model performance (Tables 2 and 3) and thus poses challenge for effective data acquisition. Our results show active testing continuing outperform uniform random testing for most combinations of surrogate and target models, with the only failure occurring when using 70B-few for both the surrogate and target models (Figure 7).
5.8 Incorrect labels cause problems for active testing
Now we investigate the failure of LURE-70B to effectively evaluate the 70B-few model on the SST-2 dataset (Section 5.2). We construct two modified versions of the SST-2 pool set: one filtered to remove inputs for which 70B-few 's negative log likelihood (NLL) is greater than 5, and one filtered with NLL threshold of 3. Then we re-run active testing with these modified pool sets.
We show that the failure case is reduced by the weaker filtering and completely resolved by the stronger filtering (Figure 8). To understand why, we inspect the filtered-out data and find it is often
8

---

WRE Lama2 708
WURE 6emmaa348
Unfoom random
708-faw
708-zero
78-few
7B-zero
1.0
1.0
1.0
1.0
Relative
0.5
0.0
0.0
0.0
0.0
Number acquired labels
Number of acquired labels
Number acquired labels
Number of acquired labels
Figure? 7
Active testing outperforms random testing in most cases on the harder MMLU dataset.
LLaMa 708 ICL CE
All pool points
Paints with loss <
Points with a lass <
Risk eetimmattooon ertor
10-7
10-1
10-4
10->
10-3
10-1
200
300
100
Num. of acquired labels
Num. of acquired labels
Num. of acquired labels
Figure 8 The failure 70B efficiently evaluating the 70B-few model on the SST-2 datasct (left) is resolved by filtering incorrectly labelled datapoints from the pool set (middle, right). Underlying this effect is the presence of incorrect labels in the original dataset: we find that inputs with high loss are often mislabelled.
mislabelled. We find 75% of the 45 examples with NLL greater than are mislabelled, and 67% of the 136 examples with NLL greater than 3. This is useful demonstration of how active testing can fail when the source of labels unreliable. Interestingly it can also be seen as a case for using less powerful surrogate models guide data acquisition: LURE-7B worked well in the exact case where LURE-70B failed. It is also worth noting that in a dataset-curation setting (Section 5.6) we have access to all the test labels and therefore have the option directly filter out high-NLL examples.
5.9 Bootstrap estimation of risk-estimation error provides an accurate performance indicator
Having assessed our proposed active-testing method, we turn to our method for estimating the mean squared error of the LURE risk estimator (Section 4). We run active testing T 100 times, generating 100 sequences of acquired indices, (m Then for K {1,2, M we perform four steps:
1. Compute a ground-truth mean-squared error, MSEK, using Equation 5 with g(i1:)) defined as a uniform distribution over the K index sequences acquired far.
2. Compute bootstrap estimate of the MSE, MSFx, using Equation 6 with B 1000 estimates.
3. Compute the MSF-estimation error, (MSEx MSFx).
Compute an approximate confidence interval, MSEK 26, using estimated standard deviation 6.
Our results show MSE-estimation error converging to low values for most runs, after an initial period (K 100) of relatively high error (Figure 9). addition this, our confidence intervals provide high levels of coverage: for a strong majority of runs we see coverage probabilities of around 94% for K 2 100, suggesting practitioner can expect their computed confidence interval contain the true mean squared error approximately 94% of the time. While there are cases where MSE estimation fails, the overall performance is promising, suggesting our proposed method could be useful practical performance indicator, helping justify real-world deployment of active testing.
Related work
The sampling-based approach to active testing that our approach builds on was proposed by Kossen et al (2021), using risk estimators from Farquhar et al (2021). Earlier work on active testing proposed

---

Target model
Surrogate
LURE-LiamaZ-70
LURE-Uamaz-7g
FPB
H5
SST-2
Subj
10-3
10-l
10-4
10-3
10-3
10.
10:7
10
MSE
10-4
10-9
10-9
10-10
200
400
200
400
0
200
400
200
400
Number of acqui red labels
Number of acquired labels
Number of acquired labels
Number of acquired labels
Subj
Coverage probability
1.00
1,00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Number acquired labels
Number acquired labels
Number acquired labels
Number of acquired labels
Figure 9 Our bootstrap estimator of the active-testing risk-cstimation error provides a useful performance indicator. Coverage probability measures how often a confidence interval contains the true risk-cstimation error.
alternative methods. Bennett Carvalho (2010), Ji et al (2021), Katariya et al (2012) and Kumar & Raj (2018) explored stratification-bane bbased techniques, such splitting the pool set into strata (based on a measure of model confidence) and sampling uniformly within each stratum. Sawade et al (2010) and Yilmaz al (2021) proposed importance-sampligg and Poisson-sampling methods related to data
that of Kossen et al (2021) but without the latter's innovations of adaptive, surrogate- acquisition and risk estimation that accounts for sampling without replacement. Nguyen et al (2018) studied a special case of active testing focused on human vetting of noisy labels.
Recent years have introduced clustering- based approach that relies on making predictions with the target model on the pool set, which we have argued a barrier to scaling up. Su et al (2024) proposed technique tailored to "dense" recognition tasks in computer vision (e.g.. image segmentation and object detection). Yu et al (2024) incorporated active testing into the process of training model. Ashury-Tahan et al (2024) and Hara et (2024) proposed methods for model selection.
various extensions of this earlier active-testing work. Huang al (2025)
The idea of reducing model-evaluation costs by using carefully selected test datasets has been highlighted a number of times in recent work. Maynez et (2023) found that the datasets they studied could be substantially reduced in size, through uniform-random subsampling, while maintaining stable rankings of the models they were comparing. Polo et al (2024), Saranathan et al (2024) and Vivek al (2024) demonstrated more sophisticated methods for achieving the same goal. The goal in these studies was to take an existing test dataset (with labels for all inputs) and reduce in a way that supports comparisons between different models. This prior work is therefore complementary to contribution: we primarily study the problem of acquiring new data with which to evaluate a model of interest; and
method that tailors data acquisition to that given model.
7 Conclusion
We have argued that a significant barrier to principled active testing of large models is the computa- tional cost of deciding which labels to acquire. To address this we have identified key contributors to the cost test-training the surrogate model, and making predictions with the surrogate and target models- and have proposed straightforward cost-saving measures. We have shown that these measures are compatible with effective active testing, producing low-error risk estimates for large language models. top of this we have demonstrated an estimator of rist-estimatiinn accuracy that can provide an on-the- fly indication of how well active testing is working given practical problem. Overall we believe this represents substantial progress in more efficiently evaluating L.MSs.
10

---

Acknowledgements
Freddie Bickford Smith was supported by the EPSRC CDT in Autonomous Intelligent Machines and Systems (EP/L015897/11. Tom Rainforth was supported by EPSRC grant EP/Y037200/1.
References
Abdin, Aneja, Bubeck, Mendes, Chen, Giorno, Eldan, Gopi, Gunasekar, Javaheripi. Kauffmann, Lee, Li, Nguyen, Rosa, Saarikivi, Salim, Shah, Santacroce, Behl, Kalai, Wang, Ward, Witte, Zhang, & Zhang (2023). Phi-2: the surprising power of small language models. Microsoft Research Blog.
Ashury-Tahan, Sznajder, Choshen, Ein-Dor, Shnarch, & Gera (2024). Label-efficient model selection for text generation. Annual Meeting of the Association for Computational Linguistics.
Bennett Carvalho (2010). Online stratified sampling: evaluating classifiers at web-scale. Interna- lional Conference on Information and Knowledge Management.
Berger (1985). Statistical Decision Theory and Bayesian Analysis. Springer.
Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Vosse Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, & Amodei (2020). Language models are few-shot learners. Conference on Neural Information Processing Systems.
Burden (2024). Evaluating AI evaluation: perils and prospects. arXiv.
De Gibert, Perez, Garcia-Pablos, & Cuadros (2018). Hate speech dataset from a white supremacy forum. Workshop on Abusive Language Online.
Efron (1979). Bootstrap methods: another look at the jackknife. Annals of Statistics.
Farquhar, Gal, & Rainforth (2021). On statistical bias in active learning: how and when to fix International Conference Learning Representations.
Ganguli, Schiefer, Favaro, & Clark (2023). Challenges in evaluating Al systems. anthropic.com index/ evwllating-ai-sttieieeeeeeeeeeee -issssssss..
Hara, Matsuura, Honda, & Ito (2024). Active model selection: a variance minimization approach. Machine Learning.
Hendrycks, Burns, Basart, Zou, Mazeika, Song, & Steinhardt (2021). Measuring massive multitask language understanding. International Conference on Learning Representations.
Huang, Song, Hu, Juefei-Xu, & Ma (2025). Active testing of large language model via multi-stage sampling. ACM Transactions on Software Engineering and Methodology.
Ji, Logan, Smyth, & Steyvers (2021). Active Bayesian assessment of black-bol classifiers. AAAI Conference Artificial Intelligence.
Kamath, Ferret, Pathak, Vieillard, Merhej, Perrin, Matejovicova, Ramé, Rivière, Rouillard, Mesnard, Cideron, Grill, Ramos, Yvinec, Casbon, Pot, Penchev, Liu, Visin, Kenealy, Beyer, Zhai, Tsitsulin, Busa-Fekete, Feng, Sachdeva, Coleman, Gao, Mustafa, Barr, Parisotto, Tian, Eyal, Cherry, Peter, Sinopalnikov, Bhupatiraju, Agarwal, Kazemi, Malkin, Kumar, Vilar, Brusilovsky, Luo, Steiner, Friesen, Sharma, Sharma, Gilady, Goedeckemeyer, Saade, Feng, Kolesnikov, Bendebury, Abdagic, Vadi, György, Pinto, Das, Bapna, Miech, Yang. Paterson, Shenoy, Chakrabarti, Piot, Wu, Shahriari, Petrini, Chen, Lan, L., Choquette-Choo, Carey, Brick, Deutsch, Eisenbud, Cattle, Cheng, Paparas, Sreepathihalli, Reid, Tran, Zelle, Noland, Huizenga, Kharitonov, Liu, Amirkhanyan, Cameron, Hashemi,
Singh, Mehta, Lehri, Hazimeh, Ballantyne, Szpektor, Nardini, Pouget-Abadie, Chan, Stanton, Wieting, Lai, Orbay, Fernandez, Newlan, Ji, Singh, Black, Yu, Hui, Vodrahalli, Greff, Qiu, Valentine, Coelho, Ritter, Hoffman, Watson, Chaturvedi, Moynihan, Ma, Babar, Noy, Byrd, Roy, Momchev, Chauhan, Sachdeva, Bunyan, Botarda, Caron, Rubenstein, Culliton, Schmid, Sessa, Xu, Stanezyk, Tafti, Shivanna, Wu, Pan, Rokni, Willoughby, Vallu, Mullins, Jerome, Smoot, Girgin, Iqbal, Reddy, Sheth, Pöder, Bhatnagar, Panyam, Eiger, Zhang,

---

Liu, Yacovone, Liechty, Kalra, Evci, Misra, Roseberry, Feinberg, Kolesnikov, Han, Kwon, Chen, Chow, Zhu, Wei, Egyed, Cotruta, Giang. Kirk, Rao, Black, Babar, Lo, Moreira, Martins, Sanseviero, Gonzalez, Gleicher, Warkentin, Mirrokni, Senter, Collins, Barral, Ghahramani, Hadsell, Matias, Sculley, Petrov, Fiedel, Shazeer, Vinyals, Dean, Hassabis, Kavukcuoglu, Farabet, Buchatskaya, Alayrac, Anil, Lepikhin, Borgeaud, Bachem, Joulin, Andreev, Hardin, Dadashi, & Hussenot (2025). Gemma technical report. arXiv.
Katariya, Iyer, & sarawagi (2012). Active evaluation of classifiers on large datasets. International Conference Data Mining.
Kossen, Farquhar, Gal, & Rainforth (2021). Active testing: sample-efficient model evaluation. International Conference on Machine Learning.
Kossen, Farquhar, Gal, & Rainforth (2022). Active surrogate estimators: an active learning approach to label-efficient model evaluation. Conference on Neural Information Processing Systems.
Kossen, Gal, Rainforth (2024). In-context learning learns label relationships but is not conventional learning. International Conference on Learning Representations.
Kumar & Raj (2018). Classifier risk estimation under limited labeling resources. Pacific-Asia Conference on Knowledge Discovery and Data Mining.
Lewis & Gale (1994). A sequential algorithm for training text classifiers. Conference on Research and Development in Information Reirieval.
Liang, Bommasani, Lee, Tsipras. Soylu, Yasunaga, Zhang, Narayanan, Wu, Kumar, Newman, Yuan, Yan, Zhang, Cosgrove, Manning, Ré, Acosta-Navas, Hudson, Zelikman, Durmus, Ladhak, Rong, Ren, Yao, Wang, Santhanam, Orr, Zheng, Yuksekgonul, Suzgun, Kim, Guha, Chatterji, Khattab, Henderson, Huang, Chi, Xie, Santurkar, Ganguli, Hashimoto, Icard, Zhang, Chaudhary, Wang. Li, Mai, Zhang, & Koreeda (2023). Holistic evaluation of language models. Trunsactions on Machine Learning Research.
Malo, Sinha, Korhonen, Wallenius, & Takala (2013). Good debt bad debt: orientations economic texts. Journal of the Association for Information Science and Technology.
detecting semantic
Maynez, Agrawal, & Gehrmann (2023). Benchmarking large language model capabilities for conditional generation. Annual Meeting of the Association for Computational Linguistics.
Nguyen, Ramanan, & Fowlkes (2018). Active testing: an efficient and robust framework for estimating accuracy. International Conference on Machine Learning.
OpenAl (2023). GPT-4 technical report. arXiv.
Pang & Lee (2004). A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. Annual Meeting of the Association for Computational Linguistics.
Polo, Weber, Choshen, Sun, Xu, & Yurochkin (2024). tinyBenchmarks: evaluating LLMs with fewer examples. International Conference Machine Learning.
Saranathan, Alam, Lim, Bhattacharya, Wong, Foltin, & Xu (2024). DELE: data efficient LLM evaluation. Workshop on "Navigating and Addressing Data Problems for Foundation Models", International Conference Learning Representations.
Sawade, Landwehr, Bickel, Scheffer (2010). Active risk estimation. International Conference on Machine Learning.
Settles (2012). Active Learning. Morgan and Claypool.
Socher, Perelygin, Wu, Chuang, Manning, Ng, & Potts (2013). Recursive deep models for seman- tic compositionality over sentiment treebank. Conference on Empirical Methods in Natural Language Processing.
Su, Li, Doan, Behpour, He, Gou, Miao, & Ren (2024). MetaAT: active testing for label-efficient evaluation of dense recognition tasks. European Conference on Computer Vision.
12

---

Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov. Batra, Bhargava, Bhosale, Bikel, Blecher, Ferrer, Chen, Cucurull, Esiobu, Fernandes, Fu, Fu, Fuller, Gao, Goswami, Goyal, Hartshorn, Hosseini, Hou, Inan, Kardas, Kerkez, Khabsa, Kloumann, Korenev, Koura, Lachaux, Lavril, Lee, Liskovich, Lu, Mao, Martinet, Mihaylov, Mishra, Molybog, Nie, Poulton, Reizenstein, Rungta, Saladi, Schelten, Silva, Smith, Subramanian, Tan, Tang, Taylor, Williams, Kuan, Xu, Yan, Zarov, Zhang, Fan, Kambadur, Narang, Rodriguez, Stojnic, Edunov, & Scialom (2023). Llama open foundation and fine-tuned chat models. arXiv.
Vivek, Ethayarajh. Yang, & Kiela (2024). Anchor points: benchmarking models with much fewer examples. Conference of the European Chapter of the Association for Computational Linguistics.
Yilmaz, Hayes, Habib, Burgess, & Barber (2021). Sample efficient model evaluation. arXiv.
Yu, Shi, & Yu (2024), Actively testing your model while it learns: realizing label-efficient learning in practice. Conference on Neural Information Processing Systems.
13

---

NeurIPS paper checklist
1. Claims
Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We support the claims in Sections 3 to 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We highlight failure cases in Section 5, particularly in Section 5.8.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-speciffcation, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-tent system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
14

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide pseudocode in Algorithmss I and 2, implementation details in Appendix and code at gi git..b.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
be accomplished in various ways. For example. if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
Depending the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide reproducibility. In the case of closed-soure models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide pseudocode in Algorithms 1 and 2, implementation details in Appendix B and code at gi github.
aeaabrieleeeerrrdaaaallllal. wwwpiiinnu--aati-teeetiieee All datasets and models used in our experiments are open-source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
15

---

Please see the NeurIPS code and data submission guidelines http::::://iis.cc//ppppposp for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:// Seaedddee/ooeuuumissseeee
nips. cttccc//puuu wtcpcpulccc//cccccccceeceeecclcl
comPolicyy cy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify data splits, hyperparameters and other implementation details in Ap- pendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars and coverage probabilities in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show tables or figures symmetric error bars that would yield results that are out of range (c.g, negative emror rates). If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
16

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification:
We provide information about computational resources in Appendix B.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics btttps://neurr: wttppippp.cccpppeeiieiieee cc/ppbblic/tthiccsuuueeeeeiee
Question:
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that this research adheres the code all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
17

---

11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used in paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit the creators of the datasets and models we use in Section 5 and Appendix B. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
18

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on requirements of your country or institution) were
obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non- standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
'The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy (https pectts://netnee /ttpprip///urss.... htt.://ppp..///eeeeeee////////// for what should or should not be described.
19

---

A Impact statement
This paper presents work whose goal is to generally advance label- and compute-efficient model evaluation. As this is not limited to particular area of application, it is difficult to evaluate the impact of this work. Generally speaking, we believe our work could play role in reducing the computational costs, and thus the CO2 footprint, associated with model evaluation. As is often the case with research, we would caution against careless use of our method in real-world applications.
B Experiment details
B.1 Datasets
Our experiments are based on five datasets: Stanford Sentiment Treebank 2 (SST2: Socher et al, 2013; unknown license), Subjectivity (Subj; Pang & Lee, 2004; Creative Commons Attribution 4.0 International License), Financial Phrase-bamt (FPB; Malo et al, 2013; Creative Commons Attribution Non Commercial Share Alike 3.0 Unported License), Hate Speech (HS; De Gibert al, 2018; Creative Commons Attribution-Shart Alike 3.0 Spain License) and Massive Multitask Language Understanding (MMLU; Hendrycks et al, 2021; MIT license).
Each method, including uniform-random testing, acquires data from the same pool set (a set of candidate data for acquiring, with labels hidden until they are acquired), although its size may vary between experiments. We compare risk estimates against a ground separate test dataset in all cases except for FPB, where we the pool set due to the small size of the original dataset. Table summarizes sizes of the pool and test sets for each dataset.
risk, which is computed on
SST-2
Subj
FPB
HS
MMLU
Pool set
10,000
6,000
2,200
6,000
7,000
Test set
10,000
4,000
4,000
7,000
Table Pool-set and test-set sizes each dataset. Sets are randomly sampled so that they are disjoint.
Stanford Sentiment Treebank 2 (SST-2) The Stanford Sentiment Treebank dataset consists of 69,000 duman-annotuue sentences extracted from movie reviews. Each sentence is labeled as either "positive" or "negative", providing benchmark for assessing sentiment-analyy capabilities. We use the predefined split of the dataset, concentrating the training set, which contains 56% positive and 44% negative labels. We randomly select two subsets of 10,000 sentences each from the training set to form the pool and test sets. The instruction used for this dataset is
Classify the sentiment Respond with "positive" or "negative". \n.
following sentence as "positive" or "negative".
The string " "Answer" is replaced by "Label" for this dataset only.
Subjectivity (Subj) The Subjectivity dataset is composed of 5,000 subjective sentences from movie reviews and 5,000 objective sentences from plot summaries. This dataset is used determine sentiment polarity. The task is to classify each sentence as either "subjective" or "objective", This dataset is balanced, with 49.5% of objective and 50.5% of subjective sentences. We randomly split the 10,000-sentence train set into a 6,000-sentence pool set and 4,000-sentence test set. Examples for leetext--eeaming are selected from the separate 2,000-sentence test set. The instruction used for this dataset is
Is the following sentence 'objective' or 'subjective' .Respond with 'objective" or "subjective \n.
Financial Phrase-bank (FPB) The Financial Phrase-bant bank dataset consists of 4,800 English finan- cial news articles that were classified as "positive", "neutral" and "negative" by human experts. We select the training subset of 2,200 sentences for which all 16 annotators agreed. This set contains 13% of negative, 25% of positive and 61 % of neutral sentences. Due to its reduced size, the whole set
20

---

is used as the pool set, and the approximation of the exact loss is the loss over the whole set. The instruction used for this dataset
Classify the sentiment of the following sentence as "negative", "neutral" or "positive". Respond with "negative", "neutral" or "positive". In
Hate Speech (HS) The Hate Speech dataset is composed of 9,900 sentences extracted from Stormfront, white-supremacist forum, from which we select the 9,600 sentences which are classified as "hate" or "no hate" to obtain a binary-classificttinn task. The dataset is highly unbalanced, with 11% classified hate speech and 89% non-hate speech. We randomly split this dataset into pool set of size ~6,000 and test set of size ~4,000 The instruction used for this dataset is
Does the sentence contain hate speech? Respond with "yes" "no". A..
Massive Multitask Language Understanding (MMLU) The Massive Multitask Language Un- derstanding dataset is composed of 1 116,000 sentences across 57 tasks asssing world knowledge and problem-solving. Each question is multiple-choice, with four choices available. We randomly split this dataset into pool set of size 7,000 and test set of size 7,000. The instruction used for this dataset is
Answer the question with the correct letter. Respond with only 'A', 'B', CC' or 'D', In.
B.2 Models
We use the 7B and 70B models from the Llama 2 family (Touvron et al. 2023; Llama Community License). We bit-quantisation for the 7B model and half- pelfi-rreiiion floating- sraiing-pinntee numbers for the 70B model; Kossen et al (2024) showed that these approximations do not significantly affect performance. We additionally use Gemma-3 4B (Kamath al, 2025; Gemma License).
B.3 Prompt formatting
To evaluate a model on a classification task, we follow the formatting guidelines from Kossen er al (2024) in how we generate the input sentence. We begin with a dataset-specific instruction, such as
Classify the sentiment of the following sentence as [[label1] or "[labe12]" Respond with [label1] or [1abe12] ". \n.
Then we introduce the sentence be classified and ask for the corresponding label. The prompt is formatted as
Sentence: [sentence]]
nnnswer:
B.4
In-context examples
We randomly select in-context examples such that each class is represented proportion to actual proportion in the dataset, which we found improved the model's accuracy. These examples are included in the input between the instruction and the prompt. Each input example is formatted as
Sentence: [sentence] \nAnswer: [label] in)n.
All few-shot models are given 50 in-context examples for all datasets, except for the MMLU dataset, for which models receive only 5 examples due to token limit. Examples are ordered randomly once and fixed for all evaluations, ensuring that all models receive the same context. The of in-context examples is therefore fixed beforehand and is common to both the target model and the surrogate model, for label efficiency and fair comparison.
B.5 Generating and processing model outputs
To obtain deterministic, reproducible token generation, we set the maximum number of tokens to 1, do set top-k top-p value and output the logits directly, from which we compute probabilities.
21

---

For each input, the model outputs a logit value for each token in the vocabulary. We select the logits corresponding the labels of the dataset and apply the softmax function to obtain final probability distribution over the possible labels. Any logit that does not correspond to one of the labels is ignored.
In cases where model represents task label with multiple tokens (Kossen al, 2024), select only the relevant token that corresponds to the core of the word. For instance, the Llama 2 tokeniser encodes the word "objective" [12091] but "subjective" [4967, 573J, which are tokens for "subject" and "ive". In this case we select [12091] for "objective" and [4967] for "subjective".
B.6 Data acquisition
Following Kossen al (2021), clip values of the acquisition function ensure inputs and thus guarantee that the LURE is unbiased. All acquisition probabilities below a = 0.1 times the probability corresponding to mnfoorm-random acquisition brought up to this limit value.
B.7 Computational resources
Our experiments are designed to be computationally efficient and therefore do not require substantial resources. Generating model outputs over the pool set is the main computational cost, typically requiring two GPUs for Llama 70B and one GPU for all other models used in this work. Aside from this, the active-testing procedure itself can be run efficiently small number of CPUs.
C
Additional results
C.1
Model performance
Here present accuracy and loss values for Llama2-7B, Llamma-700 and Gemma3-4B the five datasets we study. Results for Llama here match those published in Kossen et (2024).
Dataset
L-7B-zero
L-7B-few
L-77B-zeero
L-70B-few
G-4B-zero
G-4B-few
FPB
66.75
90.60
26.25
94.15
29.70
91.00
SST-2
63.96
92.19
76.80
93.60
57.79
91.68
Subj 49.38
89.72
54.85
96.12
50.40
93.73
HS
11.38
89.28
84.18
90.23
42.87
89.42
MMLU
35.41
42.16
60.80
65.34
52.61
$7.83
Table
Accuracy (%) of Llama (I.) 7B and 70B models and Gemma3-4B (G) model evaluated pool sets.
Dataset
L-7B-zero
L-7B-few
L-70B-zero
L-70B-few
G-4B-zero
G-4B-few
FPB
0.7847
0,3084
1.3123
0,2152
1,4695
0.2701
SST-2
0,5639
0.2050
0,5808
0.1828
0,6234
0.2107
Subj 0.7326
0.3278
0,5873
0.1235
0.7101
0.2741
HS
0.9129
0.2979
0,5386
0,2166
0.7245
0.2678
MMLU
1.3468
1.2568
0.9217
0.8384
1.0920
0.9683
Table
Loss of Llama (L) 7B 70B models and Gemma3-4B (G) model evaluated on pool scts,
C.2 Failure mode on Subjectivity
Here build our failure analysis in Section 5.8 (where we investigated why LURE-70B failed for the 70B-few target model on the SST-2 dataset), now focusing on the underperformance of LURE-7B for the 70B-few target model on the Subj dataset. We find that filtering out high-loss examples does not improve performance like it did in the SST-2 case, but we do find some insight from another
22

---

Dataset
70B (T) 70B (S)
70B (T) & 7B (S)
7B (T) & 70B (S)
7B (T) & 7B (S)
FPB
0.628
0.392
0.707
0.588
SST-2
0.285
0.224
0.527
0,454
Subj
0.364
-0.025
0.802
0.652
HS
0.461
0.146
0.555
0.384
Table Pearson's correlation coefficient between the cross-entropy of the surrogate (S) and target (T) models' predictions and the negative log likelihood of the target model's predictions. Low negative correlation indicates poor alignment between surrogate-based acquisition and optimal acquisition. Failure cases in bold.
line of investigation. In particular we compute the Pearson correlation coefficient between the cross- entropy-based acquisition scores and the optimal acquisition function (negative log likelihood of the target model's predictions), with this correlation quantifying how well surrogate- acquisition approximates optimal acquisition. Notably this the only setting where the correlation close to and is negative. This indicates that the surrogate model provides meaningful guidance, helping to explain why active testing performs similarly to uniform-random testing in this case.
C.3 Smaller surrogate models on the four core datasets
70B zeno
7Bizero
70B-tew
7B-rew
Relatve Error
1.5
1.5
Reistve Ener
1.0
Ralative
Number acquired labels
Number of acquired labals
Number of acquired labels
Number scquired labels
(a) Llama 2 target models and Phi-2 surrogate model
Gemman--1
Uniliim
706 zero
78 zero
708 few
7B ftw
15
Rosbive Entr
Relative
2.0
Belabrve
1.2
Number of acquired labels
Number of acquired labels
Number of acquired labels
Number acquired labels
(b) Llama 2 target models and Gemma3- IB surrogate model
Figure Even very small surrogate models can support effective active testing simple cases, although they struggle with evaluating our best pest-rerforiin target model, 70B-few.
Here we explore surrogate models that smaller than those used in Section 5: Phi-2 2.7B (Abdin et al, 2023; MIT License) and Gemma3- 1B (Kamath et al, 2025; Gemma License). We see both surrogate models producing strong performance in evaluating 70B-zero and 7B-zero, and more mixed performance in evaluating the two stronger target models, 70B-few and 7B-few (Figure 10),
C.4 Smaller surrogate model on MMLU
Here revisit Gemma3-1B 1B
surrogate model, the harder MMLU dataset. We see active testing failing to outperform uniform-random testing (Figure 11). This can be explained by how poorly Gemma3-1B 1B performs MMLU: it achieves approximately 25% accuracy, the base accuracy produced by uniform-randon prediction. A weak surrogate model can thus undermine active testing.
23

---

UPP Peers:315
cero on MNLU
700 cero SMUU
TB--ww on MUALU
702-1om SNLJ
Nal--othaaiin oror
Bricentmation eeroo
tort
31-2
10.2
23
232
383
200
150
106
283
Wimber of arquined luzelt
Number of acquleed Ixhelt
Sumber of aaqplred labe-
Number ofacquim babel
Figure A sufficiently weak surrogate model can cause active lesting to consistently fail on MMLU.
C.5 Dynamic-surrogate active testing
Here we explore the use of dynamic surrogate models, in contrast with the fixed surrogate models we use in our approach. In the dynamic approach the surrogate model's context incorporates newly acquired labels, leading to recomputed predictions, and thus novel acquisition probabilities each sampling step. Our experiments focus few-shot Llama2-7B and Llama2-70B target models with dynamic and fixed Llama2-7B surrogate models. We start with 10 in-context examples and run 40 steps of data acquisition. We run for 50 random-number seeds for dynamic-surrogate active testing, and for 3,000 seeds for ixed-surrogate active testing and uniform-randow active testing.
Despite being much more computationally expensive, Synamic-surrooge active testing shows no consistent improvement in risk estimation over fixed-suroggate active testing (Tables 5 and 6). Thus, while dynamic surrogate models might be useful elsewhere, the results we see for our selection of datasets support our design choice of fixing the surrogate model: it improves over uniform-randon testing while maintaining low computational cost and thus readily scaling to L.M evaluations.
Dataset
Testing method
Step 10
Step 20
Step 30
Step
91.41
45.49
29.25
20.66
Uniform random
(87.04, 95.86)
(43.53, 47.52)
(28.00, 30.51)
(19.82, 21,52)
63.25
22.79
13.52
8.62
SST-2
LURE-7B dynamic
(36.23, 96.67)
(14.56, 32.91)
(8.93, 18.80)
(5.82, 11.88)
42.71
19.99
12.42
8.74
LURE-7B fixed
(40.32, 45.11)
(19.07, 20.96)
(11.87, 13.00)
(8.38, 9.11)
135.75
65.14
40.06
27.99
Uniform random
(130.00, 141.72)
(62.37, 68.05)
(38.41, 41.76)
(26.84, 29.16)
65.73
36.40
18.33
11.53
FPB
LURE-7B dynamic
(45.79, 86.86)
(25.31, 49.18)
(11,82, 25.79)
(7.37, 16.43)
77.54
36.88
22.97
16.26
LURE-7B fixed
(73.39, 81.94)
(35.22, 38.61)
(21.99, 23.95)
(15.56, 16.96)
1.56.59
96.51
77.32
66.84
Uniform random
(152.02, 161.10)
(93.78, 99.32)
(75.14, 79.54)
(64.93, 68.76)
121.45
75.61
52.12
43.83
HS
LURE-7B dynamic
(100.99, 142.89)
(60.04, 91.73)
(40.48, 64.84)
(35,35, 52.63)
144.44
89.65
72.62
63.67
LURE-7B fixed
(139.38, 149.87)
(87.31, 91.98)
(70.73, 74.47)
(62.07, 65.31)
46.97
21.79
13.76
10.23
Uniform random
(44.97, 49.00)
(20.89, 22.70)
(13.19, 14.36)
(9.80, 10.66)
71.76
35.25
36.75
30.94
Subj
LURE-7B dynamic
(50.93, 95.54)
(20.62, 53.32)
(21.11, 55.19)
(18.43, 46.09)
40.20
19.69
12.95
9.12
LURE-7B fixed
(38.39, 42.02)
(18.89, 20.48)
(12.42, 13.50)
(8.73, 9.52)
Table
Mean squared (multiplied by 10 mean first row; 5th and 95th percentile second row) for uniform-random testing, dynamic-surrogate active testing and fixed-surrogate active testing of 7B-few.
24

---

Dataset
Testing method
Step
Step 20
Step 30
Step 40
81.75
40.04
25.01
17.92
Uniform random
(77.77, 85.85)
(38.24, 41.87)
(23.98, 26.06)
(17.19, 18.66)
28.76
17.21
12.62
8.57
SST-2
LURE-7B dynamic
(22. 13, 35.70)
(11.42, 23.89)
(8.83, 16.74)
(6.37, 11.07)
40.09
19.11
12.26
8.44
LURE-7B fixed
(38.28, 41.98)
(18.35, 19.89)
(11.80, 12.73)
(8.12, 8.76)
173.90
80.36
50.14
35.07
Uniform random
(164.57, 183,50)
(76.55, 84,22)
44.77..52.54
(33.40, 36.83)
114.11
34.89
19.71
11.10
FPB
LURE-7B dynamic
(84.88, 144.11)
(25.00, 46.22)
(13.20, 27.05)
(7.68, 14.95)
150.45
70.59
45.85
33.25
LURE-7B fixed
(140.12, 161.06)
(66.56, 74.69)
(43.59, 48.21)
(31.64, 34.85)
98.22
48.73
33.04
25.27
Uniform random
(94.33, 102.28)
(46.95, 50.54)
131.84, 34.29)
(24.36, 26,20)
88.73
41.15
23.93
16.18
HS
LURE-7B dynamic
(64.35, 116.8)
(31.37, 51,26)
(18.57, 29.54)
(12.40, 20.03)
74.57
39.31
26.87
19,93
LURE-7B fixed
(71.50, 77.75)
(37.78, 40.87)
(25.85) 27. 27.89)
(19.18, 20.69)
109.53
66.96
51,41
42.97
Uniform random
(105.47, 113.54)
(64.75, 69.11)
(49.77,53.100
(41.59, 44.36)
209.81
134.48
72.61
64.43
Subj
LURE-7B dynamic
(161.39, 260.62)
(99.07, 172.06)
(54.29, 92.06)
(48.04, 82.24)
134.87
77.31
57.14
47.24
LURE-7B fixed
(129.79, 140.22)
(74.56, 80.06)
(55.20, 59.07)
(45.64, 48.81)
Table 6
Mean squared error (multiplied by 10"; mean first row; 5th and 95th percentile on second row) for umiform-randowe testing, dynamic- surrogate active testing and fixed- fixed-surrogate active testing of 70B-few.
25

---

C.6
Risk-estimation values
In addition the plots shown in Section we present numerical values for the risk-estimation error of uniform-randone testing and active testing (with the cross-entropy acquisition function) at acquisition steps 50, 100, 200, 300 and 400 in Tables to 11.
Target model
Testing method
Step 50
Step 100
Step 200
Step 300
Step 400
Uniform random
24.3373
11.5663
5.1944
3.4854
2.4122
LURE-Llama2-7B
8.0511
3.6772
1.7589
1.0463
70B-zero
LURE-Llama2-708
4.2225
2.0045
0.98390
0.64870
7.1132
3,5176
1.6571
1.0540
Uniform random
9,4687
4.8302
2.1823
1.4524
1.0075
LURE-Llama2-78
2.5167
1,3094
0.59430
0,37380
7B-zero
LURE-LIama2-P0
1.9294
0,96850
0.42460
0.27060
LURE-Gemma3-4B
1.1421
0.5849
0.3681
0.2341
Uniform random
10.5749
5.16310
2. 2,48680
1,51150
0.977800
LURE-Llama2-7
6,3984
3,2058
1.3719
0.89110
0,63880
70B-few
LURE-Llama2-70k
5,0757
2.5057
1.1921
0.76740
0.54300
LURB-Gemma3-4B
5.0390
2.6267
1,3102
0.8344
0.5825
Uniform random
14.3272
7.2276
3,4608
2.3147
1.5246
LURE-Llama2-7B
8.9252
4.5853
2.3314
1.5486
1.1345
7B-few
LURE-Llama2-708
4.8174
2.1675
1.1186
0.65290
0.45770
LURE-Gemma3
6,5254
3.7684
1.9489
1.2371
0.8556
Table 7
Risk-estimation error (multiplied by 10"; minimum per target model in bold) for the FPB dataset.
Target model
Testing method
Step 50
Step 100
Step 200
Step 300
Step 400
Uniform random
2.1154
1,0992
0.56060
0.34910
0.26150
LURE-LIama2-77
0,5386
0.3089
0.1551
0.1023
70B-zero
LUREE-.lama2-708
0.4191
0.2308
0.1255
0.08190
JURE-Gemma3s
0.5283
0.2790
0.1542
0.1042
Uniform random
12.4455
5.49
2.9545
1.9694
1,362
LURE-Llama2-77
2.7160
1,4229
0.66190
0.48410
7B-zero
LURE-Llama2-77 2-0B
2.3740
1.2437
0.60510
0,42530
LURE-Gemma3-44
3,5371
1.6843
0.8699
0.6062
Uniform random
36.6017
16.8825
9,4717
6.3474
4,3089
LURE-Llama2-1
24.8052
14.7678
7.9162
5.2615
4.0596
70B-few
AURE-Llama2-700
40.8407
30.077
21,3892
17.1808
14.7939
4URF-GGemma3-4B
26.3377
16.8812
9.4499
7.1826
5.7175
Uniform random
19.5177
9.5602
4.9347
3.1286
2.2497
LURE-Llama2-7B
14.2747
8.3696
4.6265
3.2263
2.3764
7B-few
L.RRE-Llamaa2-70
9.7216
6,4914
3,6886
2.8971
2.1722
LURE-Gemma3-ae
13.3504
7.0426
3,9896
2.8028
1.9958
Table Risk-estimation emror (multiplied 10"; minimum per target model bold) for the SST-2 dataset.
26

---

Target model
Testing method
Step 50
Step 100
Step 200
Step 300
Step 400
Uniform random
1.2104
0.64740
0.30890
0.21 120
0.15430
0,4688
0.2311
0.1113
0.07570
70B-zero
LURE-Llama2-700
0.139
0.0758
0.0373
0.0251
0,3809
0.1897
0.0887
0.0638
Uniform random
9.7212
5.1028
2.4378
1,5612
1.1429
LURE-Llama2-73
3,4150
1.6835
0.87130
0,58730
7B-zero
LURE-L.lama2-000
1.6958
0.89970
0.44150
0,29280
LURE-Gemma3-44
2.8896
1,4536
0.7144
0.4787
Uniform random
15.5609
6,6315
3.7465
2.5614
2.1347
LUREE-.lama2-7
16,4075
8.2611
4,4942
3.1802
2.1987
70B-few
TIRE-Llama2-777
8,2473
5,0366
3,0052
2.1589
1.6293
AURE-Gemma3-44
12.615
6,7789
3,3058
2,2544
1.7645
Uniform random
7.4461
3.9287
1.8132
1.1308
0.87000
LURE-Llama2-1 7B
5.2522
2.6268
1.2944
0,84500
0.63670
7B-few
LURE-LLama2--001
1.8539
1.0921
0.62250
0,42070
0,30200
AURF-GGemma--4B
4.1246
2.1545
1.0256
0.6868
0.4941
Table Risk-estimation eror (multiplied by 10"; minimum per target model in bold) for the Subj dataset.
Target model
Testing method
Step
Step
Step
Step 300
Step 400
Uniform random
2.3826
1.1493
6.3630
0.39690
0.30590
LURE-Llama2-78
0.8497
0,4371
0.2247
0.1524
70B-zero
LURE-LLama2-705
0,5703
0.2996
0.1418
0.09900
LURE- Gemm-G-mmm3--
0.7508
0,3700
0.1822
0.1330
Uniform random
4.3455
2.3304
1.3545
0.96420
0.80470
LURE-Llama2-
3,8653
1,9395
1.1326
0.86310
7B-zero
LURE-Llama2-708
3.0157
1.7693
0.94190
0.73000
LURE-Gemma3-4p
3.5508
1.6981
0.9918
0.7766
Uniform random
24.5497
11.8593
5.8947
4.263
2.897
LURE-Llama2-IB
26.4253
13.5746
6,9755
4.249
3,0014
70B-few
LURE-LLama2-70B
19.8693
11.2704
6,1626
3,983
2,9785
LURE-GGema3344
19.4658
10.4853
5,473
3.6247
2.767
Uniform random
23.7706
11.7404
6,6989
4.6215
3,4943
LURE-Llama2-7B
19.6371
9.638
5,8037
4.0226
3.1452
7B-few
LURE-LLama2-1 13.4897
7.5196
4.1388
2.7654
2.2691
LURE-Gemma3-44
16.6135
8.7585
4.8131
3.4505
2.8044
Table 10 Risk-estimation envrr (multiplied by 10"; minimum per target model in bold) for the HS dataset.
27

---

Target model
Testing method
Step 50
Step 100
Step 200
Step 300
Step 400
Uniform random
45.2305
22.0405
11,5532
7.2541
5.288
70B-zero
AURE-Llamen
57.137
28.4008
13.3008
8,4398
6.7655
LURE-Llama2-1
30,4079
17.3273
9.5205
6.6253
4.9939
LURE-Gemma3-44
30.894
16.3988
9.0229
5.6838
4.1935
Uniform random
8.1557
4.1386
2.1924
1.4752
1.1799
LURE-Llama2-7B
6.6835
3.2991
1.6763
1.2791
0.9767
7B-zero
LURE-Llama2-7001
4.3678
2.6164
1.3063
0.9088
0.7330
LURE-Gemma3-ee
4.9507
2.6543
1,4700
0.9569
0.7622
Uniform random
80.9096
41.8724
20.7615
12.5686
8.9944
LURE-Llama2-78
94.483
51.3035
27.2732
17.5476
11.713
70B-few
LURE-
AURE-Llama2-701
84,9646
45,9222
23,3469
15.7314
12.5486
LURE-Gemma3-4B
71.571
34,8396
19,3527
12.9035
9,2558
Uniform random
20,5838
10.142
4,8003
3.0952
2,4424
L.URE-Llama2-78
16.2533
8.686
4,5997
3.047
2,2239
7B-few
LURE-Llama2-708
7.242
4.3427
2.1991
1.4639
1.1166
-B
9.6443
4,9703
2.3827
1.694
1.2064
Table 11 Risk-estimation emrrr (multiplied by 10"; minimum per target model in bold) for the MMLU dataset.
28


Reviews:
[{"review_id": "A7MtTBw1rS", "reviewer": "Reviewer_x1Pz", "summary": "The paper addresses the problem of the increasing cost of evaluating frontier LLMs. The authors revisit active testing—which selectively sends the most informative test inputs for human-labelling—and show that three simple tweaks make it practical at today’s model scale: (1) build a surrogate model through in-context learning instead of gradient training; (2) replace the target model predictions with the (smaller) surrogate model predictions; and (3) when budgets are tight, for acquisition function computation, skip target model forward pass and instead rely only on the surrogate’s entropy. Combined with a bootstrap variance estimator that works “in one shot”, these changes cut the risk-estimation error by 32% (on avg.) compared with uniform random sampling across five text-classification benchmarks and two model families.", "questions": "1. The paper does not justify its choice of datasets; outlining the selection criteria and explaining how these datasets reflect the broader evaluation landscape would strengthen the empirical study.\n\n2. How small can the surrogate model be before performance deteriorates, and how rapidly does evaluation accuracy decline as the surrogate’s quality decreases?\n\n3. Can the proposed approach be extended to regression tasks with LLMs, and what modifications would that require?\n\n4. How will this approach perform under distribution shifts?", "limitations": "Although the authors have discussed some limitations, the paper would benefit from a deeper analysis of the conditions under which the proposed estimators are reliable—and where they may fail—as well as a clearer discussion of how far the empirical findings can be generalized beyond the specific datasets and models tested.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper tackles an important problem and is both clearly written and well structured.\n\n2. The proposed method yields substantial computational savings for LLM evaluation and remains flexible across settings and models.\n\n**Weaknesses**\n\n1. The acquisition rule—uncertainty sampling via Shannon entropy—and the use of surrogate models are already commonplace in active-learning research (e.g., Settles 2010; Dongyuan et al., 2024), limiting methodological novelty.\n\n2. Freezing the surrogate model will prevent the active sampler from incorporating information gained during data collection, which can degrade efficiency.\n\n3. Uniform random sampling is a weak baseline; comparisons against stronger alternatives such as margin sampling, BALD etc. are needed to gauge true gains.\n\n4. There are no theoretical guarantees presented for the recommended estimators (using surrogate model), leaving their performance behaviors under different settings unclear.\n\n**References**\n\n1. Active Learning Literature Survey by Burr Settles, 2010\n\n2. A Survey on Deep Active Learning: Recent Advances and New Frontiers by Dongyuan et. al. 2024", "was_revised": true, "final_justification": "Authors have addressed some of my concerns. However, my concerns regarding the novelty of the work still remain. So, I still tend towards rejection (I have raised my score to 3).", "rebuttal": "Thank you for your feedback! We appreciate that you recognize the substantial computational savings and flexibility of our method. We hope our response below alleviates your concerns. \n\n> The acquisition rule–uncertainty sampling via Shannon entropy–and the use of surrogate models are already commonplace in active-learning research, limiting methodological novelty.\n\nThere is a significant difference between active learning and active testing and our core novelty is being the first paper that shows how active testing can be successfully used in LLM evaluation.  We do not claim that the use of Shannon entropy is a novelty, nor the use of surrogate which was already introduced in the original active testing paper. Our technical contribution is the overall adaptation to active testing to make it practical for LLMs at scale. Technical innovations that have not previously been done in the active testing context include: (1) using in-context learning to create surrogates that are fixed during sampling, without expensive retraining; (2) using a significantly smaller surrogate compared to the target, thereby reducing inference costs for surrogate predictions; (3) replacing the target model predictions with surrogate predictions to reduce computational costs even further when budgets are tight.\n\n> Uniform random sampling is a weak baseline; comparisons against stronger alternatives such as margin sampling, BALD etc. are needed to gauge true gains. \n\nWe disagree with the claim that uniform sampling is not an appropriate baseline, as it is the standard approach used in practice for evaluation.  The idea of using an active approach for LLM evaluation is itself novel, while approaches like margin sampling and BALD are active _learning_ acquisition strategies, not active _testing_ acquisition strategies. As established in the original active testing paper (Active testing: sample-efficient model evaluation, Kossen et al, 2021), most active learning acquisition functions are inappropriate for testing scenarios because active testing aims for accurate risk estimation rather than model improvement, which is conceptually a different problem. Our contribution lies in making active testing practical for LLMs despite prohibitive training and inference costs, demonstrating that existing acquisition functions can be useful for computational and labeling efficiency.\n\n> There are no theoretical guarantees presented for the recommended estimators (using surrogate model), leaving their performance behaviors under different settings unclear. \n\nWe strongly disagree that our method lacks theoretical guarantees. Our approach is grounded in established importance sampling theory: we employ a static importance sampler that is provably unbiased (On statistical bias in importance sampling: How and when to fix it, Farquhar et al, 2021) and samples uniformly at random (i.i.d) since the surrogate distribution remains fixed. It therefore maintains the same theoretical guarantees as uniform sampling. In particular, following guidelines from Active testing: sample-efficient model evaluation, Kossen et al, 2021, our proposal is known to be valid, and thus its convergence and unbiasedness are guaranteed. Additionally, this estimator can drastically reduce the variance of estimates compared to uniform sampling when the acquisition proposal is properly chosen. \n\nWe appreciate though that these theoretical properties could have been made clearer in the submission and will update it accordingly.\n\n> Although the authors have discussed some limitations, the paper would benefit from a deeper analysis of the conditions under which the proposed estimators are reliable—and where they may fail—as well as a clearer discussion of how far the empirical findings can be generalized beyond the specific datasets and models tested.\n\nThe variety of experiments we conducted does in fact allow for  extensive analysis of when our estimators are reliable versus when they fail. Our estimators prove reliable when the surrogate can effectively identify target model errors. When this correlation breaks down, performance gracefully degrades to uniform sampling. This can happen when the surrogates are too small for the target’s complexity (see examples with Phi-2 and Gemma-3 1B in Figures C1 and C2 of the supplementary material), for challenging tasks where surrogates struggle to capture incorrect answers (see MMLU dataset in Section 5.7), or when labels are noisy (as discussed in Section 5.8). Even in failure modes, our method degrades to uniform sampling’s performance, providing estimates that a practitioner can still rely on.\n\nOur experiments span three model families, sizes from 1B to 70B parameters and diverse tasks. The consistent 32% average improvement across these settings demonstrates broader applicability. The key insight is that effectiveness depends on the acquisition proposal and therefore the surrogate’s ability to identify the target’s error rather than absolute capabilities, making the approach transferable to new models and tasks. \n\n> Freezing the surrogate model will prevent the active sampler from incorporating information gained during data collection, which can degrade efficiency. \n\nOur decision to freeze surrogate models is empirically justified by our experimental results, leading to significant gains in error estimation, and computationally motivated, saving additional costs. Updating surrogates requires recomputing predictions over the entire sampling set, with increasingly large contexts at each step, which defeats our motivation for computational efficiency. \n\nThat said, we have now added a new ablation with dynamic updating of the surrogate to assess how big the benefits of doing this might be, see our response to reviewer cFrJ for the new results. These new results show that dynamically updating the surrogate does not provide clear consistent benefits, while prohibitively increasing computational costs: in our new experiments using dynamic surrogates for 40 steps only was over 1500 times more expensive than the frozen ones we recommend.\n\n> The paper does not justify its choice of datasets; outlining the selection criteria and explaining how these datasets reflect the broader evaluation landscape would strengthen the empirical study / Can the proposed approach be extended to regression tasks with LLMs, and what modifications would that require?\n\nThe SST-2, FPB, Hatespeech and Subjectivity datasets are very common LLM tasks, especially in the in-context learning literature (see for example In-context learning learns label relationships but is not conventional learning, Kossen et al, 2022 from which we took direct inspiration for our experiment selection). We added the more challenging MMLU dataset to confirm our results on a commonly used and more difficult classification task. While our method extends naturally to any task type, as the acquisition function can be anything, as explained in Section 2, LLM evaluation predominantly focuses on classification due to the discrete token space. With classical neural networks, Kossen et al. 2021 (Active Testing: Sample-Efficient Model Evaluation) have already explored application of active testing to regression tasks, where the acquisition function now approximates MSE.\n\n> How small can the surrogate model be before performance deteriorates, and how rapidly does evaluation accuracy decline as the surrogate's quality decreases?\n\nGood question! In general, active testing provides bigger benefits the better the surrogate is. Our analysis of small surrogates (Phi-2 in Figure C1 and Gemma-3 1B in Figure C2 of the supplementary material) identifies the practical size limits for surrogate models for our selection of tasks. For some tasks, we observe performance deteriorating when these small models evaluate much larger targets like Llama-2 70B-few. Importantly, active testing does not fail catastrophically for decreasing surrogate quality. Instead, it gracefully degrades towards the label-efficiency of standard uniform sampling. Note however, that small models do outperform uniform sampling in many settings, including when evaluating Llama-2 7B. \n\nWhile these results provide empirical evidence demonstrating the robustness of active testing under deteriorating surrogate quality, we agree it would be interesting work to further quantify this. We note, however, that any useful measurement of ‘surrogate quality’ would need to take into account the target model and evaluation task – and is thus unlikely something a practitioner could compute upfront. This is where we see our bootstrap estimates providing valuable insight for practitioners applying active testing!\n\n\n> How will this approach perform under distribution shifts?\n\nGiven that the models we used haven’t directly been trained on these tasks, we are not sure one can really think of things in terms of a “distribution shift” so much as there being tasks the LLM finds relatively harder to predict than others. We test on tasks of varying difficulty and find our approach to be helpful on both easier and harder tasks, indicating that the approach remains robust to this kind of problem.\n\n\nAgain, thank you very much for your review. We hope that our response has addressed your concerns. Please let us know if there’s anything else we can clarify."}, {"review_id": "8jay9VzPLG", "reviewer": "Reviewer_fX5b", "summary": "This paper proposes an error estimation method, aiming to address the computational bottlenecks of active testing for LLMs. By sacling up current active testing methods, sampling-based active testing is chosen. It introduces lightweight surrogate models via in-context learning, eliminating costly iterative surrogate updates. The authors validate their method on Llama-2 and Gemma-3 models, experiments show that the proposed approach demonstrates lower estimation errorr compared to uniform random sampling across. They also introduce a single-run error estimator to assess the performance of active testing, which further improve effectiveness.", "questions": "1. In few-shot scenarios, why does the method fail more often than in zero-shot setups? Is it due to mismatches between surrogate and target model capabilities?\n2. Can dynamic surrogate adaptation (e.g., updating surrogates with small label batches) mitigate errors in few-shot or complex tasks?", "limitations": "See Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Novel Idea: Instead of standard iterative training, the use of in-context learning for surrogate model construction and fixed-surrogate architecture drastically reduces computational overhead. It makes active testing feasible for LLM evaluations. It is really a good “scaling up” approach for active testing with LLM.\n2. Novel Error Estimation: The single-run bootstrap estimator achieves high coverage of true risk errors, providing critical diagnostic tools for real-world deployment where multiple runs are infeasible.\n3. Good Practical Impact: Error estimation is a critical problem for LLM, especially with limited computational and data budget. The framework’s applicability to dataset curation further extends its utility.\n4.Easy to Follow: The paper is well-structured and easy to follow. Code availability and detailed implementation enhance reproducibility.\n\n## Weaknesses\n1. Theoretical Underpinnings: While empirically effective, the reliance on in-context learning for surrogate models may need more detailed theoretical analyses for generalization error bounds.\n2. Instability in Few-Shot and Complex Tasks: Figure 2 for the 70B-few model on Subj and HS datasets, risk-estimation error is close to or worse than uniform random sampling. This may stem from surrogate models (trained via few-shot in-context learning) failing to capture the target model’s behavior when both are in few-shot modes. Figure 6  for the challenging MMLU dataset, active testing for 70B model indicating limits in surrogate model capability for complex tasks.", "was_revised": true, "final_justification": "I have read authors' response. It addressing my concerns about the theoretical guarantees andI found explanation convincing.\n\nWhile I appreciate the note on \"graceful degradation\" and agree it makes sense, the cases where performance lags behind random sampling still warrant deeper analysis. Exploring why the surrogate fails in these scenarios and including specific breakdowns of such failure cases would strengthen the work.\n\nI plan to retain my positive rating.", "rebuttal": "Thank you for your hard work and helpful feedback! We are delighted that you appreciate the novelty and practical impact of our approach for cost-effective evaluation. We hope our response below alleviates your remaining concerns.\n\n> Theoretical underpinnings: While empirically effective, the reliance on in-context learning for surrogate models may need more detailed theoretical analyses for generalization error bounds.\n\nWe respectfully disagree with the claim that the use of in-context learning for surrogate models significantly complicates obtaining theoretical guarantees: the surrogate is only being used as a proposal within a static importance sampling framework, so we inherit all the well established strong guarantees of unbiased i.i.d. Monte Carlo estimators (noting that our proposal is constructed in such a way that it is guaranteed to form a valid proposal so the estimator is unbiased and consistent). The guarantees will all be effectively the same as when using uniform sampling of new datapoints, except with a single sample variance. If our proposal is better than a uniform, this variance will be lower (and this is typically the case as shown in our experiments), if it is worse the variance will be higher.\n\n\n> Instability in Few-Shot and Complex Tasks: Figure 2 for the 70B-few model on Subj and HS datasets, risk-estimation error is close to or worse than uniform random sampling. This may stem from surrogate models (trained via few-shot in-context learning) failing to capture the target model’s behavior when both are in few-shot modes.\n\nWhile it is indeed an important point that the method can do similar or worse to uniform sampling if the surrogate performs poorly (and we have tried to ensure this is clearly shown and discussed in the paper), we emphasise that these are more graceful degradations rather than sudden instabilities.\n\nIn particular, even in these worst-case scenarios, our method gracefully degrades to uniform sampling performance, maintaining a consistent –and close to 1– relative error. This highlights the robustness of our method, and is crucial for practical deployment as practitioners can safely apply our method without risk of obtaining estimations that are significantly worse than random sampling. We also note that such failures correspond to poorly capturing the true label distribution, rather than the target model’s behaviour itself.  We would be happy to add further clarifications, discussions, and/or ablations on this if you think it would improve the paper.\n\n> In few-shot scenarios, why does the method fail more often than in zero-shot setups? Is it due to mismatches between surrogate and target model capabilities?\n\nExcellent question! The key insight is that while the surrogate model uses few-shot learning in both scenarios, the target model receives different contexts. In zero-shot settings, the target model is not given any context and therefore performs worse, making more errors that can be easily identified by the surrogate model. In few-shot settings, the target model’s improved performance reduces the number of errors and the errors that do occur can be harder to predict, making it harder for the surrogate to identify them. \n\n> Can dynamic surrogate adaptation (e.g., updating surrogates with small label batches) mitigate errors in few-shot or complex tasks?\n\nThank you for raising this important question! To test it, we explored surrogate adaptation and ran experiments with dynamic surrogate updating (detailed in our response to reviewer cFrJ). We didn’t observe clear consistent benefits in performance, but doing it drastically increased the computational costs: updating surrogates requires recomputing predictions over the entire sampling set with increasingly large contexts, which can be prohibitively costly in practice. We observed that experiments with surrogate updating took over 1500 times longer to run for 40 steps than when freezing surrogates.\n\nGiven the huge expenses associated with this uncertain performance gain, we don’t believe surrogate adaptation is beneficial to active testing in most cases. However, you are right that there might be specific applications where adaptation could be valuable, particularly when labeling costs massively exceed computational costs. We view this as an interesting direction for future work and will add discussions on it along with our new results.\n\nAgain, thank you for your review! We appreciate your questions and suggestions, which contribute to a more thorough analysis of our approach. Please let us know if there is anything else that we can clarify."}, {"review_id": "aqu50N0oWV", "reviewer": "Reviewer_FX74", "summary": "This paper presents a scalable active testing method to make evaluating large language models more efficient. The approach overcomes prohibitive computational costs by using a smaller, fixed surrogate model, constructed cheaply via in-context learning, to intelligently guide the selection of data for labeling. By avoiding repeated training and minimizing the need for predictions from the large target model, this technique is shown to substantially reduce evaluation error compared to standard random sampling, enabling more accurate LLM assessment with less labeled data.", "questions": "Could the author expand more real world applications where labelling cost is not as important as inference compute time?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths:**\n\n- The method is clearly written, with a well-introduced overview of related work.\n\n- The authors provide extensive evaluation.\n\n**Weaknesses:**\n- W1) Limited novelty: The central method primarily relies on approximating a target prediction with a cheaper prediction using ICL, an idea that by itself lacks novelty [1,2,3]. Additionally, the acquisition method presented in Section 3.3 resembles entropy-based acquisition methods commonly used in active learning. Philosophically, I do not know if we want to replace the induced test distribution by a cheap surrogate model over the specialized target model.\n\n- W2) The proposed idea could be interesting if the ICL surrogate model effectively replaces a trained, task-specialized LLM. However, the evaluation predominantly tests only within the scope of in-context learning, and in the context of classification benchmarks. Thus, if the surrogate and target models exhibit similar capabilities—even with different model sizes (e.g., 7B and 70B)—particularly in certain domains, the findings become less compelling.\n- W3) Consequently, if we assume small LLM to have similar capabilities to large LLMs, then the question is when this does not hold. I am a bit sceptic that the proposed approach \"always\" work. Since the paper is mostly empirical, I think we need to understand when does not work and when it work, which is not extensively explored.\n\n\n### References\n\n1. Diao, Shizhe, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. “Active Prompting with Chain-of-Thought for Large Language Models.” *arXiv preprint* arXiv:2302.12246, 2023.\n\n2. Zhang, Yiming, Shi Feng, and Chenhao Tan. “Active Example Selection for In-Context Learning.” In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 9134–9148. Association for Computational Linguistics, 2022.\n\n3. Margatina, Katerina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. “Active Learning Principles for In-Context Learning with Large Language Models.” In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 5011–5034. Association for Computational Linguistics, 2023.", "was_revised": true, "final_justification": "I think the word provides extensive results, because of this some reviewers find it valuable it could be accepted. However, my concern is that the idea and the paper itself is trivial in some sense. Most of acquisition are borrowed from previous work, and the one they proposed is basically assuming the surrogate behaves as well as the target model. Then if a small models performs well or have similar capabilities than the large model is obvious that we can use a small surrogate to do acquisition. This is not all the experiment they present, but many are around that idea.  It is possible that I am not weighting properly the contributions and that is why I put confidence 3.", "rebuttal": "Thank you for your review. Unfortunately, we believe there may be a fundamental misunderstanding about our work’s scope and contributions. We hope to clarify these in our response below.\n\nMost importantly, we do not propose a method to improve model _predictions_ or to reduce their cost for standard inference applications. Our work only targets LLM _evaluation_. The core challenge is that evaluating frontier LLMs on large datasets is prohibitively expensive–both computationally and in terms of labeling costs. This is a critical bottleneck as models become larger and more numerous and as evaluation datasets expand.\n\nWe do not claim novelty for in-context learning or entropy-based acquisition (W1), but rather demonstrate how to combine these techniques to make active testing practical for LLMs at scale. Our technical innovations aim to reduce the computational cost of active testing through (1) our use of fixed, in-context-learning–based, surrogate models, (2) using surrogates that are computationally cheaper than the target model, and, ultimately, (3) our proposal to replace even target model predictions with surrogate predictions for the largest computational cost savings.\n\n\n> W1) Limited novelty: The central method primarily relies on approximating a target prediction with a cheaper prediction using ICL, an idea that by itself lacks novelty [1,2,3]. Additionally, the acquisition method presented in Section 3.3 resembles entropy-based acquisition methods commonly used in active learning. Philosophically, I do not know if we want to replace the induced test distribution by a cheap surrogate model over the specialized target model. \n\nWe believe there may have been a fundamental misunderstanding here that we hope is easy to clear up. Our use of the surrogate model does not “induce” a different test distribution, nor are we trying to improve on the predictions of the original model in any way: it is being used to construct a _proposal_ over which test points to acquire labels for, as is required by our R_LURE importance sampling estimator (Eq. 1). This estimator will be unbiased for any acquisition proposal distribution q that has full support (which ours do) and thus for any surrogate. For well-chosen q, the estimate with R_LURE can be significantly lower variance than a naive random sampling baseline. As the surrogate only affects the proposal, it does not introduce any difference in test distribution.  Importantly, the true labels and target model predictions always enter the estimator via the loss term, L(f(x), y), in the estimator, regardless of the choice for q. \n\nTo reiterate, we do not suggest replacing target model predictions with surrogate predictions for general prediction scenarios. We only consider this for the purposes of computing the acquisition proposal q, as using a non-uniform proposal allows us to reduce the number of test labels we need to acquire, while using a small surrogate means this proposal is cheaper than using a large LLM. The estimator remains unbiased and estimates the test performance on the true distribution underlying the test data. It further retains the same convergence guarantees as a standard uniform Monte Carlo estimator. \n\nThe works [1, 2, 3] that you refer to study how to actively construct the subset of samples to include in ICL, aiming to improve predictions. This is entirely orthogonal to our work as we do not actively construct the subset of samples used in the ICL of the surrogate. Instead, we use a fixed and randomly selected set of samples. Our acquisition function acquires samples for use in the active testing estimator and not for inclusion in ICL. We further note that Kossen et al 2021 have previously shown that active learning acquisition strategies are not directly optimal for the active testing estimator itself as the active learning and active testing problems are quite different in which points we wish to target.  As such, the fact that strategies related to entropy have been used in active learning is tangential to our actual contributions.\n\nWe hope this clears up this misunderstanding and will clearly highlight differences and similarities to related work in a revised version of the draft.\n\n> W2) The proposed idea could be interesting if the ICL surrogate model effectively replaces a trained, task-specialized LLM. However, the evaluation predominantly tests only within the scope of in-context learning, and in the context of classification benchmarks. Thus, if the surrogate and target models exhibit similar capabilities–even with different model sizes– particularly in certain domains, the findings become less compelling.\n\nAgain, this point suggests there may have been a fundamental misunderstanding of our method: our work has nothing to do with replacing the original “task-specialized LLM” with an ICL surrogate. As explained above, the surrogate is only used as a mechanism to more efficiently evaluate the original LLM by constructing an importance sampler.  Note also that it is a surrogate of the ground truth label distribution, not of the target’s distribution, with the aim to identify incorrect predictions of the target. Even a much smaller model (eg Gemma-3 4B or Llama-2 7B) can often identify errors made by a larger model (eg Llama-2 70B), because discrepancies in predictions can detect errors independently from task performance. Our experiments demonstrate this clearly, in Figures 2, 3, and Figures C1, C2 of the supplementary material. Whether the target model and final surrogate have similar capabilities when used in general prediction scenarios is irrelevant to our contribution, as we never suggest replacing the predictions of the former with the latter.\n\n> W3) Consequently, if we assume small LLM to have similar capabilities to large LLMs, then the question is when this does not hold. I am a bit sceptic that the proposed approach “always” work. Since the paper is mostly empirical, I think we need to understand when does not work and when it work, which is not extensively explored.\n\nWe want to clarify that we never claim our approach always works independently from the surrogate’s choice, but this does not mean that we have to assume a small LLM has similar capabilities as a large LLM. We provide extensive experiments across a wide range of surrogates size, using Llama-2 7B and 70B, Gemma-3 4B and 1B as well as Phi-2 (note there are additional results in the appendix to investigate surrogate choice to those in the main paper), to analyze failure cases and boundary conditions. Our results demonstrate that active testing is generally beneficial when the surrogate is appropriately selected, even when the surrogate is a lot smaller than the target model, and fails gracefully in worst-case scenarios, with performance comparable to uniform sampling.  Even a small surrogate can provide meaningful signal in the acquisition function that allows us to improve over uniform random acquisition.\n\n> Could the author expand more real world applications where labelling cost is not as important as inference compute time?\n\nComputational costs dominate in several settings, such as model selection based on test datasets, or benchmark leaderboards when evaluating numerous models for comparison on large, already labeled datasets.\n\nAgain, thank you for your review. We hope our response helped to clarify the motivation of our work and why its contributions are highly distinct from other uses of surrogates in an LLM context. Please let us know if there is anything else that we can clarify."}, {"review_id": "l3vk3tZI84", "reviewer": "Reviewer_cFrJ", "summary": "The authors extend the technique of active testing (Kossen et al., 2021) with active surrogate estimators (Kossen et al., 2022) to the domain of LLMs. The setting is that there is a large pool of test inputs, but acquiring gold labels for test inputs is expensive. Active testing is a label-efficient way to evaluate models in this setting by carefully selecting which test inputs to acquire gold labels for. At every label acquisition step, the next test input to label is sampled from a distribution based on an acquisition function that is based on the disagreement between the target model and a surrogate model that is trained on labelled test data acquired so far.\n\nThe authors’ contributions are showing that for computing the acquisition function: (1) the surrogate model can be a smaller LLM; (2) the surrogate model can be trained using in-context learning, thus eliminating the need for gradient-based training; (3) the surrogate model does not need to be retrained with new test labels at each step, thus requiring only a single inference pass over the test inputs; (4) the target model can also be replaced with the surrogate model, thus avoiding computing inferences with the target model. The authors validate their technique using experiments using Llama and Gemma family LLMs.", "questions": "- How were the datasets for the experiments selected?\n- Did you consider running ablations for the individual components of your technique? I’m specifically curious about the ablation of adding previously acquired test labels into the in-context training of the surrogate models, rather than fixing the in-context training. I acknowledge that the ablations may have been prohibitively costly to run.", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The problem setting is very common in practical LLM evaluation settings. Gold labels are expensive because they require human labellers, and inference on the target model can be expensive for very large models. (Significance)\n- The proposed technique is very practical and easy to use. (Quality, Significance)\n- The paper is useful for introducing the active testing concept to the field of LLM evaluation, in which active testing is rarely used. (Significance)\n- The prior work around active testing is well explained, and the proposed new technique is well-justified. (Clarity)\n- The experiments are rigorous and produce convincing results. The investigation into the SST-2 dataset is explained well. (Quality)\n- The idea of replacing the target model with the surrogate model is novel, and the insight that the acquisition function reduces to predictive entropy is elegant. (Originality)\n\n### Weaknesses\n\n- The authors conclude that the surrogate and target models do not have to be from the same family. However, the experiments only used two model families, so this result may not be very general. Additionally, the advantage of using a same-family surrogate model versus a different-family surrogate model was not quantified. This may be an area for future work. (Quality)\n- The relevant importance of each of the individual components of the proposed technique is unclear, because there were no ablations for the individual components. (Quality)\n- The graphs are too dense and hard to read. The graph lines make use of both color and style (dotted, dashed, solid), which is confusing. Consider splitting the current graphs further into multiple graphs. (Clarity)\n- Some parts of the proposed technique (using in-context learning, using a smaller model) are somewhat obvious to LLM researchers. (Significance, Originality)\n- The authors ran experiments using datasets that are relatively low-difficulty for recent LLMs in 2025, and did not include datasets requiring open-ended generation. Open-ended generation datasets would have been a more realistic reflection of the problem setting, because these datasets are expensive to acquire human labels for. (Quality)", "was_revised": true, "final_justification": "Overall, the main strengths of this paper are that the main proposed method is useful, practical, and well-evaluated. The main weakness is that its novelty is questionable. The use of in-context learning was obvious to myself and reviewer FX74. The use of an entropy-based acquisition method was obvious to reviewers x1Pz and FX74 - this was not obvious to me, but it is possible that this is due to my lack of familiarity with the field of active learning. Even so, I believe that the synthesis of these ideas is interesting, and that this paper makes valuable recommendations to the community on how to do cost-efficient LLM testing. This justifies my overall rating of Accept.\n\nMy other main concern was the lack of ablations. The authors have mostly addressed this by providing experimental results for some ablations (comparing dynamic surrogates with fixed surrogates).", "rebuttal": "Thank you for your hard work and helpful feedback! We’re delighted that you appreciate the practical significance and rigorous experimental validation of our work. We hope our response below addresses your remaining questions. \n\n> Did you consider running ablations for the individual components of your technique? I’m specifically curious about the ablation of adding previously acquired test labels into the in-context training of the surrogate models, rather than fixing the in-context training. I acknowledge that the ablations may have been prohibitively costly to run. [...] The relevant importance of each of the individual components of the proposed technique is unclear, because there were no ablations for the individual components.\n\nExcellent suggestion! Following your feedback, we have conducted ablation studies comparing fixed versus dynamic surrogates, where the dynamic approach incorporates newly acquired labels into the surrogate’s context and recomputes predictions, and thus acquisition probabilities, at each sampling step. \nOur experiments evaluate the few-shot Llama-2 7B and 70B models with dynamic and fixed Llama-2 7B surrogates, starting with 10 in-context examples over 40 steps (50 runs each for dynamic surrogates, 3000 runs for fixed surrogates and uniform sampling). You can find a comparison of the mean relative error to random uniform sampling for acquisition with and without the dynamic surrogate below along with the 5th and 95th percentiles.\n\nIn summary, we find that dynamic surrogates show no clear consistent improvements in risk estimation over fixed surrogates, while imposing prohibitive computational costs. The experiments with 50 repetitions each were over 1500 times more expensive for 40 steps than the frozen ones, whose cost remains fixed as the number of steps increases. Additionally, dynamic surrogates require unique predictions at each step, eliminating the ability to precompute and share predictions across multiple evaluations. While there may be interesting applications of dynamic surrogates, for our selection of tasks and models, the results support our design choice of fixed surrogates, which achieves the same accuracy benefits while maintaining the computational efficiency that makes this active testing method practical. We are happy to add these results to the revised version of the paper. \n\n### Scaled mean squared error (x$10^{-4}$, 5th-95th percentile) for uniform random sampling, dynamic surrogate active testing, and fixed surrogate active testing on 7B-few and 70B-few across datasets\n\n| Acquisition step |                   | 10    | 20    | 30    | 40    |\n| -----------------|------------------ | ----- | ----- | ----- | ----- |\n| SST-2 7B-few  | Uniform random      | 91.41 (87.04 - 95.86) | 45.49 (43.53 - 47.52) | 29.25 (28.00 - 30.51) | 20.66 (19.82 - 21.52) |\n|                | LURE-Llama2-7B Dynamic | 63.25 (36.23 - 96.67) | 22.79 (14.56 - 32.91) | 13.52 (8.93 - 18.80) | 8.62 (5.82 - 11.88) |\n|                | LURE-Llama2-7B Fixed   | 42.71 (40.32 - 45.11) | 19.99 (19.07 - 20.96) | 12.42 (11.87 - 13.00) | 8.74 (8.38 - 9.11) |\n| FPB    7B-few | Uniform random      | 135.75 (130.00 - 141.72) | 65.14 (62.37 - 68.05) | 40.06 (38.41 - 41.76) | 27.99 (26.84 - 29.16) |\n|                | LURE-Llama2-7B Dynamic   | 65.73 (45.79 - 86.86) | 36.40 (25.31 - 49.18) | 18.33 (11.82 - 25.79) | 11.53 (7.37 - 16.43) |\n|                | LURE-Llama2-7B Fixed     | 77.54 (73.39 - 81.94) | 36.88 (35.22 - 38.61) | 22.97 (21.99 - 23.95) | 16.26 (15.56 - 16.96) |\n| HS     7B-few | Uniform random      | 156.59 (152.02 - 161.10) | 96.51 (93.78 - 99.32) | 77.32 (75.14 - 79.54) | 66.84 (64.93 - 68.76) |\n|                | LURE-Llama2-7B Dynamic | 121.45 (100.99 - 142.89) | 75.61 (60.04 - 91.73) | 52.12 (40.48 - 64.84) | 43.83 (35.35 - 52.63) |\n|                | LURE-Llama2-7B Fixed    | 144.44 (139.38 - 149.87) | 89.65 (87.31 - 91.98) | 72.62 (70.73 - 74.47) | 63.67 (62.07 - 65.31) |\n| Subj   7B-few | Uniform random      | 46.97 (44.97 - 49.00) | 21.79 (20.89 - 22.70) | 13.76 (13.19 - 14.36) | 10.23 (9.80 - 10.66) |\n|                | LURE-Llama2-7B Dynamic | 71.76 (50.93 - 95.54) | 35.25 (20.62 - 53.32) | 36.75 (21.11 - 55.19) | 30.94 (18.43 - 46.09) |\n|                | LURE-Llama2-7B Fixed     | 40.20 (38.39 - 42.02) | 19.69 (18.89 - 20.48) | 12.95 (12.42 - 13.50) | 9.12 (8.73 - 9.52) |\n\n\n\n| Acquisition step  |                   | 10    | 20    | 30    | 40    |\n| -----------------|------------------ | ----- | ----- | ----- | ----- |\n| SST-2 70B-few  | Uniform random      | 81.75 (77.77 - 85.85) | 40.04 (38.24 - 41.87) | 25.01 (23.98 - 26.06) | 17.92 (17.19 - 18.66) |\n|                | LURE-Llama2-7B Dynamic | 28.76 (22.13 - 35.70) | 17.21 (11.42 - 23.89) | 12.62 (8.83 - 16.74) | 8.57 (6.37 - 11.07) |\n|                | LURE-Llama2-7B Fixed   | 40.09 (38.28 - 41.98) | 19.11 (18.35 - 19.89) | 12.26 (11.80 - 12.73) | 8.44 (8.12 - 8.76) |\n| FPB    70B-few | Uniform random      | 173.90 (164.57 - 183.50) | 80.36 (76.55 - 84.22) | 50.14 (47.77 - 52.54) | 35.07 (33.40 - 36.83) |\n|                | LURE-Llama2-7B Dynamic   | 114.11 (84.88 - 144.11) | 34.89 (25.00 - 46.22) | 19.71 (13.20 - 27.05) | 11.10 (7.68 - 14.95) |\n|                | LURE-Llama2-7B Fixed     | 150.45 (140.12 - 161.06) | 70.59 (66.56 - 74.69) | 45.85 (43.59 - 48.21) | 33.25 (31.64 - 34.85) |\n| HS     70B-few | Uniform random      | 98.22 (94.33 - 102.28) | 48.73 (46.95 - 50.54) | 33.04 (31.84 - 34.29) | 25.27 (24.36 - 26.20) |\n|                | LURE-Llama2-7B Dynamic | 88.73 (64.35 - 116.85) | 41.15 (31.37 - 51.26) | 23.93 (18.57 - 29.54) | 16.18 (12.40 - 20.03) |\n|                | LURE-Llama2-7B  Fixed    | 74.57 (71.50 - 77.75) | 39.31 (37.78 - 40.87) | 26.87 (25.85 - 27.89) | 19.93 (19.18 - 20.69) |\n| Subj   70B-few | Uniform random    | 109.53 (105.47 - 113.54) | 66.96 (64.75 - 69.11) | 51.41 (49.77 - 53.10) | 42.97 (41.59 - 44.36) |\n|                | LURE-Llama2-7B Dynamic | 209.81 (161.39 - 260.62) | 134.48 (99.07 - 172.06) | 72.61 (54.29 - 92.06) | 64.43 (48.04 - 82.24) |\n|                | LURE-Llama2-7B Fixed     | 134.87 (129.79 - 140.22) | 77.31 (74.56 - 80.06) | 57.14 (55.20 - 59.07) | 47.24 (45.64 - 48.81) |\n\n\n\n> The authors conclude that the surrogate and target models do not have to be from the same family. However, the experiments only used two model families, so this result may not be very general. Additionally, the advantage of using a same-family surrogate model versus a different-family surrogate model was not quantified. This may be an area for future work. \n\nThat’s a good point, thank you for giving us the opportunity to clarify! Our experiments actually span three model families, namely Llama-2, Gemma-3 and Phi-2 but some of these are only in the supplement (see Figures C.1. and C.2.). Evaluations demonstrate the robustness of active testing across model families. \nWe agree that further investigation of the effects of model diversity between the surrogate and target would be interesting future work but are happy to offer some thoughts here: Interestingly, while the original active testing paper suggests that surrogate-target diversity is generally beneficial, we find that using smaller surrogates from the same model family can be highly effective. Although, there may still be scenarios where diversity between the target and surrogate model is more advantageous, it is possible this is less pronounced for LLMs, which are likely much more correlated in their predictions (e.g. due to shared architecture and parts of the training data) than the models considered by Kossen et al. (2021).\n\n> How were the datasets for the experiments selected?\n\nThe datasets we chose are common LLM tasks, especially in the in-context learning literature (see In-context learning learns label relationships but is not conventional learning, Kossen et al, 2022 from which we took direct inspiration). We added MMLU as a more challenging, widely-used benchmark to demonstrate the robustness of active testing.\n\n> The authors ran experiments using datasets that are relatively low-difficulty for recent LLMs in 2025, and did not include datasets requiring open-ended generation. Open-ended generation datasets would have been a more realistic reflection of the problem setting, because these datasets are expensive to acquire human labels for.\n\nEvaluation on tasks requiring open-ended generation represents an interesting direction for future work, but we note that this faces fundamental technical challenges that extend beyond the scope of this submission. The acquisition functions for active testing that we study require the computation of the models predictive uncertainty. For open-ended generations with LLM, uncertainty quantification is still an area of active research (see for example Detecting hallucinations in large language models using semantic entropy, Farquhar et al, 2024), in particular when considering cost-effective uncertainty estimation.\n As an alternative, we would like to point out that our classification tasks and especially MMLU remain challenging for the selection of models we studied, as shown in tables C7 and C8 of the supplementary material. \n\n> The graphs are too dense and hard to read.\n\nThank you for pointing this out! We will revise the graphs accordingly to your suggestions and make sure they appear more clearly.\n\nAgain, thank you for your review. We appreciate your questions and suggestions, which contribute to a more thorough analysis of our approach. Please let us know if there is anything else that we can clarify."}]

Author rebuttal:

Meta-review: This paper adapts active testing and active surrogate estimators to evaluate LLMs, addressing the challenge of costly label acquisition for large test sets.
While the methods are based on the combinations of the known established techniques, their integration and application to LLMs represent a novel contribution.
The authors also provided some discussion on the failure cases analysis of their approach, which further strengthen the paper.
Decision: Accept (poster)