Title: Scaling Up Active Testing to Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: active testing

active evaluation

model evaluation

data curation

large language models
Abstract: Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend the technique of active testing (Kossen et al., 2021) with active surrogate estimators (Kossen et al., 2022) to the domain of LLMs. The setting is that there is a large pool of test inputs, but acquiring gold labels for test inputs is expensive. Active testing is a label-efficient way to evaluate models in this setting by carefully selecting which test inputs to acquire gold labels for. At every label acquisition step, the next test input to label is sampled from a distribution based on an acquisition function that is based on the disagreement between the target model and a surrogate model that is trained on labelled test data acquired so far.\n\nThe authors’ contributions are showing that for computing the acquisition function: (1) the surrogate model can be a smaller LLM; (2) the surrogate model can be trained using in-context learning, thus eliminating the need for gradient-based training; (3) the surrogate model does not need to be retrained with new test labels at each step, thus requiring only a single inference pass over the test inputs; (4) the target model can also be replaced with the surrogate model, thus avoiding computing inferences with the target model. The authors validate their technique using experiments using Llama and Gemma family LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The problem setting is very common in practical LLM evaluation settings. Gold labels are expensive because they require human labellers, and inference on the target model can be expensive for very large models. (Significance)\n- The proposed technique is very practical and easy to use. (Quality, Significance)\n- The paper is useful for introducing the active testing concept to the field of LLM evaluation, in which active testing is rarely used. (Significance)\n- The prior work around active testing is well explained, and the proposed new technique is well-justified. (Clarity)\n- The experiments are rigorous and produce convincing results. The investigation into the SST-2 dataset is explained well. (Quality)\n- The idea of replacing the target model with the surrogate model is novel, and the insight that the acquisition function reduces to predictive entropy is elegant. (Originality)\n\n### Weaknesses\n\n- The authors conclude that the surrogate and target models do not have to be from the same family. However, the experiments only used two model families, so this result may not be very general. Additionally, the advantage of using a same-family surrogate model versus a different-family surrogate model was not quantified. This may be an area for future work. (Quality)\n- The relevant importance of each of the individual components of the proposed technique is unclear, because there were no ablations for the individual components. (Quality)\n- The graphs are too dense and hard to read. The graph lines make use of both color and style (dotted, dashed, solid), which is confusing. Consider splitting the current graphs further into multiple graphs. (Clarity)\n- Some parts of the proposed technique (using in-context learning, using a smaller model) are somewhat obvious to LLM researchers. (Significance, Originality)\n- The authors ran experiments using datasets that are relatively low-difficulty for recent LLMs in 2025, and did not include datasets requiring open-ended generation. Open-ended generation datasets would have been a more realistic reflection of the problem setting, because these datasets are expensive to acquire human labels for. (Quality)\n\nQUESTIONS:\n- How were the datasets for the experiments selected?\n- Did you consider running ablations for the individual components of your technique? I’m specifically curious about the ablation of adding previously acquired test labels into the in-context training of the surrogate models, rather than fixing the in-context training. I acknowledge that the ablations may have been prohibitively costly to run.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a scalable active testing method to make evaluating large language models more efficient. The approach overcomes prohibitive computational costs by using a smaller, fixed surrogate model, constructed cheaply via in-context learning, to intelligently guide the selection of data for labeling. By avoiding repeated training and minimizing the need for predictions from the large target model, this technique is shown to substantially reduce evaluation error compared to standard random sampling, enabling more accurate LLM assessment with less labeled data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The method is clearly written, with a well-introduced overview of related work.\n\n- The authors provide extensive evaluation.\n\n**Weaknesses:**\n- W1) Limited novelty: The central method primarily relies on approximating a target prediction with a cheaper prediction using ICL, an idea that by itself lacks novelty [1,2,3]. Additionally, the acquisition method presented in Section 3.3 resembles entropy-based acquisition methods commonly used in active learning. Philosophically, I do not know if we want to replace the induced test distribution by a cheap surrogate model over the specialized target model.\n\n- W2) The proposed idea could be interesting if the ICL surrogate model effectively replaces a trained, task-specialized LLM. However, the evaluation predominantly tests only within the scope of in-context learning, and in the context of classification benchmarks. Thus, if the surrogate and target models exhibit similar capabilities—even with different model sizes (e.g., 7B and 70B)—particularly in certain domains, the findings become less compelling.\n- W3) Consequently, if we assume small LLM to have similar capabilities to large LLMs, then the question is when this does not hold. I am a bit sceptic that the proposed approach \"always\" work. Since the paper is mostly empirical, I think we need to understand when does not work and when it work, which is not extensively explored.\n\n\n### References\n\n1. Diao, Shizhe, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. “Active Prompting with Chain-of-Thought for Large Language Models.” *arXiv preprint* arXiv:2302.12246, 2023.\n\n2. Zhang, Yiming, Shi Feng, and Chenhao Tan. “Active Example Selection for In-Context Learning.” In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 9134–9148. Association for Computational Linguistics, 2022.\n\n3. Margatina, Katerina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. “Active Learning Principles for In-Context Learning with Large Language Models.” In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 5011–5034. Association for Computational Linguistics, 2023.\n\nQUESTIONS:\nCould the author expand more real world applications where labelling cost is not as important as inference compute time?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an error estimation method, aiming to address the computational bottlenecks of active testing for LLMs. By sacling up current active testing methods, sampling-based active testing is chosen. It introduces lightweight surrogate models via in-context learning, eliminating costly iterative surrogate updates. The authors validate their method on Llama-2 and Gemma-3 models, experiments show that the proposed approach demonstrates lower estimation errorr compared to uniform random sampling across. They also introduce a single-run error estimator to assess the performance of active testing, which further improve effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Novel Idea: Instead of standard iterative training, the use of in-context learning for surrogate model construction and fixed-surrogate architecture drastically reduces computational overhead. It makes active testing feasible for LLM evaluations. It is really a good “scaling up” approach for active testing with LLM.\n2. Novel Error Estimation: The single-run bootstrap estimator achieves high coverage of true risk errors, providing critical diagnostic tools for real-world deployment where multiple runs are infeasible.\n3. Good Practical Impact: Error estimation is a critical problem for LLM, especially with limited computational and data budget. The framework’s applicability to dataset curation further extends its utility.\n4.Easy to Follow: The paper is well-structured and easy to follow. Code availability and detailed implementation enhance reproducibility.\n\n## Weaknesses\n1. Theoretical Underpinnings: While empirically effective, the reliance on in-context learning for surrogate models may need more detailed theoretical analyses for generalization error bounds.\n2. Instability in Few-Shot and Complex Tasks: Figure 2 for the 70B-few model on Subj and HS datasets, risk-estimation error is close to or worse than uniform random sampling. This may stem from surrogate models (trained via few-shot in-context learning) failing to capture the target model’s behavior when both are in few-shot modes. Figure 6  for the challenging MMLU dataset, active testing for 70B model indicating limits in surrogate model capability for complex tasks.\n\nQUESTIONS:\n1. In few-shot scenarios, why does the method fail more often than in zero-shot setups? Is it due to mismatches between surrogate and target model capabilities?\n2. Can dynamic surrogate adaptation (e.g., updating surrogates with small label batches) mitigate errors in few-shot or complex tasks?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of the increasing cost of evaluating frontier LLMs. The authors revisit active testing—which selectively sends the most informative test inputs for human-labelling—and show that three simple tweaks make it practical at today’s model scale: (1) build a surrogate model through in-context learning instead of gradient training; (2) replace the target model predictions with the (smaller) surrogate model predictions; and (3) when budgets are tight, for acquisition function computation, skip target model forward pass and instead rely only on the surrogate’s entropy. Combined with a bootstrap variance estimator that works “in one shot”, these changes cut the risk-estimation error by 32% (on avg.) compared with uniform random sampling across five text-classification benchmarks and two model families.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper tackles an important problem and is both clearly written and well structured.\n\n2. The proposed method yields substantial computational savings for LLM evaluation and remains flexible across settings and models.\n\n**Weaknesses**\n\n1. The acquisition rule—uncertainty sampling via Shannon entropy—and the use of surrogate models are already commonplace in active-learning research (e.g., Settles 2010; Dongyuan et al., 2024), limiting methodological novelty.\n\n2. Freezing the surrogate model will prevent the active sampler from incorporating information gained during data collection, which can degrade efficiency.\n\n3. Uniform random sampling is a weak baseline; comparisons against stronger alternatives such as margin sampling, BALD etc. are needed to gauge true gains.\n\n4. There are no theoretical guarantees presented for the recommended estimators (using surrogate model), leaving their performance behaviors under different settings unclear.\n\n**References**\n\n1. Active Learning Literature Survey by Burr Settles, 2010\n\n2. A Survey on Deep Active Learning: Recent Advances and New Frontiers by Dongyuan et. al. 2024\n\nQUESTIONS:\n1. The paper does not justify its choice of datasets; outlining the selection criteria and explaining how these datasets reflect the broader evaluation landscape would strengthen the empirical study.\n\n2. How small can the surrogate model be before performance deteriorates, and how rapidly does evaluation accuracy decline as the surrogate’s quality decreases?\n\n3. Can the proposed approach be extended to regression tasks with LLMs, and what modifications would that require?\n\n4. How will this approach perform under distribution shifts?\n\nLIMITATIONS:\nAlthough the authors have discussed some limitations, the paper would benefit from a deeper analysis of the conditions under which the proposed estimators are reliable—and where they may fail—as well as a clearer discussion of how far the empirical findings can be generalized beyond the specific datasets and models tested.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-09"}