AutoEval Done Right: Using Synthetic Data for Model Evaluation
Authors: Pierre Boyeau
Anastasios Nikolas Angelopoulos
Tianle Li
Nir Yosef
Jitendra Malik
Michael I. Jordan
Venue: ICML 2025 poster (2025)
Keywords: Prediction-powered inference
model evaluation
large language models
annotation
synthetic data
statistical inference
Primary area: general_machine_learning->evaluation
TL;DR: This paper introduces statistical methods to combine a small amount of human-labeled data with large-scale AI-generated synthetic labels to produce more precise and statistically valid model evaluations.

Abstract: The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased.

AutoEval Done Right: Using Synthetic Data for Model Evaluation
Pierre Boyeau' Anastasios N. Angelopoulos
Tianle Li'
Nir Yosef 12 Jitendra Malik' Michael I. Jordan
Abstract
This work introduces methods for autoevaluation done right. Given a small amount of human data and large amount of synthetic data, we will construct autoevaluation procedures that combine these datasets to get better esti- mates of performance.
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuning. Al-labeled synthetic data can be used to decrease the number of human an- notations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this pur- pose that improve sample efficiency while re- maining unbiased.
In other words, our methods will increase the effective sam- ple size of human data without compromising statistical va- lidity. Intuitively, we the limited human data in order to measure the bias of the synthetic data. Then, we evaluate the model on the synthetic data and correct the bias us- ing this estimate. The core statistical tool used for this de- biasing is called prediction-poweret inference (PPI) (An- gelopoulos et al., 2023a); we will describe this tool detail in the coming text. This approach can improve both metric- based evaluations (Section 2) and pairwise-comparison- based evaluations (Section 3), and can readily be applied using existing Python software. We will include code snippets throughout for producing more precise unbiased evaluations. These lower-variance evaluations are also ac- companied by confidence intervals.
1. Introduction
Our goal is to evaluate machine learning systems- assessing their accuracy, fairness, and other metrics-wi as few data points as possible. This goal is important for re- ducing the human effort required to collect extensive vali- dation datasets (Hastic et al., 2009) for such tasks. Towards that end, we will explore an approach called autoevalua- tion, wherein we evaluate models in a two-stage procedure: (i). Produce synthetic labels using AI large unlabeled dataset, and (ii). evaluate AI models using the synthetic labels.
1.1. Related Work
Autoevaluation has been a subject of interest, particularly in language modeling, well before the current wave of progress in machine learning (Corston-Olive et al., 2001; Agarwal al., 2021; Garg et al., 2022). Since the de- velopment of powerful machine learning systems such as gpt-4, 4, the accuracy of the annotations that these systems produce has started to approach that of humans (Zheng et al., 2024; Huang et al., 2024), giving substantial cre- dence to autoevaluation as an alternative to human evalu- ations (Iii et al., 2024a).
Autoevaluation can save months or years of time and po- tentially millions of dollars in anmotation costs; see, e.g., Scale AI, or the recent work of Zheng al. (Zheng et al., 2024) using gpt-4 to rank alternative language models' answers to questions with high agreement with human an- notators. However, the synthetic labels may not be trust- worthy, especially for the purpose of certifying a model's 's worst-case safety, multi-group accuracy and fairness, or to understand observed differences between models are sig- nificant. This motivates the need for serious statistical in- quiry on the general question of autoevaluation.
The prohibitive cost of human annotation has also encour- aged the development of automatic metrics used to evalu- ate model performance without human aid (Papineni et al., 2002; Lin & Och, 2004), representing distinct but re- lated approach to autoevaluation. Automatic metrics can be computed on the fly, rely on more data points and are hence less noisy, which can be more informative than hu- man evaluations when the latter are scarce (Wei & Jia, 2021). Standard autoevaluation methods generally ad hoc, and resulting estimates of model performance can sys- tematically differ from those obtained by human evalua-
DDeparment of Electrical Engineering and Com- puter Sciences, University of California, Berkeley, USA DDepartment of Systems Immunology, Weizmann Insti- tule of Science, Rehovot, Israel "Inria, Ecole Normale Supéricure, Paris, France. Correspondence to: Michael I. Jordan Emichacl.jordal @ berkeley.edu>.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
tion (Garg et al., 2022; van Breugel et al., 2023). In par- allel, classical solutions for generating confidence inter- vals, such as rank-sets (Al Mohamad et al., 2021), can- not take advantage of the Al-generated data. It has been unclear how A Al-generated data can be combined with hu- man data to improve the quality of evaluations. Towards this end, (Chaganty et al., 2018) produced lower-variance estimates of machine translation performance by combin- ing human preferences with automated metrics via control variates.
2. Autoevaluating Accuracy and other Metrics
We begin by describing how to use prediction-poweres in- ference for estimating metrics. The most commonly used metrics are accuracy and loss, so we focus on these; how- ever, our tools will be general and allow autoevaluation of any metric.
2.1. Defining the Goal
For the purpose of model training, a number of strategies have been proposed to combine human-derived ground- truth with synthetic labels, e.g., using pseudo labeling (Lee et al., 2013; Arazo et al., 2020) or consistency regulariza- tion (Bachman et al., 2014; Laine & Aila, 2016). Unlike these training methods, our work focuses the reliable evaluation of already trained models, providing statistical guarantees essential for deployment.
Basic notation We observe inputs X in some space X' such as the space of natural images, natural language, and so on. We seek to predict labels Y in some space y, such as the space of classes, next tokens, actions, etc. Towards this end, let fi...., fM denote M pretrained models mapping inputs in X' to label estimates in some third space ) We often have 5 y. in which case the model directly out- puts predictions of the label. However, we leave open the possibility that 5 is some other space-such as the space of softmax scores in the case of classification. The appropriate output space for f(X) will be easy to infer from context.
ceddiction-powered inference (PPI) is a set of estimators that incorporate predictions from machine learning mod- els (Angelopoulos et al., 2023a) to get lower-variance es- timators that remain unbiased. In our case, we employ an optimized variant, P (Angelopoulos et al., 2023b), in order to estimate metrics using synthetic data. From a sta- tistical perspective, PPI is closely related to the fields of multiple imputation and semiparametric inference, perhaps most notably the augmented inverse propensity weight- ing (AIPW) estimator (Robins & Rotnitzky, 1995; Tsiatis, 2006) (see (Angelopoulos et al., 2023b) for careful re- view). Indeed, we are not the first notice this applica- tion of PPI; the work of Saad-Falcon et al. (Saad-Falcon et al., 2023) describes an autoevaluation method for evalu- ating and ranking language models from pairwise compar- generation. A
Metries We will evaluate the performance of the models by estimating the expectation of some metric function ş y x y + I; in other words, the metric of model m will be
Mm = E (m(mm(X),
(1)
for some metric function & and every m = 1, ,..., M. We are interested in estimating the M-length vector μ (#1,-.. MM). For example, in the case of the accuracy, we would want to measure accuracy.. = E [face (X)) where (y. y') ) = 1 if y = y and 0 otherwise, for every m E ...
Accuracy is not the only quantity that can be framed within this setup. As another example, when the predictors multilabel classifiers, one performance metric of inter- est could be the average precision of the model, that is, d (ū. v) = Ignvl In the case of regression, j'mm could 1øl correspond to the mean squared or absolute error of model m, in which case ((ū.,) = (ú - y)² or Φ v) = lú y). respectively. Finally, one can imagine estimating multiple losses at once; for example, for the purpose of assessing faimess, one may want to evaluate accuracy across many groups.
isons for the purpose of netrieval-augmente preprint by Chatzi et al. (Chatzi et al., 2024), posted con- currently with ours, also considers the problem of ranking models from pairwise comparisons, and constructs approx- imate rankings with coverage guarantees. Our approach is complementary to these existing works. Our specific con- tribution is to develop an instantiation of PPI I that is prac- tical and yields tight confidence intervals, is easy to imple- ment using existing software, and compatible with exist- ing evaluation systems such as Chatbot Arena (Cha; Chiang et al., 2024). Moreover, we evaluate our PPI method on real data. Along the way, we develop an interesting exten- sion of the PPI algorithms to the case where the annotation model outputs not just a single synthetic Y, but a distribu- tion over Y.
Data We assume access to two datasets: a small human- annotated dataset. {(X,, Y)) and a large amount of un- labeled data points, {X:)) Ji=1 whose ground-truth labels (Y,"),) J are unavailable. Importantly, both datasets are i.i.d.; extensions to some limited non-i.i.d. regimes are han- dled in (Angelopoulos et al., 2023a), but we will not discuss them here. One should think of the regime where N > n: we have far more synthetic labels than real ones. For both datasets and every model, we also assume access to syn-
2

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
thetic label distribution that approximates p(Y X). We denote {P and (P } the set of synthetic la- bel distributions conditioned on the labeled and unlabeled input data points, respectively. For each and m, we will use the notation ddP..m (y) to represent the estimated PDF or PMF evaluated at label y.
where A == 1( =Y, Apr. Here, À is tuning parameter- the time being, think of À = 1. The above estimator decomposes into two natural components. Inter- preting the top softmax score as the probability of correct- ness, the first term captures the model's internal estimate of its accuracy on the unlabeled data. The second term is the bias of the first term.
For the sake of intuition, we make a few remarks regarding this data generating process. First, the synthetic data dis- tributions can be seen distributions over labels produced by one or several "annotator models", that can either be re- lated or different from the models to evaluate. In the latter case, the synthetic label distribution, for a given input, is the same for each model f fxs We do not need the subscript m in this scenario, and can simply denote the syn- thetic label distribution as P.. However, the general case of Pt,m allows for each model to have different annota- tor model, and possibly allow models to self-annotate, that is, to themselves produce synthetic labels. Second, we note that the framework we described applies directly to the case where the annotator model produces single predictions of Y instead of distributions, by setting up (v) to be a delta function the prediction (the distribution is entirely concentrated on the prediction of Y).
This estimator has two beneficial properties: unbiased- ness and variance reduction. Unbiasedness means that BlAj = μ.. This implies that the inclusion of machine learn- ing predictions in our estimator does not introduce system- atic errors for estimating the accuracy. Variance reduction means that the use of synthetic data reduces the variance of our estimator: Var (im) S Var (pehassical)
This is formally true for the optimally chosen parameter x; indeed, the optimal choice of À ensures that our estima- tor is always better than ficlassicll (in an asymptotic sense). See (Angelopoulos et al., 2023b) for details and a formal proof; refer to Supplement A to see how to compute this estimator in Python.
GENERAL METRICS
The approach we have presented for evaluating classifier accuracy is an instance of more general framework for evaluating properties of machine learning models. In par- ticular, we can use our annotator model to output an ap- proximate expectation of each label {Y:) and (Y.") in the following way:
2.2. The Algorithm
We combine the labeled and unlabeled data estimate p. In particular, we seek to benefit from the large sample size of the automatically annotated dataset to produce an esti- mator with low variance, while ensuring that this estimator remains unbiased. We will begin with the case of estimat- ing accuracy, and then generalize our algorithm to arbitrary metrics.
{ [B.. = -yEy (X.)) i,m. = -,E (XF', (v).
(3)
These expressions look complicated, but have a simple in- terpretation: the annotator model, given X,, thinks the dis- tribution of Y, is dP,, and we are simply calculating the expected metric under that estimated distribution. This ex- plains the hats on the expectation symbols; these are not real expectations, but rather, estimated expectations ac- cording to the annotator model. Indeed, in the case of clas- sification, we see that, as is intuitive, the expected accuracy of the mith model on the ith data point is equal to its top softmax score:
WARM-UP: MODEL ACCURACY
The classical approach to estimating model accuracy is to compute the fraction of correct labels:
1 a 1 10 tm -Yo,,
classical i'
Σ n
where Pi,m = argmax,, fm.((,)) (X,),, and (X,) is the soft- max output of model m. Instead, we propose estimating the accuracy of a classifier differently: by using the classi- fier's own confidence on the unlabeled data as a signal of its accuracy. Let Pt.m fm (X.) .. denote the top softmax score of model m on labeled example i, and pi ÿr 4. be defined analogously. We will use the estimator
B = f Pacco
(Ymm))(.mm u)dP i. (v) = (dP...m(... = Pi.m-
Along the same lines, our previous estimator can be gener- alized to the case of arbitrary metrics as
N
pm =
N
(2)
Am
B
n
(4)
1=1
metric on synthetic data
bius correction
accurncy
bius carrection
3

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
where now A..m :- ( Y.) -XE The first sum in the above expression is the average metric predicted by model m over all synthetic labels. If the annotator model is near-perfect and N is large, then this term will almost exactly recover the metric. However, if the synthetic label distribution is not good, this can bias our estimate of the metric. The second term corrects this bias by calculating it on the labeled dataset and subtracting off.
interval marginally for each jimi
21-a/2 -aaaiVVmmn m.
C.n ipm +
(5)
vn
The above interval comes with the following (standard) guarantee for all m 1...., M: lim,, -oo P(pm
=m)=1 1-
As an alternative to producing confidence intervals for a single coordinate pmm based on Equation (5), we might want to create confidence sets that contains the entire vector μ., that is, simultaneously valid intervals. The simultaneous interval can be constructed using the chi-squared distribu- tion as
Returning to the role of the tuning parameter: À E [0, is discount factor on our synthetic data. When the syn- thetic data is very good, we can set À = 1; when it is bad, setting À= = 0 will throw it away entirely. One can asymp- totically optimize the variance of j in order to set X, as in (Angelopoulos et al., 2023b).
:=={w:n CX Cx= (u:n V 1 = xxi-a.M }
Again, is straightforward to see that for any fixed value of X, our estimator in (4) is unbiased, meaning Ela] μ4, and will be strictly lower-varianee than its classical counterpart when À is optimally chosen.
where xi-a.M denotes the 1-a quantile of the cqi-sqe distribution with M degrees of freedom. This interval has the following (standard) guarantee:
VARIANCE AND CONFIDENCE INTERVALS
As we have explained above, the main benefit of AutoE- val to reduce the number of human-labelee data points to achieve particular variance. We can formalize this by analyzing the variance of iim. and Am Aglussical, In particular, we can write the covariance matrix of je as
lim P(NE E (())C*)- Y2,N +00
-1 -x
and thus, it can be used to rank the models by checking whether the m and m' coordinates of CX overlap for each model m and m' in M. M.
C.)((TT))) + 1 (A)) n N
1 -V= 1
2.3. Application to Rank Computer Vision Models
We applied the described methodology applies for evaluat- ing computer vision models. We considered five trained computer vision models (ResNet-19, ResNet-34, ResNet-50, ResNet-101. and ResNet-152) opti- mized over the training set of ImageNet and sourced from PyTorch (Paszke et al., 2019). We considered the task of estimating their accuracy on the validation set of Ima- geNet in a low-data regime, using a subset of labeled data points. The ground-trui model accuracies were computed as the mean accuracies evaluated over the entire validation dataset.
Aa.r) This expression admits a simple plug-in estimator; it also indicates that we should pick À to minimize V in the appropriate sense. It also al- lows for the production of non-asymptotic confidence in- tervals using concentration. We opt to use asymptotic con- fidence intervals for μ. In particular, we have that as n and N approach infinity,
where A
VnD- 1/2 (p p)) + (0, IM), where V is the plug-in estimator of V, corresponding to
We considered two different approaches estimate the curacy of these models. The first is referred to as PPI (An- gelopoulos et al., 2023a), and corresponds to (2) with À = 1. The second strategy, PPI++ (Angelopoulos et al., 2023b) optimizes À to minimize the variance, with lim- ited computational overhead (Table S3). These approaches were benchmarked against elassicca along with a standard z-test confidence interval.
nd² N +
V
L
(EE
Note that when = 0, we exactly recover g claassccal but this may not be the parameter that minimizes the variance V. Indeed, we can explicitly choose À to minimize the vari- ance. An explicit expression for this estimate can be found in (Angelopoulos et al., 2023b).
To reflect low-data regime, we randomly sampled a small number of observations to be used as labeled data points available for these approaches. The rest of the observations in the validation data were used as unlabeled data points for PPI and PPI++. Our synthetic label distribution diP,
Another beneficial aspect of the asymptotic analysis is that allows us to construct confidence intervals with which we can reliably rank models. For example, coordinatewise, the following is an asymptotically valid 1 - a confidence
4

---

AuloEval Done Right: Using Synthetic Data for Model Evaluation
a.
b.
C.
PPI
0.8
0.003
1500
Rank correlationn
classical
eff. sample size
MSE
0.002
1000
0.4
0.001
500
0.2
0
0
750 # of labeled samples
250 500
1000
0
500 1000
1500
50
100
200 300
400 500 1000
of labeled samples
# of labeled samples
Figure 1. ImtgeNNtt experiment. For every approach, we huilt confidence intervals around the average accuracy of different ResNet architectures. MSE of the point estimates of the model accuracies. b. ESS of PPI and PPI++ against the classical approach. Correlation between the estimated and true model rankings, Here, and in all following figures, obtained metries are averaged across 250 random splits of the validation data into labeled and unlabeled data.
the softmax vector of model m on labeled data point i; dPu for an unlabeled data point analogous. 1,m
using larger sample sizes, we rerun this experiment with n 10, 000 labeled data points (Table S4). This experi- ment confirmed, among other things, that PPI++ provided more accurate point estimates and tighter confidence inter- vals than the classical approach.
The mean-squared error of our estimates of the model ac- curacies improved over the classical baseline (Figure la).
Both PPT and PPI++ + had lower mean-squared errors than the baseline, no matter the size of the labeled set. Little to no difference was observed between PPI and PPI++ which probably means that the imputed accuracy scores are reliable proxies for the true quantities. Our approach hence provided more accurate point estimates of the model accu- racies. When uncertainty quantification does matter, PPI and PPI++ provided calibrated confidence intervals across all labeled set sizes, and produced tighter confidence inter- vals than the baseline (Figure 6).
2.4. Application to Evaluate Protein Fitness Prediction Models
We also used AutoEval to rank regression models, and more specifically, protein fitness prediction models. Pro- tein fitness prediction is a crucial task in computational bi- ology, aiming to predict the biological relevance of protein mutations based on their amino acid sequences. The recent development of deep learning models for protein language modeling has enabled the emergence of powerful models, trained millions of protein sequences, used to predict protein fitness in zero-shot manner (Meier et al., 2021). Unfortunately, evaluating these models for specific task remains challenging due to the scarcity of experimental data that can be used for evaluation, typically requiring ex- pensive, time-consuming. and poorly scalable wet-lab ex- periments (Laine al., 2019; Riesselman al., 2018).
The benefit of using unlabeled data can measured by computing the effective sample size (ESS) of PPI and PPI++ relative the classical approach (Figure 1b). This value can be interpreted as the equivalent number of la- beled data points for the classical approach that would be required to achieve the same level of precision as PPI or PPI++. Our ESS exceeds that of the classical approach by approximately 50%, which demonstrates the utility of unlabeled data for evaluating model performance.
We applied AutoEval ProteinGym (Notin et al., 2023), which gathers several assays containing both experimental fitness measurements, used as ground-truth labels, and pre- dicted fitness scores from various fitness predictive models. We focused on ranking protein language models for pre- dicting the fitness of mutations in the IgG-binding domain mutations of protein G based pairwise mutations (Olson et al., 2014).
Here, and in the other experiments, also evaluated our approach for the purpose of model ranking, by ranking models based on their confidence intervals after Bonferroni correction. Models with overlapping confidence intervals were considered tied. Figure 1c shows the correlation of the estimated model ranks with the ground truth ranking (computed on all data) for different n and averaged across data splits. This experiment showed dra- matic differences between the approaches. PPI++ showed much stronger correlations with the ground truth than the other approaches, meaning that its rankings were more ac- curate and less prone to ties.
assay of N 536, 962
We considered scenario where one aims to select the best model for zero-shot fitness prediction for a specific pro- tein, using a small experimental dataset and a large set of potential mutations for which fitness not measured. We focused on the Pearson correlation between predicted and experimental fitness scores validation metrie for rank-
To confirm the applicability of our approach to scenarios
5

---

AuloEval Done Right: Using Synthetic Data for Model Evaluation
a.
b. 1500
C.
0.8
PPI
classical
eff. sample size
Rank correlation
0.6
1000
MSE
0.010
500
0.005
0.2
0
250 of labeled samples
500
750
1000
500 of labeled samples
1000
1500
50
100
200
300
400
500
1000
labeled samples
Figure 2. Protein fitness experiment for building confidence intervals and point estimates for the Pearson correlation of seven protein language models with the experimental fitness scores, using held-out model produce synthetic lahels. a. MSE of the point estimates of the model correlations. b. ESS of PPI and PPI++ against the classical approach. Correlation between the estimated and true model rankings.
ing models. More specifically, we aimed to estimate the metric pmm fm (X)], where Y, X, fm are the exper- imental fitness, the protein sequence, and the m-th fitness predictor, respectively, assuming that Y and (X) have zero mean and unit variance. This fits in our general metric evaluation framework, where the metric function in Equa- tion (1) is o(y. 1/) = yy'.
with different annotator models. As expected, the better the annotator model, the higher the effective sample size. We importantly note that even with very poor annota- tor model, PPI++ performs at least well as the classi- cal approach. When the annotator labels do not correlate with the true labels, PPI++ falls back to the classical ap- proach (A = 0), effectively ignoring the synthetic labels. That being said, we observe that even mediocre annotator models, such as CARP, provide a 10% increase in effec- tive sample size compared to the classical approach. Alto- gether, these observations suggest that AutoEval can pro- vide better point estimates and tighter confidence intervals compared to the classical approach even when the annota- tor model is mediocre.
We used VESPA (Marquet et al., 2022), a protein lan- guage model, as a held-out annotator model for synthetic label generation. VESPA produce point estimates of the experimental fitness scores fxesea(X,) based on the pro- tein sequence X; aiming to approximate the experimental fitness scores Y,. While VESPA does not provide uncer- tainty estimates for its predictions, we can still use it as an annotator model our framework. Specifically, we ap- plied our estimator to estimate model m 's Pearson cor- relation with the experimental fitness scores, by setting R,,m fm (vSS) A = fm (m,)) Am (vrssx for the bias cor- rection term in Equation (4).
VESPA
650
for the synthetic term and
Progen2 ESM2
ESS
600
ESM1b
RITA
550
CARP ESM1v
The results of this experiment are shown in Figure 2. The effective sample sizes of PPI++ were systematically higher than the classical approach (Figure 2b), by approx- imately 50%. Furthermore, the ranks obtained by our ap- proach were also much closer to the true model ranks than the classical approach (Figure 2c), with a five-fold im- provement for 1000.
Unirep
0.2
0.3
0.4
0.5
corr. of the annotator pred. with GT
Figure 3. ESS of PP +++ against annotator model performance for 500 labeled points for the protein fitness experiment. The horizontal line denotes the ESS of classical.
PPI++ confidence intervals for models' correlations with the experimental fitness scores were also slightly tighter than the classical approach, yet remained calibrated (Fig- ure 6). The PPI estimator performed worse than the clas- sical approach. This is a known issue of this estimator, that PPI++ mitigates.
3. Evaluating Model Performance from Pairwise Comparisons
Characterizing the absolute performance of MI models for the purpose of ranking them is challenging. The previous section described a methodology to compare models based on a common performance metric. Unfortunately. metrics serving as proxies for model performance might either not
A question remains: how good does the annotator model need to be to allow AutoEval to work well? Figure 3 compares the effective sample size of PPI++ obtained
6

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
exist, diverge from human judgment (Ji al., 2023).
logistic regression estimate of the BT coefficients is
In such cases, assessing relative model performance might be more appropriate. This can typically be done by compar- ing different model predictions to each other. The Chatbot Arena project (Cha), for instance, allows human annotators to state preferences over different LLM predictions to the same prompt. Comparison -ased evaluation is also an ex- citing opportunity for autoevaluation (Zheng et al., 2024; Li et al., 2024b). In particular, an external LLM, prompted to serve as an annotator, agrees with human annotators with high fidelity. Still, it is unclear how biased an Al annotator might be, which drastically limits the usefulness of the val- idation data it produces. (Li et al., 2024b) conducted stud- ies into biases within Al annotators, such as stylistic and model-specific biases, highlighting the need for more ro- bust inference. This section describes how to leverage such Al-generated preferences while making statistically valid inferences about model performance.
Colassical argmin 1 Σ
(x,, Yi),
CER CERM-1 G==0 ==1
where is the binary cross-entropy loss.
3.2. Autoevaluation of Relative Performance
rrediction-poweree inference can be applied out-of-the- box to the BT model, making it possible to leverage large numbers of Al-generated preferences while controlling for their potential bias. In addition to the set of human pref- erences defined above, additionally define the unlabeled dataset {(AH, B ) On both the labeled and un- labeled datasets, we have the prompt and both models' an- swers; we use LLM in place of the human to choose be- tween the answers. This gives us a prediction Y; and Y of the pairwise comparison on both datasets. The P estimator of the BT coefficients given by
3.1. A Model to Assess Relative Performance
8- argmin 1 Σ
(x(x,,)) - - NeG(X, Ki)
The canonical model for assessing relative performance of models based on pairwise comparisons, as in a tournament. is called the Bradley-Terry (BT) model (Zermelo, 1929; Bradley & Terry. 1952; Ford Jr, 1957). The BT model used in the Chatbot Arena (Chiang et al., 2024), by the World Chess Federation, the European Go Federation, and many other competitive organizations tool for ranking players.
CERM-1 6-
7m1
(6)
N (x("" Y,"),
where À [0. 1] controls the weight we give to the AI- generated preferences. Although this estimator departs from the arguments given in Section 2, it has very similar interpretation; it constructs an unbiased and lower-variance loss function for the true logistic regression, and then min- imizes it.
Now we describe the BT model. Imagine, among M mod- els, we are trying to compare the strength of model A to the strength of model B. Towards this end, we give a prompt Q to both models, and they give us an answer. We show this answer to a human, who gives us Y = 11i if the answer of model B is better than the answer of model A, and vice versa. The assumption of the BT model is that Y follows logistic relationship.
The resulting BT coefficient estimates have the same ap- pealing properties as above. In particular, they are unbi- ased for any fixed X, and one can construct valid confidence intervals around them using PPI and PPI++; see (An- gelopoulos et al., 2023a;b) for this and other generalized linear models, as well as methods for optimally choosing x. Supplement A describes how to compute this estimator easily Python.
1
= 2Y==11A7 A, 1 + ctA CCa
with some parameter vector of length M, whose entries are called the Bradley-Terry coefficients. Each model has BT coefficient Sm which, when large relative to the other coefficients, signifies that it is more likely to win the pairwise comparison. (Also, because the model in (3.1) is invariant to addition of constant to every coordinate of Ç. we can, without loss of generality, set Çi = O, making the model identifiable.)
3.3. Autoevaluation of LLMs from Pairwise Preferences
We evaluated our approach on the Chatbot Arena project (Chiang et al., 2024). We first extracted 16K ob- servations from the Chatbot Arena dataset, in which a total of 20 recent LLMs were compared (Table S1). Each ob- servation contains a prompt written by a human, responses from two of the 20 LLMs, and the preference of the hu- man over the two responses. For each observation, we used as a judge (Zheng et al., 2024) by prompt- ing it to identify the most useful response, following the
It is well-known that, given labeled dataset of n pairwise comparisons, {A, B,, Qi, Y.). the maximum-likelibon estimator of the BT coefficients is a logistic regres- sion (Hunter, 2004). Let X, be the vector of all zeros except at indexes A, and B,, where is I and respectively. The
7

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
a
b.
C.
0.6
500
Rank correlation
0.4
MSE
0.8
sample
400
0.2
300
0.4
200
0.0
200
300
400
500
200
300
400
500
200
250
300
350
400
450
500
# of labeled samples
of labeled samples
# of labeled samples
Figure 4. LLM experiment for building confidence intervals and point estimates for the BT coefficients of different LLMss. a. MSE of the point estimates of the BT coefficients. b. ESS of PPI and against the classical approach. Correlation between the estimated and true model rankings.
GPT-4o-mini
Discussion
260
GPT-4o
AutoEval is promising direction to reduce the cost and effort of model evaluation. We have presented a method- ology that makes it possible use such synthetic data for model evaluation, improving over classical approaches in statistically rigorous way. Our implementation is available as a Python package.
Claude Sonnet
ESS
240
LLama 3.1
Gemini 1.5
220
200
0.20
0.28 corr. of the annotator pred with GT
0.22
0.24
0.26
It is worth noting that the statistical methods we have pre- sented apply beyond autoevaluation. Indeed, many setups involve trading off low-quality or unreliable validation la- bels, which are plentiful, with high-quality but scarce vali- dation labels. Our methodology applies readily to such se- tups, and could, for instance, help integrate crowd-sourced validation labels with expert validations.
Figure 5. FSS PP against annotator model performance for 200 labeled points in the I.M experiment. The horizontal line denotes the ESS classical.
One limitation of our approach that it requires the cu- rated expert and autoevaluated data to be representative of the data in production. These distributional shifts typically arise when the labeled inputs are not sampled uniformly at random from the unlabeled pool, or when labeled and unlabeled data points come from different populations al- together. such cases, AutoEval, in the form described in Equations (4) and (6), loses the statistical guarantees we outline the paper, and confidence intervals may no longer be valid. To address this issue, we derived alternative Auto- Eval estimators that are robust to covariate shifts (see Sup- plement B).
same prompting approach as (Li al., 2024b). We focused on scenarios where only few of the 16K human prefer- ences were available, and compared our approach (using both available human preferences and all gpt- 4o-t-i preferences) to the classical approach.
Results are shown in Figure 4. We observed that the BT coefficients were better estimated by PPI++ than by the classical approach, hinting that the point estimates of Auto- Eval are more accurate (Figure 4a). PPI++ also produced calibrated, and tighter confidence intervals than the classi- cal approach (Figure 6, Table S5) We also observed ESS showing a 20% to 25% improvement over the classical ap- proach (Figure 4b). Finally, we observed that the estimated rankings of the models were more correlated with the true rankings when using + (Figure 4c).
Finally, is interesting to consider other metrics by which one could evaluate models with PP I-type approaches. Herein, we handled mean estimation and logistic regres- sion, but the framework can do more. One might want to evaluate other metrics, such as fairness and bias, e.g., via estimating the least-squares coefficients relating sensitive attributes and prediction error. For any deployed machine learning system, it is important to test many of these met- rics to ensure good performance, in which case having pre- cise estimates and tight confidence intervals becomes espe- cially important.
We also studied other choices of LLM judges (Figure 5). Similar to the protein fitness experiment, the quality of the LLM judge had a large impact on the performance of our approach. Furthermore, we observed that all considered judges allowed our approach to outperform the classical ap- proach, obtaining ESS improvements between 20% to 35% depending on the L.M judge.
8

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
PPI
PPI
PPI++ classical
PPI++ classical
0.4
0.95
0.15
coverage
CI width
coverage
width
0.9
0.10
0.2
0.80
0.05
0.8
100 200 300 400 500 labeled samples
100 200 300 400 50c # of labeled samples
100 200 300 400 500 # labeled samples
100 200 300 400 500 labeled samples
(a) ImageNet: estimation of model accuracy.
(b) Protein fitness: estimation of model correlation
1.00
PPI PPI++ classical
2.4
coverage
0.95
2.2
2.0
0.90
1.8
1.6
0.85
200
300
400
500
200
400
500
# of labeled samples
# of labeled samples
(c) LLM: estimation of BT coefficients
Figure 6. Interval metrics for the different experiments. Coverage (left) and width (right) of the 90%-ccofiddnce intervals. Each experiment is described in the main text and focuses the estimation of different metric.
Software and Data
ing systems.
All code used to reproduce this work is available as supplementary materials available on OpenReview. We refer the reader to Supplement for details on the experimental setup. Tools to apply the described methodology for model evaluation Python package, available at https:/ ////ttttbbbbbb// com/ eangelopoulos/ppe srmioos/ppi_y.... experiments is available at https:: httpps:/githhbb.. com/ PierreRoyeat/auttert
References
Chatbot Arena: Benchmarking LLMs in the wild. https: //1marena.a ai/. Accessed: 2025-01-21.
available
Agarwal, M., Kalia, R., Bahel, V., and Thomas, A. Au- toeval: An NLP approach for automatic test evaluation system. 2021 IEEE 4th International Conference Computing, Power and Communication Technologies (GUCON), pp. 1-6. IEEE, 2021.
Code to reproduce the
Acknowledgements
Mohamad, D., van Zwet, E., Solari, A., and Goeman, J. Simultaneous confidence intervals for ranks using the partitioning principle. Electronic Journal of Statisties, 55:2608-6646 2021.
The authors would like to thank Tijana Zmić for her feed- back and Jessica Dai for helpful conversations. This work was supported part by the Vannevar Bush Faculty Fel- lowship program under grant number N00014-21-1-29441
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. Prediction-powere inference. Sci- ence, 382:669 674, 2023a.
Impact Statement
AutoEval provides principled strategy to facilitate model evaluation in low-data regimes, which is relevant to deploy reliable machine learning systems. AutoEval could also make easier to certify algorithmic fairness, e.g., by es- timating the least-squares coefficients relating sensitive at- tributes and prediction error. Our methodology could also enable more efficient human oversight of model evaluation, which remains crucial to responsibly deploy machine leamn
Angelopoulos, A. N., Duchi, J. C., and Zrnic, T. Ppi++: Efficient prediction poweerd inference. arXiv preprint arXiv:231 1.01453, 2023b.
Arazo, E., Ortego, D., Albert, P., O'Connor, N. E., and McGuinness, K. Pseudo-labeling and confirmation bias in deep cupi-supe learning. In 2020 International joint conference on neural networks (IJCNN) pp. 1-8. IEEE, 2020.

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
Bachman, P., Alsharif, O., and Precup, D. Learning with pseudo-ensembles. Advances in neural information pro- cessing systems, 27, 2014.
effects. Molecular Biology and Evolution, 36(1 1):2604-
2619, August 2019.
Laine, S. and Aila, T. Temporal ensembling for semi- supervised learning. arXiv preprint arXiv:1610.02242. 2016.
Bradley, R. and Terry, M. E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Lee, D.-H. al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural net- works. In Workshop on challenges in representation learning, ICML,, volume 3, pp. 896. Atlanta, 2013.
Chaganty, A. T., Mussman, S., and Liang. P. The price of debiasing automatic metrics in natural language evalua- tion. XrXiv preprint 18070000.,02202, 2018.
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., and Liu, Y. juddge:-jugge:: A comprehensive sur- vey on llm-based evaluation methods. arXiv preprint arXiv:2412.0557, 2024a.
Chatzi, I,, Straitouri, E., Thejaswi, S., and Rodriguez, M. G. Prediction-powered ranking of large language models. arXiv preprint arXiv:2402.17886. 2024.
Chiang, W.-l.., Zheng, I.., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, E., and Stoica, I. Chatbot arena: An open platform for evaluating LLMS by human preference. arXiv preprini axX1:2443.004132. 2024.
Li, T., Chiang, W.-L.., Frick, E., Dunlap, L.,, Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, From crowdsourced data high-quality benchmarks: Arena-hard and bench- builder pipeline. arXiv preprint Stti:2400.119333 2024b.
Corston-Oliver, S., Gamon, M., and Brockett, C. A ma- chine learning approach to the automatic evaluation of machine translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguis- tics, pp. 148- 155, 2001.
Lin, C.-Y. and Och, F. Looking for a few good metrics: Rouge and its evaluation. In Nicir workshop, 2004.
Marquet, C., Heinzinger, M., Olenyi, T., Dallago, C., Erek- ert, K., Bernhofer, M., Nechaev, D., and Rost, B. Em- beddings from protein language models predict conser- vation and variant effects. Human Genetics, 141(10): 1629-1647, October 2022.
Ford Jr, L. R. Solution of ranking problem from binary comparisons. The American Mathematical Monthly, 64 (8P2):28-33) 1957.
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., and Rives, A. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, pp. 0221007.0.440644 November 2021.
Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. Leveraging unlabeled data to pre- dict out-of-distribution performance. arXiv preprint arXiv:2201. 04234, 2022.
Notin, P., Kollasch, A. W., Ritter, D., van Niekerk, L., Paul, S.. Spinner, H., Rollins, N., Shaw, A., Weitzman, R., Frazer, J., Dias, M., Franceschi, D., Orenbuch, R., Gal, Y., and Marks, D. S. ProteinGym: Large-Scale bench- marks for protein design and fitness prediction. bioRxiv, pp. 2023.1 12.07 27777777 December 2023.
Hastie, T., Tibshirani, R., Friedman. J. H., and Friedman, J. H. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2009.
Huang, H., Qu, Y., Bu, X., Zhou, H., Liu, J., Yang, M., Xu, B., and Zhao, T. An empirical study of llm-as-a-judge for Ilm evaluation: Fine-tuned judge model is not a general substitute for gpt-4. arXiv preprint arXiv:2403.02839 2024.
Olson, C. A., Wu, N. C., and Sun, R. A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain. Current Biology, 24(22):2643- 2651, November 2014.
Hunter, D. R. MM algorithms for generalized Bradley- Terry models. The annals of stutistics, 32(1):384-40) 2004.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Associ- ation for Computational Linguistics, pp. 311-318, 2002.
Ji, 7.,, Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E.,, Bang, Y. Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Laine, E., Karami, Y., and Carbone, A. GEMME: A sim- ple and fast global epistatic model predicting mutational
10

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
Riesselman, A. J., Ingraham, B., and Marks, D. S. Deep generative models of genetic variation capture the ef- fects of mutations. Nature Methods, 15(10):8168222, September 2018.
Robins, J. M. and Rotnitzky, A. Semiparametric effi- ciency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 129, 1995.
Saad-Faleon, J. Khattab, O., Potts, C., and Zaharia, M. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476, 2023.
Tsiatis, A. Semiparametric Theory and Missing Data. Springer, 2006.
van Breugel, B., Seedat, N., Imrie, F., and van der Schaar, M. Can you rely on your model evaluation? improving model evaluation with synthetic test data. Advances in Neural Information Processing Systems, 36:1889-1904 2023.
Wei, J. T.-Z. and Jia, R. The statistical advantage of auto- matic NLG metrics at the system level. arXiv preprint vrXiv:2105.12437 2021.
Zermelo, E. Die berechnung der turnier-ergebnisse als ein maximumproblem der wabrscheinlcchketssrchnngg Mathematische Zeitschrift, 29(1))433 460, 1929.
Zheng, L., Chiang. W.-L.., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, E., and Stoica, I. Judging LLM-as---Judgee with MTT-Bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
A. Code snippets
This section provides code snippets to produce confidence intervals and point estimates for model accuracy and pairwise comparisons with the existing Python package ppi py (Angelopoulos al., 2023a).
from ppi_py import ppi mean Laaeaeeeeannneeeaae pointestimate, ppi mean_ opi_mean___
y labeled (n,) ground-truth values of Y_i on labeled dataset
yhat_ labeled (n,M) predicted values Y_ for each model on labeled dataset
P_ p.labeled (n,M) top softmax scores of each model on labeled dataset
p_untabeled <- (N,M) top softmax scores of each model on unlabeled dataset
# alpha <- (float) error rate of conf idence interval
corrects (yhat_ labeled = y_ y_labeled[: [NNnee) .astype (float)
hat_mu = ppt__ean_pooineeeeeeeeepee
p_ p_labeled, p_unlabeled) u_nllbblld, alpha=alpha)
ci mu ciforrects,, p labeled,
Snippet Python code to produce CIs and point estimates for model accuracy. The variable meanings are explained the code comments.
from ppi_py Import ppL_Logistic.ponntettimee api_logistic_ct
X_labeled (n,M-1) one row per battle;
model A, for model B, else y_labeled (n,) if model A wins, if model wins
#
<- (n,) predicted values of Y_t on labeled dataset
# X_unlabeleed <- (N,M-1) one row per battle; -1 for model A, 1 for model B, 0 else
# yhat_unlabeled <- (n,) predicted values of Y_i on unlabeled dataset
# alpha <- (float) error rate of confidence interval
bat_zeta ppi ppp_logistic.pponnettitttet
X labeled, y_ y_labeled, yhat, labeled,
X X_unlabeled, yhat_unlabeled
ci ppi logistic_cit cil
X. Labeeedd, y labeled, that_labeled,
x X_unlabeled, yhat_unlabeled, alphamalpha
Snippet 2: Python code produce CIs for the Bradley-Terry coefficients (without multiplicity correction). The variable meanings are explained in the code comments. For clarity, the matrix X. x.labeles has one row per pairwise comparison. The ith row is two-hot vector, with -1 at position A, and +1 at position B. The matrix X_unlabe led is analogous. Note that x_1 lbeled and X_un abe ed have only M columns, since Çi does not need to be estimated.
B. Handling covariate shifts
We here describe alternative AutoEval estimators that can be used to handle covariate shifts. We focus here on a scenario where the labeled inputs X; are obtained from different distribution Qx than the distribution of interest Px. More
12

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
particularly, assume that for any n,
Xi fnd Qx
Yi x; itd. Prix,
while X f...d. Px for j S N, and YJ" Xy ~ Prixy (though r are unobserved). The rest of the assumptions, and notations, are as in the main text.
We assume that the Radon-Nikodym derivative dPx/dQx is known, such that an importance sampling approach can be used correct for the distributional shift.
Table S1. Coverage for 0.1 in the ImageNet experiment under covariate shifts. To introduce covariate shifts, we sampled labeled data points weighted by the probability predicted by ResNet-101 on one of the 1000 ImageNet classes. Importance weights were estimated using self- normalized importance sampling.
Sample size
Unweighted
Reweighted
n 50
0.5044
0,9128
N 100
0.3780
0.9196
200
0.2444
0,9184
300
0.1992
0.9376
400
0.1692
0,9252
= 500
0.1572
0.9320
Using the same notations as the main text, the reweighted AutoEval estimator for metric estimation (Equation (4)) becomes:
Am
u 1,m
Σ
(7)
N
where A (X.)) XE4...
Similarly, for pairwise comparisons, the reweighted estimator modifies Equation (6) as
غ = argmin M-RM-1 G==0
w (X,)) - -Xx,(,,)))))xx + N l (X ('",
(8)
Table S1 shows the coverage for the reweighted AutoEval estimator in the ImageNet experiment under covariate shifts. As expected, the original AutoEval estimator does not provide calibrated confidence intervals due to the broken exchangeabil- ity assumption between labeled and unlabeled data points. The reweighted AutoEval estimator, on the other hand, provides calibrated confidence intervals.
C. Experimental details
C.1. Data acquisition and preprocessing
ImageNet We downloaded model weights from PyTorch's model zoo for the different ResNet models, trained on the training set of ImageNet. We then computed the different models' predictions on the validation set of ImageNet on a high-performance computing cluster.
Protein fitness We relied on ProteinGym' to access both the ground-truth fitness values and the predictions of the different protein language models for a specific assay corresponding IgG-binding domain mutations of protein G All fitness scores were normalized as a preprocessing step.
(SPGL.9TR8C.01son22011
Phttps:///ht http:/tttbb....ooeeeooeeee wwt.////....aaaaaae...... wow.://Aiii..aaarsssaaaaa/ooooo. nGy
13

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
LLM We considered a subset of the Chatbot Arena dataset aiming to rank twenty recent LLMs (Table S2). The data contained 16K human preferences similar procedure in (Zheng et al., 2024), we prompted judge LLM (gpt-4 1opt-4es 4o (o-mini)) to provide its own preferences for the same prompts and LLM answers.
pairs of LLM answers the same (human-provided) prompts. In parallel, using
Table S2. Overview of the evaluated language models. Models are grouped by their base architecture family//rovvier..
Family
Model
Version/Date
OpenAl
GPT-4
2023-03-14
GPT-4
2023-06-13
GPT-4-Turbo
2024-04-09
GPT-4 Preview
2023-11-00 1-06
GPT-4-Online
2024-05-13
Anthropic
Claudc-3-Opus
2024-02-29
Claude-3-Sonnet
2024-02-29
Claude-3-Haile
2024-03-07
Google
Gemini-1.55 1.5-Pro
2024-05-14 14
Gemini- 1.5--11.5FFlf
2024-05-14 14
Meta
LLaMAA---70B-Inttree
LLaMAA-3-8B-Intrrucc
Mistral AI
Mistral-Large
2024-02
Other
Command-R
Command-R- R--ula
PHI-3-Medium-44-Inneee
Qwen-1.5-72B-Cha
Owwn-2-72B-Inntrree
Starling-LM-7B-BB
Y1-1.5-34B-Chr
C.2. Methodological details
Monte Carlo trials In all experiments, we randomly split the data into labeled and unlabeled sets 250 times, and com- puted all point estimates in the main text and in this supplementary material as the average estimate over these splits.
Model ranking To rank models with the different estimators, we computed 90% confidence intervals for the different approaches after Bonferroni correction. Models with overlapping confidence intervals were assigned the same rank.
C.3. Experimental setup
All AutoEval experiments were run on a workstation with 12th generation Intel (R)Core (TM) 19-129K,KF, 128GB of RAM, and on compute cluster relying on CPU nodes with four cores. We relied on the Python package ppf py, except for the LLM experiment, for which relied on Jax implement PPI and PPI++ for the Bradley-Terry model.
14

---

AutoEval Done Right: Using Synthetic Data for Model Evaluation
D. Additional experiments
D.I. Running times
Table S3 compares the running times for AutoEval with the classical approach for the ImageNet experiment. There, the main bottleneck for AutoEval consisted in obtaining synthetic labels, with prediction time scaling approximately linearly with the number of unlabeled examples. Once these labels are obtained, AutoEval extremely fast, in few milliseconds.
Table S3. Runtime comparison between AutoEval and the classical approach for the evaluation of ResNet- 101 in the ImageNet exper- iment (n = 1, 000, N = 50, 000). This experiment was run on workstation with an Nvidia RTX 3090 GPU, 128GB RAM, and an 19-12900KF CPU.
Method Prediction (s)
Inference (ms)
Classic
5
0.3
PPI++
237
8.3
D.2. Larger sample sizes
Table S4. ImageNet experiment for 10, 000.
Method
MSE (x10-9)
Interval width
Coverage
Efficiency ratio
Classic
1.43
1.37
0.93
1.00
PPI
1.07
1.21
0.931
1.27
PPI++
1.03
1.19
0.93
1.29
D.3. Coverage analysis
Table Coverage analysis different confidence levels the LLM experiment. intervals that contain the ground truth parameter across different significance levels (a).
Values represent the proportion confidence
Method a = 0.05
a 0.1
a 0.15
a = 0.2
Classic
0.96
0.90
0.85
0.80
PPT
0.97
0.92
0.88
0.82
PPI++
0.95
0.90
0.85
0.79
15


Reviews:
[{"review_id": "ULalwETIX9", "reviewer": "Reviewer_GnRr", "summary": "The goal of this work is to reduce the cost and time of evaluating machine learning models using AI-labeled synthetic data. Introduces algorithms for auto evaluation that improve sample efficiency while remaining unbiased.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments. We appreciate your recognition of our work's practical relevance in reducing ML evaluation costs and the strength of our benchmark.\n\nPlease find below point-by-point answers to your comments.\n\n**Positioning of our approach relative to semi-supervised training strategies.**\n\n> This problem has been tackled in the literature with different names[...]. It is important to compare [...] against previously published work.\n\nThank you for the comment. Although these previously published topics handle related topics, our work is very different and does not fit in these existing areas. The fundamental reason is simple: our work is about evaluation, and your suggested topics are about model training.\n\nTo elaborate, pseudo-labeling uses predicted labels for unlabeled data as ground-truth during training (Lee 2013; Arazo et al. 2020).  Curriculum learning is another model training paradigm that progressively increases training difficulty, e.g., via multi-stage training, to improve generalization (Bengio et al. 2009). Consistency regularization aims to enforce semantically similar inputs to have similar label predictions, relevant in semi-supervised training settings (Bachman, Alsharif, and Precup 2014; Laine and Aila 2016).\n\nThe key difference is that our focus is on the reliable evaluation of already trained models, rather than the training of these models on limited data.  Our framework provides strong guarantees on the behavior of model performance estimates which is not the case for the mentioned literature. While these guarantees may not be relevant in a model training setting, they become crucial when evaluating models before deployment.\n\n> The proposed problem statement is relevant for training samples as well.\n\nWhile PPI can be employed for training purposes, this direction is beyond our scope, and we make no claims for training applications. We plan to revise our related work to cover synthetic label for model training to emphasize distinctions with our work.\n\n> Evaluation of LLM responses [...] has also been studied in measuring uncertainty using structural similarity and other pair-wise metrics.\n\nThank you for pointing towards this related research area. A growing body of literature has indeed focused on evaluating LLMs from pairwise comparisons, often using LLMs as judges. We will revise the manuscript to include a more comprehensive and detailed discussion of this literature by including the references mentioned by Reviewer f29Q.\n\n\n**Benchmark**\n\n> The effective sample sizes considered are very small, and important to evaluate the approach with larger datasets.\n\nThe labeled sample sizes in our work are small to illustrate practical settings with limited human validation; however, the results certainly hold for larger sample sizes.  To address this point head-on, we studied the behavior of AutoEval for larger sample sizes in the imagenet experiment, as shown below:\n\nTable: ImageNet experiment for $n=10,000$. \n| Method | MSE (1e-5) | Interval width | Coverage ($\\alpha = 0.1$) | Efficiency ratio ($ESS / n$) |\n|---------|-----------|----------------|-----------|-----------------|\n| classic | 1.43 | 1.37 | 0.93 | 1.00 |\n| PPI | 1.07 | 1.21 | 0.931 | 1.27 |\n| PPI++ | 1.03 | 1.19 | 0.93 | 1.29 |\n\nAbove, AutoEval compares favorably to the classical approach for pointwise performance evaluation. It also provides tighter confidence intervals, large effective sample sizes, and proper coverage.\n\n> The claim that proposed methodology provides calibrated and tight confidence intervals is clearly apparent [...]. Need to compare with other principled uncertainty metrics.\n\nTo answer your comment, we conducted additional analyses to evaluate confidence interval calibration across significance levels (0.8, 0.85, 0.9, and 0.95), and compared directly with the classical approach. The results, to be included in the revised Supplement, demonstrate our confidence intervals maintain proper calibration across these significance levels.\nEmpirical coverage closely matches the nominal coverage, indicating well-calibrated uncertainty estimates for AutoEval. We hope this addresses your concern, and would be happy to implement other metrics if you have other suggestions.\n\n> The approach is evaluated on Imagenet, protein fitness experiment and pairwise preferences for LLMs. The evaluation is limited.\n\nThe current manuscript describes several canonical applications of our framework, for a variety of tasks, in model evaluation and ranking across different domains. We believe that the current benchmark offers a comprehensive evaluation of the proposed framework in a variety of real-world settings.\n\n\n**Other comments**\n\n\n> “There is no clear theoretical justification for PPI++ metric.\n\nWe would be happy to take your suggestions into account to clarify the distinctions between PPI and PPI++ if our description lacked clarity. Could you expand on which specific section of the paper you are referring to?", "claims_and_evidence": "•\tThis problem has been tackled in the literature with different names: pseudo-labeling, curriculum learning, consistency regularization etc. It is important to compare the proposed method against previously published work.\n\n•\tEvaluation of LLM responses from pair-wise responses has also been studied in measuring uncertainty using structural similarity and other pair-wise metrics.\n\n•\tThe proposed problem statement is relevant for training samples as well.\n\n•\tThe effective sample sizes considered are very small, and important to evaluate the approach with larger datasets.\n\n•\tThe claim that proposed methodology provides calibrated and tight confidence intervals is clearly apparent (presented in supplemental material). Need to compare with other principled uncertainty metrics.", "methods_and_evaluation_criteria": "The approach is evaluated on Imagenet, protein fitness experiment and pairwise preferences for LLMs. The evaluation is limited.", "theoretical_claims": "There is no clear theoretical justification for PPI++ metric.", "experimental_designs_or_analyses": "see above sections.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LlA2FBPOlz", "reviewer": "Reviewer_3tuY", "summary": "This paper studies an important question in auto-evaluation --- how to efficiently combine model prediction (imputed output) for abundant unlabeled data with limited gold-standard data to obtain efficient estimation for expected metrics for underlying distributions. The paper's method is a direct result from a recent line of paper in prediction-powered inference, the only difference is that they used for evaluation instead of inference. Experiments have been done for interesting applications such as pairwise comparison etc.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed and thorough feedback. We are grateful for your acknowledgement of the importance of the problem we are tackling, on the clarity of our claims and the methodological soundness of our approach.\nYour review raises a number of points relating to the theoretical contributions and to the experimental design of our work. Please find below our detailed response to these points, that we hope will address your concerns.\n\n**Lack of theoretical contributions.**\n\nWe would like to address your concerns about the theoretical contributions of our work. While our approach indeed builds upon the prediction-powered inference (PPI) framework, it contains several novel theoretical and modeling aspects, relevant both in the context of auto-evaluation and PPI. \n\n\nThe first contribution is to build a general framework for model evaluation using synthetic labels. \nFor metric-based evaluation, our framework accommodates various and nontrivial synthetic annotation schemes (e.g., external model annotation, or self-annotation, see Section 2.1), and allows both for hard or soft synthetic annotations. We also extend this framework to comparison-based evaluation, enabling rigorous model performance inference from pairwise comparisons.\nOur work also introduces chi-squared confidence sets for PPI, an important extension of the original PPI machinery that is not covered in the original papers. Indeed, PPI and its extensions typically focus on simple, low-dimensional inference tasks.  The realization that chi-squared confidence sets can be used may open up new research directions to extend the PPI framework to higher-dimensional problems.\nThese developments, along with the demonstration of the practical applicability of our framework across a variety of tasks and domains, highlight that our work is more than a direct result of the PPI framework.\n\nMore importantly, we believe our work is particularly timely and of significant scientific relevance given the growing need for efficient evaluation of increasingly complex machine learning models across scientific disciplines. This need is particularly obvious in the context of LLMs, where even crowdsourced validation like Chatbot Arena (Chiang et al. 2024), studied in our work, often suffers from insufficient data to resolve ties between models.\n\n\n**Marginal improvements relative to the classical approach**\n\nThe other concern we would like to address relates to the perceived magnitude of improvement over the classical approach. In our experiments, AutoEval reaches efficiency ratios (effective sample sizes over sample sizes) between 1.25-1.40.  Another way to appreciate these numbers is to think about the width of the obtained confidence intervals. An efficiency ratio of 1.25 translates into a 11% reduction in the width of confidence intervals, which, as shown in our experiments, significantly facilitates model ranking. See for instance Fig. 1c, where tighter confidence intervals translate for most sample sizes into at least a two-fold improvement in predicted rank correlation with ground-truth.\n\nWe believe these numbers are meaningful and highlight the practical relevance of our work in a variety of ML settings. That being said, the practical relevance of this improvement depends on the financial and time costs to produce human and synthetic validation data. Our method is particularly relevant in instances where human validation cannot be obtained in a timely manner. Examples include the Chatbot arena, where human annotation depends on web traffic, which cannot be controlled, or in biological assays where obtaining validation data may take weeks or months, but where producing large amounts of synthetic data can be done in a few hours.\n\n\n**Benchmark**\n\n> But this method could also provide confidence intervals based on asymptotic normality, and this paper did not include that in most of experiments [...].\n\nWe would like to make an important clarification, as building valid confidence intervals of model performance is central to our work. In particular, Figure S1 of Appendix C shows confidence intervals not for one, but all of the main experiments (in Sections 2.3, 2.4, and 3.3) considered in this work. \n\nThese confidence intervals, which we demonstrate are well-calibrated and tighter than classical approaches in all experiments, are fundamental to our methodology as they enable statistically rigorous model comparison and selection even with limited labeled data.\n\n\n> I would like to see other metrics discussed.\n\nWe thank you for suggesting the exploration of additional metrics. In response, we have extended our analysis in the following way. We studied confidence interval coverage for multiple alpha values (0.8, 0.85, 0.9, and 0.95) beyond the original α=0.1, confirming consistent calibration across confidence levels. \nRegarding cross-entropy metrics, we would appreciate your clarification on how you envision these metrics being applied in our evaluation framework.", "claims_and_evidence": "Yes, the claims are clear and evidence seems good.", "methods_and_evaluation_criteria": "Yes, it makes sense. Mostly about directly measuring MSE and effective sample size. But this method could also provide confidence intervals based on asymptotic normality, and this paper did not include that in most of experiments, only in one in Appendix C.", "theoretical_claims": "The theoretical claims are correct but largely could be directly derived from PPI++, did not have much innovation. I wouldn't claim any innovation in theory for this paper.", "experimental_designs_or_analyses": "I think the experimental designs and fine but still limited. For instance, coverage did not include for most of the experiments, and whether other metrics beyond MSE could be considered is not discussed. For instance, some cross entropy type of metrics are very widely used, but not discussed as metrics here. I understand this work largely studies unbiased estimator, so if the final aim is MSE, then the only thing we need to do is to reduce variance. But I would like to see other metrics discussed.", "relation_to_broader_scientific_literature": "I think it is useful in practice, but also am concerned that the results by simply using gold-standard data are already good enough, the improvement seems marginal.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "I think the problem itself is important but theoretical innovation is quite limited. Also, the experimental results are not comprehensive enough (see above comments).", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "NA.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1w60fU3a1R", "reviewer": "Reviewer_uXP6", "summary": "This paper proposes an approach called \"autoevaluation\". Given a small set of human-labelled examples, and a larger set of (iid) unlabelled examples, the proposed algorithm can synthetically assign labels in a comparatively efficient and unbiased manner. The authors validate their approach using experiments on real-world tasks such as ImageNet, Protein fitness prediction, and chatbot arena.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind and insightful comments. We are grateful for your appreciation of our work, particularly our methodologically sound approach and practical demonstration of efficient model evaluation across multiple domains.\n\nRegarding your comment on the relevance of AutoEval to address benchmark saturation, we would like to highlight a central application that motivates our work. One major inspiration for this work is the chatbot arena project (Chiang et al. 2024). This project serves as a dynamic alternative to traditional static benchmarks, allowing human participants to compare and vote on large language model (LLM) responses.. \n\nDespite the availability of large amounts of human validation data, resolving ties between models remains challenging in this project. \nFirst, many users prompt models with questions but decide not to provide preferences between models. Consequently, a large number of conversations without human preferences is available, which cannot be leveraged by traditional model evaluation approaches.\n\nThere are also many cases where there is not enough validation data to resolve ties between LLMs. This can happen when a new LLM releases, requiring large amounts of votes (which may correspond to several days of traffic) to quantify its relative performance.\nUnresolved ties also occur when evaluating model performance on specific applications where relevant conversations are limited, such as coding assistance or creative writing, for instance. Helping resolve ties timely in dynamic benchmarks such as Chatbot Arena with AutoEval is one way we believe our work can help address benchmark saturation.\n\nWe agree with your observation regarding the i.i.d. assumptions in our work. In the chatbot arena, for instance, these assumptions are challenged when users ask multiple questions or repeat the same question, leading to data dependencies. To address this, future work should extend AutoEval to more general settings with data dependencies to produce reliable inferences.\n\nFinally, AutoEval does not solve the problem of designing hard inputs. It builds on the assumption that inputs are easy to sample but harder to label. While this assumption is limiting, the applications of our work—including in NLP—show that this paradigm can be useful. Beyond NLP, labeling at scale remains a significant challenge in many domains, such as biomedical applications, where we believe AutoEval can make an impact in facilitating model evaluation.", "claims_and_evidence": "Yes, the major claims made in this paper seem well-supported.", "methods_and_evaluation_criteria": "Yes the proposed PPI-based approach appears to be sound. The empirical evaluation is on standard datasets.", "theoretical_claims": "Yes the calculation of confidence intervals in section 2.2 looks correct to me. However, I am not an expert in this area so it is possible I may have missed something.", "experimental_designs_or_analyses": "The experiments presented are a straightforward application of the proposed approach and look sound to me.", "relation_to_broader_scientific_literature": "There is growing interest in synthetic data generation and annotation in the field of evaluation considering the fact that human annotation is expensive, and sometimes impractical.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "One issue I have with this line of work is that it makes a set of assumptions (1) existence of unlabelled inputs, and (2) the unlabelled inputs are iid. Considering the state of evaluation in ML wrt to large models, I think it has become increasingly important to curate more challenging evaluation benchmarks. In this respect, generating (or labelling) more examples which are iid to benchmarks that we already have (and have presumably been saturated by large models) doesn't really feel like a particularly impactful step forward towards solving our evaluation troubles. Even if we design a new (not-yet-saturated) benchmark with a few human-labelled examples, how valuable is it actually going to be to scale it up with AI-based labelling? Further note that one of the actual bottlenecks in creating effective benchmarks is designing the \"hard inputs\" (which this paper assumes we already have access to).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJdBoWnaOK", "reviewer": "Reviewer_f29Q", "summary": "- Paper addresses the problem of evaluating ML models with limited human validation data. \n- Proposes Autoeval, an approach pairing limited human data with large amounts of AI synthetically labeled data to get model eval scores\n- The primary contribution is the framework based on the existing PPI work\n- Evaluations are done to estimate performance metrics directly (e.g. accuracy) and also relative model performance in the case of BT-models\n- Results are positive compared to alternative approaches", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful comments and positive feedback on the relevance of our work, the strength of our benchmark, and of our theoretical analyses. Below are detailed answers to your comments.\n\n**Understanding when AutoEval > classic.**\n\nOur approach adjusts to the quality of synthetic labels. In particular, AutoEval can rely on power tuning to identify the optimal λ used in Equation 4 or 6. This mechanism ensures that our estimates (i) have small asymptotic variance (ii) ignore synthetic labels when these are unreliable. Thus, our method is at least as good as the classical approach and typically better with informative synthetic labels.\n\nWe also cover several scenarios in our work where multiple synthetic annotation schemes could be competing. Figures 3 and 5 show that the correlation between synthetic and true labels is a good indicator of AutoEval performance and recommend using this metric to prioritize annotation schemes.\n\nIn the context of LLMs, we indeed note overall small correlations between human and LLM judge preferences, likely due to noise in human preferences and the ambiguity of many real-world prompts (multiple valid responses may exist). For instance, agreement rates of 85-90% among experts and 75-80% between experts and crowds have been reported in the past (see Chiang et al. 2024).\nThese small correlations also highlight the limitations of LLM judges, biased by answer length, position, and stylistic similarities (Zheng et al. 2023).  Mitigating bias via careful prompting, randomization of answer order (Li et al. 2024), and stronger judges (e.g., via fine-tuning on human preferences) could improve effective sample sizes of AutoEval.\n\nYou also highlight the relevance of understanding which LLM judges produce high-quality synthetic labels and on what inputs. Future work could apply AutoEval to specialized areas for LLMs where validation data is too scarce to provide insights into underrepresented tasks, providing better insight into what constitutes good LLM judges across domains.\n\n**Covariate shifts**\n\nCovariate shifts are central to key applications of AutoEval. A common scenario arises when unlabeled inputs are drawn from a distribution different from the target distribution. Labeled inputs could themselves not be sampled from the distribution of interest. These effects are relevant to address fairness concerns. Applied to LLM evaluation, for instance, covariate shift toward specific prompts can lead to biased evaluations that are not representative of target use cases.\n\n\nWe will add a supplementary note on AutoEval estimators under covariate shifts. In the canonical case, when unlabeled and labeled inputs respectively drawn from distributions P and Q, we can apply AutoEval using traditional covariate shift adjustment methods. Key assumptions here are that P is absolutely continuous with respect to (w.r.t.) Q and that the Radon-Nikodym derivative of P w.r.t. Q is known.\n\n**Novelty**\n\nOur work is more than a repackaging of PPI for model evaluation. We outline specific theoretical and methodological novelties in our response to Reviewer 3tuY and refer you to that discussion for details.\n\n**Runtime**\n\nWe compared runtimes between AutoEval and classical in the ImageNet experiment:\n\nTable: Runtime comparison between AutoEval and the classical approach for the evaluation of Resnet101 in the ImageNet experiment ($n=1,000, N=50,000$). This experiment was run on a workstation with an Nvidia RTX 3090 GPU, 128GB RAM, and an i9-12900KF CPU.\n| Method  | prediction (s) | inference (ms) |\n| :------ | :------------- | :------------- |\n| classic | 5 | 0.3 |\n| PPI++   | 237 | 8.3 |\n\nThe main bottleneck here is synthetic label generating, typically scaling linearly with the number of samples. This cost is generally acceptable given the gains in evaluation efficiency and statistical power. We will include this analysis in the revised Supplement.\n\n\n**Other**\n\n> Coverage guarantees are not always obtained [...]\n\nUndercoverage in Figure S1b is due to insufficient Monte Carlo (MC) trials. Rerunning the experiment with $K=500$ MC trials yields:\n\nTable: Coverage for $\\alpha=0.1$ in the protein fitness experiment\n| Method  | n=50 | n=100 | n=200 | n=300 | n=400 | n=500 |\n| :------ | :--- | :---- | :---- | :---- | :---- | :---- |\n| PPI     | 0.87 | 0.89  | 0.89  | 0.89  | 0.89  | 0.88  |\n| PPI++   | 0.88 | 0.89  | 0.89  | 0.89  | 0.89  | 0.9   |\n| classic | 0.89 | 0.89  | 0.89  | 0.89  | 0.89  | 0.9   |\n\nThe coverage is now much closer to nominal levels and validates our coverage claims.\n\n> It would be useful to augment the related work [...] on LLM judges [...] synthetic data for model evaluations.\n\nWe thank you for the suggestion. Since LLM judges are core to our work, we will update the related work in the revision with the suggested references.\n\n> I recommend moving figure S1 to the main paper [...]\n\nWe thank you for this comment and will move S1 to the main paper in the revised paper.", "claims_and_evidence": "Well supported claims\n - Autoeval can improve sample efficiency without introducing bias.\n- PPI provides more accurate and reliable estimates than classical methods.\n- Autoeval extends to ranking approaches as well.\n\nUnsupported\n - Coverage guarantees are not always obtained as shown in the results and depend on the number of labeled samples (even in cases with large CI width)\n- This shows perhaps the claim on IID is not met between labeled and unlabelled", "methods_and_evaluation_criteria": "- The selection of benchmark datasets is strong and covers a wide variety of areas\n - Method is well motivated as well as the evaluation criteria\n\nJust flagging an assumption of the annotator model quality: The method assumes that the AI-generated synthetic labels are at least weakly correlated with the true human labels. \n\nThat said the correlations are pretty low. Is this a function of LLMs in general and why specifically are other LLMs better than others? i.e. where are they good and where are they bad", "theoretical_claims": "- Theoretical claims are sounds and mainly defer back to the original PPI work & its guarantees\n- However, Sec 2.2 does do some solid theoretical analysis", "experimental_designs_or_analyses": "- The experimental designs are thorough and well-executed:\n\n- The authors use a diverse set of domains (computer vision, protein fitness, LLM evaluation) to demonstrate broad applicability + good use of baselines\n\n- It would be interesting though to understand where the method works well and where it fails. e.g. where is the approach poor, where is the correlation low (is it specific types of examples). This would greatly help with understanding\n\n- Also it would be nice to have some runtime and cost information vs classical methods — to understand the trade-off of the gains", "relation_to_broader_scientific_literature": "In general pretty good\n\n- It would be useful to augment the related work to position the work with respect to recent works on:\n- LLM judges:  For example: https://arxiv.org/abs/2403.02839, https://arxiv.org/abs/2412.05579\n- Synthetic data for model evaluations. For example: https://arxiv.org/abs/2310.16524", "essential_references_not_discussed": "see above for references", "other_strengths_and_weaknesses": "Strengths: \n- clear and well-written paper\n- strong results empirically and good theoretical analysis\n\nWeaknesses:\n- Issues around novelty — seems like a repackaging of PPI for a different use case (i.e. application). \n- Computational cost uncertain\n- Needs to augment related work to position better as discussed above\n- Add discussion of how this work might be extended under distribution shift in greater detail than the current passing references. Since likely in reality this would be use-case.", "other_comments_or_suggestions": "- possible add a section on the distribution shift problem and how this could be handled\n- add a section on how this would work for fairness, as it’s mentioned but never shown\n- Deconstruct the method more to understand where it succeeds and where it fails — especially why it fails (types of situations)", "questions_for_authors": "See above points", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an evaluation method that incorporates synthetic labels along with human-labeled test data to obtain better estimate of a given model’s accuracy. The proposed method uses PPI++ to use synthetic labeled data. The proposed method is simple and applicable to various types of supervised learning settings. The efficacy of the proposed method is demonstrated by evaluation on various scenarios, including classification tasks (image recognition), regression (protein fitness prediction) and rankings (pairwise comparisons for chatbot arena). The theoretical innovations are limited. Another limitation of the proposed method as noted by reviewers is the assumption of i.i.d. data. The proposed re-weighting of the evaluation metric requires knowledge of the distribution. That said, sample-efficient evaluation of models is a very challenging task and the proposal of  applying PPI++ for model evaluation incorporating synthetic labels is a promising direction. Overall, the strengths of the submission outweigh the limitations.  

Please include the additional experimental results and discussions on covariate shifts and limitations of the current approach that came up during review-rebuttal period.
Decision: Accept (poster)