Evaluating multiple models using labeled and unlabeled data
Authors: Divya M Shanmugam
Shuvom Sadhuka
Manish Raghavan
John Guttag
Bonnie Berger
Emma Pierson
Venue: NeurIPS 2025 poster ()
Keywords: evaluation
mixture models
semi-supervised learning
Primary area: evaluation
TL;DR: We propose a new evaluation method that makes use of three sources of information (unlabeled data, multiple classifiers, and probabilistic classifier scores) to produce more accurate performance estimates than prior work.

Abstract: It is difficult to evaluate machine learning classifiers without large labeled datasets, which are often unavailable. In contrast, unlabeled data is plentiful, but not easily used for evaluation.  Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. The key idea is to estimate the joint distribution of ground truth labels and classifier scores using a semi-supervised mixture model. The semi-supervised mixture model allows SSME to learn from three sources of information: unlabeled data, multiple classifiers, and probabilistic classifier scores. Once fit, the mixture model enables estimation of any metric that is a function of classifier scores and ground truth labels (e.g., accuracy or AUC). We derive theoretical bounds on the error of these estimates, showing that estimation error decreases with the number of classifiers and the amount of unlabeled data. We present experiments in four domains where obtaining large labeled datasets is often impractical: healthcare, content moderation, molecular property prediction, and text classification. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1x relative to using labeled data alone and 2.4x relative to the next best method.

Evaluating multiple models using labeled and unlabeled data
Divya Shanmugam"
Shuvom Sadhuka"
Manish Raghavan 23
John Guttag²
Bonnie Berger 2**
Emma Pierson
'Comell University
2 Massachusetts Institute Technology
University of California, Berkeley
'quall contribution, ""Equal supervision
divyas@cornell.com edu ssadhuka@mit. edu
Abstract
It is difficult to evaluate machine learning classifiers without large labeled datasets, which are often unavailable. In contrast, unlabeled data plentiful, but not easily used for evaluation. Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. The key idea is estimate the joint distribution of ground truth labels and classifier scores using semi remi-supervised mixture model. The semi- supervised mixture model allows SSME to learn from three sources of information: unlabeled data, multiple classifiers, and probabilistic classifier scores. Once fit, the mixture model enables estimation of any metric that is function of classifier scores and ground truth labels (c.g., accuracy or AUC). We derive theoretical bounds the error of these estimates, showing that estimation error decreases with the number of classifiers and the amount of unlabeled data. We present experiments in four domains where obtaining large labeled datasets often impractical: healthcare, content moderation, molecular property prediction, and text classification. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1 x relative to using labeled data alone and 2.4x relative to the next best method.
Introduction
Large, labeled datasets are critical for evaluation in machine learning, but such datasets are often prohibitively expensive or impossible obtain [17, 24]. Exacerbating the challenge of evaluation, the number of off-the-shelf classifiers has increased dramatically through the widespread usage of model hubs. The modern machine learning practitioner thus has a myriad trained models, but little labeled data with which to evaluate them.
In many domains, unlabeled data is much advantage of this, we introduce Semi- Semi-Supervised Model Evaluation (SSME), a method that can be used evaluate multiple classifiers using both labeled and unlabeled data. The key idea is to estimate the joint distribution of ground truth classes y and continuous classifier scores s using mixture model, where different components of the mixture model correspond to different classes. The joint distribution allows us to evaluate performance on examples where we have access only to each classifier's scores, i.e. unlabeled examples. SSME can estimate any metric that is a function of class labels and probabilistic predictions, including widely-used metrics like accuracy, expected calibration error, AUC, and AUPRC. Concretely, we can evaluate classifier on an unlabeled point by using our estimate of s)y,s)
abundant than labeled data [8, 61, 49). To take
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Access to two classifier's scores on examples, some labeled.
Setting:
Stop 1: Fit mixture model to estimate joint distribution of classifier scores.
Stop 2: Compute metrics with respect to expected label for each unlabeled example.
0 LLbeled negatives
Labeled positives
??
P(s., saly 0)
sal
?
Classifer: 2 Scores 9z
?
?
Clessifier 2 Scores
?
?1
? ?
Unlabeled
Labeled
1?
?
Sample y ply =1 st, sal
Retrieve y
? ? ?0?
? 0 ?
Compute ACC y) ACCbs y) over all [s,, sol
P(sr, s
ECE/,,
ECEbo, vi
20
Classifier Scores St
Classifier Scores s:
Figure I: Using SSME with two binary classifiers. SSME applies to settings in which classifier scores available for set of examples, some labeled (left). SSME first fits a mixture model to estimate the joint density of scores and labels (where 81 is the score assigned by the first classifier, and 82 the second, middle panel). SSME next uses the mixture model compute metrics with respect to the expected label for each example (right). SSME can be applied to any number of classifiers and classes K, and supports any metric that compares classifier scores ground truth labels.
SSME is the first evaluation method to capture three key facets of modern machine learning settings: (i) multiple machine learning classifiers, (ii) probabilistic predictions over all classes, and (iii) unlabeled data. Simultancously using all three is difficult because it requires accurately estimating the (potentially high-dimensional) joint distribution P(y.s) s) with primarily unlabeled data. While prior work captures subsets of these three sources of information [69, 57, 37, 15, 11] for example, augmenting labeled data with unlabeled data to evaluate a single classifier - no existing approach accommodates all three.
We show both theoretically and empirically that by using all available data multiple classifiers, continuous scores over all classes, and unlabeled data SSME is able to produce more accurate performance estimates compared to prior work. We make three contributions:
1. We propose SSME, method that incorporates unlabeled data, continuous classifier scores, and multiple classifiers to produce more accurate performance estimates than prior work. SSME is able to estimate any metric that compares predicted probabilities to ground truth labels and accommodates varying numbers of classifiers and classes.
2. We prove that using unlabeled data and multiple classifiers improves performance estimation, providing insight into SSME's strong empirical performance across domains.
3. We show that SSME produces more accurate performance estimates compared to labeled data alone and seven baselines, reducing metric estimation error by 5.1 x relative to labeled data and 2.4x relative to the next best method. We further validate SSME's s utility in two case studies: evaluating large language models and subgroup-specific performance.
2 Problem Setting
We consider a setting which a practitioner wishes evaluate several classifiers using both labeled and unlabeled data. Formally, there are M classifiers [Si, J2....,J'ar designed for the same task, which might differ in their training data, architectures, or optimization, among other possibilities. Each classifier maps from the same input domain X' to probability distribution over K classes. Let s' = [fi (x²), fa(x')...... (aa')....ffM (x')) denote the concatenated set of classifier scores for data point æ²; Figure depicts the setting.
We assume access the set of classifiers and two datasets: (1) a small labeled dataset, Dz "') and (2) larger unlabeled dataset Du [m=)) We assume that unlabeled data far more available than labeled data, i.e. n, >> ne, and that the unlabeled samples are drawn from the same distribution as the labeled samples. Our goal is to use these two datasets to estimate classifier performance using standard metrics such as expected calibration emror (ECE) or accuracy. we knew the true label y² for each point æf, it would be straightforward to evaluate the performance of each pre-trained classifier. However, the true label not available for unlabeled examples, so we instead aim to infer a distribution over ground truth labels.

---

3 Method
Semi-supervised model evaluation (SSME) contains two key steps. First, SSME estimates the joint distribution of ground truth labels y and classifier scores - ie. Ply, s) using both labeled and unlabeled data. Second, SSME uses the estimate of P(y. , s) to compute metries with respect the posterior over y implied by classifier scores allowing classifier evaluation using both labeled data (where ground truth is available) and unlabeled data (where ground truth is inferred).
Step 1: Estimating P(y,s) SSME estimates s),,s) using a semi-supervised mixture model, where components of the mixture model correspond to different ground truth classes. Following standard procedures in mixture model estimation, we optimize parameters of the mixture model 0 by maximizing the log-likelihood of the observed data. When is unobserved, we treat it latent variable and marginalize it out. Specifically, we optimize the following expression:
Tig P(s:, 8s;
argmax Y; argmax
II P(s,
(I)
K (Po k)Pa k)]
argmnax
log (Po(s
+ho
(2)
Labeled Data Log-Likelihood
Unlabeled Data Lik-Likelibood
where Àvr controls the relative weight of the unlabeled data in the likelihood; we fix Ào = 1.
We optimize the above expression using expectation expectation-maximizzie maximization (EM) [19, 76], alternating between the E-step, which estimates which mixture component (i.c. ground truth class) each data point belongs to, and the M-step, which estimates the parameters of each component based on the soft component assignments. Algorithm details the EM procedure we use to fit our mixture model.
A technical challenge when fitting densities bounded domains, as is the case with classifier outputs, boundary bias, where estimates of density are biased near edges the domain [39]. To overcome this challenge, we transform probabilistic predictions over K classes points ("scores") in IR K- using the additive log-ratio transform, which produces a one-to-one mapping between the two spaces reparumeterization trick: they transform the classifier
[1, 53). Additive log-ratio transforms scores from a bard-to-mode space the probability simplex - to an ansier-to-mode space unbounded reals. Appendix E. provides the expression for the additive log-ratio transform.
Step 2: Estimating classifier performance given Sly,s). Once we have estimated the parameters 0 for the mixture model, procedure described Figure 1. By estimating the conditional distribution Po(y|s), SSME can measure performance with respect the expected label of unlabeled examples (where we observe only 8²). Specifically, we sample ground truth labels from the distribution (a(y))) for all unlabeled examples. Labeled examples retain their ground truth labels. We then compute metries with respect to both the sampled and ground truth labels, and average the estimated metric across samples. SSME can estimate any metric that is function of classifier scores and ground truth labels; see Algorithm for corresponding pseudocode.
the fitted density estimate metries of interest, using the
The proposed approach enjoys several properties. By using a semi- sumi-super mixture model, SSME takes advantage of unlabeled data to estimate Ply. s). SSME also naturally accommodates and benefits from additional classifiers, because they provide additional information about the ground truth label of an unlabeled example (additional classifiers simply translate to additional dimensions in Furthermore, the of the additive log-ratio transform allows SSME learn directly from continuous class probabilities for each input, as opposed to discretized labels.
3.1 Implementation
SSME can accommodate multiple ways to parametrize the class-conditional distribution of scores P(s yy as long as they can be learned the semi-supervised setting described above. We denote the parameterized distribution as Po(s y). In our experiments, we use kernel density estimator (KDE) to parameterize Po(s) y) and explore alternative parametrizations of (1)())) in Appendix E.3.

---

Algorithm 1: Estimating P(y. s) using EM
Input:
Labeled set D; = {(s" )²) Unlabeled set D., = {s |as; are ALR-transformer classifier
scores; Labeled-data weight Äv; number of classes K; total epochs 7 1000
Init: Initialize model parameters 8%
Initialize class priors p0(p = k) = pVy = k);
// empirical clase distribution
for - to 7 1 do
E-step: compute responsibilities 7a3
foreach 5 D., do
for to K do
p'v = (s
y k)
Tik f-
Σ I'V Por (5
foreach (s', (5'))== E Dj do L Tik 1,k = =
// fix to true label
M-step: update priors and 0: for 1 to K do
NE Eo. Tak Au Ev, Tak
// weighted effective for class k
+1 (y = k) c N (Mh Uhe + A(rie)::
11
Dw the
g++1 arg max
Yikk log Po y k). k=1
13
where w, Au for unlabeled points and ui otherwise;
Output: Final parameters 8 and priors p" (v)
Kernel density estimators do not make parametric assumptions about the distributional form of each component: this is useful for modeling distributions of predictions, which can vary widely across tasks and classifiers. The KDE for the kth class is parametrized by bandwidth h and the kernel type K. We weight each point by the probability it belongs to the component (i.e., a soft label). When s' is labeled, P(y" k) for the true label k, and when s' is unlabeled, we compute P(y = k|s) using the estimated density. For all experiments, we fit use Gaussian kernel and estimate using the improved Sheather-Jones algorithm [10]. We optimize the parameters using EM over 1 1000 epochs. We initialize component assignments by drawing a label for a given example according the mean classifier score across the set of classifiers.
3.2 Theoretical analysis
We motivate approach by deriving bounds for how well semi-supervised estimation procedure can estimate classifier performance in a setting where classifier scores s are drawn from a mixture of Gaussians.
Model of and s. We consider stylized binary classification setting drawn from previous work [67] where classifier scores are drawn from a balanced mixture of two Gaussians. For data in the positive class, classifier scores drawn from a multivariate Gaussian (Cc, I). For data in the negative class, classifier scores are drawn from multivariate Gaussian (00,, I). Let refer to the number of unlabeled examples, ny the number labeled examples, and d the number of classifiers. Additional classifiers increase the dimensionality of the multivariate Gaussian.
Performance estimation procedure. Having specified how classifier scores are generated, we analyze how accurately a semi-supervised performance estimation procedure similar to our own can estimate classifier performance. We consider class of semi-supervised learning algorithms frequently analyzed in prior work, UL+ [67]. UL+ algorithms first use the unlabeled data to identify decision boundaries and then use the labeled data to assign regions to classes. AUC₁ and ACC's
represent ground truth performance of classifier k; AUC, and ÁCC₁ are performance estimators that use UL+ to estimate a Gaussian mixture model, and use that model to estimate performance. We apply bounds developed by [67] for errors in mixture model estimation bound errors performance estimation.

---

Theorem 1
Let AUCk. ACC₁ be estimated using semi-supervised learning algorithm in the UL+
class of estimators. When nu Z max {d, d Te]] mine|]cc]"]lel]'>>>
log(n.)
dlogídn.)
and d 2 2, the following
||e||
bounds on error in estimated performance hold with probability 1 p.
JAUC - AUCCAI <
Ck
11
Ck - cc
(3)
V2
V2
IACC ACCAI S
Ck
$
(4)
2
2
l|cle mm
loll (1-
Co Tell
dlngany Tell
where Eo S
d n
and Co is a universal constant.
The expression for ce captures its asymptotic dependence on fh ng, d, and c. Smaller ce correspond to smaller errors in performance estimation. Section C contains the full proof, which consists of two steps: we first bound the error estimating the separation of the two Gaussian components ||c|], and then use the relationship between component separation and classifier performance to bound error in estimates of AUC and accuracy.
Analysis. Our theoretical bounds have several implications. First, unlabeled data helps: increases in n, reduce both the first term and the second term. This indicates why SSME outperforms baselines that only make of labeled data: additional unlabeled data improves our ability to accurately estimate components of a mixture model (ce ~ O(1/ /T)) Second, bounds tighten as separation between components ||c|| grows, i.e. when classifiers in the set more accurate. Third, bounds tighten as the number of classifiers increases, so long as the increase in separation ||c|| outweighs the cost of additional dimension in d; we precisely characterize this setting in Section C.I. This helps explain why SSME outperforms work that makes use of a single classifier, instead of several.
Our thcoretical results are derived for stylized setting frequently studied in prior work [59, 60, 67, 26). While this setting differs from our empirical setting for example, it uses simplified semi-supervised learning algorithm we verify that trends our theory implies hold robustly in both synthetic and real experiments (Sec. 5, D.1), including settings where these assumptions do not hold.
Experiments
4.1 Datasets and Classifier Sets
We select datasets and classifier sets to be realistic and diverse, capturing multiple modalities (EHRs. text, and graphs), domains (healthcare, content moderation, chemistry), and architectures (logistic regressions, graph neural networks, large language models). We report ground truth metrics for the binary and multiclass classifiers in Tables S1 and S2 respectively. We also include a detailed description of each dataset and classifier set in Appendix B.I. We summarize each dataset and differences between classifiers in the associated classifier set below.
We evaluate SSME on five classification datasets: (1) MIMIC-IV [38], a dataset of Boston-area electronic health records (EHRs) with three hospitalization outcomes (critical outcome, ED revisit within 30 days, and hospital admission) with classifiers drawn from prior works [50]; (2) Civil- Comments [9]. dataset of social media comments flagged as "toxic" or not by human annotators. with pretrained classifiers downloaded from the WILDS benchmark [42]: (3) OGB-SARS-CoY [33], a molecular property prediction task, with pretrained classifiers downloaded from WILDS; (4) MultiNLI, natural language task, with classifiers drawn from the SubpopBench dataset [73]; and (5) AG News [75], news classification task, using open-source and closed-source L.MSs as classifiers.
4.2 Baselines
We compare SSME to eight baselines that use both labeled and unlabeled data to estimate performance (with the exception of Labeled, the standard approach to classifier evaluation, which only uses examples for which labels are available). The remaining baselines cover broad range of approaches one could adapt to supervied evaluation and fall into three categories. For a full description of each bascline's 's implementation, please refer to Appendix B.2.
5

---

The first category fits a model of P(y.s) using labeled data alone, then applies it to the entire dataset. Each baseline does so in a different way: Pseudo-Labeled (PL) fits a logistic regression to predict the true label from classifier scores; Bayesian-Calibratin (BC) [37] fits a Bayesian model to re-calibrate each classifier's 8 predicted probabilities; AutoEval [11] learns a function to debias classifier predictions. The second category estimates the performance of individual classifiers. SPE [69] fits a parametric mixture model to a single classifier's predictions. Active-Testing [44] uses active learning to select examples to label out of pool of unlabeled examples based on single classifier's 's predictions and evaluates the classifier on those examples. The last category baselines is drawn from the multiple annotator literature, where annotations are assumed to be discrete. We compare to Dawid-Skene [18], which accepts discrete annotations from each classifier to estimate the latent true label for each example. We also compare to Majority-Vote which ensembles classifier predictions by performing an accuracy-weighted aggregation of classifier predictions.
We also provide comparison to completely unsupervised approaches (including ensembling where the "ground truth" label is drawn based on the average classifier score) in Sec. D.10, and discuss connections to weak supervision in Sec. D.9.
4.3 Evaluation
We evaluate SSME's ability to estimate four performance metrics for binary classifiers: accuracy, area under the receiver operating characteristic curve (AUC), area under the precision-recall curve (AUPRC), and the expected calibration error (ECE). For multi-class problems, we evaluate accuracy and top-label calibration error [32].
We partition each dataset into three splits: the classifier training split (used to train the classifiers whose performance SSME estimates), the estimation split (used to fit SSME and estimate classifier performance), and the evaluation split (used to produce a held-out, ground-truth measure of classifier performance). All splits sampled from the same distribution, except when estimating subgroup- specific performance, where the evaluation split contains a subset of the test distribution.
To evaluate metric estimates, we measure the absolute error of the estimated metric, computed using the estimation split, compared to the true metric, computed on the held-out evaluation split (averaging over classifiers in the set). The estimation split consists of either 20, 50, or 100 labeled examples and 1000 unlabeled examples across all experiments. The size of the evaluation split is on the order of thousands of labeled examples and varies by task (see Appendix B. for exact split sizes). To quantify uncertainty, we compute 95 % confidence intervals using the standard error of the mean across 50 random data splits (#1 .96 x SEM). In line with prior work, we report rescaled estimation error for each metric (where errors are relative to using labeled data alone), allowing us to standardize the scale of errors across datasets and metrics [28].
5 Results
5.1 SSME produces more accurate performance estimates than prior work
We now compare SSME to eight baselines in terms of its ability to estimate classifier performance on five binary tasks. All figures report rescaled metric estimation error (RMAE; lower is better) and reflect performance estimation using 20 labeled examples and 1000 unlabeled examples. Results are consistent across additional values of ne (50 and 100; see results in D.2), although labeled data grows more competitive (as expected) with larger labeled dataset sizes.
Comparison to baselines SSME achieves lower mean estimation error (averaging across tasks and metrics) than all baselines, indicating more accurate estimation of classifier performance. SSME reduces estimation error by 5.1 x relative to labeled data alone (averaged across tasks and metrics). In contrast, the next best method reduces estimation error by 2.4x. SSME also outperforms baselines on specific metrics. For accuracy, SSME reduces metric estimation error, relative to using labeled data alone, by 5.6x (averaged across tasks); the next best method for each dataset reduces metric estimation error by 2.0x. While the magnitude by which SSME beats baselines varies- -for example, SSME reduces error by 2.9x on AUC (averaged across tasks), while the next best method reduces error by 2.6x - SSME consistently outperforms baselines across metrics. SSME performs on par with or better than the strongest baseline in 51 of 60 dataset, metric, and labeled data combinations

---

Revist
X X
X00X
XX
X
XXX X
Crttial
2600
XX
X
Majority-oon
Slewi-Skene
Hospital
SPE
makily
X0X
X
AutoEval
Active- Tosting
SARS-CoV
SSME (Ours)
0.00
0.75 0. (ACC)
1.0 RMAE (AUPRC)
RMAE (ECE)
RMAF (AUC)
Figure 2: Metric estimation error on binary tasks (nz 20, n. 1000). Each point plots the rescaled mean absolute error (RMAE) across runs, where 1.0 (dashed line) represents the RMAE when using labeled data alone and lower is better. SSME (gray) achieves lower estimation error than the baselines, averaging across metrics and 50 runs, reported for five binary tasks (y-axis). Results reporting mean absolute error (unscaled) and 95% CIs are in Tables S3-S6.
we study, and significantly better in 31 settings (Tables S3-S6). Figure 2 summarizes how SSME compares to several baselines across various metrics and tasks!
Our results are also encouraging in absolute terms. With 20 labeled examples and 1000 unlabeled examples, SSME estimates accuracy within 1.5 percentage points (averaging across tasks). The closest baseline estimates accuracy within 3.4 percentage points. Results comparing SSME to the next best baseline other metrics (1.9 vs 3.8 on ECE; 3.6 4.3 on AUC; 8.5 vs 10.2 AUPRC) confirm that SSME not only achieves more accurate classifier performance estimation compared to prior work, but that the resulting performance estimates are reasonably close to the ground truth.
Consistent with our theoretical results, SSME estimates performance more accurately because makes use of all available information: multiple classifiers, continuous scores over all classes, and unlabeled data. Dawid-Skent and AvtoEval discretize classifier scores (although AutoEval does make use of classifier confidence associated with each discrete prediction). Eseudo-Labeling Bayesian- Calibration, and AutoEval each learn a mapping from s to y using only the labeled data and apply that mapping to the unlabeled data (rather than learning from labeled and unlabeled data together). By jointly learning across both labeled and unlabeled data, SSME is able to generalize much better in cases where there aren't enough labels estimate the joint distribution of classifier scores and labels from labeled examples alone. Finally, Bayesian- -alibratiin and SPE learn from a single classifier's scores, and do not learn from multiple classifiers at once.
Comparison across metrics SSME provides the greatest benefits relative to labeled data alone when measuring expected calibration error (ECE), with reduction in estimation error of 7.2x (averaging across tasks). ECE is harder to estimate with few labeled examples because it requires binning and then averaging calibration error across bins, producing greater variability when the number of labeled points per bin is small. We observe the smallest benefits relative to labeled data alone when measuring AUPRC (a reduction in estimation error of 2.2x, relative labeled data).
Comparison across amounts of labeled data SSME's performance continues to improve with more labeled data, but the advantage it confers over labeled data decreases: for example, with 20, 50, and 100 datapoints, SSME outperforms labeled data alone by 5.6x, 3.0x, and 1.6x. Similar to labeled data alone, there are diminishing but positive returns adding labeled data SSME's performance estimation procedure.
Another way to quantify SSME's benefit is measure the amount of labeled data required match SSME's performance, or the effective sample size (ESS), as introduced by prior work [11] (see Appendix B.3 for implementation details). With access to 20 labeled examples and 1000 unlabeled examples, SSME achieves an average ESS of 539 labeled examples for estimating ECE (averaging over tasks). In contrast, the next best approach achieves ESS of 110 labeled examples.
Comparison to marginal fit To validate the benefit of fitting the mixture model to multiple classifiers simultaneously, compare an ablated version of SSME fit on single classifier at time, SSME-M. SSME-M estimates the classifier-speeiff marginal distribution of P (s). and uses
Fig. omits results for Dawid-Skene (nccuracy) and Majority-Vorr (sccuracy) on hospital admission, and SPE (ECB) because their higher RMAEs distort the plotting scale. Tables S3 S6 contain complete results across baselines, including uncertainty estimates.
7

---

Labeled
Wote
PL
Dowid Skone
MuliNLI
x
AutoEval
Active-Tasting
0.00 0.25 0.50
0.75
1.00
1.25 1.50
1.75
2.00
0.0
0.2
0.4
0.6
0.8
1.0
1.2
SSME (Ours)
RMAE (ACC)
RMAE (ECE)
Figure 3: Metric estimation error for LLM classifiers (ny = 20, nu = 1000). SSME consistently
reduces estimation eπror relative to labeled data alone and does so more reliably than any baseline capable of estimating performance in multiclass settings.
this estimate to evaluate the classifier in question. Doing so results in worse performance estimates across metrics, tasks, and amounts of labeled data (Tables S3- S6) relative our full model. This agrees with our theoretical findings: each classifier provides distinct information about the ground truth label for given example, which SSME able to use.
Supplementary results The supplement contains additional results. We report mean absolute error, including standard errors and additional ny, for accuracy, ECE, AUC, and AUPRC Tables S3- S6 (broken down by classifier in Tables S7- $10). We report the sensitivity of SSME to kernel choice in Table S13. Section D.11 provides a computational cost analysis of SSME relative to the baselines. Finally, provide synthetic experiments to substantiate our theoretical findings under violations of our assumptions in Section D.I.
5.2 Case study:
Evaluating classifiers derived from large language models
Large language models (LLMs) are increasingly used to create classifiers [40, 66, 68]. with demonstrated potential to serve as data annotators [78]. However, evaluation of LLM-based classifiers is limited by the availability of labeled data. We study SSME's ability to estimate performance in this setting.
Age
Sex
Race
PL
0.92
1.06
1.13
Dawid-Skene
0.75
0.67
0.67
BC SSME (Ours)
0.45
0.33
0.41
We evaluate classifiers derived from large language models on the MultiNLI and AG News datasets: the MultiNLI classifier set contains off-the-shelf language model based classifiers, while the AG News classifier set contains four classifiers trained atop open- and closed-source large language models. Fig. 3 reports our results. We find that SSME improves over labeled data by 2.3x (averaging over tasks and metrics). In contrast, the next best baseline (Majority- Vote) improves over labeled data by
0.42
0.19
0.39
Table I: Subgroup-specific per- formance estimation (n) = 20, nu 1000). SSME achieves the lowest rescaled metric estimation error (RMAE), averaging across metrics and subgroups.
1.3x. SSME achieves significantly lower metric estimation error in 8 of the 12 combinations of task, metric, and ne, and performs comparably to the next best baseline in 3 of the remaining combinations (Tables S11, S12).
5.3 Case study: Evaluating subgroup-specine performance
SSME can be applied to measure performance within demographic subgroups, a task central to assessments of algorithmic faimess [12]. We conduct our analysis in the context of critical outcome prediction on MIMIC-IV, task with documented prediction disparities [50], and a set of commonly studied demographic groups (age, sex, race/ethnicity). We produce subgroup-specing performance estimates by sampling ground truth labels y according to our estimated p(y(s) for only those s observed among a given demographic group. We perform this analysis with respect demographic groups based on age, sex, and race/ethnicity. When there is no labeled data for a given subgroup, Labeled estimates subgroup-specifie performance global performance.
We report each method's 's reduction in estimation error relative to labeled data (averaging over metrics and subgroups) for each demographic category in Table 1, comparing to all baselines that can estimate the four performance metrics we average over. SSME reduces metric estimation error by 5.3 on sex, 2.6x race, and 2.4x age relative to labeled data, and greater extent than all baselines. SSME outperforms all baselines on all individual metries except for AUC, for which BC reduces estimation error by 3.2x as compared to 2.2x for SSME. BC is well-suited to estimating AUC because it assumes monotonicity when mapping to y. i..., Ply (1) 2 Ply 1|s(0) when
8

---

s(4) 2 2ln), When the classifiers in question have high AUCs - as the critical outcome classifiers do this is a useful assumption.
Related Work
Our work unifies ideas from two literatures: methods to 1) evaluate single classifier and 2) evaluate the accuracy of multiple discrete annotators in the presence of labeled and unlabeled data.
Classifier evaluation is a long-standing problem in machine learning. Early work on the of unlabeled data for classifier evaluation made progress via assumptions about the available unlabeled data (for example, assumptions about class priors [6. 23, 47]; conditionally independent features [64]; well-calibrated classifiers [30]; models of covariate shift [13, 45)). SSME avoids such assumptions by using both labeled and unlabeled data to learn flexible semi-supervised mixture model of classifier predictions. Interest in unsupervised evaluation has surged in the context of large language models, where models often judged by other models [77, 34]. These approaches, while promising, tailored to text generation and not applicable to broader classification tasks. A parallel line of work aims to make use of both labeled and unlabeled data to estimate classifier performance. Existing methods operate on individual classifiers and assume specific parametric forms for the prediction distributions [69, 15, 36, 37, 25]. Some works also focus on model selection in semi-supervised settings, though they do not explicitly estimate performance [46]. SSME differs from these works by naturally capitalizing on multiple classifiers, continuous classifier scores, and unlabeled data. As our results show, doing so leads to improved estimates of performance.
Evaluation of multiple discrete annotators first introduced by [18], who proposed method to estimate ground truth in the presence of multiple potentially noisy discrete annotations. Many subsequent works inherit Dawid-Skend -weiid-Skee'sss strong assumption of caas-conniiional independence of annotator errors [52, 57], including popular approaches in weak supervision [58, 4, 27], where annotators are instead user-provided labeling functions. Such assumption is plausible in certain contexts, but does not naturally translate to sets of candidate classifiers, whose predictions are likely to be correlated. However, methods from this line of work are designed to estimate the accuracy of binary annotations; they do not exploit the continuous probabilities available in multi-classifier evaluation. While some work has made progress towards accommodating continuous predicted probabilities [51, 56]. their focus is optimal aggregation, in contrast to our own, which is evaluation. Recent works use a continuous notion of classifier confidence conjunction with discrete annotations from each annotator [29, 11], but do not use the distribution of classifier scores over all classes.
Discussion
In this paper, we presented Semi-Supervised Model Evaluation (SSME), a method that supplements limited labeled data with unlabeled data to more accurately estimate classifier performance. SSME leverages three features of the current machine learning landscape: (i) there are frequently multiple classifiers for the same task, (ii) continuous classifier scores are often available for all classes, and (iii) unlabeled data is often far more plentiful than labeled data. We provide theoretical bounds on performance estimation error that help explain SSME's s advantages over prior work. We show that across multiple tasks, architectures, and modalities, SSME substantially outperforms using labeled data alone and standard baselines.
Our results, and their accompanying limitations, suggest several directions for future work. First, each of the metries examined (e.g., AUC) evaluate single classifier. But because SSME estimates the full joint distribution P(y, s), it could be used to measure properties of the classifiers as a set. For instance, recent work has highlighted the importance of measuring algorithmic monoculture [41] where all classifiers produce errors on the same instances. Second, our experiments assess settings in which the unlabeled data is sampled from the same distribution as the labeled data. Although this is common for example, when random subset of examples is annotated - many real-world scenarios violate this assumption, and extending SSME to settings with such distribution shift represents an important next step [61]. Finally, our current implementation relies on kernel density estimation, which can scale poorly in high-dimensional spaces (e.g., with many classifiers or classes). Exploring more scalable density estimators could further broaden SSME's applicability. More generally, our results indicate that SSME offers a principled and practical framework for accurate performance estimation in the absence of large labeled dataset.

---

References
[1] John Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series (Methodological), 14(2):139-160) 1982.
[2] Michael S Albergo and Eric Vanden-Fijndee Building normalizing flows with stochastic interpolants. arXiv preprint 0rx3v:2209.13577, 2022.
[3] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana (116677):69-674 2023.
Zrnic. Prediction-poween inference. Science,
[4] Stephen H Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In International Conference on Machine Learning, pages 273-282. PMLR, 2017.
[5] Christina Back, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-the-lee Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 15:19774-19289, 2022.
[6] Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. Unsupervised supervised learning ii: Margin- based classification without labels. In Proceedings of the Fourteenth Inter- national Conference on Artificial Intelligence and Statistics, pages 137-145. JMLR Workshop and Conference Proceedings, 2011.
[7] Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: https///bb..org//024444//33
[8] Tristan Bepler, Andrew Morin, Micah Rapp. Julia Brasch, Lawrence Shapiro, Alex Noble, and Bonnie Berger. Positive-umlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nature methods, (111):111331160) 2019.
[9] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491-500, 2019.
[10] ZI Botev, JF Grotowski, and DP Kroese. Kernel density estimation via diffusion. Annals of Statistics, 38(5)2291-22957, 2010.
[11] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Au- toeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008, 2024.
[12] Irene Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. Ethical machine learning in healthcare. Annual review of biomedical data science, 4(14:123-14, 2021.
[13] Lingjiao Chen, Matei Zaharia, and James Y Zou. Estimating and explaining model performance when both covariates and labels shift. Advances in Neural Information Processing Systems, 1567--1111 2022.
[14] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
[15] Alexandra Chouldechova, Siqi Deng, Yongxin Wang, Wei Xia, and Pietro Perona. Unsupervised and Semi-supervised Bias Benchmarking in Face Recognition. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, Lecture Notes in Computer Science, pages 289-306, Cham, 2022. Springer Nature Switzerland.
[16] Rob Comnssh Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International conference on machine learning, pages 2133-2143, PMLR, 2020.
10

---

[17] Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAL, volume 5. pages 746-751, 2005.
[18] Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20-28) 1979.
[19] Arthur Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society: series B (methodological), 39(1):122 1977.
[20] Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterizing prediction matrix for unsupervised accuracy estimation. In International Conference on Machine Learning, pages 7658-7674, PMLR, 2023.
[21] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15069-15078, 2021.
[22] Laurent Dinh, Jascha Dick-Diccs,,in, and Samy Bengio. Density estimation using real NVP. arXiv preprint arXiv: 160.vv:605.0800 600.08805.0 2016.
[23] Pinar Donmez. Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised Supervised Learning I: Estimating Classification and Regression Errors without Labels. 2010.
[24] Suparna Dutta and Monidipa Das. Remote sensing scene classification under scarcity of labelled samples-a survey of the state-of-the-arts. Computers & Geosciences, 171: 171:105295, 2023.
[25] Vasilii Feofanov, Malik Tiomoko, and Aladin Virmaux. Random matrix analysis to balance between supervised and unsupervised learning under the low density separation assumption. In International Conference on Machine Learning, pages 10008-10033. PMLR, 2023.
[26] Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-training converts weak learners to strong learners in mixture models. In International Conference on Artificial Intelligence and Statistics, pages 8021. PMLR, 2022.
[27] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International conference on machine learning, pages 3280-3291. PMLR, 2020.
[28] Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint N.XYv:2201.042344 2022.
[29] Hui Wen Goh, Ulyana Tkachenko, and Jonas Mueller. Crowdlab: Supervised learning to infer consensus labels and quality scores for data with multiple annotators. arXiv preprint urXiv:2210.06812 2022.
[30] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Pre- dicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1134-1144, 2021.
[31] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMIR, 2017.
[32] Chirag Gupta and Aaditya Ramdas. Top-label calibration and multiclass-to-binaryy reductions. In International Conference Learning Representations, 2022.
[33] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datasets for machine learning on graphs Advances in neural information processing systems, 33:22118-22111 2020.
[34] Hui Huang. Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. An empirical study of Ilm-as-a-judge for ilm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:2403.0 02839, 2024.
11

---

[35] Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In International conference on machine learning, pages (615-4630. PMLR, 2020.
[36] Disi Ji, Robert L Logan, Padhraic Smyth, and Mark Steyvers. Active bayesian assessment of black-box classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7935-7944, 2021.
[37] Disi Ji, Padhraic Smyth, and Mark Steyvers. Can i trust my faireess metric? assessing fairness with unlabeled data and bayesian inference. Advances in Neural Information Processing Systems, 33: 8600-18612, 2020.
[38] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: hups://physionet. August 23, 2021), pages 49-55, 2020.
orggconten/mimiii///.fueceee
[39] M Chris Jones. Simple boundary correction for kernel density estimation. Statistics and computing, 3: 34,,,,46,, 1993.
[40] Neel Kant, Raul Puri, Nikolai Yakovenko, and Bryan Catanzaro. Practical text classification with large pre-trained language models. arXiv preprint arXiv:1812.01200 2018
[41] Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 11 118022)cc018340113 2021.
[42] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in wild distribution shifts. In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[43] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Thomas Rainforth. Active surrogate esti- mators: An active learning approach to label-efficient model evaluation. Advances in Neural Information Processing Systems, 35:24557-24570, 2022.
[44] Jannik Kossen, Sebastian Farquhar. Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763, PMLR, 2021.
[45] JoonHo Lee, Jac Oh Woo, Hankyu Moon, and Kwonho Lee. Unsupervised accuracy estimation of deep visual models using comainn-ddaptve adversarial perturbation without source samples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16443- 16452, 2023.
[46] Muyang Li, Xiaobo Xia, Runze Wu, Fengming Huang, Jun Yu. Bo Han, and Tongliang Liu. Towards realistic model selection for semi-supervised learning. In Forty-first International Conference on Machine Learning, 2024.
[47] Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang. Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia Sycara. Characterizing out-of-distribution enror via optimal transport. Advances in Neural Information Processing Systems, 36:17602-17622, 2023.
[48] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between -ustt--distiibuttion and in-distribution generalization. International conference machine learning, pages 7721-7735. PMLR, 2021.
[49] Rajiv Movva, Pang Wei Koh, and Emmm Pierson. Using unlabeled data enhance fairness of medical ai. Nature Medicine, 30(4):944 2024.
[50] Rajiv Movva, Divya Shanmugam, Kaihua Hou, Priya Pathak, John Guttag, Nikhil Garg, and Emma Pierson. Coarse race data conceals disparities in clinical risk score performance. In Machine Learning for Healthcare Conference, pages 443-472. PMLR, 2023.
12

---

[51] Alfredo Nazabal, Pablo Garcia-Morent Antonio Artes- Prres-Rodr,,uzz, and Zoubin Ghahramani. Human Activity Recognition by Combining a Small Number of Classifiers. IEEE journal of biomedical and health informatics, 2055):1442-1351 September 2016.
[52] Fabio Parisi, Francesco Strino, Boaz Nadler, and Yuval Kluger. Ranking and combining multiple predictors without labeled data. Proceedings of the National Academy of Sciences, 11((4):153311555 2014.
[53] Vera Pawlowsky-Glan GGahh and Antonella Buccianti. Compositional data analysis. Wiley Online Library, 2011.
[54] F. Pedregosa,
G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[55] Emma Pierson, Pang Wei Koh, Tatsunori Hashimoto, Daphne Koller, Jure Leskovec, Nick Eriksson, and Perey Liang. Inferring multidimensional rates of aging from cross-sectional data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 97- PMLR, 2019.
[56] Gregor Pirs and Erik Strumbelj. Bayesian Combination of Probabilistic Classifiers using Multivariate Normal Mixtures. Journal of Machine Learning Research, 20(51):1-18, 2019.
[57] Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric J Horvitz. Estimating accuracy from unlabeled data: probabilistic logic approach. Advances neural information processing systems, 30, 2017.
[58] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases, volume 11, page 269. NIH Public Access, 2017.
[59] Joel Ratsaby and Santosh Venkatesh. Learning from a mixture of labeled and unlabeled examples with parametric side information. In Proceedings of the eighth annual conference on Computational learning theory, pages 412-417, 1995.
[60] Philippe Rigollet. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 8(7), 2007.
[61] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprini 12.00090, 2021.
[62] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9184-9193, 2021.
[63] Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint urXiv:1910.00 108, 2019.
[64] Jacob Steinhardt and Percy $ Liang. Unsupervised Risk Estimation Using Only Conditional Independence Structure. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[65] Vincent Stimper, Bernhard Schölkopf, and José Miguel Hernández-Lobato, Resampling base distributions of normalizing flows. In International Conference on Artificial Intelligence and Statistics, pages 4915-4936 PMLR, 2022.
[66] Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. Text classification via large language models. arXiv preprint 230.:3055 2023.
13

---

[67] Alexandru Tifrea, Gizem Yüce, Amartya Sanyal, and Fanny Yang. Can semi- supervised learning use all the data effectiveey? lower bound perspective. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21960-21982. Curran Associates, Inc., 2023.
[68] Petter Tömberg. Large language models outperform expert coders and supervised classi- fiers at annotating political social media messages. Social Science Computer Review, page 089443932412864771 2024.
[69] Peter Welinder, Max Welling, and Pietro Perona. A Lazy Man's 's Approach to Benchmarking: Semisupervised Classifier Evaluation and Recalibration. In 2013 IEEE Conference Computer Vision and Puttern Recognition, pages 3262-3269, June 2013. ISSN: 1063-6919.
[70] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 1112-1122, 2018.
[71] Xixin Wu and Mark Gales. Should ensemble members calibrated? arXiv preprint arXiv:2101.0339,, 2021.
[72] Feng Xie, Jun Zhou, Jin Wee Lee, Mingrui Tan, Siqi Li, Logasan S/O Rajnthern, Marcel Lucas Chee, Bibhas Chakraborty, An-Kwok Ian Wong, Alon Dagan, Marcus Eng Hock Ong, Fei Gao, and Nan Liu. Benchmarking emergency department prediction models with machine learning and public electronic health records. Scientific Data, 9:658, October 2022.
[73] Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look subpopulation shift. In International Conference on Machine Learning, pages 39584-39622. PMLR, 2023.
[74] Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking ilms via uncertainty quantification. arXiv preprint arXiv:2401.12794 2024.
[75] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances neural information processing systems, 28, 2015.
[76] Bingehen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16623-16633, 2023.
[77] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as as--uu-- with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46621 2023.
[78] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1):237-29) 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes]" is generally preferable to "[No]" ", it perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We acknowledge in our introduction the past work that we build upon (i.e., prior work captures subsets of information that use) while introducing our method. We also support our claims with empirical experiments and mention the top-level results in the abstract and intro.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We state clearly limiting assumptions to our theory and also note when our model doesn't 't outperform baselines.
Guidelines:
15

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state that our theory requires assuming different optimization algorithm (UL+) than the one we use (EM). We also limit our theory to the binary classification case but state this clearly.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the names and tasks of all datasets used in the paper. We are not releasing code now as per the anonymity policy but are happy provide it upon request.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soures dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is available at https://gittht/ wwtp//owaaaaan/////occacoa Most of the data we use is open-access and does not require credentials. MIMIC-IV does require credentialed access but we describe which datasets within MIMIC-IV we use. We explain our datasets in detail in the Datasets section of the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa: httpggddddd//deuubmimmieiee /CodeSubmiiiace ssi cessonnollln onPolicy) for more details.
/
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental details in Appendix B, including datasets and imple- mentations of baselines. We report splits in Section 4.3: Evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report 95% CIs using standard errors computed across 50 randomized data splits in the supplement (Table S3-S8).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments can be done standard laptop without GPU access.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurdi: http://wwrrrppp.cppppiiuii/////a htcpppllic/Ethicauuuiieeee
Answer: [Yes]
Justification: We have preserved anonymity and checked for compliance with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss how this work can be used in various settings, including healthcare, content moderation, and subgroup-specinnn performance estimation.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML.)
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer:
[NA]
Justification:
We do not think this work could be directly used for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite previous code and data with URLs and in-text paper citations.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
We do not release new assets other than code, which we discussed previous response.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not crowdsource any data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This research does not directly involve human subjects. We received proper approval for datasets involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We used LLMs to make edits to figures, tables, writing, and code. However, we did not directly use LLMs to build the core methodology presented in the paper.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neuree http:///eeuup...eee Cwwpuriisscc///oooee wcc/onfferencce/20555555 for what should or should be described.
21

---

Appendix
A
Related work
We detail related work in unsupervised performance estimation here. Works below assume access to only unlabeled data; in contrast, SSME learns from both labeled and unlabeled data.
Unsupervised performance estimation involves estimating the performance of a model given only unlabeled data. Methods designed to address this problem often focus on out-of-distribution samples, where labeled data is scarce and model performance is known to degrade. Several works have illustrated strong empirical relationships between out-of out-of-distribution generalization and thresholded classifier confidence [28], dataset characteristics [21, 30], in-distribution classifier accuracy [48], and classifier agreement [52, 5].
Several works have formalized when unsupervised model evaluation is possible [23, 13, 28, 47, 73, 20, 62 J. and propose assumptions under which estimates of performance are recoverable. [23] and [6] assume knowledge of p(y) in the unlabeled sample. [64] assume aanditionally-indeeendee
subsets of the observed features. inspired by conditional -inirinndddddeeeeenceee assumptions made in works such as [18]. [30] assume classifier calibration on unlabeled samples. [13] assume a sparse covariate shift model, in which a subset of the features' Ilass-condition distribution remains constant, [47] illustrate misestimation of p(y) in the unlabeled example, and assume that p(y) oft-o--distribution is close to p(v) in-distribution. As [28] highlight, assumptions are necessary to make any claim about the nature of unsupervised model evaluation, and the above methods are a representative sample of assumptions made Finally, there has been a surge of interest in unsupervised performance estimation in the context of large language models [77, 34J. A standard approach here is to use a large language model to adjudicate the quality of text generated by other language models. Methods in this literature are often specific to large language models, while SSME is not.
Our work is also similar, in spirit, to methods that learn to debias classifier predictions on small set of labeled data and then apply that debiasing procedure to classifier predictions on unlabeled examples. Prediction-powee procedure to ensure that unlabeled metric estimates (e.g., accuracy) are statistically unbiased. One of the baselines we compare to, AutoEval [11], is built atop prediction-powered inference.
inference [3] and double machine learning [14] both learn a debiasing
B Experimental details
We provide an extensible implementation of SSME at https:// com/ /it///aaaaaan/s... along with support for applying SSME to your own setting. Below, we provide additional detail on the experiments reported in the main text.
B.1 Real datasets and classifier sets
We provide additional detail for the five datasets we use in our work, including ground truth p(y) for each dataset and ground truth metries for each classifier in the associated classifier set in Table S1 and Table S2. As discussed, each dataset is split into a training split (provided to each classifier as training data), an estimation split (provided each performance estimation method), and evaluation split (used to compute ground truth metrics for each classifier). We determine training splits based on prior work. We then split the remaining data in half (randomly, for each run) produce the estimation and evaluation splits. We then subsample the estimation split to have n; labeled examples and nu unlabeled examples. We ensure that the labeled data always includes at least one example from each class. Thus, the estimation split contains ni + n., examples in each experiment, and the evaluation split for each task is fixed across runs (exact sample sizes reported below). No performance estimation method sees data from the evaluation split, which is used to evaluate the performance estimates.
1. MIMIC-IV: We use three binary classification tasks from MIMIC-IV [38], a large dataset of electronic health records describing 418K patient visits to an emergency department. We focus on three tasks: hospitalization (predicting hospital admission based on features available during triage, p(y = 1) = 0.45), critical outcomes (predicting inpatient mortality or transfer to the ICU within 12 hours, p(y = 1) = 0.06), and emergency department revisits (predicting a patient's 's return to the emergency department within 3 days, p(y =
22

---

1) 0.03). We split and preprocess data according to prior work [72, 50]. No patient appears in more than one split. For each task, the evaluation split contains 70,439 examples. The classifiers in the associated set differ by function class (logistic regression, decision tree, and multi-layer perceptron) and random seed (0, 1, 2).
2. Toxicity detection: The task is to predict presence of toxicity given an online comment, using data from CivilComments [9. 42] where p(y 1) 0.11. The evaluation split contains 66,891 examples. The classifiers in the associated set differ by training loss (ERM,
IRM, and CORAL) and random seed (0, 1, 2).
3. Biochemical property prediction The task is to predict presence of a biochemical property based on a molecular graph, using data from the Open Graph Benchmark [33]. We focus on the task of predicting whether molecule inhibits SARS-CoV virus maturation, where ply 1) = 0.09. We filter out examples for which no label is observed (i.e. the molecule was not screened at all) because it is impossible to evaluate our performance estimates those examples. Doing so reduces data held-out from training from 43,793 to 28,325 examples. The evaluation split then contains half, or 14,163 of those examples. The classifiers in the associated set differ by training loss (ERM, IRM, and CORAL) and random seed (0, 1, 2).
4. News classification The task is to predict one of four news types based on the title and description of an article [75]. The dataset contains 7,600 examples, with 2,550 examples reserved for the evaluation split. The classes are balanced. Classifiers differ by the base LLM used to perform news classification. The first L.M is L.ama-3.2-3B-IInttue obtain probabilities over all classes through zero-shot prompting (i.e. proving example at a time, accompanied by options) using code provided by [74]. The three remaining LLMs are closed-source models provided by OpenAl: Sext--emedding---smtee large, and eoxttembedding-aa ada-002. We train a classifier atop these embeddings by training multiclass logistic regression on 2,500 embeddings of labeled examples.
and we
text-embedding-3-
5. Sentence classification The task is to predict one of three textual entailments from a sentence [70]. The classes are balanced and the evaluation split contains 61,856 examples. Classifiers differ by training loss WReWeight, ReSample, IRM, and SqrtRe according to [73].
SSME, in contrast to prior work. makes no asumption about the correlation between classifiers because any assumption is unlikely to hold in practice. The average correlation between classifiers in our sets for each binary task is 0.53, 0.85, 0.93, 0.81, 0.77 (for ED revisit, critical outcome, hospitalization, toxicity, and SARS-COV inhibition prediction respectively). This range of values reflects natural correlation between classifiers in practice, since each of our models either an off-the-shell classifier or trained using publicly available code.
B.2 Baselines
For baselines that require discrete predictions (i.e. Dawid-Skene and AutoEval), we discretize classifier scores by assigning class according the maximum classifier score across classes. We expand on our implementation of each baseline below.
Labeled: When estimating performance over the whole dataset, we compare the classifier scores to the ground truth labels within the labeled sample. However, when estimating subgroup-specific performance, it is often the case that there are no labeled examples for given subgroup. In these instances, Labeled reverts to estimating hubgroup-speciffe performance as performance over all labeled examples.
Pseudo-Labeling (PL): We train logistic regression with the default parameters associated with the scikit-learn implementation [54], Experiments with alternative function classes (e.g. a KNN) revealed no significant differences in performance.
operates on each classifier individually. We make use of the implementation made available by [37]. Extending the proposed approach to multi-class tasks is not straightforward,
Bayesian- Calibraainiiittto (BC): Bayesian-Calibratinn
compare to CCalibaatna
only on binary tasks. We compare to extend naturally to multiclass settings.
on binary tasks, as it does not
SPE: SPE [69] ise a semi-supervised single classifier evaluation method that relies on parametric assumptions about the distribution of classifier scores. They find that the truncated
23

---

Dataset
Classifier
Acc
FCE
AUC
AUPRC
Hospital Admission
TT-Rundomlivrest-seer
74.2
1.5
81.5
76.0
MLP-ERM-sca
M-sed22
74.4
1.4
81.7
76.7
MLP-ERM-sced!
74.4
1.9
81.9
77.0
MLP-ERM-scod0
2.4
82.0
LR-LBFGS-seed2
73.3
4.0
80.7
75.5
LR-LBFGS-scod
73.3
4.0
80.7
75.5
LR-LBFGSSseed0
73,4
2.9
81.0
75.7
9T-Randomfooret-seedd
74.3
1.6
81.5
76.1
DT-
74.1
1.5
81.5
76.1
Critical Outcome
MLP-ERM-seed2
93.9
0.9
87.9
38.6
MLPeERMMsseed1
88.1
LR-LBFGGS-seed22
93.6
1.2
87.6
34.2
MI.-EERMsseed0
93.9
0.5
87.5
37.8
LR-LBFGS-scccl -ceGS
93.6
1.2
87.6
DT RandomFoorsst-oodl
0.3
87.2
38.2
DT- DT-RunddomFFrrss-seedd
94.0
0.4
87.4
38.3
DT- DT-RandommForet-sccee
94.0
0.4
87.4
38.3
LR-LBFGS seedl
1.2
87.6
34.2
ED Revisit
DT-R TTFRRuddoFFooest-seeee
1.8
$4.9
DT- OT-Randoml DT-RaandomForretttee Forest-seed
97.7
1.7
55.3
2.7
DT- DT-Random FurdtooForsstseeeed
97.7
1.8
54.9
2.7
LR-LBEGS
LR-LBFGS-seett2
97.7
0.4
59.1
M..--FRM-seedll
97.7
0.3
59.8
3.1
MLP-ERM-sced
0.3
59.8
MLP-ERM-sced2
0.5
S7.9
LR-LBFGS-seedl
97.7
0.4
59.1
3.0
Toxicity Detection
distllbat-CORAAL-seee
88.3
6.0
86.2
40.0
RisisebentIRMMseed22
10.2
91.9
RMMseedl
9.8
66.5
Bistilber-IIMM--a
88.1
10.6
91.6
65.9
Disilleet-ERM-seedd
92.1
4.9
94.1
73.3
Bisislee
93.8
72.3
92.2
6.1
93.8
72.2
Molecule Property 60
Cnn-virtull-CORAI...
92.8
5.2
90.1
61.9
gin-virtual- cos-inuul-CORAL-scee22
92.8
5.2
90.1
61.9
gin-virtual- ERiiaal-EEM---- Moodd
94.5
73.5
gin-virtal-ERRM BRa-al------eeeee
92.4
5.6
90.7
61.1
gin-virtual- ERM-ac-ERM-s-od
92.8
5.2
90.1
61.9
Nn--virtullIRMM-seeon
1.8
90.2
58.4
gin-virual-IRM--eell
5.2
83.8
43.8
Mnn-virtualIRM--seddd
91.1
5.7
82.8
44.7
Table S1: Ground truth elassifier metries on binary tasks. We report ground truth performance for classifiers in the sets associated with each binary task. Each classifier name begins with the architecture (e.g. DT represents pecisionTree), the loss or training procedure (c.g. ERM IRM), and then the seed. Note that the equivalent accuracies on ED Revisit are byproduct of both the low class prevalence and the poor classifiers.
dataset
model
Acc
ECE
AG News 1aae-3..-33--nttteeee
85.6
8.6
text-embedding 3-lage
90.1
8.8
lext-embedding-3-am
89.8
9.1
lext-embedding--an
89.0
9.3
MultiLII
distilbert-IRM
64.8
distilbert-Re distabbertReeample ReSample
81.4
distilbert-Be Reillerrh---eWehhtt
80.9
7.4
distilbert-SqrtRe. Weight
81.4
9.2
Table S2: Ground truth classifier metrics multiclass tasks. We report ground truth performance classifiers in the sets associated with each multiclass task. Each the LLMs fine-tuned for AG News are sentence transformers, while the MultiNL classifiers all use DistilBERT [63] as the base architecture.
24

---

Normal distribution demonstrates reasonable fit across datasets; accordingly, we implement SPE as mixture of truncated Normal distributions fit to each classifier's predictions individually.
Dawid-Skene: We implement Dawid-Skene with tolerance of le-5 and maximum number EM iterations of 100 (the default parameters), using the following public implementa- tion: https: / gi thub. com/ aallascard/ /d/idddwwwdd///e///ddd/ddddddddd dawid_skene. skene. Dawid-Skens accepts discrete predictions, so we discretize classifier predictions using thresholding the predicted class probability at k.
Majority- Vote: Vote:: We implement Majority- Vote as the securacy-weigheee average of discrete predictions made by each classifier. We discretize predictions by thresholding predicted probabilities at K We weight each classifier in proportion to its accuracy on the available labeled data.
Active-Testing We implement Active- where the method selects fixed number of examples to label out of pool of unlabeled examples, according to the approach proposed by [43]. We select examples according to the acquisition strategy for estimating accuracy. a metric for which a public implementation is available, and limit our comparison this metric.
AutoEval: We implement AutoEval using implementation made available by the authors [11]. The implementation, to the best of our knowledge, only supports accuracy estimation across set of classifiers, AutoEval on accuracy estimation, as additional metrics are not supported by the public implementation.
limit our comparison to this metric. We compare to
B.3 Computing effective sample size
In order to compute effective sample size, we produce 50 samples of labeled data for each increment of 5 between 10 labeled examples and 1000. We then compute the mean absolute metric estimation error of using labeled data alone, across all runs. The effective sample size of given coni-supe evaluation method is thus the amount of labeled data which achieves the most similar mean absolute metric estimation error.
C Proof of Theorem 1
We derive a hegh-probbbilityy error bound on performance estimates based on prior results in semi- supervised mixture models [67]. The proof consists of two steps: we first bound the error in estimation of component separation. We then use the relationship between component separation and classifier performance to bound the error in estimating classifier performance.
Model of and s. We consider a stylized binary classification setting drawn from previous work [67] where classifier scores are drawn from balanced mixture of two Gaussians. For data in the positive class, classifier scores are drawn from multivariate Gaussian The Gaussian assumption holds when the distribution of classifier logits follows normal distribution. NCcc I). where For data in the negative class, classifier scores are drawn from multivariate Gaussian (0, I). Let ny refer to the number of unlabeled examples, ny the number of labeled examples, and the number of classifiers. Additional classifiers increase the dimensionality of the multivariate Gaussian.
RR
Estimator We reason about SSME's performance using simplified semi-supervised learning algorithm. The estimator, UL+ first uses the unlabeled data to identify decision boundaries, and then uses the labeled data assign decision boundaries to classes. As the authors note, this is not the most efficient use of labeled data. We expect that using the labeled data in the expectation pnpectation-mxxnnee maximization procedure as SSME does weakly outperform UL+ because it uses labeled data when learning the decision boundaries. For additional detail on the estimator, please refer to Section 2.3 of [67].
Theorem 1 Let AUCr, ACC's be estimators for AUC,, ACC,, estimated using UL+ and Cs be the L2 distance between components along dimension k. Assume nu > d log(n.) d'log(dn.) and 2 2. With probability 1 p. errors in the estimated max (d, at min( eb
AUC and accuracy of classifier &s differ by at most:
25

---

|AUCk AUCHIS
$
Ck
+
Ck
cc
(5)
V
V2
ACCC
ACCAI 5
Ck
Ck
Ec
(6)
2
2
Viaal Tel
#n/lle|1
co [|il
dioring] [|ll
where S
+ ||e|le
and Co is universal constant.
Bound error separation estimation as function of ni, nuy and separation Prior work [67] has established bounds on the eπror in mean estimation in the context of semi-supervised learning algorithm which first uses the unlabeled data generate decision boundaries, and uses the labeled data to assign labels to regions. The authors prove mean estimation error can be bounded by:
+ ||c/le |mmlleel
(1-10p2
(--Co
Alan(ts)
E [enl
Tell² Tilll
||c|3n
where Cu is some universal constant and not equivalent to ||c|]. We can convert this expectation bound into high-probability bound using Markov's inequality:
d V ||c|2n
+ ||c|le 2m]lle]]
(1-
(1-10)2
Co [|oll
P
V
[ol] 2ng
S p.
Because we pin = 0, error in mean estimation is equal to error in separation estimation in this setting, i.c. Eμ Ee. So we can state that, with probability p, the mean estimation error obeys the following inequality:
d [c||3n
|clle -mellll1 (1-
dlox(an) [ellanu
Ec
S p
2. Bound error in performance estimation. The AUC of the optimal linear classifier (separating classifier predictions for the negative class from classifier predictions in the positive class) function of the Mahalanobis distance c, equivalent to the L2 distance in this setting, as prior work has shown [55]:
AUC = + (Iccl//v).
where & is the cumulative distribution function of the standard normal distribution. Similarly, the accuracy of the optimal linear classifier is:
ACC (||el//2)
Above, we derived bound on the error of separation estimation, e,, Now, we map the error in separation estimation to errors in performance estimation (AUC and accuracy). We can state that the following inequality holds:
Pr([e||| - ec < ||
||||| + Eo)
2 P
Thus:
|c||
Ec
5 $ |l@l|
< Φ
||||| + Ec
(7)
2
V2
V2
|||||
AUC v 2 ||||| + te)
Ec
(8)
V2
2
|||||
Ec
+
|||| /2
< AUC - AUC - o
|||||
+ te)
2
|||||
(9)
V2
/2
v2
26

---

Accuracy Eximation Sounda with Varsing
AUC Stimatoo Sourds ds with Varying
Citrabe ACOURII Sourdh
30
04
12
03
87 03
Figure S1: Estimation error based on original classifier set performance. We plot estimated classifier performance as function of original classifier performance for various values of c,s for accuracy (left) and AUC (right). The less accurate the classifier set, the larger the error bands.
Given that assume the classifiers are better than random (i.e. ||||| 0) the term on the far left of the inequality is larger in magnitude than the term on the far right, so we can simplify as:
||c||
|||||
te)
||||| V2
|||| E))
||c/|
AUC AUCI
max
Φ
(10)
V2
V2
V2
AUC AUC|
|
|||||
fc) Φ
(|cll
2 )
(11)
V2
Finally, error in estimated separation is necessarily larger than the emror in any one dimension, allowing us state the inequality for each dimension of the mixture. We can also follow the same logic above to derive a bound on етrоr in estimated accuracy. We can state:
IAUČ& AUCAS
|
(12)
V2
V2
IACČ& ACC&L
Ck
$
Ck
Ec
(13)
2
where the following holds with probability 1 p:
d
|m|lll] (1-
Co dloginn "mu
||c/le
||]]
[|ll
Ec p
We plot bounds in estimated classifier performance function of c in Fig. S1.
C.1 Impact of additional classifiers
Increasing the number of classifiers impacts the bound in two ways. First, an additional classifier increases the dimensionality of the mixture model, thereby creating looser bound. Second, an additional classifier increases the separation between components ||c|]. because each classifier is assumed to be independent from one another. If the marginal increase in separation outweighs the marginal increase to dimensionality, the bound on 6e tightens. Specifically, let's say that the separation of components for the new classifier (i.e. Ca+ is equal to 8. Separation ||c|| increases to /lcc|2 + 82. Dimensionality increases by 1, i.e. d 1. When |2 2 8 exceeds vd + I, the first term in the expression decreases. Specifically:
The of the augmented vector c' = [ exceeds Vd + 1 and only if
8 /
||c|22
27

---

If ||||| > vd+1, then any 8 > 0 will result in ||c'll > vd+1. Otherwise, 8 must be large enough compensate for the square root of the difference between the new dimensionality (d 1) and the squared norm (I]c||²). At extreme, if the new classifier is perfectly correlated with another, it increases dimensionality at no benefit to separation between components, resulting a looser bound.
D Supplementary results
D.1 Results on synthetic data
We corroborate theoretical findings from Section 3.2 in synthetic experiments, including settings where the assumptions do not hold. We draw classifier scores for positive examples from distributions centered on vector c, where entries of are drawn from ((c, enforcing minimum entry of 0.01 (to ensure that each classifier exhibits performance better than random). We consider Tu E 50, 100, 250, 500, 1000, 2000, d E 2, 4, 6, 8, 10, and ||c|| 0.75, 1.0, 1.25, 1.5, and sample 50 datasets for each setting. We provide 20 labeled examples to SSME across all synthetic experiments.
The three theoretical findings we make (using simplified semi-supervised learning algorithm and a Gaussian assumption) hold in our empirical setting. Our synthetic setting differs from our theoretical setting in two key respects: we use SSME to infer performance and we simulate violations of the Gaussian assumption. Our theoretical findings hold in this more realistic setting; Figure S2 plots results.
D.2 Results reporting mean absolute error
In the main text, evaluate method and all baselines using 20 labeled examples and 1000 unlabeled examples and report rescaled mean absolute error across metrics and tasks. Here, we supplement those results by reporting mean absolute error across each task and metric and expanding ny to include 50 and 100. The number of unlabeled examples remains the same (1000) to isolate the effect of additional labeled data.
Tables S3, S4, S5, and S6 report results on each binary task for accuracy, ECE, AUC, and AUPRC, respectively. Three high-level findings emerge. First, SSME-KDE achieves the lowest mean absolute error (averaging across tasks and amounts of labeled data). Second, SSME-KDE consistently outperforms the ablated version of SSME, fit to single model at time (SSME-KDE-N finally, SSME-KDE is able to produce performance estimates that are quite close, in absolute terms, to ground truth. For example, when given 20 labeled examples and 1000 unlabeled examples, SSME-KDE estimates accuracy within at most 2.5 percentage points of ground truth accuracy (across tasks)
And
Tables S11 1 and S12 report our results on the multiclass tasks, for accuracy and ECE respectively. Note that we exclude -avesiinnCallbbattion from multiclass comparisons because the method does not natively support multiclass recalibration. We also omit AutoEval from Table S12 because the implementation of expected calibration error within the framework is straightforward.
D.3 Results reporting absolute performance estimates
Results thus far have reported aggregate errors in performance estimates across classifiers in the set. Here, we include results a per-classifier basis in the context of toxicity detection, the task for which we have the largest variability in classifier quality (Tables S7, S8, S9, S10). The tables illustrate how SSME's performance manifests per-classifier basis, often producing more accurate estimates than the baselines on the classifiers with lowest performance. The tables also make evident that SSME's improvement in performance estimation can be attributed to a significant reduction in the variance of performance estimates across different data splits.
D.4 Results reporting robustness to kernel choice
We find that SSME's performance estimates are stable across three kernel choices: Gaussian kernel, an Epachenikov kernel, and an exponential kernel. Table S13 reports our results in the context of toxicity detection and accuracy, but results generalize across tasks and metrics.
28

---

Tabrlol
Majunty
PL.
SPR:
BC:
AusePcal
ActoveToumng
SSMI-M
SSMP
CHiicll Ourcore
1050
1.00
2.01 4:15
4.55-000
10:73+500 564
1.800055 00-0065
4.7%+199
5174000
177-0225
40
1030
2:00+330
11.19
6:31 1
510 0.01
4.77 -0.11
16.70.90
3.6--0.6
1.01 8.06
5.1 0:00
145+075
100
1030
2:29
g1.41
4.10
1.58
0.87
1.1:021
2,00
1.3020.0
0.77 40.33
436.2007
50
1030
2:02
0.5%
2.00
2.2
0.34
195 45.7
300 1.26
L4b:022
A.11
100
1030
1.45 0..3
2000
1.54
1 00.33
11 5:113
143 0311
1.42 10.99
2
10.16
0,57 A
Roopitll Nininis
1030
6.6
19:55 #0.5
245=044
7
1.0
0:
16.57 00.6
0.25
1.32
100
1030
3.64 a 0.55
10.66
3.01 0.53
17.91 8.33
2.4
4,00 0.55
9:24
0.45
9.33
SARS-DWY Iriibitin
20
1030
6.11
195-100
11:07-100 5.0
-.25-6.111
4.5
2.30-110 16
1030
7:04
01
0.3--031
4:01 -0.17
4.09
1.94 0.25
5.9 4.44
Tookit Detection
20
1030
5.95 0.73
4.57 x0.11
500
4:82 00009
5.29=0.9
5.2
631=025
1030
4.00
2.8 0.20
2:005
8.75
1.96
4.3
1.37
2.26 2:42
4,46
Table S3: Mean absolute error in accuracy estimation binary tasks. We report mean absolute error (MAE, averaging across classifiers) across five binary classification tasks and different amounts of labeled data. Each entry corresponds the mean MAE across 50 randomized splits of data, accompanied by the 95% CI (computed as 1.96 x the standard error) in MAE across splits. We bold the best performing method in each row, and underline the next best performing method by mean MAE. SSME-M refers to performance when fitting SSME to a single classifier's ' scores, instead of modeling the joint distribution of P(y.s) s).
Dutaset
Laheled
Majority Vose
PL
Dawid Stene
SP%
BC
SSME-M M
SSME (Ours)
Critical Chutcome
11.01 1.12
2.75 0.26
0.64
0.00
16.17 6.69
3.48 0.76
3,17 0:30
1,16
50
1000
6.22 = 0.62
3.68 2 0.34
5,38 0
2.37 a 0.09
2091 9:43
2.56 0.64
3.01 0.26
1.13 0..13
100
1000
4,20
= 0.38
3/43 = 0.28
3.65:021
2.25 + 0.30
1284 8.99
1.60 0.25
281
0.22
1.15 0.10
ED Revisit
20
1000
8.37
= 0.87
181 + 0.02
4.16-
3.25 + 1.68
14.15 10.31
5:57 0.77
1.88
0.24
0.76 0.004
50
1000
4.82 a 0:48
1.77 a 0.03
2.29 0:47
1.29 a 0.46
17.1
2.04 a 0.46
1.83
0.19
0.73 a 0.05
100
1000
3.29
0 0.24
1,59 : 0.04
1,36 0.22
1,34 a 0.21
1025 4.82
1.00
1.51
2.117
0.73 0.00
Hospini Admission
20
1000
21.76 s 1.16
21.73 e 2.75
8.10 1.78
17.31 a 0.13
16.79 0.58
5.1. 1.09
5.54
1.97 0.13
1000
12.74 =
2100 +
5.02 0.68
16.60
2.45
4.55
4.33
2.06 0.18
100
19.97
0.40
0.45
0.15
SARS-OW Inhihition
0.95
0.38
+ 0:15
11,85 5.663
0.33
2.57 0.18
3.38 0.11
a
à 0.16
8.96 2.774
1.73
2.27
3:41 1011
180
1000
2.18 = 0.32
2.36
+ 0.16
7.26 4.771
1,35
1.29 1:11
Toxicity Detection
20
1000
5.85 a 0.80
5.08 a: 0.39
5.09 a 0.79
4:40 a 0.09
5.114
4.69 1 0.34
5.67 a 0.19
2.35 0.133
50
1000
3,99
= 0.63
4.84 = 0.33
3.144 = 0.46
4.20 1007
7:5h #177
3.97 0.34
4.57
+ 00.26
2.26 0011
100
1000
2.37 + 0.37
4.63 + 0.33
1.91 a 0.27
4.10 + 0.09
6.75 0.15
5.30 + 0.25
5.49 0.20
0.19
Table S4: Mean absolute error in ECE estimation on binary tasks.
Dutaict
1y
Labelod
MajocityYooe
PL.
Duwid-Stene
SPE
BC
SSME-M
SSME (Ouss)
Critical Outcome
20
1000
10.09
& 1.34
9,45 0.39
31.73 1.10
9.39 e 0.35
10.92
2 0.25
4.72 a 0.63
2.52 0.34
50
1000
7,50
1,28
8.06 0.53
27.35
1:53
8.49 e 0.40
20.24 11.78
5.61 e 1.28
2.30 a 0.4K
1000
0.95
0.61
20.45 1.22
1.97
13.71 10.26
2.70 0.26
3.82
3.82200.5 0.48
2
ED Revisit
20
1000
1,85
2.49
0.20
8.27
11,39 0.15
5.92 0.87
50
1000
17.95
7.48 0:35
4.67
7.30 0.31
11.99
0.71
1000
1.67
14,95
7,09
2
0.33
11:28
5.08
Hospital Admission
1.29
1.00
8:9 1.65
16.70
1581
2.7.1..0
0.54
2.51
50
1000
5,05
0.97
15.47 0.56
5.59 1.20
16.18 + 0.10
00.40
0.46
3.18 =
2.51
1000
3.57
0.71
14.75 0.44
1.74
15.32
1.35
SAR5-CaV Inhibition
20
1000
9.61 2.56
0.60
30.92 e 1.21
7,50 e
13.72 5.03
1122
=
3.48
50
1000
5.84 1.01
5:22 0.39
23:71 à 1.19
7.06 e 0:29
10.67 2:28
3,62 à 0:27
5.00 e 0.52
3.41 0.47
100
1000
3.97
0.55
4,95 2 0.35
16.33
6.04 : 0.32
10.34
3.55 + 1.38
4,21 a 0.53
3.46 : 0.45
Toxtcity Detection
20
1000
6.71 0.99
5,65 0.82
17.32
6.20 = 0.11
12.38 a 5.76
11.16
6,05 = 0.28
3.34 : 0.23
50
1000
4.76 0.91
5.15 0.47
11.79 1.78
5:97 + p.09
8:23 1.65
4.76 + 0.31
4.86 - 0.29
3.15 = 0.18
100
1000
3,82 0,60
4.85 0.20
7.54 1.00
5.84
7.28 a: 0.22
4.25 à 0.27
4,15 s 0.33
3.09 a 0.22
Table S5: Mean absolute error in AUC estimation on binary tasks.
Datsset
F2
Labeled
Majority-Nite
Dawid-Skeee
SPE
SSME-M
SSME (Oum)
Ciiticll Outcome
20
1800
12. 36.86+
+ 5.05
36 + 2.09
22.08 + 1.85
30.02 1.18
45.00 6.8) 6.81
9.20 1.67
11.48 + 1.51
6.11 a 0.73
50
1000
22.51
3.65
26 12 2.77
20.48 2:23
35 + 1.11
44,67 + 7.51
9.34 1.43
11.9% + 1.43
6.17= 0.96
100
1000
15.71
2.44
24.01 2:47
16.45
2:02
33.21
1.0
47.35
6.71
8,96 1.4L
11.30 1.67
5
ID Revisit
20
1000
4.53 3.66
5.14 0.89
5.12 1.30
16,66 11:50
9.14 1.04
5.07 0.30
1.67 0.29
1000
8.85
2.26
3.12 0:79
3,04 0.77
10:06
6,03
0.30
1000
6.34
1.54
0.33
1.74
12.37
4.23 0.55
2.33
Hospital Admission
1000
9,43
1.62
6:
10.30
21.15 0.36
21.13 0.334
5.26 1.07
4:36 0.44
3.47 0.57
1000
1.31
0,80 18.73
0.74
3.70
4.12
3.40 0.60
SARS-CaV Inhibition
(000)
22:27
3.00
18:75
2.44
1.08
6.
7.54 0.76
13.81 1.53
1000)
15:02
14.10
30.29 2.61
1.07
34,97 a: 1.49
5,40 0.96
12:92 à 1.01
21.06
1800
1.56 11.53
15.72 2.66
20.34 1.79
12.61 1.05
33.60
8.27 0.899
11.01 m
30.67
Texicity Detection
1800
10
2.34
29.05
25, 3.50
26
0.36
37, 6.70
23, + 0.73
10.89
1800
13.78
30.54 3.70
20.15 3.48
25 35.24 0.36
31.94
16.84 1.58
18.00 + 1.08
9.91
100
1000
10.59
1.71
038
14.06 2.00
24.51 2
3..94
14.15 & 1.48
14.59 a: 1.26
9.88 a 0.97
Table S6: Mean absolute error in AUPRC estimation on binary tasks.
29

---

meddl
Labeled
MojoeityVote
15.
Dawrk-Serre
Ansofisal
ActieeTTosinn
SSMP (Ckrr)
Ground Truth
divilber-CRAAL
83.90 14,04
6 13:35
13:45
35.04: 10:53
$7.02 21.30
56.49 2.07
88.27:00.8
daiilber. BRM
89.30
12.04
91.3427.55
66.92.6..7
05:68 1.94
84.55
92.11 92.13.s1.88 1.88
90.79 11.63
a 16.1-
08.59
0:90::0 1.71
92.17.0.144
89 09:10+10 13:95
91 7:4
17.08-6.54
94:35.210 2.55
01 10.96
Vn. 17:14-199
90.75 17.06
05.0n 17.00
91.54 1.65
v0.17 u
daallber: 3RM--50022
8 12.42
7.55
56.67 6:72
15.78:8..844
83
2 46.57
2
9048
13.38
92.54 = 12.24
93.65 1.11
92.11 +0.15
dasilber--BR
56:00
62:92.6.99
05.27
1496
88.10
9166
+0.1
dieilhen- MRM--
86.50 13.54
00.48 9:00
6.335
05.55
86.96 15.555
14.24
87.78 13.96
$9.19 19.15
00.08 1.78
10.16
Table S7: Mean absolute error in accuracy estimation per classifier on toxicity detection.
model
Laheled
Majority-Von
Dewid-Skenc
SPE
BC
SSME (Ours)
Ground
distiieen-COAALL
13.80 + 11.38
12.60 9.19
8.78 7.05
10.78
25.34 + 53.26
7,47 8.12
8.50
+ 1.87
5.98 + 0.18
disiibber-ERM
10.37 + 11.48
8.16 7.77
6,43
4.37 2.08
14.87 + 49.40
1.97
4.96
1.47
6.14+00.17
distilbertERMM.oo
9.91 12.33
8.34 7.65
6..39
4.59 2:21
6,45 10.72
1.93
5.13+ 1.53
6.21 0.166
(istilbent-EMM-ees
10.02 11.80
7,70 7.75
10.96-6.55
1,93
14.36 46.44
2.32-2247
3.99 1.39
4.19
distilbertIIM
12.59 + 13.50
6.52 9.99
15.77
3.09 1.31
6.69 14,92
4.61 5.14
6.85 1.72
10.61 0.16
distilbert. (RM-seedd
11,93 = 13.31
6.35
15.03 6.338
2.98 1.24
4.18 12,63
3,88 3.89
6.38 1.83
9.78 0.21
distilbert (RM-seed2
12,06 = 12.00
6.71
15.55 66116
3.33 1.46
13.01
4.660
6.71 1.74
1.
Table S8: Mean absolute error in ECE estimation per classifier on toxicity detection.
modell
Labeled
Majority-Vote
PL
Dawitl-Sene
SPE
BC
SSME (Ours)
Ground Truhh
disilbet-CORAL
27.74
90.53 8.91
72,20 13.551
94.89 a 2.31
79.61 72.95
84.23 6.84
91.38
3,14
86.23 0.33
distiben-ERMM
91. 10.00+1
96.36
74:90 15.08
0.65
08.52+
1.84
95.97
0.222
distilben-ERM--ees
02.18 14.63
96.36
74.2 15.05
0.59
0..50
98.30 1.86
95.96
1.39
93.75
distilher-ERM ERM-seed2
93.32 11.62
96.55 8.886
75.03 + 15.17
98.89
0.62
90.81
57.23
98.07 + 2.20
96.16 1.33
94.08 0.19
distibent-IRA IRM
91.46 17.13
95.52
13:36
74,69 4 14.67
98.28
à 1.17
99.06
98.05
2 1.88
95.38
2:22
91.57 0.25
distibbert IRM-seodl
90.75 20.55
95.25 12,97
74.58 11.85
97.97 1.60
99.52 1.60
98.07
2,00
95.18 2.19
91.00 1.31
distilbert IRM-sead2
91.89 15,94
95.37
13.62
74.68 14.90
98.39
90.44 55.04
98.33 1.84
95.59 1.79
91.86
Table $9: Mean absolute error in AUC estimation per classifier on toxicity detection.
model
Labeled
Majerity Vote
PL
Dawid
SPG
BC
SSME (Ours)
Ground Truth
distibettCOOR CORAL
60.78 47.30
67.66
68.12
31,91
19.96
1
71.91 75.73
$0.86
21.03
60.27 10.78
40.00 0.70
distlbar-ERM
38.26
82.78 + 58.05
42.28
94.59 2.39
79.50
71.95
92.51
79.63 7.22
0.73
distilbest-RRe E
29:38
83.34
42.27
83.36
91.54
79.78 6.78
distibeatt ERM-szed2
2
83.41
55.47
42.474277.0
2.334
78.44
90.15 19669
80.06 6.54
73.33 0.69
distibert-Bn IRM
17.74.38.11
11.261
40.79 1 25.87
92.89
81.71 +: 76.90
88.23
14:48
76.90 8.27
63.86 2
distilhert -
09.21 39.17
$5.76
41.25 26.40
93.64
82.29 = 74.37
89.58 11.12
71.84 7.67
66.30 0.36
distilbert LRM sced2
39.97
85.52
40.65 26.19
92.67
77.63 77.91
90.34 12:47
77.16 7.20
65.46 0.79
Table S10: Mean absolute error in AUPRC estimation per classifier on toxicity detection.
Dosaase
Labeled
Majority- Vode
PL
Dowid-Skene
Autabval
Active-resting
SSME (Ours)
0.16
11.72
6
5.59 0.76
0.19
3.92
0.14
5.22
6.51
1.40
2.17
2,71
0.14
6.0=0.16
0.59
8.7 1.20
0.24
MulaNLI
1000
7.46 1.07
8:30
7.95 1.26
11.73
1.04
1.70
199 0.24
50
1000
4.42 = 0.55
0.14=0277
3.08 0.62
11.1
4.17+0.544
11.35 1.70
1990 =021
100
1000
3.27 a 0.46
0.30
2.47 0.51
10.72
3.17
11.63 a
2,00 0.23
Table S11: Mean absolute error in accuracy estimation on multiclass tasks.
Dataset
Ru
Labeled
Majority- Vote
Votte
PL.
Duwid- Skene
SSME (Ours)
News
1000
5.53 0:93
4,30 0.13
11.85 1.21
5.80
3.37 0.17
50
1000
3,88 = 0.76
4.18 0.11
0.77
5.70 0.10
3.28 + 0.15
100
1000
2.61 0.49
4,07 0.15
2.78 : 0.48
5.53=2..1
3,13 0.21
MultiNII
20
1000
11.57 1.13
3.87 0.18
1.84 = 1.14
2.95 = 0.08
2.06 0.24
50
1000
6,14 = 0.67
3.94 0.14
3.10 1.100000
0.10
2,06 + 0.20
100
1000
4.52 0.51
3.82
2.37 2.44
3.18
2.19
Table S12: Mean absolute error in ECE estimation multiclass tasks.
30

---

SSME gousssas
SSME -puueccuney
SSME KDE EeDponenneetial
distilbert, CORAL
86.61 1.3587
86.61 1.3586
86.60 + 1,3884
Sssilbert_RMM
L.0066
93.81
distilbert, FRM, seedl
93.76 = 0.9585
93.75 0.9584
93,73 # 0.9314
distilbert ERM__RRM_RRR seed2
93.77 0.6859
93.77 0,6859
93.75 0.6643
distilbert_ JRM
91,80 = 1.0459
91,80 = 1.0460
91.78 1: 1.0122
distilberi, seed]
92.37 = L.0333
92.37 # 1,0373
92.35 # 1.0252
Bustiberr_IRM_ees
92.33
92.33
92.31 0.9346
Table S13: Robustness of SSME to kernel choice. Because density estimation can be sensitive to kernel choice, we analyze the stability of SSME's performance estimates across three kernel types. We find no significant imapct of kernel choice on SSME's performance estimates (although different kernels do produce small changes in accuracy estimates). Results hold across metrics and tasks; we report results for each toxicity detection classifier and accuracy for brevity.
Dutiset
Labeled
MajorityNNil Note
David-Stese
AutoEval
Active Active-Testing
SSME (Ours)
SSME-NF
NG News
20
1000
5.48 0.45
5.13 1.05
11.72 a 0.63
2.17
5.59 0.50
8109 0.41
0.1
0.12
50
1000
+..22
4,96 0.007
0.22
6.51 0.36
4,36 0.40
828 0.72
1,3
0.10
100
1000
2.71 +0.23
4.7400.10
2.25
6.20 0.08
2.71 0.30
8.7 +.66
3.1
0.10
MuliNII
20
1000
8.30 0.55
7.95
11.73 0.50
0.50
10.30
1982
3.08 2..0
50
1000
4.22+00.2
8.14
3.88
0:33
11.41 0.00
4.17 0.22
11885
1.000011
2
100
1000
3:27 +.225
7.54
2.27
00.0
3.17=0.23
11:63 0.90
2.02
2.12
Table S14: Mean absolute error in accuracy estimation on multiclass tasks, including NE
parametrization. SSME-NF performs comparably or worse that SSME (Ours) across the two multiclass tasks and three labeled data settings we consider.
D.5 Results reporting performance of alternate parameterization
SSME can be fit using other functions approximate P(sly). We additionally explored the value of parameterizing each component using normalizing flow (implementation details in Sec. E.3), a technique that increasingly used to approximate complex densities [2].
We find that using a KDE outperforms the normalizing flow on each binary task, for every metric (except estimating AUPRC for SARS-CoV inhibition prediction, Table S15). We find that the KDE performs similarly to the normalizing flow for each LLM-based task, although by a smaller margin (Table S14). We hypothesize that this is due the established challenges of approximating multi-modal distributions with normalizing flows [65, 16).
For higher digher-dimensional tasks, the normalizing flow seems to outperform the KDE. In particular, we ran an experiment on ImagenetBG, which contains nine classes and four classifiers. Table S16 demonstrates that in this setting, the normalizing flow performs better than the KDE.
Dataset
Fig
Phu
Labeled
SSME-M
SSME (Ours)
SSME-NF
Critical Outcome
20
1000
5.19 4 1.07
4
0.67 = 0.13
9.80 1.41 8
50
2.90 0.59
1.65 0.25
0.78 0.13
100
1000
2.09 + 0.41
1.30 0.20
0.77 = 0.13
8.28 1.64
ED Revisit
20
1000
5.11 0.98
0.45 0.10
3.54 1.87
50
1000
2.02
1,46 0.22
0.53 0.11
1.10
100
1000
1.43 + 0.332
0.25
0.57 + 0.11
2.51 0.94
Hospital Admission
20
1000
7.32 1.25
3.29 + 0.47
1.88 + 0.29
3.19 0.95
50
1000
5.40
3.17 0.83
1,95 0.27
2.27 0.62
100
1000
3.64
1 0.55
3.06 0.45
1,51 +: 0.23
3.66 1. 2.31
SARS-CoV Inhibition
20
1000
6.111.996
3.06 0.23
2.30 2 0.16
11.67 1 5.26
50
1000
3.22 0.557
2.59 2.29 0.26
2.35 + 0.10
20.89 7.38
100
1000
2.04 + 0.38
1.84 + 0.24
2,36 0.13
9.89 4.52
Toxicity Detection
20
1000
5.95 4
6.21
2.34 0.15
0.14
50
1000
4,05
0.68
5.38 4 0.28
2,22 2 0.13
4,31
100
1000
2.44
3.80 & 0.32
2.14 0.15
0.12
Table S15: Mean absolute error in accuracy estimation on binary tasks, including NF parame- terization. We include columns for the Labeled, SSME fit to each classifier individually (SSME-M), SSME parameterized by a KDE (SSME (Ours)), and SSME parameterized by an NF (SSME-NF). While SSME-NF sometimes outperforms SSME-M, SSME-NF never outperforms SSMB-KDE on accuracy estimation in any of the tasks or amounts of labeled data we consider.
31

---

Dataset
n1
nu
Labeled
Majority- Vote
DS
AutoEval
Active-Testing
SSME (KDE)
SSM
ImnetBG
20 1000
6.62 + 2.74
2.99 + 0.90
33.45
2.96
5.78 + 0.71
6,55 2.62
10.83 5.34
8,76 + 1.00
2.65
ImnetBG
50
1000
3.98 + 1.63
3.01 + 0.61
17.88
+ 2.78
5.69
- 0.73
3.87 + 1.56
12.25 7.28
8.18 - 0.90
2.66
ImnetBG
100
1000
2.97 + 1.38
2.73 + 0.57
9.37 + 1.53
5.34
+ 0.63
2.73 + 1.13
9.08 + 4.22
8.02 0.90
2.10
Table S16: Comparison across evaluation methods on the ImmeiBG dataset with varying numbers of labeled examples (ri)) and unlabeled examples (n,)) SSME (NF) consistently achieves the lowest
MAE.
Labeled Examples Initialization
MAE, ECE
MAE, Accuracy
KNN
0.045
0.046
20
Draw
0.035
0.034
50
KNN
0.037
0.040
50
Draw
0.043
0.043
100
KNN
0,031
0.033
Draw
0.037
0.036
Table S17: Performance comparison across different numbers of labeled examples and initialization
methods (Dataset: CivilComments).
D.6 Results reporting robustness to initializations
We find that SSME's performance remains strong when using an alternative plausible initialization method: using a majority vote among k=5 nearest neighbors among the labeled examples to initialize cluster assignment for unlabeled examples. We compare this alternate initialization method to our original initialization method (which we refer to as "draw" in the table) on the CivilComments dataset with three different amounts labeled data (20, 50, and 100 points). We include our results on ECE and accuracy estimation error in the table $17, where we find that the two initialization methods perform comparably on the CivilComments dataset.
D.7 Results reporting performance as number of classifiers increases
As shown in table S18, we find that SSME's performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well one adds an inaccurate (e.g., worse than random) classifier to set.
We also find that SSME's performance does not vary greatly across classifier sets, suggesting that SSME's performance is robust to using different classifier sets. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.01 1, and the standard deviation ECE estimation error is always under 0,015, indicating consistent performance across classifier sets.
First, we find that performance does not vary greatly across classifier sets, suggesting that SSME's performance is robust to using different classifiers. In particular, the standard deviation in accuracy estimation eтror across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015.
Second, we find that SSME's performance improves the number of classifiers increases, which accords with our thcoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds inaccurate (e.g., worse than random) classifier to the set.
D.8 Results reporting performance for different bandwith selection procedures
We ran an experiment to test the robustness of our results to the choice of bandwidth. In the paper, we use the Sheather-Jones algorithm to automatically identify a bandwidth. We compare this approach to (1) another automated bandwidth selection algorithm (the Silverman algorithm) and (2) two fixed bandwidths, chosen to larger but within an order of magnitude of the Sheather-Jones selected bandwidth. We conduct experiments using the CivilComments dataset and 3 labeled dataset sizes
32

---

Classifiers
MAE, ECE (SD)
MAE, Accuracy (SD)
2
0.0457 (0.0059)
0.0471 (0.0047)
3
0,0302 (0.0107)
0.0308 (0.0098)
4
0.0268 (0.0094)
0.0273 (0.0096)
5
0.0252 (0.0072)
0.0253 (0.0078)
6
0.0235 (0.0043)
0.0235 (0.0048)
7
0.0225 (only one subset)
0.0223 (only one subset)
Table S18: Performance as the number of classifiers increases. Standard deviations (SD) are shown in parentheses. Dataset: CivilComments.
# Labeled Examples
Bandwidth Selection Procedure
MAE, ECE
MAE, Accuracy
20
Sheather-Jones
0.024
0.023
20
Silverman
0.040
0.038
20
1.0
0.037
0.036
20
2.0
0.046
0.044
50
Sheather-Jones
0.023
0.022
50
Silverman
0.066
0.065
50
1.0
0.062
0.062
50
2.0
0.045
0.044
100
Sheather-Jones
0.022
0.021
100
Silverman
0.049
0.047
100
1.0
0.056
0.055
100
0.034
0.032
Table S19: (Dataset: CivilCommnents) Comparison of bandwidth selection procedures. Sheather- Jones outperforms another automated selection method (Silverman) and two fixed bandwidths.
(20, 50, and 100 labeled datapoints). While results remain strong in all cases, we achieve the best performance when using the Sheather-Jones bandwidth selection procedure.
D.9 Comparison to baselines drawn from weak supervision
Popular approaches to weak supervision including Snorkel [58] and FlyingSquid [27] implement latent variable model equivalent to Dawid-Skene. Both works build on Dawid-Skene to incorporate information about pairwise correlations between labeling functions; [58] employs technique to infer dependencies, while [27] assume these dependencies to be user-provided. When we applied standard approach dependency inference [4] in our setting, we observed that (1) all classifiers are inferred to be dependent on one another, and (2) the number of dependencies raised issues with convergence. It is thus not feasible incorporate dependency inference, and the resulting latent variable model is equivalent to Dawid-Skene.
D.10 Comparison to ensembling
While limit the scope of our experiments in the main text to semi-supervised methods that make use of both labeled and unlabeled data, another approach would be to produce an estimate of Pr(y = k))(()) by averaging the classifier scores. This approach results in an unbiased metric estimator when the resulting ensemble is calibrated, theoretical results by [37] show. Such an approach has natural downsides: it is sensitive to the composition of the classifier set, does not improve with the introduction of labeled data, and relies that is unlikely to hold practice [71]. Here, we provide experiments to illustrate this behavior.
assumption of ensemble calibration
We use semisythetic setting to conduct comparison to ensembling. To do so, we create sets of three classifiers based on the widely-used Adult dataset [7], where the task to predict whether a person's income is above $50K. To create differences between the three classifiers in set, we train them on random fixed-size samples of 50 labeled examples from different portions of the dataset, partitioned based on age. In doing so, our semi-synthetic classifier sets mimic how training data for
33

---

Accuracy
0.1
0.12
Eximaiin Errer
0.10
0.08
0.4
0.14
Dntribtion
Student4
0.12
Laplace
Geussion
Edtimaiion Eiror
0.08
0.04
1000 2000
Number Unlabelled Examples (n.)
Number Cassifiees
Separation betweeen comporents ([]II)
Figure S2: SSME performance in response to additional unlabeled data, additional classifiers, and improved classifiers. As indicated by our theoretical results, SSME benefits from increased amounts of unlabeled data, number of classifiers, and classifier performance.
different real-world classifiers can differ in meaningful ways. We repeat this procedure to produce 500 sets of three classifiers, where sets differ in the training data provided each classifier. Our procedure naturally produces random variation classifier properties, like accuracy and calibration. As with the real datasets, produce three splits: training split learn the classifiers (50 examples), an estimation split for the performance estimation methods (20 labeled examples and 1000 unlabeled examples), and an evaluation split measure ground truth values for each metric (10,000 examples). Each classifier is logistic regression with default L.2 regularization.
We artificially increase the expected calibration error of each classifier using generalized logistic function parameterized by Specifically, we transform classifier score to be s*+(1-s)== effectively increasing overeonfidence for higher and increasing underconfidence for lower As in the semisynthetic experiments, we generate 500 semisynthetic classifier sets, where each classifier in set is trained 100 examples distinct from the training data for other classifiers in the set (results robust to this choice training dataset size). Each set contains three classifiers.
Figure S3 reports our results. As the average calibration among classifiers in a set varies, SSME consistently improves over the use of an ensemble. This aligns with our intuition, and indicates value of using labeled data in conjunction with unlabeled data. Miscalibration has little effect on the ensemble when estimating AUPRC; here, SSME and ensembling perform similarly.
D.11 Comparison of computational cost
Fitting SSME is computationally cheap and faster than the best baseline. For every dataset and experiment configuration reported in the paper, SSME can be fit in under minutes, using only CPU. SSME inherits the computational complexity of kernel density estimators (0(n)) and EM. In matched comparison (using 20 labeled examples, 1000 unlabeled examples, and 7 classifiers) the next best-performing baseline (Bayesian-Calibratton takes on average 32.1 seconds estimate performance, while SSME takes 21.5 seconds.
34

---

AUPRC
// Sample-based estimate of metric Aosoluec dfffrencc between ext estirated ated ground trith mettiNN
SSME
8.0
10.0
12.0
14:0
14.0
8.0
10.0
12.0
14.0
Avorage Cassifier Callb atton
Abcrage Clasafier Callbration
Avarage Classifeer Colibration
Avarage Classifier Colleration
Figure S3: A comparison of SSME ensembling on miscalibrated classifier SSME
consistently produces more accurate performance estimates compared to ensembling the classifiers across differently calibrated classifier sets (x-axis).
Algorithm 2: Sampling
based based metric estimation with SSME
Input: Labeled set D; {(s(1) ) Unlabeled set D,, [s(0) : are
ALR-transformed classifier scores; Fitted model Pa(y s); metric function g(p. y): Number of samples S = 500
Init: sum + 0;
for
f to S do
foreach ((6) E Du do
L g(0) Po(y s());;
// Sample true label from mixture model
D(s) + (l((),(0) UD:
( + (n, -ne) n1e=
Σ
O(ALR
(x.))ED())
sum - sum c
Output: metric = sum/S.
E Method Details
E.1
Additive Log- Log-Ratio Transform
Given vector A K-1 over K classes, let ALR(p)) log PL log P2. ,log px px
px
RK To invert, P; 1 et for < K and Px 1+ e*
The ALR transform maps unit-sum data into real space, where it easier fit mixture models. The inverse allows us map samples from the mixture model in real space back to the simplex A K-1 For details, see [53].
E.2 Metric Estimation
SSME is able to estimate any metric that is function of the classifier probabilities p and label y. We approximate the joint distribution P(y. p) with mixture model model Po s), where s refers to the ALR-trand -ratsrraffored classifier probabilities (i.e. "classifier scores")². We refer to P p) for ease of notation this section; equivalent, through invertible mapping, to P(y.s).
We denote our approximation for P(p. y) as Po(p, v). We provide a few concrete examples of SSME to measure performance metrics, given (p y) and set of unlabeled probabilistic predictions (p() (" and labeled probabilistic predictions {p', ((() 1n,,,,,,, Notationally, P refers to the jth model's probabilistic prediction of the ith unlabeled example.
how
RReaal that ALR is bijection, so use the inverse mapping ALR-1 K- + A K-1 to transform our mixture distribution in real space back probability space.
35

---

Accuracy measures the alignment between a model's (discrete) predictions and the true label To discretize predictions, practitioners typically take the argmax of p()), Using the binary case an illustrative example, the accuracy of the jth model can be written as:
Accuracy, = E [1 ly 1(p > t)]] where is indicator function and is chosen threshold, typically 0.5. our setting, we approximate this as:
1 1(p > o)]
Accuracy, nu nig
For labeled examples, we use the true label yf For unlabeled examples, we draw y() ~ (a(yp ()).
We then compute accuracy using these labels y and predictions p(+). To ensure our estimation procedure is robust to sampling noise, we average our estimated accuracy over 500 separate sampled labels for each example in the unlabeled dataset.
Alternatively, we could directly use (p)))) to estimate accuracy. That is, for each point p(*) directly compute expectation for the label, and sum this over the entire dataset.
Using the binary case an example
1
Ru +
1 = 1(p,)) > b)] (p
Accuracy; ~
E
Tu +
TE
In other words, compute the expectation that the label agrees with the predicted label for each point. This expectation is ((), This expectation is computed over Pe (y/ p) One interpret (oll)) as a "recalibration" step: given a set of classifier guesses p. what the true distribution of y?
In our experiments, we use the first of these two approaches, i.e. we sample the true label from the estimated distribution.
Expected Calibration Error (ECE) measures the alignment between model's predicted probabili- ties Py and the ground truth labels y. In particular, ECE compares the model's reported confidence to the true class likelihoods, averaged over the dataset. We write out our ECE estimation procedure for the binary case, and it extends readily to definitions of calibration in multiclass settings [32]. Binary ECE can be written as:
ECE₁ Ep, [[PTY - 1 pソ) P/
Then, to approximate the ECE with the datasets {p" V and {p' 1 one can sample y()) ~ ()) for each unlabeled sample and then use the standard histogram binning procedure [31] using both the true labels for the labeled dataset and the sampled labels for the unlabeled dataset. In this approach, we treat the sampled labels (() as true labels for unlabeled examples. To ensure our procedure is robust against sampling noise, we draw samples of (() repeatedly for fixed number of draws (500). We then compute ECE separately for each of these 500 draws and average ECE across all draws.
We this first approach, but alternatively, one could also directly use Po(ylp) estimate ECE. In particular, we can write:
ECE
Pa y
11p p0
Thu
TIg
In this approach, we sample the labels y for unlabeled examples but instead directly (y p). which provides us (an estimate of) the true distribution of y. Instead, we directly use our estimate for the conditional label distribution Pa = 1p,,))
In our experiments, we use the first approach described, i.e. sampling ()) for unlabeled examples and then using the standard binning and averaging procedure.
36

---

AUROC and AUPRC can be estimated with a similar procedure as above. In particular, we sample a label y ~ P& (y = 1 lp(()) from the conditional label distribution and compare these sampled labels to the classifier probabilities.
E.3 Alternate parameterizations
One alternative parameterization is to use a normalizing flow to model our mixture of distributions. Normalizing flows learn and apply invertible transform fo random variable D₁ obtain fo(2) ~ D2. Here, we set z ~ Dj to a Gaussian mixture model and learn a transformation such that dist.
fz(z) ~ i.c.. the transformed distribution roughly matches our classifier score distribution. By modeling explicitly as Gaussian mixture model, one can move back and forth between the two distributions, fg (Je (Z)) z, where fo is the inverse of f. Specifically, we the distribution of Z to follow a Gaussian mixture:
ZI = k)) N Ex,
Thus, the marginal distribution of Z is pz (z) Ex= Mizlue Ex) ply k) is the overall density of z. We apply our invertible transformation fe to obtain fo(z). To find p(sly = k), we follow the approach of [35]:
5ff 8x
ps = = NNf (s) Σx) det
ply =)
Intuitively, transform ()) into distribution (z. 3) which follows Gaussian mixture model. By enforcing the constraint that this transform is invertible, the joint distribution on (z, y) captures all the information in (s,y).
We use the RealNVP architecture [22] to parameterize fo using 10 coupling layers, 3 fully- fully-connected layers, and a hidden dimension of 128 between the fully connected layers. Our normalizing flow is lightweight and trains in less than a minute for each dataset our experiments section using 80GB NVIDIA A100 GPU.
Note there are two optimizations here: (1) the normalizing flow transformation fo which maps s into our latent Gaussian mixture space and (2) the Gaussian mixture model parameters μk. Eg themselves. We begin by fixing the GMM parameters μk. Σg values estimated from our classifier scores s and learning only the flow fo for 300 epochs. Afterwards, we optimize the GMM parameters Pa. Eg with EM for another 700 epochs.
37


Reviews:
[{"review_id": "2ZFvhSlAhJ", "reviewer": "Reviewer_Sq1D", "summary": "This paper introduces a method called Semi-Supervised Model Evaluation (SSME) that uses both labeled and unlabeled data to evaluate machine learning classifiers. The key to SSME is to estimate the joint distribution of ground-truth labels and classifier scores using a semi-supervised mixture model. Besides, the authors provide theoretical results to show the upper bounds of estimation error. Experiments in four domains compared with eight baselines show the effectiveness of the proposed method.", "questions": "1. In Algorithm 1, do different initializations affect the estimations of parameters?\n2. Why do you use a different algorithm UL+ in the theoretical analysis?\n3. The error bounds are tightened as the number of classifiers increases, but adding classifiers also brings additional expenses. Is there an optimal number of classifiers when considering the trade-off between performance and computational expenses? \n4. The authors provide results of the robustness of SSME to kernel choice in Table S13. How about the robustness of SSME to bandwidth h?", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written and easy to follow.\n2. The proposed method is straightforward, and the optimization procedure is clear.\n3. This paper presents theoretical results and in-depth analysis, which explains why SSME outperforms baselines.\n\nWeaknesses:\n1. The authors assume that the unlabeled samples are drawn from the same distribution as the labeled samples, which may violate reality.\n2. The authors optimize the parameters using EM over 1000 epochs, which may be time-consuming. The running time or computational complexity should be provided.\n3. Theorem 1 is derived by using the UL+ algorithm, however, the Algorithm 1 uses EM to estimate $P(y,s)$, which is not consistent.", "was_revised": true, "final_justification": "The issue of inconsistency between the theoretical analysis (UL+ algorithm) and the empirical application (EM algorithm) remains unsolved. I cannot strongly approve for this paper, and my recommendation is borderline accept.", "rebuttal": "Thank you for your positive review and recognizing both our theoretical contributions and in-depth empirical analysis. We respond to your questions and suggestions below. \n\n**Q1. Do different initializations affect the estimation of parameters?**\n\nGood question. We find that SSME’s performance remains strong when using an alternative plausible initialization method: using a majority vote among k=5 nearest neighbors among the labeled examples to initialize cluster assignment for unlabeled examples. We compare this alternate initialization method to our original initialization method (which we refer to as \"draw\" in the table below) on the CivilComments dataset with three different amounts of labeled data (20, 50, and 100 points), just as in our original paper. We include our results on ECE and accuracy estimation error in the table below, where we find that the two initialization methods perform comparably on the CivilComments dataset (with KNN slightly beating the original initialization method in 2 of 3 settings, and original beating KNN in the other). Notably, we continue to outperform all the baselines even with the alternative KNN initialization. This is useful practical guidance for implementing SSME, which we will incorporate into the paper.\n\n| Number of labeled examples | Initialization method | MAE, ECE  | MAE, Accuracy |\n| -------------------------- | --------------------- | --------- | ------------- |\n| 20                         | KNN                   | 0.045     | 0.046         |\n| 20                         | draw             | 0.035 | 0.034     |\n| **—**                      | **—**                 | **—**     | **—**         |\n| 50                         |  KNN               | 0.037 | 0.040     |\n| 50                         | draw                  | 0.043     | 0.043         |\n| **—**                      | **—**                 | **—**     | **—**         |\n| 100                        | KNN              | 0.031 | 0.033     |\n| 100                        | draw                  | 0.037     | 0.036         |\n(Dataset: CivilComments) SSME’s performance is robust to choice of initialization. The original initialization method presented in the paper is denoted \"draw.\"\n\n\n**Q2 + W3. Why use a different algorithm (UL+) in the theoretical analysis?**\n\nThank you for this question. We use UL+ for the theoretical analysis as it enables us to build on prior results in semi-supervised learning for mixture models, which use this as a standard assumption. To our knowledge, there are no results establishing analogous performance bounds (i.e., bounds relating the amount of data to parameter estimation error) for EM algorithms. The bound derived from analyzing UL+ is likely conservative, as UL+ first uses the unlabeled data to identify decision boundaries then uses the labeled data to assign regions to classes. In general we expect EM, which directly maximizes the log-likelihood of both the labeled and unlabeled data, to perform even better.\n\n**Q3. Error bounds tighten as the number of classifiers increases; but adding classifiers introduces expenses. Given this tradeoff, is there an optimal number of classifiers?**\n\nThanks for raising this question! Empirically, we find that the computational expense of an additional classifier is very small. Concretely, fitting SSME with nine classifiers takes 17.4 seconds; in comparison, fitting SSME with one classifier takes 14.3 seconds. We provide theoretical conditions under which an additional classifier provides benefit in Section C.1: specifically if the separation between components exceeds $\\sqrt{d+1}$, where $d$ represents the dimensionality, any additional classifier whose accuracy is better than random will tighten the bound on performance estimation error.\n\n**Q4. Robustness of SSME to bandwidth h?**\n\nThanks for this question. We ran an experiment to test the robustness of our results to the choice of bandwidth. In the paper, we use the Sheather-Jones algorithm [1] to automatically identify a bandwidth. We compare this approach to (1) another automated bandwidth selection algorithm (the Silverman algorithm [2]) and (2) two fixed bandwidths, chosen to be larger but within an order of magnitude of the Sheather-Jones selected bandwidth. We conduct experiments using the CivilComments dataset and 3 labeled dataset sizes (20, 50, and 100 labeled datapoints). While results remain strong in all cases, we achieve the best performance when using the Sheather-Jones bandwidth selection procedure, as is done in the original manuscript. We will incorporate these results into the manuscript. \n\n| Number of labeled examples | Bandwidth selection procedure | MAE, ECE  | MAE, Accuracy |\n| -------------------------- | ----------------------------- | --------- | ------------- |\n| 20                         | **Sheather-Jones**            | **0.024** | **0.023**     |\n| 20                         | Silverman                     | 0.040     | 0.038         |\n| 20                         | 1.0                           | 0.037     | 0.036         |\n| 20                         | 2.0                           | 0.046     | 0.044         |\n| **—**                      | **—**                         | **—**     | **—**         |\n| 50                         | **Sheather-Jones**            | **0.023** | **0.022**     |\n| 50                         | Silverman                     | 0.066     | 0.065         |\n| 50                         | 1.0                           | 0.062     | 0.062         |\n| 50                         | 2.0                           | 0.045     | 0.044         |\n| **—**                      | **—**                         | **—**     | **—**         |\n| 100                        | **Sheather-Jones**            | **0.022** | **0.021**     |\n| 100                        | Silverman                     | 0.049     | 0.047         |\n| 100                        | 1.0                           | 0.056     | 0.055         |\n| 100                        | 2.0                           | 0.034     | 0.032         |\n(Dataset: CivilComments) Sheather-Jones outperforms another automated selection method and two pre-determined bandwidths. \n\n[1] https://www.stat.cmu.edu/~rnugent/PCMI2016/papers/SurveyBandwidthSelectionWand.pdf\n\n[2] https://sites.stat.washington.edu/wxs/Stat593-s03/Literature/silverman-81a.pdf \n\n**W1: Assuming that unlabeled samples are drawn from the same distribution as the labeled samples may violate reality.**\n\nThanks for this point, which we agree is important. We have added an experiment to explore how SSME performs under violations of this assumption. Specifically, we test SSME’s performance when the unlabeled data for CivilComments is drawn from the unlabeled split provided by the WILDS benchmark [1], which is specifically designed to reflect real-world distribution shifts. SSME achieves an average estimation error (in MAE, scaled to same 0-100 scale as the paper) of 2.12 (averaging across 20, 50, and 100 labeled datapoints), compared to 4.15 for labeled data alone and 4.44 and 4.56 for the two best-performing baselines (Bayesian-Calibration and Dawid-Skene). This demonstrates that SSME still provides an advantage over baselines even in the presence of real-world distribution shifts.\n\nFurthermore, we believe the assumption that the unlabeled data is drawn from the same distribution as the labeled data is realistic in several important settings. First, the assumption is met if the annotator/practitioner chooses samples to label at random. For instance, if the labeling budget is small, a practitioner may select a random subset to label. There are also settings where this assumption approximately holds. For instance, in radiology, if one considers the “labeled” images to be all images that have been labeled in the past (e.g., for a particular hospital) and the “unlabeled” images to be the images a radiologist has yet to look at, there is no reason to expect a distribution shift. \n\n[1] https://arxiv.org/abs/2112.05090\n\n**W2. Providing running time and computational complexity.**\n\nWe will include a discussion of computational cost to accompany our empirical results; thank you for the suggestion! Specifically, we will include the following. Fitting SSME is computationally cheap and faster than the best baseline. For every dataset and experiment configuration reported in the paper, SSME can be fit in under 5 minutes, using only 1 CPU. SSME inherits the computational complexity of kernel density estimators (O(n^2)) and EM. In a matched comparison (using 20 labeled examples, 1000 unlabeled examples, and 7 classifiers) the next best-performing baseline (Bayesian-Calibration) takes on average 32.1 seconds to estimate performance, while SSME takes 21.5 seconds. \n\n**Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?**"}, {"review_id": "vOwdfivRPs", "reviewer": "Reviewer_Nk8N", "summary": "### Summary\n\nThis paper proposes a method called Semi-Supervised Model Evaluation (SSME) for evaluating the performance of multiple machine learning classifiers in scenarios where large amounts of labeled data are not available.\n\nSSME utilizes a small amount of labeled data, a large amount of unlabeled data, and the probabilistic prediction scores of multiple classifiers to estimate performance metrics such as accuracy, AUC, and ECE. By estimating the joint distribution \n𝑃\n(\n𝑦\n,\n𝑠\n)\n of true labels and classifier scores using a semi-supervised mixture model, it enables performance evaluation even for the unlabeled data under the constraint of limited labeled instances.\n\n### Contributions\n\nProposal of SSME:\nIntroduces an evaluation framework that jointly leverages labeled data, unlabeled data, and continuous prediction scores from multiple classifiers. It estimates the joint distribution \n𝑃\n(\n𝑦\n,\n𝑠\n)\n using a mixture model, enabling the calculation of various performance metrics.\n\nTheoretical guarantees:\nProvides theoretical upper bounds on the estimation error and shows that the error decreases as the number of classifiers or the amount of unlabeled data increases.\n\nEmpirical validation:\nDemonstrates effectiveness across diverse domains such as healthcare, social media moderation, molecular property prediction, and natural language processing. Compared to using labeled data only, SSME reduces estimation error by up to 5.1 times on average, and by 2.4 times compared to the second-best method.\n\nPractical applicability:\nApplies the method to real-world scenarios including classifiers based on large language models and subgroup-level performance evaluation (e.g., fairness), confirming its utility.", "questions": "The ideas behind SSME and the proposed methodology in this paper are intriguing. To better understand the contributions of this work, I would like to ask the following questions:\n\nThe paper presents experiments across five tasks, with classifier sets selected from existing benchmarks. However, I believe the choice of classifiers can significantly influence the performance of the proposed method. Do the authors consider the selection of classifiers to be a practical issue that is adequately addressed in the paper, or do they believe it is straightforward enough not to pose a concern?\n\nAs the authors themselves note, the proposed method assumes that labeled and unlabeled data are drawn from the same distribution. In practice, this assumption may often be violated. Do the authors believe that such distributional mismatch would not pose a serious issue in real-world applications? Alternatively, do they see this as a limitation that could be addressed through future methodological improvements?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* **Utilization of Rich Data Sources**: SSME leverages three sources of information simultaneously—limited labeled data, abundant unlabeled data, and multiple machine learning classifiers—which conventional evaluation methods struggle to use effectively. This integrated approach significantly improves estimation accuracy in classifier evaluation.\n\n* **High Accuracy in Performance Estimation**: SSME estimates classifier performance more precisely than competing methods. For example, it reduces estimation error by a factor of 5.1 compared to using labeled data alone, and by 2.4 times compared to the second-best method. Notably, in estimating Expected Calibration Error (ECE), it achieves an average error reduction of 7.2 times.\n\n* **Wide Applicability and Versatility**: SSME supports the estimation of various performance metrics such as Accuracy, AUC, AUPRC, and ECE. It is applicable to any number of classifiers and classes. The method has demonstrated its utility across diverse domains including healthcare, content moderation, molecular property prediction, and text classification. It also performs well in evaluating classifiers derived from large language models (LLMs) and in subgroup-specific performance evaluations.\n\n**Weaknesses**\n\n* **Simplifying Assumptions in Theoretical Analysis**: The theoretical results in the paper are based on stylized settings, such as assuming classifier scores are generated from Gaussian mixture models. While empirical evidence supports the trends predicted by the theory, its direct applicability to real-world scenarios may be limited if actual data deviates significantly from these assumptions.\n\n* **Assumption of Identical Data Distributions**: The experiments assume that unlabeled data are sampled from the same distribution as labeled data. However, in real-world settings, labeled and unlabeled data may systematically differ (e.g., covariate shift), which may limit the direct applicability of the current version of SSME.\n\n* **Performance Variability in Specific Metrics or Scenarios**: Although SSME performs well overall, the degree of error reduction is not as pronounced for some metrics (e.g., AUPRC) compared to others like ECE. Additionally, in subgroup analysis of AUC estimation, baseline methods such as Bayesian Calibration (BC) may outperform SSME under certain assumptions, such as when classifiers exhibit high AUC and monotonicity is assumed.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, the questions and concerns I had as a reviewer have been resolved, so I will improve my evaluation by one rank.", "rebuttal": "Thank you for your comments! We appreciate that you believe SSME is able to utilize multiple sources of information simultaneously, has wide applicability, and outperforms baselines in estimating performance.\n\nWe address your questions and respond to the highlighted weaknesses below.\n\n**Q1. Do the authors consider the selection of classifiers to be a practical issue that is adequately addressed in the paper, or do they believe it is straightforward enough not to pose a concern?**\n\nThanks for this question. To address it, we ran an experiment where we measured SSME’s performance as the classifier set expands and the mix of classifiers within that set changes. In particular, for each number of classifiers between 2 and 7 (inclusive), we use SSME to estimate performance for each unique set of classifiers (so for 7 classifiers, there is exactly one set; for 4 classifiers, there are 7 choose 4 = 35 sets). We performed this experiment on the CivilComments dataset.\n\nAs shown in the table below, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set.\n\nWe also find that SSME's performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifier sets. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015, indicating consistent performance across classifier sets.\n\nFirst, we find that performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifiers. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015. \n\nSecond, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set. \n\n| Number of classifiers |            MAE, ECE (SD) |       MAE, Accuracy (SD) |\n| --------------------: | -----------------------: | -----------------------: |\n|                     2 |          0.0457 (0.0059) |          0.0471 (0.0047) |\n|                     3 |          0.0302 (0.0107) |          0.0308 (0.0098) |\n|                     4 |          0.0268 (0.0094) |          0.0273 (0.0096) |\n|                     5 |          0.0252 (0.0072) |          0.0253 (0.0078) |\n|                     6 |          0.0235 (0.0043) |          0.0235 (0.0048) |\n|                     7 | 0.0225 (only one subset) | 0.0223 (only one subset) |\n\n(Dataset: CivilComments) MAE for ECE and accuracy across classifier set choice, along with the standard deviation in MAE across classifier sets.\n\n**Q2 + W2 On the assumption that labeled and unlabeled data are IID: do the authors believe that such distributional mismatch would not pose a serious issue in real-world applications? Alternatively, do they see this as a limitation that could be addressed through future methodological improvements?**\n\nThanks for this question! We performed an experiment to assess SSME’s performance under violations of this assumption and found that SSME continues to outperform baselines at small amounts of labeled data. The experiment draws the unlabeled data for CivilComments from the unlabeled data provided by WILDS, which is specifically designed to exhibit realistic distribution shifts. We report the average error in accuracy estimation for each method. SSME achieves an average estimation error (in MAE, scaled to same 0-100 scale as the paper) of 2.12 (averaging across 20, 50, and 100 labeled datapoints), compared to 4.15 for labeled data alone and 4.44 and 4.56 for the two best-performing baselines (Bayesian-Calibration and Dawid-Skene). This demonstrates that SSME still provides an advantage over baselines even in the presence of real-world distribution shifts.\n\nFurthermore, we believe the assumption that the unlabeled data is drawn from the same distribution as the labeled data is realistic in several important settings. The assumption is met if the annotator/practitioner chooses samples to label at random. For instance, if the labeling budget is small, a practitioner may select a random subset to label. There are also settings where this assumption approximately holds. For instance, in radiology, if one considers the “labeled” images to be all images that have been labeled in the past (e.g., for a particular hospital) and the “unlabeled” images to be the images a radiologist has yet to look at, there is no reason to expect a distribution shift. \n\n**W1. Simplifying assumptions in theoretical analysis.**\n\nThank you for this question. We use UL+ for the theoretical analysis as it enables us to build on prior results in semi-supervised learning for mixture models, which use this as a standard assumption. To our knowledge, there are no results establishing analogous performance bounds (i.e., bounds relating the amount of data to parameter estimation error) for EM algorithms. The bound derived from analyzing UL+ is likely conservative, as UL+ first uses the unlabeled data to identify decision boundaries then uses the labeled data to assign regions to classes. In general we expect EM, which directly maximizes the log-likelihood of both the labeled and unlabeled data, to perform even better.\n\n**W3. Performance variability in specific metrics and scenarios**\n\nThe reviewer points out that, although SSME performs well overall, it reduces the error more for some metrics than for others. The reviewer is correct that SSME’s performance is strong, outperforming baselines across all metrics and tasks. In particular, SSME outperforms the next-best baseline, Bayesian-Calibration, in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation) averaged across tasks. \n\nWe agree that the performance gains are not uniform across metrics. This behavior is consistent with our theory, which shows that error bounds on AUC are looser than those on accuracy (because the error bound on estimated AUC scales differently with separation, see the $\\sqrt{2}$ denominator instead of 2 in Equations 3 and 4). \n\nFurthermore, the reviewer points out that under certain assumptions, Bayesian-Calibration might outperform SSME. We agree that Bayesian-Calibration may outperform all other methods if its strong assumptions on the classifier set (monotonic relationship between classifier scores and Pr(Y=1)) exactly hold. However, as our empirical results demonstrate, in real-world settings the assumptions Bayesian-Calibration requires do not hold well enough for it to outperform SSME, which achieves better performance for every metric.  Additionally, Bayesian-Calibration is restricted to settings with binary classifiers, a significant limitation that SSME addresses.\n\n\n**Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?**"}, {"review_id": "rR6SD4VKym", "reviewer": "Reviewer_HJAd", "summary": "The paper introduces Semi-Supervised Model Evaluation (SSME), a simple EM-based mixture model that learns the joint distribution of true labels and the score vectors of multiple classifiers from a few labeled examples plus abundant unlabeled data; once fitted, this density lets can be used to infer any performance metric (accuracy, AUC, calibration, etc.) on the unlabeled pool. The authors prove that the estimation error shrinks with more unlabeled samples and more diverse classifiers, then show across five real-world domains (including LLM-based text classifiers) that SSME cuts metric-estimation error by about five-fold compared with using labels alone.", "questions": "1. It seems that currently, this method is mainly applied for Natural Language classification tasks, how will it performs on image classification tasks?\n\n2. How will the diversity and the coverage of the classifiers being tested impact the estimation accuracy? Intuitively, more classifiers you considered, the more unbias your joint distribution $P(Y,s)$ should be, I encourage authors to conduct a experiments to test this.", "limitations": "N.A.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper considers a vital yet relatively overlooked problem - model selection under weak supervision.\n\n2. The proposed method seems new and interesting - we are leveraging the fact that we have multiple models to evaluate, and we are using their predictions to predict pseudo labels for evaluation purposes.\n\n3. The experiments contain a diverse setting spanning from logistic regression to language models.\n\n**Weaknesses:**\n\n1. The baselines being considered in this paper are incomplete and outdated, and many relevant works in the domain of unsupervised accuracy estimation are not compared (while some are discussed), to name some examples: [1, 2, 3, 4]. The topic being proposed here has stronger assumptions than unsupervised accuracy estimation; if not compared empirically, then this makes the technique proposed in this paper not very convincing. \n\n2. This method only works for classification tasks, which, to an extent, limits the significance and contribution of the proposed method.\n\n3. The proposed method is consistently worse than some baselines, such as Bayesian Calibration. \n\n[1] Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation, ICML 2023. \n\n[2] Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samples, ICCV 2023. \n\n[3] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density, ICCV 2021.\n\n[4] Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data, KDD 2023.", "was_revised": true, "final_justification": "My major concerns before rebuttal is the fact that recent and relevant baselines are not compared or discussed, the authors provided somewhat convincing arguments in their rebuttal - their focus in performance estimation and focus beyond model ranking, some of the baselines I mentioned can only achieve the latter. Therefore, I believe there is no significant reason for me to oppose the acceptance of this paper.", "rebuttal": "Thank you for your comments! We appreciate that you recognize that SSME is able to utilize multiple sources of information simultaneously, has wide applicability, and outperforms baselines in estimating performance.\n\nWe address your questions and respond to the highlighted weaknesses below.\n\n**Q1. It seems that currently, this method is mainly applied for Natural Language classification tasks, how will it performs on image classification tasks?**\n\nThanks for this question! SSME outperforms baselines across three settings we test: natural language data, tabular data, and graph data. In response to your question, we have added an experiment where we analyze SSME’s performance on image data in the context of CIFAR-3 [1, 2, 3] (i.e., a dataset with three classes from CIFAR-10), where classifiers are models pre-trained on ImageNet and finetuned on CIFAR. We found SSME outperforms baselines that can be applied in this setting, achieving an mean absolute error in accuracy estimation of 4.50 (SD: 2.12), while the next best baseline (labeled data alone) achieves a mean absolute error of 5.33 (SD: 2.77). We find similar results for ECE estimation, where SSME achieves a mean absolute error in ECE estimation of 3.75 (SD: 1.21), while the next best baseline (pseudo-labeling) achieves a mean absolute error of 6.36 (SD: 4.78).\n\n[1] https://ewsn2022.pro2future.at/paper/sessions/ewsn2022-final3.pdf \n\n[2] https://kclpure.kcl.ac.uk/ws/portalfiles/portal/332763034/2501.06066v3.pdf\n\n[3] https://proceedings.mlr.press/v189/pandey23a/pandey23a.pdf\n\n**Q2. How will the diversity and the coverage of the classifiers being tested impact the estimation accuracy? Intuitively, more classifiers you considered, the more unbias your joint distribution  should be, I encourage authors to conduct a experiments to test this.**\n\nThanks for this question. We ran an experiment to measure the benefit of each additional classifier, as well as how much performance varies across classifier sets, and include the results below. In particular, for each number of classifiers between 2 and 7 (inclusive), we use SSME to estimate performance for each unique set of classifiers (so for 7 classifiers, there is exactly one set; for 4 classifiers, there are 7 choose 4 = 35 sets). We performed this experiment on the CivilComments dataset.\n\nYour intuition is correct: additional classifiers do produce better estimates of performance, with diminishing returns. As shown in the table below, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set. \n\nWe also find that SSME's performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifier sets. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015, indicating consistent performance across classifier sets.\n\n| Number of classifiers |            MAE, ECE (SD) |       MAE, Accuracy (SD) |\n| --------------------: | -----------------------: | -----------------------: |\n|                     2 |          0.0457 (0.0059) |          0.0471 (0.0047) |\n|                     3 |          0.0302 (0.0107) |          0.0308 (0.0098) |\n|                     4 |          0.0268 (0.0094) |          0.0273 (0.0096) |\n|                     5 |          0.0252 (0.0072) |          0.0253 (0.0078) |\n|                     6 |          0.0235 (0.0043) |          0.0235 (0.0048) |\n|                     7 | 0.0225 (only one subset) | 0.0223 (only one subset) |\n\n(Dataset: CivilComments) MAE for ECE and accuracy across classifier set choice, along with the standard deviation in MAE across classifier sets.\n\n**W1. Comparison to additional methods**\n\nThanks for your comment! In the original manuscript, we compare to and outperform eight state-of-the-art baselines designed for semi-supervised evaluation. We agree that unsupervised accuracy estimation methods are related, and Appendix A describes connections to past work in this literature. However, these baselines cannot be directly applied to estimate the metrics we wish to estimate. For example: three of the baselines you mention ([1], [3], and [4]) discuss heuristics that correlate well with model accuracy, but it is not clear how to map these heuristics to absolute estimates of other metrics like ECE, AUC, and AUPRC, as SSME does. It is also unclear how to extend the ideas in the other baseline you mention [2], which fits a custom pseudo-labeling function to image data specifically, to our setting, where it is unclear what such a pseudo-labeling function should be for a generic dataset. We agree the connection is interesting and believe it merits future work, and will include the citations you provided in Appendix A; thank you for bringing them up!\n\n[1] Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation, ICML 2023.\n\n[2] Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samples, ICCV 2023.\n\n[3] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density, ICCV 2021.\n\n[4] Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data, KDD 2023.\n\n**W2. Limitation to classification**\nWhile our method primarily focuses on semi-supervised performance estimation in classification, we note that we can very naturally extend our approach to incorporate regression settings as well. The key idea behind our method is to estimate a joint distribution for P(s, y) with a semi-supervised model. In classification settings, it is natural to model this using a mixture model. In a regression case with a continuous y, however, there are other joint density estimators for P(s,y) that could be reasonable, including KDEs or autoencoders. Thanks for flagging this, which we agree is a natural extension of our method and an interesting direction for future work!\n\n**W3. Proposed method is consistently worse than Bayesian-Calibration**\nThank you for this comment. To clarify, our results show that SSME decisively outperforms Bayesian-Calibration across the diverse datasets that we study. In particular, our results show that SSME outperforms Bayesian-Calibration in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation), averaging across datasets and tasks. Our tables in the appendix provide further details substantiating this result. Bayesian-Calibration also does not generalize to settings with more than two classes, an important limitation SSME overcomes. \n\n**Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?**"}, {"review_id": "hDekfLC2wU", "reviewer": "Reviewer_4KCW", "summary": "This paper introduces Semi-Supervised Model Evaluation (SSME), a method that leverages both labelled and unlabeled data to improve evaluation accuracy. SSME uses a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier scores, reducing error through the combination of multiple classifiers and probabilistic scores. Authors perform experiments across diverse domains to demonstrate that SSME provides more accurate estimates than existing methods, reducing error by 5.1x relative to labelled data and 2.4x relative to the next best method.", "questions": "How does the author’s contribution improve the state of the art?\nHow do authors ensure the lack of data leaks in the datasets they used?", "limitations": "Limitations are poorly discussed in the manuscript. I suggest outlining a specific section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\nThe authors propose an approach for an extremely poignant problem in ML and Data Science - the lack of well-annotated datasets. Even though new datasets appear constantly, there's arguably never enough data. Therefore searching for alternative approaches is important. The manuscript seems to be sound theoretically.\n\nWeaknesses:\n\n1) By their own admission in the checklist, the concept of the authors does not outperform the benchmarks. The advantage of the contribution approach has to be made clear.\n\n2) Overall, the concept seems to borrow from the fields of semi-supervised learning and active learning. However, the existing concepts have not been sufficiently discussed in the related work section.\n\n3) The writing of the manuscript and the appendix could be significantly improved: it is impossible to verify the sources and adequacy of the tasks. \n\n4) Furthermore, the supplement does not include the source code, despite the statement in the checklist. Lack of source code and clear references to the datasets limits my ability to review this work.", "was_revised": true, "final_justification": "I discussed the paper with the authors. However, while some issues were resolved, there are some limitations of the work that the authors won't be able to further address here. These limitations include poor scalability to high-dimensional data. This would be important to tackle in this day and age. Given my generally high initial score and in the face of hard limitations of the proposed methodology, I will retain my fairly high score for this work.", "rebuttal": "Thank you for your comments and the positive review! We appreciate that you recognize the importance of semi-supervised evaluation and believe our method is theoretically sound.\n\nWe address your questions and respond to the highlighted weaknesses below.\n\n**Q1. How does the author’s contribution improve the state of the art? How do authors ensure the lack of data leaks in the datasets they used?**\n\nOur approach improves upon the state-of-the-art in semi-supervised performance estimation by framing the problem as semi-supervised mixture model estimation; we show empirically that our method outperforms eight baselines across a wide range of tasks and metrics. We ensure there is no dataset leakage by using distinct sets of data to train the classifiers (the training split), to estimate performance (the estimation split), and to measure ground truth performance (the evaluation split). We detail this procedure in Section B.1 and will better incorporate this information into the main text.\n\n**Q2. Adding a limitations section**\n\nThank you for this suggestion. We will add a dedicated limitations section to clearly discuss the scope and potential constraints of our approach, including the difficulty of estimating KDEs in high dimensions and the assumption that the labeled and unlabeled data are drawn from the same distribution. \n\n**Q3. Paper organization: moving the related work to be right after the problem setting. Addressing missing bibliography for the appendix.**\n\nThank you for the suggestion and for flagging the reference issue in the appendix. We will shift the related work to appear after the introduction. The appendix references map to the main text bibliography (we manually split the files to submit the supplementary material). We will ensure that the references properly accompany the appendix. \n\n\n**W1. Outperforming baselines**\n\nWe’d like to clarify: SSME consistently outperforms the eight baselines we compare to across metrics. In particular, SSME outperforms the next-best baseline in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation) averaged across tasks. We will clarify our statement in the checklist, which was intended to transparently state that we discuss limitations of our work and settings where baselines could outperform SSME (e.g. at Line 310 we discuss settings that would favor Bayesian-Calibration). Overall, however, we decisively outperform all baselines we compare to. \n\n**W2. Clarifying relationship to work in semi-supervised learning and active learning**\n\nThanks for raising this! Our work indeed builds off prior work in semi-supervised learning and, to a lesser extent, active learning. All the baselines we compare to use either semi-supervised (Bayesian-Calibration, AutoEval, SPE, Dawid-Skene) or active learning (Active-Testing) to derive their performance estimates. Unlike prior methods, we are able to simultaneously incorporate continuous scores, labeled and unlabeled data, and multiple classifiers, whereas previous works only use at most two of these three data sources. We will further clarify this point in the related work.\n\n\n**W3. Improve writing to make clear the sources of the tasks; include references to datasets.**\n\nThanks for the suggestion; while we do include references for each dataset in Section 4.1, we will make the connections more explicit. Our code, once public, will also support downloading and processing each of the five public datasets used in our experiments. \n\n**W4. Inclusion of source code**\n\nOur apologies, we misunderstood the checklist to mean the code must be made public upon publication. We will make the code available upon publication, including all code required to reproduce experimental results on five public datasets. Unfortunately, NeurIPS reviewing guidelines do not allow anonymized code URLs during the rebuttal process, so we cannot provide the code right now. The codebase will also support practitioners who wish to apply SSME to new settings. \n\n**Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?**"}]

Author rebuttal:

Meta-review: This paper introduces SSME, a semi-supervised method for performance estimation without labeled data, based on modeling the joint distribution of predictions and labels using KDE and classifier mixtures.

Reviewers generally acknowledge the novelty and practical relevance of the problem setting and experimental validation. Several concerns were raised, including limited coverage of related work, the gap between theory and implementation, and scalability in high-dimensional settings. The authors provided a detailed rebuttal, and some reviewers acknowledged these responses and updated their scores accordingly. Some remained concerned about clarity, especially in the writing and appendix.

While certain aspects could benefit from further refinement, I find the paper addresses an important and under-explored problem with a well-motivated approach and strong empirical results. I recommend acceptance.
Decision: Accept (poster)