Evaluating multiple models using labeled and unlabeled data

Authors: Divya M Shanmugam, Shuvom Sadhuka, Manish Raghavan, John Guttag, Bonnie Berger, Emma Pierson
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: classifier evaluation, semi-supervised learning, unlabeled data

Abstract:
It remains difficult to evaluate machine learning classifiers in the absence of a large, labeled dataset. While labeled data can be prohibitively expensive or impossible to obtain, unlabeled data is plentiful.
Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. SSME is the first evaluation method to take advantage of the fact that: (i) there are frequently multiple classifiers for the same task,  (ii) continuous classifier scores are often available for all classes, and (iii) unlabeled data is often far more plentiful than labeled data. 
The key idea is to use a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier predictions.
We can then use this model to estimate any metric that is a function of classifier scores and ground truth labels (e.g., accuracy or expected calibration error). 
We present experiments in four domains where obtaining large labeled datasets is often impractical: (1) healthcare, (2) content moderation, (3) molecular property prediction, and (4) image annotation. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1x relative to using labeled data alone and 2.4x relative to the next best competing method. SSME also improves accuracy when evaluating performance across subsets of the test distribution (e.g., specific demographic subgroups) and when evaluating the performance of large language models.

Decision: reject
Reviewer scores: 8, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730219634104

thread_type:
official_review

<thread_object>
creator: reviewer_w43e

<content>
summary:
The paper studies a practical setting which is to evaluate the performance of $M$ classifiers pre-trained targetting the same tasks on two subsets, one with labels and the other without labels. The solution proposed in the paper is to employ semi-supervised learning following a generative modelling through a mixture model to estimate the joint probability of the ground truth labels of unlabelled data and the predictions made by those $M$ classifiers. The ground truth is them inferred from such a modelling to evaluate the performance. The empirical experiment presented in the paper covers a number of datasets.

strengths:
The paper introduces a setting often encountered in practice: evaluating performance of $M$ classifiers pre-trained for a task on a new dataset. In addition, that dataset has a small portion labelled, while the remaining is unlabelled. The paper is well-written to explain the idea at high-level.

weaknesses:
Despite the easy-to-understand being of the paper at high level, the paper poorly explains how the main idea can be done in technical details (including the text in section 4 and the one in Appendix C). The explanation is too vague for me to understand how the paper is doing to achieve its goal.

According to the problem setting in section 3, the problem is equivalent to infer the ground truth labels on unlabelled data using the labelled data and some additional pre-trained classifiers. If one knows the distribution of the ground truth for each sample, it is easier to evaluate the performance of those pre-trained classifiers. Stating this equivalence would significantly improve the clarity of the paper.

The main idea of the paper is to follow the generative approach in semi-supervised learning to find the joint probability between label $y$ and another variable (in the paper, it is the concatenated predictions of all the classifiers, denoted as $\mathbf{s}$). The equation at line 167 can be rewritten in a more understandable form as follows:

$$
\max_{\theta} \ln \Pr(S, Y, S^{\prime}; \theta) =  \max \ln \prod_{i = 1}^{n_{l}} \Pr(\mathbf{s}\_{i}, y\_{i}; \theta) \prod_{j = 1}^{n\_{u}} \Pr(\mathbf{s}\_{j}; \theta) = \max \sum_{i = 1}^{n\_{l}} \ln \Pr_{\theta}(\mathbf{s}\_{i}, | y\_{i}) \Pr(y\_{i}) + \sum_{j = 1}^{n\_{u}} \ln \sum_{k = 1}^{K} \Pr_{\theta}(\mathbf{s}\_{j} | y\_{j} = k) \Pr(y\_{j} = k).
$$

Here, it is not explained why the vector concatenated predictions of all the classifiers, $\mathbf{s}$, is used as one of the variables, instead of the raw input data $\mathbf{x}$ as in conventional semi-supervised learning. This may lead to a bad estimation if most of the classifiers are bad. My guess is that the setting does not allow to access to the raw data $\mathbf{x}$. However, in section 3, the raw input data $\mathbf{x}$ is accessible.

Another concern is how to know the label distribution $\Pr(y\_{j})$ of the unlabelled data (the very last term in the equation above). In the text, the authors only explain how to calculate $\Pr(\mathbf{s} | y)$ at line 181, without specifying how to calculate $\Pr(y\_{j})$ of the unlabelled data term. This distribution must be known to estimate $\Pr(y | \mathbf{s}$ from the learnt $\Pr(\mathbf{s}, y)$, which is claimed by the paper, but is never explained how to perform such inference.

questions:
My concern is the main method proposed to infer the ground truth labels of unlabelled data. Could the authors clarify how such ground truth labels are inferred in more details? The current explanation is too vague to understand.

In addition, could the authors clarify why using $\mathbf{s}$ instead of $\mathbf{x}$ in the formation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730511835066

thread_type:
official_review

<thread_object>
creator: reviewer_sycl

<content>
summary:
This paper introduces **Semi-Supervised Model Evaluation (SSME)**, a method that leverages both labeled and unlabeled data to evaluate multiple machine learning classifiers. SSME is built on the premise that unlabeled data is often more abundant than labeled data,  addressing scenarios where multiple classifiers are available but labeled data is scarce. Using a semi-supervised mixture model, SSME estimates the joint distribution of ground truth labels and classifier scores, which allows it to assess metrics such as accuracy and expected calibration error  without requiring extensive labeled datasets. Through experiments in healthcare, content moderation, molecular property prediction, and image annotation, SSME is shown to reduce estimation error in the metrics compared to baselines.

strengths:
1. The paper presents a novel approach to evaluate classifiers by leveraging unlabeled data alongside limited labeled data, addressing a common bottleneck in evaluating models. The method is simple and the paper is well written overall.

2. SSME’s performance is evaluated across multiple domains and tasks, such as healthcare and content moderation.  The results indicate improvements over baselines in estimating standard metrics, providing evidence of the method's applicability.

3.  The study also shows methods ability to evaluate subgroup-specific metrics, which is beneficial for fairness assessments. This application is particularly relevant for sensitive domains like healthcare, where performance disparities among demographic groups can be critical.

weaknesses:
1.  There is no theoretical analysis provided,  limiting our understanding of when and why the method may succeed or struggle in different data distributions or model configurations e.g. high or low accuracy models.

2.  The main empirical results in Figures 2, 3 and Table 1, report absolute error in estimates. I'd like to see the actual estimates, I believe if the models are highly accurate then it would be easier to infer the groundtruth label and hence the estimates are expected to be better but for models with low accuracy, or models with less correlation the observations could be different. 

3. There are several baselines that have not been considered for evaluation. Some of them are based on weak supervision and active testing. You can use the same amount labeling budget as $n_l$ to run active testing and use $n_l$ points to estimate source quality or write labeling functions in weak supervision pipelines. See some of the references below,

https://arxiv.org/pdf/1711.10160

https://arxiv.org/abs/2107.00643

https://arxiv.org/abs/2002.11955

https://arxiv.org/pdf/2211.13375

https://arxiv.org/abs/2103.05331

questions:
See the weakness above. I have a few more questions,

1. The estimated joint model is specific to the number of classifiers. So if one comes up with a new classifier how is that going to be evaluated? This modeling choice does not seem flexible to me.

2. The assumptions on the classification models are not clear. In particular, how accurate should they be and what's the correlation between them?

3. How will this approach work if the user gives classifiers one at a time, i.e. you don't see all the classifiers ahead of the time but instead they arrive sequentially. For instance, during model training we iteratively improve a single classifier and even during the developement cycle we maintain and update a single model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598300346

thread_type:
official_review

<thread_object>
creator: reviewer_hobm

<content>
summary:
The paper provides a semi supervised method for evaluation of a set of classifiers. Given a small amount of labeled data, a large amount of unlabeled data on which a set of classifiers can provide scores on any input, estimate accuracies of these classifiers. In particular for unlabeled data $x^1, \dots x^n$ and classifiers $1, \dots m$ , we have scores $s^1, \dots s^n$ where $s^i$ is the set of scores for $x^i$, $s^i = \{s_1^i, \dots s_m^i\}$. For classifier $j$,  $s_j^i$ is the score vector in $\Delta^K$ where $\Delta^K$ is the $K$ dimensional simplex, and $K$ is the number of classes.

Their method comprises on fitting a mixture model that estimates true class probabilities given classifier scores, i.e. learn $P(y | s_1, \dots s_m)$. The mixture model is fit by minimizing a joint distribution over scores and true class probabilities parameterized as 
$$P(y, s) = \lambda_L \sum_{i\in D_L} \log P(s^i|y^i) + \sum_{i\in D_U} \log \left(\sum_{k=1}^K P(s^i|y^i = k)P(y^i = k)\right)$$
where $P(s^i|y^i = k)$ is parameterized as 
$$P(s | y = k) = \frac{1}{\sum_i P(y^i=k|s^i)} \sum_i \mathcal{K}_h(s - s^i) P(y^i = k |s^i)$$

An EM algorithm is used to estimate $P(y^i = k |s^i)$. In the E step given, the mixture component each data point belongs to, i.e. $P(y^i = k |s^i)$ is estimated by maximizing the objective keeping $\mathcal{K}_h$ fixed, and in the M step, $\mathcal{K}_h$ is optimized keeping $P(y^i = k |s^i)$. 

Finally given the fitted $P(y^i | s^i)$, any metric $m$ such as classification accuracy is estimated as  $1/n \sum_{i=1}^n E_{y^i \sim P(y^i | s^i)}[m(s_j^i , y_i)]$.

To evaluate their method, the authors use a 6 publicly available datasets. Their evaluation procedure involves first getting a set of classifiers by training on a split of the data, then estimating $P(y^i | s^i)$ on another estimation split and finally comparing the estimated metric against the true metric from a larger evaluation split of the data. 

The authors show that their method performs superior to a number of baselines including labeled, Pseudo labeled, David-Skene, and Bayesian Calibration. Further they provide some ablations on their method 1) when only one instead of many classifier is available, and 2) using a normalization flow based model instead of Kernel Density estimation for the mixture model.

strengths:
1. The presentation of the estimation method given a set of classifiers was clear and easy to understand. 

2. The method of fitting a mixture model is sound however not novel.

3. The results are superior compared to baselines. 

4. Adequate ablations were performed such as using just one classifier instead of many, providing insights on estimation error with classifier accuracy, and studying the estimation error of method by partitioning into different subgroups.

weaknesses:
1. Under what assumptions does the particular joint model over classifier scores and true latent distribution makes sense is not described.  

2. The details regarding how are different classifiers distinct from each other is not given. If I understand correctly, each classifier is trained on the same training data, so I assume the only difference arises from different random initialization ? In that case, it appears that different classifier scores should be very similar to each other. Then the point of using multiple classifiers is not clear. 

3. A simple baseline such as majority vote weighted by some function of classifier accuracy could be provided.

3. The exact implementation details of baselines such as David Skene is not provided.

4. Assumption of availability of multiple classifiers appears like a strong assumption. 

5. The experiment section could be structured better. Instead of figure 2 and figure 3, tables could have provided more information. Some tables in appendix should be used in the main paper.

questions:
Describe more details on how the baselines were used and how are the set of trained classifiers different from each other.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701272604

thread_type:
official_review

<thread_object>
creator: reviewer_aeyd

<content>
summary:
The paper introduces Semi-Supervised Model Evaluation (SSME), a novel method designed to evaluate machine learning classifiers using both labeled and unlabeled data, addressing the challenge of obtaining large labeled datasets. SSME leverages the availability of multiple classifiers, continuous classifier scores, and abundant unlabeled data to estimate the joint distribution of true labels and classifier predictions through a semi-supervised mixture model.  The authors validate SSME across four challenging domains—healthcare, content moderation, molecular property prediction, and image annotation—showing that it significantly reduces error rates in performance estimation compared to traditional methods.

strengths:
Originality: SSME is a pioneering approach that creatively combines labeled and unlabeled data for model evaluation, filling a critical gap in existing methodologies.
Clarity: The abstract effectively communicates the core concepts and findings, making it accessible to a broad audience while maintaining technical depth.
Significance: This work addresses a significant challenge in machine learning evaluation, potentially benefiting many fields where labeled data is scarce. Its implications for improving model assessment and performance understanding are substantial.

weaknesses:
In fact, there has been a lot of work in recent years on evaluating language models, but none of these are mentioned in this paper. Moreover, since the datasets and classifiers selected for the experiment also involve the architecture of large language models, these methods of evaluating language models are not mentioned in the comparison method.

questions:
What is the complexity of the method in this paper? In other words, is this approach worth the computational resources we consume compared to tolerating the errors that come with traditional evaluation methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732133231919

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for your feedback!

comment:
We thank all reviewers for their constructive feedback, and for their recognition of the importance of the problem we address – measuring model performance with little labeled data. We appreciate their saying that we “fill a critical gap,” “address a common bottleneck,” and “introduce a setting often encountered in practice.” The reviewers also seem convinced that we improve over the 5 baselines we compare to: one reviewer states that the results are "superior compared to baselines," another notes that our method "significantly reduces error rates... compared to traditional methods" and a third notes that the "results indicate improvements over baselines."

The reviewers also provided a number of useful suggestions for improving the manuscript, all of which we have addressed by performing additional experiments and submitting a revised draft. (Note that the updated draft appears to be over the page count because we used track changes to make clear what the edits are, but it is within the page count.) We summarize our major changes below and provide additional details in the point-by-point response to each reviewer. 

* Reviewers suggested comparisons to additional baselines. We have reviewed all suggested baselines, and added comparisons to the 2 additional baselines which are comparable to our method and not previously included in our evaluation. Our method outperforms these baselines. We have also clarified that several suggested baselines are already included in our evaluation. 
* Reviewer SyCL requested discussion of the theoretical properties of our method; we have added this in Section E.3 by drawing from the semi-supervised mixture model estimation literature.
* While several reviewers found the work clear and easy to understand, others suggested details to clarify which we have now addressed, including how classifiers within a set differ from one another (Section B.1), connections to LLM evaluation (Lines 889 - 893), why we model the distribution of classifier scores *s* instead of inputs *x* (Lines 164-170), and our procedure for inferring ground truth labels (Lines 240-246). In particular, this addresses the main concern of reviewer w43E, which was lack of methodological clarity.
* Reviewers asked about the scalability of our method. We have provided additional details on the method's computation time: SSME fits all configurations of (n_labeled, dataset, data split) reported in the paper in under 5 minutes, creating minimal computational overhead. We have also clarified that the scalability of the method depends on the model class used to parameterize the mixture model; hence, for extremely large datasets, efficient model classes (e.g., fast implementations of KDEs) could be used.

We believe the updated manuscript to be much stronger because of the reviewers’ feedback and look forward to continued discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary

The paper proposes a semi-supervised evaluation method for classifier performance using unlabelled data, addressing the lack of annotated test datasets. It introduces a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier predictions. The method uses continuous classifier scores, multiple classifiers, and abundant unlabeled data to infer both true labels and classifier performance. Empirical studies across four domains show that this approach improves evaluation accuracy. 

Strengths


The paper addresses a practical and underexplored problem, making it highly relevant for real-world scenarios where labelled test data is unavailable. Its strengths include strong experimental results that validate the proposed method and its application across multiple domains, demonstrating robustness and generalizability. The paper also offers a scalable solution. 

Weaknesses

The main concern is that the method may lack novelty, as it appears to be generative modelling through a mixture model to estimate the joint probability of the ground truth labels of unlabelled data and the predictions made by classifiers. The assumption underlying the joint modelling of classifier scores and ground truth labels is not guaranteed to hold universally, potentially limiting its applicability. 

Reasons for decision

Overall, the primary concern with the paper lies in its lack of novelty. Neither the paper itself nor the rebuttal successfully convinced reviewers of its methodological originality, as the techniques employed are perceived as a combination of existing methods, albeit with different objectives. While the paper explores an interesting and potentially impactful direction, addressing the challenge of insufficient labelled test data, it currently falls short in practicality due to concerns about its assumptions and technical implementation. These issues, highlighted by reviewers, need to be thoroughly addressed in future versions of the work.