Semi-supervised learning objectives as log-likelihoods in a generative model of data curation
Authors: Stoil Krasimirov Ganev
Laurence Aitchison
Venue: ICLR 2022 Submitted (2022)
Keywords: Bayesian neural network
semi-supervised learning
Bayesian inference

Abstract: We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to give a proof-of-principle for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any ``out of distribution'' examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.

Under review as a conference paper at ICLR 2022
SEMI-SUPERVISER LEARNING OBJECTIVES AS LOG-LIKELIHOODS IN A GENERATIVE MODEL OF DATA CURATION
Anonymous authors Paper under double-blind review
ABSTRACT
We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development e.g. Bayesian SSL. Here, we note that bench- mark image datasets such CIFAR-I0 are carefully curated, and formu- a generative model of data curation
late SSL objectives as a log-likelihood that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-ard techniques similar to FixMatch can be understood as lower-bounds on our prin- cipled log-likelihood. We are thus able to give proof-of-princpple for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL. performs better on clean datasets without any "out of distribution" examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2
INTRODUCTION
To build high-performing deep learning models for industrial and medical applications, is neces- sary to train on large human-labelled datasets. For instance, Imagenet (Deng al., 2009), a classic benchmark dataset for object recognition, contains over million labelled examples. Unfortunately, human labelling is often prohibitively expensive. In contrast obtaining unlabelled data is usually very straightforward. For instance, unlabelled image data can be obtained in almost unlimited vol- umes from the internet. Semi-supervised learning (SSL) attempts leverage this unlabelled data to reduce the required number of human labels (Seeger) 2000 Zhu 2005 Chapelle et al.) 2006 Zhu & Goldberg 2009; Van Engelen & Hoos 2020). One family of SSL methods - those based on low-density separation assume that decision boundaries lie in regions of low probability density, far from all labelled and unlabelled points. To achieve this, pre deep learning (DL) low-density separation SSL methods such entropy minimization and pseudo-labelling (Grandvalet & Bengio, 2005 Lee 2013) use objectives that repel decision boundaries away from unlabelled points by en- couraging the network make more certain predictions on those points. Entropy minimization (as the name suggests) minimizes the predictive entropy, whereas pseudo-labelling treats the currently most-probable label as pseudo-label, and minimizes the cross entropy to that pseudo-label. More modern work uses the notion of consistency regularisation, which augments the unlabelled data (e.g. using translations and rotations). then encourages the neural network to produce similar outputs for different augmentations of the same underlying image (Sajjadi et al.) 2016 Xie et al.) 2019: Berth- elot al., 2019b] [Sohn et al. 2020). Further developments of this line work have resulted in many variants/combinattions of these algorithms, from directly encouraging the smoothness of the classifier outputs around unlabelled datapoints (Miyato et al. 2018)) to the "FixMatch" family of algorithms (Berthelot et al. 2019b a Sohn et 2000), which combine ppeudo-labelling and con- sistency regularisation by augmenting each image twice, and using one of the augmented images to provide pseudo-label for the other augmentation.
Our code: http:://aaannnnnnnssasaasssccncc
MIT Licensed

---

Under review as a conference paper at ICLR 2022
Y1 2/2 Y33 train Y train
Y1 1/2 Y3 bus
Y1 = Y/ train;
Y/ bus
Y bus
Y None
Figure 1: A depiction of the generative model of data curation, with S Annotators in- structed to classify images as trains or buses. The left-hand image is clearly a train, so the annotators agree and consensus is reached. The middle image is clearly bus, annotators agree and consen- sus is reached. The right-hand image, however, is ambiguous or even has an ill-defined class label. So annotators disagree, consensus is not reached and the image is excluded from the dataset.
A
X
B
X
C
X
D X
sup
[1,)==
[-- C
0
0
Figure 2: Graphical models under consideration. The observed variables are highlighted in red. A The generative model for standard supervised learning with no data curation. Note that the label, Youp E V. only takes on values in the label set, so it differs from our label, Y, which could also be None. The generative model for standard supervised learning, omitting the label. Under this model P (0IX) reduces to P (0) (see Sec. 2.3 C The generative model with data curation for labelled points, note that there consensus, E E and if there is no consensus, = None. D The generative model with data curation for unlabelled points. C' is a random variable representing whether or not consensus was reached, so we have C = if consensus was reached (i.e. Y y) not reached (i.e. Y None). Critically, C acts "label", so
and C if consensus the posterior over parameters, P [Ø]X, , C)), does not reduce to (0), unlike in the case of standard supervised learning when we omit the label (panel B).
However, some of the biggest successes deep learning, from supervised learning to many gen- erative models, have been built principled statistical framework maximum (marginal) like- lihood inference (e.g. the cross-entropy objective in supervised learning can be understood as the log-likelihood for Lategorical-softman model of the class-label MacKKy 2003). Low-density sep- aration SSL. methods such as pseudo-labellin and entropy minimization are designed primarily to encourage the class-boundary lie in low-density regions. Therefore they cannot be understood as log-likelshoos and cannot be combined with principled statistical methods such as Bayesian inference.
Here, we give formal account of SSL methods based on low-density separation (Chapelle et al,, 2006) as lower bounds on a principled log-likei.hod. In particular, we consider labudo-labelling (Lee 2013). entropy minimization (Grandvalet & Bengio (005),, and modern methods similar to FixMatch (Sohn et al., 2020). This log-likelihood arises from a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison) (021). Critically, this approach gives an explanation for previous findings that SSL most effective when unlabelled data obtained by throwing away labels from the carefully curated training set, and is less effective when unlabelled data is taken from uncurated images, especially those that do not depict one of the classes of interest (Cozman al. 2003) Oliver al. 2018)) Chen et al. 2020; Guo et al.) 2020). We confirmed the importance of data curation for SSL on toy data generated from a known model and on real data from Galaxy Zoo 2 (Willett et al. 2013)
2

---

Under review as a conference paper at ICLR 2022
deciion boundary conGenBUE: class
na conserGus
COTSSTISSS class
8
2-
2
4
2
30
Figure 3: Our generative model of data curation applied to a simple 2D dataset. Data from each class sampled from different Gaussian, and the true decision boundary (green dashed line) was given by the posterior probability of class given (æo, æ1). A Datapoints far from the decision boundary are unambiguous, so annotators agree and consensus is reached (red and blue points), Datapoints close to the decision boundary are ambiguous, so consensus is not reached (grey crosses). The consensus datapoints thus exhibit artificially induced low-density separation. benchmark datasets such as CIFAR-10, the unlabelled points (yellow triangles) are selected from the consensus points (red or blue points) as the noconensus points are not available. The unlabelled points therefore also exhibit artificially induced low-density separation.
When using
BACKGROUND
Our work brings together many disparate areas. Here, we give an introduction to generative model of data curation (Aitchison 2021) initially developed to explain the cold posterior effect (Wenzel eet al.) 2020), and the treatment of unlabelled points in the standard supervised learning setup.
and entropy minimization (Grandvalet & Bengio 2005 Lee 2013),
2.1 A GENERATIVE MODEL OF DATA CURATION
To develop model of data curation, remember that image datasets including CIFAR-10 and Im- ageNet are curated to ensure they only contain images whose class-labels are unambiguous. For instance, in CIFAR-10, annotators were instructed that "It's worse include one that shouldn't be included than to exclude one.". and Krizhevsky (2009) "personally verified every label submitted by the annotators" In creating ImageNet, Deng et (2009) made sure that number of Amazon Mechanical Turk annotators agreed upon the class before including an image in the dataset.
Thus, these datasets have two odd properties. First, consensus labels exist only for subset of im- ages, e.g. for white-noise image, consensus cannot be reached and the image cannot be labelled. Second, inclusion of an image in a dataset like CIFAR-10 informative and itself, as indi- cates that the image shows unambiguous example of one of the ten classes. To understand these odd properties of curated datasets, consider simplified generative model consensus-formatiion draw random image, X. from the distribution over images. P (X), and ask S human annotators, indexed s, to give label, {Y, 18 (c.g. using Mechanical Turk). Importantly, every annotator is forced to label every image and if the image ambiguous they should give a random label. If all the annotators agree, Yı=Y==
=Ys, they have consensus and the datapoint is included in the dataset. However, the case of any disagreement, consensus is not reached and the datapoint is excluded (Fig. D. Concretely, the final label, Y is Y; (which is the same as all the other labels) if consensus was reached and None otherwise (Fig. 2C).
Yi
i =Ys
VI(Y)
(1)
None
otherwise
Taking )' to be the label set, we have Y, E V. and the final label, Y, could be any of the underlying labels in V, or None if consensus not reached, so y [None). When consensus was reached, the likelihood is.
P PY P ((Y. ((Y-v))& (x -II P PYY-=y/X,0 (Y -\x0) = (X))
3

---

Under review as a conference paper at ICLR 2022
where we have assumed annotators IID, and Px((X) PY(Y, is the single-am annotaaor probability. From here, it is possible to see how this model might be taken to give an account of tempering. have taken the underlying single-annotator likelihood, Py(X) to the power S (for further details see Aitchison 2021
2.2 LOW-DENSITY SEPARATION SEMI-SUPERVISED LEARNING OBJECTIVES
The intuition behind low-density separation objectives for semi-supervised learning is that decision boundaries should be in low-density regions away from both labelled and unlabelled data. As such, is sensible to "repel" decision boundaries away from labelled and unlabelled datapoints and this can be achieved by making the classifier as certain as possible on those points. This happens au- tomatically for labelled points as the standard supervised objective encourages the classifier to be as certain as possible about the true class label. But for unlabelled points we need a new objective that encourages certainty, and we focus on two approaches. First, and perhaps most direct is entropy minimization (Grandvalet & Bengio, 2005)
Centropy (X) = Σ (X( (x))
(3)
yEY
where, following the typical probabilistic approach, we write the negative entropy an objective to be maximized. Alternatively, we could use pseudo-labelling, which takes the current classification, y", to be the true label, and maximizes the log-probability of that label (Lee) 2013).
(pe)nd((] = log Py- (X)
y = arg max log p,y yEY
(4)
Lee (2013) regarded pseudo-labelling as closely related to entropy miminization as the optimal value of both objectives is reached when all the probability mass assigned one class. However, they are not formulated as principled log-likelihood, which gives rise to at least three problems. First, these methods cannot be combined with other principled statistical methods such as Bayesian inference. Second, it is unclear how to combine these objectives with standard supervised objectives, except by taking a weighted sum and doing hyperparameter optimization over the weight. Third, these objectives risk reinforcing any initial poor classifications and it is unclear whether this is desirable.
2.3
IN STANDARD SUPERVISED LEARNING, UNLABELLED POINTS SHOULD BE UNINFORMATIVE
is important to note that under the standard supervised-learre generative model (Fig. 2 unla- belled points should not give any information about the weights. Omitting the label, Ysu we obtain the graphical model in Fig. B This model emphasises that the images, X, and the model param- eters, 0, are marginally independent, so cannot obtain any information about 0 from X alone (Fig. 2B). Formally, the posterior over 0 conditioned on X is equal to the prior,
(0IX) P ((,, Eyey P (0, Yup P(X) P P(O) wey 11,0 = P
(5)
(())
(
as 1 - PXm To contirm this result is intuitively sensible, note that are many situations where encouraging the decision boundary to lie in low density regions would be very detrimental to performance. Consider a classifier with two input features: To and æ1 (Fig. AA). The class boundary lies in the high-density region crossing both clusters, so obtain reasonable re- sult, the classifier should ignore the low-density region lying between the clusters. However, strong low-density separation SSL. terms in the objective may align the cluster boundaries with the class boundaries, leading the classifier to wrongly believe that one cluster is entirely one class and the other cluster is entirely the other class. In contrast, supervised learning without SSL will ignore clustering and obtain a reasonable answer close to the grey dashed line. Importantly, this is just an illustrative example to demonstrate that without further assumptions, the standard supervised ap- proach of ignoring unlabelled data is sensible; supervupe learning without loss of performance in such settings has been studied and is known as Safe SSL (Li & Zhou 2014) & Loog 2014 Krijthe & Loog 2016).
Kawakita & Takeuchi 2014
Loog 2015
4

---

Under review as a conference paper at ICLR 2022
A
more Invariant NN
loss Invartant NN
true boundary
E [pyomrjp (400))
22
-4 -3
0
2
Xp
Figure 4: A. A toy dataset generated to illustrate the dangers of using the clustering of the input points to inform classification boundaries. The input features, zp and æ1 are plotted on the and y- and the class is represented by colour. B. A schematic diagram demonstrating the effect of our principled likelihood incorporating anttettgmenttiionn on the certainty of predictions for different degrees of invariance. More invariant NNs (left) give similar predictive distributions for different augmentations (blue), and hence a certain averaged predictive distribution (bottom; orange). Less invariant NNs (right) give different predictive distributions for different augmentations (blue), and hence highly uncertain averaged predictive distributions (bottom: orange).
THEORY
SSL methods are usually applied to benchmark datasets such as CIFAR-10 or ImageNet. These datasets were first carefully curated during the labelling process: (Fig. BA), implying that ambiguous images close to the decision boundary were excluded. Critically, unlabelled points for these bench- mark datasets are obtained by taking labelled points (which have reached consensus) and throwing away their labels (Fig. 33). The likelihood for consensus (Y (None))
P ANonel X, 0) = -yEe (py(X))
(6)
This prohability close to (for S > 1) if the underlying distribution, (Py(X))) puts most of its mass onto class, and the probability is smaller if the mass is spread out over classes. As such, the likelihood "repels" decision boundaries away from unlabelled points, which is the common intuition behind low-density separation SSL methods, and which should be beneficial if class boundaries indeed in regions of low probability density away from both labelled and unlabelled points.
If noconsensus images observed (Fig. 2C), can include likelihood term for those images, P (Y None X, 0) - -Eve (py(X))
(7) If noconsensus images are not observed, we could in principle integrate over the underlying dis- tribution over images, P P(X= =x). However, we not even have samples from the underlying distributions over images (and if we did, would have the noconsensus images Eq. 77.. As such this term is usually omitted (c.g. Aitchison 202T), but the use out-of--istribution (OOD) datasets surrogate noconsensus points important direction for future work.
P PYY= X)0)
could use
3.1 ENTROPY MINIMIZATION AND PSEUDO-LABELS ARE LOWER BOUNDS ON OUR PRINCIPLED LOG-LIKELIHOOD
To prove that entropy minimization forms lower-bound log-likelihood (Eq. 6), begin by writing the log-likelihood of consensus in terms of an expectation over labels,
log P (Y ANone X, -log Σ (X) (p, log Ex,, (x) (x)))))
yEy
Applying Jensen's incquality, the negative entropy gives lower-bound our likeliiidd
log P (Y~Nonel X, 2 Ep,(x) log (X)(
(S - 1) Pa (X log (X) = (S 1) L (X)
(9)
5

---

Under review as a conference paper at ICLR 2022
This bound is tight for uniform predictive distribution,
log (Y NNone X, 8) log Evey (x,(X)) = log (3) (S 1) log S - 11 Settroy (X) - 1) log EveyPs (X) = (S 1) log
(10)
(11)
Pseudo-labelling forms an alternative lower bound on the log- likkliihoood which is obtained by noting that all (X)) are positive, so selecting any subset of terms in the sum gives lower bound,
log P (Y*None|X, X, 0) = log Evey (Py(x))² 2 log (Py- (X)) = 5logg Pv- (x) = S X)-
(12)
The inequality holds if we choose y² to be any class, but will be tightest if we choose the highest probability class. This bound is tight for a predictive distribution that puts all its mass on y", so Py-(X)
and Pytv
log P (oneeee|X, 0) log Evey (X)) = log (x)) = llo 1 = 0
(13)
Sppscud (X) S (X) = Slog = = 0.
(14)
As such, entropy minimization and pseudo-labelling optimize different lower-bounds our princi- pled log-likelihood, log P (Y*None|X, f), which gives a potential explanation for the effectiveness of pseudo-labelling and entropy minimization. Additionally, low-density separation SSL objectives encourages class-labels to be more certain. We can therefore expect pseudo-labelling to be the more relevant bound, that bound is tight when the predictive distribution puts all its mass onto one class. In contrast, the entropy maximization bound is tight when the predictive distribution is uniform, which is discouraged by all low-density separation SSL objectives. This provides po- tential explanation for the use of psuedo-labelling rather than entropy regularisation in modern SSL. approaches such as (Sohn al. 2020
3.2 DATA AUGMENTATION PRIORS AND FIXMATCH FAMILY METHODS
FixMatch family methods combine data augmentation and pseudo-labelling To understand Fix- Match as a bound on a principled log-likelihoode we therefore need a principled account of data augmentation likelihood. Inspired by Wenzel et al.) (2020) (their Appendix K), consider distribution, P (X) over augmented images, X', given the underlying unaugmented image, X. We choose the single-annotator predictive distribution the average over predictive distributions for many different augmented images,
P PY x 0) E (X')
(15)
where py (X') the predictive probabilities resulting from applying the neural network to the aug- mented image, and remember E {1...., S} indexes the annotator. This is sensible prior because we expect the neural network to be invariant under luta-auumentttion, and if the predictions are approximately invariant, then averaging the predictive distributions has little impact (Fig. 4 left). However, if the predictions do vary dramatically with different data augmentations then we should trust the network's classifications (i.e. we should have uncertain predictive distribution), and averaging over very different predictive distributions for different augmentations indeed gives rise to broader, more uncertain predictions (Fig. 4 right).
To obtain a tractable objective in the supervised setting, we use a multi-sample version of Jensen's inequality, with K augmented images denoted X²-
log P (Y, =) 2 E |log 1 Xx)
(16)
Combining this single-annotator probability with our generative model of curation, we obtain,
log P (Y =y] 0) = S log P xY,=)
S log E [,, (X') [x]] 2 S E [log (x))(
(17)
The resulting objective for unlabelled points is, log P (Y~None) X, f) = log ΣyEy P (Y x
= log EyEDE PP, (X') xo log Dye C (x)))
(18)

---

Under review as a conference paper at ICLR 2022
test log ag-likelihodd
lest log ggg-lleelhoodd
0.06
85
00.1
acouracy
2
0.04
accuracy
-0.2
0.02
75
-0.3
0.00
1000
1000
1000
0
1000
unlabelled points
unlabelled points
unlabelled points
unlabelled points
Figure 5: Test log-like log-likelihood and accuracy for Langevin sampling for Bayesian SSL on toy datasets sampled from the model as a function of the number of unlabelled points.
where we approximate the expectation with K different samples of X', denoted X Unfortunately, this approach does not immediately form bound on the log-likelihood due the convex nonlinear- ity in taking the power of S. Nonetheless, one key problem with approximating machine learning losses is that the optimizer learns to exploit approximation errors find pathological solution that makes the objective unboundedly large. We appear to be safe from that pathology here, we are simply forming predictions by averaging over K augmentations of the underlying image. Nonethe- less, to form a lower bound, we can follow FixMatch family algorithms by pseudo-labelle taking only one term in the sum for class y² FixMatch chooses y" by using the highest-pros -hobbst-probbbiiitty class for weakly-augnut augmented image. An alternative approach is to choose the y" giving the tightest bound, i.e. arg max, K ExPr(X)). In either case,
i.e. by
log P (Y ANone X, 0) 2 log E [,y- (X') 2 SE log R E (X][X),
(19)
If K = 1 and y is chosen using a separate "weak" augmentation, then this is exactly equal to the FixMatch objective for unlabelled points.
Note that both of these objectives (Eq. 18 and 19) promote reduced predictive uncertainty. Im- portantly, this does not just increase confidence in the single-augmentation predictive distributions, (X,)), but also increases alignment between the predictive distributions for different augmenta- tions (Fig. 4). In particular, the single-augmentation predictives are all highly confident, but place that high-confidence on different classes, then the multi-augnu aullii-uummentation predictive formed by averag- ing will have low-confidence (Fig. A right). The only way for the auglmnttugmtttnooonn predictive to have high confidence is if the underlying single-augmentation predictive distributions have high confidence in the same class (Fig. 4 left), which encourages the underlying network to become more invariant. This makes sense: if data-augmentation changes the class predicted by the neural network, then any predictions should be low confidence. And implies that combining principled data augmentation with generative model of data curation automatically gives rise to objective encouraging invariance.
RESULTS
We begin by giving a proof-of-principle for Bayesian SSL on a toy dataset generated from a known model. Next, tested our theoretical results (rather than trying achieve SOTA performance) on real-world datasets. In particular, our theory gives one explanation for why SSL is typically more effective when unlabelled data is taken from the original, curated training To confirm these results, we used Galaxy Zoo 2 as this was real-world dataset which allowed us to generate matched curated and uncurated datasets.
4.1 BAYESIAN SSL ON A GENERATED DATASET
Our formulation of SSL likelihood implies that it should be possible to take entirely novel approaches, such as using low-density separation SSL in a Bayesian neural network (BNN).
We considered toy dataset generated from "true" neural network model with hidden layer and 30 hidden units, 5 dimensional inputs and output classes. We generated inputs IID from a Gaussian, then passed them through the "true" neural network, then sampled multiple categorical
7

---

Under review as a conference paper at ICLR 2022
4
lest log glliillhhodd C
exact
test log log-likeihood
Norm test error
1.0
uncurated
test error (%)
pseudo
uncurated
-2"
50-
exact
curated
pseudo
0.0
curated
0K
3K
0K
3K 6K
3K
8K
0K
3K
unlabelled points
unlabelled points
unlabelled points
unlabelled points
Figure 6: Test log-likelihood and error for curated and uncurated GZ2 datasets as a function of the number unlabelled points. Exact corresponds to Eq. (18) (which is exact in the limit as K oo)
and pseudo corresponds to the pseudo labelling version of the augmented objective (Eq. 19)
class-labels corresponding to different annotators. If all the simulated annotators agreed, consen- sus was reached and any simulated annotators disagreed, consensus was not reached. We used 100 labelled datapoints, though not all of them will have reached consensus, and we used up to 1600 unlabelled points, though again not all of them will have reached consensus. Note that as the consensus/nocoosensus status of point arises from the generative model, we cannot independently specify the number of consensus/noconsensus points. We used Eq. (2) as the likelihood for labelled points, Eq. (6) as the likelihood for unlabelled points and Eq. 7 the likelihood for noconsensus points. We sampled (and trained networks on) 500 datasets in parallel. We trained using Langevin dynamics with all data simultaneously (no minibatching) with no momentum and rejection.
For generative model with 5 = 1, consensus is always reached and the problem is equivalent to standard supervised learning. As such, found henefits from including unlabelled points for S = 1. In contrast, for any setting of S >> 1 we found that increasing the number of unlabelled points improved the test log-likelihood (Fig. 5A-B) and the test accuracy (Fig. (C-D).
4.2 GALAXY Zoo 2
Our data curation based theory predicts that low-density separation based SSL should be much more effective on curated than uncurated data. To test this prediction on real-world data, we turned to Galaxy Zoo 2 (GZ2) (Willett et al. 2013) which uses images from the Sloan Digital Sky Survey. This dataset is particularly useful for us as it has received only very minimal filtering based on criteria such object brightness and spatial extent. We defined labels by truncating the complex decision tree followed by the annotators (for further details see Aitchiison 2021). Further, as each GZ2 image has received 50 labels, we define consensus coefficient by taking the fraction of annotators that agreed upon the highest probability class. We can then define a curated dataset by taking the images with consensus coefficient above some threshold within each class. Note that needed to select images per-class basis, because annotators tend to be more confident on some classes than others, so taking the highest consensus coefficients overall would dramatically change the class balance. In particular, we used the top 8.2% of images, which gave a full curated dataset of just over 20,000 images. Of those, we randomly selected 2000 as labelled examples, 10000 as test examples, and () - 6000 as unlabelled examples. The images were preprocessed by center-cropping to 212 212 and then scaled to 32 x 32. We applied a FixMatch-inspired semi-supervised learning algorithm, with standard supervised objective, with unlabelled objective given by Eq. (18) with K 2. Data augmentation was given by vertical and horizontal flips, rotations from -880 to 180", translations by up 40% on both axes and scaling from 20% to 180%. Note that trying to mirror the standard SSL setup, we did not include noconsensus points in the objective. We trained R8 with our maximum likelihood objective using SGD with batch size of 500, learning rate of 0.01 and 1500 epochs. We used an internal cluster of n Vidia 1080 and 2080 GPUs, and the experiments took roughly 300 GPU hours.
were
We found that the test-log-likllhhodd for curated data improved slightly as more unlabelled points were included, whereas the les-llog-ikelihoodd for uncurated dramatically declined as unlabelled points were added (Fig. (A.B). We saw strong improvements in test accuracy with the number of
8

---

Under review as a conference paper at ICLR 2022
unlabelled points for curated datasets (Fig. GC-D). Note that in Fig. 6C the error rate for curated datasets is already very small, so to see any effect we needed to plot the test error, normalized to the initial test error (Fig. (D). For uncurated data, the inclusion of large numbers of unlabelled points dramatically worsened performance, though the inclusion of a small number of unlabelled points gave very small performance improvements (Fig. (C-D). Thus, this experiment is consistent with the idea that the effectiveness of SSL arises at least in part from curation of the underlying dataset.
5 RELATED WORK
There are least three main approaches to semi-supervised learning (Seeger 2000 Zhu 2005
Chapelle et al.] 2006; Zhu & Goldberg, 2009). First there is low-density separation, where we assume that the class boundary lies in region of low probability density away from both labelled and unlabelled points. This approach dates back at least to transductive support vector machines (SVMs) where the model is to be tested on a finite number of known test locations (Vapnik) 1998 Chapelle et al. 1999). Those known test locations are treated as unlabelled points, and we find the decision boundary that perfectly classifies the limited number of labelled points, while at the same time being as far as possible from labelled and unlabelled data. Alternative approaches include pseudo-labelling and entropy minimization (Grandvalet & Bengio 2005 Lee| 2013). Second, there are graph- based methods such as (Zuu & Ghahramani 2002) wh which are very different from the methods considered here. Third, there are approaches that use unlabelled points to build a generative model of the inputs and leverage that model to improve classification (e.g. Kingma et al., [014]]
Odenal 2016; Gordon & Hernández-Lobate (017). This approach was originally explored
considerable body of classical work (e.g. McLachlan 1975: Castelli & Cover 1995 Druck et al., 2007] for a review, see [Seeger (2000) and references therein. These approaches are fundamentally different from the SSL approaches considered here, as they require generative model of inputs, while low-density separation methods do not. Generative modelling can be problematic as training generative model can be more involved than training a discriminative model and because the even when the model can produce excellent samples, the high-level representation may be "entangled" (Higgins al. 2017) in which case it may not offer benefits for classification.
DISCUSSION
Our theory provides a theoretical understanding of past results showing that SSL is more effective when unlabelled data is drawn from the original, curated training set (Cozman et al. 2003 Oliver jet al., 2018 Chen et al.) 2020 Guo et al., 2020). In the extreme, our theory might be taken to imply that if data has not been curated, then SSL cannot work, and therefore that low-density sepa- ration SSL methods will not be effective in messy, uncurated real-world datasets. However, this is not the complete picture. Low-density separation SSL methods, including our log-likelihood, fun- damentally exploit bausdboossasiis lying in low-density regions. As such, low-density separation could equally come from the real underlying data could artificially induced by data curation (Fig. 3). None of these methods are able to distinguish between these different underlying sources of low-density separation and as such any of them may work on uncurated data where the underly- ing distribution displays low-density separation. However, the possibility for curation to artificially induce low-density separation does imply that we should cautious about overinterpreting spec- tacular results obtained on very carefully curated benchmark datasets such as CIFAR-10.
Surprisingly, the generative model of data curation used here also explains the cold-posterior fect in Bayesian neural networks (Wenzel et al. 2020: Aitchison 2021). revealing profound and previously unsuspected connection.
In conclusion, we showed that low-density separation SSL objectives can be understood as lower- bound on a log-poobayility which arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective curated data, which we confirmed by developing a Bayesian SSL model applied to toy data, using GZ2, which allowed to consider completely uncurated dataset.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. ICL.R, 2021.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint 1nXiv:1911.09785 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 5050-5060, 2019b.
Vittorio Castelli and Thomas M Cover. On the exponential value of labeled samples. Pattern Recognition Letters, 16(1):105-111, 1995.
Olivier Chapelle, Vladimir Vapnik, and Jason Weston. Transductive inference for estimating values of functions. NIPS, pp. 421-427. Citeseer, 1999.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. 2006.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, pp. 3569-3576. AAAI Press, 2020.
Fabio Gagliardi Cozman, Ira Cohen, Marcelo Cesar Cirelo, et al. Semi-supervised learning of mixture models. In ICML, volume 4, pp. 24, 2003.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Gregory Druck, Chris Pal, Andrew McCallum, and Xiaojin Zhu. Semi Sumi-supe classification with hybrid generatieeddisrimnnatiee methods. In Proceedings of the 13th ACM SIGKDD inter- national conference Knowledge discovery and data mining, pp. 280-289, 2007.
Jonathan Gordon and José Miguel Hermández-Lobato. Bayesian semisupervised learning with deep
generative models. arXiv preprint 1rXiv:1700.09751, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Ad- vances in neural information processing systems, pp. 529 536, 2005.
Lan-Zhe Guo, Zhenyu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 3897-3906. PMLR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2017.
Masanori Kawakita and Jun ichi Takeuchi. Safe semi-supervised learning based on weighted likeli- hood. Neural Networks, 53:1461164, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27: 3581-3589, 2014.
Jesse H Krijthe and Mareo Loog. Implicitly constrained semi-supervised linear discriminant analy- sis. In 2014 22nd International Conference on Pattern Recognition, pp. 3762-3767. IEEF, 2014.
Jesse H Krijthe and Marco Loog. The pessimistic limits and possibilities of margin-based losses in supervisd learning. arXiv preprint arXiv:1612.08875, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech. report, 2009.
Dong- Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 2, 2013.
10

---

Under review as a conference paper at ICLR 2022
Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. IEEE transactions pattern analysis and machine intelligence, 37(1):175 18888,88888 2014.
Marco Loog. Contrastive pessimistic likelihood estimation for semi-supervised classification. IEEE transactions on pattern analysis and machine intelligence. 38(3)462-475, 2015.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university
press, 2003.
Geoffrey McLachlan. Iterative reclassification procedure for constructing asymptotically opti- mal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365-366 1975.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supe-supe learning. IEEE transactions on pattern analysis and machine intelligence, 1178):1979-1993, 2018.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint 160.:11606.01583,, 2016.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep sumi-supervised learning algorithms. In NeurlPS, pp. 9239-3250, 2018.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-1171, 2016.
Matthias Seeger. Learning with labeled and unlabeled data. 2000.
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07665, 2020.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing. 109(2):373-440, 2020.
V Vapnik. Statistical learning theory. NY: Wiley, 1998.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiętkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint 20ti:200..022055 2020.
Kyle W Willett, Chris Lintott, Steven Bamford, Karen Masters, Brooke D Simmons, Kevin RV Casteels, Edward M Edmondson, Lucy F Fortson, Sugata Kaviraj. William C Keel, et al. Galaxy zoo 2: detailed morphological classifications for 304 122 galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 435(4))2835-2860 2013.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. arXiv::904. 12848, 2019.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propa- gation. 2002.
Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
11


Reviews:
[{"review_id": "gTMQbsRembj", "reviewer": "Reviewer_vb5u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Under the generative model of data curation, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. These results are potentially interesting. \n\nIn toy experiments and Galaxy Zoo data experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data. However, this is not directly the consequence of the theoretical results. I am not clear on this connection. The theory shows that log likelihood is an upper bound on the entropy, pseudo labelling loss under the generative model of curated data. It does not say what happens when the data is uncurated. Also, I am not clear how the authors measured the test likelihood.\n\nBesides, the technical contribution, both theoretical and experimental are limited. On the experiments, it would be nicer to see the consequences of the theoretical results on modern semi supervised learning approaches and datasets they use.\n\n", "summary_of_the_paper": "The paper is built on the previous work (Aitchison, 2021) that provides a generative model of data curation. In this model, the likelihood of a labeled data obtained with consensus is given by product of the probability that each labeler labels correctly. Under this model, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. In toy experiments and Galaxy Zoo experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data.", "summary_of_the_review": "The theoretical and experimental contribution of the work are not enough to justify publication at the current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "XuskuPaAZyE", "reviewer": "Reviewer_FjiX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The theoretical results seem sound and connect popular semi-supervised objectives to the proposed generative model for curated datasets.\n- Using a generative model of the curation process for popular datasets in the context of semi-supervised learning is insightful, even though the model itself has already been proposed in previous work.\n\n### Weaknesses\n- The paper shows entropy minimisation and pseudo-labelling form a lower bound on the log-likelihood of the proposed generative model. However, optimising lower bounds is only useful when the bounds are tight, which does not seem to be the case here. The authors could expand their discussion on the insights these bounds provide.\n- The theoretical results motivates, to some extent, the use of semi-supervised learning when curated data is available but do not tell us much about uncurated data. We could argue that uncurated data does not match the proposed generative model, but then that is just a case of model misspecification which is a rather well-known problem and not directly connected to the theoretical results of the paper. In that light, I do not find the experimental results very insightful.\n- There is no discussion about other works that tried to explain the occasional poor performance of semi-supervised methods. I know of at least one [2] that provides theoretical results showing that unlabelled data (even when matching the distribution of labelled data) might hurt performance if the model is misspecified. They cover classifiers based on generative models, but there might still be parallels worth discussing.\n- Even though the text is well written, I do not find the exposition very clear. For instance, it was not clear if $\\theta$ refers to the parameters of 'true' data-generating process or the parameters of the classifier. Moreover, when I checked [1] for further background information, I found the exact same description, almost word for word, of the generative model for curated datasets. I find this problematic, as at best this is a bad case of text recycling.\n\n### Questions\n- In section 4.1., it is not clear how the annotators are simulated. Could the authors give more details on how the data was generated? Were the annotations just sampled from the categorical distribution defined by the 'true' neural network?\n- Still in section 5.1., how was the test data constructed? This is not entirely clear in the text.\n- Could there not be other factors influencing the poor performance when using uncurated data? For instance, uncurated data could be simply harder to model, with more ambiguous and noisier samples. Did the authors control for this somehow?\n\n### Minor Issues\n- Equation 5 seems a bit redundant. The result follows directly from the assumptions in the generative model, and this equation does not add much to the discussion.\n\n[1] Aitchison, Laurence. \"A statistical theory of cold posteriors in deep neural networks.\" International Conference on Learning Representations. 2021.\n\n[2] Cozman, Fabio Gagliardi, Ira Cohen, and Marcelo Cesar Cirelo. \"Semi-supervised learning of mixture models.\" ICML. Vol. 4. 2003.", "summary_of_the_paper": "The paper analysis semi-supervised methods taking into account the curation process employed in the creation of popular datasets, like ImageNet. The authors model this process via a generative model which they use to show two common objectives in semi-supervised learning, namely entropy minimisation and pseudo-labelling, are actually lower-bounds on the log-likelihood of the data under this generative model. The paper also shows experimentally, that unlabelled data improves the accuracy of classifiers when the dataset has been curated but actually hurts performance when uncurated datasets are used.", "summary_of_the_review": "The paper shows that existing semi-supervised objectives are proper lower-bounds on the log-likelihood of a generative model of curated data. These are interesting and valid results, but I am not confident they are very insightful. The experiments essentially show that uncurated data is harder to model, but unless I am missing something, this is not a surprising fact in itself nor is it a direct consequence of the bounds provided. Finally, I do think taking the curation process into account in semi-supervised learning (and other fields in machine learning) is quite promising, but the main methods of the paper, namely the generative model of curated data, are not new to this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P3P3oni_8oB", "reviewer": "Reviewer_B51g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Minor comments (language mistakes, etc.):\nFigure 1 caption: The first ``annotators\" should have its first letter capitalized.\nSection 2.1: \"ensure sure\" should be \"ensure\".\nSection 4.1: ``Fig. 5AB\" and ``Fig. 5CD\" should be written as ``Fig. 5A-B\" and ``Fig. 5C-D\" (i.e., with a hyphen inserted between).", "summary_of_the_paper": "This paper shows that low-density separation semi-supervised learning (SSL) objectives can be understood as a lower-bound on a log-probability that arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective when unlabelled data is obtained by throwing away labels from the carefully curated training set, and is less effective when unlabeled data is taken from uncurated images. Experiments on toy data generated from a known model and on real data from Galaxy Zoo confirm the importance of data curation for SSL.", "summary_of_the_review": "The paper is generally very well written, clearly presented, and a pleasure to read. The problem is well-motivated and well-defined with respect to the background. The technique look sound and novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JihCIM-r7Dy", "reviewer": "Reviewer_1qyP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n* Focus on the curated aspect of common SSL benchmark datasets\n* Goal of relating objectives to a proper generative model is valuable\n* Insightful use of the same generative model to offer support for 3 different common SSL objectives arising from last 20 years of SSL research\n* Nice toy data experiments in Figure 5, showing impact of more unlabeled data on SSL methods for both curated and uncurated unlabeled sets\n\n# Weaknesses\n\n* Missing previous known connections between Pseudolabel and Entropy regularization\n\nSee W1 below.\n\n* Results on Galaxy Zoo (Fig. 6) could use more explanation: why does performance benefit from a few (not too many) unlabeled examples even in noconsensus case?\n\nSee Q3 below.\n\n\n### W1: Missing previous known connections between Pseudolabel and Entropy regularization\n\nIn the presented background in Sec. 2, entropy regularization in Eq 3 and pseudolabel in Eq 4 are presented as seemingly separate alternatives. I think the authors could do a better job explaining that pseudolabel was originally motivated by entropy minimization -- the original PseudoLabel paper dedicates all of Sec. 3.2 to explaining that \"... our method is equivalent to Entropy Regularization.\"\n\nNot a dealbreaker, but making this connection could help the audience understand why both methods might be lower bounds to the same log likelihood.\n\n\n\n# Questions for Rebuttal\n\n### Q1: What insight about which SSL objectives to prefer is offered by your lower-bound arguments?\n\nThe paper offers justification of several SSL objectives -- pseudolabel, entropy minimization, and fixmatch --- by deriving each as a lower bound of the log likelihood of the label-consensus model.\n\nHowever, there is little insight here about when each bound might be expected to be \"the best\".... does FixMatch always dominate? \n\nThis may seem a bit out of scope, but I think offering some analysis here would help practitioners in thinking about which methods to apply in which circumstances.\n\n\n### Q2: Questions about Fig. 6\n\nWhat do \"exact-\" and \"pseudo-\" prefixes mean here? Can you adjust caption in revision to clarify?\n\n### Q3: Results on Galaxy Zoo (Fig. 6) could use more explanation\n\nIn Fig. 6, seems like both log likelihood and accuracy improve *slightly* with modest unlabeled data in uncurated case, but then decline with large unlabeled data. The authors acknowledge this in the paper by saying \" though the inclusion of a small number of unlabelled points gave very small performance improvements (Fig. 6CD)\".\n\nIs there any insight about why the curves look U shaped here, rather than a strict decline as the theory might suggest? Perhaps the data augmentation related to FixMatch used here is driving this? Would the same trend be seen if we just used entropy minimization?\n\n\n# Minor Presentation Comments\n\nNo need to mention these in rebuttal, but I hope you consider them to improve the paper\n\n### Comments on Figure 2\n\n* I felt Fig. 2 could communicate more than it did. Perhaps in the right-most panel that shows your consensus model, you could somehow visually emphasize that for unlabeled examples, you are assuming that \"consensus\" was reached, and thus that the distribution $p(\\theta | X)$ does not reduce to the prior marginal $p(\\theta)$ under the assumed model?\n\n### Superscript Notation could be improved\n\nThis is somewhat minor and perhaps a personal nitpick, but I dislike the notation $p^S_y(X)$ in Eq. 2. \n\nI'd prefer to see it like this:\n\n$$\n( p_y(X) )^S\n$$\n\nThis makes it clear that the quantity is raised to the $S$-th \npower. The current notation is a bit less clear to me on that front.\n\n\n### Lots of use of Y_s without defining $s$\n\nIn many cases, the paper writes $P( Y_s = y | X, \\theta)$ without defining a value for the index $s$ (as an example, see Eq. 2).  I guess I found this a little confusing, since elsewhere $s$ requires a concrete value to be well understood. I'd rather there was some explicit comment like \"where s can be any value in 1 to S\" or instead some other notation.  \n\n\n### Coments on Figure 3\n\n* Why in Figure 3 are points labeled \"bus\" and \"train\"? Aren't these just toy examples?\n* Not sure that both panels (3A and 3B) are needed in this figure. Could use just one, and verbally explain how unlabeled data would be selected. Or, at very least, the second panel could remove the noconsensus points, to help drive home visually the separation that occurs in the unlabeled set.\n\n\n### Comments on Sec. 4.2\n\nCurrent text says: \"data-curation based theory predicts that SSL should be much more effective on curated than uncurated data\".\n\nI'd clarify that the theory only applies to *SSL methods using the low-density separation principle*. Other SSL principles (e.g. those based on generative models) aren't applicable.", "summary_of_the_paper": "\nThis paper uses a generative model for \"curated\" labeled datasets that was initially developed by Aitchison 2020 for another seemingly unrelated purpose (explaining the \"cold posterior\" effect). The assumed task is classification, where there is a set of mutually exclusive possible class we wish to assign to each image. To obtain labels for a training set given specific images (or other example features), the generative model assumes that each example is assigned a label from S independent, identically-distributed annotators. Only if *all* annotators agree is consensus reached and the class label is provided, otherwise the image is considered \"noconsensus\" and is usually not included. It is suggested that many common SSL datasets (e.g. CIFAR-10) use such a consensus curation process, and that even unlabeled sets are subjected to consensus curation.\n\nUsing this model, they provide a principled explanation for the empirical success of several well-known semi-supervised learning (SSL) objectives -- entropy minimization, pseudo-label, and FixMatch -- applicable to discriminative deep learning. The key idea is that each objective can be viewed as a lower bound of the log likelihood of the unlabeled set under the proposed model for curation, assuming that the unlabeled set contains only \"consensus\" images.\n\nIn Sec. 4.1, the paper presents a Bayesian SSL analysis on a toy dataset. The key result is that if unlabeled data is generated without multi-annotator curation, then the unlabeled data does not improve classifier accuracy when included in training using SSL. However, if only consensus examples are included in the unlabeled set, then reasonable improvements in accuracy might be expected from SSL vs. labeled-set-only learning.\n\nIn Sec. 4.2, the paper analyzes astronomy images with 9 class labels, prepared using the Galaxy Zoo dataset (to my knowledge not a common dataset for SSL). The existence of multiple annotators for this dataset allows creating both curated and uncurated versions. Again, they find that for *curated* datasets (where many annotators agree), both test-log-likelihood and test accuracy improve from the inclusion of unlabeled examples. However, for *uncurated* datasets, including too many unlabeled examples \"dramatically worsened\" test-set performance, though small unlabeled sets help.", "summary_of_the_review": "I recommend accepting the paper. First, it draws attention to an important (yet under-studied) aspect of common SSL datasets like CIFAR-10: they are heavily curated and thus their so-called \"unlabeled\" sets are really just labeled sets with labels redacted. Second, it offers insight into why several widely-used SSL methods (entropy regularization, pseudolabel, and FixMatch) work, by framing each as a lower bound of a log-likelihood for a generative model for how many annotators produce consensus. I thought this was insightful and backed by nice toy data experiments and real data experiments (perhaps the curated vs uncurated versions of GalaxyZoo used here can become nice benchmarks for future SSL methods).\n\nIn the rebuttal and revisions, I hope the authors can address my questions about experimental results and why the bounds say about when each SSL method might be preferred over alternatives.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission has generated sufficient debate, including some messages that, in my viewpoint, have the wrong tone. It may well be that different colleagues see the work in different ways. It is very hard to evaluate submissions in a short time and mistakes can happen. In this case, I think there were and still are misunderstandings and unclarity wrt very crucial points of the paper. This does not mean that the work overall is weak not that there is no contribution. If the content is so interesting (as discussed by authors and multiple reviewers) in some way (which it seems to be), then a better presentation and argumentation will lead to a publication elsewhere soon, but based on all the data that I have here, I recommend rejection. I see to reason to list details about the content and possible concerns, as they should be clear from the multiple messages among authors and reviewers. Best of luck.
Decision: Reject