SEMI-SUPERVISED LEARNING OBJECTIVES AS LOG-LIKELIHOODS IN A GENERATIVE MODEL OF DATA CURATION
Year: 2021

Abstract:
We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to give a proof-of-principle for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any “out of distribution” examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.1

heading: 1 INTRODUCTION
text: To build high-performing deep learning models for industrial and medical applications, it is necessary to train on large human-labelled datasets. For instance, Imagenet (Deng et al., 2009), a classic benchmark dataset for object recognition, contains over 1 million labelled examples. Unfortunately, human labelling is often prohibitively expensive. In contrast obtaining unlabelled data is usually very straightforward. For instance, unlabelled image data can be obtained in almost unlimited volumes from the internet. Semi-supervised learning (SSL) attempts to leverage this unlabelled data to reduce the required number of human labels (Seeger, 2000; Zhu, 2005; Chapelle et al., 2006; Zhu & Goldberg, 2009; Van Engelen & Hoos, 2020). One family of SSL methods — those based on low-density separation — assume that decision boundaries lie in regions of low probability density, far from all labelled and unlabelled points. To achieve this, pre deep learning (DL) low-density separation SSL methods such as entropy minimization and pseudo-labelling (Grandvalet & Bengio, 2005; Lee, 2013) use objectives that repel decision boundaries away from unlabelled points by encouraging the network to make more certain predictions on those points. Entropy minimization (as the name suggests) minimizes the predictive entropy, whereas pseudo-labelling treats the currently most-probable label as a pseudo-label, and minimizes the cross entropy to that pseudo-label. More modern work uses the notion of consistency regularisation, which augments the unlabelled data (e.g. using translations and rotations), then encourages the neural network to produce similar outputs for different augmentations of the same underlying image (Sajjadi et al., 2016; Xie et al., 2019; Berthelot et al., 2019b; Sohn et al., 2020). Further developments of this line of work have resulted in many variants/combinations of these algorithms, from directly encouraging the smoothness of the classifier outputs around unlabelled datapoints (Miyato et al., 2018) to the “FixMatch” family of algorithms (Berthelot et al., 2019b;a; Sohn et al., 2020), which combine pseudo-labelling and consistency regularisation by augmenting each image twice, and using one of the augmented images to provide a pseudo-label for the other augmentation.
1Our code: https://anonymous.4open.science/r/GZ_SSL-B6CC; MIT Licensed
However, some of the biggest successes of deep learning, from supervised learning to many generative models, have been built on a principled statistical framework as maximum (marginal) likelihood inference (e.g. the cross-entropy objective in supervised learning can be understood as the log-likelihood for a Categorical-softmax model of the class-label MacKay, 2003). Low-density separation SSL methods such as pseudo-labelling and entropy minimization are designed primarily to encourage the class-boundary to lie in low-density regions. Therefore they cannot be understood as log-likelihoods and cannot be combined with principled statistical methods such as Bayesian inference.
Here, we give a formal account of SSL methods based on low-density separation (Chapelle et al., 2006) as lower bounds on a principled log-likelihood. In particular, we consider pseudo-labelling (Lee, 2013), entropy minimization (Grandvalet & Bengio, 2005), and modern methods similar to FixMatch (Sohn et al., 2020). This log-likelihood arises from a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison, 2021). Critically, this approach gives an explanation for previous findings that SSL is most effective when unlabelled data is obtained by throwing away labels from the carefully curated training set, and is less effective when unlabelled data is taken from uncurated images, especially those that do not depict one of the classes of interest (Cozman et al., 2003; Oliver et al., 2018; Chen et al., 2020; Guo et al., 2020). We confirmed the importance of data curation for SSL on toy data generated from a known model and on real data from Galaxy Zoo 2 (Willett et al., 2013).

heading: 2 BACKGROUND
text: Our work brings together many disparate areas. Here, we give an introduction to a generative model of data curation (Aitchison, 2021) initially developed to explain the cold posterior effect (Wenzel et al., 2020), pseudo-labelling and entropy minimization (Grandvalet & Bengio, 2005; Lee, 2013), and the treatment of unlabelled points in the standard supervised learning setup.

heading: 2.1 A GENERATIVE MODEL OF DATA CURATION
text: To develop a model of data curation, remember that image datasets including CIFAR-10 and ImageNet are curated to ensure they only contain images whose class-labels are unambiguous. For instance, in CIFAR-10, annotators were instructed that “It’s worse to include one that shouldn’t be included than to exclude one.”, and Krizhevsky (2009) “personally verified every label submitted by the annotators”. In creating ImageNet, Deng et al. (2009) made sure that a number of Amazon Mechanical Turk annotators agreed upon the class before including an image in the dataset.
Thus, these datasets have two odd properties. First, consensus labels exist only for a subset of images, e.g. for a white-noise image, consensus cannot be reached and the image cannot be labelled. Second, inclusion of an image in a dataset like CIFAR-10 is informative in and of itself, as it indicates that the image shows an unambiguous example of one of the ten classes. To understand these odd properties of curated datasets, consider a simplified generative model of consensus-formation: draw a random image, X , from the distribution over images, P (X), and ask S human annotators, indexed s, to give a label, {Ys}Ss=1 (e.g. using Mechanical Turk). Importantly, every annotator is forced to label every image and if the image is ambiguous they should give a random label. If all the annotators agree, Y1=Y2= · · · =YS , they have consensus and the datapoint is included in the dataset. However, in the case of any disagreement, consensus is not reached and the datapoint is excluded (Fig. 1), Concretely, the final label, Y is Y1 (which is the same as all the other labels) if consensus was reached and None otherwise (Fig. 2C),
Y |{Ys}Ss=1 = { Y1 if Y1=Y2= · · · =YS None otherwise
(1)
Taking Y to be the label set, we have Ys ∈ Y , and the final label, Y , could be any of the underlying labels in Y , or None if consensus is not reached, so Y ∈ Y ∪ {None}. When consensus was reached, the likelihood is,
P (Y =y|X, θ) = P ( {Ys=y}Ss=1|X, θ ) = ∏S s=1 P (Ys=y|X, θ) = P (Ys=y|X, θ) S = (py(X)) S
(2)
where we have assumed annotators are IID, and py(X) = P (Ys=y|X, θ) is the single-annotator probability. From here, it is possible to see how this model might be taken to give an account of tempering, as we have taken the underlying single-annotator likelihood, py(X) to the power S (for further details see Aitchison, 2021).

heading: 2.2 LOW-DENSITY SEPARATION SEMI-SUPERVISED LEARNING OBJECTIVES
text: The intuition behind low-density separation objectives for semi-supervised learning is that decision boundaries should be in low-density regions away from both labelled and unlabelled data. As such, it is sensible to “repel” decision boundaries away from labelled and unlabelled datapoints and this can be achieved by making the classifier as certain as possible on those points. This happens automatically for labelled points as the standard supervised objective encourages the classifier to be as certain as possible about the true class label. But for unlabelled points we need a new objective that encourages certainty, and we focus on two approaches. First, and perhaps most direct is entropy minimization (Grandvalet & Bengio, 2005)
Lentropy(X) = ∑ y∈Y py(X) log py(X) (3)
where, following the typical probabilistic approach, we write the negative entropy as an objective to be maximized. Alternatively, we could use pseudo-labelling, which takes the current classification, y∗, to be the true label, and maximizes the log-probability of that label (Lee, 2013),
Lpseudo(X) = log py∗(X) y∗ = argmax y∈Y log py(X). (4)
Lee (2013) regarded pseudo-labelling as closely related to entropy miminization as the optimal value of both objectives is reached when all the probability mass is assigned to one class. However, they are not formulated as a principled log-likelihood, which gives rise to at least three problems. First, these methods cannot be combined with other principled statistical methods such as Bayesian inference. Second, it is unclear how to combine these objectives with standard supervised objectives, except by taking a weighted sum and doing hyperparameter optimization over the weight. Third, these objectives risk reinforcing any initial poor classifications and it is unclear whether this is desirable.

heading: 2.3 IN STANDARD SUPERVISED LEARNING, UNLABELLED POINTS SHOULD BE UNINFORMATIVE
text: It is important to note that under the standard supervised-learning generative model (Fig. 2A), unlabelled points should not give any information about the weights. Omitting the label, Ysup, we obtain the graphical model in Fig. 2B. This model emphasises that the images, X , and the model parameters, θ, are marginally independent, so we cannot obtain any information about θ from X alone (Fig. 2B). Formally, the posterior over θ conditioned on X is equal to the prior,
P (θ|X) = P (θ,X) P (X) =
∑ y∈Y P (θ,X, Ysup=y)
P (X) (5)
= P (θ) P (X) ∑ y∈Y P (Ysup=y|θ,X) P (X) = P (θ) .
as 1 = ∑
y∈Y P (Ysup=y|θ,X). To confirm this result is intuitively sensible, note that are many situations where encouraging the decision boundary to lie in low density regions would be very detrimental to performance. Consider a classifier with two input features: x0 and x1 (Fig. 4A). The class boundary lies in the high-density region crossing both clusters, so to obtain a reasonable result, the classifier should ignore the low-density region lying between the clusters. However, strong low-density separation SSL terms in the objective may align the cluster boundaries with the class boundaries, leading the classifier to wrongly believe that one cluster is entirely one class and the other cluster is entirely the other class. In contrast, supervised learning without SSL will ignore clustering and obtain a reasonable answer close to the grey dashed line. Importantly, this is just an illustrative example to demonstrate that without further assumptions, the standard supervised approach of ignoring unlabelled data is sensible; semi-supervised learning without loss of performance in such settings has been studied and is known as Safe SSL (Li & Zhou, 2014; Krijthe & Loog, 2014; Kawakita & Takeuchi, 2014; Loog, 2015; Krijthe & Loog, 2016).

heading: 3 THEORY
text: SSL methods are usually applied to benchmark datasets such as CIFAR-10 or ImageNet. These datasets were first carefully curated during the labelling process: (Fig. 3A), implying that ambiguous images close to the decision boundary were excluded. Critically, unlabelled points for these benchmark datasets are obtained by taking labelled points (which have reached consensus) and throwing away their labels (Fig. 3B). The likelihood for consensus (Y 6=None) is
P (Y 6=None|X, θ) = ∑
y∈Y (py(X)) S . (6)
This probability is close to 1 (for S > 1) if the underlying distribution, (py(X)) S puts most of its mass onto one class, and the probability is smaller if the mass is spread out over classes. As such, the likelihood “repels” decision boundaries away from unlabelled points, which is the common intuition behind low-density separation SSL methods, and which should be beneficial if class boundaries indeed lie in regions of low probability density away from both labelled and unlabelled points.
If noconsensus images are observed (Fig. 2C), we can include a likelihood term for those images, P (Y =None|X, θ) = 1− P (Y 6= None|X, θ) = 1− ∑
y∈Y (py(X)) S . (7)
If noconsensus images are not observed, we could in principle integrate over the underlying distribution over images, P (X=x). However, we do not even have samples from the underlying distributions over images (and if we did, we would have the noconsensus images so we could use Eq. 7). As such this term is usually omitted (e.g. Aitchison, 2021), but the use of out-of-distribution (OOD) datasets as surrogate noconsensus points is an important direction for future work.

heading: 3.1 ENTROPY MINIMIZATION AND PSEUDO-LABELS ARE LOWER BOUNDS ON OUR PRINCIPLED LOG-LIKELIHOOD
text: To prove that entropy minimization forms a lower-bound on our log-likelihood (Eq. 6), we begin by writing the log-likelihood of consensus in terms of an expectation over labels, y,
log P (Y 6=None|X, θ) = log ∑ y∈Y py(X) (py(X)) S−1 = logEpy(X) [ (py(X)) S−1 ] . (8)
Applying Jensen’s inequality, the negative entropy gives a lower-bound on our log-likelihood, log P (Y 6=None|X, θ) ≥ Epy(X) [ log (py(X)) S−1 ]
= (S − 1) ∑
y∈Y py(X) log py(X) = (S − 1)Lentropy(X) (9)
This bound is tight for a uniform predictive distribution, log P (Y 6=None|X, θ) = log ∑
y∈Y (py(X)) S = logS ( 1 S )S = (S − 1) logS (10)
(S − 1)Lentropy(X) = −(S − 1) log ∑ y∈Ypy(X) log py(X) = (S − 1) logS. (11)
Pseudo-labelling forms an alternative lower bound on the log-likelihood which is obtained by noting that all (py(X)) S are positive, so selecting any subset of terms in the sum gives a lower bound,
log P (Y 6=None|X, θ) = log ∑
y∈Y (py(X)) S ≥ log (py∗(X))S = S log py∗(X) = SLpseudo(X).
(12)
The inequality holds if we choose y∗ to be any class, but will be tightest if we choose the highest probability class. This bound is tight for a predictive distribution that puts all its mass on y∗, so py∗(X) = 1 and py 6=y∗ = 0
log P (Y 6=None|X, θ) = log ∑
y∈Y (py(X)) S = log (py∗(X)) S = log 1 = 0 (13)
SLpseudo(X) = S log p∗y(X) = S log 1 = 0. (14) As such, entropy minimization and pseudo-labelling optimize different lower-bounds on our principled log-likelihood, log P (Y 6=None|X, θ), which gives a potential explanation for the effectiveness of pseudo-labelling and entropy minimization. Additionally, low-density separation SSL objectives encourages class-labels to be more certain. We can therefore expect pseudo-labelling to be the more relevant bound, as that bound is tight when the predictive distribution puts all its mass onto one class. In contrast, the entropy maximization bound is tight when the predictive distribution is uniform, which is discouraged by all low-density separation SSL objectives. This provides a potential explanation for the use of psuedo-labelling rather than entropy regularisation in modern SSL approaches such as (Sohn et al., 2020).

heading: 3.2 DATA AUGMENTATION PRIORS AND FIXMATCH FAMILY METHODS
text: FixMatch family methods combine data augmentation and pseudo-labelling. To understand FixMatch as a bound on a principled log-likelihood, we therefore need a principled account of data augmentation as a likelihood. Inspired by Wenzel et al. (2020) (their Appendix K), we consider a distribution, P (X ′|X), over augmented images, X ′, given the underlying unaugmented image, X . We choose the single-annotator predictive distribution as the average over predictive distributions for many different augmented images,
P (Ys=y|X, θ) = E [py(X ′)|X] (15) where py(X ′) is the predictive probabilities resulting from applying the neural network to the augmented image, and remember s ∈ {1, . . . , S} indexes the annotator. This is a sensible prior because we expect the neural network to be invariant under data-augmentation, and if the predictions are approximately invariant, then averaging the predictive distributions has little impact (Fig. 4B left). However, if the predictions do vary dramatically with different data augmentations then we should not trust the network’s classifications (i.e. we should have an uncertain predictive distribution), and averaging over very different predictive distributions for different augmentations indeed gives rise to broader, more uncertain predictions (Fig. 4B right).
To obtain a tractable objective in the supervised setting, we use a multi-sample version of Jensen’s inequality, with K augmented images denoted X ′k,
log P (Ys=y|X, θ) ≥ E [ log 1K ∑ kpy(X ′ k) ∣∣X] . (16)
Combining this single-annotator probability with our generative model of curation, we obtain,
log P (Y =y|X, θ) = S log P (Ys=y|X, θ) = S logE [py(X ′)|X] ≥ S E [ log 1K ∑ kpy(X ′ k) ∣∣X] , (17)
The resulting objective for unlabelled points is, log P (Y 6=None|X, θ) = log ∑
y∈Y P (Y =y|X, θ) = log ∑
y∈YE [py(X ′)|X] S ≈ log ∑ y∈Y ( 1 K ∑ kpy(X ′ k) )S , (18)
where we approximate the expectation with K different samples of X ′, denoted X ′k. Unfortunately, this approach does not immediately form a bound on the log-likelihood due to the convex nonlinearity in taking the power of S. Nonetheless, one key problem with approximating machine learning losses is that the optimizer learns to exploit approximation errors to find a pathological solution that makes the objective unboundedly large. We appear to be safe from that pathology here, as we are simply forming predictions by averaging over K augmentations of the underlying image. Nonetheless, to form a lower bound, we can follow FixMatch family algorithms by pseudo-labelling, i.e. by taking only one term in the sum for class y∗. FixMatch chooses y∗ by using the highest-probability class for a weakly-augmented image. An alternative approach is to choose the y∗ giving the tightest bound, i.e. argmaxy 1 K ∑ kpy(X ′ k). In either case,
log P (Y 6=None|X, θ) ≥ logE [py∗(X ′)|X] S ≥ S E [ log 1K ∑ kpy∗(X ′ k) ∣∣X] , (19)
If K = 1 and y∗ is chosen using a separate “weak” augmentation, then this is exactly equal to the FixMatch objective for unlabelled points.
Note that both of these objectives (Eq. 18 and 19) promote reduced predictive uncertainty. Importantly, this does not just increase confidence in the single-augmentation predictive distributions, py(X ′ k), but also increases alignment between the predictive distributions for different augmentations (Fig. 4B). In particular, if the single-augmentation predictives are all highly confident, but place that high-confidence on different classes, then the multi-augmentation predictive formed by averaging will have low-confidence (Fig. 4B right). The only way for the multi-augmentation predictive to have high confidence is if the underlying single-augmentation predictive distributions have high confidence in the same class (Fig. 4B left), which encourages the underlying network to become more invariant. This makes sense: if data-augmentation changes the class predicted by the neural network, then any predictions should be low confidence. And it implies that combining principled data augmentation with a generative model of data curation automatically gives rise to an objective encouraging invariance.

heading: 4 RESULTS
text: We begin by giving a proof-of-principle for Bayesian SSL on a toy dataset generated from a known model. Next, we tested our theoretical results (rather than trying to achieve SOTA performance) on real-world datasets. In particular, our theory gives one explanation for why SSL is typically more effective when unlabelled data is taken from the original, curated training set. To confirm these results, we used Galaxy Zoo 2 as this was a real-world dataset which allowed us to generate matched curated and uncurated datasets.

heading: 4.1 BAYESIAN SSL ON A GENERATED DATASET
text: Our formulation of SSL as a likelihood implies that it should be possible to take entirely novel approaches, such as using low-density separation SSL in a Bayesian neural network (BNN).
We considered a toy dataset generated from a “true” neural network model with one hidden layer and 30 hidden units, 5 dimensional inputs and 2 output classes. We generated inputs IID from a Gaussian, then passed them through the “true” neural network, then sampled multiple categorical
class-labels corresponding to different annotators. If all the simulated annotators agreed, consensus was reached and if any simulated annotators disagreed, consensus was not reached. We used 100 labelled datapoints, though not all of them will have reached consensus, and we used up to 1600 unlabelled points, though again not all of them will have reached consensus. Note that as the consensus/noconsensus status of a point arises from the generative model, we cannot independently specify the number of consensus/noconsensus points. We used Eq. (2) as the likelihood for labelled points, Eq. (6) as the likelihood for unlabelled points and Eq. (7) as the likelihood for noconsensus points. We sampled (and trained networks on) 500 datasets in parallel. We trained using Langevin dynamics with all data simultaneously (no minibatching) with no momentum and no rejection.
For a generative model with S = 1, consensus is always reached and the problem is equivalent to standard supervised learning. As such, we found no benefits from including unlabelled points for S = 1. In contrast, for any setting of S > 1 we found that increasing the number of unlabelled points improved the test log-likelihood (Fig. 5A–B) and the test accuracy (Fig. 5C–D).

heading: 4.2 GALAXY ZOO 2
text: Our data curation based theory predicts that low-density separation based SSL should be much more effective on curated than uncurated data. To test this prediction on real-world data, we turned to Galaxy Zoo 22 (GZ2) (Willett et al., 2013) which uses images from the Sloan Digital Sky Survey. This dataset is particularly useful for us as it has received only very minimal filtering based on criteria such as object brightness and spatial extent. We defined 9 labels by truncating the complex decision tree followed by the annotators (for further details see Aitchison, 2021). Further, as each GZ2 image has received ∼ 50 labels, we can define a consensus coefficient by taking the fraction of annotators that agreed upon the highest probability class. We can then define a curated dataset by taking the images with consensus coefficient above some threshold within each class. Note that we needed to select images on a per-class basis, because annotators tend to be more confident on some classes than others, so taking the highest consensus coefficients overall would dramatically change the class balance. In particular, we used the top 8.2% of images, which gave a full curated dataset of just over 20,000 images. Of those, we randomly selected 2000 as labelled examples, 10000 as test examples, and 0 – 6000 as unlabelled examples. The images were preprocessed by center-cropping to 212× 212 and then scaled to 32× 32. We applied a FixMatch-inspired semi-supervised learning algorithm, with a standard supervised objective, with unlabelled objective given by Eq. (18) with K = 2. Data augmentation was given by vertical and horizontal flips, rotations from −180◦ to 180◦, translations by up to 40% on both axes and scaling from 20% to 180%. Note that as we were trying to mirror the standard SSL setup, we did not include noconsensus points in the objective. We trained a ResNet18 with our maximum likelihood objective using SGD with a batch size of 500, a learning rate of 0.01 and 1500 epochs. We used an internal cluster of nVidia 1080 and 2080 GPUs, and the experiments took roughly 300 GPU hours.
We found that the test-log-likelihood for curated data improved slightly as more unlabelled points were included, whereas the test-log-likelihood for uncurated dramatically declined as unlabelled points were added (Fig. 6A–B). We saw strong improvements in test accuracy with the number of
2https://data.galaxyzoo.org; www.sdss.org/collaboration/image-use-policy/
unlabelled points for curated datasets (Fig. 6C–D). Note that in Fig. 6C the error rate for curated datasets is already very small, so to see any effect we needed to plot the test error, normalized to the initial test error (Fig. 6D). For uncurated data, the inclusion of large numbers of unlabelled points dramatically worsened performance, though the inclusion of a small number of unlabelled points gave very small performance improvements (Fig. 6C–D). Thus, this experiment is consistent with the idea that the effectiveness of SSL arises at least in part from curation of the underlying dataset.

heading: 5 RELATED WORK
text: There are at least three main approaches to semi-supervised learning (Seeger, 2000; Zhu, 2005; Chapelle et al., 2006; Zhu & Goldberg, 2009). First there is low-density separation, where we assume that the class boundary lies in a region of low probability density away from both labelled and unlabelled points. This approach dates back at least to transductive support vector machines (SVMs) where the model is to be tested on a finite number of known test locations (Vapnik, 1998; Chapelle et al., 1999). Those known test locations are treated as unlabelled points, and we find the decision boundary that perfectly classifies the limited number of labelled points, while at the same time being as far as possible from labelled and unlabelled data. Alternative approaches include pseudo-labelling and entropy minimization (Grandvalet & Bengio, 2005; Lee, 2013). Second, there are graph-based methods such as (Zhu & Ghahramani, 2002) which are very different from the methods considered here. Third, there are approaches that use unlabelled points to build a generative model of the inputs and leverage that model to improve classification (e.g. Kingma et al., 2014; Odena, 2016; Gordon & Hernández-Lobato, 2017). This approach was originally explored in a considerable body of classical work (e.g. McLachlan, 1975; Castelli & Cover, 1995; Druck et al., 2007) for a review, see Seeger (2000) and references therein. These approaches are fundamentally different from the SSL approaches considered here, as they require a generative model of inputs, while low-density separation methods do not. Generative modelling can be problematic as training a generative model can be more involved than training a discriminative model and because the even when the model can produce excellent samples, the high-level representation may be “entangled” (Higgins et al., 2017) in which case it may not offer benefits for classification.

heading: 6 DISCUSSION
text: Our theory provides a theoretical understanding of past results showing that SSL is more effective when unlabelled data is drawn from the original, curated training set (Cozman et al., 2003; Oliver et al., 2018; Chen et al., 2020; Guo et al., 2020). In the extreme, our theory might be taken to imply that if data has not been curated, then SSL cannot work, and therefore that low-density separation SSL methods will not be effective in messy, uncurated real-world datasets. However, this is not the complete picture. Low-density separation SSL methods, including our log-likelihood, fundamentally exploit class-boundaries lying in low-density regions. As such, low-density separation could equally come from the real underlying data or could be artificially induced by data curation (Fig. 3). None of these methods are able to distinguish between these different underlying sources of low-density separation and as such any of them may work on uncurated data where the underlying distribution displays low-density separation. However, the possibility for curation to artificially induce low-density separation does imply that we should be cautious about overinterpreting spectacular results obtained on very carefully curated benchmark datasets such as CIFAR-10.
Surprisingly, the generative model of data curation used here also explains the cold-posterior effect in Bayesian neural networks (Wenzel et al., 2020; Aitchison, 2021), revealing a profound and previously unsuspected connection.
In conclusion, we showed that low-density separation SSL objectives can be understood as a lowerbound on a log-probability which arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective on curated data, which we confirmed by developing a Bayesian SSL model applied to toy data, using GZ2, which allowed us to consider a completely uncurated dataset.

Review summaries:
The paper analysis semi-supervised methods taking into account the curation process employed in the creation of popular datasets, like ImageNet. The authors model this process via a generative model which they use to show two common objectives in semi-supervised learning, namely entropy minimisation and pseudo-labelling, are actually lower-bounds on the log-likelihood of the data under this generative model. The paper also shows experimentally, that unlabelled data improves the accuracy of classifiers when the dataset has been curated but actually hurts performance when uncurated datasets are used.

 This paper uses a generative model for "curated" labeled datasets that was initially developed by Aitchison 2020 for another seemingly unrelated purpose (explaining the "cold posterior" effect). The assumed task is classification, where there is a set of mutually exclusive possible class we wish to assign to each image. To obtain labels for a training set given specific images (or other example features), the generative model assumes that each example is assigned a label from S independent, identically-distributed annotators. Only if *all* annotators agree is consensus reached and the class label is provided, otherwise the image is considered "noconsensus" and is usually not included. It is suggested that many common SSL datasets (e.g. CIFAR-10) use such a consensus curation process, and that even unlabeled sets are subjected to consensus curation.  Using this model, they provide a principled explanation for the empirical success of several well-known semi-supervised learning (SSL) objectives -- entropy minimization, pseudo-label, and FixMatch -- applicable to discriminative deep learning. The key idea is that each objective can be viewed as a lower bound of the log likelihood of the unlabeled set under the proposed model for curation, assuming that the unlabeled set contains only "consensus" images.  In Sec. 4.1, the paper presents a Bayesian SSL analysis on a toy dataset. The key result is that if unlabeled data is generated without multi-annotator curation, then the unlabeled data does not improve classifier accuracy when included in training using SSL. However, if only consensus examples are included in the unlabeled set, then reasonable improvements in accuracy might be expected from SSL vs. labeled-set-only learning.  In Sec. 4.2, the paper analyzes astronomy images with 9 class labels, prepared using the Galaxy Zoo dataset (to my knowledge not a common dataset for SSL). The existence of multiple annotators for this dataset allows creating both curated and uncurated versions. Again, they find that for *curated* datasets (where many annotators agree), both test-log-likelihood and test accuracy improve from the inclusion of unlabeled examples. However, for *uncurated* datasets, including too many unlabeled examples "dramatically worsened" test-set performance, though small unlabeled sets help.

This paper shows that low-density separation semi-supervised learning (SSL) objectives can be understood as a lower-bound on a log-probability that arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective when unlabelled data is obtained by throwing away labels from the carefully curated training set, and is less effective when unlabeled data is taken from uncurated images. Experiments on toy data generated from a known model and on real data from Galaxy Zoo confirm the importance of data curation for SSL.

The paper is built on the previous work (Aitchison, 2021) that provides a generative model of data curation. In this model, the likelihood of a labeled data obtained with consensus is given by product of the probability that each labeler labels correctly. Under this model, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. In toy experiments and Galaxy Zoo experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data.