Semi-supervised learning objectives as log-likelihoods in a generative model of data curation

1 INTRODUCTION . To build high-performing deep learning models for industrial and medical applications , it is necessary to train on large human-labelled datasets . For instance , Imagenet ( Deng et al. , 2009 ) , a classic benchmark dataset for object recognition , contains over 1 million labelled examples . Unfortunately , human labelling is often prohibitively expensive . In contrast obtaining unlabelled data is usually very straightforward . For instance , unlabelled image data can be obtained in almost unlimited volumes from the internet . Semi-supervised learning ( SSL ) attempts to leverage this unlabelled data to reduce the required number of human labels ( Seeger , 2000 ; Zhu , 2005 ; Chapelle et al. , 2006 ; Zhu & Goldberg , 2009 ; Van Engelen & Hoos , 2020 ) . One family of SSL methods — those based on low-density separation — assume that decision boundaries lie in regions of low probability density , far from all labelled and unlabelled points . To achieve this , pre deep learning ( DL ) low-density separation SSL methods such as entropy minimization and pseudo-labelling ( Grandvalet & Bengio , 2005 ; Lee , 2013 ) use objectives that repel decision boundaries away from unlabelled points by encouraging the network to make more certain predictions on those points . Entropy minimization ( as the name suggests ) minimizes the predictive entropy , whereas pseudo-labelling treats the currently most-probable label as a pseudo-label , and minimizes the cross entropy to that pseudo-label . More modern work uses the notion of consistency regularisation , which augments the unlabelled data ( e.g . using translations and rotations ) , then encourages the neural network to produce similar outputs for different augmentations of the same underlying image ( Sajjadi et al. , 2016 ; Xie et al. , 2019 ; Berthelot et al. , 2019b ; Sohn et al. , 2020 ) . Further developments of this line of work have resulted in many variants/combinations of these algorithms , from directly encouraging the smoothness of the classifier outputs around unlabelled datapoints ( Miyato et al. , 2018 ) to the “ FixMatch ” family of algorithms ( Berthelot et al. , 2019b ; a ; Sohn et al. , 2020 ) , which combine pseudo-labelling and consistency regularisation by augmenting each image twice , and using one of the augmented images to provide a pseudo-label for the other augmentation . 1Our code : https : //anonymous.4open.science/r/GZ_SSL-B6CC ; MIT Licensed However , some of the biggest successes of deep learning , from supervised learning to many generative models , have been built on a principled statistical framework as maximum ( marginal ) likelihood inference ( e.g . the cross-entropy objective in supervised learning can be understood as the log-likelihood for a Categorical-softmax model of the class-label MacKay , 2003 ) . Low-density separation SSL methods such as pseudo-labelling and entropy minimization are designed primarily to encourage the class-boundary to lie in low-density regions . Therefore they can not be understood as log-likelihoods and can not be combined with principled statistical methods such as Bayesian inference . Here , we give a formal account of SSL methods based on low-density separation ( Chapelle et al. , 2006 ) as lower bounds on a principled log-likelihood . In particular , we consider pseudo-labelling ( Lee , 2013 ) , entropy minimization ( Grandvalet & Bengio , 2005 ) , and modern methods similar to FixMatch ( Sohn et al. , 2020 ) . This log-likelihood arises from a generative model of data curation that was initially developed to explain the cold-posterior effect ( Aitchison , 2021 ) . Critically , this approach gives an explanation for previous findings that SSL is most effective when unlabelled data is obtained by throwing away labels from the carefully curated training set , and is less effective when unlabelled data is taken from uncurated images , especially those that do not depict one of the classes of interest ( Cozman et al. , 2003 ; Oliver et al. , 2018 ; Chen et al. , 2020 ; Guo et al. , 2020 ) . We confirmed the importance of data curation for SSL on toy data generated from a known model and on real data from Galaxy Zoo 2 ( Willett et al. , 2013 ) . 2 BACKGROUND . Our work brings together many disparate areas . Here , we give an introduction to a generative model of data curation ( Aitchison , 2021 ) initially developed to explain the cold posterior effect ( Wenzel et al. , 2020 ) , pseudo-labelling and entropy minimization ( Grandvalet & Bengio , 2005 ; Lee , 2013 ) , and the treatment of unlabelled points in the standard supervised learning setup . 2.1 A GENERATIVE MODEL OF DATA CURATION . To develop a model of data curation , remember that image datasets including CIFAR-10 and ImageNet are curated to ensure they only contain images whose class-labels are unambiguous . For instance , in CIFAR-10 , annotators were instructed that “ It ’ s worse to include one that shouldn ’ t be included than to exclude one. ” , and Krizhevsky ( 2009 ) “ personally verified every label submitted by the annotators ” . In creating ImageNet , Deng et al . ( 2009 ) made sure that a number of Amazon Mechanical Turk annotators agreed upon the class before including an image in the dataset . Thus , these datasets have two odd properties . First , consensus labels exist only for a subset of images , e.g . for a white-noise image , consensus can not be reached and the image can not be labelled . Second , inclusion of an image in a dataset like CIFAR-10 is informative in and of itself , as it indicates that the image shows an unambiguous example of one of the ten classes . To understand these odd properties of curated datasets , consider a simplified generative model of consensus-formation : draw a random image , X , from the distribution over images , P ( X ) , and ask S human annotators , indexed s , to give a label , { Ys } Ss=1 ( e.g . using Mechanical Turk ) . Importantly , every annotator is forced to label every image and if the image is ambiguous they should give a random label . If all the annotators agree , Y1=Y2= · · · =YS , they have consensus and the datapoint is included in the dataset . However , in the case of any disagreement , consensus is not reached and the datapoint is excluded ( Fig . 1 ) , Concretely , the final label , Y is Y1 ( which is the same as all the other labels ) if consensus was reached and None otherwise ( Fig . 2C ) , Y | { Ys } Ss=1 = { Y1 if Y1=Y2= · · · =YS None otherwise ( 1 ) Taking Y to be the label set , we have Ys ∈ Y , and the final label , Y , could be any of the underlying labels in Y , or None if consensus is not reached , so Y ∈ Y ∪ { None } . When consensus was reached , the likelihood is , P ( Y =y|X , θ ) = P ( { Ys=y } Ss=1|X , θ ) = ∏S s=1 P ( Ys=y|X , θ ) = P ( Ys=y|X , θ ) S = ( py ( X ) ) S ( 2 ) where we have assumed annotators are IID , and py ( X ) = P ( Ys=y|X , θ ) is the single-annotator probability . From here , it is possible to see how this model might be taken to give an account of tempering , as we have taken the underlying single-annotator likelihood , py ( X ) to the power S ( for further details see Aitchison , 2021 ) . 2.2 LOW-DENSITY SEPARATION SEMI-SUPERVISED LEARNING OBJECTIVES . The intuition behind low-density separation objectives for semi-supervised learning is that decision boundaries should be in low-density regions away from both labelled and unlabelled data . As such , it is sensible to “ repel ” decision boundaries away from labelled and unlabelled datapoints and this can be achieved by making the classifier as certain as possible on those points . This happens automatically for labelled points as the standard supervised objective encourages the classifier to be as certain as possible about the true class label . But for unlabelled points we need a new objective that encourages certainty , and we focus on two approaches . First , and perhaps most direct is entropy minimization ( Grandvalet & Bengio , 2005 ) Lentropy ( X ) = ∑ y∈Y py ( X ) log py ( X ) ( 3 ) where , following the typical probabilistic approach , we write the negative entropy as an objective to be maximized . Alternatively , we could use pseudo-labelling , which takes the current classification , y∗ , to be the true label , and maximizes the log-probability of that label ( Lee , 2013 ) , Lpseudo ( X ) = log py∗ ( X ) y∗ = argmax y∈Y log py ( X ) . ( 4 ) Lee ( 2013 ) regarded pseudo-labelling as closely related to entropy miminization as the optimal value of both objectives is reached when all the probability mass is assigned to one class . However , they are not formulated as a principled log-likelihood , which gives rise to at least three problems . First , these methods can not be combined with other principled statistical methods such as Bayesian inference . Second , it is unclear how to combine these objectives with standard supervised objectives , except by taking a weighted sum and doing hyperparameter optimization over the weight . Third , these objectives risk reinforcing any initial poor classifications and it is unclear whether this is desirable . 2.3 IN STANDARD SUPERVISED LEARNING , UNLABELLED POINTS SHOULD BE UNINFORMATIVE . It is important to note that under the standard supervised-learning generative model ( Fig . 2A ) , unlabelled points should not give any information about the weights . Omitting the label , Ysup , we obtain the graphical model in Fig . 2B . This model emphasises that the images , X , and the model parameters , θ , are marginally independent , so we can not obtain any information about θ from X alone ( Fig . 2B ) . Formally , the posterior over θ conditioned on X is equal to the prior , P ( θ|X ) = P ( θ , X ) P ( X ) = ∑ y∈Y P ( θ , X , Ysup=y ) P ( X ) ( 5 ) = P ( θ ) P ( X ) ∑ y∈Y P ( Ysup=y|θ , X ) P ( X ) = P ( θ ) . as 1 = ∑ y∈Y P ( Ysup=y|θ , X ) . To confirm this result is intuitively sensible , note that are many situations where encouraging the decision boundary to lie in low density regions would be very detrimental to performance . Consider a classifier with two input features : x0 and x1 ( Fig . 4A ) . The class boundary lies in the high-density region crossing both clusters , so to obtain a reasonable result , the classifier should ignore the low-density region lying between the clusters . However , strong low-density separation SSL terms in the objective may align the cluster boundaries with the class boundaries , leading the classifier to wrongly believe that one cluster is entirely one class and the other cluster is entirely the other class . In contrast , supervised learning without SSL will ignore clustering and obtain a reasonable answer close to the grey dashed line . Importantly , this is just an illustrative example to demonstrate that without further assumptions , the standard supervised approach of ignoring unlabelled data is sensible ; semi-supervised learning without loss of performance in such settings has been studied and is known as Safe SSL ( Li & Zhou , 2014 ; Krijthe & Loog , 2014 ; Kawakita & Takeuchi , 2014 ; Loog , 2015 ; Krijthe & Loog , 2016 ) .

Summary:  This paper uses a generative model for "curated" labeled datasets that was initially developed by Aitchison 2020 for another seemingly unrelated purpose (explaining the "cold posterior" effect). The assumed task is classification, where there is a set of mutually exclusive possible class we wish to assign to each image. To obtain labels for a training set given specific images (or other example features), the generative model assumes that each example is assigned a label from S independent, identically-distributed annotators. Only if *all* annotators agree is consensus reached and the class label is provided, otherwise the image is considered "noconsensus" and is usually not included. It is suggested that many common SSL datasets (e.g. CIFAR-10) use such a consensus curation process, and that even unlabeled sets are subjected to consensus curation.  Using this model, they provide a principled explanation for the empirical success of several well-known semi-supervised learning (SSL) objectives -- entropy minimization, pseudo-label, and FixMatch -- applicable to discriminative deep learning. The key idea is that each objective can be viewed as a lower bound of the log likelihood of the unlabeled set under the proposed model for curation, assuming that the unlabeled set contains only "consensus" images.  In Sec. 4.1, the paper presents a Bayesian SSL analysis on a toy dataset. The key result is that if unlabeled data is generated without multi-annotator curation, then the unlabeled data does not improve classifier accuracy when included in training using SSL. However, if only consensus examples are included in the unlabeled set, then reasonable improvements in accuracy might be expected from SSL vs. labeled-set-only learning.  In Sec. 4.2, the paper analyzes astronomy images with 9 class labels, prepared using the Galaxy Zoo dataset (to my knowledge not a common dataset for SSL). The existence of multiple annotators for this dataset allows creating both curated and uncurated versions. Again, they find that for *curated* datasets (where many annotators agree), both test-log-likelihood and test accuracy improve from the inclusion of unlabeled examples. However, for *uncurated* datasets, including too many unlabeled examples "dramatically worsened" test-set performance, though small unlabeled sets help.