Semi-supervised learning with a principled likelihood from a generative model of data curation
Authors: Stoil Krasimirov Ganev
Laurence Aitchison
Venue: ICLR 2023 poster (2023)
Keywords: Bayesian deep learning
Bayesian neural networks
principled likelihoods

Abstract: We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation. We show that SSL objectives, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to introduce a Bayesian extension of SSL, which gives considerable improvements over standard SSL in the setting of 40 labelled points on CIFAR-10, with performance of $92.2\pm 0.3\%$ vs $88.6\%$ in the original FixMatch paper. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any ``out of distribution'' examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.

Published as a conference paper at ICLR 2023
SEMI-SUPERVISED LEARNING WITH A PRINCIPLED LIKELIHOOD FROM A GENERATIVE MODEL OF DATA CURATION
Stoil Ganev and Laurence Aitchison Department of Computer Science University of Bristol,
Bristol, UK laurence a aitchiso@@rristtl.cccm
ABSTRACT
We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-ikkelihodss, which precludes the development of e.g. Bayesian SSL. Here, we note that bench- mark image datasets such as CIFAR-10 are carefully curated, and formulate SSL objectives as a log-likelihooe in a generative model of data curation. We show that SSL objectives, from entropy minimization and pseudo-labelling, to state-of- the-art techniques similar to FixMatch can be understood as lower-bounds on our principled lieelool....d. We are thus able introduce Bayesian extension of SSL, which gives considerable improvements over standard SSL in the setting of 40 labelled points on CIFAR-10, with performance of 92. 4 0.3% 88.6% in the original FixMatch paper. Finally, our theory suggests that SSL is effective in part due the statistical patterns induced by data curation. This provides an explana- tion of past results which show SSL performs better on clean datasets without any "out of distribution" examples. Confirming these results we find that SSL gave much larger performance improvements on curated than uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2
INTRODUCTION
To build high--erttiee bigh-performing deep learning models for industrial and medical applications, it is neces- sary to train large hamen-labelled datasets. For instance, Imagenet (Deng al. 2009). classic benchmark dataset for object recognition, contains over million labelled examples. Unfortunately. human labelling is often prohibitively expensive. In contrast obtaining unlabelled data is usually very straightforward. For instance, unlabelled image data can be obtained almost unlimited vol- umes from the internet. Semi-supervised learning (SSL) attempts leverage this unlabelled data reduce the required number of human labels (Seeger 2000 Zhu 2005 Chapelle et al., 2006 Zhu & Goldberg 2009; Van Engelen & Hoos 2020). One family SSL methods those based on low-density separation - assume that decision boundaries lie in regions of low probability density, far from all labelled and unlabelled points. To achieve this, pre deep learning (DL. low-density separation SSL methods such entropy minimization and pseudo-labelling (Grandvalet & Bengio, 2005 Lee 2013) use objectives that repel decision boundaries away from unlabelled points by en- couraging the network make more certain predictions those points. Entropy minimization (as the name suggests) minimizes the predictive entropy, whereas pseudo-labelling treats the currently most-probable label pseudo-label, and minimizes the cross entropy to that paeudo-label. More modern work uses the notion consistency regularisation, which augments the unlabelled data (e.g. using translations and rotations). then encourages the neural network produce similar outputs for different augmentations of the same underlying image (Sajjadi et al.] 2016 Xie et al.) 2019 Berth- clot al., 2219b] [Sohn et al. 2020). Further developments of this line work have resulted in many variants/comminationn of these algorithms, from directly encouraging the smoothness of the classifier outputs around unlabelled datapoints (Miyato et al. 2018 the "FixMatch" family of
Our code: http:://anooymmooss...pppcccccce FDDE MTT Licensed

---

Published as a conference paper at ICLR 2023
algorithms (Berthelot et al. 2019b a Sohn et al. 2020), which combine apeudo-labelling and con- sistency regularisation by augmenting each image twice, and using one of the augmented images to provide pseudo-label for the other augmentation.
However, some of the biggest successes of deep learning, from supervised learning to many gen- erative models, have been built principled statistical framework maximum (marginal) like- lihood inference (e.g. the cross-entropy objective in supervised learning can be understood as the log-likelihood for aration SSL methods such as pseudo-labelling and entropy minimization are designed primarily to encourage the Hass-boundary lie in low-density regions. Therefore they cannot be understood as log- lieliiiel and cannot be combined with principled statistical methods such as Bayesian inference.
model of the class-label MacKay 2003). Low-density sep-
Here, we give formal account of SSL methods based on low-density separation (Chapelle et al,, 2006) as lower bounds on a principled log-likelihood. In particular, we consider pseudo-labelling [Lee] 2013), entropy minimization (Grandvalet & Bengio (005), and modern methods similar to FixMatch (Sohn al., 2020). Thus, we introduce a Bayesian extension of SSL which gives 92.2 + 0.3% accuracy, vs 88.6% in the case of 40 labelled examples in the original Fix.Matc paper. We confirm the importance of data curation for SSL on real data from Galaxy Zoo 2 (also see Cozman
ct al.
2003 Oliver al. 2018 Chen et al. 2020 Guo al. 2020
BACKGROUND
The intuition behind low-density separation objectives for semi-supervised learning is that decision boundaries should be in low-density regions away from both labelled and unlabelled data. As such, it is sensible to "repel" decision boundaries away from labelled and unlabelled datapoints and this can be achieved by making the classifier as certain as possible on those points. This happens au- tomatically for labelled points as the standard supervised objective encourages the classifier to be as certain as possible about the true class label. But for unlabelled points need a new objective that encourages certainty, and we focus on two approaches. First, and perhaps most direct is entropy minimization (Grandvalet & Bengio, 2005)
Cueg entropy (X)
Σ (x( (ogP,(X) BEY
where X is the input, y is the on particular label and y is the set of possible labels. Here, have followed the typical probabilistic approach in writing the negative entropy as an objective to be maximized. Alternatively, we could use pseudo-labelling, which takes the current elassification, y", to be the true label, and maximizes the log-probability of that label (Lee 2013).
Cpucadu (X) log Pg- (X)
y = arg max log (X). yEy
(2)
Leel (2013) regarded pseudo-labelling as closely related entropy miminization as the optimal value of both objectives is reached when all the probability mass is assigned to one class. However, they are not formulated as principled log-likelihood, which gives rise to at least three problems. First, these methods cannot be combined with other principled statistical methods such Bayesian inference. Second, it is unclear how to combine these objectives with standard supervised objectives, except by taking a weighted sum and doing hyperparameter optimization over the weight. Third, these objectives risk reinforcing any initial poor classifications and is unclear whether this is desirable.
2.1
IN STANDARD SUPERVISED LEARNING, UNLABELLED POINTS SHOULD BE UNINFORMATIVE
It is important to note that under the standard supervised-learnieg generative model unlabelled points should not give any information about the weights. The typical supervised learning setup assumes that the joint probability factorises as,
P
(3)
2

---

Published as a conference paper at ICLR 2023
A
more Invariant NN
loss Invartant NN
true boundary
E [pyomrjp (400))
22
-4 -3
0
2
Xp
Figure 1: A. A toy dataset generated to illustrate the dangers of using the clustering of the input points to inform classification boundaries. The input features, zg and æ1 are plotted on the and y- and the class is represented by colour. B. A schematic diagram demonstrating the effect of our principled likelihood incorporating anttettgmenttiionn on the certainty of predictions for different degrees of invariance. More invariant NNs (left) give similar predictive distributions for different augmentations (blue), and hence a certain averaged predictive distribution (bottom; orange). Less invariant NNs (right) give different predictive distributions for different augmentations (blue), and hence highly uncertain averaged predictive distributions (bottom; orange).
as the prior over weights, is usually chosen to be independent of the inputs (e.g. IID Gaussian). Thus, X and 0 are marginally independent and we cannot obtain any information about from X alone. Formally, the posterior over conditioned on X is equal to the prior,
P (0IX) P (0, Evey (0, P x,0, (X) UEyy P PY- (X)
-P (0).
((X)
P(X)
1 1 EyEy P(Y=y004 X). To confirm this result intuitively sensible, note that there many situations where encouraging the decision boundary to lie in low density regions would be very detrimental to performance. Consider a classifier with two input features: 2y and ø (Fig. [A). The class boundary lies in the high- region crossing both clusters, sult, the classifier should ignore the low-density region lying between the clusters. However, strong low-density separation SSL. terms in the objective may align the cluster boundaries with the class boundaries, lcading the classifier to wrongly believe that one cluster is entirely one class and the other cluster is entirely the other class. In contrast, supervised learning without SSL will ignore clustering and obtain a reasonable answer close to the grey dashed line. Importantly, this is just an illustrative example demonstrate that without further assumptions, the standard supervised ap- proach of ignoring unlabelled data is sensible; semi- coni--upervieed learning without loss of performance in such settings has been studied and is known as Safe SSL (Li Zhou 2014 Krijthe & Loog 2014
obtain reasonable
Kawakita & Takeuchi 2014
Loog 2015
Krijthe & Loog 2016 Guo & Li. 2018
Li et al. 2019).
2.2 A GENERATIVE MODEL DATA CURATION
Here, we give details on the generative model of data curation from Aitchison 2021 The key obser- vation behind this model is that the dataset curators for image datasets like CIFAR-10 and ImageNet were very careful to include only images with clear and unambiguous class label. For instance, for image datasets like CIFAR-10 10 and ImageNet, annotators were instructed that "It's worse to include one that shouldn's t be included than to exclude one", and Krizhevsky (2009) "personally verified every label submitted by the annotators". number of Amazon Mechanical Turk annotators agreed upon the class before including an image in the dataset. Remarkably, this means that not all images have a valid class label. For instance, an image of a radio would not have valid CIFAR-I0 class-label, and hence it would have been rejected from the CIFAR-IB 10 dataset. And further, because an "intelligent" decision has been made on whether not include the datapoint in the dataset, it might actually be possible extract useful information from merely inclusion of the point in the dataset (which is precisely what SSL does).
creating ImageNet, [Dng al. (2009) made sure that
3

---

Published as a conference paper at ICLR 2023
deciioon boundary conGenBUE: class
na consersus
COTGSTISUS class
8
2
4
2
30
The generative model of data curation applied to a simple 2D dataset. Data from each class sampled from different Gaussian, and the true decision boundary (green dashed line) was given by the posterior probability of class given (æo, æ1). A Datapoints far from the decision boundary are unambiguous, so annotators agree and consensus is reached (red and blue points), Datapoints close to the decision boundary are ambiguous, so consensus is not reached (grey crosses). The consensus datapoints thus exhibit artificially induced low-density separation. benchmark datasets such as CIFAR-10, the unlabelled points (yellow triangles) are selected from the consensus points (red or blue points) as the noconensus points are not available. The unlabelled points therefore also exhibit artificially induced low-density separation.
Figure 2:
When using
Formally, /Aitchisn (021) considers simplified generative model of consensus-formation between multiple annotators, in Deng et al. (2009). In particular, for any image, X, S human annotators, indexed i, are asked to give label, [Y,] (e.g. using Mechanical Turk). Every annotator is forced to label every image and if the image is ambiguous they are instructed to give random label. If all the annotators agree, Y1 yı=Y==->
=Ys. they have consensus and the datapoint is included in the dataset. However, in the case of any disagreement, consensus is not reached and the datapoint is excluded. Concretely, the final label, is Y; (which the same as all the other labels) consensus was reached and Undef otherwise,
Yi Y
=Y= =Ys
Undef otherwise
Taking J' to be the label set, we have Y, E D, and the final label, Y, could be any the underlying labels in D, or Undef f if consensus is not reached, Y E y {Undef). When consensus was reached, the likelihood is,
P (Y (Y= = ((Y=yy) 9) = II P (Y/=)/X,0)
= =P - 8)5 (x)()))
(5)
where have assumed annotators are IID, and px( P (Y, = X, is the single-annotator probability.
3 METHODS
Now, we apply this model of data curation to SSL. The key question is what likelihood to use for unlabelled points. To understand this, remember that SSL methods are usually applied to benchmark datasets such as CIFAR-10 or ImageNet; these datasets were first carefully curated during the la- belling process, implying that all points the dataset have reached consensus. Critically, typical benchmarks, unlabelled points are obtained by taking labelled points (which have reached consen- sus) and throwing away their labels (Fig. B). Under the generative model of consensus described above, we can obtain a probability of consensus by summing over class labels,
P (Y X = Eyey (py(S))
This probability is close to 1 (for S > class, and the probability is smaller if the mass is spread out over classes. As such, the
1) if the underlying distribution, (p, (X)) puts most of its
4

---

Published as a conference paper at ICLR 2023
likelihood "repels" decision boundaries away from unlabelled points, which is the common intuition behind low-density separation SSL methods, and which should be beneficial if class boundaries indeed lie in regions of low probability density away from both labelled and unlabelled points.
noconsensus images are observed, we can include a likelihood term for those images.
(Y UUdefn (x x, = P (Y Unde 0) = =1 Eyey (Py(X))
If noconsensus images are not observed, we could in principle integrate over the underlying distribu- tion over images, (X). However, we do not even have samples from the underlying distributions over images (and if we did, we would have the noconsensus images so we could use Eq. 7 As such we omit this term, but the use of out-o--distribution (OOD) datasets as surrogate noconsensus points an important direction for future work.
3.1 ENTROPY MINIMIZATION AND PSEUDO-LABELS ARE LOWER BOUNDS ON OUR PRINCIPLED LOG-LIKELIHOOD
Now, we try to understand whether our log-likelihood for noconsensus points bears any relation to previous successful methods for low-density separation based SSL. In particular, we begin by proving that entropy minimization forms a lower-bound on our log-likelihood (Eq. [6) First, we write the log-likelihood of consensus as an expectation over labels, y,
log P (Unddef) 0) log
Σ
(X) (x)( log (x) (1))()))
(8)
yEy
Applying Jensen's inequality, the negative entropy gives a lower-bound on our log- ligelikkd
log P (Y/Undef]X,, 2 (x) [log (p, (P)(
= (S - 1)E Pa (X) log Py, Px) = =(S Lng entropy (X)
(9)
This bound is tight for a uniform predictive distribution,
log P (Y/Undef] X, 0) = llog Eyey (P)( = =log S (3)
= - -1 1) gogg =(S (S- Lngg entropy (X)
(10)
Pseudo-labelling forms alternative lower bound on the log-likelihood which is obtained by noting that all (Py( are positive, so selecting any subset of terms in the sum gives lower bound,
log P (Unde X,0) = log EyEy (x)(
> log (py- (X)) Slog Py (X) SCpscudo
(11)
The inequality holds if we choose y² to be any class, but will tightest if we choose the highest probability class. This bound is tight for predictive distribution that puts all its mass on y", so py (X) = and Pyzv"
S log P (ndde X, 0) = log E,Ey (p, (x)) = log (xlos=
Sppeeuo (X) = S log (X) = Slog = 0.
As such, entropy minimization and pscudo-labelling optimize different lower-bounds on our prin- cipled log-likelihood, log ff 0), which gives a potential explanation for the effec- tiveness of pseudo-labelling and entropy minimization. Additionally, low-density separation SSL objectives encourages class-labels to be more certain. We can therefore expect pseudo-labelling to be the more relevant bound, as that bound is tight when the predictive distribution puts all its mass onto one class. In contrast, the entropy maximization bound is tight when the predictive distribution is uniform, which is discouraged by all low-density separation SSL objectives. This provides po- tential explanation for the use of psuedo-labelling rather than entropy regularisation in modern SSL approaches such (Sohn al. 2020
5

---

Published as a conference paper at ICLR 2023
3.2 DATA AUGMENTATION PRIORS AND FIXMATCH FAMILY METHODS
The simple bounds above do not include data augmentation. However, modern FixMatch family methods combine data augmentation and pseudo-labelling- To understand FixMatch as bound on a principled log-likelihood, we therefore need a principled account of data augmentation as a likelihood. Inspired by and Wenzel et al. (2020) (their Appendix K) and Nabarro et al. (2022) we consider a distribution, P (X'IX), over augmented images, X', given the underlying unaugmented image, X. We choose the single-annotator predictive distribution as the average over predictive distributions for many different augmented images,
Paug (Y; _ 0) = E (x,(X') X]
(12)
where (X') is the predictive probabilities resulting from applying the neural network to the aug- mented image, and remember {1................. S) indexes the annotator. To obtain tractable objective
in the supervised setting, Nabarro et al. (2022) recommend using a multi-sample version of Jensen's s inequality, with K augmented images denoted X
log Pang (Y; - X, 0) 2 E [log æ (X))(X
(13)
Combining this single-annotator probability with our generative model of curation, we obtain,
log P aug (Y y yx, x) S log P (Y, y X, 0) S log E [X,
S E loog R ExPr (XX)
(14)
The resulting objective for unlabelled points is, logPaug Paug (Unuddef[X)) (X, 0) = log logg Evey Paug (Y (Y = log EvEy E [D (X')
2 log Ewe>> ( (x)) (15)
where we approximate the expectation with K different samples of X', denoted X Unfortunately, this approach does not immediately form bound on the log-likelihood due to the convex nonlinear- ity in taking the power of S. Nonetheless, one key problem with approximating machine learning losses is that the optimizer learns to exploit approximation errors to find pathological solution that makes the objective unboundedly large. We appear to be safe from that pathology here, we are simply forming predictions by averaging over K augmentations of the underlying image. Nonethe- less, to form a lower bound, we can follow FixMatch family algorithms by pseudo-labelling, i.e. by taking only one term in the sum for class y² FixMatch chooses y² by using the highest-probability class for weakly- bound, i.e. arg max, K (Xp) In either case,
image. An alternative approach is to choose the y" giving the tightest
log P aug (Y [UXunde]|X, 0) 2 log lp (X')) 2 S [log (X:))(((()X))
(16)
If K = 1 and y² is chosen using a separate "weak" augmentation, then this is exactly equal to the FixMatch objective for unlabelled points.
Note that both of these objectives (Eq. 15 and 16) promote reduced predictive uncertainty. Im- portantly, this does not just increase confidence in the single-augmentation predictive distributions, (x)),, but also increases alignment between the predictive distributions for different augmenta- tions (Fig.[B) In particular, the single-augmentation predictives are all highly confident, but place that high- high-confidence on different classes, then the multi-augmmentaiinn predictive formed by averag- ing will have low-confidence (Fig. [ right). The only way for the multi-auggenttaiin predictive to have high confidence is if the underlying single-augmentation predictive distributions have high confidence in the same class (Fig. [ left), which encourages the underlying network to become more invariant. This makes sense: if fata-auumentttion changes the class predicted by the neural network, then any predictions should be low confidence. And it implies that combining principled data augmentation with generative model of data curation automatically gives rise to objective encouraging invariance.
3.3 A BAYESIAN EXTENSION OF SSL
Once we have derived principled likelihood for SSI., P (Y XX 0). we can combine that likelihood with standard priors over Bayesian neural network weights, P (0), (in particular, we use independent
Gaussians), and standard inference methods, specifically SGLD (Welling & Teh 201I). This allows us to draw samples from the Bayesian posterior, P (0)X,0) 0)
P Y) x P (Y X, (0).
(17)

---

Our work opens the possibility of combining Bayesian neural networks with SSL, We therefore de- signed Bayesian variant of FixMatch, using the bound on the likelihood in Eq. (6)), where we take y" from the weak augmentation and take the probabilities from the strong augmentation (remember Published as a conference paper at ICLR 2023
4
test log reg-ilkllhhodd
-0.1
teat accuracy
95
-0.2
85
1000
1000
unlabelled points
unlabelled points
Figure 3: Test log- ligeliieldoodd (A) and accuracy (B) for Langevin sampling for Bayesian SSL on toy datasets sampled from the model as a function of the number of unlabelled points. Note the lines for different values of S are not directly comparable, as they use different data: each line uses data curated with a different value of 5, and we increase we curate more aggressively. More aggres- sive curation will improve performance it eliminates ambiguous difficull-to-classify points.
RESULTS
We began by giving proof-of proof-of-principle principle for a Bayesian extension of SSL on toy generated dataset, then we assessed the performance of Bayesian SSL on CIFAR-10 10. Finally, we confirmed the impor- tance of curation for low-density separation based semi-supervised learning using Galaxy Zoo 2 as this was a real-world dataset which allowed us to generate matched curated and uncurated datasets.
4.1 BAYESIAN SSL ON GENERATED DATASET
Our formulation of SSL likelihood implies that it should be possible take entirely novel approaches, such as using low-density separation SSL a Bayesian neural network (BNN). We considered toy dataset generated from "true" neural network model with one hidden layer and 30 hidden units, 5 dimensional inputs and 2 output classes. We did Bayesian inference in a network with the same architecture, with independent Gaussian prior over the weights, with prior variance 1/fan-in, which is necessary to ensure that the outputs have sensible scale. We generated inputs IID from a Gaussian, then passed them through the "true" neural network. then sampled multiple categorical class-labels corresponding to different annotators. If all the simulated annotators agreed, consensus was reached and if any simulated annotators disagreed, consensus was not reached. We used 100 labelled datapoints, though not all of them will have reached consensus, and we used up to 1600 unlabelled points, though again not all of them will have reached consensus. Note that as the consensus/noconseesss status of point arises from the generative model, we cannot independently specify the number of consensus/nocoonsesuu points. We used Eq. ( as the likelihood for labelled points, Eq. (6) the likelihood for unlabelled points and Eq. points. We sampled (and trained networks on) 500 datasets in parallel. We trained using I Langevin dynamics with all data simultaneously (no minibatching) with momentum and rejection.
the likelihood for noconsensus
For generative model with 5 1, consensus always reached and the problem equivalent to standard supervised learning. As such, found no benefits from including unlabelled points for = 1. In contrast, for any setting of S > 1 we found that increasing the number of unlabelled points improved the test log- likelihood (Fig. AA and the test accuracy (Fig. B))
4.2 BAYESIAN SSL ON CIFAR-10
the bound is valid for any choice of y)) We used the same network, augmentation (RandAugment) and optimization schedule as in the original FixMatch paper, with the only modifications being to use 300 epochs rather than 1024 epochs, due to time constraints, and to add noise to the gradient up- dates. These noisy gradient updates convert SGD principled Bayesian inference method, SGLD (Welling & Teh 2011). We used an isotropic Gaussian prior over the weights with variance 10 1 as that roughly corresponded the usual 1 1/fan-in variance initialisation, and chose temperature

---

Published as a conference paper at ICLR 2023
Accuracy of standard vs Bayesian Fix for different numbers of unlabelled points. Three seeds were used for all FixMatch and Bayesian FixMatch experiments except for FixMatch with 40 labelled points, where we used 12 seeds because the noise was much higher than that for any other setting. Bold indicates statistically significant improvements.
Table 1:
number of labelled points:
40
250
4000
FixMatch
88.5 + 0.7
93.9 + 0.2
94.6 + 0.1
Bayesian FixMatch
92.2 + 0.3
93.9 + 0.1
94.9 + 0.0
FixMatch (unlab CIFAR-100)
41.35
62.63
86.66
of = 1003 Wenzel et al. 2020 Aitchison 2021). The accuracy was computed by averaging over the last 100 epochs. As expected, found that Bayesian Fix.Maach performed considerably better with the smallest number of unlabelled points (including better than the original FixMatch paper result of 88.6%), where uncertainty was highest, and gave similar performance otherwise (Tab. I Finally, we also tried training on on labelled points from CIFAR-10 and unlabelled points from CIFAR-100. As expected, we found catastrophic drop in performance (Tab. I
4.3 GALAXY Zoo 2
Our data curation based likelihood predicts that low-density separation based SSL should be much more effective curated than uncurated data. To test this prediction on real-world data, turned to Galaxy Zoo 2 (GZ2) (Willett et al.) 2013) which uses images from the Sloan Digital Sky Survey. This dataset is particularly useful for us as it has received only very minimal filtering based on criteria such as object brightness and spatial extent. We defined 9 labels by truncating the complex decision tree followed by the annotators (for further details see Aitchison 202T). Further, as each G72 image has received ~ 50 labels, we can define a consensus coefficient by taking the fraction of annotators that agreed upon the highest probability class. We can then define a curated dataset by taking the images with consensus coefficient above some threshold within each class. Note that needed to select images on a per-class basis, because annotators tend to be more confident on some classes than others, taking the highest consensus coefficients overall would dramatically change the class balance. In particular, we used the top 8.2% images, which gave a full curated dataset of randomly selected 2000 labelled examples, 10000 test
just over 20,000 images. Of those, examples, and - 6000 as unlabelled examples. The images were preprocessed by center-cropping 212 212 and then scaled to 32 x 32. We applied a FixMatch-inspired semi-supervised learning algorithm, with a standard supervised objective, with unlabelled objective given by Eq. (15) with
K = 2 and = Data augmentation was given by vertical and horizontal flips, rotations from - 180 to 180% translations by up to 40% on both axes and scaling from 20% to 180%. We trained ResNet18 with our maximum likelihood objective using SGD with batch size of 500, a learning rate of 0.01 and 1500 epochs. We used an internal cluster of vVdia 1080 and 2080 GPUs, and the experiments took roughly 300 GPU hours.
We found that the test-log- best-log-likelinng likelihood for curated data improved slightly as more unlabelled points were included, whereas the sost-loglikellhodd for uncurated dramatically declined as unlabelled points were added (Fig. AA-B). We saw strong improvements in test accuracy with the number of unlabelled points for curated datasets (Fig. 4C-D). Note that in Fig. 4C the error rate for curated datasets is already very small, so initial test error (Fig. (.. For uncurated data, the inclusion of large numbers of unlabelled points dramatically worsened performance, though the inclusion of small number of unlabelled points gave very small performance improvements (Fig. (C-D). Thus, this experiment is consistent with the idea that the effectiveness of SSL arises at least in part from curation of the underlying dataset.
any effect needed plot the test error, normalized to the
5 RELATED WORK
There are least three main approaches to semi-supervised learning (Seeger 2000 Zhu 2005 Chapelle et al.) 2006 Zhu & Goldberg. (009). First there is low-density separation, where we assume that the class boundary lies in region of low probability density away from both labelled
8

---

Published as a conference paper at ICLR 2023
4
lest log glliilhhhodd C
exact
test log log-likeihood
Norm test error
1.0
uncurated
test error (%)
pseudo
uncurated
-2"
50
exact
curated
pseudo
0.0
curated
0K
3K
0K
3K 6K
3K
8K
0K
3K
unlabelled points
unlabelled points
unlabelled points
unlabelled points
Figure 4: Test log-likelihood and error for curated and uncurated GZ2 datasets as a function of the number unlabelled points. Exact corresponds to Eq. (15) (which exact in the limit as K oo) and pseudo corresponds to the pseudo labelling version of the augmented objective (Eq. 16 Note the green line (exact curated) is under the red line in A-C.
and unlabelled points. This approach dates back at least to transductive support vector machines (SVMs) where the model be tested on a finite number of known test locations (Vapnik) [998] Chapelle et al. 1999). Those known test locations are treated as unlabelled points, and we find the decision boundary that perfectly classifies the limited number of labelled points, while at the same time being as far as possible from labelled and unlabelled data. Alternative approaches include
pseudo-labelling and entropy minimization (Grandvalct & Bengio 2005 Lee 2013). Second, there graph- based methods such as (Zhu & Ghahramani 2002) which very different from the methods considered here. Third, there are approaches that use unlabelled points to build a generative model of the inputs and leverage that model to improve classification (e.g. Kingma al., [014 Odenal 2016): Gordon & Hernández- Lobato (017). This approach was originally explored in a considerable body of classical work (e.g. McLachhan 1975 Castelli & Cover 1995 Druck al., 2007) for a review, see [Seeger (2000) and references therein. These approaches are fundamentally different from the SSL approaches considered here, as they require generative model of inputs, while low-density separation methods do not. Generative modelling can be problematic as training a generative model can more involved than training a discriminative model and because the even when the model can produce excellent samples, the high-level representation may be "entangled" HHggins al.) 2017) in which case it may not offer benefits for classification.
DISCUSSION AND CONCLUSION
In conclusion, we showed that low-density separation SSL objectives can be understood as lower- bound on log-probability which arises from a principled generative model of data curation. This allows us to develop Bayesian SSL, which performs better settings where there are a very small number of unlabelled points.
Our theory provides a thcoretical understanding of past results showing that SSL is more effective when unlabelled data is drawn from the original, curated training set (Cozman et al. 2003 Oliver ett al,, 2018 Chen et al.) 2020 Guo et al.] 2020). In the extreme, our theory might be taken to imply that if data has not been curated, then SSL cannot work, and therefore that low-density sepa- ration SSL methods will not be effective in messy, uncurated real-world datasets. However, this is not the complete picture. Low-density separation SSL methods, including our log-likelihood, fun- damentally exploit class-boundaries lying in low-density regions. As such, low-density separation could equally come from the real underlying data could be artificially induced by data curation (Fig. 2) None of these methods are able to distinguish between these different underlying sources of low-density separation and as such any of them may work on uncurated data where the underly- ing distribution displays low-density separation. However, the possibility for curation to artificially induce low-density separation does imply that we should cautious about overinterpreting spec- tacular results for low-density separation based SSL obtained on very carefully curated benchmark datasets such as CIFAR-10.

---

Published as a conference paper at ICLR 2023
REFERENCES
Aitchison, L. A statistical theory of cold posteriors in deep neural networks, ICLR, 2021.
Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. Remix- match: Supi-super learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: /9/ 09788,, 2019a.
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. Mixmatch:
A holistic approach to semi-supervised learning. In Advances in Neural Information Prucessing Systems, pp. 5050-5060, 2019b.
Castelli, V. and Cover, T. M. On the exponential value of labeled samples. Pattern Recognition Letters, ((1):105-111, 1995.
Chapelle, O., Vapnik, V., and Weston, J. Transductive inference for estimating values of functions. In NIPS. pp. 421-427. Citeseer, 1999.
Chapelle, O., Scholkopf, B., and Zien, A. semi-supervised learning. 2006.
Chen, Y., Zhu, X., Li, W., and Gong, S. Semi-supervised learning under class distribution mismatch. In AAAI, pp. 3569-3576. AAAI Press, 2020.
Cozman, F. G., Cohen, I., Cirelo, M. C., et al. Semi-supervised learning of mixture models. In ICML, volume 4, pp. 24, 2003.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248- 255. Icee, 2009.
Druck, G., Pal, C., McCallum, A., and Zhu, X. Semi-supervised classification with hybrid genera- tive/discriminative methods. Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 280-289, 2007.
Gordon, J. and Hernández-Lobato, J. M. Bayesian semisupervised learning with deep generative models. arXiv preprint arXiv:1706.0977 2017.
Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 529-536. 2005.
Guo, L., Zhang, Z., Jiang, Y., Li, Y., and Zhou, Z. Safe deep semi-supervised learning for unseen- class unlabeled data. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 3897-3906. PMLR, 2020.
Guo, L.-Z. and Li, Y.-F. general formulation for safely exploiting weakly supervised data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Higgins, I. Matthey, L., Pal, A.. Burgess, C.. Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.
Kawakita, M. and Takeuchi, J. Safe semi-supervised learning based weighted likelihood. Neural Networks, 53:146-166 2014.
Kingma, D. P., Mohamed, S., Jimenez Rezende, D., and Welling, M. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27:3581-3589, 2014.
Krijthe, J. H. and Loog, M. Implicitly constrained semi-supervised linear discriminant analysis. In 2014 22nd International Conference on Pattern Recognition, pp. 3762-3767. IEEE, 2014.
Krijthe, J. H. and Loog, M. The pessimistic limits and possibilities of margin-based losses in semi- supervised learning, arXiv preprint arXiv:16/2.08875, 2016.
Krizhevsky, A. Learning multiple layers of features from tiny images. Tech. report, 2009.
10

---

Published as a conference paper at ICLR 2023
Lee, D.-H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. JCML, volume 3, pp. 2, 2013.
Li, Y.-F. and Zhou, Z.-H. Towards making unlabeled data never hurt. IEEE transactions on pattern analysis and machine intelligence, 37(1):175- 1888 2014.
Li, Y.-E., Guo, L.-Z.., and Zhou, Z.-H. Towards safe weakly supervised learning. IEEE transactions on pattern analysis and machine intelligence, 43(1):334-346, 2019.
Loog, M. Contrastive pessimistic likelihood estimation for semi-supervised classification. IEEE transactions on pattern analysis and machine intelligence, 38(3)4662-45,, 2015.
MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003.
McLachlan, G. J. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350): 365-369 1975.
Miyato, T., Maeda, S.-i., Koyama, M., and Ishii. S. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):1979-199 2018.
Nabarro, S., Ganev, S., Garriga-Alonso, A., Fortuin, V., van der Wilk, M., and Aitchison, L. Data augmentation bayesian neural networks and the cold posterior effect. In Uncertainty Artifi- cial Intelligence, pp. 1434-1444. PMLR, 2022.
Odena, A. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583, 2016.
Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., and Goodfellow, I. Realistic evaluation of deep semi-super learning algorithms. In NeurlPS, pp. 3239-3250, 2018.
Sajjadi, M., Javanmardi, M., and Tasdizen, T. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-11711 2016.
Seeger, M. Learning with labeled and unlabeled data. 2000.
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., and Raffel, C. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07683 2020.
Van Engelen, E. and Hoos, H. H. survey on semi-supervised learning. Machine Learning, 109 (2):373 440, 2020.
Vapnik, V. Statistical learning theory. NY: Wiley, 1998.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Pro- ceedings of the 28th international conference on machine learning (ICML-1I), pp. 681-688 2011.
Wenzel, F., Roth, K., Veeling, B. S., Swigtkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R., and Nowozin, S. How good is the bayes posterior deep neural networks really? arXiv preprint arXiv:2002.02400. 2020.
Willett, K. W., Lintott, C. J.. Bamford, S. P., Masters, K. L., Simmons, B. D., Casteels, K. R., Edmondson, E. M., Fortson, L. F., Kaviraj, S., Keel, W. C., et al. Galaxy zoo 2: detailed morpho- logical classifications for 304 122 galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 435(4):2835-2))) 22600, 2013.
Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., and Le, Q. V. Unsupervised data augmentation for consis- tency training. arXiv: / /2848, 2019.
Zhu, X. and Ghahramani, Z. Learning from labeled and unlabeled data with label propagation. 2002.

---

Published as a conference paper at ICLR 2023
Zhu, X. and Goldberg, A. B. Introduction to semi supervised learning. Synthesis lectures artifi- cial intelligence and machine learning, 3(1):1-130, 2009.
Zhu, X. J. Semi-supervised learning literature survey, 2005.
12


Reviews:
[{"review_id": "m5fMHP8VBA", "reviewer": "Reviewer_3m8s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to formulate the low-density separation term in SSL as a log-likelihood term from a generation model of data curation. Based on which, a Bayesian modeling is possible, and some new insight on why SSL works for curated data set.", "summary_of_the_review": "Overall, I feel the paper is more like playing with a math trick, but can not deliver more insight. The low-density separation SSL assumes the decision boundary is not close to any data point, thus, even without any math, it implies that all data, including unlabelled data are away from decision boundary, i.e., they must be curated data. A mathematical formulate should provide us with more subtle insight of this matter, and this paper seems fail to do it.", "strength_and_weaknesses": "Strength: The likelihood for curated/uncurated data set seems interesting, and it successfully establishes a relationship to entropy minimization and  pseudo-labelling. \n\nWeakness: The construction of likelihood is too artificial, aiming to appeal low density separation idea. How to choose S? How to determine a labelled/unlabelled observation is curated or not, beyond common data sets such as CIFAR10? Unlike penalty term which are artificial designed to incorporate prior knowledge, likelihood should reflect data generation rule and tuning-free. \n\nOne experiment shows the improvement when unlabelled curated data is added, and the performance drop with more uncurated data. But this is not a convincing justification of the main of idea of the paper. The theory shows that log likelihood is lower bounded by the entropy or pseudo labelling loss under the generative model of curated data. It does not say what happens when the data is uncurated. To establish empirical justification, one also needs to show that the performance will increase with added uncurated data if likelihood (7) is used.\n\nThe paper claims it is a Bayesian approach, but it never mention anything about prior setting. For all the results displayed, are they based on posterior MAP or posterior mean?\n\nFigure 3, even when there is no unlabelled data, the test accuracy is different w.r.t. the choice of S. S=0 yields the best test accuracy. Please comment on this\n\nWhat is the test likelihood (Figure 3/4)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tz7l725teH", "reviewer": "Reviewer_JPM3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper looks at the topic of semi-supervised learning for classification, through a probabilistic lens. The starting point is the observation that many common SSL datasets (e.g. CIFAR) are assembled via multi-annotator agreement protocol (images are only included if many humans agree on their class label). This leads to a multi-annotator consensus generative model (previously described in Aitchison 2021). Eq. 6 gives this model's likelihood for an image to be a *consensus* image (e.g. images that would be included even in unlabeled set because many annotators agree).\n\nThe new thing in this paper is the realization that this likelihood can be lower bounded by many common SSL objectives, included entropy (Eq 9) or pseudolabel (Eq 11) or FixMatch (Eq 16). This gives a nice interpretation of what these objectives are doing (maximizing the likelihood of a model that is aware that consensus is needed).\n\nExperiments on toy data demonstrate properties of this model, including:\n\n1) Fig 3 suggests that if S =1 (multiple annotators are not needed), unlabeled data does not help under this model, while if S > 1 (several annotators need to agree), unlabeled data can help\n\n2) Tab 1 shows that a Bayesian version of FixMatch, using their view of FixMatch's loss as a likelihood, does better than FixMatch on CIFAR-10 with few labeled images\n\n3) Fig 4 shows that on Galaxy Zoo data, datasets that are *curated* show gains from unlabeled data while *uncurated* sets show losses not gains in classifier performance as more unlabeled data are added\n\n", "summary_of_the_review": "Overall I like the core ideas here: the observation that curation is behind many standard SSL datasets is potentially powerful though underused currently: the methods here suggest that multi-annotator curation can be exploited (if present) via SSL but that datasets with only single annotator curation may not be usefully pursued with SSL methods. \n\n<strike>\nHowever, there's too many writing clarity issues and overall puzzling results to give this a high score currently. I hope discussion period clarifies things, I'd be willing to raise my score given a satisfactory response.\n</strike>\n\n**Update after discussion**: Clarity issues have largely been resolved, and the past improvements suggest remaining ones will also be resolved in good faith. Given the significance of the contributions here, I would vote for and argue for acceptance.", "strength_and_weaknesses": "# Strengths\n\n* Highlights that curation (multi-annotator agreement) is behind many popular SSL benchmarks, which is less widely understood/used than it should be\n* Principled approach to defining likelihoods that correspond to bounds on popular SSL objectives\n* Showing how to use these likelihoods within a Bayesian model seems valuable\n* Analysis in Sec 2.1 and Experiments on toy data are convincing: if data comes from a single-annotator model, hopes for SSL are dim (any unlabeled data is uninformative!)\n\n# Weaknesses\n\n* Why does accuracy get worse as S increases in Fig 3 on toy data?\n* Use of a bound (not exact likelihood) in CIFAR experiments may hide role of augmentation\n* Unclear what prior is used in the Bayesian FixMatch\n* Use of a temperature in the Bayesian FixMatch result needs more explanation\n* Writing needs some work: Lots of abrupt transitions and confusing notation\n\nThese weaknesses are elaborated further below (under Quality/Clarity)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NCPA-Pufkcv", "reviewer": "Reviewer_y8jM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors have formulated Semi Supervised Learning (SSL) objective as a principled log-likelihood estimate in the context of a generative model of data curation. The distribution of data (in the classification context) has multiple or at least two levels of uncertainty: one is the distribution of the classes and the other is the labels themselves. The authors characterize the supervised setting as learning by using curated data labels (the lower bound of the SSL setting) and propose SSL as an extended case with curated plus non-curated and unlabelled data (coins the concept of undefined class while there is a disagreement on labels or no label available).\n\nThe proposed methodology has been tested on some synthetic, CIFAR, and Galaxy Zoo 2 datasets. Statistically significant improvements have been claimed over the FixMatch, a known technique in SSL.\n", "summary_of_the_review": "I have gone through the paper more than once; overall, the idea sounds good and the results support some of the claims of this research. If we exclude certain sections, the paper is well written. The experiment section is missing some details and therefore found little difficult to follow.\n\nThe reported results are found to be interesting and provide us some insights on the difficulty of non curated and unlabelled data. I think this work has some value if some of the above mentioned limitations can be resolved/improved.\n", "strength_and_weaknesses": "Strengths: The main idea of this paper is to derive a Bayesian formulation of the SSL setup by using labeled and unlabeled data. Using curated labeled data they have formulated/defined a lower bound of the solution. They also have proposed two likelihood estimates that are applicable to the curated labeled and unlabeled data and then combine them in a coherent fashion for model prediction.\n\nThe proposed methodology has been tested on CIFAR-10 and Galaxy Zoo datasets. Reported results are found to be better than FixMatch, especially when trained with less number of labeled data points (in the SSL mix).\n\nWeaknesses: This work lacks a proper Bayesian formulation of the SSL problem which requires selection of some suitable priors and a proper estimation of the posteriors. Given that unlabeled data class labels are unknown it adds additional complexity to the modelling. There has been some prior work through the definition and usage of a null category (similar to the definition of the undefined class in this paper) such as [1, 2] which may help improve the Bayesian aspect of the formulation.\n\n[1] https://papers.nips.cc/paper/2004/file/d3fad7d3634dbfb61018813546edbccb-Paper.pdf [2] http://www.bmva.org/bmvc/2011/proceedings/paper3/paper3.pdf)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JR2Y3b0cw4q", "reviewer": "Reviewer_fBnW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the fact that our common benchmark data sets are curated, in the sense of their data points being preselected by the consensus of a set of labellers. Aitchison (2021) already noted this and designed a proper likelihood that takes account of this fact. The authors build upon his work by moving into the semi-supervised setting where they show how to formulate two common approaches (entropy minimization, pseudo-labelling) as lower bounds of this likelihood. The approach is then evaluated in several experiments ranging from synthetic to real-world.", "summary_of_the_review": "While the paper contains some interesting formulations and results that are well worth further investigation in its current form it is rather incremental and contains several technical flaws (derivation and esp. wrt reproducibility) that need to be addressed.", "strength_and_weaknesses": "## Strengths\nThe paper is well argued for and shows clear results upon the baseline the authors are considering and demonstrates in varied ways their assumptions\n\n## Weaknesses\n- The strong emphasis on this being _Bayesian SSL_ seems to me to be too strong. While the proposed approach gives us a proper likelihood the method and approach lack any intrinsic \"Bayesianity\". Dropping the Bayes terms the paper could have just as well been formulated as a novel likelihood for a maximum likelihood approach. The authors themselves 'simply' use it in this function in their third experiment. \n- Inconsistent notation and minor errors (see below)\n- A lot of the hyperparameters and experimental details are missing, or partially available but then hidden in the provided code. E.g., Sec 4.1 mentions an architecture for creating the toy data set but lacks any information on whether the trained networks have the same architecture. Sec 4.3 lacks a complete discussion on how the authors arrived at their final nine labels, making replicability and reproducibility impossible. If the preprocessing follows Aitchison (2021) then please properly cite and discuss this fact. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster