Semi-supervised learning objectives as log-likelihoods in a generative model of data curation

Abstract: We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to give a proof-of-principle for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any ``out of distribution'' examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.

Reviews and author responses:

 I thank the authors for going through my review and responding to some of my concerns.

### Tightness of the bounds
I thank the authors for expanding the discussion on this point, which I believe adds to the relevance of these bounds.

### Model mismatch
I am not questioning the novelty or the complexity of the model. My point is that model mismatch is not a very insightful observation in this context. Given the known phenomenon that uncurated data is less useful for semi-supervised learning, any model that captures the data curation process well enough would allow us to draw the same conclusion that uncurated data hurts performance due to a misspecified model. In that sense, I fail to see a meaningful contribution in the experiments of section 4.2, as the claim that "data curation based theory predicts that low-density separation based SSL should be much more effective on curated than uncurated data" is technically correct but not very interesting. That is, it is not very surprising that unlabelled data is more effective when sampled from the same generative process as labelled data.

### Prior art
Indeed, the existence of other theoretical approaches addressing similar questions does not reduce the insights of the paper, but that does not mean the paper would not benefit from a discussion on competing explanations for the observed phenomena. In particular, I think my question about other factors that could make uncurated data difficult to model remains pertinent.

I only suggested the paper by Cozman et al. because I thought it provides a rigorous theoretical development proving why semi-supervised learning might lead to poor results. This is relevant in this context and contrasting their results with the ones in this paper could be enlightening. I was not just asking for a citation for those authors and the way their work is currently cited in the latest version of the paper is inaccurate, as Cozman et al. do not talk about curated data.

### Parameters of the true data generating process
I know that mathematically $\theta$ is an arbitrary set of parameters, but it might help the exposition to introduce it in the text, especially if one wants to convey the idea of model mismatch.

### Section 4.1
My question was exactly because I found this part of the text unclear.

Overall, I still find that framing common SSL objectives as log-likelihoods is potentially interesting. However, I find the insights provided in the paper unconvincing or poorly connected to the theoretical results. For instance, the results seem much more promising for studying Bayesian SSL than for explaining why uncurated data is less amenable to traditional SSL methods, but the paper focus on the latter rather than the former. All in all, I remain unconvinced of the relevance of the results in the paper and, while I remain open to change my mind in the remaining of the review process, for now I maintain my original review.

 For the record, giving a science lecture to reviewers who assign low scores to one's work might not come across as very humble or might even be seem as a distraction from the main criticisms of the paper. In fact, novelty was not the main concern of my review—I mentioned it only in passing when describing a positive point.

 Regarding the list of the 3 requested items from Reviewer vb5u above, as a separate reviewer, my views are:

1. The contribution is not the model itself, but the interpretation of a model not originally designed for SSL as a powerful way to explain when SSL methods like pseudolabel will be effective

2. The current model in the paper is *specifically designed* to match a data curation process in which multiple annotators need to agree. Multiple annotator agreement is the process used to assemble CIFAR and ImageNet, as argued in the current revision's Sec. 2.1

3. The submitted paper *does* develop the Bayesian view of the proposed model. See Fig. 5 of the latest revision. 

I think what the reviewer is asking for has been adequately provided already. I'm happy to discuss more if there's any confusion or further concerns.

 Just wanted to stop by to say I think most of my questions / concerns (like W1) have been addressed.  I continue to like this work and advocate for its acceptance at this venue.

A few quick responses:

### RE discussion of bound tightness

Thanks for this revision, this is just what I was looking for.

### RE Q3 "the dip"

Agreed on reflection now that while it's interesting to try to explain, it's not essential to focus on it too much.

### New updated bounds and tightness arguments

Looks good! FYI a few equations (like 10 and 13) are hard to distinguish what is inside the log vs outside

Current
$$
\log S \frac{1}{S}^S 
$$

Suggested:
$$
\log \left( S \frac{1}{S}^S \right)
$$

### RE new Fig 2 and Fig 3
I like the new diagrams and captions here much better, thanks! FYI Fig 3 has a typo in the caption:"noconensus"


 We absolutely agree that showing how to do Bayesian SSL would be significant.  We did just that in Fig. 5 of the main text of the original submission, and we emphasised this result in the original abstract:
> We are thus able to give a proof-of-principle for Bayesian SSL on toy data.

The extension to a large scale example is of course trivial (merely a change to the network architecture).  

Remember that Bayesian inference involves combining a prior, a likelihood, and an inference method.  We know how to do inference in neural networks (e.g. using SGLD), and we have sensible neural network priors.  What we don't know is how to specify an SSL-like objective as a likelihood, which can then be incorporated into Bayesian inference.  It is this difficulty that we resolve.

We also agree on the importance of ensuring that the generative model was accurate and appropriate model of the true data generating process.  Which is why the experiments (e.g. for GZ2 in Fig. 6) focus on exactly this question.  We confirmed that the generative model was an accurate and appropriate model of the true data generating process, and therefore no further changes to the generative were necessary.

 ### Tightness of the bounds
We have added a discussion of those bounds in the text. The entropy minimization bound is tight for a uniform predictive distribution (corresponding to maximal uncertainty) and the pseudo-labelling bound is tight if the predictive distribution puts all its mass onto one class (corresponding to maximal certainty).  This gives us insight into which bound might be most suitable in practice.  In particular, low-density separation SSL objectives encourage predictive distributions to be more certain, suggesting that the pseudo-labelling bound will be more relevant in practice.  And indeed, pseudo-labelling is used in modern approaches such as FixMatch.

### Model mismatch
One of the key insights of the framework is that uncurated data indeed corresponds to the well-understood problem of model mismatch.
And the fact that model-mismatch is well-understood is a massive advantage of the framework.
As scientists, our goal is to build a simple framework that explains as diverse a range of phenomena as possible.
We could do a whole bunch of complex math, hugely specialised to this particular problem. 
But that wouldn't help us to "build a simple framework that explains as diverse a range of phenomena as possible".
But reducing uncurated data in SSL to the known, well-understood problem of model-mismatch does help.

### Parameters of the true data generating process
Almost everywhere $\theta$ appears, it is used in the likelihood, $P(Y|X, \theta)$.  This is the probability of $Y$, conditioned on a particular value of $X$ and $\theta$.  You can think about it as a function that maps from $X$ and $\theta$ to a distribution over $Y$ (or if $X$ and $Y$ are fixed by the data, you can think of it as a function that maps $\theta$ to a scalar probability).  And because it is like a function, it doesn't really make sense to ask whether $\theta$ is "truly" sampled from the prior, or learned.  It could be either (or neither, e.g. part way through the optimization).  In summary, $\theta$ is really just an argument to a function, and we're reasoning about properties of the function.

### Prior art.
The original manuscript contained a discussion of previous empirical results (Oliver et al., 2018; Chen et al., 2020; Guo et al., 2020) and theoretical results surrounding Safe SSL (Li & Zhou, 2014; Krijthe & Loog, 2014;Kawakita & Takeuchi, 2014; Loog, 2015; Krijthe & Loog, 2016).
We have added your [2] to these discussions.
To reiterate, the claim we make in the title and first sentence of the abstract is:
> We  currently  do  not  have  an  understanding  of  semi-supervised  learning  (SSL)objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL.

This exercise happens to provide a simple, elegant account in terms of model mismatch for why uncurated data gives worse performance when used in semi-supervised learning objectives.
The existence of other, very different and much more complex theoretical approaches to this question does not reduce the insights given by our work.

### Section 4.1
The original manuscript gave a comprehensive overview of the approach:
>We considered a toy dataset generated from a "true" neural network model with one hidden layer and 30 hidden units, 5 dimensional inputs and 2 output classes.
We generated inputs IID from a Gaussian, then passed them through the "true" neural network, then sampled multiple categorical class-labels corresponding to different annotators.
If all the simulated annotators agreed, consensus was reached and if any simulated annotators disagreed, consensus was not reached.
We used 100 labelled datapoints, though not all of them will have reached consensus, and we used up to 1600 unlabelled points, though again not all of them will have reached consensus. 
Note that as the consensus/noconsensus status of a point arises from the generative model, we cannot independently specify the number of consensus/noconsensus points.  We used Eq. (2) as the likelihood for labelled points, Eq. (6) as the likelihood for unlabelled points and Eq. (7) as the likelihood for noconsensus points.  We sampled (and trained networks on) 500 datasets in parallel.  We trained using Langevin dynamics with all data simultaneously (no minibatching) with no momentum and no rejection

We have added the word "categorical" to this description  

### Section 5.1
There is no Sec. 5.1 (Sec. 5 is the related work). If you meant Sec. 4.1, please see the extensive description in the paragraph quoted from the manuscript in the previous answer.

### Eq. 5
We agree that for certain readers, Eq. 5 won't add much.  Given that this a slightly odd conceptual point, we do feel it is worth driving home with a formal derivation.

 So, if the test likelihood is negative of cross entropy, then the insight from the Fig. 6 is that as we include more curated data, the classification becomes better even with unlabeled data, but that is not the case with the uncurated data. But, this does not clearly support the lower bound obtained theoretically. Also, the lower bounds have nothing to say about uncurated data.

 Certainly, the Bayesian methods and probabilistic methods have made significant impact on the machine learning field. There is no doubt about that.  However, the contribution of this paper is not clear:
1. The probabilistic formulation of data generating process is due to Aitchison, 2021. It is not the contribution of this paper.
2. It would be better if you could show how to modify the generative model to match the true data generating process.
3. It would be significant if you could show how to obtain the Bayesian version and reason under uncertainty.

In the current form, the paper talks about how to interpret the operations commonly used in SSL as a lower bound on log likelihood. The connection to Bayesian methods or uncertainty is not very clear. It is not clear how we could use these ideas with uncertainty. There is no connection with Bayesian ideas like posterior distribution computation or integrating out the distribution etc. 


 Thanks for your comments.

### Scientific novelty
What should we demand of a scientific theory?  We should demand that they are as powerful as possible, in that they predict/explain a broad range of empirical phenomena.  And by Occam's Razor, we should demand that the theory is also as simple as possible.  This is the case with e.g.\ quantum mechanics (QM). We don't care about QM because the math is pretty: it just isn't interesting in and of itself. It is only interesting to the extent that it provides a relatively simple explanation of/prediction for an broad range of empirical phenomena.  And this is reflect in almost all of the published work on QM.  The work doesn't do theoretical development, because the theory is settled.  It is using QM to understand an ever broader range of empirical phenomena. Because as scientists, that's what we ultimately care about: explaining empirical phenomena. And if QM allows you to come up with simple explanations for phenomena that were previously not understood, then that's good.  Its infinitely better than coming up with separate, complex theories for each phenomenon separately.

How does our work fit in this framework?  First, novelty is driven not by developing new theories (which are irrelevant in and of themselves), but in explaining empirical phenomena. So the right question is: has there been any previous explanation of SSL as maximum likelihood inference in a well-defined probabilistic generative model? The answer is no.
Second, is it better to use simple theory (probabilistic generative model of data curation) that also explains other phenomena such as the cold posterior effect?  Or would is it better to develop complex theory that only applies to SSL?  Well, our goal is scientific: to explain as many phenomena as possible using theories that are as simple as possible.  So our approach: developing a simple theory that gives strong insight and explains many different empirical phenomena is clearly **scientifically** preferable.

Indeed, the links that we raise between semi-supervised learning and the cold-posterior effect are pretty shocking in and of themselves.  There is no reason to believe there should be any sort of connection between them.  The cold-posterior effect is a weird and highly-specific empirical phenomenon in Bayesian neural networks for supervised image classification.  Why on earth would there be a link to the whole area of semi-supervised learning?  If nothing else, the cold-posterior effect is for supervised, not semi-supervised networks.  Indeed, prior to this work, we don't believe that _anyone_ had made any sort of link between semi-supervised learning and cold-posteriors. We believe that this link is interesting and surprising enough in and of itself to be deserving of wider dissemination through publication.

 Thanks for your thorough and helpful comments!

### W1
We have included a note about the relationships between pseudo-labelling and entropy regularisation noted in the original pseudo-labelling paper.  Note though that Lee (2013) only seem to formally claim equivalence of the optimal predictive distributions for pseudo-labelling and entropy minimization, which puts all probability mass into one class (from Lee 2013; Sec. 3.3):
>Our method encourages the predicted class probabilities to be near 1-of-K code via training with unlabeled data and Pseudo-Labels, so the entropy of (17) is minimized. Thus our method is equivalent to Entropy Regularization.


### Q1: Guidance to the practitioner.
[Update].  In response to a very late review, we found that the entropy minimisation bound is tight for a uniform predictive distribution (maximum uncertainty), and the pseudo-labelling bound is tight for a predictive distribution that puts all its mass on one class (maximum certainty).  Presumably, the pseudo-labelling bound is more relevant, as low-density separation SSL objectives push predictive distributions towards greater certainty.  And this might be a potential explanation for use of pseudo-labelling rather than entropy minimisation in recent methods such as FixMatch.

### Q2
We should have clarified that exact corresponds to Eq. (14) (which is exact in the limit as $K \rightarrow \infty$) and pseudo corresponds to the pseudo labelling version of the augmented objective (Eq. 15). We have added this to the legend. 

### Q3: the dip
We agree that the small "dip" is an interesting finding.  We are hesitant to focus on it too much, as it isn't clear that it will be robust across datasets etc.  To give one explanation for the dip, remember that what really matters is the degree of separation between the classes.  If the classes are well-separated, then any SSL method (including our approaches, pseudo-labelling, entropy minimization and FixMatch) should work, regardless of where that separation came from. The separation could be inherent to the data, or it could be induced by data curation.  We guess that there might be a small degree of separation inherent to the GZ2 data, allowing there to be a small benefit from limited semi-supervised learning.  However, as the amount of unlabelled data and SSL increases, pathologies set in.  In contrast, the strong separation induced by data curation would give a much stronger improvement with unlabelled data.

We'd expect all low-density SSL methods to exhibit roughly similar performance.  Indeed, empirically in Fig. 6, "pseudo" (labelling) (Eq. 15) seems to match very closely to exact (Eq. 15), so we'd expect similar behaviour with entropy minimization too.

### Minor presentation comments
Thanks for these suggestions: we've implemented all of them.

**Fig. 2**
We have added a panel to Fig 2, showing the generative model of consensus for an omitted label, and showing that the posterior over $\theta$ under this model doesn't reduce to the prior.  Note that this requires us to introduce a new random variable, $C$, representing whether or not consensus was reached.  (We did have an earlier version of the manuscript which used $C$ in the derivations, but it turned out to be _much_ simpler to eliminate it in the derivations, and just work with $Y=None$ or $Y\neq None$.)

**Superscripts**
We have moved to $(p_y(X))^S$.

**Y_s without defining s**
This notation is introduced in Sec. 2.1, which already has references to the range of $s$.
We have switched to using $Y_\text{sup}$ rather than $Y_s$ in Sec. 2.3.
And we added a reminder about s=$\\{1,...,S\\}$ at the start of Sec. 3.2.
That *should* be everywhere we use this notation, but please let us know if there's anywhere we've missed.

**Fig. 3**
"Bus" and "train" were chosen to correspond to the toy example in Fig. 1.  We have changed them to "consensus: class 1" and "consensus: class 2", which is clearer.
We have deleted the no-consensus points in panel B, does seem to make the panel much cleaner.

**Sec 4.2**
Corrected to: 
> Our data curation based theory predicts  that  low-density  separation  based  SSL should  be  much more effective on curated than uncurated data.

The paper analysis semi-supervised methods taking into account the curation process employed in the creation of popular datasets, like ImageNet. The authors model this process via a generative model which they use to show two common objectives in semi-supervised learning, namely entropy minimisation and pseudo-labelling, are actually lower-bounds on the log-likelihood of the data under this generative model. The paper also shows experimentally, that unlabelled data improves the accuracy of classifiers when the dataset has been curated but actually hurts performance when uncurated datasets are used. ### Strengths
- The theoretical results seem sound and connect popular semi-supervised objectives to the proposed generative model for curated datasets.
- Using a generative model of the curation process for popular datasets in the context of semi-supervised learning is insightful, even though the model itself has already been proposed in previous work.

### Weaknesses
- The paper shows entropy minimisation and pseudo-labelling form a lower bound on the log-likelihood of the proposed generative model. However, optimising lower bounds is only useful when the bounds are tight, which does not seem to be the case here. The authors could expand their discussion on the insights these bounds provide.
- The theoretical results motivates, to some extent, the use of semi-supervised learning when curated data is available but do not tell us much about uncurated data. We could argue that uncurated data does not match the proposed generative model, but then that is just a case of model misspecification which is a rather well-known problem and not directly connected to the theoretical results of the paper. In that light, I do not find the experimental results very insightful.
- There is no discussion about other works that tried to explain the occasional poor performance of semi-supervised methods. I know of at least one [2] that provides theoretical results showing that unlabelled data (even when matching the distribution of labelled data) might hurt performance if the model is misspecified. They cover classifiers based on generative models, but there might still be parallels worth discussing.
- Even though the text is well written, I do not find the exposition very clear. For instance, it was not clear if $\theta$ refers to the parameters of 'true' data-generating process or the parameters of the classifier. Moreover, when I checked [1] for further background information, I found the exact same description, almost word for word, of the generative model for curated datasets. I find this problematic, as at best this is a bad case of text recycling.

### Questions
- In section 4.1., it is not clear how the annotators are simulated. Could the authors give more details on how the data was generated? Were the annotations just sampled from the categorical distribution defined by the 'true' neural network?
- Still in section 5.1., how was the test data constructed? This is not entirely clear in the text.
- Could there not be other factors influencing the poor performance when using uncurated data? For instance, uncurated data could be simply harder to model, with more ambiguous and noisier samples. Did the authors control for this somehow?

### Minor Issues
- Equation 5 seems a bit redundant. The result follows directly from the assumptions in the generative model, and this equation does not add much to the discussion.

[1] Aitchison, Laurence. "A statistical theory of cold posteriors in deep neural networks." International Conference on Learning Representations. 2021.

[2] Cozman, Fabio Gagliardi, Ira Cohen, and Marcelo Cesar Cirelo. "Semi-supervised learning of mixture models." ICML. Vol. 4. 2003. The paper shows that existing semi-supervised objectives are proper lower-bounds on the log-likelihood of a generative model of curated data. These are interesting and valid results, but I am not confident they are very insightful. The experiments essentially show that uncurated data is harder to model, but unless I am missing something, this is not a surprising fact in itself nor is it a direct consequence of the bounds provided. Finally, I do think taking the curation process into account in semi-supervised learning (and other fields in machine learning) is quite promising, but the main methods of the paper, namely the generative model of curated data, are not new to this work.

 ### Empirical relates to theoretical
The improvement of test log likelihood on curated but not on uncurated data is a direct consequence of our theoretical results.
In particular, there are many classical results showing that the effectiveness statistical inference methods including maximum likelihood and Bayesian inference depends strongly on the accuracy of the generative model.
For maximum likelihood:
* White H. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society. 1982 Jan 1:1-25. (citations: 6255).

For Bayesian inference:
* Savage LJ. The foundations of statistics. (1954) (citations: 19,032).

Therefore our work does give strong theoretical reason to believe that semi-supervised learning, which can be understood as maximum likelihood inference under a generative model incorporating data curation, is going to work better with curated data following that data generating process, and that semi-supervised learning will work far less well with uncurated data that does not follow the modelled data generating process.

### Calculating test log-likelihood
The test log-likelihood is equivalent to the (negative) cross entropy.  More concretely, the data is split into a training and a test set.  The neural network estimates the probability that a particular image is in each class.  For each test image, we take the log-probability for the true class label under the neural network's predictive distribution.  For further details, see:
* Goodfellow I, Bengio Y, Courville A. Deep learning. MIT press; 2015.  Chapter 6.

### Experiments
Our experiments are specifically addressed to validating our hypothesis as cleanly as possible, by e.g. taking datasets like GZ2 which are not initially curated, but do have enough labels to allow us to create our own, matched curated datasets.  Importantly, datasets like uncurated GZ2 many actually be better than carefully curated benchmark datasets like CIFAR-10 for understanding messy, real-world settings where data is typically not curated.  

There is past empirical work looking at the consequences of using uncurated (or "OOD") data in typical benchmark semi-supervised learning settings:
* Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, pp. 3239–3250, 2018

This work is broadly in agreement with our work: that SSL methods typically work much better on curated datasets.

 Thanks for your comments!

To begin, it is worth discussing your final comment: about the differences between a technical, theoretical and experimental contribution.

Our work descends from a long line of theoretical work that takes intuitively reasonable algorithms or objectives, and shows that they can be understood as inference in a well-defined probabilistic model. 
This is really important *theoretical* work, and through the long history of this approach, it has had a huge impact in statistics and machine learning.
There are two really classic examples of this type of theoretical work.

Perhaps the cleanest example is PCA.  In 1999, Chris Bishop and Mike Tipping discovered that PCA can be understood as maximum likelihood inference in a latent variable model:

* Tipping ME, Bishop CM. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 1999;61(3):611-22. (citations: 3,983)

That allowed Chris Bishop to develop Bayesian PCA that was able to reason about uncertainty:

* Bishop CM. Bayesian PCA. NeurIPS (1999) (citations: 422)

And then enabled them to develop improved variants of PCA by modifying the generative model:

* Tipping ME, Bishop CM. Mixtures of probabilistic principal component analyzers. Neural computation. 1999 Feb 15;11(2):443-82. (citations: 2,082)

Tracing this line of work, even further back, Harold Jeffreys offered in *1939* an understanding of a number of orthodox statistical methods from Fisher and Neyman-Pearson as Bayesian inference in probabilistic generative models:

* Jeffreys H. The theory of probability. (1939) (citations: 15,238)

The advantages of these approaches, in terms of clarifying assumptions about the data generating process, improved uncertainty estimation and additional flexibility were extensively discussed in:

* Jaynes ET. Probability theory: The logic of science. (2003) (citations: 7971).

This is important theoretical work that has had huge impact, as evidenced by the number of citations.

These are some of the most prominent early examples, but there are other examples, including:
* The connection between kernel methods and Gaussian processes which allows us to reason about uncertainty, and to offer improved methods by modifying the generative process.
    * Rasmussen CE. Gaussian processes in machine learning. (2003; citations: 23,515). 
    * Williams CK, Rasmussen CE. Gaussian processes for machine learning. (2006; citations: 1724). 
* ICA 
  * MacKay DJ. Maximum likelihood and covariant algorithms for independent component analysis. (1999; citations: 203)
  * Hyvärinen A. Sparse code shrinkage: Denoising of nongaussian data by maximum likelihood estimation. Neural computation. (1999; citations: 533)
* Slow feature analysis 
  * Turner R, Sahani M. A maximum-likelihood interpretation for slow feature analysis. Neural computation. (2007; citations: 80).


As we can see from some of the work mentioned previously, this theoretical work gives three key practical advantages:
1. The formulation as a probabilistic model makes clear our assumptions about the data generating process (those assumptions are there in the original formulation of the algorithm, they just aren't explicit).  We can then see the extent to which real data matches this data-generating process, which might suggest improvements or alternative approaches.
2. We may be able to obtain improved or generalised methods by modifying the generative model to make it match the true data generating process more closely.
3. We can readily obtain a Bayesian version of the method that is able to reason under uncertainty.

However, there is a conspicuous gap in the literature when it comes to modern deep learning methods, including semi-supervised learning.
This is a gap that our paper seeks to fill.
Theoretically, we show that semi-supervised learning can be understood as inference in a well-defined generative model.
This work thus offers three potential practical advantages:
First, we have been able to write down a probabilistic generative model under which SSL methods perform (approximate) inference.  This model involves data curation, and thus highlights that we expect SSL to perform better on curated than uncurated data, which has important practical implications.
Second, we were able to improve the model by introducing data augmentation, which led us to an algorithm mirroring FixMatch.
Third, we were able to introduce Bayesian SSL, allowing future work to reason under uncertainty in small datasets.

 Thanks!  We fixed those minor points.  Please let us know if you think of anything else!



This paper uses a generative model for "curated" labeled datasets that was initially developed by Aitchison 2020 for another seemingly unrelated purpose (explaining the "cold posterior" effect). The assumed task is classification, where there is a set of mutually exclusive possible class we wish to assign to each image. To obtain labels for a training set given specific images (or other example features), the generative model assumes that each example is assigned a label from S independent, identically-distributed annotators. Only if *all* annotators agree is consensus reached and the class label is provided, otherwise the image is considered "noconsensus" and is usually not included. It is suggested that many common SSL datasets (e.g. CIFAR-10) use such a consensus curation process, and that even unlabeled sets are subjected to consensus curation.

Using this model, they provide a principled explanation for the empirical success of several well-known semi-supervised learning (SSL) objectives -- entropy minimization, pseudo-label, and FixMatch -- applicable to discriminative deep learning. The key idea is that each objective can be viewed as a lower bound of the log likelihood of the unlabeled set under the proposed model for curation, assuming that the unlabeled set contains only "consensus" images.

In Sec. 4.1, the paper presents a Bayesian SSL analysis on a toy dataset. The key result is that if unlabeled data is generated without multi-annotator curation, then the unlabeled data does not improve classifier accuracy when included in training using SSL. However, if only consensus examples are included in the unlabeled set, then reasonable improvements in accuracy might be expected from SSL vs. labeled-set-only learning.

In Sec. 4.2, the paper analyzes astronomy images with 9 class labels, prepared using the Galaxy Zoo dataset (to my knowledge not a common dataset for SSL). The existence of multiple annotators for this dataset allows creating both curated and uncurated versions. Again, they find that for *curated* datasets (where many annotators agree), both test-log-likelihood and test accuracy improve from the inclusion of unlabeled examples. However, for *uncurated* datasets, including too many unlabeled examples "dramatically worsened" test-set performance, though small unlabeled sets help. # Strengths

* Focus on the curated aspect of common SSL benchmark datasets
* Goal of relating objectives to a proper generative model is valuable
* Insightful use of the same generative model to offer support for 3 different common SSL objectives arising from last 20 years of SSL research
* Nice toy data experiments in Figure 5, showing impact of more unlabeled data on SSL methods for both curated and uncurated unlabeled sets

# Weaknesses

* Missing previous known connections between Pseudolabel and Entropy regularization

See W1 below.

* Results on Galaxy Zoo (Fig. 6) could use more explanation: why does performance benefit from a few (not too many) unlabeled examples even in noconsensus case?

See Q3 below.


### W1: Missing previous known connections between Pseudolabel and Entropy regularization

In the presented background in Sec. 2, entropy regularization in Eq 3 and pseudolabel in Eq 4 are presented as seemingly separate alternatives. I think the authors could do a better job explaining that pseudolabel was originally motivated by entropy minimization -- the original PseudoLabel paper dedicates all of Sec. 3.2 to explaining that "... our method is equivalent to Entropy Regularization."

Not a dealbreaker, but making this connection could help the audience understand why both methods might be lower bounds to the same log likelihood.



# Questions for Rebuttal

### Q1: What insight about which SSL objectives to prefer is offered by your lower-bound arguments?

The paper offers justification of several SSL objectives -- pseudolabel, entropy minimization, and fixmatch --- by deriving each as a lower bound of the log likelihood of the label-consensus model.

However, there is little insight here about when each bound might be expected to be "the best".... does FixMatch always dominate? 

This may seem a bit out of scope, but I think offering some analysis here would help practitioners in thinking about which methods to apply in which circumstances.


### Q2: Questions about Fig. 6

What do "exact-" and "pseudo-" prefixes mean here? Can you adjust caption in revision to clarify?

### Q3: Results on Galaxy Zoo (Fig. 6) could use more explanation

In Fig. 6, seems like both log likelihood and accuracy improve *slightly* with modest unlabeled data in uncurated case, but then decline with large unlabeled data. The authors acknowledge this in the paper by saying " though the inclusion of a small number of unlabelled points gave very small performance improvements (Fig. 6CD)".

Is there any insight about why the curves look U shaped here, rather than a strict decline as the theory might suggest? Perhaps the data augmentation related to FixMatch used here is driving this? Would the same trend be seen if we just used entropy minimization?


# Minor Presentation Comments

No need to mention these in rebuttal, but I hope you consider them to improve the paper

### Comments on Figure 2

* I felt Fig. 2 could communicate more than it did. Perhaps in the right-most panel that shows your consensus model, you could somehow visually emphasize that for unlabeled examples, you are assuming that "consensus" was reached, and thus that the distribution $p(\theta | X)$ does not reduce to the prior marginal $p(\theta)$ under the assumed model?

### Superscript Notation could be improved

This is somewhat minor and perhaps a personal nitpick, but I dislike the notation $p^S_y(X)$ in Eq. 2. 

I'd prefer to see it like this:

$$
( p_y(X) )^S
$$

This makes it clear that the quantity is raised to the $S$-th 
power. The current notation is a bit less clear to me on that front.


### Lots of use of Y_s without defining $s$

In many cases, the paper writes $P( Y_s = y | X, \theta)$ without defining a value for the index $s$ (as an example, see Eq. 2).  I guess I found this a little confusing, since elsewhere $s$ requires a concrete value to be well understood. I'd rather there was some explicit comment like "where s can be any value in 1 to S" or instead some other notation.  


### Coments on Figure 3

* Why in Figure 3 are points labeled "bus" and "train"? Aren't these just toy examples?
* Not sure that both panels (3A and 3B) are needed in this figure. Could use just one, and verbally explain how unlabeled data would be selected. Or, at very least, the second panel could remove the noconsensus points, to help drive home visually the separation that occurs in the unlabeled set.


### Comments on Sec. 4.2

Current text says: "data-curation based theory predicts that SSL should be much more effective on curated than uncurated data".

I'd clarify that the theory only applies to *SSL methods using the low-density separation principle*. Other SSL principles (e.g. those based on generative models) aren't applicable. I recommend accepting the paper. First, it draws attention to an important (yet under-studied) aspect of common SSL datasets like CIFAR-10: they are heavily curated and thus their so-called "unlabeled" sets are really just labeled sets with labels redacted. Second, it offers insight into why several widely-used SSL methods (entropy regularization, pseudolabel, and FixMatch) work, by framing each as a lower bound of a log-likelihood for a generative model for how many annotators produce consensus. I thought this was insightful and backed by nice toy data experiments and real data experiments (perhaps the curated vs uncurated versions of GalaxyZoo used here can become nice benchmarks for future SSL methods).

In the rebuttal and revisions, I hope the authors can address my questions about experimental results and why the bounds say about when each SSL method might be preferred over alternatives.

This paper shows that low-density separation semi-supervised learning (SSL) objectives can be understood as a lower-bound on a log-probability that arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective when unlabelled data is obtained by throwing away labels from the carefully curated training set, and is less effective when unlabeled data is taken from uncurated images. Experiments on toy data generated from a known model and on real data from Galaxy Zoo confirm the importance of data curation for SSL. Minor comments (language mistakes, etc.):
Figure 1 caption: The first ``annotators" should have its first letter capitalized.
Section 2.1: "ensure sure" should be "ensure".
Section 4.1: ``Fig. 5AB" and ``Fig. 5CD" should be written as ``Fig. 5A-B" and ``Fig. 5C-D" (i.e., with a hyphen inserted between). The paper is generally very well written, clearly presented, and a pleasure to read. The problem is well-motivated and well-defined with respect to the background. The technique look sound and novel.

The paper is built on the previous work (Aitchison, 2021) that provides a generative model of data curation. In this model, the likelihood of a labeled data obtained with consensus is given by product of the probability that each labeler labels correctly. Under this model, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. In toy experiments and Galaxy Zoo experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data. Under the generative model of data curation, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. These results are potentially interesting. 

In toy experiments and Galaxy Zoo data experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data. However, this is not directly the consequence of the theoretical results. I am not clear on this connection. The theory shows that log likelihood is an upper bound on the entropy, pseudo labelling loss under the generative model of curated data. It does not say what happens when the data is uncurated. Also, I am not clear how the authors measured the test likelihood.

Besides, the technical contribution, both theoretical and experimental are limited. On the experiments, it would be nicer to see the consequences of the theoretical results on modern semi supervised learning approaches and datasets they use.

 The theoretical and experimental contribution of the work are not enough to justify publication at the current form.

Meta-review: This submission has generated sufficient debate, including some messages that, in my viewpoint, have the wrong tone. It may well be that different colleagues see the work in different ways. It is very hard to evaluate submissions in a short time and mistakes can happen. In this case, I think there were and still are misunderstandings and unclarity wrt very crucial points of the paper. This does not mean that the work overall is weak not that there is no contribution. If the content is so interesting (as discussed by authors and multiple reviewers) in some way (which it seems to be), then a better presentation and argumentation will lead to a publication elsewhere soon, but based on all the data that I have here, I recommend rejection. I see to reason to list details about the content and possible concerns, as they should be clear from the multiple messages among authors and reviewers. Best of luck.

Decision: Reject