Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper analysis semi-supervised methods taking into account the curation process employed in the creation of popular datasets, like ImageNet. The authors model this process via a generative model which they use to show two common objectives in semi-supervised learning, namely entropy minimisation and pseudo-labelling, are actually lower-bounds on the log-likelihood of the data under this generative model. The paper also shows experimentally, that unlabelled data improves the accuracy of classifiers when the dataset has been curated but actually hurts performance when uncurated datasets are used.
main_review: ### Strengths
- The theoretical results seem sound and connect popular semi-supervised objectives to the proposed generative model for curated datasets.
- Using a generative model of the curation process for popular datasets in the context of semi-supervised learning is insightful, even though the model itself has already been proposed in previous work.

### Weaknesses
- The paper shows entropy minimisation and pseudo-labelling form a lower bound on the log-likelihood of the proposed generative model. However, optimising lower bounds is only useful when the bounds are tight, which does not seem to be the case here. The authors could expand their discussion on the insights these bounds provide.
- The theoretical results motivates, to some extent, the use of semi-supervised learning when curated data is available but do not tell us much about uncurated data. We could argue that uncurated data does not match the proposed generative model, but then that is just a case of model misspecification which is a rather well-known problem and not directly connected to the theoretical results of the paper. In that light, I do not find the experimental results very insightful.
- There is no discussion about other works that tried to explain the occasional poor performance of semi-supervised methods. I know of at least one [2] that provides theoretical results showing that unlabelled data (even when matching the distribution of labelled data) might hurt performance if the model is misspecified. They cover classifiers based on generative models, but there might still be parallels worth discussing.
- Even though the text is well written, I do not find the exposition very clear. For instance, it was not clear if $\theta$ refers to the parameters of 'true' data-generating process or the parameters of the classifier. Moreover, when I checked [1] for further background information, I found the exact same description, almost word for word, of the generative model for curated datasets. I find this problematic, as at best this is a bad case of text recycling.

### Questions
- In section 4.1., it is not clear how the annotators are simulated. Could the authors give more details on how the data was generated? Were the annotations just sampled from the categorical distribution defined by the 'true' neural network?
- Still in section 5.1., how was the test data constructed? This is not entirely clear in the text.
- Could there not be other factors influencing the poor performance when using uncurated data? For instance, uncurated data could be simply harder to model, with more ambiguous and noisier samples. Did the authors control for this somehow?

### Minor Issues
- Equation 5 seems a bit redundant. The result follows directly from the assumptions in the generative model, and this equation does not add much to the discussion.

[1] Aitchison, Laurence. "A statistical theory of cold posteriors in deep neural networks." International Conference on Learning Representations. 2021.

[2] Cozman, Fabio Gagliardi, Ira Cohen, and Marcelo Cesar Cirelo. "Semi-supervised learning of mixture models." ICML. Vol. 4. 2003.
summary_of_the_review: The paper shows that existing semi-supervised objectives are proper lower-bounds on the log-likelihood of a generative model of curated data. These are interesting and valid results, but I am not confident they are very insightful. The experiments essentially show that uncurated data is harder to model, but unless I am missing something, this is not a surprising fact in itself nor is it a direct consequence of the bounds provided. Finally, I do think taking the curation process into account in semi-supervised learning (and other fields in machine learning) is quite promising, but the main methods of the paper, namely the generative model of curated data, are not new to this work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
This paper uses a generative model for "curated" labeled datasets that was initially developed by Aitchison 2020 for another seemingly unrelated purpose (explaining the "cold posterior" effect). The assumed task is classification, where there is a set of mutually exclusive possible class we wish to assign to each image. To obtain labels for a training set given specific images (or other example features), the generative model assumes that each example is assigned a label from S independent, identically-distributed annotators. Only if *all* annotators agree is consensus reached and the class label is provided, otherwise the image is considered "noconsensus" and is usually not included. It is suggested that many common SSL datasets (e.g. CIFAR-10) use such a consensus curation process, and that even unlabeled sets are subjected to consensus curation.

Using this model, they provide a principled explanation for the empirical success of several well-known semi-supervised learning (SSL) objectives -- entropy minimization, pseudo-label, and FixMatch -- applicable to discriminative deep learning. The key idea is that each objective can be viewed as a lower bound of the log likelihood of the unlabeled set under the proposed model for curation, assuming that the unlabeled set contains only "consensus" images.

In Sec. 4.1, the paper presents a Bayesian SSL analysis on a toy dataset. The key result is that if unlabeled data is generated without multi-annotator curation, then the unlabeled data does not improve classifier accuracy when included in training using SSL. However, if only consensus examples are included in the unlabeled set, then reasonable improvements in accuracy might be expected from SSL vs. labeled-set-only learning.

In Sec. 4.2, the paper analyzes astronomy images with 9 class labels, prepared using the Galaxy Zoo dataset (to my knowledge not a common dataset for SSL). The existence of multiple annotators for this dataset allows creating both curated and uncurated versions. Again, they find that for *curated* datasets (where many annotators agree), both test-log-likelihood and test accuracy improve from the inclusion of unlabeled examples. However, for *uncurated* datasets, including too many unlabeled examples "dramatically worsened" test-set performance, though small unlabeled sets help.
main_review: # Strengths

* Focus on the curated aspect of common SSL benchmark datasets
* Goal of relating objectives to a proper generative model is valuable
* Insightful use of the same generative model to offer support for 3 different common SSL objectives arising from last 20 years of SSL research
* Nice toy data experiments in Figure 5, showing impact of more unlabeled data on SSL methods for both curated and uncurated unlabeled sets

# Weaknesses

* Missing previous known connections between Pseudolabel and Entropy regularization

See W1 below.

* Results on Galaxy Zoo (Fig. 6) could use more explanation: why does performance benefit from a few (not too many) unlabeled examples even in noconsensus case?

See Q3 below.


### W1: Missing previous known connections between Pseudolabel and Entropy regularization

In the presented background in Sec. 2, entropy regularization in Eq 3 and pseudolabel in Eq 4 are presented as seemingly separate alternatives. I think the authors could do a better job explaining that pseudolabel was originally motivated by entropy minimization -- the original PseudoLabel paper dedicates all of Sec. 3.2 to explaining that "... our method is equivalent to Entropy Regularization."

Not a dealbreaker, but making this connection could help the audience understand why both methods might be lower bounds to the same log likelihood.



# Questions for Rebuttal

### Q1: What insight about which SSL objectives to prefer is offered by your lower-bound arguments?

The paper offers justification of several SSL objectives -- pseudolabel, entropy minimization, and fixmatch --- by deriving each as a lower bound of the log likelihood of the label-consensus model.

However, there is little insight here about when each bound might be expected to be "the best".... does FixMatch always dominate? 

This may seem a bit out of scope, but I think offering some analysis here would help practitioners in thinking about which methods to apply in which circumstances.


### Q2: Questions about Fig. 6

What do "exact-" and "pseudo-" prefixes mean here? Can you adjust caption in revision to clarify?

### Q3: Results on Galaxy Zoo (Fig. 6) could use more explanation

In Fig. 6, seems like both log likelihood and accuracy improve *slightly* with modest unlabeled data in uncurated case, but then decline with large unlabeled data. The authors acknowledge this in the paper by saying " though the inclusion of a small number of unlabelled points gave very small performance improvements (Fig. 6CD)".

Is there any insight about why the curves look U shaped here, rather than a strict decline as the theory might suggest? Perhaps the data augmentation related to FixMatch used here is driving this? Would the same trend be seen if we just used entropy minimization?


# Minor Presentation Comments

No need to mention these in rebuttal, but I hope you consider them to improve the paper

### Comments on Figure 2

* I felt Fig. 2 could communicate more than it did. Perhaps in the right-most panel that shows your consensus model, you could somehow visually emphasize that for unlabeled examples, you are assuming that "consensus" was reached, and thus that the distribution $p(\theta | X)$ does not reduce to the prior marginal $p(\theta)$ under the assumed model?

### Superscript Notation could be improved

This is somewhat minor and perhaps a personal nitpick, but I dislike the notation $p^S_y(X)$ in Eq. 2. 

I'd prefer to see it like this:

$$
( p_y(X) )^S
$$

This makes it clear that the quantity is raised to the $S$-th 
power. The current notation is a bit less clear to me on that front.


### Lots of use of Y_s without defining $s$

In many cases, the paper writes $P( Y_s = y | X, \theta)$ without defining a value for the index $s$ (as an example, see Eq. 2).  I guess I found this a little confusing, since elsewhere $s$ requires a concrete value to be well understood. I'd rather there was some explicit comment like "where s can be any value in 1 to S" or instead some other notation.  


### Coments on Figure 3

* Why in Figure 3 are points labeled "bus" and "train"? Aren't these just toy examples?
* Not sure that both panels (3A and 3B) are needed in this figure. Could use just one, and verbally explain how unlabeled data would be selected. Or, at very least, the second panel could remove the noconsensus points, to help drive home visually the separation that occurs in the unlabeled set.


### Comments on Sec. 4.2

Current text says: "data-curation based theory predicts that SSL should be much more effective on curated than uncurated data".

I'd clarify that the theory only applies to *SSL methods using the low-density separation principle*. Other SSL principles (e.g. those based on generative models) aren't applicable.
summary_of_the_review: I recommend accepting the paper. First, it draws attention to an important (yet under-studied) aspect of common SSL datasets like CIFAR-10: they are heavily curated and thus their so-called "unlabeled" sets are really just labeled sets with labels redacted. Second, it offers insight into why several widely-used SSL methods (entropy regularization, pseudolabel, and FixMatch) work, by framing each as a lower bound of a log-likelihood for a generative model for how many annotators produce consensus. I thought this was insightful and backed by nice toy data experiments and real data experiments (perhaps the curated vs uncurated versions of GalaxyZoo used here can become nice benchmarks for future SSL methods).

In the rebuttal and revisions, I hope the authors can address my questions about experimental results and why the bounds say about when each SSL method might be preferred over alternatives.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows that low-density separation semi-supervised learning (SSL) objectives can be understood as a lower-bound on a log-probability that arises from a principled generative model of data curation. This gives a theoretical understanding of recent results showing that SSL is more effective when unlabelled data is obtained by throwing away labels from the carefully curated training set, and is less effective when unlabeled data is taken from uncurated images. Experiments on toy data generated from a known model and on real data from Galaxy Zoo confirm the importance of data curation for SSL.
main_review: Minor comments (language mistakes, etc.):
Figure 1 caption: The first ``annotators" should have its first letter capitalized.
Section 2.1: "ensure sure" should be "ensure".
Section 4.1: ``Fig. 5AB" and ``Fig. 5CD" should be written as ``Fig. 5A-B" and ``Fig. 5C-D" (i.e., with a hyphen inserted between).
summary_of_the_review: The paper is generally very well written, clearly presented, and a pleasure to read. The problem is well-motivated and well-defined with respect to the background. The technique look sound and novel.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is built on the previous work (Aitchison, 2021) that provides a generative model of data curation. In this model, the likelihood of a labeled data obtained with consensus is given by product of the probability that each labeler labels correctly. Under this model, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. In toy experiments and Galaxy Zoo experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data.
main_review: Under the generative model of data curation, the paper shows 1) the log likelihood of the consensus is lower bounded by entropy or pseudo labeling, 2) the log likelihood of the consensus is also lower bounded by the FixMatch type average likelihood of the augmented data. These results are potentially interesting. 

In toy experiments and Galaxy Zoo data experiments, authors show that the test likelihood improves when unlabelled curated data is added, but decreases on the uncurated data. However, this is not directly the consequence of the theoretical results. I am not clear on this connection. The theory shows that log likelihood is an upper bound on the entropy, pseudo labelling loss under the generative model of curated data. It does not say what happens when the data is uncurated. Also, I am not clear how the authors measured the test likelihood.

Besides, the technical contribution, both theoretical and experimental are limited. On the experiments, it would be nicer to see the consequences of the theoretical results on modern semi supervised learning approaches and datasets they use.


summary_of_the_review: The theoretical and experimental contribution of the work are not enough to justify publication at the current form.
review_title: null

Meta-review: This submission has generated sufficient debate, including some messages that, in my viewpoint, have the wrong tone. It may well be that different colleagues see the work in different ways. It is very hard to evaluate submissions in a short time and mistakes can happen. In this case, I think there were and still are misunderstandings and unclarity wrt very crucial points of the paper. This does not mean that the work overall is weak not that there is no contribution. If the content is so interesting (as discussed by authors and multiple reviewers) in some way (which it seems to be), then a better presentation and argumentation will lead to a publication elsewhere soon, but based on all the data that I have here, I recommend rejection. I see to reason to list details about the content and possible concerns, as they should be clear from the multiple messages among authors and reviewers. Best of luck.
Decision: Reject