Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors have formulated Semi Supervised Learning (SSL) objective as a principled log-likelihood estimate in the context of a generative model of data curation. The distribution of data (in the classification context) has multiple or at least two levels of uncertainty: one is the distribution of the classes and the other is the labels themselves. The authors characterize the supervised setting as learning by using curated data labels (the lower bound of the SSL setting) and propose SSL as an extended case with curated plus non-curated and unlabelled data (coins the concept of undefined class while there is a disagreement on labels or no label available).

The proposed methodology has been tested on some synthetic, CIFAR, and Galaxy Zoo 2 datasets. Statistically significant improvements have been claimed over the FixMatch, a known technique in SSL.

strength_and_weaknesses: Strengths: The main idea of this paper is to derive a Bayesian formulation of the SSL setup by using labeled and unlabeled data. Using curated labeled data they have formulated/defined a lower bound of the solution. They also have proposed two likelihood estimates that are applicable to the curated labeled and unlabeled data and then combine them in a coherent fashion for model prediction.

The proposed methodology has been tested on CIFAR-10 and Galaxy Zoo datasets. Reported results are found to be better than FixMatch, especially when trained with less number of labeled data points (in the SSL mix).

Weaknesses: This work lacks a proper Bayesian formulation of the SSL problem which requires selection of some suitable priors and a proper estimation of the posteriors. Given that unlabeled data class labels are unknown it adds additional complexity to the modelling. There has been some prior work through the definition and usage of a null category (similar to the definition of the undefined class in this paper) such as [1, 2] which may help improve the Bayesian aspect of the formulation.

[1] https://papers.nips.cc/paper/2004/file/d3fad7d3634dbfb61018813546edbccb-Paper.pdf [2] http://www.bmva.org/bmvc/2011/proceedings/paper3/paper3.pdf)
clarity,_quality,_novelty_and_reproducibility: Overall, the structure of the paper is organized; however, certain parts are difficult to follow. Data curation and principled likelihood estimates are two important concepts of this work. Although these two concepts have dependency, they have been used (the spread) over many different places in the paper which seems to be redundant. The readability of the paper can be improved if such redundancy can be simplified. The experiment section 4.1 is difficult to follow as it is missing some details such as the Gaussian parameters and how those were chosen, etc.

The paper contains some novelty: estimating the SSL lower bound in the context of data curation and experimenting and explaining the complexity as the uncertainty increases due to the increase of un-curated and unlabelled data.

As the code has been shared, it is expected the results can be reproduced although some additional details in section 4 could be helpful.

summary_of_the_review: I have gone through the paper more than once; overall, the idea sounds good and the results support some of the claims of this research. If we exclude certain sections, the paper is well written. The experiment section is missing some details and therefore found little difficult to follow.

The reported results are found to be interesting and provide us some insights on the difficulty of non curated and unlabelled data. I think this work has some value if some of the above mentioned limitations can be resolved/improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to formulate the low-density separation term in SSL as a log-likelihood term from a generation model of data curation. Based on which, a Bayesian modeling is possible, and some new insight on why SSL works for curated data set.
strength_and_weaknesses: Strength: The likelihood for curated/uncurated data set seems interesting, and it successfully establishes a relationship to entropy minimization and  pseudo-labelling. 

Weakness: The construction of likelihood is too artificial, aiming to appeal low density separation idea. How to choose S? How to determine a labelled/unlabelled observation is curated or not, beyond common data sets such as CIFAR10? Unlike penalty term which are artificial designed to incorporate prior knowledge, likelihood should reflect data generation rule and tuning-free. 

One experiment shows the improvement when unlabelled curated data is added, and the performance drop with more uncurated data. But this is not a convincing justification of the main of idea of the paper. The theory shows that log likelihood is lower bounded by the entropy or pseudo labelling loss under the generative model of curated data. It does not say what happens when the data is uncurated. To establish empirical justification, one also needs to show that the performance will increase with added uncurated data if likelihood (7) is used.

The paper claims it is a Bayesian approach, but it never mention anything about prior setting. For all the results displayed, are they based on posterior MAP or posterior mean?

Figure 3, even when there is no unlabelled data, the test accuracy is different w.r.t. the choice of S. S=0 yields the best test accuracy. Please comment on this

What is the test likelihood (Figure 3/4)?

clarity,_quality,_novelty_and_reproducibility: I feel that the paper has some reproducibility issue, since not all the detail implementations are given for the experiments. For example, what is S value for the Galaxy zoo experiment; page 7, it says "We used Eq. (5) as the likelihood for labelled points, Eq. (6) as the
likelihood for unlabelled points and Eq. (7) as the likelihood for noconsensus points ",  I have no idea what is the likelihood for no-consensus unlabelled points. 
summary_of_the_review: Overall, I feel the paper is more like playing with a math trick, but can not deliver more insight. The low-density separation SSL assumes the decision boundary is not close to any data point, thus, even without any math, it implies that all data, including unlabelled data are away from decision boundary, i.e., they must be curated data. A mathematical formulate should provide us with more subtle insight of this matter, and this paper seems fail to do it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper looks at the topic of semi-supervised learning for classification, through a probabilistic lens. The starting point is the observation that many common SSL datasets (e.g. CIFAR) are assembled via multi-annotator agreement protocol (images are only included if many humans agree on their class label). This leads to a multi-annotator consensus generative model (previously described in Aitchison 2021). Eq. 6 gives this model's likelihood for an image to be a *consensus* image (e.g. images that would be included even in unlabeled set because many annotators agree).

The new thing in this paper is the realization that this likelihood can be lower bounded by many common SSL objectives, included entropy (Eq 9) or pseudolabel (Eq 11) or FixMatch (Eq 16). This gives a nice interpretation of what these objectives are doing (maximizing the likelihood of a model that is aware that consensus is needed).

Experiments on toy data demonstrate properties of this model, including:

1) Fig 3 suggests that if S =1 (multiple annotators are not needed), unlabeled data does not help under this model, while if S > 1 (several annotators need to agree), unlabeled data can help

2) Tab 1 shows that a Bayesian version of FixMatch, using their view of FixMatch's loss as a likelihood, does better than FixMatch on CIFAR-10 with few labeled images

3) Fig 4 shows that on Galaxy Zoo data, datasets that are *curated* show gains from unlabeled data while *uncurated* sets show losses not gains in classifier performance as more unlabeled data are added


strength_and_weaknesses: # Strengths

* Highlights that curation (multi-annotator agreement) is behind many popular SSL benchmarks, which is less widely understood/used than it should be
* Principled approach to defining likelihoods that correspond to bounds on popular SSL objectives
* Showing how to use these likelihoods within a Bayesian model seems valuable
* Analysis in Sec 2.1 and Experiments on toy data are convincing: if data comes from a single-annotator model, hopes for SSL are dim (any unlabeled data is uninformative!)

# Weaknesses

* Why does accuracy get worse as S increases in Fig 3 on toy data?
* Use of a bound (not exact likelihood) in CIFAR experiments may hide role of augmentation
* Unclear what prior is used in the Bayesian FixMatch
* Use of a temperature in the Bayesian FixMatch result needs more explanation
* Writing needs some work: Lots of abrupt transitions and confusing notation

These weaknesses are elaborated further below (under Quality/Clarity)
clarity,_quality,_novelty_and_reproducibility: 
## Novelty

I think there's plenty of contributions here: 

1) the observation that most existing SSL benchmarks are tightly curated (e.g. require multi-annotator agreement) is not widely known/mentioned or used within approaches, but this paper makes a clear case that it should be better understood.

2) the interpretation of negative entropy and pseudolabel loss functions as likelihoods appears new to me (despite these ideas being popular for a decade plus) and it nicely offers an explanation of why pseudolabel is more in favor now (its bound is tighter).

3) building a "Bayesian" FixMatch that outperforms standard FixMatch at small datasets sizes (Tab 1) is nice to see


## Quality

### Why does accuracy get worse as S increases in Fig 3?

Fig 3C clearly shows that S=4 annotators (dark purple) reach 70-73% accuracy, while S=1 reaches ~85% accuracy. Can the authors please explain?

As the number of annotators S increases, I would expect that on *labeled* test data the task of classification becomes easier, as only examples where the true class distribution gave many samples of the same class are included. Requiring more consensus should discard examples near decision boundaries, and thus make learning sharp boundaries easier.

I guess the test dataset itself is changing as S is changing? Is the ratio of pos/neg examples changing? The size of test set changing?
I think would be useful to clarify this in the figure/caption.

Because panels B and D are somewhat redundant re-visualizations of the same information in panels A and C, I wonder if the figure would be better off only showing A and C, while also showing some other visualization that makes the nature of the task a bit more clean (e.g. show a scatterplot in 2d of the examples, colored by class label, somehow indicating which examples are "consensus" or not under each of the S values)

### Would results improve if used exact likelihood instead of the bound?

Bayesian FixMatch uses Eq 16 as the likelihood in CIFAR-10 experiments.
In earlier toy experiments, Eq 6 is used as likelihood, and Eq 16 is only a lower bound of Eq 6.

Could we use Eq 6 (the ideal likelihood, not a bound) in CIFAR-10 and get better results than Bayesian FixMatch in Table 1? 
My guess is the augmentation hiding inside FixMatch is delivering generalization gains that wouldn't be possible with Eq 6, even though it is the "ideal" likelihood, but the explanation in this paper for why the Bayesian approach is better does not seem to account for that.


### Unclear what prior is used in the Bayesian FixMatch

Naturally, Bayesian methods require a prior and likelihood. I don't see a clear description of the prior used in Sec. 4.2. Can the authors please elaborate?

Probably it is just an independent Gaussian on each weight, but really should be specified in the main paper.

### Use of a temperature in the Bayesian FixMatch result needs more explanation

The use of the temperature (as in cold posterior paper by Wenzel et al. 2020) to me spoils the interpretation as truly "Bayesian"... I'd like to see some analysis of how necessary this is (what is performance with temperature 1.0?)

### Clarify purpose of training on CIFAR-100 images in Table 1?

I'm not sure why the authors report the third line in Table 1, showing FixMatch (a baseline) trained on unlabeled data from a different dataset (CIFAR-100). 

Naturally, it is reassuring to know that using different unlabeled data does worse, as we might expect, but it's not clear how this experiment lends credibility to the presented Bayesian approach or the overall interpretation of this paper that a model of "curation" is useful to SSL.


## Clarity 

Overall I found the manuscript's high-level points came through, but it was unfortunately tough to read in places, with awkward/abrupt transitions and frequent notation switching.

I'd recommend that transitions between subsections get revised to help the reader keep a narrative thread of where this is all going. For example, Sec. 2.2. immediately dives into a generative model of data curation without explaining how it fits in, and Sec. 3.1 dives into a proof of a bound without motivating why it is useful.

Here's a few further problematic clarity issues:

* Sec 2 never defines the symbols X or Y (easy enough to guess what they mean)
* Symbol Y_sup in Equation below Eq 2 is never defined, not clear if different from Y without subscript
* Sec. 2 uses "n" to denote number of annotations, this switches to "S" in Sec. 3 without explanation. I'd suggest keeping S, since "n" often denotes training data size.

summary_of_the_review: Overall I like the core ideas here: the observation that curation is behind many standard SSL datasets is potentially powerful though underused currently: the methods here suggest that multi-annotator curation can be exploited (if present) via SSL but that datasets with only single annotator curation may not be usefully pursued with SSL methods. 

<strike>
However, there's too many writing clarity issues and overall puzzling results to give this a high score currently. I hope discussion period clarifies things, I'd be willing to raise my score given a satisfactory response.
</strike>

**Update after discussion**: Clarity issues have largely been resolved, and the past improvements suggest remaining ones will also be resolved in good faith. Given the significance of the contributions here, I would vote for and argue for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the fact that our common benchmark data sets are curated, in the sense of their data points being preselected by the consensus of a set of labellers. Aitchison (2021) already noted this and designed a proper likelihood that takes account of this fact. The authors build upon his work by moving into the semi-supervised setting where they show how to formulate two common approaches (entropy minimization, pseudo-labelling) as lower bounds of this likelihood. The approach is then evaluated in several experiments ranging from synthetic to real-world.
strength_and_weaknesses: ## Strengths
The paper is well argued for and shows clear results upon the baseline the authors are considering and demonstrates in varied ways their assumptions

## Weaknesses
- The strong emphasis on this being _Bayesian SSL_ seems to me to be too strong. While the proposed approach gives us a proper likelihood the method and approach lack any intrinsic "Bayesianity". Dropping the Bayes terms the paper could have just as well been formulated as a novel likelihood for a maximum likelihood approach. The authors themselves 'simply' use it in this function in their third experiment. 
- Inconsistent notation and minor errors (see below)
- A lot of the hyperparameters and experimental details are missing, or partially available but then hidden in the provided code. E.g., Sec 4.1 mentions an architecture for creating the toy data set but lacks any information on whether the trained networks have the same architecture. Sec 4.3 lacks a complete discussion on how the authors arrived at their final nine labels, making replicability and reproducibility impossible. If the preprocessing follows Aitchison (2021) then please properly cite and discuss this fact. 
clarity,_quality,_novelty_and_reproducibility: - Sec 2.2 uses `n` annotators and switches to `S` for the remainder of the paper
- (10) contains an error. It should be $\log(S (1/S)^S) = (1-S)\log S$, i.e., you are missing a negative sign. Similarly, given that the derivation is about lower terms to a likelihood that is to be maximized (9,10) would read better if they were formulated in terms of an entropy objective that is to be maximized instead of the current negative entropy, giving then $(1-S)L_\text{entr}$ for both. (I.e., as in (11) which is explicitly formulated as a maximization task)

**Clarity and Quality:** The paper itself is well-written but contains an error in the derivation  

**Novelty:** The approach heavily builds on the prior work by Aitchison (2021) and is a rather minor extension of it.   

**Reproducibility:** In its current form reproducibility is not given for most of the experiments.  



## Specific questions
- Q1: Fig 2 speaks of _our generative model_. Can the authors clarify that part, i.e., to which degree does it differ from the example and model of Aitchison (2021) Figure 3?
- Q2: Fig 3. Can the authors comment on why the test log-likelihood seems to improve with more samplers (A) but the test accuracy decreases? Similarly, the caption lacks an explanation wrt. what the change in B, D consists of which has to be guessed by the reader.
- Q3: As stated above, the proposed approach could be used in a deterministic setting as well. Can the authors comment on how a maximum likelihood approach would perform in Sec 4.1 and 4.2 and vice versa how a BNN would perform in Sec 4.3 (e.g., Aitchison (2021) relies on an SGLD approach in that experiment)?

## Minor comments
- The method is restricted to the low-density approach of semi-supervised learning. But this restriction is acknowledged in Sec 5 and properly defended.
- Sec 2 
- The experiments as summarized in Fig 3/4 lack standard deviations

## Typos
- Inconsistency in the equation typography, i.e., some end with `.`,`,` others don't
- End of Sec 3.1: _psuedo-labeling_
summary_of_the_review: While the paper contains some interesting formulations and results that are well worth further investigation in its current form it is rather incremental and contains several technical flaws (derivation and esp. wrt reproducibility) that need to be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that this paper makes a very useful contribution in terms of proposing a new likelihood for semi-supervised learning. However, the framing of the paper as a new _Bayesian_ approach to semi-supervised learning is not convincing: more methodological development and motivation, plus corresponding experiments would be needed to justify this. My judgment is that the paper is still a very useful one: the title should be changed (i.e. remove the word Bayesian) and the framing should simply state that Bayesian extensions are possible, but emphasize the main contribution in terms of a new likelihood.
justification_for_why_not_higher_score: Some reviewers don't think the paper is ready in its current form.
justification_for_why_not_lower_score: It might make sense to reject, see above.
Decision: Accept: poster