Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Unlabelled samples are labelled with credal sets (set of probability distributions), better capturing the 'unknowledge' about the label.

The methodology integrates seamlessly several state-of-the-art ideas to achieve top performance in the experimental evaluation.


main_review: The paper is well-written and technically sound.
The novelty of the paper is enough.
Although the improvements in accuracy are not impressive, the contribution can be of interest to a wide range of NeurIPS participants.

The ablation study could include an analysis of the importance of the weighting in Eq (6) to the final results. What would be the results with the straightforward alphas?
limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: For the task of semi-supervised learning, this work proposes the use of credal sets to model uncertainty in pseudo-labels. By proposing simple scheme for generating and learning from these credal sets, the authors show how the FixMatch baseline can be extended and improved in settings where the number of labeled examples is small. Furthermore, they show that this approach can improve the efficiency of this semi-supervised learning approach and reduce calibration errors. 
main_review: The authors proposed used of credal sets to model uncertainty in the pseudo-labeling pipeline of semi-supervised learning is original, well-motivated, and demonstrated to be effective. The authors clearly motivate their approach and through description and accompanying pseudo code present the work in a manner than can be replicated and re-implemented. As both semi-supervised learning and uncertainty quantification are important and timely areas of interest within the machine learning community, this work has significance based on its ability to successfully combine important approaches in both areas. 

The paper seems to lack ablative experiments around various aspects of CSSL, such as exponential moving averaging, which would help clarify which portions of the proposed approach contributed to the improved performance the most significantly.

The paper is well-written, addresses an important shortcoming of existing ssl approaches (efficiency and generalization) a and presents a way to improve upon competitive baselines. 
limitations_and_societal_impact: The authors do not discuss limitation or societal impacts at all in their work, this is unfortunate as it represents and area where the authors could point to the importance of their work in various real-world settings.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to modify the pseudo-labeling procedure in semi-supervised learning. Student model predictions are penalized by distance to a credal set of teacher predictions. The size of credal sets is determined heuristically. Experiments demonstrate improvement over baselines mainly in scarce-label scenarios.
main_review: The central idea is clear, and the experiments demonstrate improvements over the SOTA method Fixmatch. However, there are a few weaknesses / questions:

1. The techniques seem like empirical hacks and lacks theoretical support. Can the authors demonstrate the advantage of the proposed method on a simple data generative model?
2. No ablations on the choice of $\alpha_i$. Equation 6 seems especially ad hoc and requires justification.
3. An important missing baseline is 'Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training' by Tai et al.
4. It is not clear to me why distributional matching doesn't work, whereas credal self-training does.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studied the credal sets, i.e., sets of probability distributions, in self-supervised learning. The author claimed that the increased expressiveness can improve performance and reduce biases. The proposed method is evaluated empirically on the image classification task.
main_review: ## Originality

The use of such credal probability and so-called "possibility distributions" seems rare in machine learning. The proposed method for learning from credal labels is based on the minimal value of the KL-divergence between probability in the set and the prediction, which originated from existing work (optimistic superset loss, infimum loss). The author then used credal labels in self-supervised learning with FixMatch. 

To the best of my knowledge, although its components are from existing work (credal label, infimum loss, FixMatch), the application of learning from credal labels in self-supervised learning is new.

---

## Soundness

My major concern is about the usefulness of the concept of credal sets in this setting. Indeed, hard labels may cause problems in neural network training. The author suggested that soft labels, probability distributions, are still not enough. However, the evidence is not convincing enough.

In traditional statistical learning, we want to learn a decision function that outputs labels, where both training data and prediction are deterministic ("hard label"). Since it's NP-hard to optimize it, we can resort to probabilistic models that output probabilities of labels, where the training data is still deterministic. It is possible because the expectation over the empirical distribution converges to the expectation over the true distribution as the size of the sample increases. There are a lot of attempts to let the model provide more uncertainty measures using, for example, Bayesian methods. The supervision may be still deterministic.

The direction of this paper is different from the ones above. Here, the uncertainty of the supervision is increased, from labels to probabilities of labels, to sets of probabilities of labels. Meanwhile, if I understand correctly, the output is still a single-point probability. My concern is, is the same effect can be achieved by, for example, regularizations of the weight or the output? Another straightforward approach would be using distributions over distributions of categorical labels, such as Dirichlet distribution, as the format of supervision. What's the advantage of the proposed method using credal labels?

---

## Technical quality

Another issue is that this method is verified solely empirically. Theoretical characterization is limited or from existing work.
limitations_and_societal_impact: The author discussed some limitations of this work through experiments in Section 4.
review_title: null

Meta-review: After the discussion, the reviewers agree that the paper proposes an interesting approach of using credal sets as a generalization of using probability estimates in self-supervision approaches. The resulting loss function is relatively simple and the authors describe one practical way to implement it. 

The paper is still borderline because the experimental results, even though they seem slightly better than the initial FixMatch approach, are not significantly better than other baselines, and additional ablations could be performed regarding some design choices, such as the effect of the weighting in Eq 6 or the exponential moving average of model weights. These ablations would be good to have, but since these come from previous work on the topic and the baselines are included in the main table, I do not feel that these are decisive arguments for rejection, even though they would be good to have. 

Overall, the approach proposed in the paper provides a cleaner way of using pseudo-labeling than the confidence matching, which is illustrated for instance in section on efficiency (learning curves might be better than a fixed number of epochs here), where we see the advantage of not requiring the confidence threshold as additional hyperparameter. In that respect, the approach can be used in practice even if it does not significantly improves performance compared to the optimally tuned competitors.
Decision: Accept (Poster)