Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: As noted in (Guo et al., 2017), modern neural networks are often miscalibrated. Pseudo-labeling based Semi-Supervised learning schemes are predicated on high confidence predictions from these neural networks. This paper posits that this miscalibration may lead to inferior results in confidence-based pseudo-labeling approaches. By taking into account the uncertainty of models and only using pseudo labels from high-confidence instances with low uncertainty this work presents a model that significant improves on other PL strategies and is competitive with consistency-based regularization strategies that comprise the current state-of-the-art.

Not only does this approach generate competitive results from with pseudo labeling strategy (which is compelling due to the history of PL approaches), but it is also less reliant on domain-specific augmentations that current consistency-based regularization approaches rely on. This is important for extending approach beyond domains where specific types of augmentation have been extensively studied. The results on the video domain offer some evidence of the usefulness in under-explored domains.


For the CIFAR (10/100) results presented it is unclear what data augmentations where used to produce the UPS results and how important these augmentations are to presented performance. As a key claim in this paper is de-emphasis of domain-specific augmentations would be beneficial to see a highlighted result strengthening this claim. 

Also, it seems the model is re-initialized and trained to convergence after each round of pseudo labeling. This would extremely compute intensive for large-scale semi-supervised learning tasks, which is a motivating use-case for semi-supervised learning. Providing results and comparisons on the compute requirements for this approach and its comparisons with competitors would be beneficial. As the re-initialization, is likely a major component in compute resources would also be beneficial for community to understand how essential this is to the improved performance numbers presented in the paper.
review_title: Through taking into account miscalibration in modern neural networks, this works produces a competitive Semi-Supervised Learning approach based Pseudo-Labeling 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strength:

The paper notices the problem in pseudo-labeling methods of semi-supervised learning, which is the erroneous prediction of pseudo-labeling. Pseudo-labeling is an easy-to-implement method for semi-supervised learning does not require constraints needed by consistency regularization methods, so improving pseudo-labeling methods can promote the practical use of semi-supervised learning.

The paper is well-written and easy to follow. 

The experimental results on datasets of different domains such as image, video show that the proposed method outperforms previous semi-supervised learning methods.



Constraint:

The paper proposes a new method to predict pseudo-labels of unlabeled data by confident threshold. However, confidence threshold is a widely-used method to decide pseudo-labels. And the threshold of confidence is hard to decide since different backbone network and different datasets have different confidence levels.

Furthermore, using both positive and negative labels for classification is also used in (Kim et al., 2019). The authors fails to discuss the difference of the usage of positive and negative labels between their method and (Kim et al., 2019). The authors also need to discuss what is the special challenge of using positive and negative labels in pseudo-labeling for semi-supervised learning.

The paper argues that the calibration error is greatly reduced with more certain predictions. However, the paper only empirically shows the relation but fails to demonstrate the claims or give intuition on the relation. Also, confidence itself is also an uncertainty measure, but the authors do not use the confidence for uncertainty but use a new uncertainty measure. Could the authors explain what uncertainty measurement do they use and why using the new measurement instead of the confidence?

The paper uses a fixed set of threshold hyper-parameters for all the experiments. However, the confidence-level for different datasets should be different. For example, the confidence for a real difficult dataset should be much lower than an easy dataset. The authors need to show that why using a set of hyper-parameters is enough and how to select the hyper-parameters. For example, showing the relation between the accuracy of pseudo-labels and the confidence.
review_title: The paper proposes a new pseudo-label selection method for pseudo-labeling based semi-supervised learning methods. The selection process labels positive and negative samples based on confidence and filter samples based on uncertainty. Experimental results show that the proposed method outperforms previous semi-supervised learning methods.

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary
This paper proposes uncertainty aware pseudo-labelling for semi-supervised learning, extending previously known methods by negative labels.

# Score justification
Well written paper and with extensive experiments with some points of improvement. The method should be positioned against others using confidence filtering and the role of calibration needs to be studied further.


# Strong and weak points
## Pros
Extensive experiments on different datasets and domains.
Broadly applicable method, that aims to improve conventional pseudo-labelling. Method is independent of uncertainty estimate and data augmentation.
Combining uncertainty estimates and confidence estimates, as well as adding negative learning are interesting approaches.

## Cons
Confidence filtering for pseudo-labels has already been suggested, see e.g. suggestions given in detailed comments.

I am unfortunately not convinced by the role of calibration, details given below. The authors could enhance the ablations studies with and without calibration and -importantly- adjusted thresholds, to clarify that point.



# Questions to the authors
The authors should consider e.g. https://arxiv.org/pdf/2002.02705 and https://www.microsoft.com/en-us/research/uploads/prod/2020/06/uncertainty_self_training_neurips_2020.pdf and position their work against those.

"Learning with UPS" have you treated pseudo-labels and original labels equally when training $f_{\theta,1}$, if so, how about fine-tuning on the original labels?

Please elaborate on the effect of different thresholds and how they were chosen. Especially with regards to calibration. 

# Detailed comments
The authors should consider e.g. https://arxiv.org/pdf/2002.02705 and https://www.microsoft.com/en-us/research/uploads/prod/2020/06/uncertainty_self_training_neurips_2020.pdf

Augmentations for other domains are only not effective, if they are not suitable for that domain. Augmentations should use domain specific invariances. From the paper it becomes clear, you did use data augmentation, not sure why you argue so strongly against it.

I am not sure if calibration plays a role here. Calibration only moves the distribution in shape, by setting a confidence threshold $\tau$ the threshold would be changed, but a suitable threshold could be found before and after calibration.

Section 3.1 small type: "psuedo" --> "pseudo"
review_title: Interesting approach with extensive experiments, some gaps remain

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper is in defense of simple semi-supervised learning (SSL) with pseudo-labeling (PL): authors demonstrate with experiments on 4 vision datasets (CIFAR-10, CIFAR-100, Pascal VOC and UCF-101) that pseudo-labeling can perform on par with consistency regularization methods. Authors argue that PL doesn't work well because of poor network calibration: because of that the high confident predictions are wrong leading to noisy training and poor generalization. The main contribution of the paper is the usage of prediction uncertainty selection in addition to the confidence-based selection which provides high accuracy of PL used in the further training. Besides this PL is generalized to create negative labels: with this authors perform and show effectiveness of negative learning and multi-label classification. Proposed approach performs in the same ballpark as state-of-the-art methods on CIFAR-10 and CIFAR-100, while it achieves the new state-of-the-art results on video dataset and multi-label task. It is worth to notice that proposed approach is independent from the domain while consistency regularization methods extensively are based on the specific augmentation techniques for the vision/datasets.

Pros:
- Cool idea on predictions uncertainty based selection for pseudo-labeling, no any dependence on domain-specific augmentations for the method
- Analysis of correlation between model calibration and prediction uncertainty
- Analysis of UPS compared to the conventional PL and confidence-based PL
- Ablation study on each component of the method, and dependence on the network architecture
- Well-designed (fair comparison) extensive experiments and comparisons on 4 dataset for multiclass and multi-label classification with better performance than other methods

Cons:
- Absent of large-scale experiments with ImageNet

Comments:
- typo page 3 "would lead to binary psuedo-labels" -> "would lead to binary pseudo-labels"
- "For the multi-label case, $\gamma = 0.5$ would lead to binary pseudo-labels, in which multiple classes can be present in one sample." - this sentence is not clear. If $\gamma = 0.5$ it could be only one or two classes presented in the pseudo-label vector.
- Eq. (2), it is obvious but still please specify that $\tau_p >= \tau_n$
- Figure 1 (b) and (c) - on which data is this analysis done?
- Do authors use the same network $f_{\theta, k}$ on each PL iteration $k$ and just randomly reinitialize it?
- What is $\gamma$ value in experiments?
- For Table 1 would be good to have a clarification on baselines: which are consistency regularization based, which are PL. 
- typo in footnote 2 on page 7: add dot at the end of sentence.
- typo page 7 "experimental set-ups." -> "experimental setups.", "labeled samples Both" -> "labeled samples. Both"
- Did authors try experiments on ImageNet too?
- On which data is study in Fig.2 performed?
- typo page 8 "unique in that it can be easily" -> "unique in that: it can be easily" (or any punctuation here)
- For ECE computation is percentile binning used?
- some possible relevant works: https://arxiv.org/pdf/2003.03773.pdf, https://arxiv.org/abs/2006.07733, simCLR v2, https://arxiv.org/abs/2006.09882

It is very well written paper with extensive experiments and ablations (except large-scale experiment), which prove the method efficiency and generalization. Hope, this will push the study of simple SSL approach, pseudo-labeling, with the new competitive results not only in vision but in other domains too.
review_title: Pseudo-labeling can be on par or better than consistency regularization methods. Method which is **not** based on specific to vision data augmentation.

Meta-review: The paper is written in defense of pseudo-labeling. The authors `aim at demonstrating that pseudo-labeling based methods can perform on par with consistency regularization methods which have been show to achieve strong performance.

The paper is well-written and easy-to-follow. The reviewers are generally positive about the contribution. It has to be underline, however, that pseudo-labeling is still a controversial approach with a very limited theoretical understanding. This paper does not provide any further understanding of it, but proposes several "heuristics", intuitively well-motivated, but justified only experimentally. Nevertheless, the results are promising and the paper is an important voice in the general discussion around learning with weak labels and semi-supervised learning, two crucial problems in many practical applications. Taking this into account I recommend to accept the paper as a poster. 

The reviewers have raised several problems that the authors have been exhaustively discussing in their rebuttals. The one remaining issue is the interaction between calibration and the threshold. This problem has to be clarified in the final version of the paper, as indeed calibration usually does not change the order.
Decision: Accept (Poster)