Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper leverages approximate Bayesian techniques, such as approximate Bayesian neural networks to quantify the uncertainty and improve the performance on the uncalibrated models. This paper is mainly a summarization of recently introduced techniques in SSL and the
contribution in terms of exploited techniques being commonly used in the SSL framework
strength_and_weaknesses: Strengths:
The paper is well-written and clear.

Weaknesses:
The main weaknesses rest on the novelty of the introduced techniques.
clarity,_quality,_novelty_and_reproducibility: Many of the techniques introduced in the paper have been previously introduced for semi-supervised learning. There are also some issues regarding the paper presentation and claims which are detailed below:

1- The authors introduced ECE metric and they claimed a strong correlation between SSL accuracy and ECE. However, Figure 1 does not show a strong correlation between ECE and accuracy. For example, for FixMatch while the ECE for the threshold of 0.99 is the lowest, the accuracy of this specific case is lower than the threshold of 0.95 with the ECE over 0.18.

2- The paper claims "propose and explore weight averaging approaches" which is the commonly-used teacher-student framework in SSL.

3- Following the introduction of ECE metric, a loss called ELBO based on negative log-likelihood is introduced. However, it is not clear and never discussed how this loss can control ECE.

4- Regarding the experiments the authors "consider the median of 20 checkpoints around the best accuracy checkpoint as the convergence criteria, and report this value as the test accuracy". However, I believe the test accuracies should be reported in the standard format of SSL methods by reporting the variance of the results of different runs with different seeds.

5- It is not clear in the experiments how M weights from the BNN layer are sampled. Did you use drop-out for it?

6- In the abstract, the author claimed up to 15.9% accuracy across different datasets however the improvement is only limited to BAM-UDA on CIFAR-100 with 400 labels.

7- In the experiments, for BAM-PAWS over CIFAR-10 with 250 labels application of SWA or EMA dramatically decreased the accuracy which is unusual. Weight averaging has shown established improvements in the SSL framework. Similarly, for CIFAR-100 with 4000 labels, the results show the same ECE 0.193 while showing different test accuracies.

8- In the experiments, the results should be compared with uncertainty quantification methods.

minor issues

-- In Figure UDA, FixMatch, BAM-UDA across training on CIFAR-100 with 400 labels are compared. However, FixMatch needs to be compared with BAM-FixMatch instead of BAM-UDA.
-- The definition of L_d^l and \delta needs to be presented in Corollary 1.
-- typo: BAyesian Model averaging (BAM)
-- In the supplements, for the proof of Theorem 1 the terms with 1/N_u and 1/N_l cannot be merged to 1/N.  
summary_of_the_review: The contributions from this paper are not significant and many of the discussed topics and techniques for SSL are well explored in the literature. In the experiments, the variance of different runs with different random seeds needs to be reported.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to improve the pseudo-label in semi-supervised learning with a well-calibrated model. To that end, it improves the calibration with Bayesian model averaging. It also provides a theoretical result for the benefit of the calibration in semi-supervised learning for the task of classification. The approach is evaluated on CIFAR-10, CIFAR-100 and ImageNet where it clearly demonstrates the advantage of maintaining a calibrated classifier. 

strength_and_weaknesses: Strength:

- The paper is well-written and easy to follow. The presented contribution is clearly presented and thus it is compared with the prior work. 

- The presented observations can have an impact on a wide range of semi-supervised learning tasks. Moreover, the theoretical result further supports the paper.

- The results are convincing in the three evaluations. 

- The paper presents several ablation studies to support why improving the classifier calibration would improve the performance.


Weaknesses: 

- It would be interesting to examine more calibration approaches. It is interesting to know if only the presented calibration approach has a positive impact on semi-supervised learning or in general any calibration approach would improve it. Making a more general claim would add significant value to the paper. 

- The approach is applied only for classification. Similarly, it could have been applied to the classifier of an object detector. 

- Although the paper makes interesting observations, it does not show new elements. For instance, the EMA update is known, as well as the rest parts of the presented methodology. However, the combination of the calibration with semi-supervised learning is a solid contribution.
clarity,_quality,_novelty_and_reproducibility: The paper is well-presented and makes a practical contribution for semi-supervised learning. Reproducibility is possible through the provided code, but more implementation details can be helpful. 
summary_of_the_review: The paper shows strong evidence of why the model calibration can improve the performance in semi-supervised learning. It is well-written, with complete related work and a detailed experimental section. Overall, it has potential to be accepted, but there are still some parts to be further worked.


Post-rebuttal comments:

The rebuttal discusses all open questions from the reviews. Regarding my concerns: 

- The argumentation that is not possible to compare with another approach does not help the paper. It is necessary to provide more references of comparison for understanding the practical contribution of the paper. 

- The lack of novelty is mentioned in another review too. As mentioned, I find the combination a solid contribution but not enough for acceptance. A strong evaluation would add more contribution to the paper (above point).

For these reasons, I will lower my score. The paper needs additional work at the current stage.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work is motivated by the poor model calibration which affects semi-supervised learning. Bayesian network and weight-averaging techniques are adopted to improve model calibration and it has been demonstrated to improve semi-supervised learning efficacy.

strength_and_weaknesses: Strength:

1. Deep learning model are poorly calibrated. This could affect semi-supervised learning and this work addressed this important problem.

2. Demonstrating a strong correlation between SSL performance and model calibration.

Weakness:

1. The Bayesian network only considers the last layer. In particular, for the pseudo label approach only the classifier head is sampled multiple times. Does this really reveal the uncertainty of a network? This is equivalent to perturbing the decision boundary, the samples that are close to decision boundary are more likely to yield high variance and will be pruned out for self-training. In another words, this can be seen as another way of thresholding.

2. The improvement from FixMatch is very marginal. Does this suggest BAM is another way of doing thresholding.

3. The threshold on Bayesian prediction exploits the distribution of standard deviation on unlabeled data. In contrast, FixMatch does not exploit this information. Thus, it is vital to verify whether the improvement is attributed to a more carefully chosen threshold.

4. EMA parameter update has been studied in existing SSL methods, e.g. mean teacher.

5. It has been reported that confirmation bias is more severe at low label regime. If improving model calibration is effective, it is recommended to evaluate at even lower label regime, e.g. CIFAR-10 @ 40 labeled.

clarity,_quality,_novelty_and_reproducibility: This paper presents the idea clearly. The technical novelty is limited. The addressed problem of poor model calibration is a realistic problem. The solutions should be easy to reproduce.

summary_of_the_review: Overall, this paper addressed an important problem, e.g. poor model calibration affects SSL performance. However, some components, e.g. using the quantile to estimate threshold, of the proposed method might give unfair advantages over the existing method. The improvement from existing methods is also marginal, thus hard to quantify the significance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper centers around one problem of improving calibration under the scenario of semi-supervised (deep) learning. Towards this end it proposed two methods: the bayesian model averaging based and the weight averaging based. Empirical verifications are conducted on several tasks which demonstrate the improvement both in terms of test set accuracy and the calibration (measured by ECE).

It also points out the correlation between calibration and accuracy as a side outcome.
strength_and_weaknesses: **Strength**

1. The problem this paper considers is important: improving calibration under SSL. Calibration is indeed a very important metric in many real-world industry settings (like computational advertising) while is a comparatively neglected thing in deep learning research, let alone for the specific setting of semi-supervised learning. It's great that the authors could dive deeply into this important problem which might benefit the success of (semi-supervised) deep learning method in real-world scenarios. 

2. This paper is stated in a very clear way, and its technical soundness is good. I enjoy reading the whole contents which give readers a good background introduction and "logical flow".

3. The empirical evaluation looks comprehensive and sound.

**Weakness**

1. The technical innovations seems limited: it seems strait-forward to directly adopt the two methods (i.e., bayesian based and weights averaging based) to the scenario of SSL. There is no specific innovations here given no unique challenges (not in terms of theoretical but in terms of algorithmic) when we want to combine the two worlds, that is previous "basic" methods already established and SSL.
clarity,_quality,_novelty_and_reproducibility: **Quality and Clarity**

Good. As stated above, this paper is technically sound and comprehensive with an excellent background introduction. Furthermore its writing is also user-friendly. 

**Novelty/Originality**

Seems limited as it is a work brining some already built techniques to a particular scenario. It would be a good application paper but not a paper showing great ML technical depth.

Furthermore the relationship of calibration and accuracy seems not new as well despite that it might be the first-time to state it in SSL. It is a not a rarely known thing in typically supervised learning scenario that calibration at least correlates with accuracy. A (maybe not that good) example is the good (training time) calibration of a simple logistic regression model is just theoretically derived under the assumption that it goes to a perfect solver state (i.e., gradient coming to zero for its bias term).

**Reproducibility**

Good

summary_of_the_review: As stated above, this paper is very well motivated, technically sound and written in a very clear way. Meanwhile its major limitation is on its marginal technical innovation since it is a work connecting the "raw" methods to a specific ML application scenario.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper proposes a new semi-supervised learning algorithm based on the observation that classifier accuracy and calibration are correlated in the context of SSL. Reviewers generally appreciated the importance of studying calibration in the SSL context, the clear presentation, and promising results in the evaluations. However, they had concerns about the technical innovation of the method given that its elements are well-studied in the SSL context, as well as the lack of comparison to other techniques that can also improve calibration. The authors provided responses to these concerns, but did not include additional comparisons to other techniques; reviewers did not feel their concerns were adequately addressed by the response. Overall, reviewers unanimously lean towards rejection due to the novelty concerns and lack of comparison to other techniques. 

The AC appreciates the perspective that the combination of techniques used to address the identified calibration issue in SSL is novel and interesting, but also agrees that other calibration techniques should be included in the experiments. This could help to further motivate the design of the proposed method and to provide more evidence that calibration is indeed the mechanism behind the improved performance; these were also points of concern for reviewers. The empirical evidence used to support the correlation between calibration and accuracy could also be further strengthened by evaluating over more settings (datasets, models, label budgets). The authors are encouraged to strengthen the submission accordingly for submission to a future venue.
justification_for_why_not_higher_score: - Lack of comparisons to other calibration techniques 
- Lack of specific novel techniques to address the identified problem
Decision: Reject