Mitigating Confirmation Bias in Semi-supervised Learning via Efficient Bayesian Model Averaging
Authors: Charlotte Loh
Rumen Dangovski
Shivchander Sudalairaj
Seungwook Han
Ligong Han
Leonid Karlinsky
Marin Soljacic
Akash Srivastava
Venue: Accepted by TMLR ()

Abstract: State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data, often via self-training or pseudo-labeling. During pseudo-labeling, the model's predictions on unlabeled data are used for training and may result in confirmation bias where the model reinforces its own mistakes. In this work, we show that SOTA SSL methods often suffer from confirmation bias and demonstrate that this is often a result of using a poorly calibrated classifier for pseudo labeling. We introduce BaM-SSL, an efficient Bayesian Model averaging technique that improves uncertainty quantification in SSL methods with limited computational or memory overhead. We demonstrate that BaM-SSL mitigates confirmation bias in SOTA SSL methods across standard vision benchmarks of CIFAR-10, CIFAR-100, giving up to 16% improvement in test accuracy on the CIFAR-100 with 400 labels benchmark. Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.

Published in Transactions on Machine Learning Research (08/2023)
Mitigating Confirmation Bias in Semi-supervised Learning via Efficient Bayesian Model Averaging
Charlotte Loh MIT EECS MIT-IBM Walson Al Lab
cloh@mmit edu
Rumen Dangovski
rumenrd@mii. edu
MIT EECS
Shivchander Sudalairaj MIT-IBM Watson AI Lab
shiv.er@ibbm.com com
Scungwook Han
swhann@ibm com
MIT EECS
MIT-IBM Watson Al Lab
Ligong Han
Damligong@gmaiil.com com
Rutgers University
MIT-IBM Watson AI Lab
Leonid Karlinsky
leonidka@ibm
MIT-IBM Watson AI Lab
Marin Soljačié
soljacio@mit. edu
MIT Physics
Akash Srivastava MIT-IBM Watson Al Lab
abash.erivvsttruu@B0m..cmm
Reviewed on OpenReview: https: // openreview. net/ forum? id= PBrK0aDQ:00
Abstract
State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful leveraging a mix of labeled and unlabeled data, often via self-training or pseudo-labeling, During pseudo-labeling, the model's predictions on unlabeled data are used for training and may result in confirmation bias where the model reinforces its mistakes. In this work, show that SOTA SSL methods often sulfer from confirmation bias and demonstrate that this is often a result of using a poorly calibrated classifier for pseudo labeling. We introduce BaM- SSL, an efficient Bayesian Model averaging technique that improves uncertainty quantification in SSL methods with limited computational or memory overhead. We demonstrate that BaM- SSL mitigates confirmation bias SOTA SSL methods across standard vision benchmarks of CIFAR-10, CIFAR- 100 and ImageNet, giving up 16% improvement test accuracy on the CIFAR-100 with 400 labels benchmark. Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class- inaslimnal datasets and in photonics science.

---

Published in Transactions on Machine Learning Research (08/2023)
1 Introduction
While deep learning has achieved unprecedented success in recent years, its reliance on vast amounts of labeled data remains a long standing challenge. Semi-supervised learning (SSL) aims to mitigate this by leveraging unlabeled samples in combination with a limited set of annotated data. In computer vision,
two powerful techniques that have emerged are consistency regularization (Bachman et al. 20044] Sajjadi ct al. 2016) and pscudo-labcling (also known as self tolf-training) (Rosenberg et al., 2005] Xie et al. 2019b/. Broadly, y; consistency regularization enforces that random perturbations of the unlabeled inputs produce similar predictions, while pscudo-labeling assigns artificial labels to unlabeled samples, which are then used to train the model. These two techniques are typically combined by minimizing the cross-entropy between pseudo-labels and predictions that are derived from differently augmented inputs, and have led to strong
performances on vision benchmarks (Sohn et al. 2020 Assran et al. 2021
In many SOTA SSL methods, a selection metric (Lee)
2013 Sohn et al. 2020 based on the model's confidence
is often used in conjunction with pseudo-labeling, where only confident pseudo-labels are selected to update the model. As such, there is need for proper confidence estimates; in other words, the calibration of the model should be of paramount importance. Model calibration (Guo et al. 2017 can be understood as measure of how a model's ss output truthfully quantifies its predictive uncertainly, i.e. it denotes the aligmment between its prediction confidence and its fround-trutl accuracy. Apart from the importance of calibration arising from the selection metric, the use of cross-cntropy minimization objectives common in SSL implics that models will naturally be driven to output high-confidence predictions (Grandvalet & Bengio 2004). Having confdcooeeeence predictions is highly desirable in SSL since we want the decision boundary to lie low-density regions of the data manifold, i.e. away from labeled data points (Murphy) (022).. However, without proper calibration, model would easily become over-confident. This is highly detrimental as the model would be encouraged to reinforce its mistakes, resulting in the phenomenon commonly known as confirmation bias (Arazo et. al. 2019
In this work, we propose to mitigate confirmation bias in semi-supervised learning by incorporating approximate Bayesian techniques, which have been widely known to improve uncertainty estimates (Wilson & Izmailov 2020). Our main approach, BaM- performs Bayesian Model averaging during pseudo-labeling by incorporating a Bayesian last layer to the model and is illustrated in Fig. 1 Broadly, BaM- can be described as follows; instead of single-fixed-value weights, the last. layer are parameterized Gaussian random variables and two main modifications are made during pseudo-labeling: 1) multiple weight samples are drawn from the layer and averaged to derive the predictions and 2) the selection criterion is based on their postcrior variances (further details follow in Section 4.1). We further contextualize the novelty of BaM- against prior art in the SSL literature in Table 1 In contrast to prior hedho BaM- is designed to specifically target improving model calibration in order to mitigate confirmation bias.
Our contributions are summarized as follows:
1. We introduce BaM-, which is designed to mitigate confirmation bias via Bayesian model averaging in SOTA SSL methods based upon selection metric. BaM- incorporates two new features to improve uncertainty estimation: 1) bayesian averaging over multiple weight samples and 2) selection metric based on the variance of the predictions
2. We empirically demonstrate that BaM- effectively improves model calibration, resulting in better performances standard benchmarks like CIFAR-10 and CIFAR-100, notably giving up to 16% gains in test accuracy.
We also demonstrate the efficacy of BaM- in more challenging and realistic sccnarios, such as class-imbalanced datasets and real-world application in photonic science.
2 Related Work
Semi-supervised learning (SSL) and confirmation bias. A fundamental problem in SSL methods based pseudo-labeling (Rosenberg et al. 2005) is that of confirmation bias (Taraainen & Valpola 2017
2

---

Published in Transactions on Machine Learning Research (08/2023)
Weak
Average many predictions
Boyesian layer
Acceptance criteria
Unlabeled
Pseudo-label
1 H(p.q)
Strong
Model
Prediction
Figure 1: Illustration of our Bayesian Model averaging (BaM) approach. The last layer of the model has weights that are represented by probability distributions rather than the typical single fixed value. BaM- incorporates two main modifications to traditional semi-supervised learning approach: 1) During pseudo-labeling of each unlabeled data point, multiple weights are sampled and averaged to derive the prediction; and 2) the selection criteria is based upon the variance over predictions.
Table 1: Comparison of techniques used in BaM- with techniques used in prior art of SSL.
SSL method
Augmentation
Pseudo-label
Selection metric
Temporal ensemble (Laine & Aila
2016
Weak
Model from earlier step
Mean teacher (Tarainen & Val pola
201
Weak
FMA
UDA (Xie et al.
2209a
Weak & strong
Sharpen
Logit thresholding
MixMatch Belth al.
2019
Weak
Averaging aug sharpen
FixMatch Sohn al. 2020
Weak strong
Hard labels
Logit thresholding Class-wise logit threshold
Fltx
Zhang et al. 2021
Weak & strong
Hard labels
Our main method BaM-
Weak strong
Posterior sampling sharpen Variance thresholding
Murphy 2022). i.c. the phenomenon where a model overfits to incorrect pseudo-labels. Several strategies have
emerged to tackle this problem: Guo et al. (220))
and and|Ren et al. (020) looked into weighting unlabeled samples,
Thulasidasan et al. (2019) and Arazo et al.
(2019 proposes use augmentation strategies like MixUp (Zhang et al. 2017). while Cascante-Bonilla et al.
(2020) proposes to re-initialize the model before every iteration to
overcome confirmation bias. Another popular technique is to impose a selection metric (Yarowsky 1995 to
retain only the highest quality pseudo-labels, commonly realized via fixed threshold the maximum class probability (Xie et al.) 2019aa Sohn et al.) 2020). Recent works have further extended such selection metrics to be based on dynamic thresholds, either time (Xu et al.) 2021) or class-wise (Zou et al.) 2018 Zhang ee al. 2021). Different from the above approaches, our work proposes to overcome confirmation bias in SSL by directly improving the calibration of the model through approximate Bayesian techniques.
Model calibration and uncertainty quantification. Proper estimation of a network's prediction uncertainty of practical importance (Amodei et al.) 2016) and has been widely studied. A common approach to improve uncertainty estimates is via Bayesian marginalization (Wilson & Izmailov 2020 i.e. by weighting solutions by their posterior probabilities. Since exact Bayesian inference is computationally intractable for neural networks, a series of approximate Bayesian methods have emerged, such as variational methods (Graves)
2011 Blundell et al.) 2015 Kingma et al.) 2015), Hamiltonian methods (Springenberg et al. 2016) and
Langevin diffusion methods Welling & Teh 2011). Other methods to achieve Bayesian marginalization also
exist, such deep ensembles (Lakshminarayanan et al. 2016) and efficient versions of them (Wen et al. 2020
Gal & Ghahramani] 2015), which have been empirically shown improve uncertainty quantification. The concept of uncertainty and calibration are inherently related, where calibration is commonly interpreted the frequentist notion of uncortainty. It is known that a well specified Bayesian model (i.c. one where the prior captures the model uncertainty) has well-calibrated posterior (Gelman et al. 2021). Motivated by this, in our work we adopt some approximate bayesian techniques specifically for the context of semi-supervised learning in order to improve model calibration during pscudo-labeling and empirically validate their effectiveness. While other methods for improving model calibration exists (Platt) 1999: Zadrozny & Elkan 2002 Guo
et al. 2017), these most. commonly achieved in post-hoc manner using held-out validation set; instead,
3

---

Published in Transactions on Machine Learning Research (08/2023)
seek to improve calibration during training and with scarce set of labels. In the intersection of SSL and calibration, Rizve et al. (2211) proposes to leverage uncertainly to select better calibrated subset of pseudo-labels. Our work builds on similar motivation, however, in addition to improving the selection metric with uncortainty estimates, we show that directly incorporating approximate Bayosian techniques into SSL methods can indeed improve calibration via its better-calibrated approximate posterior. Finally, our work also bears some close resemblance to acquistion functions used in Bayesian active learning. There, we sook data points for which the parameters under the posterior disagree about the outcome the most (Houlsby et al,) 2011 Kirsch al. 2019). In contrast, in semi-supervised learning where the goal is to reduce confirmation bias by selecting samples where the network is most certain about predicting, we instead seek data points for which the parameters under the posterior agree the most.
3 Notation and Background
Given a small amount of labeled data L = {(111 y1 ) (here, yr E {0,1) are one-hot labels) and a large amount of unlabeled data u - { i.e. N₂ > Ni, in SSL, we seek to perform a K-class classification task. Let f(;,6) 0y) be a backbone encoder (e.g. ResNet or WideResNet) with trainable parameters fy, likewise let h(,00p (x) be linear classification head, and H denote the standard cross-entropy loss.
SSL methods based on a selection metric. Many SSL methods such as Pseudo-Labels Lee 2013), UDA (Xie et al., 2019a) and FixMatch (Sohn et al. 2020) use a selection metric in conjunction with pscudo-labeling to achieve SOTA performance. These methods minimizes a cross-entropy loss on augmented copies of unlabeled samples whose confidence exceeds a pre-defined threshold. Let aj and a2 denote two augmentation transformations and their corresponding network predictions for sample to be qi = (a (x)) and 92 = ho / ((z(2))), the total loss on a batch of unlabeled data has the following form:
BB
B.B Σ
1 (max((1..) 2
+) H) (2.u)
(1)
where B denotes the batch-size of labeled examples, / a scaling hyperparameter for the unlabeled batch-size, E [0, 1] is threshold parameter often set close to 1. Pr is either a sharpening operation on the pseudo-labels, i.c. [x<())] == [/ -= when soft-pseudo-labels when hard pscudo-labels are used. In the latter, only about the class where the maximum logit occurs. Pt also implicitly includes a "stop-gradient" operation, i.e. gradients are not back-propagated from predictions of lsbudo-labels. Lu is combined with the expected cross-entropy loss on labeled examples, 1 Σe, (y.. 91,1) to form the combined loss L + ALu, with a scaling hyperparameter A. Differences L = B between Pseudo-Labels, UDA and FixMatch are detailed in Appendix D.I
used, a one-hot operation (i.e. t - 0)
Calibration metrics. A popular empirical metric to measure a model's calibration is via the Expected Calibration Error (ECE). Following (Guo et al. 2017 Minderer et al. 2021), we focus on a slightly weaker condition and consider only the model's most likely class-rredccton, which can be computed as follows. Lot Jmax denote the model's confidence, or the prediction at the most likely class (i.c. the maximum logit value after the softmax), the model's confidence on batch of N samples are grouped into M equal-interval bins, i.e. Bn, contains the set of samples with Imax E (m-1 m J. ECE is then computed the expected difference between the accuracy and confidence of cach bin over all N samples:
M
Bm acc(BB))conftee conf(Bm))
ECE Σ
where acc( = (L/ Eies... (argmax((g) = y1) and cBnf(Bm)) = (L/ EieB.. 4max, with 3/4 the true label of sample In this work, we estimate ECE using M 10 bins. We also caveat here that while ECE is not free from biases (Minderer et al. 2021),, we chose ECE over alternatives (Brier] 1950 DeGroot & Fienberg 1983) due to its simplicity and widespread adoption.
4

---

Published in Transactions on Machine Learning Research (08/2023)
4 Mitigating Confirmation Bias in Semi-supervised learning
As we see from Eq. (1)
the model's confidence (the maximum softmaxed logit value) is used to determine if the pseudo-label of a particular unlabeled data point is used to update the model; as such it is important for the model to have proper confidence estimates, i.e. to be well-calibrated. More recently, temperature
scaling (Guo et al. 2017) or other similar methods (Platt 1999 Zadrozny & Elkam 2002) have been highly effective towards model calibration. However, such methods pose several challenges in the semi-supervised setting; 1) they operate post-hoc, ie.. after training is completed, while in SSL the model needs to be calibrated constantly during training to reduce confirmation bias of pseudo-labels 2) these methods use a held-out labeled set (typically around 10% of the total datasct size (Guo et al.) (017 ) to perform calibration,
while common benchmarks of SSL typically have label percentages less than that (up to as little as <1%),
making it challenging to create a held-out set with so little data to begin with.
4.1 Mitigating Confirmation Bias with Bayesian Model Averaging
Given the above limitations of post-hoc calibration methods, in this work we propose to incorporate approxmate Bayesian methods such as Bayesian Neural Networks (BNN) into cxisting SSL methods. Bayesian models, when well-specified (i.e. where the prior captures the model's uncertainty); are known to produce well-calibrated posteriors (Gelman et al. 2021) and approximate Bayesian techniques have been widely empirically shown to produce well-calibrated uncertainty estimates in deep neural networks (Wilson &
Izmailov 2020
Lakshminarayanan et al. 2016 Blundell et al. 2015).
Implementing a last-layer BNN. In order to minimize the computational overhead and reduce the risk of overall poorer model accuracy arising from a full Bayesian approach Wenzel et al. (2020), we propose to only replace the final layer of the network, i.e. the linear classification head h, with a BNN layer. While there may be several options towards the implementation of the BNN layer, we propose to use a BNN with a variational posterior trained via stochastic variational inference (SVT) for computational efficiency. This would allow one to optimize both the non- bayesian backbone and the BNN layer simulataneously in a single backward pass, opposed to other Bayesian approaches such Hamiltonian Monte Carlo NNcal 2012) which may require separate optimization loops.
Stochastic variational inference in BaM- For notation convenience, we denote the input embedding to the BNN layer to be 0 in this section. In SVI, we first assume a prior distribution on weights p(0n).. Given some training data Dx == (X, Y), we seek to calculate the posterior distribution of weights, p(0% p(0%)Dx), Dx), which can then be used to derive the posterior predictive p(yle, Dx) = p(ylv, 0n (p0, Dd0), This process is also known as " "Bayesian model averaging" or "Bayesian marginalization" " (Wilson & Izmailov 2020). Since exact. Bayesian inference is computationally intractable for neural networks, we adopt variational approach following [Bundell et al. (2015), where learn Gaussian variational approximation to the posterior qo (0, Φ), paramcterized by d. by maximizing the evidence lower-bound (ELBO) (sce Appendix C.1 for details). The ELBO = E,, (( (Y (X||X;)) - K [(q(O)))(lp(() consists of a log-likelihood (dala-dependent) term and KL (prior-dependent) term. We provide some preliminary theoretical connections to generalization bounds via Corollary I in Appendix A Corollary I shows that the generalization error is upper bounded by the negative ELBO, i.c. by maximizing the ELBO we may improve generalization. Apart from the last layer, the rest of the network is non-bayesian and are point values which are trained via regular Maximum Likelihood Estimation (MLE).
Pseudo-labeling via BaM- As depicted in F Fig. 1 pseudo-labeling in BaM- proceeds in two-stages: 1) M weights from the BNN layer are sampled and predictions are derived from the Monte Carlo estimated posterior predictive, i.e. g = (1/M) EM h(v, ((m)). and 2) the selection criteria is based upon their variance, -f = (1/M) ΣM (h(v, o(m))) i)2, at the predicted class c = argmax [o]e. This constitutes a more intuitive measure of model uncertainty compared to the maximum logit value commonly used in prior SSL methods which does not have an uncertainty interpretation. In section 7, we verify through ablations that the variance of predictions is indeed more effective than the maximum logit value for mitigating confirmation bias. The variance is also highly intuitive - if the model's prediction has a large variance, it is highly uncertain
5

---

Published in Transactions on Machine Learning Research (08/2023)
and the pseudo-label should not be accepted. We later show that better uncertainty estimates from BaM- effectively mitigates confirmation bias. In practice, og decreases across training, we use a simple quantile Q over the batch to define the threshold where pseudo-labes of samples with of < Q are accepted, with Q as a hyperparameter. Algorittm 1 shows a snippet of pseudo-code to highlight the main modifications introduced by BaM- during pseudo-labeling (a more complete version of the pseudo-code can be found Appendix C.
We explore the effectiveness of BaM- by modifying upon SOTA SSL methods and denote them with the BaM- suffix, i.e. "BaM-X" incorporates approximate Bayesian Model averaging (BaM) during pseudo-labeling for SSL method X.
5 Experimental Setup
In all our experiments, we begin with and modify upon the original implementations of the baseline SSL methods. The backbone encoder f a Wide ResNet-28-2 and Wide ResNet- for the CIFAR-10 and CIFAR-100 benchmarks respectively. We use the default hyperparameters and dataset-specifie settings (learning rates, batch size, optimizers and schedulers) recommended by the original authors for both the baselines and in BaM- We set the weight priors in BaM- as unit Gaussians and optimizer for the BNN layer with learning rate 0.01, weight decay and impose the same cosine learning rate scheduler as the backbone. We set Q = 0.75 for the CIFAR-100 benchmark and Q 0.95 for the CIFAR-10 benchmark; which are both linearly warmed-up from 0.1 in the first 10 epochs. As Q is computed across batches, we improve stability by using a moving average the last 50 quantiles.
separate Adam
ECE and test accuracy evaluation. In our experiments, we found that the test accuracy exhibits considerable amount of noise across training, especially in label-scarce settings. Sohn et al. (2020) proposes to take the median accuracy of the last 20 checkpoints, while Zhang et al. (2021) argues that this fixed training budget approach is not suitable when convergence speeds of the algorithms are different (as the faster converging algorithm would over-fit more severely at the end) and thus report also the overall best accuracy. our experiments, we adopt a balance between the two aforementioned approaches: we consider the median of checkpoints around the best accuracy checkpoint as the convergence criteria, and report this value as the test. accuracy. The ECE reported when the model reaches this convergence criteria (one could also also aggregate the ECEs up till convergence we found this gave similar trends and thus report the simpler metric).
Algorithm Snippet of PyTorch-style pseudocode showing pseudo-labeling in BaM-UDA.
D: quantile paraneter
nun samples: number of reight sanples
q. g_ltst 0
for s_veak, x_strong in unlabeled loader:
z_veak, z_strong
incoder(x_ven) bayes
encoder (x_ (x_atrong) get representations
nean_veak, veak
(bayes_classiffi 2_veak) # get nean and atd of predictions accept_ (quaniilleeeeee (std_ q))
q. g. (0) if (an(q,,isst) > 50 keep 50 nost recent quantiles g.list a.pieddpppedd..... append
std_veak.leqq,2att rean
acceptance
samples
conpute unlabeled loss using soft peeudo llbbll accepted samples loss, unlab cross_ croas_entropy entropy_ (bayes, (x, sharpen (nean_weak)) accept loss_kl ki = "i_loss (hayee classifirr)) (byaeaaaaaccaasstiierrr loss loss_ unlab los8_x1
# dependden (data- data-inndepndent) loss
def gyvas_predict(h z):
outputa = atack stack((h(2) (Ia(2)) aoftnax(-1) for range (wun_ samples)] sanple weights return outputs m)an(), oud mean std of predictions

---

Published in Transactions on Machine Learning Research (08/2023)
Table 2: BaM- in SSL showing "Test accuracy (%) / ECE". BaM- improves calibration and result in better test accuracies, consistently across all benchmarks and across two SSL methods, FixMatch (FM) (Sohn et al. 2020) and UDA (Xie et al, 2019a). For each benchmark, results are averaged over random dataset splits.
CIFAR-10
CIFAR-100
250 libels
2500 labels
400 Jabels
1000 labels
10000 lobels
FM (tepro) BoM-FM (ours)
16 0.0466a0m
96. 73..0 0.009
3..4... / 0.366.
74.2.0 0.183
78.1 / 0.
95.1 0.04izmoo
95.7 / 0.000ase
58.0s14 (*2.6) / 0.331soms
74.8-mo
(10.6) / 0.177a00m
78.1s02 / 0.1399aa
UDA (repre)
94.1 / 0.033
75.7 / 0.039
44.1 / 0.473
72.9ge011 /
77.2s0a / 0. 0.1154g..00
BaM-UDA (ouns)
95.2. (f1.) c.042.c.o
93.9 on (10.2 / 0.038
60.3. (116.2) / 0.314
76.2101 (12.3) / 0.165
78.3 (11.) / 0.138
Test Accuracy
ECE
Selection Purity (%)
70
0.60
80
(a)
(b)
(c)
63
0.55
57
67
51
0.50
1.00
(d)
Max. logit
45
0.45
55-
0.95
38
FM
0.40
0.90
32
UDA
42-
26
BaM-FM lours)
0.35
BaM UDA (aurs)
20
0.30
30
0.80
0k 200k
400k 600k
so0k 1000k
200k 400k 600k 80Ok 1000k
200k 480k 600k sook 1000k
500k
1000k
Iterations
Iterations
terations
Iterations
Figure 2: FixMatch, UDA, BaM-FM and BaM-UDA across training on CIFAR-100 with 400 labels (a) Test accuracies and (b) ECE as a function of training time. (c) Selection purity shows the accuracy of unlabeled samples that are accepted by the selection metric and (d) Max. logit shows the model's confidence, the average maximum class probability of all unlabeled samples.
6
Results
6.1 BaM- improves model calibration and consistently leads to better SSL performances
Results on the CIFAR-10 and CIFAR-100 benchmarks for various number of labels are depicted in Table 2 Table 2 demonstrates that BaM- successfully reduces the ECE over the baselines across all the benchmarks and as a result, we also attained significant improvements in test accuracies, notably up to 16.2% (for UDA on CIFAR-100-400 labels). Interostingly, while the bascline FixMatch outperforms UDA across all the benchmarks, improving calibration UDA (i.e. BaM-UDA) allows it to out perform both FixMatch and BaM-FixMatch. A key difference between FixMatch and UDA is the use of hard pseudo-labels in FixMatch (i.c. -- 0 in Pr defined in |Section 3/ versus soft pscudo-labels in UDA (with t 0.4); this suggests that Bayesian classifier is more effective in conjunction with soft pseudo-labels. Leveraging this insight, set t = 0.9 in BaM-UDA (also see Section 7.2 for ablations on f), resulting in consistently better performances over the two SSL baselines across all CIFAR benchmarks.
Overall, we found that the improvements in both calibration and test accuracy are more significant for label-scarce settings, as expected, since the problem of confirmation bias is more acute there and BaM- can provide greater benefits by mitigating this. We explore even more extreme settings in Appendix I when there are only 1 or labels per class in CIFAR-100. In the setting with only label per class, BaM-UDA gives decent 35.2% accuracy, In contrast UDA gives a poor 8.2% accuracy in such challenging setting, Furthermore, the additional computation overhcad incurred from BaM- is minimal, adding approximately only 2-5% in wall-clock time for the CIFAR-100 settings (see Appendix H.. While previous works (Sohn et al.) 2020) also include extreme low label settings such as CIFAR-1O-40 labels, we found this benchmark to be highly sensitive the random initialization and different splits of the data, giving up to 2% variance with the exact same SSL method and thus we exclude them in our study, Such variance in CIFAR-10 is significant, given that all hodh are giving accuracies 95-96% and we do not realistically expect gains of 2% more.
7

---

Published in Transactions on Machine Learning Research (08/2023)
Table 3: Long-tailed CIFAR-I0 & CIFAR-100 showing "Test accuracy (%) / ECE". We use 10% labels from each class. For better interpretation, we also show the supervised (100% labels) accuracy reported in Cao et al. 2019 which use a different architecture, ResNet-32, and an algorithm targeted for long-tailed problems.
CIFAR-IOLIT
CIFAR-100-11
10
100
10
100
FixMatch BaM-UDA (ours)
91.3 / 0.073
70.0 / 0.26
48.8 / 0.38
28.6 / 0.55
91.6 (10.3) / 0.067
71.2 (11.2) / 0.24
53.6 (14.8) / 0..2
31.9 (13.3) / 0.50
Supervised (reported)
88.2
77.0
58.7
42.0
6.2 BaM- improves performances by reducing confirmation bias
To further understand how BaM- mitigates confirmation bias, we track the test accuracy, ECF,, model confidence and ground-truth accuracy of accepted pseudo-labels (i.e. the "selection purity") over the course of training for the baselines and BaM-, as shown in Fig. 2 While the baselines learn effectively for the initial stages of training, learning is eventually hindered. Due to the entropy minization objective, the model is encouraged to output increasingly confident predictions (as evident from the growing model confidence in Fig. 2).. Thus, in the absence of explicit calibration, the baselines quickly become over-confidedd, resulting in confirmation bias where the model reinforces its mistakes. Confirmation bias in the baselines is particularly evident from the selection purity (i.e. the ground truth accuracy of accepted pseudo-labels) in Fig. 2 after short amount of training, the selection purity starts to drop suggesting that the model begins to accept pseudo-labels that it makes mistakes on. In contrast, BaM- successfully miligates confirmation bias as evident from the constantly improving selection purity, thus promoting learning for longer periods to result in better final performances. Further ablation studics are discussed in Section 7.2 and Appendix G
6.3 BaM- is more effective in class-imbalancee settings
Long-tailed image datasets. Datasets in the real world are often long-tailed or class-imbalanced, where some classes are more commonly observed while others are rare. We curate long-tailed versions from the CIFAR datasets following Cao et al. (2019), where a indicates the imbalance ratio (i.e. ratio between the sample sizcs of the most frequent and loast frequent classes). We randomly select 10% of the samples each class to form the labeled set; see further details in [Appendix Ej We use the best performing baseline baseline and compare against BaM-UDA. Results shown
method from Table 2 Le. FixMatch (FM), in TTable 3, where BaM-UDA achicves consistent improvements over FM in both calibration and accuracy across all benchmarks. Notably, gains from BaM- are more significant than those in the class-balanced balanced settings (for e.g. BaM-UDA improves upon FM by a smaller margin of 1.5% on the CIF4R-100-4000 labels benchmark which is also approximately 10%). Further analysis are provided in Appendix E.2 where test samples were separated into three groups depending on the number of samples per class and test accuracies are plotted for each group.
Table 4: Photonic crystals (PhC) band gap prediction showing "Test accuracy (%) / ECE". The fully-supervised (100% labels) accuracy is 88.5%.
PhC-10%
PhC-1%
FixMatch
78.8 / 0.098
55.0 / 0.385
BaM-UDA 81.0 (12.2) / 0.052 56.9 (11.9) / 0.356
Photonics science. A practical example of real-world domain where long-tailed datascts are prevalent is that of science samples with the desired properties are often much rarer than trivial samples. Further, SSL is highly important in scientific domains since labeled data is particularly scarce (owing to the high resource cost needed for data collection). To demonstrate the effectiveness of BaM-, we adopt a problem photonics (Loh al. 2022),, where the task a 5-way classification of photonic crystals (PhCs) based on their band gap sizes. A brief summary and visualization of this dataset are available in Appendix F We explored
8

---

Published in Transactions on Machine Learning Research (08/2023)
approach similar to FixMatch for the baseline and similar to BaM-UDA for ours (some modifications were needed in the augmentation strategies to respect the correct physics of this problem; see Appendix F). Results are shown in Table 4 where we demonstrate the consistency of BaM-UDA's effectiveness in improving calibration and accuracies in this real-world problem.
7 Ablation studies on BaM
7.1 Ablating the key components of BaM
BaM- consists of two main features; 1) several weight samples are taken from the BNN layer and averaged to derive the predictions, and 2) the selection criteria is modified to be based upon the variance of the samples. To further investigate the effect of cach feature, we perform ablation studies on BaM-UDA to isolate the contribution of averaging predictions from the contribution of replacing the selection metric. Results are shown in Table 5) rows indicated with "BNN no T² show experiments using a BNN layer in BaM-UDA only for averoging predictions while maintaining the original selection metric, i.e. pseudo-labels are accepted if the 0.95. Comparing the first two rows, indood we
maximum prediction class probability greater than see that by not using the uncertainly estimates from the BNN, already get some improvements (minor improvements in cases where labels are not scarce). The difference between the last two rows show the effect of replacing the selection metric and indeed we observe larger and consistent gains across the benchmarks from doing so.
Table 5: Uncertainty estimate by BNN. Ablating the importance of uncertainty estimate provided by the variance of BNN predictions. "BNN no 22n indicates that the BNN layer is only used for bayesian model averaging, i.c. predictions are replaced by the posterior predictive but selection metric still follows the baseline, i.e. pseudolabels are accepted if maximum logit value after the softmax >> 0.95. Cyan indicates the default configuration.
CIFAR-100
400
4000
UDA (t=0.4) BaM-UDA, BNN no o² (1=0.4
44.0
0.491
72.9 / 0.185
48.3
0.418
73.0 / 0.184
BaM-UDA, BNN no d² 7== BaM-UDA (t=0.9)
54.2
0.368
74.5
/ 0.170
59.7
0.327
75.3
/ 0.167
7.2 Importance of sharpening temperature
From our results [Section 6 we found that BaM was more effective in conjunction with soft pseudo-labels. In Table we show ablation experiments on the temperature of the sharpening operation on the CIFAR-100- 400 labels benchmark. Overall, observe trend that softer pseudo-labels (i.e. reducing the sharpening of pseudo-labels) led to better calibration and improved test performance. As such, in our experiments we modify upon the original sharpening parameter of UDA and set f = 0.9 for BaM-UDA in all our benchmarks.
Table 6: Ablation of sharpening temperature in BaM-UDA. Dataset is CIFAR-100 with 400 labels. Highlightod in cyan is the main configuration used.
t
Test Accuracy
ECE
0.4
57.9
0.344
0.8
58.1
0.340
0.9
59.7
0.327
1.0
59.2
0.334
9

---

Published in Transactions on Machine Learning Research (08/2023)
8 Conclusion and Broader Impact
Since confirmation bias is a fundamental problem in SSL, in this work we showed that it is imperative for the model to have proper uncertainty estimates, or be well-calibrated, to mitigate this problem. In particular, we empirically demonstrated that approximate Bayesian techniques such as a last Bayesian layer or weight averaging approaches can be used to improve a model's uncertainty estimates which can result in better model performance across a variety of SSL methods. We further underscore their importance in more challenging real-world datasets. We hope that our findings can motivate future research directions to incorporate techniques targeted for optimizing calibration during the development of new SSL methods. Furthermore, while the primary goal of improving calibration is to mitigate confirmation bias during pseudo-labeling, an auxiliary benefit brought about by our approach is a better calibrated network, i.e. one that can better quantify its uncertainty, which is highly important for real-world applications. A potential limitation in our work lies in the use of ECE as a metric to measure calibration which, while commonly used across literature, are not free from flaws (Nixon et al.) 2019).. However, in our work, we empirically demonstrate that despite their flaws, the ECE metric still provides good correlations to measuring confirmation bias and test accuracy. We provide further discussion of the socictal impact and ethical considerations of our work in Appendix J
Acknowledgments
We thank Hao Wang, Pulkit Agrawal, Tsui- Wei Weng, Rogerio Feris, Samuel Kim and Serena Khoo for fruitful conversations and foedback.
This work was sponsored in part by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was also sponsored in part by the the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http:/ http://iaifi.oo /iaafffi..go/)) and in part by the Air Force Office of Scientific Research under the award number
C.L. also acknowledges financial support from the DSO National Laboratories, Singapore. L.D. is supported in part by the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750-19-C-1001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA.
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016. URT /tt///aawiw..rr///////////6 0666
Eric Arazo, Diego Ortego, Paul Albert, Noel F. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning, 2019. URL
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by non-parametrically prodicting view assignments with support samples. CoRR, abs/2104.13963, 2021. URL httpp::://aaxii.or//abbb///
13963
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Z. Ghahramani, Aduances in Newral Information Processing
M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances Systems, volume 27. Curran Associates, Inc., 2014. URL http:::/ppooeeeiigg...oe. wwwpeeeiiiiinnnnnnirpeeee wttpp/ppp...pppppp///////// .pff
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygekimer,
10

---

Published in Transactions on Machine Learning Research (08/2023)
F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
kcc1ebb0000004666/eeebbbbaaaaae. pdf
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural
networks, 2015. URL https: http:///////a/55555 httep://aarrii.oorrraab//115... 05424
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalilies: A Nonasymplolic
Theory of Independence. Oxford University Press, 02 2013. ISBN 9780199535255. doi: 10. 10.1093/aco /acprof:
0so/97801995338 (978801993335555000000000
URL https: org/ / 10. 10pr//cpro/ooooooooo http://ooooooo////00099900935330
00100001100000000 0001
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1
3, 1950. doi: 100.77//...000un////00000000.010
ametsoo ametsoc 2ht___00011oooto___111111 vofeit 2_0__c__0 2___co___x___ xm
URL
org/ http://ww//ooaaaaaaaaaaaaa/////0 1950 001995000880000 078_0001111100000000000000000000
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with labol-distribution-wwaee margin loss, 2019. URL
wttp:/wwww.rg//b//09/0////// 07413
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand
Joulin. Emerging properties in self-supervised vision transformers. CoRR, abs/2104.14294, 2021. URL
14294
Paola Bascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Self-pacod
pseudo-labeling for semi-supervised learning. CoRR, abs/2001.06001,, 2020. URL https://arxiv.org
abs/2001.06001
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020a. URL htttp:://axxiv.or//ab//////
Ting Chen, Simon Korublith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners, 2020b. URL htttps://rxii... wwtp:/iw...rgaa////////0// 10029
Thomas Christensen, Charlotte Loh, Stjepan Picek, Domagoj Jakobović, Li Jing, Sophie Fisher, Vladimir Ceperic, John D. Joannopoulos, and Marin Soljačíé. Predictive and generative machine learning models for photonic crystals. Nanophotonics, ((13):4183-4192 October 2020. ISSN 2192-8614. doi: 10.1515/
202001-2000-0197. URL https://www httt::://wwwdogrrutterrr com cot///bcmmmt/////////0..0 http://0...55.//aaao---00022---0 html
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data
augmentation with reduced search space, 2019. URL betp:://arriv.orr/aab////// 13719
Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the
Royal Statistical Sociely, Series D (The Statistician), 32(1/2):12 22, 1983. ISSN 00390526, 14679884. URL
http:://ww.jjstt org/stable/2987588
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2015. URL
02142
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson, Loss surfaces, mode connectivity, and fast ensembling of dnns, 2018. URLtstps::/aaxitt
10026
Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin. Bayesion Dala Analysis, Third Edition. CRC press, 2021.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-bayesian theory meets bayesian inference. In D. Lee, M. Sugiyama, U. Luxburg. Guyon, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:
11

---

Published in Transactions on Machine Learning Research (08/2023)
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou (eds.), Aduances in Newral Information Processing Sys- tems, volume 17. MIT Press, 2004. URL https://proceediing www.oeiiinnnn..uis..... 9822500066611af02777222228888-83
Alex Graves. Practical variational inference for neural networks. In Shawe-Taalorr R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- ume 24. Curran Associates, Inc., 2011. URL
ppff
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-21284, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017. URL https://arxix: org/ abs/ 1706. 04599
Zhe Guo, Zhen-Xo Yu Zhang, Yuan Jiang, Yu- Li, and Zhi-Hua Zhou. Safe deep cani-supervieed lcarning for unseen-class unlabeled data. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 97th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-3906 PMLRR, 13 18 Jul 2020. URL htttps:/pproceedings .ml.. Seate.pressvvvve https///19/////11.1 httpgooo20.....
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729 9738, 2020.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning, 2011.
Pavel Tzmailov, Dmitrii Podopríkhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization, 2018. URL https://arxiv.orrgg
J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade. Photonic Crystals: Molding the Flow of Light. Princeton University Press, 2 edition, 2008. URL
Steven G. Johnson and J. D. Joannopoulos. Block-iterative frequency-domaiin methods for Maxwell's equations a planewave basis. Optics Express, 8(3)1133 190, January 2001. ISSN 1094-4087. doi: 00.134/0EE..000177 URL htttt:://ww/w www.:o/www..iii.oooooooooo/////e http:///aabstattt.... -fm-
Diederík P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick, 2015. URL https: http://wwww...ra/////aa////////e abs/1566 02557
org/
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, 2016. URL https : ///axtv..or//ab///500.0222 02242
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell Simple and scalable predictive uncertainty estimation using deep ensembles, 2016. URL http:://arriv..orraaa/////1/ 01474
Dong- hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks, 2013.
Charlotte Loh, Thomas Christenson, Rumen Dangovski, Samuel Kim, and Marin Soljacic. Surrogate- and invariance-boost contrastive learning for data-scarce applications in science. Nat Commun, 13, 2022. http://tgeee/eeerrr////////////0
12

---

Published in Transactions on Machine Learning Research (08/2023)
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. CoRR, abs/2106.079998, 2021. URL
07998
Kevin Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml ai
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media,
2012.
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning, 2019. URL
wttgg//b//99..01.../1 01685
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Zhongzheng Ren, Raymond A. Yeh, and Alexander G. Schwing. Not all unlabeled data are equal: Learning weight data in semi-supervised learning, 2020. URL https://arxiv.org/ wwwggabs/2007.00 012933
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pscudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021. URL https://openreviier http:p///ppppooiiiieeeeeeorr/rra
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In 2005 Seventh IEEE Workshops Applications of Computer Vision (VACV/MOTION'')) Volume 1, volume 1, pp. 29-36, 2005. doi: 10.1109/ 0111009/ACVOT.20000
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep cum-ssupervise learning. CoRR, abs/ 111166606648888 2016. URL htttp//aaxiv.orr/
Abb/1606.04580
Kihyuk Sohn, David Berthelot, Chun-Liang Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- seni-supervised learning with consistency and confidence, 2020. URL httpps://arti wwtrriiv.orrg/oooo oww.oogg///oeooooooooooooooosser abbs/2001.000
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:
//proceedings.not httpa///wcc///ooooo..//ccaaaaaae
Antti Tarvainen and Harri Valpola. Weight-averaged consistency targets targottssssttotttooo improve somi-supervised deep
learning results. CoRR, abs/ abs/1703.01780, 2017. CRL http:///arriv.orr/a////111.....3
Sunil Thulasidasan, Gopinath Chenmupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks, 2019. URL
attps://arxiv.orgg wrgg/b/////g....o./ 11001
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp. 681 688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306199
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. 2020. doi: 10.48550/ ARXIV 2002.06675. URL htttps///rxiv..or http://..wrrraa/////////// hbsp//aas//00...ee////// 06715
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiętkowski, Linh Tran, Stephan Mandt, Jasper Snock, Tim Salimans, Rodolphe Jenatton, and Scbastian Nowozin. How good is the bayes postcrior in deep neural networks really?, 2020.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. CoRR, 188/2002.08791,, 2020. URL
13

---

Published in Transactions on Machine Learning Research (08/2023)
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmenta- tion. CoRR, 0ass/194..128.0 2019a. URI http:::/aaaxiv.oog//bb//////////
12848
Qizhe Xie, Minh-Thame Luong, Eduard Hovy, and Quoc V. Le. Self with noisy student improves imagenet classification, 2019b. URL httpe///axxt..or///////111.....1
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding, 2021. URL https://arxiv.. wwtg/a/////aaaaeee 00650
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33md Annual Meeting of the Association for Computational Linguistics, pp. 189 196, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics. doi: 10.3115/96 //1///886688868888 URL https://
aclanthology 1rannttoooggyooo/PP5---0000 org/P95-1026
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, pp. 694 699, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X. doi: 10.1145/775047.7755555 775151.. URL https: :tops://ddoai /tt.. org/10 11451777771 7771/77754777715511
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, 2021. URL
https://arxiv wwtpr/www.rggbb//110....om 2002//2211000033
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2017. URL https://arxiv.. orgg/bs/////a 2 09412
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balas balanced self-training. In Proceedings of the European conference on computer vision (RCCV), pp. 289 305, 2018.
14

---

Published in Transactions on Machine Learning Research (08/2023)
A Theoretical Connections to Generalization Bounds
In the training of BaM-, we adopt a variational approach and maximize the evidence lower-bound (ELBO),
ELBO B log KLL(((())ll(())
with notation consistent with those used in Section 4.1 of the main text. We motivate this approach by using Corollary 1 below, derived from the PAC-Bayes framework (for the complete proof, see Appendix A.1 below). The statement shows the generalization error bounds on the variational posterior in the SSL setting and suggests that this generalization error upper bounded by the negative ELBO. This motivates our approach, i.e. by maximizing the ELBO we improve generalization by minimizing the upper bound to the generalization error. The second term on the right side of the inequality characterises the SSL setting, and vanishes in the supervised setting.
Corollary 1 Let 'D be data distribution where i.i.d. training samples sampled, of which No labeled, uI, DN: and Nu are unlabeled, (x, DN=, where úi (yi) denotes the model-assigned pseudo-labed (true labels) for input 24. For the negative log likelihood loss function E, assuming that is sub- Gaussion (see Germain et al. (2016)) with variance factor s², then with probability at least. - 8, the generalization error (denoted as DD) of the variational posterior ((f()) is given by,
No.
log p((()1);;) 0) p(ui)2;;))
1
1
s2
E (5()) S [EEBBO] (0(0))
+ log + N
(3)
Nu
2
A.1 Proof of Corollary
PAC-Bayesian theory aims to provide bounds on the generalization risk under the assumption that samples are i..dd. While PAC-Baycsian bounds typically apply to bounded losses, Germain et al. (2016) extends them to unbounded losses (which necessary in our work since it uses the unbounded log-loss). Their theoretical result is reproduced below in Theorem 1 under assumptions for sub-Gaussian losses. These bounds commonly assume the supervised setting; in this proof, our goal is to extend them to the semi-supervised learning setting where in addition to labels from an empirical set, our loss is also mediated by pseudo-labele assigned by the model.
Let (X,Y) ~ DW denote iiddd training samples from the data distribution D, where (X,Y) { [( ((2,,8)) (()).)))))) 0.)) represents the set of N₁ labeled input-output pairs and N₂ pairs of unlabeled input and their model-assigned pscudo-labels g. Here, Nr N. N. Let the loss function be e(f, x. y) RR the generalization risk on distribution D, iee. (D),,, and the empirical risk on the training set, iee x is given by:
N cx = ((f.,,y)) + Nu {f.
1 N
cL(f)
E {ffz,y); (a.n)~
N Σ
On assumptions that the loss is sub- Gaussian (see Boucheron et al. 2013) section 2.3), i.c. a loss function is sub- sub-Gaussian with variance s2 under a prior æ and D if it can be described by a sub- Gab--Gaussian random variable v = ( 2(f,x, y), the generalization error bounds are given by Theorem |Germain et al. 2016 below.
Theorem (Corollory 4 from Germain al. (206))) Let π be the prior distribution and e be the posterior. If the loss is sub- Ganssias with variance Jactor s² with probabilily at least. 1-6 over the choice of (X, (X,Y) D.
E(C(()) E 8ー 8ーB
E
(x)(( + log(1/))) log $²
Beginning from Theorem 1 log likelihood loss and prior p(0) in our SSL setting can be found by replacing á with q(0))) and & with
the generalization error of the variational posterior ((1))) under the negative
15

---

Published in Transactions on Machine Learning Research (08/2023)
(f vi) == - -log (ogp(ys/24:3 8);
< E L xxxx (5) + 1 N (L(((()))lb(() + log(1/0)) +
1
B. CD())
f~g
fmg
2
N N
1
Eg
logp((((=;;0) +
log pog
N
N (KI(q(0())l((0)) + log (1))) + 1
2
log p(ūx(as;0)
1 N
E2
log p(p: x1
log bogg(uss p(ys(2;;00)
p(ya(1r;
M
1
+ log(1100 (/0)) +
N
2
p00: 0)
1 (KI((((())l)(0)) + log(1/8)) 1 $²
Eg
log
log ((vi 0)
+ N
N
plya 1zy;
N
NO.
p( Ix 0)
1 5 + (KTT((())))p(()) log(1/6))
1
E,g
log
+ Eng
log ( (1
N
p(ua [.: 0)
N
i=1
Nu
p(0: (x:
E (b) S Egg f~
log
+ N
|- ELBO + N log(1/0) + 8
No
p(y (x;
2
1=1
where we define ELBO R))))) - (1((0)))) P(O).. In the later sections, Appendix C.1 maximizing in our loss objective. By maximizing the ELBO, L.e.
will see that this is the term minimizing the negative ELBO, we are minimizing the upper bound to the generalization error of our variational posterior. The first term in the last line adds a divergence measure between the pseudo-label prediction distribution and the ground truth distribution - in the fully supervised setting Ú: + y/ and this term vanishes.
B Exploring SSL methods without selection metric
In the main text, we have looked at two popular SSL methods that are based on a selection metric. More recently, there is also a family of SSL methods based upon representation lcarning (Assran et al. 2021 Chen let al.) 2020b; Caron et al.] 2021), where the model is first trained to produce useful representations and subsequently tunetuned on limited labeled data. This give rise to some interesting questions: Is confirmation bias still an issue in SSL methods without selection metric? And if so, can BaM- or other approximate Bayesian techniques be used to mitigate confirmation bias? We explore these questions in this section, using canonical example for the family of SSL methods based on representation learning approach. Details about the training loss used in PAWS are explained in Appendix B.1 The main difference PAWS has in contrast to the methods studied above is in its prediction generation technique instead of a classification head, PAWS uses a non-parametrie nearest neighbour classification met over its representations.
PAWS (Assran al. 2021p
B.1 SSL methods based on representation learning
PAWS. More recently, there has been a newer family of SSL methods based upon visual representation learning Chen et al. (2020a b); Assran et al. (2021). We use PAWS Assran et al. (2021) as a canonical example of a SSL method from this family. A key difference of PAWS as compared to other selection-mette based SSL methods is the lack of parametric classfier layer. Instead, predictions are derived from a non-parametric hoft-nearest-neighe
classifier based on representations. Let 21 = fx)) and 22 = =f(2((z)) be the representations for the two views from the backbone encoder, their pseudo-labels (q,, 92) and the
16

---

Published in Transactions on Machine Learning Research (08/2023)
unlabeled loss are given by:
d(2 2x) yy
RB Σ
4 - ma(ii (z,)) - Σ ==1 E ((22,22 25) Lu = B u=1
H( + () (4)
where {z,) is the set of representations of the labeled examples, d(a.b) exp(a b/ b/[[||||||]]T] is similarity metric with temperature hyperparameter Tp and all other symbols have the same meanings defined before in Section 3 of the main text. The combined training loss for PAWS is Iu + Tmme-mxx where the latter is a regularization term Lmo-max = (ÿ) that seeks to maximize the entropy of the average of predictions g = (1/(2pB)) Eie (p((guu) p((2).))))
B.2 Exploring other approximate Bayesian techniques
Since BaM- replaces the classification head and PAWS does not have one, we incorporate BaM- into PAWS by using& Bayesian last layer in the encoder and "Bayesian marginalizing" over the representations (also see Appendix B.4.1 for detailed formulation). Nonethhless, apart from Bayesian marginalization, BaM- is originally desirable towards improving uncertainty estimates in the selection metric. Since PAWS does not use a selection motric, instead of Bayesian marginalizing or aggregating over just one layer, one could seok to aggregate over the entire network. This can be most commonly acheived by approximale bayesian techniques such as model ensembling approaches Lakshminarayanan et al. (2016), which has been highly successful improving uncertainty estimation. However, training multiple networks would add immense computational overhead. To perform this aggregation computational efficiently, we instead explored weight averaging approaches, such as Stochastic Weight. Averaging (Tzmailov et al. 2018) (SWA) and Exponential Moving Averaging (EMA) (Tarvainen & Valpola 2017 He et al.) 2020; Grill et al. 2020) and studied their role in mitigating confirmation bias during pseudo-labeling. Weight averaging however differs from traditional bayesian model averaging approaches typically used in ensembles since we are averaging in the weight-space (i.c. averaging model weights) instcad of function-space (i.c. averaging predictions). The approximation of SWA to Fast Geometric Ensembles |Garipov al. 2018)) has been shown by previous works (lzmailov et al.) 2018); however, the best of our knowledge the connection of EMA to ensembling has not been formally shown.
Weight averaging during pseudo-labeling We maintain separate set of non-trainable weights 0g containing the aggregated weight average of 0n which is used to produce pseudo-labels throughout training. In SWA, we update them via 0g + (na0, + 0 + 1) at every iteration, where n. represents the total number of models in the aggregate and , are the trainable parameters of our backbone encoder as before. In practice, we switch on SWA only after some amount of training. In EMA, this update is 0g < -0 + (1-7)00, where y is a momentum hyperparameter controlling how much memory 0,, should retain at each iteration (see Appendix C.2 for pscudocode). While EMA has been previously explored in the context of SSL (Tarvainen) & Valpola| (017)) the best of our knowledge SWA has not been explored in SSL and more importantly, their link to mitigating confirmation bias in pscudo-labeling has not been explicitly shown.
Table 7: "Test accuracy (%) / ECE" for PAWS Assran et al. 2021 and our methods. 'BaM-" refers to Bayesian Model averaging via BNN final layer, while +SWA and +EMA are using weight averaging techniques. For each benchmark, results averaged over 3 random dataset splits.
CIFAR-10
CIFAR- 100
2500 labels
4000 labels
10000 labels
PAWS (repro)
95.3...1
0.046.0.001
71.5a0.3 / 0.23210.005
75.6 / 0.198
B-W-PAWS
95.420.1
(f0.1) 0.046220001
72.520.3
(f1.0) / 0. 0.228=0.003
76.520.2 (f0.9) / 0.195
PAWS-SWS
95.6.0.1
(10.3)
0.046=20.01
74.5
(13.0) / 0.193
78.440.4
(12.8) 0.164
(10.5) 95.820.0
0.042 000
73.5=022 (*2.0) 0.1
77.240.1
(11.6) 0.168220.00
17

---

Published in Transactions on Machine Learning Research (08/2023)
CIFAR10-2500: Test Acc.
CIFAR10-2500 ECE
PAWS
95
0.11
(T 200)
0.10
PAWS+EMA
93-
0.09
92
0.08
0.07
89
PAWS
0.06
PAWS++SW
200)
0.05
PAWS +MA
87
0.04
200
400
600
100
200
300
400
500
600
Epochs
Epachs
Figure 3: Weight averaging techniques on PAWS for CIFAR-10 with 2500 labels displays highly consistent. relationships between increasing test accuracy and decreasing ECE.
B.3 BaM- is useful, but weight averaging is more effective when a selection metric is absent
Results are shown in Table 7 while BaM- gave some improvements over the baselines across all benchmarks, the improvements from weight averaging approaches are consistently larger. This is unsurprising, since BaM- relics on stochasticity only in the last layer while weight averaging aggregates over the entire model. Since PAWS does not use a selection metric and simply "accepts all unlabeled samples", the "selection purily" is absent and thus we investigate the issue of confirmation bias by tracking the accuracy and model calibration on held-out set as shown in Fig. Incorporating weight averaging results in better accuracies on the held-out set compared to the baseline at every stage of training. This suggests that even in the absence of selection metric, proper uncertainty estimates (or improved model calibration) can lead to improved SST performance. The effectiveness of improved uncertainty estimates is particularly evident through PAWS+SWA when SWA was switched on (at Tswa+ = 200 epochs), model calibration was immediately improved (i.e. the ECE quickly dips) and the accuracy also correspondingly spikes. Furthermore, we also observe a significant improvement in convergence rates resulting from the weight averaging techniques, i.c. better accuracics can be attained in just a third of the number of training iterations of the baseline. Ablations Towa and momentum schedules are presented in? ??.
Effective also in large-scale datasets like ImageNet. A strength of representation learning SSL methods is in large-scale datasets like ImageNet, where PAWS outperforms selection metric based SSL methods (i.e. 73.9% vs 71.5% on the best performing selection metric based method, FixMatch for the ImageNet-10% benchmark). In this work, we use the 10% labels setting of Assran et al. (2021) with slight modifications to the default setting in order to fit on our system (see Ap pendix D.2 for implementation details) and explored weight averaging techniques SWA and EMA, given their effectiveness over BaM- in PAWS. Results are shown in Table 8] where our methods provide consistent improvements in calibration and test accuracy over the baseline.
Table 8: ImageNet-10% showing "Top-1 test accuracy (%) / ECE" for PAWS and our methods, trained for 200 epochs and fine-tuned with lin- ear head.
IN-10%
PAWS +SWA (ours)
74.4 / 0.186
74.5 0.186
EMA (ours)
75.1 / 0.183
B.4 Further experimental details
B.4.1
Improving calibration in PAWS with BaM- (i.e. BaM-PAWS)
Because of the lack a classifier layer, incorporating BaM- requires some modifications. Instead, we use BaM- the last layer in the encoder (which is a linear layer) and pseudo-labels are obtained by 'Bayesian marginalizing' over the representations. More specifically, let be the Bayesian version of this layer, parameterized by 02 and let e be the input to this layer, pseudo-labels of BaM-PAWS (i.e. Eq. (4) are
18

---

Published in Transactions on Machine Learning Research (08/2023)
defined as:
q: = =a (z,)) = Σ
B
d( 2s) Us
M 2 = (1/M)
Σ m
h(c, )
(5)
E=== d(2, 2x)
s=1
where M the number of samples laken from the Bayesian layer.
B.4.2 Further training details for BaM-PAWS, PAWS+EMA, PAWS+SWA
BaM-PAWS.
Unlike BaM- in selection-metrie based SSL methods, we do not use separate optimizer and simply impose one one-minusccosine scheduler (i.e. scheduler (2) from the next paragraph) for the coefficient to the KI-divergence loss that goes from O to 1 in T epochs where we simply picked the best from T E {50, 100}.
PAWS+SWA and PAWS+EMA. We delay the onset of SWA to after some amount of training has clapsed (i.c. Towa epochs) since it is undesirable to include the randomly initialized weights to the aggregate. We set Tawe 200 and Towa = 100 for CIFAR-10 and CIFAR-100 respectively. After SWA was switched on, the weight aggregate was simply updated after every batch iteration. In contrast, EMA has a natural curriculum to "forget" older model parameters since more recent parameters are given more weight in the aggregate. We experimented with two schedules for y: 1) using linear warm-up from 0.996 in 50 epochs and then maintaining 7 at 0.996 for the rest of training and 2) using a one-minus-cosine scheduler starting from 0.05 and decreasing to 0 resulting in a 0.95 to range for 7, We visualize these schedulers and include ablation studies on them and on Tswn in ??.
C Formulation details and pseudocode of our methods
C.1 Bayesian model averaging via a BNN final layer
Following the notations Scction 4.1, we denote the BNN layer be h and an input embodding to this layer to be o in this section. We assume prior distribution on weights P(Ox) and seek to calculate the posterior distribution of weights given the impirical/training data, P(0, (x)), where Dx (X, which can then be used to compute the posterior predictive during inference. As exact Bayesian inference is intractable for neural networks, we adopt variational approach following (Blundell et al.) 2015) to approximate the posterior with a Gaussian distribution parameterized by Φ, ()) From now, we will drop the h index in On for brevity. To learn the variational approximation, we seek to minimize the Kullback- Leibler (KL) divergence between the Gaussian variational approximation and the posterior:
6º = argmin KL (9(0 Φ) p(0]X,
= argmin /
9(0) ())
((0l))log
d8
p00 X, Y)
( (Y())
argmin 1 log
X ())(
argmingE, [l
( ()) logp(Y
- logp(0)] + logp (X:0) (x:
argmin (L(g(0 أ) (l)) - (2 logr x + lYg
argmin ([FELBO) + logg(Y(X))
lp(0) is the evidence lower-bound which con- sists of a log- lieeliiiooood (data-dependent) term and KL (preor-de))ndent) term. Since KL (g(0l)) @) (()))x) is intractable, we maximize the ELBO which is equivalent to minimizing the former up to the constant, Xx)..
where in the last line, ELBO By(ess [logy( (Y
Each variational posterior parameter of the Gaussian distribution, o, consists of the mean (n) and the standard deviation (which is parametrized as a = log(1 + exp(p)) so that a is always positive Blundell et al. 2015)), i.c. o= o= (). p). To obtain a sample of the weights 0, we use the reparametrization trick (Kingma et al, 2015) and sample (0, T) to get μ + log(1 exp(p)) where denotes elementwise multiplication.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm PyTorch-style pseudocode for Bayesian model averaging in UDA or FixMatch.
I: backbone encoder network
b: bayesian classifier
loss: tern in evidence loxer-bound
B: croas -etrroyy loas
Q: quantile paraneter
nun san_sanples: number reight sanples
nethod: "UDA" or "FM" sharpen operation
g_list -
for (xl, labels), in zsp(iabeled_ loader, unlabeled loader))
x_lab, x_uX, x_ (2)), veak_augnent augong__ z_lab, 2,49, 2,u5 =(x__aab), (00),(xx000 f(x_uv), f(x_us) # get representations
nean_uR, atd uv aayee_predict z_uv)
g.list. (eonoddquoiil 1.list.appen mask Red.uu.le(q_listtmmann
len(q.liat) list)
keep 50
quantiles
Q))
compute losses
less + KL. 11_loss(0 (h) prior-dependent (data-independen loss losa_lab =H(h(__lab,, labels)
if method 'UDA';
l0s2 uosa_unlab H(h(2_ua), abp(mean_uv) mask # sharpened soft paeudo -labell elif nethod 'PN';
losa_unlab =Hhh(__us, mean us
mean_uv. argnax (-1))
pseudo -aabell labels
loss lobs_lab iosa_unlab 1oas_k1
loss backvard() optinizer.
def bayes_predict(b, 2):
range (nun, samples)] sanple weights return outputa... ()) autputa.atd() mean and atd of predictions
outputs stack softmax
In other words, we double the number of learnable parameters in the layer compared a non-Bayesian approach; however, this does not add huge computational cost since only the last layer is Bayesian and the dense backbone remains non-Baycsian.
Algorithm 2 shows the PyTorch-styy pseudo-code for BaM- in selection-metric based methods (here showing asymmetric augmentation applicable for UDA or FixMatch).. The main modifications upon the baseline methods include 1) computation of an additional KL loss term (between two Gaussians, i.e. the variational approximation and the prior), 2) taking multiple samples of weights to derive predictions and 3) replacing the acceptance criteria from using maximum probability to using standard deviation of predictions.
C.2 Algorithm for SWA & EMA in PAWS
Algoritmm 3 shows the pseudocode for the implementation of PAWS+SWA and PAWS+EMA in PyTorch. For brevily, leave out details of the multicrop strategy, mean entropy maximization regularization and soft nearest neighbour classifier formulation which are all replicated from the original implementation. We defer readers to the original paper (Assran et al. 2027 h for these details.
20

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm PyTorch-style pseudocode for PAWS-SWA and PAWS-EMA.
I: backbone encoder netvork
g: veight aggregated encoder network
ahp: aharpen operation
snn: PANS soft mearest neighbour classifier
B:
entropy loss
nax: nean entropy naxinization regularization loss
N: total nuaber of spochs
boolean to
SWA
use_ena: boolean to soa sva_apochar apochs: number of epachs before avitching on SWA ganma: nomentun paraneter for EMA
8 Capanns pararns initialized as copy s.parans.reqirre__gee
False # remove gradient computations
nun_ava
for in ran:
far in loader:
augnent(x), augnent augnentations
pl, p2 enn() (fn(f
gl. q2 snn snn(g(x)), snn(g(x)))
loss H(pi, shp (2 H(p2, shp ME (g_t 1oas ()ckvard Perinizer.stop
if _
if : # update aggregate sum Eva
parana (g-parana nun_ava p.paass)) / (nun_ava + 1)
else:
parans parans weights are just copied
alif use
una_ana: ana:
parans ganna 8 parans (1-ganua) params update nomentun aggregate
else:
g_parans parans PAWS baseline
Further implementation details
D.1 Hyperparameters for various selection-metric based methods
All selection-metric based methods in this study uses an optimization loss function of the form of Eq. (1) from the main text, with differences in the hyperparameters X, т, Pt and a. We use the hyperparameters from the original implementations. For Pseudo-Labels (Lee) 2013), 1, r = 0.95 and Pt=o (i.e. hard labels); for UDA Xie et al. 2019a),, μ = 7, T = 0.8, Pt=0.4 (i.e. soft laeudo-labels sharpened with temperature of 0.4); for FixMatch (Sohn al., 2020). f 7, F 0.95, Pi=o (i.c. hard labels). All methods use UDA and FixMatch uses asymmetric transforms for the two legs of sample and pseudo-label prediction, i.e. 0n weak transform (based on the standard flip-and-shift augmentation) and 02 is strong transform (based on Ragnengment (Cubuk et al.] 2019)). Pseudo-Labels uses symmetric weak transforms for both legs.
In addition,
In our calibrated versions of all these methods (i.e. "BaM-X"), we maintained the exact same hyperparameter configuration as its corresponding baseline. The only exception is the sharpening temperature of BaM-UDA, which uses = 0.9 instead of 0.4 in UDA, as we found that calibration enables, and was highly effective with, the use of soft pseudo-labels..
D.2 Implementation details for ImageNet experiments
We maintain the default 64 GPU training configuration recommended by the authors and make slight modifications to the default implementations to fit on hardware. On set up (which does not use Nvidia's Apex package due to installations issues), we found training to be unstable had to use full-precision training. In order to fit into memory, we had to decrease the number of images per class from to 6, resulting in a slightly lower baseline performance from the reported for 200 epochs of training (see Table 4 in Assran et al. (2021) for the study on the correlation between the number of images per class and the final accuracy). On all ImageNet experiments on PAWS, we follow the validation and testing procedure of PAWS (Assran al. (2021) and swept over the same of hyperparameters during fine-tuning of the linear head. We report the F.F. at the final checkpoint.
half-precision and
21

---

Published in Transactions on Machine Learning Research (08/2023)
E Long-tailed CIFAR-10 and CIFAR-100
E.1 Dataset preparation
We create long-tailed versions of CIFAR-10 and CIFAR-100 following the procedure from Cao et al. (2019) L.e. by removing the number of training examples per class from the standard training set with 50,000 samples. We create the class- imbalance unlabeled dataset with an exponential decay where the severity the imbalance given by the imbalance ratio a = (M..)//mmmmmmmmmmmm unlabeled examples for class i. The number of samples in the most frequent class is 5,000 for CIFAR-10 and 500 for CIFAR-100. To create the labeled set, randomly select 10% of samples from each class, under the constrain that least I sample for each class is included in the labeled set, i.e. 7y, min(1, 0.1 Ra,i), where T4, is the number of labeled examples for class i. The total number of labeled and unlabeled examples for the different benchmarks
E {10, 100), where nu the number of
CIFAR-10-1 and CIFAR-100-TT are summarized in Table g The test set remains unchanged, i.e. we use the original balas--blanced) test set of the CIFAR datasets with 10,000 samples.
Table 9: Dataset statistics for CIFAR-IO-LT & CIFAR-100-LT showing total number of examples in the labeled and unlabeled datasets for each benchmark.
CIFAR-
CIFAR-100-LI
a 10
a 100
a a= 10
a 100
Labeled
2,041
1,236
1,911
1,051
Total
20,431
12,406
19,573
10,847
We used the exact same configuration and hyperparameters as the original (non-long-tailed) CIFAR bench- marks, Lee the architecture is WideRessNet-285 for CIFAR-1O-LT and WideResNet-28-- for CIFAR-100-L11 All training hyperparameters used to obtain the results for FM and BaM-UDA in Table of the main text also follow the ones from the original CIFAR benchmarks.
E.2 Additional classwise results
Fig. shows the test accuracies for the baseline (FM) and ours (BaM-UDA) after separating samples from the test set into three groups the "Many" group which contains classes with more than 100 samples, the "medium" group which contains classes between 10 and 100 samples and the "few" group which contains classes less than 10 samples, for the benchmark of CIFAR-100-LT with 10 and 10% labels. We see that BaM-UDA outperforms FM on every group: in addition, the gap between BaM-UDA and FM increases as the number of samples are more scarce, highlighting BaM-UDA's 's utility in improving accuracy over the baseline in the more difficult classes.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Test Accuracy (%)
100
BAM-UDA: many
BAM-UDA: medium
BAM-UDA:
FM: many
FM: medium
80
60
20
100
200
300
400
500
600
700
Iterations
Figure 4: CIFAR100-LT, Accuracies for samples based on their their classwise sample frequency, "many" for classes with >00 samples, "medium" for classes between 10-100 samples and "few" for classes between <10 samples. Dataset is CIFAR-100-LL with a 10 and 10% labels.
F Semi-supervised Learning in Photonics Science
Semi-supervised learning highly important and applicable to domains like Science where labeled data is particularly scarce, owing to the need for computationally expensive simulations and labor-intensive laboratory experiments for data collection. To demonstrate the effectiveness of our proposed techniques in the Science domain, we use an example problem in Photonics, adopting the datasets from Loh et al. (2022) The task we studied in this work is a 5-way classification of photonic crystals (PhC) based on their band gap sizes.
PhCs are periodicalll--struutuued materials engineered for wide ranging applications by manipulating light waves (Joannopoulos et al.] 2008) and an important property of these crystals is the size of their band gap (often, engineers seek to design photonic crystals that host substantial band gap (Christensen et al., 2020 ). We adopt the dataset of PhCs from Loh et al. (2022), which consists of 32,000 PhC samples and their corresponding band gaps which had been pre-computed through numerical simulations (Johnson & Joannopoulos 2001). Examples of PhCs and an illustration of band gap from the dataset is shown in Fig. 5) We binned all samples in the dataset into 5 classes based on their band gap sizes; since there was a preponderance (about 25,000) of samples without a band gap, we only selected 5,000 of them in order to limit the severity of class imbalance and form new dataset with just 11,375 samples (see Fig. 30). Notably the long-tailed distribution of this dataset is characteristic of many problems in science (and other real-world datasots), where samples with the desired propertics (larger band gap) are much rarer than trivial samples. From this reduced dataset, we created two PhC benchmarks with 10% labels (PhC-10% and 1% labels (PhC-1%), where 10% and 1% of samples from each class are randomly selected to form the labeled set respectively. The class-balas balanced test set is fixed with 300 samples per class (total of 1,500 samples), the unlabeled set consists of 9,876 samples and the labeled sets consists of 1,136 samples and 113 samples for PhC-10% and PhC-1% respectively.
For these benchmarks, we used a WideltesNet-28-2 architecture, with single channel for the first CNN layer, and made the following changes upon FixMatch and BaM-UDA. We set μ 1 and instead of 2 iterations, trained for 300 epochs (where each epoch is defined as iterating through the unlabeled set once). For BaM-UDA, we set Q = 0.9 and use a onc-minus-cosine warm-up scheduler of 50 epochs to the KL loss coefficient (resulting in a coefficient going from 0 to 1 in 50 epochs). For each benchmark, we swept the learning rate across (0.01, 0.001} and select the best model for both the baseline and ours. The standard image angmentations usod in vision problems cannot. be applied integrity of the data (e.g. cropping the PhC input would result in completely different band gap profile). Instead, used the augmentations proposed in Loh et al. 2022 (periodie translations, rotations and mirrors)
this problem, since destroys the scientific
23

---

Published in Transactions on Machine Learning Research (08/2023)
(a) Examples 2D Photonic crystals
(c) Dataset statistics showing the class boundaries
6000
class O: bg = 0 % (5000)
1: % bg < (3822)
5000
class 2: 5% bg 12% (1343)
class 3: 12% < bg 20% (701)
of samples
class 4: bg 20% (509)
(b) Illustration of band gap for sample
4000
Total samples = 11375
3000
Densty of states
band gap
2000
1000
Frequency (a.u.)
""izee band gap (measured in
Max Bandgap (%)
Figure 5: Photonics dataset. (a) Examples of 2D periodic photonic crystals (PhC), (b) illustration of the band gap size of single PhC seen from its density-of-states, a common spectrum of interest for PhCs (Loh) le al. 2022), and (c) dataset statistics for the task used in this work where the samples were binned into classes based on their band gap (bg) with class boundaries detailed the inset. Numbers in parenthesis show the number samples in each class, showing strong class imbalance.
and applied them symmetrically (i.e. there distinction of strong and weak augmentations). In addition, included these augmentations to the labeled samples as well.
G Ablation Studies
G.1 Further ablation studies BaM-
Number of weight samples. Bayesian model averaging produces better calibrated predictions through "Bayesian marginalization", i.e. by averaging over multiple predictions instead of using single prediction of the model. In Table 10] we show ablation studies on the number of weight samples taken from the variational posterior in the BNN layer used for "Bayesian marginalization", i.e. for computing the posterior predictive. We observe an overall trend that increasing the number of weight samples load to better calibration and final test. accuracies, providing direct evidence for the effectiveness using Bayesian model averaging towards improving calibration. In order limit the computational overhead arising from taking large number of weight samples, given that pscudo-labeling happens every iteration, limit M to 50 in our study. The computational overhead for M 50 is discussed in Appendix H where see that incorporating Bayesian model averaging incurs negligible computational overhead.
Table 10: Ablation of number of weight samples, M, taken from the variational posterior in BaM-UDA. Dalasel is CIFAR-100 with 400 labels. Highlighted in cyan is the main configuration used.
M
Test Accuracy
ECE
2
50.0
0.403
5
58.1
0.342
10
58.5
0.336
50
59.7
0.327
A basic semi-supervised learning setup. Many SOTA semi-supervised learning methods incorporate several techniques like weak and strong augmentation, sharpening of pseudo-labels and selection criteria to
24

---

Published in Transactions on Machine Learning Research (08/2023)
achieve the best performance. An interesting study is to investigate basic semi-supervised learning setup where many of these techniques are removed. In Table 5 of the main text, investigated the importance of the different features in BaM- and found that even without a selection metric, BaM still leads to some performance gains. Here, we study a simple set up of semi-supervised learning, where we remove sharpening and the selection metric and compare this basic baseline with one where the last. layer replaced with a BNN via BaM-; results are shown in Table 11 Results show that performance gains via BaM- are consistent. and are effective independent from the techniques commonly introduced in SSL methods.
Table 11: Basic semi-supervised learning setup without selection criteria.
CIFAR-100-400
CIFAR-100-4000
Basic setup
53.5
74.0
Basic sctup + BaM
54.6
74.9
H Computational Requirements and Additional Computational Costs
All CIFAR-10 and CIFAR-100 experiments in this work were computed using a single Nvidia V100 GPU. ImageNet experiments were computed using 64 Nvidia V100 GPUs. A key aspect of proposed calibration methods is the requirement of adding minimal computational cost to the baseline approaches. In the following paragraphs we list the additional computational cost (based on wall-clock time on the same hardware) for our proposed methods.
Computational cost of Bayesian model averaging. The main additional computational cost comes from executing several (in our case, 50) forward passes through the weight samples of the BNN layer when deriving predictions for the unlabeled samples. This additional overhead is minimal since only the final layer, which consists of a small fraction of the total network weights, is (approximate) Bayesian. On the benchmarks of CIFAR-100 where gains are larger, we found the BNN versions to take only around 2 5% longer in wall-clock time on the same hardware when compared to the baseline, for the same total number of iterations. This justifies the BNN layer as a plug-in calibration approach with low computational overhead. The recorded run times for 500 epochs for each method on the same hardware are shown in Table 12
Table 12: Wall-clock run time (in hours) for 500 epochs for each method.
CIFAR-10
CIFAR-100
250 labels
2500 labels
400 labels
4000 labels
10000 labels
FM
30.8
19.3
77.2
74.7
71.9
BaM-FM
36.5
21.5
79.7
75.1
73.7
UDA
40.8
23.8
77.8
73.7
73.2
BaM-UDA
50.1
25.6
80.6
75.4
74.5
Computational cost of SWA & EMA. Both SWA and EMA requires storing another network of the same architecture, denoted g in the main text, which maintains the aggregate (exponentially weighted aggregate) of past network weights for SWA (EMA). Rather than using representations from the original backbone, f, representations from 9 are used to derive the better calibrated predictions, and thus the main computational overhead comes from the second forward pass needed per iteration. We timed the methods and found that PAWS+SWA and PAWS+EMA took around 18% longer in wall clock training time (on the same hardware) when compared to the baseline PAWS method for each epoch of training. However, as shown in the main text, PAWS+SWA and PAW+EMD resulted in a significant speed up in convergence, cutting training time by > 60% and additionally giving better test performances. Hence rather than an overhead, improved calibration in PAWS+SWA and PAWS+EMA in fact reduces the computation cost needed for the original approach.
25

---

Published in Transactions on Machine Learning Research (08/2023)
Extreme low-label settings in CIFAR-100
Table 13 compares BaM-UDA against FixMatch and UDA in the extremely low-label settings for CIFAR-100 such as 100 labels and 200 labels. These are settings with extremely small number of labels since there is only label per class (the lowest possible) for the 100 labels setting and 2 labels per class for the 200 labels setting. In the extreme setting of label per class, BaM-UDA gives decent 35.2% accuracy, a whopping 27% improvement from the UDA bascline (without the last bayesian layer) and 19% improvement from the next best method (FixMatch).
Table 13: Extremely low labels in CIFAR-100.
CTFAR-100 200 labels
100 labels
400 labels
FM
16.0+3.3
45.6+0.3
56.4+1.6
UDA
8.2+0077
19.0#0.1
44.1+0.7
BaM-UDA
35.211.3
51.2.10.2
60.3
J Societal Impact and Ethical Considerations
Semi-supervised learning is arguably one of the most important deep learning applications, in real life we often have access to an abundance of unlabeled examples, and only few labeled datapoints. Our work highlights the importance of calibration in semi-supervised learning methods which could yield benefits for real-world applications in two main ways: 1) a better deep learning model that is more data efficient and 2) a model that is better calibrated and can quantify uncertainty better. The latter is highly important for crucial societal applications such as in healthcare. However, as with any deep learning application, there may be biases accumulated during dataset collection or assimilated during the training process. This issue may be more acute in semi-supervised setting where the small fraction of labels may highly misrepresent the ground truth data. This may lead to unfair and unjust model predictions ions and give rise to ethical concerns when used for socictal applications.
26


Reviews:
[{"review_id": "3BW8VwqpiK", "reviewer": "Reviewer_v2un", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-written and easy to follow.\n- The paper is tackling an important problem in SSL, and the suggested solution is a natural way to go.\n- The performance of the proposed method, compared to the baselines with the selection criteria, is good.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a Bayesian treatment for reducing confirmation bias in semi-supervised learning (SSL). The main idea is to estimate the uncertainties of the predictions and use them to build a selection criterion for pseudo-labels. The paper uses the variational approximation to the posterior of BNN with only the last layer treated random, and shows that the variance in the predictions computed with multiple samples from the variational posterior could serve as a decent selection criterion for SSL. For the SSL methods without selection criteria, the paper proposes to use EMA or weight-averaged parameters for pseudo-labeling, and demonstrate that those methods outperform the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- While I agree with the generic idea of using better-calibrated predictions for SSL and building such calibrated predictions from BNNs, I'm a bit skeptical about the choice the authors made for the actual implementation. The proposed method (BaM) is based on the last-layer BNN with variational posterior. One may choose one of those options for scalability issues but why use both? For instance, if the variational approximation is employed, BNN with full stochasticity for all weights are still amenable (unless you are using extremely large neural networks), or if the last layer BNN is employed one may consider more accurate posterior approximations such as HMC. Since the quality of uncertainty estimation is crucial for the efficacy of SSL, it would be important to justify the choice made in the paper.\n\n- I'm a bit confused to see the experiments since PAWS+SWA or PAWS+EMA outperform BaM+PAW and the authors actually admit that. What is the main contribution of the paper? Is it only the BaM (last layer BNN + variational approximation + variance-based selection criterion), or the generic concept of applying uncertainty-based predictions for SSL? It would be important to clarify this, and if the latter is the main contribution, the paper might need some rewriting.\n\n- The philosophy of the proposed method, especially the part where the uncertainty of the prediction is measured through some sort of disagreement (variance) between the predictions computed from BNN posteriors. This is somewhat similar in the way the acquisition functions are computed in uncertainty-based active learning, to give an example, BALD or BatchBALD. It would be good to have a discussion in the related work part.", "broader_impact_concerns": "The authors addressed societal impact and ethical considerations in the appendix.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cwAhG3geYX", "reviewer": "Reviewer_pZ95", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is very difficult to work though.  There is a lot going on and a lot of moving pieces. \n\nThe introduction of the paper focuses on the ability of bayesian models and the marginalization inherent in the posterior predictive as affording better calibration and uncertainty quantification which ought to be useful for SSL.  Why not focus on that?  I want to know whether the bayesian model averaging aids SSL, but given all of the moving pieces here I am not convinced by the evidence provided that it was BAM and any calibration it provided that provided the benefits.\n\nI feel as though the simplest baseline and first direct test of the efficacy of BAM would be to start with a very basic UDA setup, where you simply take unlabeled data and two augmentations for the data and try to match the predictive distributions.  No selection criteria, no sharpening, just minimize the KL between the predictive distributions for the two augmented inputs.\n\nAs it stands, even though spending significant time with the paper, I'm not convinced I fully understand all of the details of what was done, would not be able to replicate the experiments and so am unconvinced that the gains are for the reasons described.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The high level idea appears to be using a last-layer bayesian neural network (trained with elbo) to give better calibrated and better uncertainty aware pseudo labels for semi supervised learning.\n\nHowever, the paper is a bit of a grab bag of a lot of little pieces.  First, there is a selection criteria used based on the variance of the posterior predictive to decide whether an example is worth training on or not, also most of the results in the paper are obtained by combining the last layer bayesian model averaging with Unsupervised Data Augmentation (UDA), and a strong and weak augmentation scheme, along with a sharpening operation applied to the posterior.\n\nLater in Section 7, weight averaging is used with PAWS for another set of experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "How does BAM by itself perform at SSL?  I would have liked to see ablations without the acceptance criteria applied, merely using the soft pseudo-labels from the posterior predictive as the target.  If the posterior predictive has higher variance, that will already effect the KL loss if the soft labels are used, I don't see why you would also need a hard selection criteria, that seems to be at odds with the notion that the posterior predictive is well calibrated.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vcDVDXfgU3", "reviewer": "Reviewer_nAoU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The proposed method is simple to understand and implement. \n- The experimental results cover multiple datasets such as CIFAR-10. CIFAR-100, and ImageNet. \n\nWeaknesses\n- In my opinion, the problem of semi-supervised learning in a low-shot regime is not significant. For example, the recent low-shot learning methods like CLIP-adapter and CaFo are much easier to use and can provide better performance. On the other hand, the recent fashion of zero-shot transfer enabled by large pre-trained models can already yield competitive enough results. Namely, I don't think the \"16% improvement in test accuracy on the CIFAR-100 with 400 labels benchmark\" is promising. \n\n- The technique novelty is limited. A last-layer Bayesian treatment is introduced, stochastic VI is used for learning, and the uncertainty estimates are leveraged for data selection. The community has thoroughly investigated every part of this. The only novelty arises from that the selected data is used back for model training. But, I want to point out that this paper makes a close analogy with the Bayesian active learning pipeline developed by Yarin Gal. The difference is that you use the current model to label the selected data, while Gal uses an additional expert for data labeling. \n\n- After reading the introduction, I thought the authors explored some theoretical connections between the confirmation bias in SSL and a Bayesian treatment. But, in fact, the findings are purely empirical. A deeper theoretical understanding is required to make the paper more convincing. \n\n- For Section 7, I don't agree that SWA and EMA are approximate Bayesian inference or Bayesian model averaging approaches. You should make clear the difference between the weight-space average and the function-space average. So, I think including this section under the current paper titled \".. Efficient Bayesian Model Averaging\" is improper. You can try SWAG, though I still don't regard it as a principled Bayesian inference way.\n\n- The performance gain is not substantial. In particular, I really want to see a significant gain on the imbalanced CIFAR-10-LT.\n\nMinor issues\n- The $\\rho_t$ is not an argmax operation but a one-hot operator at $t=0$. Right?\n\n- \"Bm contains the set of samples with ρ0(q) ∈ (m−1/M ,m/M ]\". ρ0(q) is a model prediction or model confidence or a one-hot vector? The notations are confusing.\n\n- In section 4, you should clarify how the other weights of the model are trained. They remain point estimates, so MLE or MAP is applied? You can find a unified interpretation in related BNN papers. \n\n- \"This constitutes a more accurate measure of model uncertainty compared to the maximum logit value\". Is there a citation explaining why it is more accurate? Besides, other uncertainty estimates, such as mutual informantion and softmax variance, also apply here. You can empirically evaluate their effectiveness. \n\n- Is the method hyperparameter-sensitive? What hyperparameters should be carefully tuned in a new task? Is there an ablation regarding this? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper replaces the last layer of a regular NN with a Bayesian one to compute calibrated uncertainty estimates for the data selection in semi-supervised learning. The techniques are simple, and the results are thorough. The paper also explores a different family of SSL methods that do not rely on a selection metric and makes some algorithmic modifications. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns detailed in Weaknesses and Minor issues.", "broader_impact_concerns": "None ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study popular semi-supervised learning (SSL) methods that produce pseudolabels during training, such UDA and FixMatch. They observe that the main difficulty here is that if the model is miscalibrated, low-confidence pseudolabels are included in training and so will reinforce downstream error. Their proposed solution is Bayesian model averaging, which is a standard way to improve calibration. This is done by inserting a Bayesian neural network at the last layer trained via variational inference. An additional innovation is to use a selection step to filter out high-variance pseudolabels.  The authors compare this two-step approach, obtaining small but consistent improvements over popular SSL methods in several datasets along with improved calibration.

There were a number of comments from reviewers; the authors provided a substantial response to all of these points and made a fairly solid revision that improves the clarity of the work.

Overall I believe the paper meets the bar. The basic principles introduced by the authors (ie, improving uncertainty quantification is crucial for pseudolabeling-based SSL methods) are both sound and useful (as also noted by several of the reviewers). The proposed implementation is a reasonable way to achieve these. 

Some concerns raised by the reviewers include,
- The value of SSL as a whole given zero-shot models. The authors have a reasonable answer here---and in any case, the pseudolabels produced by SSL can always be useful for fine-tuning zero-shot models.
- Small improvements from chaining multiple tricks. This is almost inevitable in a fairly mature field like SSL, where a huge number of ideas have been proposed and combined over the last 20+ years.

For this reason, the paper is heading towards acceptance. 

I recommend the following minor revisions. The paper, while making its basic points, could stand to be a bit more informative in some of its experiments, especially in two aspects: wall-clock time across experiments (rather than a few quoted ones), and the number of % of labeled VS unlabeled data. These two factors are important for the SSL audience. I suggest
- Adding a table in the appendix that shows change in wall-clock time for all of the settings , 
- The authors mention that the very low-labeled setting case is too sensitive to initalization. Fair enough---what is the gain when the smallest proportion of labeled VS unlabeled data where reasonable conclusions can be drawn? This should be in more favorable to BaM.




Decision: Accept (poster)