On the Importance of Calibration in Semi-supervised Learning
Authors: Charlotte Loh
Rumen Dangovski
Shivchander Sudalairaj
Seungwook Han
Ligong Han
Leonid Karlinsky
Marin Soljacic
Akash Srivastava
Venue: Submitted to ICLR 2023 (2023)

Abstract: State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining techniques of consistency regularization and pseudo-labeling. During pseudo-labeling, the model's predictions on unlabeled data are used for training and thus, model calibration is important in mitigating confirmation bias. Yet, many SOTA methods are optimized for model performance, with little focus directed to improve model calibration. In this work, we empirically demonstrate that model calibration is strongly correlated with model performance and propose to improve calibration via approximate Bayesian techniques. 
We introduce a family of new SSL models that optimizes for calibration and demonstrate their effectiveness across standard vision benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up to 15.9\% improvement in test accuracy. Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.

Under review as a conference paper at ICLR 2023
ON THE IMPORTANCE OF CALIBRATION IN SEMI-SUPERVISER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-art (SOTA) semi- comi-suprrvised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining tech- niques of consistency regularization and pseudo-labeling. During pseudo-labeling, the model's predictions on unlabeled data are used for training and thus, model calibration is important in mitigating confirmation bias. Yet, many SOTA methods are optimized for model performance, with little focus directed to improve model calibration. In this work, we empirically demonstrate that model calibration is strongly correlated with model performance and propose to improve calibration via approximate Bayesian techniques. We introduce family of new SSL models that optimizes for calibration and demonstrate their effectiveness across standard vision benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up 16.2% improvement in test accuracy on the CIFAR- 1 0004000-labels benchmark. Further- more, we also demonstrate their effectiveness in additional realistic and challenging problems, such as mmallaeebde datasets and in photonics science.
INTRODUCTION
While deep learning has achieved unprecedented success in recent years, its reliance on vast amounts of labeled data remains a long standing challenge. Semi-supervised learning (SSL) aims to mitigate this by leveraging unlabeled samples in combination with limited set of annotated data. In computer vision, two powerful techniques that have emerged are pseudo-labeling (also known as self-training) (Rosenberg et al. 2005; Xie et al.) 2019h) and consistency regularization (Bachman (et al., 2014 Sajjadi et al. 2016). Broadly,
labeling the technique where artificial labels are assigned to unlabeled samples, which are then used to train the model. Consistency regularization enforces that random perturbations of the unlabeled inputs produce similar predictions. These two techniques are typically combined by minimizing the cross-entropy between pseudo-labels and predictions that are derived from differently augmented inputs, and have led to strong performances on vision benchmarks (Sohn et al. 2020 Assran et al.) 22021).
Intuitively, given that pseudo-labels (i.c. the model's predictions for unlabeled data) are used to drive training objectives, the calibration of the model should be of paramount importance. Model calibration (Guo et al. 2017) is a measure of how a model's output truthfully quantifies its predictive uncertainty, i.e. can understood as the alignment between its prediction confidence and its ground- truth accuracy. In some SSL methods, the model's ss confidence is used as a selection metric (Lee) 2013)] Sohn al. 2020) to determine pseudo-label acceptance, further highlighting the need for proper confidence estimates. Even outside this family of methods, the use of cross cross-entropy minimization objectives common in SSL implies that models will naturally be driven to output high congidceneeeece predictions (Grandvalet & Bengio, 2004). Having high-confidence predictions is highly desirable in SSI. since we want the decision boundary to lie in low-density regions of the data manifold, i.e. away from labeled data points (Murphy (022). However, without proper calibration, model would easily become over-confident. This is highly detrimental as the model would be encouraged to reinforce its mistakes, resulting in the phenomenon commonly known confirmation bias (Arazo al. 2019).
Despite the fundamental importance of calibration in SSL, many -ftee-of-thearrr (SOTA) methods have thus far been empirically driven and optimized for performance, with little focus techniques that specifically target improving calibration to mitigate confirmation bias. this work, we explore the generality of the importance of calibration in SSL by focusing on two broad families of SOTA

---

Under review as a conference paper at ICLR 2023
Weak
Bayesian layer
Averoge many predictions
Acceptance criteria
Unlabeled
Pseudo-label
H(p.q)
Strong
Model
Prediction
Figure Illustration of BAyesian Model averaging (BAM) approach
SSL methods that both use pscudo-labeling and consistency regularization: 1) threshold-medieee methods (Sohn al., 2020: Xie al.
2019a Lee 2013) where the model selectively accepts pseudo- labels whose confidence exceed a threshold and 2) "representation learning" methods adopted from self-supervised learning (Assran et al. 2021) where pseudo-labed are non-selective and training consists of two sequential stages of representation learning and fine-tuning.
To motivate our work, we first empirically show that strong baselines like FixMatch (Sohn et al. 2020) and PAWS (Assran et al.] (021), from each of the two families, employ a set of indirect techniques to implicitly maintain calibration and that achieving good calibration is strongly correlated to improved performance. Furthermore, we demonstrate that it is not straightforward to control calibration via such indirect techniques. To remedy this issue, we propose techniques that are directed to explicitly improve calibration by using approximate Bayesian techniques that designed to capture model uncertainty. In our work, we explored approximate Bayesian neural networks (Blundell et al./ 2015) and weight-ensembling approaches (Izmailov al. 2018). Such Our modification forms a new family of SSL methods that improve upon the SOTA on both standard benchmarks and real-world applications. Our contributions are summarized as follows:
1. Using SOTA SSL methods calibration is strongly correlated to better model performance in SSL.
studies, empirically show that maintaining good
2. We propose to use approximate Bayesian techniques to directly improve calibration and provide theoretical results on generalization bounds for SSL to motivate our approach.
3. We introduce new family of methods, BAM-, that improves calibration via BAyesian model averaging (see|Fig. 1) and demonstrate their improvements upon variety of SOTA SSL methods on standard benchmarks, notably giving up to 16.2% gains in test accuracy.
4. We further explored weight averaging techniques, one of which (i.e. EMA) being well- established in SSL and show that their effectiveness can be understood from improving pseudo-label calibration.
5. We further demonstrate the efficacy of BAM- in more challenging and realistic scenarios, such as class-imbalanced datasets and a real-world application in photonic science.
RELATED WORK
Semi-supervised learning (SSL) and confirmation bias. A fundamental problem in SSL methods based on pseudo-labeling (Rosenberg al. (0055) is that of confirmation bias Tarvainen & Valpola 2017, Murphy 2022). i.e. the phenomenon where a model overfits to incorrect pseudo-labels. Several strategies have emerged to tackle this problem: Guo et al.] (2020) and Renn et al. (2020) looked into weighting unlabeled samples, Thulasidasan et al. (2019) and Arazo et (2019) proposes to use augmentation strategies like MixUp (Zhang et al. (017). while Cascante-
et al. (2020) proposes to re-initialize the model before every iteration to overcome confirmation bias. Another popular technique is to impose a selection metric (Yarowsky 1995) to retain only the highest quality pseudo-labels, commonly realized via a fixed threshold on the maximum class probability (Xie et al.) 2019a Sohn et al.) 2020). Recent works have further extended such selection metrics to be based dynamic thresholds, either in time (Xu et al. (021) class-wise (Zou et al. 2018 8 Zhang et al.) 2021). Different from the above approaches, our work proposes to overcome confirmation bias in SSL by directly improving the calibration of the model through approximate Bayesian techniques.
2

---

Under review as a conference paper at ICLR 2023
Model calibration and uncertainty quantification. Proper estimation of a network's prediction uncertainty is of practical importance (Amodei et al. (016)) and has been widely studied. A common approach to improve uncertainty estimates is via Bayesian marginalization (Wilson & Izmailov 2020), i.e. by weighting solutions by their posterior probabilities. Since exact Bayesian inference computationally intractable for neural networks, series of approximate Bayesian methods have emerged, such as variational methods (Graves) 2011 Blundell et al., 2015 Kingma et al.) 2015).
Hamiltonian methods (Springenberg et al. 2016) and Langevin diffusion methods (Welling & Teh 201T). Other methods to achieve Bayesian marginalization also exist, such as deep ensembles (Laksh minarayanan al. 2016) and efficient versions of them (Wen et al. 2020 Gal Ghahramani] 2015). which have been empirically shown to improve uncertainty quantification. The concept of uncertainty and calibration are inherently related, where calibration commonly interpreted as the frequentist notion of uncertainty. In our work, we will adopt some of these techniques specifically for the context of semi-supervised learning in order to improve model calibration during pscudo-labeling other methods for improving model calibration exists (Plati) 1999; Zadrozny & Elkan 2002 Guo et al. 2017). these are most commonly achieved in a post-hoc manner using a held-out validation set; instead, we seek to improve calibration during training and with a scarce set of labels. Finally, in the intersection of SSL and calibration, Rizve et al. (2021) proposes to leverage uncertainty to select better calibrated subset of pseudo-labels. Our work builds on similar motivation, however, in addition to improving the selection metric with uncertainty estimates, we further seek to directly improve calibration via Bayesian marginalization (i.e. averaging predictions).
While
NOTATION AND BACKGROUND
Given small amount of labeled data L = ((1) 31 ) (here, y1 (0. (0, one-hot labels) and a large amount of unlabeled data 2 = { i.e. No. >> N₁, in SSL, we seek to perform a K-class classification task. Let f(- fy) be backbone encoder (e.g. ResNet or WideResNet) with trainable parameters f,. h(,, 0%) be a linear classification head, and H denote the standard cross-entropy loss.
Threshold-mediatee methods. UDA (Xie et al.] 2019a) and FixMatch (Sohn et al. 2020) minimizes a cross-entropy loss on augmented copies of unlabeled samples whose confidence exceeds a pre-defined threshold. Let o and ag denote two augmentation transformations and their corresponding network predictions for sample z to be 91 ho (z)) and 92 = ho f (x))))), the total loss on batch of unlabeled data has the following form:
Threshold-mediater methods such as Pseudo-Labels (Lee) 2013).
aB
1
L
(max((1).w) 2 T) (2.u)
BB
where B denotes the batch-size of labeled examples, μ scaling hyperparameter for the unlabeled batch-size, E [0, 1] is a threshold parameter often set close to 1 and pt is either a sharpening operation on the pseudo-labels, i.e. [pr( := / >/>>">=]>> +- 0). Pt also implicitly includes estop-gradient" operation, i.e. gradients are not back prok-propagated from predictions of pseudo-labels. L,, is combined with the expected cross-entropy loss on labeled
or an argmax operation (i.e.
examples, L1 3/2 Σ (3n, 91,1) to form the combined loss L, + XL.. with hyperparameter X.
Differences between Pseudo-Labels, UDA and FixMatch are detailed in Appendix D.I
Representation learning based methods. We PAWS (Assran et al.) 2021 as a canonical exam- ple for this family. A key difference from threshold-mediatee methods is the lack of the parametric classifier h. which is replaced by a non-parametric soft-neares, neighbour classifier (Ta) based on a labeled support set { Let 21 = ((1(z)) and 22 = f(a((z)) be the representations for the two views from the backbone encoder, their pseudo-labels (q1, (2) and the unlabeled loss are given by:
B
aB
a. = Te(=,, (FoH)
Σ ==1
d(zi, 2x) Us d(z,, 23.) E
Lu = BnB
1
Σ
H(pl(a.): H (42,w), (1,))
(2)
where d(a, = exp(a [||||||||ll> is a similarity metric with temperature hyperparameter Tp and all other symbols have the same meanings defined before. The combined loss is Tan, + Time mix
3

---

Under review as a conference paper at ICLR 2023
(a) FixMatch
(b) PAWS
0.30
72
0.99
Temp.
Sharpen ME-Max
Test
ECE
71
Acciracy
73.0
0.05
0.1
v
69.3
0.225
0.50
0.05
0.25
v
71.6
0.213
72.5
0.50
0.05
0.25
*
67.0
0.257
68
69
72.0
0.01
0.1
0.25
v
72.0
0.229
0.180
0.185
0.190
0,195
0.1
0.25
x
68.2
0.262
0.23
ECE
Figure 2: Empirical study CIFAR-100, 4000 labels. (a) Fix Test accuracy (%) against ECE (lower ECE is better calibration) when varying the threshold (value shown beside cach scatter point). (b) PAWS: effect of various parameters on test accuracy and ECE; scatter plot explores wider range of parameters, Temp. E [0.03, 0.1] and Sharpen E [0.1, 0.5].
where the latter regularization term Lmn-max (ÿ) that seeks maximize the entropy of the average predictions = (1/(2pB)) Σn (pr(g1,)) + Pc (c(2..))
model's calibration is via the Expected Calibration Error (ECE). Following (Guo et al. 2017 Minderer et al. (021), we focus on slightly weaker condition and consider only the model's most likely class-prediction, which be computed as follows. Let Po(4) denote the model's prediction (where Po is the argmax operation) as defined before, the model predictions on batch of N samples are grouped into M equal-interval bins, i.c. B,, contains the set of samples with (o(4) E (m-1 m expected difference between the accuracy and confidence of each bin over all N samples:
Calibration metrics. A popular empirical metric
ECE is then computed as the
M
18ml N
ECE Σ
where acc(Bm) = (1/(B,ml) Eigs,, (pv(g:) = y4) and conf( = (1/lBm))) EEEB max qi with y; the true label of sample i. In this work, we estimate ECF using M bins. We also caveat here that while ECE is not free from biases (Minderer et al.) [021), we chose FCF over alternatives (Brier 1950: DeGroot Fienberg 1983) due to its simplicity and widespread adoption.
CALIBRATION IN SEMI-SUPERVISED LEARNING
Better calibration correlated better performances. To motivate our work, we first perform detailed ablation studies SOTA SSL methods. In particular, we used FixMatch (Sohn et al., 2020 and PAWS (Assran al., (02T), each from the two families, due to their strong performance. FixMatch relies using high value of the selection threshold r to mitigate confirmation bias by accepting only the most credible pseudo-labels; on the other hand, PAWS uses multiple techniques that seem to have effect shaping the output prediction distribution and implicitly controlling calibration - these include label smoothing (Müller et al. 2019)), mean-entropy maximization (Joulin & Bach 2012), sharpening (Berthelot et al. 2019h Xie et al. (019a) and temperature scaling (Guo et al., 2017). In Fig. 2] we ablated on all of these parameters and observe an overall trend, for both methods, that model performance is strongly correlated better calibration (i.e. lower ECE). However, these trends are inevitably noisy since none of these parameters tunes for calibration in isolation but are instead optimized towards performance. Therefore, in this work, we aim explore techniques that predominantly adjusts for model calibration in these methods to clearly demonstrate the direct effect of improving calibration towards model accuracy.
4.1 IMPROVING CALIBRATION WITH BAYESIAN MODEL AVERAGING
Bayesian techniques have been widely known to produce well-calibrated uncertainty estimates Wil son & Izmailov 2020), thus in our work, we explored the use approximate Bayesian Neural Networks (BNN). To minimize the additional computational overhead, propose only replace the final layer of the network with a BNN layer. We propose a novel family SSL methods termed BAM- (for BAyesian Model averaging) and provide illustration of this approach in Fig...
4

---

Under review as a conference paper at ICLR 2023
For meddatte methods the BNN layer is the linear classification head h and for representa- tion learning methods this is the final layer of the projection head. For brevity we will simply denote this BNN layer to be h and an input embedding to this layer to be v in this section. Following Bayesian approach, we first assume a prior distribution on weights p(On). Given some training data Dx= = (X,Y) Y), we seek calculate the posterior distribution of weights, p(0,, (Dx),, which can then be used to derive the posterior predictive ply e, Px) = p(ylu, 0,
(Dx)ddn. This process is also known "Bayesian model averaging" "Bayesian marginalization" (Wilson & Izmailov 2020). Since exact Bayesian inference is computationally intractable for neural networks, we adopt variational approach following (Blundell et al.) 2015). where we learn a Gaussian variational approximation to the posterior ga (0)) parameterized by é, by maximizing the evidence lower- bound (ELBO) (see Appendix for details). The ELBO = F.g logp(Y 0) (L(((0)))) consists of log-likelihood (data-dependent) term and KI. (prior-dependent) term.
Theoretical results. We motivate our approach by using Corollary below, derived from the PAC-Bayes framework (see Appenndxx for the proof). The statement shows the generalization error bounds on the variational posterior in the SSL setting and suggests that this generalization error is upper bounded by the negative ELBO. This motivates our approach, i.e. by maximizing the ELBO we improve generalization by minimizing the upper bound to the generalization error. The second term on the right side of the inequality characterises the SSL setting, and vanishes in the supervised setting.
Corollary Let D be a data distribution where i.i.d. training samples are sampled, of which N₁ are labeled, (I. ~ DN: and Nu are unlabeled, (z, 5) "DN= where & (y.) denotes the model- assigned pseudo-labels (true labels) for input Iy For the negative log likelihood loss function E, assuming that & is sub-Gaussian (see Germain et al. (2016)) with variance factor s², then with probability at least 1 - 6, the generalization error (denoted as ( is given by,
of the variational posterior q(fl))
1 N
N
1 N
1 5
$² 2
Byy (g) S N
-ELBO] - (9(0110
log
+
log
(4)
As depicted in Fig. 1 pseudo-labeling proceeds in two-stages: 1) M weights from the BNN layer are sampled and predictions are derived from the Monte Carlo estimated posterior predictive, i.e. q (1/M) End h(u o(()), ) and 2) the selection criteria, if present, is based upon their variance, of = (1/M) Em (h(v, ofmi) g)² at the predicted class c argmax_ [g]e.. This constitutes better uncertainty measure as compared to the maximum logit value used in threshold-mediated methods and is highly intuitive if the model's 's prediction has large variance, it is highly uncertain and the pseudo-labe should not be accepted. In practice, as o² decreases across training, we use a simple quantile Q over the batch define the threshold where pseudo-labels of samples with of < Q are accepted, with Q as a hyperparameter (see Appendix C.T for pseudocode). In representation learning methods where predictions are computed using a non- non-parametric nearest neighbour classifier. we use the "Bayesian marginalized" versions of the representations, i.c. q: = TT(2,, {&+}) in Eq. (2)
where 2 = (1/M) ΣM M h(, e(m)). We explore the effectiveness of BAM by modifying upon SOTA SSL methods and name them "BAM-X", which incorporates approximate BAyesian Model averaging (BAM) during pseudo-labeling for SSL method X.
4.2 IMPROVING CALIBRATION WITH WEIGHT AVERAGING TECHNIQUES
A BNN classifier has two desirable characteristics: 1) multiple predictions are "Bayesian marginalized" and 2) better measure of uncertainty (i.e. confidence estimates) using the variance across predictions. Representation learning SSL methods that do not use a selection metric cannot explicitly benefit from (2) and in these cases, instead of aggregating over just one layer, one could seek to aggregate over the entire network. To explore the generality of the importance of calibration in SSL beyond Bayesian layer approach, we further explored well-established weight averaging approaches such as Stochastic Weight Averaging (Izmailov et al. 2018) (SWA) and Exponential Moving Averaging (EMA) (Tarvainen & Valpola 2017; He et al. calibration during pseudo-labeling.
2020 Grill et al. 2020a) and studied their role in
To do so, we maintain a separate set of non con-trainable weights 6, containing the aggregated weight average which used throughout training produce better calibrated pseudo-labels. In SWA,
5

---

Under review as a conference paper at ICLR 2023
these weights are updated via 0g < (na0g+01)/(n, + 1) every iteration, where na represents the total number of models in the aggregate and fy are the trainable parameters of our backbone encoder as before. The update of EMA only differs slightly: 0g f 70g + (1 7)0/, where r is a momentum hyperparameter controlling how much memory 0g should retain at each iteration (see Appendix C.2 for pseudocode). While EMA has been previously explored in the context of SSL (Tarvainen & Valpola| 2017), to the best of our knowledge SWA has not been explored in SSL and more importantly, their link pseudo-label calibration has not been explicitly shown. In this work, we demonstrate that the effectiveness of such weight averaging approaches can be understood from an improved calibration of pseudo-labels labels.
EXPERIMENTAL SETUP
In our experiments, we begin with and modify upon the original implementations of the baseline SSL methods. The backbone encoder f is Wide ResNet-28-2. Wide ResNet-28-8 and ResNet-50 for the CIFAR-10, CIFAR-100 and ImageNet benchmarks respectively. Where possible, the default hyperparameterrs and dataset-specific settings (learning rates, batch size, optimizers and schedulers) recommended by the original authors were used and where unavailable (for e.g. CIFAR- 100 on PAWS), the baseline first optimized to the best we could. To ensure fairness, we use the same configuration and same set of hyperparameters when comparing our methods against the baselines.
Bayesian model averaging with a BNN final layer. We set the weight priors as unit Gaussians and use separate Adam optimizer for the BNN layer with learning rate 0.01, no weight decay and impose the same cosine learning rate scheduler as the backbone. We set Q = 0.75 for the CIFAR-I00 benchmark and Q = 0.95 for the CIFAR- 10 benchmark; which are both linearly warmed-up from 0.1 in the first 10 epochs. As Q is computed across batches, we improve stability by using a moving average of the last 50 thresholds. For representation learning methods without a parametric classifier, we do not use a separate optimizer and simply impose a one-minus-cosine scheduler (i.e. scheduler (2) from the next paragraph) for the coefficient to the KL--divergence loss that goes from 0 to 1 in T' epochs where we simply picked the best from T E {50, 100}.
Weight averaging. We delay the onset of SWA to after some amount of training has elapsed (i.e. Tewa epochs) since it is undesirable to include the randomly initialized weights to the aggregate. We set Tewa = 200 and Tewa = 100 for CIFAR-10 10 and CIFAR- respectively. In contrast, EMA has natural curriculum to "forget" older model parameters since more recent parameters are given more weight in the aggregate. We experimented with two schedules for yy: 1) using linear warm-up from to 0.996 in 50 epochs and then maintaining 7 at 0.996 for the rest of training and 2) using a one-minus-cosine scheduler starting from 0.05 and decreasing to 0 resulting in a 0.95 to I range for y. We visualize these schedulers and include ablation studies on them and on Tswa in Appendix G.3
ECE and test accuracy evaluation. In our experiments, we found that the test accuracy exhibits a considerable amount of noise across training, especially in label-scarce settings. Sohn et al. (2020) proposes to take the median accuracy of the last 20 checkpoints, while Zhang al. (2021) argues that this fixed training budget approach is not suitable when convergence speeds of the algorithms are different (as the faster converging algorithm would over-fit more severely at the end) and thus report also the overall best accuracy. In our experiments, we adopt a balance between the two aforementioned approaches: we consider the median of 20 checkpoints around the best accuracy checkpoint as the convergence criteria, and report this value as the test accuracy. The ECE is reported when the model reaches this convergence criteria (One could also also aggregate the ECEE up till convergence we found this gave similar trends and thus report the simpler metric.) For PAWS, we follow Assran et al. (20TT) and report the test accuracy and ECE from the soft nearest neighbour classifier on the CIFAR benchmarks and after fine-tuning a linear head for the ImageNet benchmark.
6 RESULTS
Bayesian model averaging. Our main results on CIFAR-10 and CIFAR-100 in Table 1 show the use of a BNN layer to improve calibration for all methods in our study. Table demonstrates that incorporating BNN final layer successfully reduces the ECE over the baselines across all the

---

Under review as a conference paper at ICLR 2023
Table 1: Calibration in SSL showing "Test accuracy (%) / ECE", Improving calibration improves test accuracies consistently across all benchmarks and across variety of SSL methods, Pseudolabel
Lee 2013). FixMatch (FM) (Sohn et al. 2020), UDA (Xie et al. 2019a) and PAWS (Assran a.. 2021 ). Rows with the prefix "BAM-" refers to BAyesian model averaging via BNN final layer, while +SWA and +EMA are using weight averaging techniques. For each benchmark, results are averaged over 3 random dataset splits. Entries with dashes (-) indicate that training did not converge.
(PL)
CIFAR-10
CIFAR-100
lahels
2500 labels
400 labels
4000 labels
10000 labells
Threshold-mediineed methods
PL (repro)
53.8 0.395
$1.0 0.197
0,336
664/ 0.23)
BAM-PL (ours)
81.5 (10.5) /0.19 0.190
50.8 (11.7) 0.342
66.4 0.229
FM trepro)
94.2 0.051
95.7amm / 000009
56.4816/ / 0.366amm7
74,2a53 / 0,1883a000
78.1 00.147
BAM-FM (ours)
95.0 (10.5) / 0.045
95.7=aa / 0.033e2s
59.0e14
0.331
74.8sa29 (10.6) 0.171=
78.1 0.139tuux
UDA (repeo)
94.3/ 0,050
; 0.039
44.
/ 0.473amm
72.9 0.189a
2003 / 05.
BAM-UDA (ouns)
95.3 (11.0) D.042
95.8 (10.1) 0.088
(L3 0.314
(1.2)) (0..2.50/00.65.... 165
78.3 (11.1) / 1.138,0000
Rep-lcarning methods
PAW'S (repro)
90.5 00001
95.3 / 0.046
44.5 / 0.458
71.5g33 / 0.232gga05
75.6.t / 00.198aooe
BAM-PAWS (ours) 94.5 (14.0) (D084
(f0.1) / 0.046
47 (12.9) / 0,372
72.5 (11.0) / /0.288
76.5 (10.9) / 0.195
PAWS-SWA (ours)
90.9 (10.4) / 0.088
95.6 (10.3) 0.046
45.2((7.7) / 00,44
74. 74.5axx((330) 0,1993uux
78.4 (f2.) 0.164
PAWS-EMA 90.9 (10.4), D.067
95.8 (10.5)
46.2((11.7)// 0.445
73.5gen(12.0) 19,19
77.2 (11.6) / 0.168
benchmarks and as a result, we also attained significant improvements in test accuracies, notably up to 16.2% (for UDA on CIFAR-100-400 labels). Interestingly, while the baseline FixMatch outperforms UDA across all the benchmarks, improving calibration in UDA (i.e. BAM-UDA) allows it outperfom FixMatch and even the calibrated version of FixMatch (i.e. BAM-FM), A key difference between FixMatch and UDA is the use of hard pseudo-labels in FixMatch (i.e. + 0 in Pt defined infeeciion 3) versus soft pseudo-labels in UDA (with f 0.4): this suggests that a Bayesian classifier is more effective in conjunction with soft pseudo-labels. Intuitively, this makes sense, since better calibration could potentially allow the model to leverage on information about its prediction on classes apart from the one it's s most confident of. This further underscores the importance of calibration - while information about the model's prediction on other classes is useful, the lack of proper calibration prohibits the model from using this information, resulting in stronger confirmation bias compared to the case where this information is not used (i.e. UDA vs FixMatch).
Leveraging on this insight, we significantly reduce the sharpening of the pseudo-labels in BAM-UDA by setting f = 0.9 (see) (pppendix G.2 for ablations on t). BAM-UDA consistently outperforms all threshold-mediiteed baselines across all CIFAR benchmarks. Overall, we found that the improvements in both calibration and test accuracy are more significant for label-scarce settings, as expected, since the problem of confirmation bias is more acute there and thus calibration can provide greater benefits by mitigating this. The additional computation overhead incurred from our methods is minimal, adding approximately only 2-5% in wall-clock time (see (pppendix H). While previous works (Sohn eet al., 2020) also include extreme low label settings such as CIFAR- 10-40 labels, we found this benchmark to be highly sensitive to random initialization and thus exclude them in our study.
Why does improving calibration improve performance? To further understand the dynamics between calibration and test accuracies, we track the test accuracy and ECE over the course of training for UDA. FixMatch and BAM-UDA and plot them in Fig. 3 While the baselines learn effectively for the initial stages of training, learning is eventually hindered - as the model is encouraged to output increasingly confident predictions due to the entropy minimization objective, in the absence of explicit calibration, the baselines inevitably become over-confident, resulting in drop in test accuracy. This phenomenon of confirmation bias is evident from that 1) the baselines' output predictions become increasingly over-confident (Fg.. 3/1) and 2) they make increasingly more mistakes on the accepted pseudo-labels (Fig. 3d). In contrast, explicit calibration via Bayesian model averaging allows BAM- AAMA to improve its calibration throughout training, resulting in a constantly improving purity rate and effectively mitigating confirmation bias, thus promoting learning for longer periods to result in better final performances. Further ablation studies are discussed in Appendix G.2
Weight averaging techniques. The effect of better calibration is also clearly demonstrated in the weight averaging approaches [Tig. 4). This is particularly evident through SWA was switched on (at Tawa = 200 epochs), the ECE quickly dips and the test accuracy also correspondingly spikes. Furthermore, we also observe a significant improvement in convergence rates resulting from the weight averaging techniques, ie. better test accuracies can be attained in just third of the number of training iterations of the baseline. Ablations Towa and momentum
7

---

Under review as a conference paper at ICLR 2023
Test Accuracy
ECE
Purity rate
UDA
63-
0.35
57-
BAM--M (ours)
BAM LDA (ours)
0.50
0.-
45-
0.45
Max
(d)
38
UDA
0.40
0.95
FiaMatchh
(ours)
0.35
BAK-UDA tours]
0.85
20
6:0k
BOOK
0.30
0.80
500K
1000c
Eerations
Iterations
Figure 3: UDA, FixMatch, BAM-FM and BAM-UDA across training on CIFAR- 100 with 400 labels (a) Test accuracies and (b) ECE as function of training time. (c) Purity rate shows the accuracy rate of unlabeled samples that are accepted by the selection metric and (d) Max prob shows the model''s confidence, i.e. the average maximum class probability of all unlabeled samples.
CIFAR10-2500: Test ACC.
CIFARIO-2500: ECE
97
0.12
RAW'
RAWS-SNNA (7am - 200)
34-
0.10
PAWSHEMM
92-
3.08
59-
0.06
PAWS- 1WA TT 200)
PAMSS EMA
57
100
200
300
400
300
600
0.04
100
200
330
400
5.00
600
Epochs
Eoochs
Figure 4: Weight averaging techniques PAWS for CIFAR-10 with 2500 labels displays highly consistent relationships between increasing test accuracy and decreasing ECE.
schedules are presented in Appenddx G.3 Our motivation for exploring weight averaging techniques in PAWS was attributed to its lack of a selection metrie; however, given that almost all SSL. methods have a "stop-gradient operation on pseudo-labels, one could potentially also explore the use of weight averaging techniques in threshold-meding where we studied the use of EMA on the best performing threshold-mediene Notably, we found that EMA consistently under-performs Bayesian model averaging, highlighting the superiority of a BNN classifier in meddattedd methods. This could be attributed to the fact that, in addition to averaging predictions, BNN classifier provides better uncertainty estimates to improve the selection metric in these family of methods (see Appenddix G.T for further discussion).
methods. Results are shown in Appendixx G.I
baseline, i.c. FixMatch.
6.1 LARGE-SCALE REAL-WORT. DATASETS
ImageNet. We further verified the effectiveness of calibration on the large-scale dataset ImageNet. Out of all the baseline methods in our study, PAWS reports the best performance on the ImageNet- 10% benchmark (i.e. 73.9% for 100 epochs vs 71.5% on the best perform- ing fireshold-mediatee
Table 2: ImageNet-100s show- ing "Top-1 test accuracy (%) ECE" for PAWS and our meth- ods, trained for 200 epochs and fine-tuned with linear head.
method, FixMatch). Due to computational resource limitations, we focus experiments only the best baseline PAWS and explored calibration improvements using SWA and EMA. We use the 10% labels setting Assran et al. (2021) with slight modifications to the default setting in order to fit on our system (see Appendix 13.2 for implementation details). In particular, maintain the exact same configuration across the baseline and our methods. Results are shown in Table 2| where our methods provide consistent improvements in calibration, and correspondingly, we also obtain consistent improvements on the test accuracy.
IN-10%
PAWS +SWA (ours)
74.4 / 0.186
74.5 / 0.186
+EMA (ours)
75.1 / 0.183
Long-tailed datasets. Datasets in the real world are often long-tailed or class-imbalanced, where some classes more commonly observed while others are rare. We curate long- versions from the CIFAR datasets following Cao et al. (2019). where a indicates the imbalance ratio (i.c. ratio between the sample sizes of the most frequent and least frequent classes). We randomly select 10%
8

---

Under review as a conference paper at ICLR 2023
Table 3:
Long-tailed CIFAR-10 & CIFAR-100 showing "Test accuracy (% / ECE". We 10% of labels from each class. For better interpretation, we also show the supervised (100% labels) accuracy reported in Cao et al. 2019)) which use a different architecture, ResNet-32, and an algorithm targeted for long-tailed problems.
CIFAR-I0-11
CIFAR- 100-TT
10
100
100
FM BAM-UDA (ours)
91.3 0.073
70.0 0.26
48.8 / 0.38
28.6 / 0.55
91.6 (10.3) / 0.067
71.2 (11.2) 0.24
53.6 (14.8) / 0.32
31.9 (13.3) 0.50
Supervised
88.2
77.0
58.7
42.0
of the samples in each class to form the labeled set; see further details in Appendix E] We use the best performing baseline method from Table i.e. FixMatch (FM), as our baseline and compare against BAM-UDA. Results are shown in Table 3] where BAM-UDA achieves consistent improvements over FM in both calibration and accuracy across all benchmarks. Notably, gains from improving calibration more significant than those in the class-balanced settings (for e.g. BAM-UDA improves upon FM by a smaller margin of 1.5% on the 10%), further underscoring the utility of improving calibration in more challenging SSL problems. Further results are in AApenddix E.2] where test samples were separated into three groups depending on the number of samples per class and test accuracies plotted for each group.
labels benchmark which is also approximately
Photonics science. A practical example of real-world domain where long-tailed datasets are prevalent is that of science samples with the desired properties are often much rarer than trivial samples. Further, SSL is highly important in scientific domains since labeled data par- ticularly scarce (owing to the high resource cost needed for data collection). To demonstrate the effectiveness of calibration, we adopt a problem in photonics (Loh et al. 2022), where the task is 5-way classification of photonic crystals (PhCs) based on their band gap sizes. A brief summary and visualization of this dataset are detailed in Appendix F] We explored an approach similar to FixMatch for the baseline and similar BAM-UDA for ours (with modifications needed in the augmentation strategies; see Appendix F). Results are shown in Table 4| where we demonstrate the consistency of BAM-UDA's elfectiveness in improving calibration and accuracies in this real-world problem.
Table 4: Photonic crystals (PhC) band gap predic- tion showing "Test accuracy (%) / ECE". The fully- supervised (100% labels) accuracy 88.5%.
PhC-10%
PhC-1% %
78.8 0.098 BAM-UDA 81.0 (12.2) / 0.052 56.9 (11.9) 0.356
FM
55.0 / 0.385
CONCLUSION
In this work, we demonstrate that calibration plays a crucial role in SSL methods specifically, since confirmation bias is fundamental problem in SSL, it is imperative for the model to be well- calibrated to mitigate this problem. In particular, we demonstrated the strong correspondence between calibration and model performance in SSL methods and proposed to use approximate Bayesian techniques to directly improve calibration in state-of-the-art SSL methods. Notably, we demonstrate their effectiveness across a broad range of SSL methods and further underscore their importance in more challenging weal-world datasets. We hope that our findings can motivate future research directions to incorporate techniques targeted for optimizing calibration during the development of new SSL methods. Furthermore, while the primary goal of improving calibration is to mitigate confirmation bias during pseudo-labeling, an auxiliary benefit brought about by our approach is a better calibrated network, i.e. one that can better quantify its uncertainty, which is highly important for woald applications. A potential limitation in our work lies in the use of ECE as a metric to measure calibration which, while commonly used across literature. ct al., (211).. However, in our work, we empirically demonstrate that despite their flaws, the ECE metric still provides good correlations measuring confirmation bias and test accuracy.
not free from flaws (Nixon)

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
To ensure that our reported results are reproducible, all source code will made available in a publicly hosted repository soon, We further list all details training configurations and hyperparameter settings in Section and Appendix
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.
06565
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning, 2019. URL httpt//ss///axiii.o.rrtt
abs/ abs/1908. 02983
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by non-paramettically predicting view assignments with support samples. CoRR, 0os/2104.13963. 2021. (tRhttpss
//axxiv.org/abs//20000 13963
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL (bbela24cc0611112222222222222-22
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation abs /1111.91.... 09785
anchoring. CoRR, abs/1911.00 0ts//991.00985, 2019a. URL httptt::://xxi../o///
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beyyelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019b. URL [cd138d04 9e1.0004996888880 11c2334444..351117777000eee22223
neurips wwtpuuriisscccpppeeeeee httpp//appr/219//111
Charles Blundell, Julien Comneiie, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks, 2015. URL http://ttts//aaxxxvvr///////// /ttp///0aaab555055554444
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: Nonasymp-
totic Theory of Independence. Oxford University Press, 02 2013. ISBN 9780199535255.
doi: 10. 001108//aamnfsss0777011155555.50 0000... URL https:
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1): - 3, 1950. doi: 101177//500000511500000000000000
2.0.CO;2.
1520-0493.0 01w1:4093111100_0_00000888801111 0tt_::/___0001_orrtt__0___.0___0 xm1
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced
datasets with bibellddsttibuton-awaaaa margin loss, 2019. URL httppss//aaxxi..orgg///
1906.07413
Paola Cascante-Bonilke Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Self- paced pseudo-labeling for semi-supervised learning. CoRR, abs/2001 106001, 2020. URI https
06001
Thomas Christensen, Charlotte Loh, Stjepan Picek, Domagoj Jakobović, Li Jing, Sophie Fisher, Vladimir Ceperic, John D. Joannopoulos, and Marin Soljačić. Predictive and generative ma-
chine learning models for photonic crystals. Nanophotonics, (13))4183-411 4192, October 2020.
ISSN 2192-8614, doi: 10. htt.:1/////ammo-222000011111 document//oii/ http:///ooooooo.111ooaaaooooogpe
WtRRLttps:: http:ss:///www.// /www http://wwwwww......... com/ html
10

---

Under review as a conference paper at ICLR 2023
Ekin D. Cubuk, Barret Zoph. Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with a reduced search space, 2019. URL bttpp://arxiv.org/ abs/
1909. 13719
Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society. Series D (The Statisticiam), 32(1/2): 1983. ISSN 00390526, 14679884. URL http: www. jstor. org/ staallalsa..... /292
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model 021422
uncertainty in deep learning, 2015. URL
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-hayesian the- ory meets bayesian inference. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL httpss://proceeednggg www.:pi.......ur.....oom..... httpg///....rr.oo///////////////
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004, URL htttps://proceediiiii .nt.eeeeedinnnn.............. nwwrips....... cc cc//papet//0044/51.l..
9612550bbB61addf3eeee ott.://bceeaaaaaaa22222222222883 Paper. Paperr.d.
Alex Graves.
Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 201 1. URL
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural 2020a.
Information Processing Systems,
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020b. URL WRRLhttts://arrt htt.aaawwi...gaa/////a/aaaa
07733
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017. htLt:ttps:/// http::::///.xv..oooo/////////// 1at.s/176...000
Lan-Zhe Guo, Zhen-Yo Yu Zhang, Yuan Jiang. Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 3897-3906. PMLR, 13-18 Jul 2020. URL
httpss:/proceeeings mlr. www.s/pesssse.. v1 http://11199u22o..1coo www.gu....h..mm
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averaging weights leads to wider optima and better generalization, 2018. httstt/s:
orq/abs/1803 05407
D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. Meade. Photonic Crystuls: Molding the Flow of Light. Princeton University Press, 2 edition, 2008. URL htttp://abiinttii. mit
edu//boo//
Steven G. Johnson and J. D. Joannopoulos. Block-iterative frequency-domain methods for Maxwell's equations in a planewave basis. Optics Express, 8(3): 190, January 2001. ISSN 4094-4087. doi: 00.1364/0E..000017 URL https:////// www. wwwp::/wwwwbllshh...... org/ oe/ ott.://abstracttttteeeeoe absttact
c

---

Under review as a conference paper at ICLR 2023
Armand Joulin and Francis Bach. A convex relaxation for weakly supervised classifiers, 2012, URL fettps://rxivz http://aaxxvv....aabbbb attg/bbs//20/ 6413
Diederik P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- awwiv wwt.//gg///.b.o.ooo..o..o 1106. 02557
zation trick, 2015.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, 2016. URL
https: www./rggg/.ooooo aaa//bbss11000 02242
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2016. URL
01474
Dong-hyun hyuu Lee. Pseudo-label: The simple and efficient comi-super learning method for deep neural networks, 2013.
Charlotte Loh, Thomas Christensen, Rumen Dangovski, Samuel Kim, and Marin Soljacie. Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science. Nat Commun, www.oo.............o............
13, 2022. URL https://doii http:///ooooogg////////
022 31915-y
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. CoRR, abs/2106.07998, 2021. URL
wttp:/ivivvrrg/a/saaaaaaaaaaa abspa 07998
Kevin P. Murphy. Probabilistic Machine Learning: An introduction. probml ai
MIT Press, 2022. URL
Rafael Müller, Simon Komblith, and Geoffrey Hinton. When does label smoothing help?, 2019. URL
attpss//aarxt http:///xxww.v...aaa/.. htrg//bb/19906 02629
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning, 2019. URL httpss://axii..orrrg
abs/1904. 01685)
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Zhongzheng Ren, Raymond A. Yeh, and Alexander G. Schwing. Not all unlabeled data are equal: Learning to weight data in semi-supervised learning, 2020. TRL//ttta////rxiv..ttm http://awxxx...../b//// 2007. 01293
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference Learning Representations, 2021. URL httpsss//oppeeeeeeeee
net//orum: -0
Chuck Rosenherg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'0)) Volume 1, volume 1, pp. 29-36, 2005. doi: 10.110/AAVVMOT.205.0077
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep wemi--uperrvsee learning. CoRR, (tbs//606.04586 2016. URL
http::/arxi org/ abw.aba//l/ 04586
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence, 2020. //tp:tttpss////xiioorrrr wwwp:/wwwwwwwww.////////oaae
07685
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016, URL / [oLl/atttpa:/prreeecingg http://pooceccii...e...eeeeeeeee pdf
a a9663a f 14 http://wwww............e.ppapppp 2/15// ffile/
12

---

Under review as a conference paper at ICLR 2023
Antti Tarvainen and Harri Valpola. Weight-averaged consistency targets improve semi-supervised deep learning results. CoRR, abs/1703.01180,, 2017. URL http::ttt:/attttt wwwpxxii.or//aaaa. 01780
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks, 2019. URL httpttts::/// //ppsi///rxiv. /22g///bb1990 11001
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML'1 11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 978 7781450306193
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. 2020. doi: 10445500/RRXIV2200206777 URL https://
arxiv.org/ abs 2wwaaa///2220... 06715
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and probabiliistic perspective of generalization. CoRR, abs/2002.08791, 2020. URL httpp::/aaxx...rrrraaa/////////
08791
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh- Minh-Thang Thang Luong, and Quoc V. Le. Unsupervised data augmentation. CoRR, abs/1904. 12848, 2019a. URL httt:::/aaxxv..rg///o 12848
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2019b. URL
1911. 04252
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding, 2021. URL. nttts::/arxiv..o
2bs/2109. 00650
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Anmual Meeting of the Association for Computational Linguistics. pp. 189-196, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics. doi: 10.3115/99, 1588.
981684. URL httpt:tt////. a wnt.oolooy..oooooooo wrg. 1026
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass prob- ability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, KDD '02, pp. 694-699, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X, doi: 10. 10.1145/775047.775555 URI.
https://doi.
11445775777777 775151
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, 2021.
URL http:httss///axxi..ooo http:///aaaxxi..oo///////1//1/ 08263
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez- Paz. mixup: Beyond empirical risk minimization, 2017. IRtptt:/s///aa.. org/ abs 17t.//9111.9112
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. Proceedings of the European conference computer vision (ECCV), pp. 289-305, 2018
13

---

Under review as a conference paper at ICLR 2023
A COMPARISON WITH PRIOR ART IN SSL
Table 5: Comparison of techniques used in BAM- with techniques used in prior art of SSL.
SSL method
Augmentation
Pseudo-label
Selection metric
Temporal ensemble aane
2016
Weak
model from earlier step
Mean teacher (Tarviienn Valpola 2017
Weak
EMA
UDA (Xic al.
2019%
2019
Weak & strong
Sharpen
Logit thresholding
MixMarch Berthelo ol al.
2019h
Weak
Averaging aug sharpen
FixMatch Sohn ct al. 2020
Weak & strong
Hard labels
Logit thresholding
FlexMatch Zhang et al 2021
Weak & strong
Hard labels
Class-wisc logit threshold
Our main method BAM-
Weak & strong
Posterior sampling sharpen Variance thresholding
B PROOF OF COROLLARY
PAC-Bayesian theory aims to provide bounds on the generalization risk under the assumption that samples are i.i.d. While PAC-Bayesian bounds typically apply to bounded losses, Germain al. (2016) extends them to unbounded losses (which is necessary in our work since it uses the unbounded log-loss). Their theoretical result reproduced below in Theorem 1] under assumptions for sub- Gaussian losses. These bounds commonly assume the supervised setting; in this proof, our goal is to extend them to the semi-supervised learning setting where in addition to labels from an empirical set, loss is also mediated by pseudo-labels assigned by the model.
Let (X,Y) ~ DN denote i.idd training samples from the data distribution D, where (X,Y) [1 ((I,,)))) ) {1x,, ):)) represents the set of N₁ labeled input-output pairs and N, pairs of unlabeled input and their model-assigned pseudo-labels ÿ. Here, N₁ + N,, = N. Let the loss function be ((f, xx 8) + R. the generalization risk on distribution D, i.e. (f), and the empirical risk on the training set, ie. (f), is given by:
C E {(f,z,v); (x-(f)= 1 N (a.y)D
Σ (ffrz,1))
1 No Σ (f....0)
N
On assumptions that the loss is sub-Gaussian (see Boucheron et J. (2013) section 2.3), i.e. a loss function is uub-auussian with variance s2 under a prior and D can be described by sub-Gaussian random variable = (f) {(f. x, v), the generalization error bounds are given
by Theorem Germain al. 2016 below.
Theorem (Corollary from Germain et al. (016 Let x be the prior distribution and p be the posterion. If the loss is ab--Gasssian with variance factor s², with probability least 1 over the choice of (X, Y) ~ N
E (Dffs E p fーP
(x)(())+ N (KL(p) + log(1/6)) =
Beginning from Theorem 1 the generalization error of the variational posterior q(fl)) under the negative log likelihood loss and prior p(0) in our SSL setting can be found by replacing with g(0)))
14

---

Under review as a conference paper at ICLR 2023
and l with e Fz. u4)
log logp(us(e;;0))
2 2
E
(f)
E
C (f)+
f~q
f~
N
Bgg
logr(ū;)2;;
log logp(vi
N
N
(KL(g((())ll(()) + log (1/6)) =
N
N
plye T. p(ú:l= 0)
N; log ((gp(ui)) +
Eg
log
log p(u 2,; 0)
N
(KT(q(0())]l((()))
log (1/5)) logg(10))
$²
N
No
log p(ue 10; mr; 0)
Bg
log (ogp(vs(=z;0)
+ (KL(((0)))Ip(0O)> +og((1/)) + #2
Nu
p(v 24: 0)
N
p(iiaan 0) log
1 (KL()(0)))lи))) + log(11/)) + 1 = +
Byg
E.,
log logp logp(Y(X:0))
N u
20 0)
N
log p( 0)
1 + N 1og(1/0)+ 1 2
E (D) s By f~g
ELBO]
N
a1 0)
where define ELBO Bi( [oog(( Appendix C.I. we will see that this the term we are maximizing in our loss objective. By maximizing the ELBO, ie. minimizing the negative ELBO, are minimizing the upper bound to the generalization error of our variational posterior. The first term the last line adds a divergence measure between the pseudo-label prediction distribution and the ground truth distribution in the fully supervised setting Dt + y: and this term vanishes.
KL(ç(0 | ()))) In the next section,
C FORMULATION DETAILS AND PSEUDOCODE OF OUR METHODS
C.I BAYESIAN MODEL AVERAGING VIA A BNN FINAL LAYER
Following the notations in Section 4.1 we denote the BNN layer be and an input embedding to this layer to be v in this section. We assume a prior distribution on weights P(Ox) and seek to calculate the posterior distribution of weights given the empirical/training data, P(0, [Dx).. where Dx == (X, Y). which can then be used to compute the posterior predictive during inference. As exact Bayesian inference intractable for neural networks, we adopt a variational approach following (Blundell et al. 2015) to approximate the posterior with Gaussian distribution parameterized by Φ, q(0 Φ)) From now, we will drop the h index 0, for brevity. To learn the variational approximation, we seek to minimize the Kullback-Leblle (KL) divergence between the Gaussian variational approximation and the posterior:
t = argmin y KL (g(0 Φ) p(0]Xx
argmin /
((01))log g(0|0) d8
p00 X,
argmin E (0 log {0) (Y
p(Y|X; p0))
argmin E [lgg(0) () logp(Y - logp(0)] + lYgp(Y|X))
=
argmin
(rgningKL(g( (g(0 ())) - lopp() (X:6) 6) + lYg
= argmin # ([-ELBO] + log X))
where in the last line, ELBO E,,), [Yogr(Y| 0) (L Φ) [lp())) is the evidence lower- bound which consists of likeliiol (data-dependent) term and KI. (prior-dependent) term.
15

---

Under review as a conference paper at ICLR 2023
Algorithm PyTorch-ttyle pseudocode for Bayesian model averaging in UDA or FixMatch.
# E: backbone encoder network
bayesian classifier
loss: XL term evidence lower-bound
A Н: cross-entropy loss
quantile parameter
samples:
weight samples
# method: 'UDA' or 'FM"
sharpen operation
threshold 1ist
[] for (x1, labels), Llb, uw, us x l_b, R_w,, 2_ a f(x lab), (x_uw), [x_us) # get representations
in cip(labeled loader, unlabeled loader):
(ugment(xx); weak augment (xu), strong augment (xu)
mean std uw = bayes baavesspredicttieeeeee thresheld list. (ostt)pp(0)) if (threshhold list)
x z_uw)
keep
thresholds
threshold list aapstt.pppdddeeee -quend (std (st
2))
uw. mask ut (thresshdl list. list.mean())
compute losses
loss kl Kt. loss # prior-dapendent (data-indapendent) loss loss lab H( (ab),, labels)
method
'UDA':
loss unlab = (((((z_ss)) shp (aean__ww) mask # sharpened soft pseudo-labels 'FM":
elif mathod
loss unlab H (z us), mean ww. www.rgmax (-1)) mask hard pseudo-labels
loss loss lab + loss_unles unlab + loss_kl kl
loss. bas.kaackarr(() optimizer.stepl.
def eryeeeppeeict (h, outputs stack ([h softmax
in range (num samples) sample weights return outputs mean(), outputs. mean and std of predictions
Since KL (g(0))) p())))) is intractable, maximize the ELBO which equivalent minimiz- ing the former up to the constant, lo..(
Each variational posterior parameter of the Gaussian distribution, ø. consists of the mean (p) and the standard deviation (which is parametrized as = log(1 + exp(p)) so that a is always positive (Blun- dell et 2015)) )) i.e. á = (p., p). To obtain a sample of the weights we use the reparametrization trick (Kingma et al. 2015) and sample ((O I) to get 0 μ + log(1 + exp(p)) a where other words, we double the number learnable parameters in the layer compared to a non-Bayesian approach; however, this does not add huge computational cost since only the last layer Bayesian and the dense backbone remains non- non-Bayesian.
denotes elementwise multiplication.
Algorithm 1] shows the Py Torch-style pseudo-code for BAM- in cireshold-medittee methods (here showing asymmetric augmentation applicable for UDA or FixMatch). The main modifications upon the baseline methods include 1) computation of additional KL loss term (between two Gaussians, ie. the variational approximation and the prior). 2) taking multiple samples of weights to derive predictions and 3) replacing the acceptance criteria from using maximum probability to using standard deviation of predictions.
C.2 ALGORITIM FOR SWA & EMA IN PAWS
Algorithm 2 shows the pseudocode for the implementation of PAWS+SWA and PAWS+EMA in Py Torch. For brevity, we leave out details of the multicrop strategy, mean entropy maximization regularization and soft nearest neighbour classifier formulation which are replicated from the original implementation. We defer readers to the original paper (Assran et al. 2021) for these details.
16

---

Under review as a conference paper at ICLR 2023
Algorithm 2 PyTorch-style pseudocode for PAWS-SWA and PAWS-EMA
# f: backbone encoder network
weight aggregated encodar network
shp: sharpen operation
# snn: PANS soft nearest neighbour classifier
entropy loss
max: mean entropy maximization regularization loss
# N: total number of epocha
boolean
SMA
use ama: boolean
# swa wwa_epochs: number of epochs before switching on SWA gamma: momentum parameter for EMA
5-params f. f.params $ initialized as copy
S.paras.rrquiree. grad = False remove gradient computatioon
swa
for in range (N)
for in loader:
augment (x), augment augmentations
pl, p2 snn (x1)), enn (xx2))
ql, snn (xi)), snn (x22)
loss (pl, shp (42)) /2 (p2, shp /2 /2 + max (cat (2), loss. backward optimizer.step steep())))
if use swa: num :pochh:::: update aggregate
papaaass (g-params num f.params) (num_sva 1)
else:
g-params params weights just copied
olff use ema:
g. gamma params (1-gamma) f.params update momentum aggregate
g params params PAMS baseline
D FURTHER IMPLEMENTATION DETAILS
D.1 HYPERPARAMETERS FOR VARIOUS THRESHOLD MEDDIATED METHODS
All preshold-mediatee methods this study uses an optimization loss function of the form of Eq. (1)
with differences in the hyperparameters μ4, X, Pt. and We use the hyperparameters from the original implementations. For Pseudo-Labels (Lee) 2013), 1 1, = 0.95 and Pr=o (i.e. hard labels); for UDA (Xie et al. 2019a), / temperature 0.4); for FixMatch (Sohn et al. 2020). = 7, T 0.95, Pt=o (i.e. hard labels). All methods À = 1. In addition, UDA and FixMatch uses asymmetric transforms for the two legs of sample and pseudo-label prediction, i.e. on is weak transform (based on the standard flip-and-shift augmentation) and Or is strong transform (based on RandAugment (Cubuk al. ). Pscudo-Labels uses symmetric weak transforms for both legs.
0.8, Pt=0.4 (i.e. soft pseudo- sharpened with
In our calibrated versions of all these methods (i.e. "BAM-X"), we maintained the exact same hyperparameter configuration as its corresponding baseline. The only exception is the sharpening temperature of BAM-UDA which uses 0.9 instead of 0.4 in UDA, as we found that calibration enables, and was highly effective with, the use of soft pseudo-labels (see discussion Section 6).
D.2 IMPLEMENTATION DETAILS FOR IMAGENET EXPERIMENTS
We maintain the default 64 GPU training configuration recommended by the authors and make slight modifications the default implementations fit on our hardware. On our set up (which does use Nvidia's 's Apex package due to installations issues), we found training to be unstable and had to use full-precision training. In order to into memory, we had to decrease the number of images per class from 7 to 6, resulting in slightly lower baseline performance from the reported for 200 epochs training (see Table 4 in Assran et al. (202T) for the study on the correlation between the number of images per class and the final accuracy). On all ImageNet experiments
half-precision
PAWS, follow the validation and testing procedure of PAWS (Assran et al., [022] and swept over the same set of hyperparameters during fine-tuning of the linear head. We report the ECE at the final checkpoint.
17

---

Under review as a conference paper at ICLR 2023
E LONG-TAILED CIFAR-10 AND CIFAR-100
E. DATASET PREPARATION
We create long-tailed versions of CIFAR-10 10 and CIFAR-100 following the procedure from Cao et al.) (2019), i.e by removing the number of training examples per class from the standard training set with 50,000 samples. We create the class-imbalance unlabeled dataset with an exponential decay where the severity of the imbalance is given by the imbalance ratio max, (rea,, (H.,,)) mi, (Rw,a) E {10, 100), where T the number of unlabeled examples for class The number of samples the most frequent class is 5,000 for CIFAR-10 10 and 500 for CIFAR- 100. To create the labeled set, randomly select 10% of samples from each class, under the constrain that at least sample for each class is included in the labeled set, L.c. Ry,; min(1, 0.1 i.,,,,,), where 7u,, is the number of labeled examples for class i. The total number of labeled and unlabeled examples for the different benchmarks in CIFAR- 10-LT and CIFAR-100-LE are summarized in (Table 6 The test remains unchanged, i.e. we use the original (class-balanced) test set of the CIFAR datasets with 10,000 samples.
Table 6: Dataset statistics for CIFAR-10-LT & CIFAR-100-LT showing total number of examples in the labeled and unlabeled datasets for each benchmark.
CIFAR-IO-LT
CIFAR-100-LT
Q 10
100
a 100
Labeled
2,041
1,236
1,911
1,051
Total
20,431
12,406
19,573
10,847
We used the exact same configuration and hyperparameters as the original (non-long-tailed) CIBAR benchmarks, i.e. the architecture is WideRessNet-28-2 for CIFAR- -A-RR-O-LT and WideResNee-22--10 for CIFAR-IO0-LT. All training hyperparameters used to obtain the results for FM and BAM-UDA in Table also follow the ones from the original CIFAR benchmarks.
E.2 ADDITIONAL CLASSWISE RESULTS
shows the test accuracies for the baseline (FM) and ours (BAM-UDA) after separating samples from the test set into three groups - the "Many" group which contains classes with more than 100 samples, the "medium" group which contains classes between 10 and 100 samples and the "few" group which contains classes less than 10 samples, for the benchmark of CIFAR- 100-LT with 10 and 10% labels. We see that BAM-UDA outperforms FM on every group; in addition, the gap between BAM-UDA and FM increases as the number of samples are more scarce, highlighting BAM-UDA's utility in improving accuracy over the baseline in the more difficult classes.
Fig. 5
18

---

Under review as a conference paper at ICLR 2023
Test Accuracy (%)
100
BAM-UDA: many
medium
UDA:
FM: many
FM: medium
FM: Tew)
herations
Figure 5: CIFARI00-LT. Accuracies for samples based on their their classwise sample frequency, "many" for classes with >100 samples, "medium" " for classes between 10-100 samples and "few" for classes between <10 samples. Dataset is CIFAR-100-LT with 10 and 10% labels.
F SEMI-SUPERVISED LEARNING IN PHOTONICS SCIENCE
Semi-supervised learning is highly important and applicable to domains like Science where labeled data is particularly scarce, owing the need for computationally expensive simulations and labor- intensive laboratory experiments for data collection. To demonstrate the effectiveness of our proposed techniques in the Science domain,
example problem in Photonics, adopting the datasets from Loh et al. (2022). The task we studied in this work is a 5-way classification of photonic crystals (PhC) based their band gap sizes.
PhCs are perioddically-strucuurrd materials engineered for wide ranging applications by manipulating light waves (Joannopoulos al., 2008) and important property of these crystals is the size of their band gap (often, engineers seek to design photonic crystals that host a substantial band gap (Christensen et al., (2020)) We adopt the dataset of PhCs from Loh al. (0022), which consists of 32,000 PhC samples and their corresponding band gaps which had been pre-computed through numerical simulations (Johnson Joannopoulos 2001). Examples of PhCs and an illustration of band gap from the dataset is shown in Fig. 6 We binned all samples in the dataset into 5 classes based on their band gap sizes; since there was a preponderance (about 25,000) of samples without band gap, we only selected 5,000 of them in order to limit the severity of class imbalance and form new dataset with just 11,375 samples (see Fig. 6b). Notably the long-tailed distribution of this dataset characteristic of many problems in science (and other real- world datasets), where samples with the desired propertics (larger band gap) are much rarer than trivial samples. From this reduced dataset, created PhC benchmarks with 10% labels (PhC-10%) and 1% labels (PhC-1%) where 10% and 1% of samples from each class are randomly selected to form the labeled set respectively. The baaanedl test set fixed with 300 samples per class (total of 1,500 samples), the unlabeled set consists of 9,876 samples and the labeled sets consists of 1,136 samples and 1 113 samples for PhC-10% and PhC-1% 1% respectively.
For these benchmarks, we used a WideResNet-28-2 architecture, with a single channel for the first CNN layer, and made the following changes upon FixMatch and BAM-UDA. We set μ 1 and instead of 220 iterations, we trained for 300 epochs (where each epoch is defined as iterating through the unlabeled set once). For BAM-UDA, we set Q 0.9 and use a one-minus-cosine warm-up scheduler of 50 epochs to the KL loss coefficient (resulting in a coefficient going from 0 to 50 epochs). For each benchmark, we swept the learning rate across {0.01, 0.001} and select the best model for both the baseline and ours. The standard image augmentations used in vision problems cannot be applied to this problem, since destroys the scientific integrity of the data (e.g. cropping the PhC input would result in completely different band gap profile). Instead, used the augmentations proposed in Loh et al. 2022) (periodic translations, rotations and mirrors) and applied them symmetrically (i.e. there is no distinction of strong and weak augmentations). In addition, included these augmentations to the labeled samples as well.
19

---

Under review as a conference paper at ICLR 2023
(a) Examples of 2D Photonic crystals
(c) Dataset statistics showing the class boundaries
S
class D: bg - 01% (5000)
class 1:0 % < bg 5% (3R22)
5000
bg
(1343)
class 3: 12% bg < 20% (701)
samples
class 4: 20% (509)
(b) Illustration of band gap for sample
4000
Total samples
3000
Densty of states
band
1000
Frequincy (AA))
"too of band gap (measured in
0
0
10
20
30
40
Bandgap (96
Figure 6: Photonics dataset. (a) Examples of 2D periodic photonic crystals (PhC), (b) illustration of the band gap size of single PhC seen from its density-of oftit---o--tttess... a common spectrum of interest for PhCs (Loh et al., 2022), and (c) dataset statisties for the task used in this work where the samples were binned into 5 classes based their band gap (bg) with class boundaries detailed in the inset. Numbers in parenthesis show the number samples each class, showing strong class imbalance.
G ABLATION STUDIES
G.1 WEIGHT AVERAGING THRESHOLL--EDDATTEO METHODS
For our experiments incorporating EMA during pseudo-labeling for FixMatch, we use the second momentum scheduler defined in Section 5 i.e. a one-minus-cosing scheduler starting (1 x) and decreasing to O, resulting in x range for y. While PAWS is effective with high y values (i.e. x 0.95), we found that FixMatch needed much smaller momentum values at the start of training and set 0.25. Note that in the default implementation of FixMatch, an EMA of constant experiments,
0.999 is also used, however it is used only for test evaluation and not during training. In our
the EMA weight aggregate for pseudo-labeling throughout training and the same weight aggregate after training for test evaluation. Results are shown in Table 7] where we see that using weight averaging also effective improving calibration in FixMatch and can lead to improved test accuracies. However, EMA weight averaging was not so effective when the labels are less scarce and further, consistently under-performs the Bayesian model averaging approach.
Table 7: FM+EMA compared against the baseline FixMatch (FM) well the Bayesian model average approach on FixMatch (BAM-FM).
CIFAR
CIFAR-100
40
250
400
4000
FM
91.7
0.078
94.2
0.051
56.4
0.366
74.2 / 0.183
BAM-FM
93.6 / 0.058
95.0 / 0.044
59.0 / 0.331
74.8 / 0.171
FM+EMA
93.5 0.060
94.8 0.047
56.8
0.350
74.0 / 0.188
The desirability of Bayesian model averaging over EMA weight averaging for fhrehhld-mediiee SSL methods could be attributed to replacing the selection metric with better uncertainty estimates given by the BNN layer, which is not available for EMA weight averaging approaches. To further investigate this claim, we perform ablation studies on BAM-UDA to isolate the contribution of averaging predictions from the contribution replacing the selection metric. Results are shown in Table 8 rows indicated with "BNN no 02 show experiments using BNN layer in BAM-UDA only for averaging predictions while maintaining the original selection metric, i.e. pseudo-labes labels are accepted if the maximum prediction class probability is greater than 0.95. Comparing the first two rows, indeed we see that by not using the uncertainty estimates from the BNN, we get marginal improvements especially when labels are not scarce (72.9 73.0 for CIFAR-100 4000 labels),
20

---

Under review as a conference paper at ICLR 2023
corroborating observations with FM+EMA. The difference between the last two rows show the effect of replacing the selection metric and indeed we observe consistent gains across the benchmarks from doing
Table 8: Uncertainty estimate by BNN. Ablating the importance of uncertainty estimate provided by the variance of BNN predictions. "BNN no 0² indicates that the BNN layer is only used for bayesian model averaging, i.e. predictions are replaced by the posterior predictive but selection metric still follows the baseline, i.e. pseudolabels are accepted maximum logit value > 0.95.
CIFAR-100
400
4000
UDA (I=0.4)
44.0
0.491
72.9
0.185
BAM-UDA, BNN no o² (t=0.4, r=0.95)
48.3
0.418
73.0
0.184
BAM-UDA, BNN no o² (t=0.9, 7=0.95)
54.2
0.368
74.5
0.170
BAM-UDA (1-0.9)
59.7
0.327
75.3
0.167
G.2 ABLATION EXPERIMENTS ON BAYESIAN MODEL AVERAGING
From our main results Section 6 we found that our Bayesian model averaging approaches were more effective in conjunction with soft pscudo-labels. In Table 9 we show ablation experiments on the lemperature of the sharpening operation on the CIFAR- 100-400 labels benchmark. Overall, we observe trend that softer pseudo-labels (i.e. reducing the sharpening of pseudo-labels) labels) led to better calibration and improved test performance. As such, in our experiments we modify upon the original sharpening parameter of UDA and set 0.9 for BAM-UDA in all our benchmarks.
Table 9: Ablation of sharpening temperature in BAM-UDA, Dataset is CIFAR-100 with 400 labels. Highlighted in cyan is the main configuration used.
Test Accuracy
ECE
0.4
57.9
0.344
0.8
58.1
0.340
0.9
59.7
0.327
1.0
59.2
0.334
Bayesian model averaging produces better calibrated predictions through "Bayesian marginaliza- tion", i.e. by averaging over multiple predictions instead of using single prediction of the model. Table 10, we show ablation studies on the number of weight samples taken from the variational posterior in the BNN layer used for "Bayesian marginalization", i.e. for computing the posterior predictive. We observe overall trend that increasing the number of weight samples lead to better calibration and final test accuracies, providing direct evidence for the effectiveness in using Bayesian model averaging towards improving calibration. In order to limit the computational overhead arising from taking a large number weight samples, given that pscudo-labeling happens at every iteration, we limit M to 50 in study. The computational overhead for M 50 is discussed in Appendix H where we see that incorporating Bayesian model averaging incurs negligible computational overhead.
Table 10: Ablation of number of weight samples, M, taken from the variational posterior in BAM- UDA. Dataset is CIFAR-100 with 400 labels. Highlighted in cyan is the main configuration used.
M
Test Accuracy
ECE
2
50.0
0.403
58.1
0.342
10
58.5
0.336
50
59.7
0.327
21

---

Under review as a conference paper at ICLR 2023
G.3 ABLATION EXPERIMENTS ON PAWS-SW. SWA & PAWS-EMA
Training times before switching on SWA. Fig. 7 shows the test accuracy and ECE across training for PAWS-SWA and PAWS on the CIFAR-J0 10 benchmark with 4000 labels and explores the effect of varying the time in which SWA was switched on. In particular, we found that switching on SWA later during training more desirable. Unlike EMA which has natural curriculum to give more weight to the more recent network parameters in the aggregate, SWA weighs all models in the aggregate equally and thus including the poorly optimized weights from the start of training is undesirable.
Test Acc. (%)
ECE
97
0.12
SWA EP:10
95
0.11
EP:50
EP:100
94
0.10
SWA EP:200
93
0.09
baseline
92
0.08
90
SWA EP:10
0.07
SWA EP:50
EP:100
EP:200
baseline
100
200
300
400
500
600
100
200
300
400
500
600
Epochs
Epochs
Figure 7: PAWS-SWA for different training times before switching on SWA. We experimented with switching on SWA after {10, 50, 200) epochs of training (of the baseline). Dataset is CIFAR-10 with 4000 labels.
Momentum schedules for PAWS-EMA. Table Table 11 explores different schedules for the parameter used in PAWS-EMA the CIFAR-100, 4000 labels benchmark. For brevity, we consider a simple schedule where y is linearly warmed up from 0 to Ymaa in the first W epochs, and maintains Ymaa thereafter. In Table 11 we varied W {0, 10, and "maz E {0.5, 0.9, 0.99, 0.999). We observe that using very high 7 (i.e. 0.999) from the start of training is detrimental to both test performance and ECE, while fixing y to a moderately high value (i.e. 0.9) across training was also less desirable than stepping up ry in the first 50 epochs. Notably, these observations highlight distinct differences with the EMA commonly used in dual-network methods of self-supervised learning (Grill et al. [2020b] He et al. 2020) where very high momentum value (0.996 to 0.999) is usually recommended.
With this insight, in this work, we explored two schedules for 1) linear warm-up from 0 to 0.996 for 50 epochs and maintaining at 0.996 (highlighted in cyan in Table IT} and 2) using one-minus-cosim schedule starting from 0.05 and decreasing to O, resulting in 0.95 to 1.0 range for y. These two schedules are visualized in Fig. 8 We use (1) for the CIFAR benchmarks and we found (2) to be more effective the ImageNet benchmark.
1.0
0.8
0.6
0.4
0.2
warmup=10, const@0.996 cos_schedule 0.95 to 1.0
0.0
50
100
150
200
Epochs
Figure 8: PAWS-EMA: Visualization of the momentum schedules used in this work.
22

---

Under review as a conference paper at ICLR 2023
Table 11:
Ablation on momentum schedule. Dataset is CIFAR-100 with 4000 labels. Highlighted in eyan is the main configuration used for the CIFAR benchmarks.
warm-up epochs
Ymmaa
Test Accuracy
ECE
0
0.5
72.1
0,215
0
0.9
72.4
0.218
0
0.99
72.0
0.208
0
0.999
59.2
0.266
10
0.99
72.6
0.198
10
0.999
68.5
0,214
50
0.99
73.2
0.193
50
0.999
73.3
0,200
50
0.996
73.6
0.189
H COMPUTATIONAL REQUIREMENTS AND ADDITIONAL COMPUTATIONAL COSTS
All CIFAR-10 10 and CIFAR-100 experiments this work were computed using single Nvidia V100 GPU. ImageNet experiments were computed using 64 Nvidia V100 GPUs. A key aspect of our proposed calibration methods the requirement adding minimal computational cost to the baseline approaches. In the following paragraphs we list the additional computational cost (based on wall-clock time on the same hardware) for our proposed methods.
Computational cost of Bayesian model averaging. The main additional computational cost comes from executing several (in our case, 50) forward passes through the weight samples of the BNN layer when deriving predictions for the unlabeled samples. This additional overhead is minimal since only the final layer, which consists of a small fraction of the total network weights, (approximate) Bayesian. We tested this on the benchmarks of CIFAR-100 and found the BNN versions to take only around 2 - 5% longer in wall-clock time on the same hardware when compared to the baseline, for the same total number of iterations. This justifies the BNN layer plug-in calibration approach with low computational overhead.
Computational cost of SWA & EMA. Both SWA and EMA requires storing another network of the same architecture, denoted in the main text, which maintains the aggregate (exponentially weighted aggregate) of past network weights for SWA (EMA). Rather than using representations from the original backbone, f. representations from g are used to derive the better calibrated predictions, and thus the main computational overhead comes from the second forward pass needed per iteration. We timed the methods and found that PAWS+SWA and PAWS+EMA took around 18% longer in wall clock training time (on the same hardware) when compared the baseline PAWS method for each epoch of training. However, as shown in the main text, PAWS+SWA and PAW+EMA resulted in significant speed up in convergence, cutting training time by > 60% and additionally giving better test performances. Hence rather than an overhead, improved calibration in PAWS+SWA and PAWS+EMA in fact reduces the computation cost needed for the original approach.
I SOCIETAL IMPACT AND ETHICAL CONSIDERATIONS
sumi-supervised learning arguably one of the most important deep learning applications, as real life we often have access to an abundance of unlabeled examples, and only a few labeled datapoints. Our work highlights the importance calibration in semi-supervised learning methods which could yield benefits for real-world applications in two main ways: 1) better deep learning model that is more data efficient and 2) a model that is better calibrated and can quantify uncertainty better. The latter is highly important for crucial societal applications such as in healthcare. However, with any deep learning application, there may be biases accumulated during dataset collection or assimilated during the training process. This issue may be more acute in sepi-supe setting where the
23

---

Under review as a conference paper at ICLR 2023
small fraction of labels may highly misrepresent the ground truth data. This may lead to unfair and unjust model predictions and give rise to ethical concerns when used for societal applications.
24


Reviews:
[{"review_id": "UZG42p3-V-w", "reviewer": "Reviewer_ZkiF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper centers around one problem of improving calibration under the scenario of semi-supervised (deep) learning. Towards this end it proposed two methods: the bayesian model averaging based and the weight averaging based. Empirical verifications are conducted on several tasks which demonstrate the improvement both in terms of test set accuracy and the calibration (measured by ECE).\n\nIt also points out the correlation between calibration and accuracy as a side outcome.", "summary_of_the_review": "As stated above, this paper is very well motivated, technically sound and written in a very clear way. Meanwhile its major limitation is on its marginal technical innovation since it is a work connecting the \"raw\" methods to a specific ML application scenario. ", "strength_and_weaknesses": "**Strength**\n\n1. The problem this paper considers is important: improving calibration under SSL. Calibration is indeed a very important metric in many real-world industry settings (like computational advertising) while is a comparatively neglected thing in deep learning research, let alone for the specific setting of semi-supervised learning. It's great that the authors could dive deeply into this important problem which might benefit the success of (semi-supervised) deep learning method in real-world scenarios. \n\n2. This paper is stated in a very clear way, and its technical soundness is good. I enjoy reading the whole contents which give readers a good background introduction and \"logical flow\".\n\n3. The empirical evaluation looks comprehensive and sound.\n\n**Weakness**\n\n1. The technical innovations seems limited: it seems strait-forward to directly adopt the two methods (i.e., bayesian based and weights averaging based) to the scenario of SSL. There is no specific innovations here given no unique challenges (not in terms of theoretical but in terms of algorithmic) when we want to combine the two worlds, that is previous \"basic\" methods already established and SSL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N3TJY0zhyi", "reviewer": "Reviewer_x2Kw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is motivated by the poor model calibration which affects semi-supervised learning. Bayesian network and weight-averaging techniques are adopted to improve model calibration and it has been demonstrated to improve semi-supervised learning efficacy.\n", "summary_of_the_review": "Overall, this paper addressed an important problem, e.g. poor model calibration affects SSL performance. However, some components, e.g. using the quantile to estimate threshold, of the proposed method might give unfair advantages over the existing method. The improvement from existing methods is also marginal, thus hard to quantify the significance.\n", "strength_and_weaknesses": "Strength:\n\n1. Deep learning model are poorly calibrated. This could affect semi-supervised learning and this work addressed this important problem.\n\n2. Demonstrating a strong correlation between SSL performance and model calibration.\n\nWeakness:\n\n1. The Bayesian network only considers the last layer. In particular, for the pseudo label approach only the classifier head is sampled multiple times. Does this really reveal the uncertainty of a network? This is equivalent to perturbing the decision boundary, the samples that are close to decision boundary are more likely to yield high variance and will be pruned out for self-training. In another words, this can be seen as another way of thresholding.\n\n2. The improvement from FixMatch is very marginal. Does this suggest BAM is another way of doing thresholding.\n\n3. The threshold on Bayesian prediction exploits the distribution of standard deviation on unlabeled data. In contrast, FixMatch does not exploit this information. Thus, it is vital to verify whether the improvement is attributed to a more carefully chosen threshold.\n\n4. EMA parameter update has been studied in existing SSL methods, e.g. mean teacher.\n\n5. It has been reported that confirmation bias is more severe at low label regime. If improving model calibration is effective, it is recommended to evaluate at even lower label regime, e.g. CIFAR-10 @ 40 labeled.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ldwj_GrmYkK", "reviewer": "Reviewer_bpxv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper leverages approximate Bayesian techniques, such as approximate Bayesian neural networks to quantify the uncertainty and improve the performance on the uncalibrated models. This paper is mainly a summarization of recently introduced techniques in SSL and the\ncontribution in terms of exploited techniques being commonly used in the SSL framework", "summary_of_the_review": "The contributions from this paper are not significant and many of the discussed topics and techniques for SSL are well explored in the literature. In the experiments, the variance of different runs with different random seeds needs to be reported.", "strength_and_weaknesses": "Strengths:\nThe paper is well-written and clear.\n\nWeaknesses:\nThe main weaknesses rest on the novelty of the introduced techniques.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CmYLxU_W4m", "reviewer": "Reviewer_11ZK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to improve the pseudo-label in semi-supervised learning with a well-calibrated model. To that end, it improves the calibration with Bayesian model averaging. It also provides a theoretical result for the benefit of the calibration in semi-supervised learning for the task of classification. The approach is evaluated on CIFAR-10, CIFAR-100 and ImageNet where it clearly demonstrates the advantage of maintaining a calibrated classifier. \n", "summary_of_the_review": "The paper shows strong evidence of why the model calibration can improve the performance in semi-supervised learning. It is well-written, with complete related work and a detailed experimental section. Overall, it has potential to be accepted, but there are still some parts to be further worked.\n\n\nPost-rebuttal comments:\n\nThe rebuttal discusses all open questions from the reviews. Regarding my concerns: \n\n- The argumentation that is not possible to compare with another approach does not help the paper. It is necessary to provide more references of comparison for understanding the practical contribution of the paper. \n\n- The lack of novelty is mentioned in another review too. As mentioned, I find the combination a solid contribution but not enough for acceptance. A strong evaluation would add more contribution to the paper (above point).\n\nFor these reasons, I will lower my score. The paper needs additional work at the current stage. ", "strength_and_weaknesses": "Strength:\n\n- The paper is well-written and easy to follow. The presented contribution is clearly presented and thus it is compared with the prior work. \n\n- The presented observations can have an impact on a wide range of semi-supervised learning tasks. Moreover, the theoretical result further supports the paper.\n\n- The results are convincing in the three evaluations. \n\n- The paper presents several ablation studies to support why improving the classifier calibration would improve the performance.\n\n\nWeaknesses: \n\n- It would be interesting to examine more calibration approaches. It is interesting to know if only the presented calibration approach has a positive impact on semi-supervised learning or in general any calibration approach would improve it. Making a more general claim would add significant value to the paper. \n\n- The approach is applied only for classification. Similarly, it could have been applied to the classifier of an object detector. \n\n- Although the paper makes interesting observations, it does not show new elements. For instance, the EMA update is known, as well as the rest parts of the presented methodology. However, the combination of the calibration with semi-supervised learning is a solid contribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject