A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC 
Authors: Masoumeh Javanbakhat
Christoph Lippert
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this paper we present a practical Bayesian formulation for self-supervised learning method with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC). Within this framework, we place a prior over the parameters of a self-supervised learning model and use cSGHMC to approximate the high dimensional and multimodal posterior distribution over the embeddings. By exploring an expressive posterior over the embeddings, the Bayesian self-supervised learning produces interpretable and diverse representations. Marginalising over these representations results improvement in semi-supervised learning and out-of-distribution detection tasks. We provide experimental results on multiple classification tasks in semi-supervised learning including Cifar10 and Cifar100. Moreover we demonstrate the effectiveness of the proposed method in out-of distribution detection task using SVHN dataset.

Under review as a conference paper at ICLR 2023
A PROBABILISTIC APPROACH TO SELF-SUPERVISED LEARNING USING CYCLICAL STOCHASTIC GRADIENT MCMC
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper we present a practical Bayesian self-supervised learning method with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (eSGHMC). Within this framework, we place a prior over the parameters of a self-supervised learn- ing model and use cSGHMC to approximate the high dimensional and multi- modal posterior distribution over the embeddings. By exploring expressive posterior over the embeddings, Bayesian self-supervised learning produces in- terpretable and diverse representations. Marginalizing over these representations yields an improvement in performance, calibration and out-of-distribution detec- tion in downstream task. We provide experimental results multiple classifi- cation tasks on five challenging datasets. Moreover, we demonstrate the effec- tiveness of the proposed method in out-of-distriutioon detection using the SVHN dataset.
INTRODUCTION
Self-supervised learning is a learning strategy where the data themselves provide the labels (Jing & Tian (2020)). The aim of sulf-suue learning is to learn useful representations of the input data without relying on human annotations (Zbontar et al. (202I)). Since they do not rely on annotated data, they have been used an essential step many areas such as natural language processing, computer vision and biomedicine(Uospin et al, (2020)).
Contrastive methods (Chen et al. (2020) are one of the promising self-supervised learning ap- proaches which learn representations by maximizing the similarity between embeddings obtained from different distorted versions an image (Zbontar et al. (202I)). Several tricks are proposed to overcome the issue of feature collapse. These include using negative samples in simCLR (Zbontar et al. (02T)) and stop gradient in BYOL (Grill et al. (2020)).
Self Self-supervised models often trained using stochastic optimization methods which approximate the distribution over the parameters with point mass ignoring the uncertainty in the parameter space. Indeed if the regularizer imposed the model parameters is viewed as the the log of prior on the distribution of the parameters, optimizing the cost function may be viewed as a maximum a-posteriori (MAP) estimate of model parameters (Li et al. (2016)). Bayesian methods provide principled alternatives that model the whole posterior over the parameters and account for model uncertainty in the parameter space (Zhang al. (2020)).
Exact Bayesian learning of Deep Neural Networks is generally intractable, hence Bayesian deep learning models use approximation methods like variational inference (VI, Blundell al. (2015) or MCMC methods (Neal (2012)) to capture the posterior over the parameters and estimate model uncertainty. While VI methods usually approximate single mode, MCMC methods are used for sampling from different modes (Jospin et al. (2020). In a recent line of work, Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods (Welling & Teh (2011). Chen et al. (2014), Ma al. (2015)) were proposed which couple MCMC with SGD to provide promising sampling approach to inference in Bayesian deep learning for large datasets (Welling & Teh (201T)). In another work, Zhang al. (2000) proposed Cyclical Stochastic Gradient MCMC (cSG-MCMC) in order to explore a highly multimodal parameter space given a realistic computational budget (Zhang
et al. (2020)).

---

Under review as a conference paper at ICLR 2023
In this paper aim adapt Bayesian supervised learning concepts to self- self-supervised learning to make a self-suuerrvsed learning model fully probabilistic using cSG-MCMC. Our motivation comes from the fact that the posterior distribution over the parameters of a self-supervised learning model may be multimodal and thus insufficiently represented by single point estimate. By exploring the posterior distribution
the parameters instead of point mass aim to improve performance in downstream tasks. Moreover, the optimization step in cSG-MCMC involves injection of Gaussian noise the parameter update of SGD which helps alleviate the feature collapse issue in the contrastive methods and make the features more informative (Li et al. (2016)).
In this paper, we propose a simple Bayesian formulation for self supervised learning with a specific family of cSG-MCMC methods called Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cS- GHMC) (Zhang al. 2020)). Within this framework, we use BYOL as a self-supervised learning model which allows to incorporate Bayesian learning of an approximate posterior over the param- eters instead of MAP. Our experimental results indicate that by integrating a Bayesian learning we can achieve better performance in downstream tasks including classification and out-of-distribution detection. The simplicity of the proposed approach is one of its greatest strengths.
PROBLEM STATEMENT
Given a dataset D, a sulf-supe learning model Fo parameterized by 6, aims to produce a rep- resentation Ze by solving a predefined proxy task. In this paper wish to learn distribution over the emheddings Zo by placing a prior over the parameters 0 and using Bayesian learning instead of MAP estimation. For learning the representations we use BYOL, recent self-supervised learn- ing method based on contrastive learning. To obtain the distribution over the embeddings, we use eSGHMC. In the following, first we describe the self-supervised learning model to learn represen- tations. Then, we describe cSGHMC and highlight how it allows to obtain distribution over the embeddings.
2.1 SELF SUPERVISED LEARNING
The aim of contrastive learning is to learn representations by contrasting two augmented views of an image. Particularly B BYOL learns representations by reducing a contrastive loss between two neural networks referred as online network Fe (parameterized by 0) and target network Fe (pa- rameterized by {). Each network consists of three components, an encoder ʃ() (e.g., Resnet-18), projection head (.) (e.g., an MLP) and a prediction head q(.) (e.g., an MLP). For a given mini- batch X = { sampled from a dataset D it produces two distorted views, t(X) and f(X), via distribution of data augmentations T. The two batches of distorted views then are fed to the online network and the target network, producing batches of embeddings, Zo and Zç. respectively. These features are then transformed with the projection heads into Y, and Yg- The online network then outputs prediction Qe of Y& using the prediction head (-). Finally the following mean squared error between the normalized predictions Qo and target projections Ye is defined:
Co.g = 11Qe - Yell- = 2 2. (Qo. Ye)
1Q01--112
Čo.s computed by separately feeding t'(X) the online network Fg and t(X) the target network Fg. Then, at each training step, stochastic optimization step is performed to minimize
CBYOL Ce.g + Lo.s
The gradient is taken only with respect A. So, the parameter update is as follows:
0 f- optimizer((.. V Ve CYOL).
(3)
{ + 7 7),
where the weights& & an exponential moving average of the online network's parameters 0 with a target decay rate 7 E [o, 1]. At the end of training, the encoder fo(.) is used for the downstream task. During training only the parameters 0 of the online network Fg are updated.
2

---

Under review as a conference paper at ICLR 2023
2.2 POSTERIOR SAMPLING USING CSGHMC
In the Bayesian paradigm, for a given dataset D = { and a 0-parameterized model, the following a-posterior distribution over 0 is computed using Bayes' rule as: p(O p(D(0)p(0). where p(0) is prior assigned to the parameters 0 and p(D(0) is the likelihood.
In MAP optimization, the prior has the role of regularizer and the likelihood has the role of cost function. An optimizer is optimized to find the MAP solution which is amenable to the parameter update:
l AO 2 Σ Vo logp(a;;0) Ve log (()),
for a given randomly sampled mini-bated X f D and learning rate
In contrast to MAP optimization, in the Bayesian paradigm the model explores the distribution over the model parameters. |Welling & Teh (201T) showed that this distribution can be approximated using Stochastic Gradient Langevin Dynamics (SGLD) by injecting Gaussian noise to the parameter updates of SGD so that they do not collapse to just the MAP solution. This leads to the following parameter update:
E
AA
Ve logp(x;;00) + V Volog((0)) + ((0
(5)
2
SGHMC (Chen al. (2014) improved counterpart of SGLD which introduces momentum variable m. The posterior sampling is done using the following update rule:
N Vo log p(z; 0) + Vo logg((()) + cx B)er)
n
m =Bm
(6)
2
N
= 0 + m
where 3 the momentum term. Equations and 6 guarantee convergence to true posterior as long the learning rate f decreases towards zero. Zhang et al. (2020) showed that replacing the traditional decreasing learning rate schedule in SGHMC with a cyclical variant allows explore multimodal posterior distributions and developed cSGHMC. In this paper apply cSGHMC to take samples from the posterior distribution.
3 POSTERIOR OVER REPRESENTATIONS
To infer a posterior over the embeddings, we place a prior a over the parameters of the online network Fo. By placing distribution over induce distribution networks Fg. This results in a distribution over embeddings Za. Sampling from this distribution corresponds to sampling from the following conditional posterior:
infinite space of online
7 x X) p(X)))p(0)o).
where X is a mini-batch. Equation 7 can be interpreted intuitively as follows. We sample weights 0 from the prior p(O)ar). Then condition on this sample of the weights form particular online network Fe which is used to produce embedding Zg by minimizing the loss function C
Byol
To take samples from the posterior in equation we use cSGHMC and the update rule in equation 6 Algorithm describes our proposed method to sample from the posterior. Algorithm produces samples from the posterior over the parameters of the online network Fo and consequently distribution over embeddings Ze. since compute the gradients of the loss with respect to the parameters that we are sampling.
Our proposed probabilistic approach is natural Bayesian generalization of MAP optimization. Indeed, if one performs MAP optimization using SGD in Algorithm | insteed of posterior sampling, approximates the whole posterior over 0 with a point mass. Sampling from the whole posterior over embeddings versus approximating this with a point mass allows estimating uncertainties the embedding space.
3

---

Under review as a conference paper at ICLR 2023
Algorithm Probabilistic Self-Supervised Learning
Require: l initial learning rate, momentum term, number of training iterations
Ensure: sequence 0',0² for
K do
sample mini-batch X = f and augmentations t T,,' ~
compute CBYOL. based on equation 2 B,
+ C(:)e
update learning rate using cyclic modulation
m + 3m
V VoLos
BBYOL
+e; ex 3)
0 + m
update parameters 0 using update rule
& - TE (1-
end of cycle then
yield
end if
end for
Marginalizing over representations: Once pretraining is done, can marginalize the posterior over for downstream tasks. To compute the predictive distribution for a new instance x² we use model average over all collected samples with respect to the posterior over 0.
T Elu'le la", ,(()), g(h) & p((D))
plu" x=, D) = --looooll pl ply"1="00)p001
(8)
We will see that this model average is effective for improving performance, calibration as well as out-of-distribution detection in downstream task.
EXPERIMENTS
this section, we present our experimental results. evaluate proposed method on several tasks including cemi-supervised learning and out-of-distribution detection. First, describe exper- imental setup. Then, we evaluate our model using semi-supervised setting, and lastly, we evaluate model using out-of-distribution examples. We implemented the code using PyTorch (Paszke et al. 2017)). Our code is available upon request.
4.1 EXPERIMENTAL. SETUP
Datasets We conduct our experiments on five challenging image datasets. brief description of these datasets summarized in Table Both the training and set are used for CIFAR- 10, CIFAR-100 (Krizhessky & Hinton (2009)), and STL-10 (Coates et al. (201I). while only the training set used for ImageNet-IU 10 (Chang et (017)), and Tiny-ImageNet (Le Yang (201)). For Imagenet-Il 10 and Tiny-ImageNe the Validation set of Imagenet- 10 is used for evaluation, since the Test set of these datasets do not have ground-truth For STL-10, its 100,000 unlabeled samples are used for pretraining.
Implementation Details We adopt ResNer-18 (He et al. (2016) as the backbone for the self- supervised learning model. Following the original setting of BYOL, we use 2-layer MLPs as the
Table 1: A summary of datasets used for evaluations.
Dataset
Pretrain
Fine-tune
Test
Samples
Classes
Image Size
CIFAR-10
Train
Train
Test
60000
10
32
CIFAR-100
Train
Train
Test
60000
100
32
STL-10
Unlabeled
Train
Test
113000
10
96
ImageNet- 10
Train
Train
Validation
13000
10
224
Imag-Im
Train
Train
Validation
103925
10
64
4

---

scheduler. The batch Under review as a conference paper at ICLR 2023
80
02
EDYOL
EDIOL
EDYCL
IDVIC
20
80
Cydla
Figure 1: NLL and ACC in downstream task for embeddings of last 4 cycles of both models. left: Results on CIFAR-10, 10, right: Results on ImageNet-10. Embeddings from BBYOL show consistanly lower NLL and higher ACC.
projection and prediction heads. We apply the standard ResNet without modification on the input images of given sizes in for all datasets which produces feature vector of size 512 for each sample. We use terms "representations" or "embeddings" interchangeably for this feature vector. We use the same set of data augmentations in Grill et al. (2020) on all datasets for pretraining, con- sists of random cropping and resizing with random horizontal flip, followed by a color distortion and grayscale conversion.
For cSGHMC, a normal prior (O, I) is set on the parameters of the online network and cosine decay learning rate (Zhang al. (2020) with cycle length of 50 is used size is set to 256 and we use momentum term of 0.9. For all datasets we train the model from scratch for 1000 epochs with the initial learning rate 0.1 for Tiny-ImageNer and 0.2 for the rest of datasets. We inject Guassian noise epoch 45 and collect sample per cycle for the last cycles resulting in samples. We scale Guassian noise and prior with the dataset size (Florian et al. (2020))
Evaluation Metrics Tow widely-used metrics including Accuracy (ACC), and Negative Log Likelihood (NLL) are utilized to evaluate our method. Higher value of ACC indicate better per- formance of the model and lower value of NLL indicate better calibration.
Baselines We compare proposed Bayesian self-supervised model referred as BBYOL with its de- terministic counterpart in terms of described evaluation metrics. For training BYOL we use SGD optimizer with fixed learning rate schedule and momentum term 0.9. No weight decay is used. Other parameters are as the same as BBYOL. The initial learning rate for each dataset is described in Appendix [ We train BYOL on all datasets for 1000 epochs and take 4 samples last 200 epochs with the same interval 50 epochs. For both BYO and BBYOL if we marginalize over representations we use terms BOO--ENS and BBYOL-ENS respectively.
The experiments are carried out on Nvidia A40 48 GB and it takes about 7 gpu-hours to train the model on CIFAR-10 and CIFAR-100, 21 gpu-hours on STL-10, 9 gpu-hours on ImageNet-10, 24 gpu-hours on ImageNn. We repeat experiments for 3 random seeds and report average NLL and ACC over runs with the standard error from the mean predictor.
4.2 IMAGE CLASSIFICATION
We evaluate the performance obtained when fine-tuning both BYOL and BBYOL's representations on a classification task. In this task a pretrained model is fine-tuned on subsets of original training dataset with labels. We fine-tune on 100% of labeled training data described in Table [ We follow protocol in Grill et al.] (2020) detailed in Apendix B We report both ACC and NLL on five chal- lenging datasets in Table 2 and Table 3 According to the results shown in Table 2 and 3 BBYOL significantly outperforms BYOL and BYOL-ENS
all five datasets. In particular, BBYOL im- proves NNL on all datasets by large margin compared to its deterministic counterpart. Moreover, marginalizing over the representations in BB YOL-ENS also improves performance in terms of both ACC and NLL. For 10 and ImageNet we marginalize over two embeddings of last two cycles.
5

---

Under review as a conference paper at ICLR 2023
Table ACC 1 and NLL | on five object image benchmarks. The best results are shown boldface.
Dataset
CIFAR-10
CIFAR-100
STLI0"
Metrics
ACC
NLL
ACC
NLL
ACC
NLL
BYOL
7..0
0.4
4..6
2.0.0
6.5
96.2 2 3.6
BYOL-ENS
7.0
0.8
4.2
0.4
65.6
94.6 - 4.4
BBYOL
0.4
6.9
4..6
202.0
77.9
60.4 0.5
BBYOL-ENS
77.3 0.4
65.0 0.8
47.5
193.3 2.3
78.1 + 0.2
60.4 10.4+000 0.6
Table 3: ACC 1 and NLL on five object image benchmarks. The best results are shown in boldface.
Dataset
ImageNet-10
Tiny- ImagelmaeeNet
Metries
ACC
NLL
ACC
NLL
BYOL
$0.10.11
61.0 E 1.5
69.8 1.3
94.14.
BYOL-ENS
80.6 + 0.3
58.8 + 0.7
71.1 0.3
89.0 + 1.5
BBYOL
83.0 + 0.3
51.3 + 1.0
76.3 0.2
74.3 + 0.4
BBYOL-ENS
83.0 = 0.0
50.6
7.1
72.5 = 0.2
We observe that marginalizing over the representations in BYOL-ENS also improves performance. It is due to the nature of contrastive loss which induces diversity the parameter space. Whenever the loss is not too high, marginalizing over these representations improve the performance.
In Figure 1, we plot ACC and NLL of embeddings obtained from last cycles for CIFAR-I0 and ImageNet-10, 10. We observe that BBYOL consistently yields lower error and higher accuracy, demon- strating that embeddings obtained from a BBYO are more informative.
Ensemble Size In some applications, it may be beneficial to vary the size of the ensemble dynam- ically test time depending on available resources. Figure 2 displays the performance BB YOL on CIFAR-10 10 and CIFAR-10 100 datasets as the effective ensemble size, is varied. Although ensem- bling more models generally gives better performance, we observe significant drops in error when the second and third models are added to the ensemble. In most cases, an ensemble of two models outperforms the baseline model.
4.3 NUTTOF--DISTRBBUTOO DETECTION
To further explore the diversity of embeddings, consider the out-of-distribution (OOD) detection task. A pretrained model on CIFAR- 10 was fine-tunned on its training data with labels and tested on SVHN dataset (Netzer al. (20TT); and pretrained model on CIFAR- 1-0 was fine-tunned on its labeled training data and was tested on SVHN. We estimate the entropy of the predictive distribution on SVHN. Figure (left) presents the empirical CDF of the predictive entropy for CIFAR-10 (in-
CIFR-110
CFAR-100
0.5
NLL
2.00
0.64
1.5
0.63
1.9
1
3
Number cycles
Number cycles
Figure NLL as a function number ensembles on CIFAR-10 and CIFAR-100.

---

Under review as a conference paper at ICLR 2023
1.0
BBYOL-ENS (4)
1000
BBYOL BBYOL-ENS (6)
BBYOL-ENS (6)
Emperical CDF
800
0.6
Count
0.4
400
0.2
200
0.0
0.5
1.5
0.50
0.75
1.00
1.25
1.50
175
2.00
2.25
Entropy
Entropy
Figure 3: left: Empirical CDF for the predictive entropy of SVHN pretrained on CIFAR-10. right: Hitogram for the predictive entropy of SVHN pretrained on CIFAR-100. BYOL-ENS (4/6) is marginalizing over 4/6 embeddings.
distribution) and SVHN (out-of-distribution)) Figure (right) indicates the histogram predictive entropy for CIFAR-100 (in-distribution) and SVHN
For the unseen data, we expect the model indicates max entropy and the probablity of low entropy should be lower (Zhang et al. (2020).. It also implies that the mode of histogram focuses at higher value. We observe that all models assign high entropy unseen data, since self-supervised learning improves model uncertainty (Hendrycks et al. (2019)). In particular the predictive uncertainty im- proves on unseen classes, as the ensemble size increases. indicates that the embeddings produced by sampling from the posterior come from different modes and provide different characterization of training data. Indeed different embeddings can provide different predictions on OOD, leading to better uncertainty estimation.
Table summarizes the results for BYOL, BBYOL and BBYOL-ENS with different ensemble size. BBYOL-ENS (6) indicates marginalizing over 6 embeddings collected from last cycles. In BBL- (12") we took 3 samples per cycle in last 4 cycles. Consistent with our previous results BBYOL improves BYOL in terms of calibration (lower NLL) and uncertainty estimation (higher Entropy). The improvement in calibration and uncertainty estimation consistently increases by increasing the number of ensemble size.
Table 4: Results for OOD detection for various settings. Pretrained models are fine-tunned training labeled data of CIFAR- 10 and CIFAR-I0K 100 and tested on SVHN. All values are in percent- ages. indicates larger value is better, and indicates lower value is better.
In-Distribution
Out-of-Distribution
Method
NLL +
AUROC 1
Mean-Ent f
BYOL
5.22
47.13
0.99
BBYOL
5.12
47.24
0.98
Cifar-10 10
SVHN
BBYOL-ENS (4)
5.12
47.04
1.05
BBYOL-ENS (6)
4.99
47.82
1.10
BBYOL-ENS (12*)
5.02
47.74
1.09
BYOL
2.62
49.75
2.0
BBYOL
2.55
50.02
2.06
Cifar-100
SVHN
BBYOL-ENS (4)
2.36
49.83
2.23
BBYOL-ENS (6)
2.34
50.0
2.25
BBYOL-ENS (12*)
2.30
50.11
2.30

---

Under review as a conference paper at ICLR 2023
5 DISCUSSION AND FUTURE WORK
In this paper, we propose to explore the distribution over the representations using cSGHMC instead of estimating the posterior with a MAP solution. Our experimental results indicate that samples taken from the posterior provide meaningfully different representations which leads to improved accuracy, calibration and uncertainty estimation in downstream tasks. Moreover through our exper- imental results we indicate that using cSGHMC instead of a deterministic counterpart improves the quality of representations in downstream task in terms of both accuracy and calibration. Capturing the posterior over the embeddings can also provide the possibility of uncertainty estimation in the embedding space, opening new directions for practitioners in the field.
For future work we would like to further investigate the following: (i) We would like to analyze the notion of uncertainty in the embedding space and its relation with uncertainty in the predictive space. (ii) We would like to explore the possibility of using Bayesian self-supervised learning model prior for Bayesian downstream task.
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In Proceedings of the IEEE international conference computer vision, pp. 5879-5887, 2017.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Ma- chine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1683-1691, Be- jing, China, 22-24 Jun 2014. PMLR. URL https: pttttp::/pproceedll ngs ml wtt..m.sss...//.....
cheni ht.ml
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Wenzel Florian, Roth Kevin, Veeling Bastiaan S. Swiatkowski Jakub, Tran Linh, Mandt Stephan, Snoek Jasper, Salimans Tim, Jenatton Rodolphe, and Nowozin Sebastian. How good is the bayes posterior in deep neural networks really? arXiv preprint Ntt::2000.02200. 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1109/CVPR.20169900
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(1 43(11)5407744058 2020.
Laurent Valentin Jospin, Wray L. Buntine, Farid Boussaid, Hamid Laga, and Mohammed Ben- namoun. Hands-on bayesian neural networks a tutorial for deep learning users. CoRR, abs/2007.06823, 2020. URL http.h:ts::/aattt www.vvoorroggooooooooooo.oooo.oe htrgg/bbs/000 220.220077702223333
8

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report 0, University of Toronto, Toronto, Ontario, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3 2015.
Chunyuan Li, Andrew Stevens, Changyou Chen, Yunchen Pu, Zhe Gan, and Lawrence Carin. Learn- ing weight uncertainty with stochastic gradient meme for shape classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5666-5675, 2016.
Yi-An Ma. Tianqi Chen, and Emily B. Fox. A complete recipe for stochastic gradient MCMC. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1917-2925, 2015. URL httpsttt///eeeeeed//ieee neurips.. wwtpapper/215//aaaee.e.e 9a440050 0tw.o0leeebb299992288888888888 obtpr//9e2eeeettooccoaooaaaa..ee html
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Workshop Autodiff, 2017. URL /tttpss:/epeereeviiee net// Corum?id-BJJsiai
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp. 68166888 1888, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306199
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representattons, 2020.
A APPENDIX
For training BYOL using SGD optimizer we set initial learning rate e= 3e - 4 for STL.-10 and Kiny-ImageNer We set 3e 3 for the rest of datasets. For eSGHMC, as mentioned in Zhang et al. (2020), tempering helps. We use temperature 0.1 for all datasets.
B APPENDIX
For fine-tunnin tunnitunning, we follow the protocol Grill et al. (2200). We first initialize the network with the parameters of the pretrained representation, and fine-tune it with a subset of original datasets with labels. We do not loss using SGD with Nesterov momentum with batch size of 100, and a momentum of 0.9. We sweep over the learning rate (2e - 5, le 5, le 2e 4, 3e 4, 4e 4, 5e 4), weight decay (0, 5e 4} and the number of epochs (50, 60) and select the hyperparameters achieving the best performance on our local validation set to report test performance. Table 5 describes parameters for each dataset.
any data augmentation during fine-tumnin fune-tunning. We optimize the cross-entropy

---

Under review as a conference paper at ICLR 2023
Table 5: Parameters used in fine-tunning.
Dataset
learning rate
weight decay
number of epochs
CIFAR-10
2e
50
CIFAR-100
1c 4
0
50
STL-10
28
0
50
ImageNet-10
2E 4
0
50
Tiny-ImageNet
2e 4
5e 4
50
10


Reviews:
[{"review_id": "qxI90Qymny_", "reviewer": "Reviewer_6qBF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a Bayesian version of the self-supervised learning algorithm (BYOL), where the cyclical stochastic gradient MCMC (cSGMCMC) is employed for approximate posterior inference. Instead of point estimates, the proposed approach constructs posterior distributions of encoder parameters and uses them for downstream tasks. The proposed approach is demonstrated to outperform non-Bayesian baselines on benchmark datasets.", "summary_of_the_review": "I think in the current form the submission is not ready for publication, mainly due to the fact that it is a straightforward application of cSGMCMC to BYOL without much care.", "strength_and_weaknesses": "Strengths\n- Bayesian self-supervised learning is relatively underdeveloped; the paper is tackling an interesting problem.\n\nWeaknesses\n- Limited novelty. I see no contribution other than a mere application of cSGMCMC to BYOL.\n- Bayesian inference requires Monte-Carlo approximation which requires multiple forward passes through encoders. This is a critical downside, considering that we typically employ deep neural networks with millions of parameters for encoders. I'm not sure whether the benefit from the performance improvement outweighs the increased inference cost.\n- Experiments were done only for relatively small-scale datasets (CIFAR-10 and CIFAR-100).\n- When training Bayesian neural networks, one should be careful about using data augmentations, since the resulting model might not be interpreted as a valid Bayesian model. To bypass this technical difficulty, existing works studying BNNs often discard data augmentations; I think training BYOL is even harder than vanilla BNNs to be cast as a Bayesian inference problem, especially due to the use of slow-update parameter $\\xi$ (which also is a function of $\\theta)$. For instance, given the data augmentation policy and moving average parameter $\\xi$, what would be a valid likelihood? This is not an easy question to answer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Br1HQj-frdO", "reviewer": "Reviewer_tf9n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "propose a Bayesian technique by enforcing a prior over parameters for self-supervised learning. The main idea is based on the BYOL to learn the representation and combine that with Cyclical SGHMC. Paper applies the method to two datasets for semi-supervised classification and one dataset for out-of-distribution detection. ", "summary_of_the_review": "Even though the problem is rather important but this paper is highly incremental as it comes down to two basic ideas that were introduced and used by others. Experimental results are insufficient and it does not sufficiently compare this framework with other well-understood methods. ", "strength_and_weaknesses": "Strength: \n- Rather an important problem. \n- It is simple to follow and straightforward. \n- Enough background is provided.\n-------------------------------------------------------\nWeakness: \n- This paper is highly incremental meaning it is a direct combination of two things. Using BYOL for representation learning and then using a variation of HMC to sample from the distribution both of which were introduced in other papers. \n- All the techniques for predictive distribution approximation and etc are very straightforward and the novelty is very minor. \n- Simulation results are extremely insufficient. It would have been useful if a comparison with other state-of-the-art frameworks was provided. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "B1HUnmtiQB", "reviewer": "Reviewer_YvPk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose introducing Bayesian inference to self-supervised learning for improved model generalization capability. Specifically, they view the traditional learning of the BYOL as a MAP estimation; accordingly, Bayesian posterior sampling methods like the cSGHMC can be leveraged to replace the vanilla SGD optimizer for posterior inference of the model parameters. Experiments on semi-supervised classification tasks are conducted.", "summary_of_the_review": "(1) As mentioned in the Abstract, a Bayesian self-supervised learning method is proposed. So why use semi-supervised classification tasks to evaluate its effectiveness? Similarly, how would those experiments justify your goal of ``improving performance in downstream tasks,'' as mentioned in the introduction?\n\n(2) In Eq. (7), what are the definitions of the likelihood and the prior, when we talk about a self-supervised learning method? Besides, since X is a mini-batch, Eq. (7) is actually quite confusing; you are not performing cSGHMC on the mini-batch X, right? Similarly, there are other notations that are confusing/wrong.\n\n", "strength_and_weaknesses": "Strength.\n(1) The research direction is interesting.\n\nWeaknesses.\n(1) The underlying logic and notations are in a mess. \n(2) The novelty is quite limited.\n(3) The presented techniques are not convincing overall.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject