Incorporating Unlabelled Data into Bayesian Neural Networks
Authors: Mrinank Sharma
Tom Rainforth
Yee Whye Teh
Vincent Fortuin
Venue: Accepted by TMLR ()

Abstract: Conventional Bayesian Neural Networks (BNNs) are unable to leverage unlabelled data to improve their predictions. To overcome this limitation, we introduce Self-Supervised Bayesian Neural Networks, which use unlabelled data to learn models with suitable prior predictive distributions. This is achieved by leveraging contrastive pretraining techniques and optimising a variational lower bound. We then show that the prior predictive distributions of self-supervised BNNs capture problem semantics better than conventional BNN priors. In turn, our approach offers improved predictive performance over conventional BNNs, especially in low-budget regimes.

Published in Transactions on Machine Learning Research (07/2024)
Incorporating Unlabelled Data into Bayesian Neural Networks
Mrinank Sharma University of Oxford, UK
marinank@roboss.oo..cco
Tom Rainforth
rainforth@ssttts.or.cmm uk
University of Oxford, UK
Yec whye Tch University of Oxford, UK
w.W.lch@stals.oo ow.ac.ca..... uk
Vincent Fortuin
vincent..fortuinn@tam. de
Helmholtz AI, Munich, Germany
Technical University of Munich, Germany
Reviewed on OpenReview: http:://ppeereeeeeeee//////////s Owmmii
Abstract
Conventional Bayesian Neural Networks (BNNs) are unable leverage unlabelled data to improve their predictions. To overcome this limitation, we introduce Sapp-Supeee Bayesian Newrol Networks, which use unlabelled data to learn models with suitable prior predictive distributions. This is achieved by leveraging contrastive pretraining techniques and optimising a variational lower bound. We then show that the prior predictive distributions of self-supervised BNNs capture problem semantics better than conventional BNN priors. In lurn, our approach offers improved predictive performance over conventional BNNs, especially in low-budget regimes.
1 Introduction
Bayesian Neural Networks (BNNs) are powerful probabilistic models that combine the flexibility of deep neural networks with the theoretical underpinning of Bayesian methods (Mackay, 1992; Neal, 1995). Indeed, as they place priors over their paramcters and perform posterior inference, BNN advocates consider them a principled approach for uncertainty estimation (Wilson Ixmailov, 2020; Abdar et al., 2021), which can be helpful for label-efficient learning (Gal al., 2017). has even recently been argued that improving them will be crucial for large language models (Papamarkou et al., 2024) and generative Al as a whole (Manduchi et al., 2024).
Conventionally, BNN rescarchers have focused on improving predictive performance using human-crafted priors over network parameters or predictive functions (e.g,, Louizos et al., 2017; Tran et al., 2020; Matsubara Fortuin et al., 2021a). However, several concerns have been raised with BNN priors (Wenzel et al., 2020; Noci et al., 2021). also stands to reason that the vast store of semantic information contained in unlabelled data should be incorporated into BNN priors, and that the potential benefit of doing so likely exceeds the benefit of designing better, but ultimately human-specified, priors over parameters functions. Unfortunately, standard BNNs explicitly only models for supervised prediction, they cannot leverage such semantic information from unlabelled data by conditioning on it.
et al., 2021;
To overcome this shortcoming, we introduce Self-Superoised Bayesion Neural Networks ($3), which use unlabelled data to learn improved priors over functions. In other words, our approach improves the BNN prior prodictive distribution (which we will just call prior predictive in the remainder of the paper) by incorporating unlabelled data into it. This contrasts with designing different but ultimately speaffeee priors, which is the prevalent approach.

---

Published in Transactions on Machine Learning Research (07 (27/2024)
(a) Unsupervised Prior Learning
Learn prior s.t. augmented images have the same label
Learn prior s.t. distinct images have different labels
(b) Self-Supervised BNN Prior Predictive
(c) Conventional BNN Prior Predictive
Semantically incomsistent images
Semunticaaly consistent images and semantically inconsistent images are similarly likely have same label x
less likely
Semantically consistent images more likely to have the label
Demity
Demity
loss likelly
more likely
less likedy
more likely
Probability image has the lobel under the prior predictive
Probability image has the labol under prior predictive
Figure 1: Sepervuue Bayesian Neural Networks. (a) Pre-training in self-supervised BNNs corresponds to unsupervised prior learning. We learn model with a prior distribution such that augmented images likely have the same label and distinct images likely have different labels under the prior predictive. (b) Self-supervised BNN priors assign higher probabilities to semantically consistent image pairs having the same label compared to semantically inconsistent image pairs. Here, semantically consistent image pairs have the same ground-truth label, and semantically inconsistent image pairs have different ground-truth labels. The plot shows kernel density estimate of the bog-probability that same-class and different-class image pairs are assigned the same label under the prior. (c) Unlike self-supervisel prior predictives, conventional BNN prior predictives assign similar probabilities to semantically consistent and semantically inconsistent image pairs having the same label.
In practice, self-supervised BNNs generate pseudo-labelled data using unlabelled data and data augmentation, similar contrastive learning (Oord et al., 2019; Chen et al., 2020a;b; Grill et al., 2020; Hénaff al., 2020). We use this generated data to learn models with powerful prior predictive distributions. To do this, we perform unsupervised model learning by optimising lower bound of log-marginal likelihood dependent on the pseudo-labelled data. This biases the prior towards functions that assign augmented image pairs a larger likelihood of having the same label than distinct images (Fig. la). Following pretraining, we perform inference in the learnt model to make predictions.
We then further demonstrate that self-supervised BNN prior predictives reflect input-pair semantic similarity better than normal BNN priors ($4). To do so, we develop methodology to better understand the prior predictive distributions of BNNs. Our approach is to measure the probability of poirs of data points having the same label under the prior. Intuitively, pairs of points that are more semantically similar should be more likely to have the same label under the prior predictive. Applying this methodology, we see that the functional priors learned by self-supervised BNNs distinguish same-class input pairs and different-class input pairs much better than conventional BNNs (Fig. 1b).
Finally, we empirically demonstrate that the improved prior predictives of self-supervised BNNs translate improved predictive performance, especially in problem settings with few labelled examples (55).
2

---

Published in Transactions on Machine Learning Research (07/2024)
2 Background: Bayesian Neural Networks
Let fo( be a neural network with parameters 0 and D (as W))); be an datasct. We want to predict y from x. A BNN specifies prior over parameters, p(0), and likelihood, p(y]fo(x)) which in turn define the posterior p(0 & p(0) II, ((vslf((zz)) To make predictions, we approximate the posterior predictive p,,, D) = ppter (p(p
Improving BNN priors has been a long-standing goal for the community, primarily through improved human- designed priors. One approach to improve the prior over the network's s parameters (Louizos et al., 2017; Nalisnick, 2018). Others place priors directly over predictive functions (Flamn-Sepperd et al., 2017; Sun et al., 2019; Matsubara et al., 2021; Nalisnick et al., 2021; Raj et al., 2023). Both approaches, however, present challenges- the mapping between the network's parameters and predictive functions is complex, while directly specifying our beliefs over predictive functions is itsclf a highly challenging task. For these reasons, as well as convenience, isotropic Gaussian priors over network parameters remain the most common choice (Fortuin, 2022), despite concerns (Wenzel et al., 2020). In contrast to these works, we propose to learn better functional priors from unlabelled data via contrastive lcarning.
3 SuperSupe BNNs
Conventional BNNs are unable to use unlabelled data to improve their predictions. To overcome this limitation, we introduce Self-Superoised BNNs. At a high level, self-supervised BNNs allow unlabelled data to be incorporated by using it to learn a powerful prior that captures known similarities between inputs. In practice, we can utilise ideas from contrastive learning to learn models with prior predictive distributions that reflect the semantics of different input pairs. The high-level idea is thus to use prior knowledge in the form of data augmentations, for which we believe that the semantic content of the data should be invariant to them. We can then use variational method to learn a function-space prior that assigns more weight. to functions, whose outputs unlabelled data are also invariant to these augmentations.
Problem Specification. Suppose D" {am))= is an unlabelled dataset of examples x, R". Let = be labelled dataset correspouding
D' =
((+ ((// target associated with z/. We want to use both D" and D' to train deep learning model for predicting y given x with probabilistic parameters 0, where all information about the data is incorporated through the distribution on 0. That is, we predict using p(yla 0) for a given 0.
a supervised "downstream" task, where of is the
3.1 Incorporating Unlabelled Data into BNNs
The simplest way one might proceed, is to place a prior on 0 and then condition on both Du and Dt, leading to a posterior p( x p(0)D") p(D*'DD, p())D"))(D''D" 0). However, if we are working with conventional BNNs, which are explicitly models for supervised prediction, then p(0[D") = p(0). Further, as the predictions depend only the parameters, P(D' (D", 8) = ()), which then means that D" D³) p(A)D'). Thus, cannot incorporate D by naïvely conditioning on it.
To get around this problem, we propose to instead use D" to generate hypothetical labelled data and then condition our predictive model on it. In other words, we will use D" to guide a self-supervised training of the model, thereby incorporating the desired information from our unlabelled data. To do this, we will draw on data, augmentation (Yacger al., 1996; Krizhevsky et al., 2012; Shorten & Khoshgoftaar, 2019) and contrastive learning (Oord et al., 2019; Chen et al., 2020b;a; Grill et al., 2020; Hénaff et al., 2020; Chen & He, 2020; Foster et al., 2020; Miao et al., 2023).
Indeod, such approaches that use data augmentation provide effective means for making use of prior information. Although it is difficult to encode our prior beliefs with hand-crafted priors over neural network parameters, we can construct augmentation schemes that we expect to preserve the semantic properties of different inputs. We thus also expect these augmentation schemes to preserve the unknown downstream labels of different inputs. The challenge is now to transfer the beliefs iimefiillly defined through our data augmentation scheme-inte into our model.
3

---

Published in Transactions on Machine Learning Research (07 2024)
(a) Conventional BNN
0
z
1,.... T
Downstrear Datasel
(b) Supe-Supe BNN
I....T T
Downstreem Detssel
0²
v
Contrastive Datasel(s)
1,..., 2M
fj
1...
Figure 2: BNN Probabilistic Models. (a) Probabilistic model for conventional BNNs. (b) Probabilistic model for sule-supervised BNNs. We share parameters between different tasks, which allows to condition generated self-supervised data. indexes self-supervised tasks, i indexes datapoints.
One simple way to do this would be to just augment the data in D' when training standard BNN with standard data augmentation. However, this would ignore the rich information available in D Instead, we will use a construct from contrastive learning to generate psoudo-labelled data from D", and then condition 0 on both this pseudo data and D
Concretely, suppose we have a set of data augmentations A {a R² R"} that preserve semantic content. We use A and D to generate controstive datoset De that reflects our subjective beliefs by:
Drawing M examples from D" at random, { where indexes the subset, not D": For each i sampling a aa A and augmenting, giving 2 ( and x = a (á,); 3. Forming De by assigning ā.A and aB the same class label, which is the subset index i.
We thus have D === {x V = ((M U ((*,, where the labels are between and M. The lask associated with our generated data thus to predict the subset index corresponding to each augmented example. We can repeat this process times and create a set of contrastive task datasets, {D Here, consider the number of generated datasets L to be fixed, finite hyper-parameter, but discuss the implications of setting L = oo in Appendix A. Note that rather than using hand-crafted prior to capture our semantic beliefs, we have instead used data augmentation in combination with unlabelled data.
Next, to link each Dj with the downstream predictions, we use parameter sharing (sec Fig. 2). Specifically, we introduce parameters 85 for each Dj, parameters 8¹ for D', and saared-parameters that are used for both the rownstream and the contrastive tasks. D thus informs downstream predictions through 0², via (D))== For example, 0¹ and of could be the parameters of the last layer of neural network, while 0" could be the shared parameters of carlier layers.
Learning in this Framework. We now discuss different options for learning in this framework. Using the Bayesian approach, one would place priors over 0=, 0', and each 0g. This then defines posterior distribution given the observed data (Dz)-z and Dr. To make predictions on the downstream task, which depend on 0 and Ht only, would then use the posterior predictive:
plvllz,, (DI1/E,.)) D') -p(o-11D HDD;), 1000000 0",0"")
(1)
4

---

Published in Transactions on Machine Learning Research (07/2024)
where we have (i) noted that the downstream task parameters H are independent of (D)) given the shared parameters 8 and (ii) integrated over each ff and 0' in the definition of (D0) D)).
Alternatively, one can learn a point estimate for 6*, e.g., with MAP estimation, and perform full posterior inference for 0 and 03 only. This would be partially stochastic network, which Sharma et al. (2023) showed often outperforms fully stochastic networks while being more practical. In the case where all parameters are shared up to the last (linear) layer, this is also known as the neural linear model (Lázaro- Figueiras- Vidal, 2010), which has been shown to have many desirable properties (Ober & Rasmussen, 2019; Harrison et al., 2023). Learning in this way is also known as model learning, as used deep kernels and variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014; Wilson et al., 2015; 2016), where in the case of Gaussian processes (GPs), point estimate of kernel parameters is learned to also define learned prior over functions. Note that approach can also be considered kernel learning, where the representations of input data after the shared layers H* define the reproducing kernel Hilbert space (RKHS) and the kernel is given by their inner products. In learning a point estimate for 0%, one is learning a suitable model to perform inference in. This model has the following prior predictive:
&
plu (D;)) = Eptorrppy (r,,0 (),
(2)
where 02 is the learnt value of 0=.1 We would then update our beliefs over 0 in light of observed data. Through 0% we are using D to effectively learn prior over the functions that can be represented by the network in combination with 0r. Learning a point estimate for 0 is thus our main approsch.
3.2 Supe-Supe BNNs in Practice
We now use our framework to propose a practical two-step algorithm for sclf-supervised BNNs.
Preliminaries. We focus on image-classificttion problems. We use an encoder so- (-) that maps images to representations and is shared across the contrastive tasks and the downstream task. The shared parameters O thus are the base encoder's parameters. We also normalise the representations produced by this encoder. For the downstream dataset, we use a linear readout layer from the encoder representations, i.e., we have and y softmax fo- (ax)) + B¹). The rows of WJ are thus class template vectors, that is, data point will achieve the highest possible softmax probability for certain class if its representation is equal to (a scaled version of) the corresponding row of the weight matrix. For the contrastive lasks, we use a linear layer without biases, i.e., 07- = Wf, and j indexes contrastive tasks. We place Gaussian priors over 6*, 0¹, and each 8%.
0² = -W",
Pre-training 0" (Step I). Here, we learn a point estimate for the base encoder parameters 0%, which induces a functional prior over the downstream task labels (see Eq. 2). To learn 0", we want to optimise the (potentially penalised) log-likelihood log p( (( D' (')) but this would require integrating over H and each 05 Instead, we use the evidence lower bound (ELBO):
23(0") E [ogo( 091 DKL (a (g(0))l)(()) s log logp(D))0").
(3)
where ((f) is a variational distribution over the contrastive task parameters.
Rather than learning a different variational distribution for each contrastive task j, we amortise the inference and exploit the structure of the contrastive task. The contrastive task is to predict the corresponding source image index for each augmented image. That is, for the first pair of augmented images in a given contrastive task dataset, we want to predict class "1", for the second pair, we want to predict class "2", and so forth. The label is the index within the contrastive dataset, not the full datasct. To predict these labels, we use a linear layer applied to an encoder that produces normalised representations. We want a suitable variational distribution for this linear layer.
To make progress, we define 2 = for (a:1) and 2 = fo- (T.)), which are the representations of given images. To solve the contrastive task well, we want to map z1 and z to class "1", 21 and z to class "2", and so forth.
I Alternatively, this approach can be understood as learning the prior p(8".0' (0)) = p(0"))00-pp where 8 the Dirac delta function.
5

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Algorithm 1 Self-Supervised BNNs
Input: augmentations A, unlabelled data D", task data D', contrastive prior p(W))
for j I,...,L do
> Unsupervised prior learning
Draw subset {4: fitl set Dj = {
for i= 1...... M do
Di Create contrastive task
Sample a² ,a,a A
1 = ( a (i:)
2 (i; (i,). 2 for (b.))
Wy 0.5(2 + 2
Add (5 i) and (i 10) to Dj.
end for
whIl T + with e ~ NN0, 021) [(T, 2 0*) log ZM Fowggo(D3100 Wf) Der.l(W3))l((W)))
=-= wf
Update ,,, to maximise L(T, 0*) end for Approximate p(0' 09) ~ g(0)) Predict using Eg ( x,,00 0" 8']]
b Evaluation
We define W; = 0,5 + =f)); ) ie., wi is the mean representation for each augmented pair of images. Because the rows of the linear layer weight matrix Wy are effectively class templates, we use u(WfiT,o* = (u] o² wHi] /r. In words, the mean representation of each augmented image pair is the class template for each source image, which should solve the contrastive task well. Note that since this makes the last-layer weights dala-dependent,, it also renders them softmax outputs invariant to the arbitrary ordering of the data points in the batch, since if one permutes the Iú, and thus 2; this also automatically permutes the Wi and thus rows of W in the same way. Also, recall that the Wf are auxiliary parameters that are just needed during the contrastive learning to learn A that induces a good functional prior, but discarded afterwards and not used for the actual supervised task of interest. The variational parameters 7 and o² determine the magnitude of the linear layer and the per-parameter variance, vary throughout training, are shared across contrastive tasks j. and are learnt by maximising Eq. (3) with reparametcrisation gradients (Price, 1958; Kingma & Welling, 2013).
with 145- = [w] [w]
Both the contrastive tasks and downstream task provide information about the basc encoder parameters 0² One option would be to learn the base encoder parameters 0* using only data derived from Du (Eq. 3), which would correspond to standard superviee learning setup. In this case, the learnt prior would be task-agnostic. An alternative approach is to use both D' and D to learn 8", which corresponds to a semi-supervised setup. To do so, we can use the ELBO for the downstream data:
('((0) = Eg(or)logp(D11g (=') Dratlg(0") S logr(D')0*),
(4)
where q(0') = NO00 E²) is a variational distribution over the downstream task parameters; Σ is diagonal. We can then maximise Σ, (0*) + a (00),, where is a hyper-parameter that controls the weighting between the downstream task and contrastive task datasets. We consider both variants of our approach, using Sell-Supervised BNNs to refer to the variant that pre-trains only with (D9) and Self-Supervised BNNs to refer to the variant that uses both {D5}5= and Dt
Downstream Inference (Step II). Having learnt a point estimate for 0*, we can use any approximate inference algorithm infer 0¹. Here, we use a post-hoc Laplace approximation (Daxberger et al., 2021).
Algorithm 1 summarises Self-Supervised BNNs, which Iearn # with D" only. We found tempering with the meea--per-paamneee KL divergence, DKL, improved performance, in line with other work (e.g., Krishnan et al., 2022). Moreover, we generate a new Dj per gradient step so corresponds to the number of gradient steps. As shown on Algorithm 1, our full loss is L(T, 2 0*) log p(0)) 2 By IP( Wf] - -KLL ((W)))
6

---

Published in Transactions on Machine Learning Research (07/2024)
The first term of this loss is a prior over the shared parameters, in our case Gaussian prior, which is equivalent. to weight decay. The second term is where the actual contrastive learning happens, namely it is an expected Categorical log-likeling oggliikllhood (i.e., cross-entropy) over the softmax logits under Wj. Recall that the rows of this weight matrix are the mean embeddings vectors wi: so this likelihood encourages the inner products wi 2j to be large for i = j, that is, drawing two augmentations of the same image towards their mean and thus each other, and to be small for i + j, that is, pushing augmentations of different images away from each other. Finally, the third KL term places a Gaussian prior on We, which in our case means that the W; that make up this matrix (and thus the embeddings 2j) cannot grow without bounds to maximize the likelihood score, but have to stay reasonably close together. Moreover, following best-practice for contrastive learning (Chen et al., 2020a), we use a non-linear projection head (-(+) only for the contrastive tasks. For further details, see Appendix A.
Pre-training as Prior Learning. In this work, our central aim is to incorporate unlabelled data into BNNs. To achieve this, in practice, we perform model learning using contrastive datasets generated from the unlabelled data and data augmentation. This corresponds to an unsupervised prior learning step. Since our objective function during this is a principled lower bound on the log- marginal likelihood, it is similar to type-II maximum likelihood (ML), which is often used to learn parameters for deep kernels (Wilson et al., 2015) of Gaussian processes (Williams & Rasmussen, 2006), and recently also for BNNs (Immer et al., 2021a). As such, similar to type-II MI., our approach can be understood as a form of prior learning. Although we learn only a point-estimate for 0", this fixed value induces a prior distribution over predictive functions through the task-specific prior p(0*). However, while normal type-II ML learns this prior using the observed data itself, our approach maximises a marginal likelihood derived from unsupervised data.
4 How Good Are SuperSupe BNN Prior Predictives?
We showed our approach incorporates unlabelled data into the downstream task prior predictive distribution (Eq. 2). We also argued that, as the generated contrastive data encodes our beliefs about the semantic similarity of different image pairs, incorporating the unlabelled data should improve the functional prior. We now examine whether this is indeed the case.
Unfortunately, prior predictive checks are hard to apply to BNNs because of the high dimensionality of the input space. We will therefore introduce our own novel metric to assess the suitability of the prior predictive.
The basis for our approach is to note that, intuitively, a suitable prior should reflect a belief that the higher the semantic similarity between pairs of inputs, the more likely these inputs are to have the same label. Therefore, rather than inspecting the prior predictive at single points in input space, we examine the joint prior predictive of pairs of inputs with known semantic relationships. Indeed, it is far easier reason about the relationship between examples than to reason about distributions over high-dimensional functions.
Note that this is of course only reasonable assumption in cases where we believe to have sufficiently good knowledge of semantic similarity in our data domain. That is, we need to have set of data augmentations for the contrastive tasks, for which we can be reasonably certain that the true labels in our downstream task will be invariant to them. Recent results in contrastive learning suggest that this is indeed the case for natural images paired with the augmentations used in SimCLR (Chen et al., 2020a), which why we use these in our experiments.
To compute our proposed metric, we consider different groups of input pairs. Each group is comprised of input pairs with known semantic similarity. For example, for image data, we could use images of the same class as a group with high semantic similarity, and image pairs from different classes as a group with lower semantic similarity. To investigate the properties of the prior, we can evaluate the probability that input pairs from different groups are assigned the same label under the prior predictive. We can qualitatively investigate the behaviour of this probability across and within different groups. For a prior to be more adapted to the task than an uninformative one, input pairs from groups with higher semantic similarity should be more likely to have the same label under the prior predictive.
7

---

Published in Transactions on Machine Learning Research (07 (2742)
5
Augmented
Density
Same class
DifT. class
0
-1.0
0.5
-3
2
-1
log (x)
log z)
Figure 3: BNN Prior Predictives. We investigate prior predictives by computing the probability p that particular image pairs have the same label under the prior, and examining the distribution of p across different sets of image pairs. We consider three sets of differing semantic similarity: (i) augmented images; (ii) images of the same class; and (iii) images of different classes. Left: Conventional BNN prior. Right: Self-supervised BNN learnt prior predictive. The self-supervised learnt prior reflects the semantic similarity of the different image pairs better than the BNN prior, which is reflected in the spread between the different distributions.
Table 1: Prior Evaluation Scores. Mean and standard deviation across three seeds shown. Self-supervised priors are better than standard BNN priors.
Prior Predictive
Prior Evaluation Score
BNN
Gaussian
0.261 + 0.024
BNN
- Laplace
0.269 + 0.007
Self-Supent Supervise BNN
0.680 + 0.063
Moreover, extend this methodology to quantitatively evaluate the prior. Suppose have G' groups of input pairs, G, = ((af,.9) |اol with g = I,..., G, and suppose G1 is the group with the highest semantic similarity, G2 is the group with the second highest semantic similarity, and so forth. We define p(x, &) as the probability that inputs x have the same label under the prior predictive, ie., p(zr, () = (g(p(((x) (()) where v(x) is the label corresponding to input æ. We then define the prior evaluation score, a, as:
a E[l(p('', (all(l't')= .... > p(x ,80)),,
(5)
where we compute the expectation sampling (x,,**) ~ G1 and so forth. This the probability that the prior ranks randomly sampled input pairs correctly, in terms of semantically similar groups being assigned higher probabilities of their input pairs having the same label. We now use this methodology to compare conventional BNNs and self-supervised BNNs.
Experiment Details. We investigate the different priors on CIFARIO. For the BNN, we follow Izmailov et al. (2021b) and use a ResNet-20-FRN with a (0, 1/5) prior over the parameters. For the self-supervised BNN, we learn a base encoder of the same architecture with Du only and sample from the prior predictive using Eq. (2). 8² are the parameters of the lincar readout layer. For the image pair groups, we use: (i) an image from the validation set (the "base image") and an augmented version of the same image; (ii) a base image and another image of the same class; and (iii) base image and an image of different class. As these image pair groups have decreasing semantic similarity, we want the first group to be the most likely to have the same label, and the last group to be the least likely. See Appendix B.3 for more details.
Graphical Evaluation. First, we visualise the BNN and self-supervised BNN prior predictive (Fig. 1 and 3). The standard BNN prior predictive reflects belief that all three image pair groups are similarly likely have the same label, and thus does not capture semantic information well. In contrast, the superiid prior reflects belief that image pairs with higher semantic similarity are more likely to have the same label.
8

---

Published in Transactions on Machine Learning Research (07/2024)
In particular, the self-supervised prior is able to distinguish between image pairs of the same class and of different classes, even without access to any ground-trulh labels.
Quantitative Evaluation. We now quantify how well different prior predictives reflect data semantics. In Table 1, we see that conventional BNN priors reflect semantic similarity much less than self-supervised BNN priors, matching our qualitative evaluation. Note that this measure has of course been designed by us to capture the kind of property in the prior that our contrastive training is meant to induce, and should therefore just be seen as confirmation that our proposed approach works as expected. There are, naturally, many other properties that one could desire in prior, which are not captured by this metric.
5 Self-Supervised BNNs Excel in Low-Label Regimes
In the previous section, we showed that self-supervised BNN prior predictives reflect semantic similarity of input pairs better than conventional BNNs ($4). One hopes that this translates to improved predictive performance, particularly when conditioning on small numbers of labels, which is where the prior has the largest effect (Gelman et al., 1995; Murphy, 2012). We now show that this is indeed the case. Self-supervised BNNs offer improved prodictive performance over standard BNNs, with especially large gains when making predictions given small numbers of observed labels.
5.1 Semii-Suprrvsed Learning
Training Datasets. We evaluate the performance of different BNNs on the CIFAR1O and CIFAR100 datasets, which are standard benchmarks within the BNN community. We evaluate the performance of different basclines when conditioning on 50, 500, 5000, and 50000 labels from the training set.
Algorithms. As baselines, we consider the following BNNs: MAP, SWAG (Maddox et al., 2019), deep ensemble with 5 ensemble members (Lakshminarayanan et al., 2017), and last-layer Laplace (Daxberger et al., 2021). The conventional baselines use standard data augmentation and were chosen because they support batch normalisation (Tolfe & Szegedy, 2015). We consider two variants of self-supervised BNNs: Self-Supervised BNNs pretrain using DV only, while Self-Supervised BNNs also use D'. Both variants use a non-linear projection head when pretraining and the data augmentations suggested by Chen et al. (2020a). We use a post-hoe Laplace approximation for the spsk-sp parameters (the last-layer parameters). We further consider ensembling self-supervised BNNs.
Evaluation. To evaluate the predictive performance of these BNNs, we report the negative log-likelihood (NLL). This is a proper scoring rule that simultaneously measures the calibration and accuracy of the different networks, and is thus an appropriate measure for overull predictive performance (Gneiting & Raftery, 2007). We also report the accuracy and expected calibration error (ECE) in Appendix Table D.1. We further assess out-of-distributiin (OOD) generalisation from CIFARIO to CIFAR10-C (Hendrycks & Dietterich, 2019). Moreover, we evaluate whether these BNNs can detect out-o--distribution inputs from SVHN (Netzer et al., 2011) when trained on CIFAR10. We report the area under the receiver operator curve (AUROC) metric using the predictive entropy. We want the OOD inputs from SVHN to have higher predictive entropies than the in-distribution inputs from CIFARIO.
Results. In Table 2, we report the NLL for each BNN when making predictions with different numbers of labelled examples. We see that self-super BNNs offer improved predictive performance over the baselines. In fact, on the full CIFAR-10 test set, a single self-supervised BNN outperforms a deep ensemble, whilst being 5x cheaper when making predictions. We also show that self-supervised BNNs can also be ensembled to further improve their predictive performance. Incorporating the labelled training data during pretraining (SS BNN*) usually improves predictive performance. Self-supervised BNNs also offer strong performance out-of-distribution and consistently are able to perform out-of-distribution detection. Indood, they are the only method with an AUROC exceeding 90% at all dataset sizes. In a further analysis in Appendix Table D.1, we also see that the improved NLL of self-supervised BNNs is in large part due to improvements in predictive accuracy, and that incorporating labelled data during pretraining also boosts accuracy. Overall, these results accord with our earlier findings about the improved prior predictives of self-supervised BNNs compared to standard BNNs, and highlight the substantial benefits of incorporating unlabelled data into the BNN pipeline.
9

---

Published in Transactions on Machine Learning Research (07/2024)
Table 2: BNN Predictive Performance. We measure the performance of different BNNs for different numbers of labels. We consider in-distribution prediction, out-of-distribution (OOD) generalisation, and OOD detection. Shown is the mean and standard error across 3-5 seeds. The dusti--disttibuttin generalisation results average over all corruptions with intensity level five from CIFAR10-C (Hendrycks & Dietterich, 2019). Recall that the SS BNN is performing the contrastive learning separately from and the SS BNN* jointly with the downstream task. We see that self-supervised BNNs offer improved predictive performance over conventional BNNs, especially in the low-data regime.
J. Negative Log Likelihood
labelled points
Dataset
MAP
LL Taplace
SWAG
SS BNN
SS BNN
Deep Ensemble
SS BNN
SS BNN
Ensembble
Ensemble
CIFARIO
50
7.594
2.259
2,332
1.047
0.996
iama
3,689
0.980
0.953
500
2,501
22.022
1889
23
2,072
10.03
0.154
20.00
0.441
2000
1 1.806
ceaas
0.399 10:00
0.384 10.000
5000
1.570
1.327
1.028
20.0
0.361
0.369
gnons
0.846
0.309 m0
0.292 200000
50300
0.613
0.424
0.312
0.325
0.256
0.272
0.270
0.204
CIFARI00
50
11.86
4.585
6.840
10.300
4.505
-0.00
4.496
10.38
4.450 iises
4.492
500
5.536
4.359
2330
5,252
10.300
2.640
20.000
2.614
100000
4.807
2000
2.533
2.510
10.000
5000
4.319
3.362
3.518
1.689
1.910
3.052
1.524
1.844
50000
1.834
1.469
1,250
1.433
1.139
1,088
1.212
0.929
$4.003
CIFARIO to CIFARIOCC
50
7.140
2.275
2.353
1.723
1.697 Limm
3,970 -oLLas
1,638 10.000
1.603
10.000
(OOD Generalisaion)
500
2.838
83.055
2.045
20001
2.355
1.272
200014
1.260
20.000
2.101
1.164
1.113
5000
2.423
1.844
1.705
1.235
1.237
1.382
1.103
1.096
50000
1.944
1.244
1.215
1.287
1.225 so0m
0.984
1.126
1.048
AARROC (%)
CIFAR1O vs SVHN
50
51.11 14:5
48.1
52.3
87.1 10:0s
92.4
53.6
91.3
90.9
(OOD Detection)
500
61.2 2014
61.1 20:3
51.1 20:00
94.9 20:2
94.2 20:40
62.1 20:3
96.2
20:00
96.9
20:47
$3.3 sa
84.6 20.
59.6
2010
96.1 2n.
94.6 2x.00
92.9 g00s
97.0 gaar
96.9
20:33
50200
93.8
92.6
78.4
95.6
95.5
96.8
97.0
97.7
Moreover, we perform an ablation of our variational distribution q(W)) in Appendix Table C.2, where we see that our data-dependent mean is indeed needed for good performance.
Note that our goal in this experiment is mainly to compare our self-supervised BNNs against other BNN methods on equal grounds, not necessarily to reach state-of-the-art performance on the used benchmark datasets. Indeed, reaching higher performances usually requires computationally expensive hyperparameter tuning (which we have not systematically performed) as well as using many engincering tricks, such as data augmentation and batch normalization. These tricks generally affect the likelihood in complicated ways and are thus often omitted from Bayesian neural networks (sce, e-g., the discussions in Nabarro et al. (2021) and Krishnan et al. (2022)). This is why our results are empirically on par with many recent papers in Bayesian deep learning (e.g., Immer et al., 2021b; Ober & Aitchison, 2021; Izmailov et al., 2021b). However, it should be noted that some recent attempts have been made to reconcile BNNs with common practical deep learning tricks to reach high performance (c.f., Rudner et al., 2023). Adding these orthogonal idcas to our proposed framework would be a promising avenue for improving its performance to reach stale-of-the-arl levels.
5.2 Active Learning
We now highlight the benefit of incorporating unlabelled data in an active learning problem. We consider low-budget active learning, which simulates a scenario where labelling examples is extremely expensive. We use the CIFARIO training set as the unlabelled pool set from which to label points. We assume an initial train set of 50 labelled points, randomly selocted, and validation set of the same size. We acquire 10 labels per acquisition round up 500 labels and evaluate using the full test set. We compare self-supervised BNNs to deep ensemble, the strongest BNN baseline. We use BALD (Houlsby et al., 2011) as the acquisition function for the deep ensemble and self-supervised BNN, which provide epistemic uncertainty estimates. We further compare to SimCLR using predictive entropy for acquisition because SimCLR does not model epistemic uncertainty.
In Fig. 4, we see that the mothods that leverage unlabelled data perform the best. In particular, the self-supervised BNN with BALD acquisition achieves the highest accuracy across most numbers of labels, and substantially outperforms the deep ensemble. This confirms the benefit, of incorporating unlabelled data in
10

---

Published in Transactions on Machine Learning Research (07 (07/2022 2024)
Self-Supervised BNN (Ours)
80
Tost accuracy (%)
Ensemble
SimCLR
60
40
20
100
200
300
400
500
Number of acquired labels
Figure 4: Low-Budget Active Learning CIFARIO. We compare (i) self-supervised BNN, (ii) SimCLR, and (iii) deep ensemble. For the self-supervised BNN and the ensemble, we acquire points with BALD. We use prodictive entropy for SimCLR, which does not provide epistemic uncertainty estimates. Mean and std. shown (3 seeds). The methods that incorporate unlabelled data perform best by far, with our method slightly outperforming SimCLRR
active learning settings, which by definition are semi-supervised and include unlabelled data. Moreover, our approach slightly outperforms SimCLR, suggesting that our Bayesian treatment of contrastive learning yields better uncortaintics than conventional non-Baycsian contrastive learning. This is also confirmed in Appendix Fig. C.I, where we see that our approaches yield consistently lower calibration errors than SimCLR.
6 Related Work
Improving BNN Priors. We demonstrated that BNNs have poor prior predictive distributions ($4), a concern shared by others (e.g., Wenzel et al., 2020; Noci et al., 2021; Izmailov et al., 2021a). The most common approaches to remedy this are through designing better priors, typically over network parameters (Louizos al., 2017; Nalisnick, 2018; Atanov al., 2019; Fortuin et al., 2021b) or predictive functions directly (Sun et al., 2019; Tran et al., 2020; Matsubara et al., 2021; D' D'Angelo & Fortuin, 2021, Fortuin (2022) for overview). In contrast, approach incorporates vast stores of unlabelled data into the prior distribution through variational model learning. Similarly, other work also learns priors, but typically using labelled data e.g., by using meta-learning (Garnelo et al., 2018; Rothfuss et al., 2021) or type-1l maximum likelihood (Wilson et al., 2015; Immer et al., 2021a; Dhahri et al., 2024), or by using transfer learning in an ad hoc way (Shwartz- Ziv et al., 2022). Notably, function-space variational inference methods (Sun et al., 2019; Rudner et al., 2023) often also use unlabelled data, which has been shown potentially improve the out-of-distributioo performance of these models (Linn et al., 2023). However, in this case, the unlabelled data is only used for evaluating the KL divergence in function space, a practice which has theoretically been shown to be insufficient Burl et al. (2020). Conversely, our work uses semantic information from the unlabelled data to actually inform the function-space prior. Another related line of work is concerned with learning invariances from data in Bayesian models using the marginal likelihood (van der Wilk et al., 2018; Immer et al., 2022). This case is essentially the opposite of our setting, as there, the labels are known but the augmentations are learned, while our case, the augmentations constitute our prior knowledge, but we do not know the data labels.
A Perspective on Contrastive Learning. We offer Bayesian interpretation and understanding of contrastive learning (63). Under our framework, pretraining is understood as model learning technique for finding probabilistic models with prior predictive distributions that capture our semantic beliefs. There has been much other work on understanding contrastive learning (c.g., Wang & Isola, 2020; Wang Liu, 2021). Some appeal the InfoMax principle (Becker & Hinton, 1992). Zimmermann et al. (2022) argue that contrastive learning inverts the data-generating process, while Aitchison (2021) cast. InfoNCE, as the objective
11

---

Published in Transactions on Machine Learning Research (07/2024)
of self-supervised variational auto-encoder. Ganev & Aitchison (2021) formulate several semi-supervised learning objectives as lower bounds of log-likelihoods in a probabilistic model of data curation.
Semi-Supervised Deep Generative Models. Deep generative models (DGMs) are a fundamentally different approach for label-efficient learning (Kingma Welling, 2013; Kingma al., 2014; Joy al., 2020). A semi-supervised DGM models the full distribution p(z, y) with generative modelling, and so incorporates unlabelled data by learning to generate it. Unlike BNNs, we can condition the parameters of DGM on unlabelled data. In contrast, to construct pseudo-labelled tasks that encode our prior beliefs. Self-supervised BNNs are discriminative models, which tend to be more scalable and perform better for discriminative tasks compared to full generative modelling (Ng & Jordan, 2001; Bouchard & Triggs, 2004). Finally, Sansone & Manhaeve (2022) try to unily selfrsupe learning and generative modelling under one framework.
approach does not model the data distribution- the unlabelled data is used
7 Conclusion
We introduced Self-Superoised Boyesian Neural Networks, which allow semantic information from unlabelled data to be incorporated into BNN priors. Using novel evaluation scheme, we showed that self-supervised BNNs learn functional priors that better reffect the semantics of the data than conventional BNNs. In turn, they offer improved predictive performance over conventional BNNs, especially in low-data regimes. Going forward, we believe that effectively leveraging unlabelled data will be critical to the success of BNNs in many, if not most, potential applications. We hope our work encourages further development in this crucial area.
Acknowledgments
MS was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems JCP/S024050/1), and thanks Rob Burbea for inspiration and support. VF was supported by a Postdoc Mobility Fellowship from the Swiss National Science Foundation, a Research Fellowship from St John's College Cambridge, and Branco Weiss Fellowship.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavanzadeh, Paul Ficguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
Laurence Aitchison. InfoNCE is a variational autoencoder, July 2021. URL http::/aaxxiv.orr/aass 2107. 02495. arXiv:2107. 1524:22007.02495 02499 Jcs, stat).
Andrei Atanov, Arsenii Ashukha, Kirill Struminsky, Dmitry Vetrov, and Max Welling, The Deep Weight Prior. arXie:1810.0600 06943 fes, statl, February 2019. URL 1810.06943.
wrtp://///a/aa.0.00// 06943. arXiv:
Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random- dot stereograms. Nature, 366(6336) January 1992. ISSN 1476-4687. doi: 10.1038/3551 1003/35511610. 61. URL tttps://wwwt www.ww.aaaa.....a Group.
Number: 6356 Publisher: Nature Publishing
Guillaume Bouchard and Bill Triggs. The tradeoff between generative and discriminative classifiers. 16th
IASC International Symposium on Computational Statistics (COMPSTAT'OL) pp. 721-728, 2004.
David Burt, Sebastian W Ober, Adrià Garriga-Alone and Mark van der Wilk. Understanding variational inference in function-space. In Thind Symposium on Advances in Approzimate Bayesian Inference, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv:2002.0517 (cs, stat], June 2020a. URL http://arxix: org/ 0ts/20002.05709 arXiv: 2002.05709.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self- Sele--uupe Models are Strong Semi-Supervised Learners. arxix:0006. 10029 fes, stat), October 2020b. URL http: //aaxxv.org/abs/220000.. 10029. arXiv: 2006.10029.
Xinlei Chen and Kaiming Hc. Exploring Simple Siamese Representation Learning, November 2020. URL 10566. arXiv:2011.10566 [cs].
Francesco D' D'Angllo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information. Processing Systems, 34:3451-3465, 2021.
Erik Daxberger, Agustimus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-cffortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089 20103, 2021.
Rayen Dhahri, Alexander Immer, Betrand Charpentier, Stephan Günnemann, and Vincent Fortuin. Shaving weights with occam's razor: Bayesian sparsification for neural networks using the marginal likelihood. arXiv preprint arXiv:2402.1500 15978, 2024.
Daniel Plam-Shepherd, James Requeima, and David Duvenaud. Mapping gaussian process priors to bayesian neural networks. In NIPS Bayesion deep learning workshop, volume 3, 2017.
Vincent Fortuin. Priors in bayesian deep learning: A review. International Statistical Review, 2022.
Vincent Fortuin, Adrià Garriga-Alonso Mark van der Wilk, and Laurence Aitchison. BNNpriors: A library for Bayesian ncural network inference with different prior distributions. Software Impacts, 9:100079, 2021a.
Vincent Fortuin, Adrià Carriga-Alonso Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian Neural Network Priors Revisited. arXix:2102.0 06571 [es, stat), February 2021b. URL http:://rxiiii http:///axvvv.rgaaaaaa/ attg:/bs/210000 06571. arXiv: 2102.06571.
Adam Foster, Rattana Pukdee, and Tom Rainforth. Improving transformation invariance in contrastive representation learning. In International Conference on Learning Representations, 2020.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In PMLR, 2017.
International Conference on Machine Learning, pp.
Stoil Ganev and Laurence Aitchison. Semi-supervised learning objectives as log- likelihoods a generative model of data curation, October 2021. URL http://aariivorrg/aa///0066 05913. 2eXiv:2008.05913
[cs, stat].
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural Processos, July 2018. URL http://axxi..rr///////////0../ 01622 arXiv:1807.01222 [cs, stat].
Androw Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesion data analysis. Chapman and HaR//CRC, 1995.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359- 378, 2007.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised Learning, September 2020. URL htt.h/ppp/aarvv.oo/////////.... [cs, stat].
0070//0000/2000.0733. 07733. arXiv:2006.07733
James Harrison, John Willes, and Jasper Snoek. Variational bayesian last layers. In The Twelfth International Conference Learning Representations, 2023.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Pertarbations. In International Conference on Learning Regresenlations, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification 2011.
and preference learning, Xiv preprint or Xrxiv:112...
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Docrsch, S. M. Ali Eslami, and Aaron van den Oord. Data-Efficient Image Recognition with Contrastive Predictive Coding, July 2020. URL http:://axxv..rrg/bbs...
09272. arXiv:1905.0922 [cs].
Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Khan Mohammad Emtiyaz. Scalable marginal likelihood ostimation for model selection in deep learning. In International Conference on Machine Learning, pp. 4563 4573. PMLR, 2021a.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization, In International conference on artificial intelligence ond statistics, pp. 703 113-711. PMLR, 2021b.
Alexander Immer, Tycho van der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark van der Wilk. Invariance learning in deep neural networks with differentiable laplace approximations. Adoonces in Neural Information Processing Systems, 35:12449 1142249--1463,, 2022.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Pavel Izmailov, Patrick Nicholson, Sanae Lotfi, and Andrew Gordon Wilson. Dangers of Bayesian Model Averaging under Covariate Shift. arXiv:2106.1100 fcs, stat], June 2021a. URL http:://axxi..rrrt arXiv: 2106.17 11905.
abs/2106. 11905.
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What Are Bayesian Neural Network Posteriors Really Like? 11x4::2104.14421 [es, statf, April 2021b. URL http://arxiv.. org/ htt//abs/210044 14421. arXiv: 2104.14421.
Tom Joy, Sebastian M Schmon, Philip HS Torr, N Siddharth, and Tom Rainforth. Capturing label character- istics vaes. arXiv preprint 200: 10102, 2020.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXtv:1312.6114 2013.
Diederik P. Kingma, Danilo Rezende, Shakir Mohamed, and Max Welling. Semi-Supervised Learning with Deep Generative Models, October 2014. URL http://arxiv.org/abss. (cs, stat).
5298. arXiv:1406.5298
Ranganath Krishnan, Pi Esposito, and Mahesh Subedar. Bayesian-Torch: Bayesian neural network layers for uncertainty estimation, January 2022. URL
thub. wwtp/Inteeeaaabbbbaeeeeetteee torch.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, 1lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty cstimation using deep ensembles. Advonces in neural information processing systems, 30, 2017.
Miguel Lázaro-Gredille and Anibal R Figuciras- Vidal. Marginalized neural network mixtures for largo-scale regression. IEFF transactions on neural networks, 21(8):1345-135) 2010.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Jihao Andreas Lin, Joe Watson, Pascal Klink, and Jan Peters. Function-space regularization for deep bayesian classification. In Fifth. Symposium on Advances in Approsimate Bayesion Inference, 2023.
Christos Louizos, Karen Ulirich, and Max Welling. Bayesian compression for deep learning, Advances in neural information processing systems, 30, 2017.
David J C Mackay. Bayesian Methods for Adaptive Models. Technical report, 1992.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Laura Manduchi, Kushagra Pandey, Robert Bamler, Ryan Cottcrell, Sina Däubener, Sophie Fellenz, Asja Fischer, Thomas Gärtner, Matthias Kirchler, Marius Kloft, Yingzhen Li, Christoph Lippert, Gerard de Melo, Eric Nalismick, Björn Ommer, Rajesh Ranganath, Maja Rudolph, Karen Ullrich, Guy Van den Broeck, Julia E Vogt, Yixin Wang, Florian Wenzel, Frank Wood, Stephan Mandt, and Vincent Fortuin. On the challenges and opportunities generative ai. arXiv preprint arXiv:2403. 00000, 2024.
Takuo Matsubara, Chris Oales, and François- Xavier Briol. The ridgelet prior: covariance function approach prior specification for bayesian neural networks. The Journal of Machine Learning Research, 22(1):7445-7101, 2021.
Ning Miao, Tom Rainforth, Emile Mathicu, Yann Dubois, Yee Whye Tch, Adam Foster, and Hyunjik Kim. Learning instafcc-specific augmentations by capturing local invariances. International Conference Machine Learning, 2023.
Kevin Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Scth Nabarro, Stoil Ganev, Adrià Garriga-Alonso, Vincent Fortuin, Mark van der Wilk, and Laurence Aitchison. Data augmentation in Bayesian neural networks and the cold posterior effect. arXio:2106.0550 055886 [es, statf, June 2021. URL http: wwwp:ii..rrgabb//////// 05586. arXiv: 2106.05586.
Eric Nalisnick, Jonathan Gordon, and José Miguel Hernández-Lobato. Predictive complexity priors. In International Conference on Artificial Intelligence and Statistics, pp. 694-702. PMLR, 2021.
Eric Thomas Nalisnick. On priors for Bayesian newral networks. University of California, Irvine, 2018.
Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. Technical report, 1995.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Lorenzo Noci, Kevin Roth, Gregor Bachmann, Sebastian Nowozin, and Thomas Hofmann. Disentangling the Roles of Curation, Data-Anummentation and the Prior in the Cold Posterior Efffect. arXiv:2106.0660 06.0669666666 [es], June 2021. URL
06596. arXiv: 2106.06596.
Scbastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for bayesian neural networks and deep gaussian processes. In International Conference on Machine Learning, pp. 8248-8259. PMLR, 2021.
Sebastian W Ober and Carl Rasmussen. Benchmarking the neural linear model for regression. In Second Symposium on Advances in Approzimate Bayesian Inference, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, January 2019. URL
wwt.:/orrgg///////eeee.oo. 0ttps/1887.07444 03748. arXiv:1807.0371 [cs, stat).
15

---

Published in Transactions on Machine Learning Research (07/2024)
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nometh, Michael A Osborne, Tim GJ Rudner, David Rügamer, Yee Whye Tch, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position paper: Bayesian deep learning the age of large-scale ai. arxw preprint or Xix:24402 00809, 2024.
Robert Price. useful theorem for nonlinear devices having gaussian inputs. IRE Transactions on Information Theory, 4(2):69 72, 1958.
Vishmu Raj, Tianyu Cui, Markus Heinonen, and Pekka Marttinen. Incorporating functional summary information in bayesian neural networks using a dirichlet process likelihood approach. In International Conference on Artificial Intelligence and Stalistics, pp. 6741 6763. PMLR, 2023.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1286.
PMLR, 2014.
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta-learning with pac-guarantees. In International Conference on Machine Learning, pp. 9116-9126 PMLR, 2021.
Tim GJ Rudner, Sanyam Kapoor, Shikai Qiu, and Andrew Gordon Wilson. Function-space regularization in neural networks: A probabilistic perspective. In International Conference on Machine Learning, pp. 29275-29299 PMLR, 2023.
Emanuele Sansone and Robin Manhacve. Gedi: Generative and discriminative training for self-supervised learning. Xiv preprint arXiv:2212.13/ 2022.
Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In International Conference on Artificial Intelligence and Statistics, pp. 7694-7722. PMLR, 2023.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and An- drew Gordon Wilson. Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors, May 2022. URL http:://rxxiii org/ ///g//bb//a/ 220022255.00 10279. arXiv:2205.102 [es].
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. arXiv preprint arx:::1903.0577 2019.
Ba-Hien Tran, Simone Rossi, Dimitrios Milios, and Maurizio Filippone. All you need is a good functional prior for bayesian deep learning. arXiv preprint arXtv:2011.1829, 2020.
Mark van der Wilk, Matthias Bauer, ST John, and James Hensman. Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems, 31, 2018.
Feng Wang and Huaping Liu. Undorstanding the Behaviour of Contrastive Loss. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2495 2504, Nashville, TN, USA, June 2021. IEEE. ISBN 278-1-66544-500-2. doi: 10.1109/CVPR464490 2021..02222...222. URL https:// ieeexplore ieee. org wtt.oo//boceeee//9976//////////
Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings the 37th International Conference on Machine Learning, pp. 9929 9939. PMLR, November 2020. URL htt.https///proceeeiiii wttp:/eciiinnn..t...reee//////// ISSN: 2640-3498.
.ar.gggsm.r......... pttpp//e1e//waneeeeeeeeeeee html.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient ggeudo-independen weight. perturbations on mini-batches. arXie preprint 1rXxi:1883.04380 2018.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How Good is the Bayes Posterior in Deep Neural Networks Really? arXiv, February 2020. URL Publisher: Xrviv.
02400
Christopher KI Williams and Carl Edward Rasmussen, Gaussian processes for machine learning, volume 2.
MIT press Cambridge, MA, 2006.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of gencralization. Advances in neural informalion processing systems, 33:4697-4700 2020.
Andrew G Wilson, Zhiting Hu, Russ R Salakhuldinov, and Eric P Xing. Stochastic variational deep kernel learning. Advances in newral information processing systems, 29, 2016.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep Kernel Learning, arXiv:1511.02222 [cs, stat].
November 2015. URL http::t/t///t/// httt:://arxx..or/aas//55....
Larry Yaeger, Richard Lyon, and Brandyn Webb. Effective training of neural network character classifter for word recognition. Advances in neural information processing systems, 9, 1996.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint Xiv: 100::100.000 03888, 6(12):6, 2017.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process, April 2022. URL http://arxii..rrre wttp:/wwwww..gg////////////// 08850. arXiv:2102.08833 [cs].
17

---

Published in Transactions on Machine Learning Research (07 2224)
A Self- Sufervuuee BNNs: Further Considerations
We introduced Self-Supeeeieed BNNs ($3), which benefit from unlabelled data for improved prodictive performance within the probabilistic modelling framework. To summarise, our conceptual framework uses data augmentation to create a set of contrastive datasets {Di}}=1 In our probabilistic model, conditioning on this data is equivalent to incorporating unlabelled data into the task prior predictive. We now discuss further considerations and provide further details.
Thcorctical Considerations. In $3.1, we treated the number of contrastive task datasets, L, as a fixed hyper-parameter. However, one could generale an potentially infinite number datasels, which case, the posterior [D] x p(0*). pp(")00) - will collapse to a delta function, and will be dominated by the contrastive tasks. This justified learning point estimate for 0%, but if one wanted to avoid this behaviour, one could re-define the posterior:
L "(1(D))),,'')) x p(0)) p(D'100) II P(D3000111
(A.1)
The log-posterior would equal, up to a constant:
log 00110))))) D') llgg(pp)) + log (())p(D)))))))
2 L
Σ log p (D))00))
(A.2)
Here, the total evidence contributed by the contrastive datasets is independent of L, and instead controlled by the hyper-parameter ry This is equivalent to posterior tempering. The final term of the above equation could also be re-defined as an average log-likelihood when sampling different D=, i.e, we could use:
log D" D*) = p(0") + l('' 0*) + +/BD= [PD]))(()')'').
(A.3)
In this case, look for distribution algorithm samples a different De per gradient step, and instead weights the D' term with a, which is similar to the above approach if we set = 1/y and modify the prior term as needed. Re-defining the framework in this way allows it to support potentially infinite numbers of generated contrastive datasets. We further note that this framework could naturally be extended to multi-task sconarios.
0* where De has high likelihood average. Our practical
Practical Considerations. In practice, we share parameters 0 across all tasks (i.e., across the generated contrastive tasks and the actual downstream task), and learn a Ht separately for each task. We focus on image classification problems and let 0= be the parameters of a base encoder, which produces a represent ation z. 8t are the parameters of linear readout layer that makes predictions from z. We discussed learning a point-estimate for 8 by optimising an ELBO derived from the unlabelled data only:
(7)00)) E 1et05) [Dg00 09) DKI. (1))1(0)))))) S log logg(1))0"".
(A.4)
We (optionally) further include an ELBO derived using task-specific data:
L'((0) = R (ogpo 0*)] DKL. (g(ll((0))) ((00)(())))l)(0))) < logr(D')00)
(A.5)
Our final objective is:
L( = 1og (Z)))) [10))-(00((()))(0110)))----11113 [2})
(A.6)
where a = 0 would learn the base-encoder parameters only using the unlabelled data. a controls the weighting between the generated contrastive task data and the downstream data. We see the above objective is closely related to (lower bound of) Eq. (A.3). We further modify this objective function, following best practice, either in the contrastive learning Bayesian deep learning communities, and improve performance by:
18

---

Published in Transactions on Machine Learning Research (07/2024)
1. We add non-linear projection head to the base encoder architecture only for the contrastive task dalasels. As such, we use = (f (z))/ (x))]l for DD go(-) is the projection head, and the representation is normalised. For the downstream tasks, we use == fe- (z), i.e., we "throw-away" the projection head. This is best practice within the contrastive learning community (Chen et al., 2020a:b).
We temper the KL divergence term, using the (+l))- Tempering necessary for several Bayesian deep learning algorithms to perform well (Wenzel et al., 2020; Krishnan et al., 2022).
KL divergence (denoted
3. We generate new contrastive task dataset per gradient step, update on that dataset, and then discard it. This follows slandard contrastive learning algorithms (Chen et al., 2020a).
We rescale the likelihood terms in the ELBOs (0*)) and 27(0") to be average per-datapoint bog-likelhhoods, e.g., we use [D
1 log p(D5 10 0*)
Instead of having an explicit prior distribution over 0¹, we use standard weight-decay for training, i..., we specify a penalty on the norm of the weights of the encoder per gradient step.
Together, these changes yield the objective function used in Algorithm 1. Finally, we note that different practical algorithms ensue depending on the choice 8;, the choice of 8*, and the techniques used to perform approximate inference. We employ variational inference and learn point estimate for 0* but there are other choices possible.
B Experiment Details
We now provide further experiment details and additional results. The vast majority of experiments were run on an internal compute cluster using Nvidia Tesla V100 or A100 GPUs. The maximum runtime for an experiment was less than 16 hours.
B.1 Semi- Supervsee Learning (55)
B.1.1 Datasets
We consider the CIFAR10 and CIFAR100 datasets (Krizhevsky et al., 2009). The entire training set is the unsupervised set, and we suppose that we have access to different numbers of labels. For the evaluation protocols, we reserve a validation set of 1000 data points from the test set and evaluate using the remaining 9000 labels.
To assess distr-iiittiibutionn generalisation, we further evaluation on the CIFAR-10-C dataset (Hendrycks & Dictterich, 2018). We compute the average performance across all corruptions with intensity level five.
B.1.2 Sele-Supe BNNs
Base Architecture. We ResNet-18 architecture, modified for the size of CIFAR10 images, following Chen al. (2020a). The reprosentations produced by this architecture have dimensionality 512. Further, for the non-linear projection head, we use a 2 layer multi-layer perceptron (MLP) with output dimensionality 128.
Contrastive Augmentations. We follow Chen al. (2020a) and compose a random resized crop, a random horizontal flip, random colour jitter, and random grayscale for the augmentation. These augmentations make up the contrastive augmentation set A. We finally normalise the images to have mean 0 and standard deviation per channel, as is standard.
Hyperparameters. We use (O, + prior over the linear parameters 8*, and tune Tp for each dataset. As such, 7 can be understood as the prior temperature. We use Tp 0.65 for CIFAR10 and Tр 0.6 for CIFAR100. We weight decay le 6 for the base encoder and projection head parameters.
19

---

Published in Transactions on Machine Learning Research (07/2024)
Variational Distribution. That i8, we have = expo.
We parameterize the temperature and noise scale using the log of their values.
Optimisation Details. We use the LARS optimiser (You et al., 2017), with batch size 1000 and momentum 0.9. We train for 1000 epochs, using a linear warmup cosine anncaling learning rate schedule. The warmup starting learning rate for the base encoder parameters is le with maximum learning rate of 0.6. For the variational parameters, the maximum learning rate is le - 3, which found to be important for the stability of the algorithm.
Laplace Evaluation Protocol. We find a point estimate for 6 found using the standard lincar evaluation protocol (i.e., SGD training)- We then apply a post-hoc Laplace approximation using the generalised Gauss- Newton approximation to the Hessian using the laplace library (Daxberger et al., 2021). For CIFAR10, use full covariance approximation and for CIFARI00 the Hessian of the last layer weights and biases. We tune the prior precision by maximising the likelihood of validation set. For predictions, we use the (extended) probit approximation. These choices follow recommendations from Daxberger et al. (2021).
Krocnkor-factorise approximation for
B.1.3 Sele-Supe BNNs*
Self-Supervised BNNs* additionally leverage labelled data when training the base encoder by including an additional ELBO L' that depends D
We use mean-field variational inference over Ht with a Gaussian approximate posterior and Flipout (Wen et al., 2018). We use the implementation from Krishnan et al. (2022), and temper by setting 3 = 1/ meaning we use the average per-parameter KL divergence. a is a hyperparameter that controls the relative weighting between the generated contrastive task datasets and the obscrved label data, and is tuned. For CIFAR1O, We use 5 10-5 when we have fewer than 100 labels, and 5 10-3 otherwise. For CIFAR100, We use 5 10-5 when have fewer than 1000 labels, and 5 10-3 otherwise. For p(0'), we use NO, 1) prior. For downstream evaluation, we use the Laplace evaluation protocol.
All other details follow Self-Supervised BNNs.
B.1.4 BNN Baselines
All baselines use the same R8 architecture, which modified for the image size used in the CIFAR image datasets. The baselines considered were chosen because they are all compatible with batch normalisation, which is included in the base architecture. We provide further details about the baselines below.
MAP. For the maximum-a-posterroo network, we use the Adam optimiser with learning rate 10-3, default weight decay, and batch size 1000. We train for minimum of 25 epochs and maximum of 300 epochs, terminating training carly if the validation loss increases for 3 epochs in row.
Last-Layer Laplace. For the Last-Layer Laplace baseline, we perform a post-hoc Laplace approximation to a MAP network trained using the protocol above. We use the same settings as for the self-supervised BNN's Laplace evaluation.
Deep Ensemble. For the deep ensemble baseline, train 5 MAP networks starting from different inilialisations using the above protocol, and aggregate their predictions.
SWAG. For the SWAG baseline, we first a MAP network using the above protocol. We then run SGD from this solution for 10 epochs, taking snapshots per epoch, and using K 20 as the rank of the covariance matrix. We choose the SWAG learning rate per run using the validation set, and consider 10-2, 10-3, and 10-1
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
B.2 Active Learning ($5)
We simulate low-budget active learning selling, For each method, we use their default implementation details as outlined in this Appendix, With regards to the active learning setup, we assume that we have access to a small validation set of 50 labelled examples and are provided 50 labelled training examples. Wo acquire 10 examples per acquisition round up to a maximum of 500 labelled examples, which corresponds to 1% of the labels in the training set.. We evaluate using the full test set. The deep ensemble and self-supervised BNNs provide epistemic uncertainty estimates, so we perform active learning by selecting the points with the highest BALD metric (Houlsby et al., 2011). For SimCLR, entropy, commonly used baseline (Gal et al., 2017). For this experiment, we use only the CIFAR10 dataset. SimCLR and the self-supervised BNNs here are pretrained on 500 epochs, not 1000 epochs default.
acquire points using the highest predictive
B.3 Prior Predictive Checks (54)
BNN Prior Predictive. We use the ResNee-20-FFRN architecture, which is the architecture used by Izmailov et al. (2021b). Note that this architecture does not include batch normalisation, which means the prior over parameters straightforwardly corresponds to a prior predictive distribution. We use a @N {)) prior over all network weights, again following Iamailov et al. (2021b), and sample from the prior predictive using 8192 Monte Carlo samples.
Self-Supervised Prior Predictives. We primarily follow the details outlined carlier, except we use the samee ResNet-20-FRN architecture used for the BNNs and batch size of 500 rather than 1000. To sample from the prior predictive, we use Eq. (2) and have y softmax fe- (z)), where normalise the representations produced by the base encoder to have zero mean and unit variance, and we have W ~ ~N(00, 20), with the prior precision chosen by hand. We neglect the biases because they introduce additional variance. The prior evaluation scores are not sensitive to the prior variance choice, and are evaluated by sampling images from the validation set, which was not seen during training. We used 4096 Monte Carlo samples from the prior.
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
C Ablation Studies
Effect of Batch Sizc. We study the effect of the pre-training batch size on the performance of our
self-supervised BNNs. We run one seed for 100 epochs across three different batch sizes on CIFARIO. We see in Table C.I, the performance on CIFAR10 is robust to reducing the batch size. We hypothesise this is due to the noise injected during pre-training.
Table C.1: Effect of pretraining batch size on self-supervised BNN.
Batch Size
CIFAR10 Accuracy (%)
100
0.81
500
0.81
1000
0.80
Effect of Variational Distribution. We run an ablation study changing the variational distribution mean CIFARIO. We evaluated using one sced, training for 100 epochs only. We consider sctting the mean of the variational distribution for image wi, to be: 0.5(z,* + =f),, #d and O. We see that a suitable mean is required for good performance.
Table C.2: Effect of the pretraining variational distribution self-supervised BNN performance CIFAR10. We see that some variant of our dala-dependent mean is needed for good performance.
Variational Dist. Mean CIFAR10 Accuracy (%)
0.19
2 0.5(z + R)
0.79
0.80
Effect of Pretraining and Inference. To better understand the effect of the pretraining objective and the approximate inference scheme used, we performed an ablation study on CIFAR10. We considered both variants of our variational pretraining, and additionally deterministic pretraining that uses the NT-XENT loss. We also consider either using Laplace approximate inference MAP estimation for the task parameters. Using the NT-XENT loss and MAP inference corresponds to SimCLR (Chen et al., 2020a), as widely used in the self-supervised learning community. For NT-XENT, we use 5 0.45 for CIFAR10 and = 0.3 for CIFAR100, For these experiments, we only train for 500 epochs.
In Fig. C.I, we see that incorporating the labelled data during pretraining boosts accuracy, but surprisingly decreases calibration. Relative to SimCLR, all of the self-supervised BNNs offer improved calibration at all dataset sizes. All approaches have high accuracy at low data regimes, highlighting the benefit. of leveraging unlabelled data. Both deterministic pretraining and variational pretraining bchave similarly, but performing approximate inference over task parameters substantially improves calibration.
22

---

Published in Transactions on Machine Learning Research (07/202) /024)
Test accuracy (%)
10
90
Test ECE (%)
80
5
70
10
10
10
10
10
10
Num. Labels
Num. Labels
Self-Supervised BNN (Ours)
SimCLR
Self-Supervised BNN (Ouns)
Self-Supervised BNN
Deterministic Pretraining (Ours)
Figure C.1: Effect of Pretraining and Inference on CIFAR1O. On the left plot, red, green, and blue lines overlap. On the right plot, blue and red lines overlap. Recall that the SS BNN is performing the contrastive learning separately from and the SS BNN* jointly with the rownstream lask. We see that our SS BNN* slightly outperforms the other approaches in terms of accuracy and that all of our approaches yield better-calibrated uncertainties than SimCLR.
Additional Results
We additionally report the in-distribution accuracy and expected calibration error (ECE) of diffcrent BNNs when observing different numbers of labels. These metrics, unlike the log- likelihood, are interpretable. But note that for useful classifier, need to have both accurate and well-calibrated predictions.?
that self-supervised BNNs substantially outperform conventional BNNs in terms of in-distribution accuracy. The gains are particularly large at smaller dataset sizes, precisely where improved priors are expected to make the biggest difference. Moreover, in terms of calibration, they consistently offer well-calibrated uncertainty estimates. Even though LL Laplace offers well-calibrated uncertainty estimates at low numbers of labels, the predictions are much less accurate than self-supervised BNNs. We also see that incorporating labelled data during pretraining or ensembling self-supervised BNNs boosts accuracy, but surprisingly harm calibration. Curiously, find that the calibration of ensemble methods also sometimes worsens as condition on more data.
In Table D.1,
2There perfectly calibrated but useless classifiers, e.g., 70% examples class and 30% are class predicting píclass A) 0.7 on every input achieves perfect ECE but does discriminate between examples
23

---

Published in Transactions on Machine Learning Research (07 (07/2022 2224)
Table D.1: Bayesian Neural Network Predictive Performance. Here, relative to the main results table, we report the accuracy and expected calibration error of different methods separately. Recall that the SS BNN is performing the contrastive learning separately from and the SS BNN* jointly with the downstream task.
Accuracy (%)
# labelled points
Dataset
Deep Ensemble
SS BNN
SS BNN
MAP
LL Laplace
SWAG
SS BNN
SS BNN
Ensemble
Ensemble
CIFARIO
14.6
14.9
14.8
66.3
68.3
19.0
68.9
69.5
500
31.2
32.4
31.5
84.8
86.2
39.2
87.0
87.6
5000
56.5
53.4
66.4
87.7
88.6
72.1
89.6
90.9
50000
83.4
85.7
90.5
88.6
91.7
91.8
90.7
93.2
CIFARI00
3.6
3.6
3.1
14.5
14.7
16.2
16.5
500
6.1 10.
38.5
9.2
40.8
41.3
5000
22.7
21.2
25.6
54.5
56.0
28.2
58.4
62.5
50000
60.5
59.6
67.6
59.9
69.2
70.7
66.4
74.9
CIFARTO to CIFARIU-C
13.6
13.8
14.0
45.5
45.5
16.9
47.7
48.0
(OOD Generalisation)
500
26.7
27.2
26.2
57.8
59.0
32.0
60.3
61.8
5000
42.1
43.2
50.5
59.6
60.3
55.1
62.9
64.0
50000
59.2
62.1
69.0
59.8
63.1
70.1
63.6
66.8
Expected Calibration Error (ECE; %)
CIFARIIO
65.9
2.7
1.7
1.9
35.3
1.9
2.0
500
30.0
3.2
13.4
1.5
10.6
2.3
2.0
5000
21.0
4.5
10.0
1.1
2.1
3.8
2.3
2.6
50000
8.4
1.5
2.8
0.8
1.2
3.7
2.8
2.1
CIFAR100
61.6
17.5
11.6
11.7
38.5
13.4
13.6
500
25.3
1.8
22.0
2.1
14.9
3.3
2.9
5000
32.3
1.4
16.4
1.5
7.8
4.4
3.2
12.3
50000
17.8
1.6
8.9
1.7
2.5
4.7
6.5
6.6
CTFAR1O to CIFARIO-C
50
67.6
4.0
8.4
9.2
37.3
6.1
5.9
(OOD Generalisation)
500
33.7
7.4
18.0
8.1
15.5
7.2
7.6
5000
30.4
9.2
19.1
7.4
7.8
6.9
5,0
6.5
50000
24.1
11.1
13.6
9.1
12.2
6.0
6.1
6.8
24


Reviews:
[{"review_id": "ALDrfIvnsK", "reviewer": "Reviewer_84pJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### **Strenghts.**\n\n- the idea and exploitation of unlabelled data for obtaining better priors in BNNs is definitely powerful and interesting. \n- authors did a good review of the state-of-the-art, and I find the related work and references particularly well written.\n- even if i have concerns with certain points, I did not find critical mistakes in the equations and derivations, and notation fits with the ideas and methods proposed.\n- experiments are sufficient, with a decent comparison wrt other SOTA methods from BNN literature.\n- the paper has still some issues with clarity, in my opinion, but it is general readable and easy to follow.\n\n#### **Weaknesses.**\n\nI would like to focus my concerns on the concept that the paper brings a lot: \"BNN prior predictive / prior predictive distribution\". From a Bayesian perspective, this prior predictive is kind of odd to me, because it basically ignores data (there is no inference there), and one samples weights/parameters from a \"tuned\" prior for later averaging standard likelihood predictions over the test data $y_\\star$ (I'm looking to Eq. (2) for instance). This point confuses me a lot because some predictive properties/performance are claimed all along the paper bc this prior predictive distribution is well obtained.\n\nHaving some experience in BNNs, I've not seen this prior predictive thing a lot, even if in many cases one wants to do model selection (i.e. fitting hyperparameters of the prior wrt to marginal likelihood) or just posterior approximations for later performing well with the (posterior) predictive estimates. \n\nBefore Eq. 2, it is claimed that learning in this way is known as model learning, and used in Wilson et al. 2015, 2016. However, I've not found in these references something similar to the prior predictive framework proposed by the authors, so I have important doubts around the validity of such claims.\n\nI feel that the introduction of self-supervised learning/ contrastive methods with unlabelled data for improving BNNs is definitely a big topic to come, but from section 3.1 and partially section 3.2, I feel that some Bayesian principles are not entirely respected or well adjusted to what authors would like to propose in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents advances in Bayesian neural networks (BNNs) leveraged by unlabelled data. For this task, a self-supervised framework is designed around the prior predictive distribution for improving on the NN predictive tasks. Results show that the priors learned by leveraging unlabelled data work better than conventional BNN priors.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I want to remark that I have a positive impression of the self-supervised and contrastive learning approach proposed. However, I still feel that it does not fit very well in the current focus and framework of BNNs, to the point, that it makes me doubt if the method is actually a Bayesian technique. So, having some answers and improvements on what I wrote before and sections 3.1 and 3.2 would be a plus. Including some extra clarity in the presentation of the methodology.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wo73Sillmr", "reviewer": "Reviewer_R7PH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have several major concerns regarding the contrastive learning procedure considered in the paper.\n1. The contrastive loss is not specified in the paper. The only description of the loss function is as follows.\n```\nThe contrastive task is to predict the corresponding source image index for each augmented image. That is, for the first pair of augmented images in a given contrastive task dataset, we want to predict class “1”, for the second pair, we want to predict class “2”, and so forth. ... To predict these labels, we use a linear layer applied to an encoder that produces normalized representations.\n```\nIt is unclear how exactly the authors use the linear layer to classify pairs of images.  \n2. Furthermore, it is not clear why predicting the index of the pair within the given batch results in contrastive loss. Indeed, the model will be penalized for predicting different labels (for the original and augmented images), however, it will also be penalized for predicting the same label for both images but different from their index in the batch. This design choice does not have any motivation in the paper.  \n3. The variational distribution for the contrastive head layer does not correspond to the probabilistic model used in eq. (3). Indeed, the parameters of the variational family described in Section 3.2 depend on the input data, while, in eq. (3), the authors use it as a conventional variational family that does not depend on inputs.\n\nI also have some minor concerns regarding the presentation:\n- throughout the paper, the authors use the word \"predictive\" as a noun, which I can interpret as \"predictive distribution\", but it adds confusion; \n- it is not clear what \"class template vector\" means;\n- the paper has grammar mistakes, for instance, in Section 3.1, see the second sentence of the first paragraph;\n- the paper has spelling mistakes, for instance, in Section 3.1, see the last sentence of the first paragraph.\n\nThe empirical evaluation of the proposed approach can be significantly improved.\n1. The paper does not contain an ablation study for the introduced design choices. For instance, the choice of the variational distribution and the contrastive loss (which is not properly discussed) are not motivated and are not studied properly.\n2. The proposed method is compared only against the methods that do not use the unlabelled data. The comparison against other semi-supervised techniques (not necessarily Bayesian) is required.\n3. In Table 5, reported accuracies are much lower than other conventional and Bayesian models. For instance, see Table 9 in [2]. This raises a major concern regarding the quality of the baselines.\n\n[2] Maddox, Wesley J., Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. \"A simple baseline for bayesian uncertainty in deep learning.\" Advances in neural information processing systems 32 (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a way to perform semi-supervised learning of Bayesian Neural Networks (BNNs).\nThe proposed approach mostly follows the contrastive learning technique from [1], i.e. the authors train an encoder on the labeled data via the constrastive loss, which then can be used for supervised learning on downstream tasks with labeled data. The main focus of the paper is on bringing the Bayesian perspective to this framework. Namely, the authors propose to use Bayesian linear layers as \"heads\" for both the contrastive learning and downstream tasks. The encoder model precisely follows [1], though.\n\nFor the empirical evaluation, the authors evaluate their approach on CIFAR-10 and CIFAR-100, from which they partially remove the labels. For the baselines, the authors choose several Bayesian methods that do not incorporate unlabelled data into their training procedure.\n\n[1] Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. \"A simple framework for contrastive learning of visual representations.\" In International conference on machine learning, pp. 1597-1607. PMLR, 2020.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The paper requires a major revision. See the list of major concerns for the requested changes.", "broader_impact_concerns": "The paper does not require including a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5gkDg9LKR2", "reviewer": "Reviewer_Eww4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOne of the motivations of Bayesian methods is that the priors should be a good initialization for a given ML task. It makes sense to learn priors that allow fast adaption to tasks in the low-data regime.\nThe proposed approach does appear to show strong sample-efficiency in the experiments, in large-ish scale experiment settings such as CIFAR 10. It is also good at out-of-distribution and active learning.\n\n**Weaknesses**\n\n**Function-space VI.**\nThroughout the paper, the authors claim that BNN methods cannot use unlabeled data. In functional / function-space VI (Sun et al. 2019, which the authors cite), the function-space KL is enforced by a 'measurement set'. This measurement set is essentially samples from the data domain, so it could easily include unlabelled data. In fact, there was an AABI paper last year that used FVI to improve OOD performance of BNN using unlabelled OOD data [A]. The authors need to revise their claims and potentially compare them to FVI methods if they are relevant. \n\n**Actual relevance to BNNs.**\nBy the end of the paper, I was questioning how relevant this paper was to actual *Bayesian* NNs and not just NNs more generally. Since it seems the shared 'torso' network is trained to obtain a point estimate of the shared parameters, it's not really an interesting prior distribution over the weights, and the initial motivation of the work feels a bit misleading. You could focus the paper on the value of this pre-trained torso network and just use BNNs for the evaluation. \nAt the very least, I would motivate the work through features /kernels rather than weight priors as done in the introduction. \n\n**Prior design and inventing a metric that the method was designed to solve.** \nIt's not a good idea to invent a new metric to evaluate a model since there is a strong motivation to bias it towards the proposed model's strengths. \nI don't agree that 'similar inputs should predict the same class' is a sound prior we want models to have, and Section 4 seems to suggest it's an inherently good idea when it is, in fact, a subjective design decision made by the authors. \nI think a good BNN prior for classification should pick no specific class, and each input should have a close-to-uniform distribution over labels. \nThis is the motivation for several works using stationary Gaussian processes-like architectures for BNNs [B, C]. \nThe authors could consider combining their method with the architecture of [B, C] to get the best of both worlds. \nIn summary, the authors have conflated a model having invariances with a model's prior, which are not exactly the same. A model can have invariances without it necessarily being evident in it's prior. For example, you could use sine and cosine features for angular data in a stationary Gaussian process to encode rotational invariance, but this doesn't change the stationary nature of the prior and the predictive distribution is unchanged. \n\nI think a more principled metric would be the 'effective dimension' (c.f. [D]) of the kernel induced by the pre-trained feature space. \nFor $n$ data points, the effective dimension of the kernel (basically the rank of the data matrix, with the kernel being the outer product of the features) is $n$ if the features are unique and $1$ if the feature /data point is just repeated $n$ times, therefore minimizing the effective dimension for augmented inputs would be a more principled way of encoding this prior knowledge into the model and focuses on the fact that the authors are optimizing a deterministic feature space rather than a weight distribution. \n\n**The neural linear model.**\nThe authors don't seem to know about the neural linear model / Bayesian last layer, which is a Bayesian linear regression on a neural network feature space. It's one of the oldest approaches to BNNs [E] and has lots of attractive properties, such as tractable marginal likelihoods in the regression settings [F] and also tractable ELBOs [G]. Throughout the paper, the authors refer to 'last-layer Laplace' and 'partial stochasticity' when the neural linear model is an older and more explicit way of describing this approach. \n\n**Invariance learning with the marginal likelihood.** I'm not very familiar with the body of work, but I know that the marginal likelihood has been used to train invariances with Bayesian neural networks [H, I]. This body of work seems relevant enough to discuss as related work, and even relevant enough to evaluate against it since the pre-trained feature space is also essentially learning invariances.  \n\n**The experiments mix up approximate inference with architectures.**\nThe experimental evaluation, while thorough, was a bit confused. The central question that was tackled by the experiments was 'Does pre-training the feature space improve BNN uncertainty quantification?'. This is invariant to the choice of approximate inference for the BNN itself. Therefore, the experiments could rather use the pre-trained network as the torso for the MAP, SWAG and ensemble and see if the torso brings a net benefit across approximate inference methods. Comparing two very different architectures and approximate inference methods is not that meaningful. \n\n**Clarity**\nA few clarity issues:\n* The ''# Labels' column in Table 2 is a bit confusing since I believe it's referring to the number of training data points, but it reads like the dimensionality of the classification problem is increasing \n*  The definition of SS BNN* should be repeated a few times (e.g. the caption of Table 2) because the name is vague and it's easy to miss/forget while reading the main text.\n\n[A] Function-Space Regularization for Deep Bayesian Classification, Lin et al.\n\n[B] Periodic Activation Functions Induce Stationarity, Meronen et al \n\n[C] Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness, Liu et al\n\n[D] Bandit optimisation of functions in the Matern kernel RKHS, Janz et al \n\n[E] Marginalized neural network mixtures for large-scale regression, Lázaro-Gredilla et al\n\n[F] Benchmarking the Neural Linear Model for Regression, Ober et al\n\n[G] Variational Bayesian Last Layers, Harrison et al\n\n[H] Learning Invariances using the Marginal Likelihood, van der Wilk et al \n\n[I] Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations, Immer et al", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes using unlabelled data to pretrain a BNN prior for greater sample efficiency in downstream tasks. \nThey propose a pre-training scheme that uses a contrastive-style ELBO objective combined with data augmentation.  \n\nThey look at image classification, out-of-distribution detection, and active learning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the feedback above, my major concerns can be summarized as follows:\n\n1) Retract claims on BNNs and unlabelled data w.r.t. FVI and discuss FVI with greater accuracy \n\n2) Disentangle architecture and approximate inference in the experimental section \n\n3) Rewrite paper to motivate instead by learning invariant feature spaces rather than learning weight prior distributions, or do experiments that learn weight priors for the pretrained network torso\n\n4) Remove Section 4 and use Figure 3 and Table 1 as evidence for the behavior of Algorithm 1.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As discussed above, the paper is currently primarily lacking in its empirical evaluation, which needs to be rectified. The aim of this revision should not be to chase state-of-the-art results in the fully supervised case, i.e., 50000 labels, but to ensure that the reported results for each setup mirror their prior reported performances for the reader to accurately interpret the results. 

I recommend this as a minor rather than as a major revision, as I expect the relative performance improvements to persist in the more relevant semi-supervised setups. 
While not mandatory for acceptance this would also allow the authors to provide the reader with error bars on all reported results, which would further improve the interpretability of the results.

I therefore recommend _Acceptance with minor revision_.
Decision: Accept (poster)