Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling
Authors: Hyungi Lee
Giung Nam
Edwin Fong
Juho Lee
Venue: ICLR 2024 poster (2024)
Keywords: Transfer Learning
Nonparametric Learning
Bayesian Neural Network
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Posterior sampling method for transfer learning scenarios using nonparametric prior

Abstract: Transfer learning has recently shown significant performance across various tasks involving deep neural networks. In these transfer learning scenarios, the prior distribution for downstream data becomes crucial in Bayesian model averaging (BMA). While previous works proposed the prior over the neural network parameters centered around the pre-trained solution, such strategies have limitations when dealing with distribution shifts between upstream and downstream data. This paper introduces nonparametric transfer learning (NPTL), a flexible posterior sampling method to address the distribution shift issue within the context of nonparametric learning. The nonparametric learning (NPL) method is a recent approach that employs a nonparametric prior for posterior sampling, efficiently accounting for model misspecification scenarios, which is suitable for transfer learning scenarios that may involve the distribution shift between upstream and downstream tasks. Through extensive empirical validations, we demonstrate that our approach surpasses other baselines in BMA performance.

Published as a conference paper at ICLR 2024
ENHANCING TRANSFER LEARNING WITH FLEXIBLE NONPARAMETRIC POSTERIOR SAMPLING
Hyungi Lee
Giung Nam Edwin Fong? Juho Lee
KAIST AI The University Hong Kong *AITRICS
{1hk2708, giung, jubolee)kkaist.acc chefongåhku.hk
ABSTRACT
Transfer learning has recently shown significant performance across various tasks involving deep neural networks. In these transfer learning scenarios, the prior distribution for downstream data becomes crucial in Bayesian model averaging (BMA). While previous works proposed the prior over the neural network pa- rameters centered around the pre-trained solution, such strategies have limitations when dealing with distribution shifts between upstream and downstream data. This paper introduces nonparametric transfer learning (NPTL), flexible poste- rior sampling method to address the distribution shift issue within the context of nonparametric learning. The nonparametric learning (NPL) method is a recent approach that employs nonparametric prior for posterior sampling, efficiently accounting for model misspecification scenarios, which is suitable for transfer learning scenarios that may involve the distribution shift between upstream and downstream tasks. Through extensive empirical validations, we demonstrate that our approach surpasses other baselines in BMA performance.
INTRODUCTION
In Bayesian deep learning, we regard the parameters of a deep neural network as random variables. Instead of optimizing for single-point estimate of these parameters, this approach involves infer- ring the posterior distribution of these parameters given the provided training data and predefined parameter prior distribution. After have the posterior distribution, make predictions through Bayesian model averaging (BMA). BMA entails computing predictions from multiple parameter values and weighting them based their respective densities within the posterior. BMA effectively integrates both data uncertainty and model uncertainty into the prediction process, leading to more accurate and resilient predictions (Hoeting et al., 1999).
The success of Bayesian deep learning often depends on the choice of the prior distribution. While is common practice to employ simple zero-mean Gaussian prior for neural network parameters, there is an ongoing discussion regarding the adequacy of these zero mean Gaussian priors (Wenzel al., 2020; Fortuin et al., 2022). Meanwhile, in the context of transfer learning scenarios, these concerns about prior configurations are further intensified. The fundamental idea behind the trans- fer learning process is that when model parameters are pre-trained using sufficiently extensive and versatile upstream data, they inherently capture biases related to the data modality, which can be beneficial in related downstream tasks (Mikolov et al., 2013; Girshick et al., 2014). In this case, there is doubt that the Gaussian prior neural network parameters can sufficiently capture solely the "prior knowledge" " embedded the upstream data.
To address this issue, directed our attention to nonparametric posterior sampling method called Bayesian nonparametric learning (NPL; Lyddon et al., 2018; Fong et al., 2019). NPL enables the use of statistical models without assuming the model true. It utilizes a nonparametric prior, such mixture of Dirichlet Processes (Antoniak, 1974), centered around parametric model then updates a nonparametric posterior for the parameters of the parametric model. The NPL approach effectively accounts for the model misspecification by an implicitly defined prior. This prior is chosen without
Equal contribution

---

Published as a conference paper at ICLR 2024
any reliance the specific model of interest, which is suitable in the context of misspecified model as we do not have confidence in the existence of a true model.
Our primary contribution is utilizing the NPL approach, which effectively accounts for the model misspecification, within the context of transfer learning. While the fundamental premise of transfer learning is that the "prior knowledge" obtained from upstream data is advantageous for downstream tasks, empirical studies have indicated that prior regularization centered around pre-trained model in the weight space can hinder the downstream learning process in cases where there mismatches between the upstream knowledge and downstream tasks (Xuhong et al., 2018; Wan et al., 2019; Li al., 2020). In this regard, we were intrigued by the prospective benefits presented by NPL's efficacy in addressing model misspecification, especially in transfer learning situations where there can be distribution shift between upstream and downstream data. This done by combining the upstream and downstream information in noпparametric fashion, resembling form of weighted summation between the downstream dataset and the pseudo dataset.
We summarize contribution as follows:
We proposed nonparametric transfer learning (NPTL): posterior sampling method that adapts the NPL method into the transfer learning scenario where both the parameter of interest and the downstream dataset are very large.
Our proposed posterior sampling algorithm can be easily parallelized. We highlight that such parallelization is not straightforward for stochastic gradient Markov Chain Monte Carlo (MCMC) methods, which are commonly used in Bayesian deep learning for posterior sampling, due to their sequential nature (Neiswanger et al., 2014; 4; De Souza et al., 2022).
We empirically validate that the proposed algorithm shows better performance compared to the other baseline posterior sampling methods.
PRELIMINARIES
2.1 BAYESIAN DEEP LEARNING
In Bayesian inference, the goal is to sample from the posterior distribution p(0 over the parame- ters 9 after observing the data D. The prediction for a new datum x is then given by Bayesian model averaging (BMA),
plye, D) = p(yla, O)p(O
(1)
which we can approximate by Monte Carlo integration ply x. D) so Em=1pt P(y|z, 0m) with posterior samples ..,.... O p(0 D). In practice, typically introduce several sampling meth- ods for p(0 to compute the BMA integration since cannot be expressed closed form for modern deep neural networks.
In order to work with the posterior p)0 it is essential to first establish the prior p(0). The most common choice of the uninformed prior distribution is zero mean isotropic Gaussian, i.e. p(0) N( 0, I which is equivalent to L2 weight decay regularization (Krogh & Hertz, 1991). A straightforward way to enhance such prior to select the mean of the isotropic Gaussian with a pre-determined value (Chelba & Acero, 2006; Daumé III, 2007; Grachten & Chacón, 2017; Xuhong et al., 2018), iee. p(0) N0 prrar I²), where 0ptior is some informative weight with condensed information of the source task (e.g. MAP solution pre-trained on the source task). Re- cently, Shwartz-Ziv et al. (2022) proposed a method to build an informative prior from a pre-trained model, where they compute both mean and covariance matrix of the Gaussian prior using SWA- Gaussian (Maddox et al., 2019) procedure. However, none of these methods take into account the downstream dataset when creating the weight prior distribution.
2.2 TRANSFER LEARNING
Modern deep learning algorithms often employ the pre-training and fine-tniin paradigm (He et al., 2019): treitrain a model from scratch on a large source dataset, then fine-tune the model for a specific target task. This is based on the belief that there some transferable information between
2

---

Published as a conference paper at ICLR 2024
source and target tasks, i.e. we expect features learned from source data to be useful target tasks. Compared to training from scratch on a target task where the optimization starts from a randomly initialized parameter, transfer learning starting from the pre-trained parameter converges quicker to a reasonable solution. While it is common to fine-tune all parameters in pre-trained model, depending on the target task and computational resources available, it may be possible to fine-tune only subset of parameters while keeping the rest fixed. An example of this is linear probing, where the feature extractor is frozen, and only the last linear layer is newly trained for the target task.
2.3 NONPARAMETRIS LEARNING
We assume that downstream dataset D = (zi, is drawn i.i.d. from some data distribution Fo. and we are interested in a class of parametric models Fe == (F' E )} where Fe denotes the model data with 0 E θ C IRP (e.g. neural network with parameter 0). Here, we do not assume that our target distribution Fu belongs to Fe, indicating the possibility of model misspecification.
The NPL framework defines parameter of interest functional of Fo, that is, O, ((F)) where
((F) = arg min / {(0 x2 y) dFo y),
(2)
and E(0;ar æt y) is loss function of interest. In supervised learning, we {(0; 20 y) log so x, 0) as the negative log density of Fe, which gives 0, = arg ming DKL [Fo (Walker, 2013). Since we do not know Fo. we treat as a random measure F, and elicit Dirichlet process (DP; Ferguson, 1973) prior following Fong et al. (2019), that is F ~ DP Fx)- Here, F- is a center- ing probability measure acting as our prior guess on Fo, and E R4 is a scalar hyperparameter indicating the strength of this prior belief. By conjugacy, the posterior of F given D is also a DP:
a Fx a a a+n n
D DP
a + n,
The NPL posterior p(0) | D) is then the pushforward measure of p(F' D) through (2), from which
one can sample by drawing FG3 ~ p(F D) and computing /63 (F()). Draws from p(F D) can be carried out using skeleton of T atoms from F, and appropriate Dirichlet weights, which outline later. When is highly non-convex, Fong et al. (2019) explores two possible heuristics for feasible sampling. one is interested in sampling from multiple modes, the global minima in (2) can be estimated via random restart optimization with multiple random initializations. Alternatively, one may target local mode by initializing each optimization at some fixed init of interest around which we want to quantify uncertainty. Finally, Lyddon et al. (2018); Fong et al. (2019) show that the NPL posterior is robust to model misspecification through the adoption of nonparametric model, and gives asymptotically superior predictions to the regular Bayesian posterior on 6.
3 NONPARAMETRIC TRANSFER LEARNING
In this section, we provide details on our proposed approach for posterior sampling called NPTL, tailored for common transfer learning scenarios involving single pre-trained parameter. The key feature of these situations is that downstream solutions are typically found near the pre-trained pa- rameter (Neyshabur al., 2020). Consequently, this aligns with the second scenario described in $ 2.3, which focuses on exploring local mode centered around the pre-trained parameter.
3.1 BASE MEASURE
To form our initial estimate F, in the DP prior, it is essential to incorporate our prior understanding of the true data distribution Fg. This prior knowledge can be derived from the upstream task in trans- fer learning scenarios, which we outline below. Formally, we define an informative base measure F (Ir, y) by utilizing the empirical distribution of the upstream dataset (1 v,, Ji=1
up
1 Rap
F (z =
Σ
S ( (2.
Trp i=1
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Nonparametric Transfer Learning.
Require: Pre-trained feature extractor parameters o and the linear-probed head W for down- stream data ( y/ HEs- The linear-probed model Sprobed is parameterized by (Ø" W
Ensure:
Downstream posterior samples {@(m) 1m=1-
1: Define [x) prohed (x)).
2: for m = to M do
3: Generate prior pseudo-samples (zi, fx fx;))
4: Randomly construct index mapping function u [n] [4] x [4]
5: Draw (0i x(m)) 2L Dir (1. ................................
6:
Replace (Wi..... Wn, W.,..., wn) ) = (Bu(I)I))....... [1) 11 ..... Bu(a))2) w)
Optimize ((m²) arg ming { [ (m)
((0:z,. + Σi= t6 zk, (x())
7:
8:
end for
where 0(x.y) denotes a discrete point measure at (r, v). However, in many real-world machine learning scenarios, this approach faces challenges for two primary reasons. Firstly, the upstream datasets are often kept private and undisclosed, e.g. JFT-300M and JFT-3B datasets (Sun et al., 2017). Secondly, handling such extensive datasets can demand substantial memory resources, mak- ing it unfeasible to directly build the empirical distribution.
To overcome these issues, we introduce an alternative approach that solely depends on pre-trained parameters and downstream data for constructing informative base measure. Specifically, first partition the model parameters into feature extractor parameters & and task-specific head parameters W, i.e. 0 (Φ, W), address the variability in output dimensions across different tasks. Next, we utilize linear probing to transfer the upstream knowledge to the downstream setting, as the pre-trained feature extractor parameters d² should capture the essence of the upstream data generation process. Using the linear-probed model fpoobed and downstream inputs ( we define the informative base measure F (I. y)
F F+ v) = Σ 1=1
(mttaaaa (2,, U).
(3)
Our DP prior on F, then becomes DP(a, F2), where is the strength of our belief. The value of a chosen through a validation set to regulate the proper degree of our belief, outlined below.
Empirical Bayes. We highlight that both Fx and in the DP prior dependent on the downstream data D, which indicates that our method can be understood as an empirical Bayes approach (Rob- bins, 1956). In particular,
key aspects: 1) the choice of loss function and the data for training W and 2) the selection of an appropriate a. To ensure that fprobed emulates the downstream data generating process based on our prior knowledge, function !(0; xr y). that is the negative log-likelihood, on the downstream dataset to learn W. For the selection of a, we minimize the negative log- likelihoooo on held-out validation set which con- stitutes 10% of the entire training dataset, which has close connections to maximizing the marginal likelihood (Fong & Holmes, 2020).
address
employed the same loss
3.2 BLOCK DIRICHLET DISTRIBUTION
Recall the posterior of F after observing the downstream dataset D {0r, 2e
FID DP n, a T2 Σ ((ss)))
F+
(4)
a a
When dealing with continuous base measure, sampling exactly from the posterior DP is not pos- sible with finite computational resources. Fortunately, the informative base measure Fz defined in Equation 3 only involves finite inputs from the downstream dataset. This allows exact sam- pling using finite Dirichlet distribution, where we only need to sample weights (Wisn W1.)) from Dirichlet distribution defined over R2n with concentration parameters (1,...>
4

---

Published as a conference paper at ICLR 2024
Here, Winn and Win denote the weights associated with the downstream data points and the pseudo data points originating from the base measure Fa respectively.
The provided Dirichlet distribution, denoted as Dir(I,..... 1, aaaaa.....,/a............ generates weight vector (Wi:ns win) with a dimension of 2n. However, due to the numerical issues, accurately generating these weight samples becomes challenging when the number of downstream training data points, denoted as n, becomes large. To address this issue, we adopt a block Dirichlet dis- tribution (Shin et al., 2021) to handle the dimensionality of the concentration parameter. To use a block Dirichlet distribution as an alternative to the non-block version, we need to map each el- ement in the set [n] to two pairs of blocks as we independently block the downstream dataset and the pseudo dataset. This mapping is performed by a function u [n] - [L] x [4], where u(1)) (1, Here, we denote u(i) [] as 4 and u(i)[2] as 12- We assign weights w; = Vu(iJu for the ith training data point and wi is assigned as (1(()22] for the ith pseudo data point, where
(U1.Z, 01.1) Dir(1,...., 1,a/,,...................... a/n) in I... After finishing mapping (W1:n, WI:n), the sum Ej=1 w/ + Rk=1 Wi becomes 2 this point. This index mapping function is randomly constructed for each posterior sample. Please refer to Appendix A for details on the implementation and analysis regarding the asymptotic convergence between the target distributions of the blocked Dirichlet distribution and the non-blocked Dirichlet distribution.
3.3 SAMPLING FROM POSTERIOR DISTRIBUTION
To perform posterior sampling, we need to sample total of M instances of F(m) from the dis- tribution F,D, where m ranges from to M. We then need to solve M instances of Equation 2, substituting ((m) for Fg. Since our measure has finite support, our objective can be expressed as:
J ((;: ()dF(m) (I, = w E(O E(0; Bj)
(m) k
1(0; fprobed (zz)).
(5)
As finding the global minima is impractical for deep neural networks and using the entire dataset demands significant memory resources, we have employed Stochastic Gradient Descent (SGD; Rob- bins & Monro, 1951) methods to optimize the objective described in Equation 5. In order to stabilize the optimization procedure, we maintain the scale of the sum Σラ= w/m") Σx= wi(") as 2n by
multiply 2T. to the variables (u(m)) (((m). This scaling adjustment can be easily achieved by sam- pling from distribution, denoted (m) , (ffm) 2L x Dir(1,.... .,,11,,, a,nn........ a/n). Unlike posterior sampling methods based on MCMC, our approach does not require sequential sampling for each instance, allowing for parallelized posterior sampling. Please refer Algorithm 1 for summary of the NPTL algorithm.
4 RELATED WORKS
Posterior sampling in deep neural networks. While there exist several posterior sampling meth- ods guaranteed by statistical theory, e.g. Hamiltonian Monte Carlo (HMC; Duane et al., 1987; Neal et al., 2011), they are ill-suited for the scale of modern deep neural networks. Consequently, recent advances in Bayesian deep learning have been established upon SGD based methods (Robbins & Monro, 1951), a foundation of modern machine learning algorithms. We can highlight the follow- ing research areas for Bayesian deep learning: 1) Variational inference introduces variational dis- tribution, which are tractable approximate posteriors, and then samples from them (Graves, 2011; Ranganath et al., 2014; Blundell et al., 2015; Khan et al., 2018). 2) Stochastic gradient Markov chain Monte Carlo directly explores the posterior distribution using stochastic gradients by using various continuous dynamics (Welling & Teh, 201 I; Chen et al., 2014; Ma et al., 2015). 3) Particle optimization variational inference evolves a set of particles towards the high-density regions of the posterior distribution (Liu & Wang, 2016; D' Angelo & Fortuin, 2021). However, most of the inves- tigations into posterior sampling are conducted in from-scratch scenarios, where models start with random initializations, and it is considered crucial to specify suitable priors even in the absence of observed data (Fortuin, 2022). Consequently, in transfer learning situations that utilize pre-training data, the significance of establishing priors that consider this becomes even more pronounced.
5

---

Published as a conference paper at ICLR 2024
Learning priors from data. Several works have proposed to learn prior distribution from data. The work by Krishnan et al. (2020) focuses on adapting a prior distribution by maximizing the marginal likelihood using Maximum Likelihood Estimation solution. On the other hand, Fortuin et al. (2022) takes a different approach by using actual summary statistics of SGD trained weights as weight prior distribution. Tomezak & Welling (2018) proposed Variational Mixture of Pos- teriors (Vamp) prior, which utilizes mixture of variational posteriors learned from data as a prior. While these approaches aim create carefully tailored weight prior distribution based on data, our method establishes an informative nonparametric prior to the true data distribution. This distinction enhances NPTL's resilience for the model misspecification scenarios.
EXPERIMENTS
In this section, we present empirical evidence that demonstrates the effectiveness of our proposed posterior sampling method in practical transfer learning scenarios. $ 5.1 includes experiments con- ducted on vision tasks, while 5.2 contains experiments conducted on language tasks. In Š 5.3, we suggest computationally efficient method to decrease inference cost while maintaining the perfor- mance of NPTI.. Across all the result tables, a boldfaced underline highlights the best value, while an underline denotes the second-best value in each column. The last 'Avg.' column summarizes the overall results for each method across all datasets or intensity levels.
As baseline for assessing the quality of the posterior samples from our proposed NPTL method, we consider the following two representative algorithms for practically implementing BMA; 7) Stochas- tic Gradient Hamiltonian Monte Carlo (SCHMC): It is an approximate way simulate HMC by employing stochastic gradients and is guaranteed to have the same stationary distribution as the original HMC (Chen et al., 2014). Unless otherwise indicated, we utilized 30 posterior samples from the SGHMC sampler for calculating BMA. 2) Ensemble: Despite its simplicity, it serves as a compelling approach to the practice of BMA (Wilson & Izmailov, 2020). Unless specified, we employed 10 model copies to make ensemble predictions. In the baseline BMA procedure, we em- ployed L2SP (Xuhong et al., 2018) and PTYL (Shwartz-Ziv et al., 2022) approaches, which provide Gaussian priors centered around the pre-trained solution. Appendix B.2 provides more detailed explanation of these techniques and their specific hyperparameter settings.
5.1 EMPIRICAL ANALYSIS ON VISION TASKS
Our experimental setups for visual tasks encompass the following pre-trained models: /) ResNet- 20x4, a residual network with depth of 20 and projection shortcuts (He et al., 2016). To better adapt to the transfer learning scenario, we enhanced the model capacity by increasing the number of convo- lutional filters by four (Zagoruyko & Komodakis, 2016). We then performed supervised pre-training using the downsampled variant of the ImageNet dataset, where the image size is 32x32 (Chrabaszez et al. 2017). 2) ResNer-50, residual network with depth of 50 and projection shorteuts (He et al., 2016). Fine-tuning the ResNet-50 network, pre-trained on the ImageNet dataset with a 224x224 image size (Russakovsky al., 2015), is a well-established procedure in computer vision experi- ments (Girshick et al., 2014; He et al.. 2019). 3) ViT-B/16, a vision transformer (VīT) with 16 x 16 input patch size (Dosovitskiy et al., 2021). Considering the recent advancements in computer vision achieved through ViT architectures, we also contemplate scenarios in which we fine-tune the ViT- B/16 network, preitrai on the ImageNet-21k dataset with a 224x224 image size. We believe the experimental results obtained through these setups will be convincing to the readers.
We verify the effectiveness of our posterior sampling method on range of downstream image clas- sification tasks when utilizing the aforementioned pre-trained model in transfer learning context. In addition to measuring classification accuracy (ACC), we also assess the performance of resulting categorical predictions using negative log-likeline likelihood (NLL). NLL provides additional insights into the quality of the posterior predictive distribution obtained through the BMA procedure. Throughout the paper, the values displayed in the tables represent averages with standard deviations calculated over multiple runs. Please refer to Appendix for a thorough description of downstream datasets, evaluation metrics, and precise hyperparameter information.
Results for image classification tasks. We start by evaluating the performance of NPTL on image classification tasks, using the ResNet-20x4 model for the following four datasets with image size

---

Published as a conference paper at ICLR 2024
Table 1: Main results with BMA for ResNet-220x. Evaluation results of NPTL and baseline methods for BMA on four image classification datasets, including C10, C100, F101, and D120. Results with standard deviations represent the average of three runs.
Datasets
Metric
Methods
C10
C1CO
F101
D120
Avg.
SGHMC L2SP
0,95520.000
0.797a0.00
0.625
0,61240:011
0.747
SGHMC PTYL
0,958a0001
0.801
0.632a0001
1
0.751
ACC (1)
Ensemble L2SP
0.963:0.000
0.81550.000
0.646::0.001
0.62410:003
0.762
Ensemble PTYL
0,95810.001
0.600000000
0, 0.600s0.008
0. 0.63240.022
0.749
NPTL. (ours)
0.964-8.001
0.818-0.00
0.64440.002
0.634-00001
0.765
SGHMC L2SP
0.138000000
0.686x0.000
1.447a0.001
1.390
0.915
SGHMC PTYL
0. 128
0.66510.001
1. 1.420s0.000
1.385
0,900
(1)
Ensemble L2SP
0. 007000000
0.617+0000
+.330
1.357=0.000
0.853
Ensemble PTYL.
0. 0..00000
0.644-0000
1.513+0.000
1.320-n.00
0,899
NPTL (ours)
0.102
0.606s00002
1.347a0001
1.297
0.838
Table 2: Main results with BMA for ResNet-50.
Evaluation results of NPTL and baseline meth- ods for BMA on five image classification datasets, including B200, C1 C101, D47, F 02, and P37. 7. Results with standard deviations represent the average of three runs.
Datasets
Metric
Method
3200
c101
D47
F102
P37
Avg.
SGHMC + L2SP
0.78220.001
0,88720001
0.69920.008
0.903s0.001
0.930.0.00
0.840
SGHMC + PTYL
0.784=0001
0.884
0.697=0.004
0.904xn.011
0,930-aum
0.840
ACC (1)
Fasemble + I.2SP
0.807
0.899.0007
0.702-0.000
0.918x0.022
0.924+amm
0.850
Ensemble + PTYL
20805
0.895a00us
0. 0.704=0.008
0.917a0.000
0,925=am03
0,849
NPTL (ours)
0.811
0.901
0.709x0.002
0.921
0.930.0.00:
0.854
SGHMC + L.2SP
0. 0.868x0.001
0.350
1.209
0.401
0.240
0.614
SGIIMC + PTYL
20.03
0.358x0024
1.22990.005
0,39240.000
0.235
0.615
NLL (4)
Ensemble L2SP
0.773
0.318+0023
1.295=0.00s
0.327xn.02
0.260+am06
0.595
Ensemble PTYL
0.773
0.33
1.288
0.323an.001
0.253
0,594
NPTL (ours)
0.738
0.317
1.16720.02
0.313
0,234zouxz
0.554
of 32 32: 1) CIFAR-1O. 2) CIFAR-100, 3) Foods-101, and 4) Stanford Dogs. For brevity, these datasets are abbreviated as C1 0, c100, F101, and D120, respectively. Table clearly demonstrates that NPTL outperforms the SGHMC baselines, which serve as strong competitor in posterior sampling. Although it is widely accepted that ensembles of deep neural networks typically achieve superior performance compared to Bayesian methods (Ashukha et al., 2020; Ovadia et al., 2019), NPTL demonstrates competitive outcomes in terms of ACC and outperforms them in terms of NLL when compared to the ensemble baselines.
We further verify the scalability of our approach through experiments involving the ResNet-50 and VTT-B/16 models. These experiments conducted the following five datasets with an image size of 224 x 224: 1) Caltech-UCSD) Birds 200, 2) Caltech-101, 3) Describable Textures Dataset, 4) Oxford Flowers 102, and 5) Oxford-IIIT Pet. To simplify, we use the abbreviations 3200, C101, D47, F102, and P37 for these datasets, respectively. The results presented in Table 2 and Table exhibit consistent trend with the ResNet-20x4 case. is evident that NPTL demonstrates signifi- cant performance advantage over posterior sampling baselines and surpasses the ensemble baselines. These experimental results highlight that NPTL exhibits superior posterior sampling quality in terms of BMA performance in transfer learning scenarios for image classification tasks.
Robustness to common corruptions. We also assess calibration performance by utilizing CIFAR- 10-C, corrupted version of CIFAR-10 (Hendrycks & Dietterich, 2019). This benchmark aims to evaluate the robustness of CIFAR-10 10 classification models when exposed to 15 common corrup- tions across five different severity levels. We used posterior samples or ensemble members from the CIFAR-10 classification task to evaluate the performances. Table 4 clearly shows our approach consistently outperforms other baselines across all intensity levels and evaluation metrics we mea- sured. This result shows that posterior samples from NPTL are more robust on distribution shift between the training dataset and the evaluation dataset compared other baselines. Please refer

---

Published as a conference paper at ICLR 2024
Table 3: Main results with BMA for VIT-B/16. Evaluation results of NPTL and bascline methods for BMA on five image classification datasets, including B200, C1C1, D47, F F102, and P37. Re- sults with standard deviations represent the average of three runs.
Datascts
Metric
Method
B200
C101
D47
F102
P37
Avg.
Ensemble L2SP NPTL (ours)
0.876
0.905=0.001
0.765
0.992.0.000
0.939
0.895
ACC (t)
0.885
0.923-0.01
0.770xo.02
0.992
0.944
0.903
Ensemble L2SP
0.457
0.290s0.022
1.005
0.048
0.200a0001
0,400
NLL (4)
NPTL (ours)
0.420.0.000
0.264-0.05
0.941 40.006
0.049:0.001
0.184:0.003
0.372
Table 4: Results on CIFAR-10-C benchmark for ResNet-20xx. Evaluation results of NPTL and baseline methods for BMA under five different levels of common corruptions. Results with standard deviations represent the average across 15 different corruption types.
Intensity levels
Metrics
Methods
2
Avg.
SGHMC 1.2SP
0.844+a
0.785 136
007
0.667-0.290
0.574-0.236
0.721
SGHMC PTYL
0.849a0144
0.79
0.740a0.06
0.670so.26
0.575+0.243
0.725
ACC (1)
Ensemble L2SP
0.875#a116
0.823s0.15
0.778=018p
0.713=0.21
0.625+0220
0.763
Ensemble PTYL
0.85320137
0.797.0.170
0.749
0,68
0.59
0.734
NPTL. (ours)
0.89
0.846
0.805
0.750=0.19
0.672
0.793
SGHMC L2SP
0.475=0445
0.643ao.53
0.789
1.016
L.339+0.79
0.852
SGHMC PTYL
0.46520.55
0.63120.522
a.777
1.01720755
1.345a0.829
0.847
NLL ())
Ensemble I,2SP
0.432x0455
0.62 so
+.798
1.076x0.880
1.455x0.981
0.876
Ensemble PTYL
0.499
0.703 666
0.887
1.185
1.55621.01
0.966
NPTL. (ours)
0.348
0.500=0455
0.645=0.992
0.855
1.14+00757
0.698
to Appendix B.3 for more detailed results, including the Expected Calibration Error (ECE; Naeini al., 2015) metric for measuring the calibration.
5.2 EMPIRICAL ANALYSIS ON LANGUAGE TASKS
We further confirm that the applicability of our proposed approach beyond vision models by con- ducting experiments on language tasks. More precisely, we perform the BMA procedure when fine-tuning the pre-trained RoBERTa-Boo Base model (Liu et al., 2019) for the following three subtasks from the GIUE benchmark (Wang et al., 2019): /) CoLA, 2) MRPC, and 3) RTE. These datasets have relatively limited training data (8.55k, 3.67k, and 2.49k, respectively), making the transfer learning process particularly crucial.
Table 5 provides evidence that our proposed NPTL results in decrease in NLL, implying that it offers an enhanced approach to compute BMA in comparison to the ensemble baseline. addition to assessing NLL, which measures the quality of the posterior samples used in the BMA process, we also present results using specific evaluation metrics tailored to each dataset within the GLUE benchmark. It is worth noting that NPTL. method also outperforms in terms of GLUE metries. Here, we have omitted the SGHMC baseline because practical experiments have shown that us- ing SGD with momentum in transformer architectures yields subpar outcomes (Zhang al., 2020; Liu et al., 2020). However, we believe the Ensemble approach would serve as a strong baseline, demonstrated before in vision experiments. Please refer Appendix B, for more information regarding datasets, evaluation metrics, and specific hyperparameter details.
5.3 SOUPS OF NPL SAMPLES
Despite the theoretical basis of the BMA procedure, a factor that hinders its practicality is the in- ference cost the multiple forward passes required for multiple model copies in BMA calculations make applications in resource-com consurcieec real-world deployments challenging. Several methods have been suggested mitigate this cost issue in ensemble modeling (Izmailov al., 2018; Malinin et al., 2020; Hobbhahn et al., 2022), and we further introduce NPTL-Soup. a practical variation of NPTL involving weight averaging of posterior samples from NPTL., in line with those attempts.
8

---

Published as a conference paper at ICLR 2024
Table 5: Main results with BMA for RoBERTa-Base. Evaluation results of NPTL and baseline methods for BMA on three text classification datasets, including cola, mrpc, and rte.. Results with standard deviations represent the average of three runs. IIt denotes the use of commonly em- ployed evaluation metrics for each dataset. See Appendix B for further details.
Datasets
Metric
Method
cola
nrpe
rte
Avg.
Ensemble + L2SP NPTL (ours)
0.64320.03
0,930.0.005
0.794
0.789
GLUE (t))
0.645:n006
0.934
0.812+0.006
0.797
Ensemble + 1.2SP
0.452+n.00
0.263+0.000
0.51 8+0.09
0,411
NLL (L)
NPTL (ours)
0.397
0.245+0.00
0.472
0.371
Table 6: Results for soups of NPTL posterior samples. Evaluation results of NPTL-Soup com- pared to NPTL. NPTL-Soup requires only one forward pass, resulting in about ten times lower inference cost than the BMA computation of NPTL. 'It represents the best value of each metric for each test dataset among the ten fine-tuned models, which can be seen as an unfair advantage given to the 'Fine-tune' baseline.
Datasets
Metrics
Method
B200
C101
D47
F102
P37
Avg.
NPTL
0.901=0002
0.709s0002
0.921=0mm
0.930
0.854
NPTL-Soup
0.800
0.894=0004
0.705=o004
0.917-000
0.923+o.us
0.848
ACC
Fine-tune
0.780+a006
0.879=0009
0.681
0.909-om0
0.906=088
0.831
Fine-tune (oracle)
0.785
0.893
0.695
0.912
0.921
0.841
NPTL
0.738.0.002
0.317
1.167.0.002
0.313-0001
0.234.0.002
0.554
NPTL-Soup
2.755
0.357
1.329
0.321
0.25420.009
0.603
NLL
Fine-tune
0.88 12.011
0.42420057
1.683
0.358-0016
0.34920.03
0.739
Fine-tune (oracle)?
0.862
0.340
1.612
0.334
0.288
0.686
Notably, in the transfer learning scenario, fine-tuned solutions from the same initial pre-trained model tend to reside the same basin on the loss surface (Neyshabur et al., 2020), and we can thus obtain a single solution achieving competitive performance by appropriately averaging weights of NPTL posterior samples. We implemented this NPTL-Soup procedure by employing the Greedy Soup algorithm (Wortsman et al., 2022).
Table demonstrates that NPTL-Soup can achieve performance competitive to that of the BMA computation while reducing the computational demands. Notably. NPTL-Soup achieves superior results compared to the 'Fine-tune (oracle)' baseline, which represents the best-performing fine- tuned solution in terms of test performance. While NPTL-Soup does not strictly carry out the BMA procedure, its importance lies in offering a practical single solution through the weight-space en- semble of samples with high posterior density over neural network weights. One could apply alternative methods such as ensemble distillation (Malinin et al., 2020) to mitigate the inherent in- ference cost associated with the BMA procedure. However, emphasize that such approaches not the primary focus of our paper, which primarily introduces the posterior sampling method. We consider them as potential avenues for future research.
CONCLUSION
In this paper, proposed novel posterior sampling approach for transfer learning scenarios, which we call NPTL. NPTL leverages the nonparametric learning (Lyddon et al., 2018; Fong et al., 2019) framework. which effectively accounts for the model misspecificationn by adapting nonparamet- ric prior. Our method involves constructing an informative base measure using empirical Bayes techniques and proposing a numerically stable posterior sampling algorithm based on the block Dirichlet distribution. We conducted empirical validations across various tasks and models to vali- date the performance of NPTL. The results consistently demonstrate that NPTL outperforms other existing methods, indicating its superior ability to produce high-quality posterior samples in transfer learning scenarios.

---

Published as a conference paper at ICLR 2024
Ethics statement. This paper does not include any ethical issues. This paper proposes posterior sampling algorithm for the transfer learning scenario which does not cause ethical issues.
Reproducibility statement. We described our experimental details in Appendix B which covers dataset description, used librarics, and hardware.
ACKNOWLEDGEMENT
This work was partly supported by Institute of Information communications Technology Promo- tion(IITP) grant funded by the Korea (MSTT) 2019-0-00075, Artificial Intelligence Graduate School Program(KAISTT), the National Research Foundation of Korea grant funded by the Korea (SST)) 1104-2002211A577733909111111 N and KAIST-NAVER Hypercreative Center.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learn- ing heterogeneous systems, 2015. URL https:/ /www/ www.tetssrffo.oorggg Software available from tensorflow.org. 18
Charles Antoniak. Mixtures of dirichlet processes with applications to bayesian nonparametric problems. The annals of statistics, pp. 1152-1174, 1974.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning
Representations (ICLR), 2020. 7
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojeiech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http: / g thub. com deepmi nd. 18
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge, 2006. 19
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The
fifth PASCAL recognizing textual entailment challenge, 2009. 19
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. Proceedings of The 32nd International Conference on Muchine Learning (ICML), 2015. 5
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision (ECCV), 2014. 19
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/ google/ jaleee/j.. 18
Ciprian Chelba and Alex Acero. Adaptation of maximum entropy capitalizer: Little data can help a lot. Computer Speech & Language, (0(4):382-399 2006. 2
10

---

Published as a conference paper at ICLR 2024
Liang- Chihh Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017. 23
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of The 31st International Conference Machine Learning (ICML), 2014. 5,
Patryk Chrabaszez, Iyya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:/707.066 19, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
19
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp- 177-190. Springer, 2006.
Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Hal Daumé III. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 256-263, 2007.
Daniel A De Souza, Diego Mesquita, Samuel Kaski, and Luigi Acerbi. Parallel meme without embarrassing failures. International Conference on Artificial Intelligence and Statistics, pp. 1786-1804. PMLR, 2022. 2
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005. 19
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 6
Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid monte carlo.
Physics letters B, (95())216-222) 1987. 5
Stefan Flfwing, Fiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018. 19
Thomas S Ferguson. A bayesian analysis of some nonparametric problems. The annuls of statistics, pp. 209-230, 1973.
Edwin Fong and Chris C Holmes. On the marginal likelihood and cross-validation. Biometrika, 107 (2):489 496, 2020. 4
Edwin Fong, Simon Lyddon, and Chris Holmes. Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In Proceedings of The 36th International Conference on Machine Learning (ICML), 2019. 1, 3,3,9 9, 23
Vincent Fortuin. Priors in bayesian deep learning: A review. International Statistical Review, 90(3):
563-591, 2022. 5
Vincent Fortuin, Adrià Garriga-Alonso, Sebastian W. Ober, Florian Wenzel, Gunnar Ratsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network pri-
ors revisited. In International Conference on Learning Representations (ICLR), 2022. 1,
Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparametric Bayesian inference,
volume 44. Cambridge University Press, 2017. 16, 17
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguisties, 2007. 19

---

Published as a conference paper at ICLR 2024
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for curate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 1,
Maarten Grachten and Carlos Eduardo Cancino Chacón. Strategies for conceptual change convo- lutional neural networks. arXiv preprint
2017. 2
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems (NIPS), 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2019. 2,
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error arXiv: 08415, 2016. 19
linear units (gelus). arXiv preprint
Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for clas- sification with bayesian deep networks. In Uncertainty in Artificial Intelligence, pp. 822-832. PMLR, 2022. 8
J. A. Hoeting, D. Madigan, A. E. Raftery, and C. T. Volinsky. Bayesian model averaging: a tutorial. Statistical Science, 14(4):382-40) 401, 1999.
CC Holmes and SG Walker. General bayesian updating and the loss-likelihood bootstrap. Biometrika, 106(2):465-478, 2019. 23
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing in- ternal covariate shift. In Proceedings of The 32nd International Conference Machine Learning (ICML), 2015. 20
P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty Artificial Intelligence 2018, UAI 2018, pp. 876-885, 2018. 8
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In Proceedings of The 38th International Conference on Machine Learning (ICML), 2021.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by pertuttpenttrration in adam. In Proceedings The International Conference on Machine Learning (ICML), 2018. 5
The 35th
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011. 19
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 20
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Specifying weight priors in bayesian deep neural networks with empirical bayes. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4477-4484, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. 18
12

---

Published as a conference paper at ICLR 2024
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems (NIPS), 1991. 2
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022. 19
Hao Li, Pratik Chaudhari, Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations (ICLR), 2020.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the diffi- culty of training transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 5747-5765 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in Neural Information Processing Systems (NIPS), 2016.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907:19997.1192 2019. 8, 19
Simon Lyddon, Stephen Walker, and Chris C Holmes. Nonparametric learning from bayesian mod- els with randomized objective functions. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 1, 3, 9, 24
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient meme. In Advances in Neural Information Processing Systems (NIPS), 2015. 5
W. Maddox, Garipov, P. Izmailov, D. Vetrov, and A. G. Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 2
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In Inter- national Conference on Learning Representations (ICLR), 2020. 8,
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In International Conference Learning Representations (ICLR), 2013.
Ulrich K Müller. Risk of bayesian inference in misspecified models, and the sandwich covariance matrix. Econometrica, 1005518805, 2013. 24
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. 8
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011. 5
Willie Neiswanger, Chong Wang, and Eric P Xing. Asymptotically exact, embarrassingly parallel meme. Proceedings of the Thirtieth Conference on Uncertuinty Artificial Intelligence, pp- 623-632, 2014. 2
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in transfer learn- ing? In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008. doi: 10.1 10111//VV//..00 47. 19
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snock. Can you trust your model's ' uncertainty? evaluating predictive uncertainty under dataset shift. In Advances Neural Information Process- ing Systems (NeurIPS), 2019.
13

---

Published as a conference paper at ICLR 2024
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
19
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 4(5):1-17, 1964. 20
Jens Præstgaard and Jon A Wellner. Exchangeably weighted bootstraps of the general empirical process. The Annals of Probability, pp. 2053-2086, 1993. 18
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv: 2017. 19
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. intelligence and statistics, pp. 814 822. PMLR, 2014. 5
Artificial
H. Robbins. An empirical Bayes approach to statistics. Berkeley Symposium Mathematical Statistics and Probability, 1956. 4
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225 2015. 6, 19
Minsuk Shin, Hyungjoo Cho, Hyun-seok Min, and Sungbin Lim. Neural bootstrapper. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 5, 16, 17, 18
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew Gordon Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 6, 21, 23
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch de- pendence in the training of deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 20
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable fectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843 852, 2017.
Tomczak and M. Welling. VAE with a VampPrior. In Proceedings of The 21st International Conference Artificial Intelligence and Statistics (AISTATS 2018), 2018. 6
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958 1970, 2008 doi: 10.1 109/TPAMM..000 128. 18
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd
birds-200-201 dataset, Jul 2011. 19
Stephen G Walker. Bayesian inference with misspecified models. Journal of statistical planning and inference, 143(10):1621-1333 2013. 3, 23
Ruosi Wan, Haoyi Xiong, Xingjian Li, Zhanxing Zhu, and Jun Huan. Towards making deep transfer learning never hurt. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 578- 587. IEEE, 2019.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019. URL https: htk..ttt////peeeeiiiii /ppnreview. fett/forum? i dd= r 444kk255t 7.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805. /2471, 2018. 19
14

---

Published as a conference paper at ICLR 2024
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of The 28th International Conference on Machine Learning (ICML), 2011.
F. Wenzel, K. Roth, B. S. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Je- natton, and Nowozin. How good is the Bayes posterior in deep neural networks really? In Proceedings of The 37th International Conference Machine Learning (ICML., 2020. 1,20
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 6
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-aa art natural language processing. In Proceedings of the 2020 Conference Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https:/ /www.w............./.... a acl web. org/ aatholooyyy. 2020. momml--dmoss 6. 18
Empirical Methods in
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time. Proceedings The 39th International Conference on Machine Learning (ICML), 2022. 9
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of The 35th International Conference on Machine Learning (ICML), 2018. 2
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, In Procedings of the British Machine Vision Conference 2016. British Machine Vision Association, 2016. 6
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems (NeurIPS), 2020.
15

---

Published as a conference paper at ICLR 2024
A DISCUSSION ON BLOCK DIRICHLET DISTRIBUTION
A.1 IMPLEMENTATION
Let Ii...,, I denote the randomly constructed pairwise disjoint index sets where U 1, [n] where [1,] = E for all i E [L] and n is the number of the training dataset. Furthermore, let
....... .I denote another randomly constructed pairwise disjoint index sets where U I [n]
where |1 - - for all i e [L]. Now let index mapping functions u : [n]] + L x L where l and u(i)
u(i) (1, 12) i It, and It.- Here we denote u(i)[1]
l2. Then,
given some scaled Dirichlet weights (V1...., Cn U1...., UL) ~ 2L x Dir(1,.... 1, a.,,..., a/n) on
R²L we assign the weights for the ith training datum and i-th pseudo datum as wy [x( and
W0; = [2
A.2 CONVERGENCE
In this section, we will show that a simplified version of the blocked posterior bootstrap is asymp- totically equivalent to the non-blocked bootstraps based on the assumptions in Theorem A.1 of Shin et al. (2021). The simplification involves keeping T fixed and not blocking the pseudo data. We leave the extension of our theory to the implemented case for future work. Formally, let / /,,..., Tr.
denotes the randomly constructed pairwise disjoint index sets where U J, = [n] and n is the size of the training dataset, and again [T]] = L for all i E [L]. Now define the index mapping functions u: [nx] + [L] where u(t) i E 4t Here we denote uui) I. Then, given some Dirichlet weights (ui.... Vn. .... ir) ~ Dir(1 ..., 1, ar a//T,
a/T) on TR we assign the weight w; Pu(s) for the i-th training datum and w ūj for the j-th pscudo datum. We have omitted the scaling term for the theoretical study, and assume that L is a fixed constant.
We assume that the data points Zı, Z2... are i..dd from the probability measure Po. where write Z = (X,, Yj). We then denote the empirical probability measure for the n observations by P,, Ei=ı ōz,- For any probability measure P and P-measurable function g, let Pg denote I gdP. The non-blocked weighted posterior distribution be written
72
F
65 j=1
Fa and (Si...., Sn, "........777 (r) ~ Dir (1,...,,, o......... a//T) independently. Here, we assume a and T are constant and Fş is fixed distribution from which we draw T samples. We will also write f- 1 Σラ oz,- The blocked weighted posterior distribution is then
where 73 l.d.
T
FB
wrōz,
EL Wi + E
i=
where (Wi,..., Wny W1 WT } is described just above. Note that we have
n
T wi Σ ii
Σ
- 1 (",
U4-
We will now show that F and Fg are asymptotically equivalent.
First, we study F following Theorem 12.2 from Ghosal & Van der Vaart (2017).
Proposition A.1 (Ghosal & Van der Vaart (2017)). Let H be class of functions that is -Donskee with envelope function H2z) = suPREM [h(2)] which satisfies both P [ < co and F [H?] < co. The process vnl (F Pn). where F is the non- blockee weighted posterior, converges conditionally in distribution given Z, Zz.... .. in Loe (F) to G as n - 0o Po -a.s,, where G a Brownian bridge process.
16

---

Published as a conference paper at ICLR 2024
Proof. First, note from Proposition G.10 of Ghosal & Van der Vaart (2017) that F can be written as
= V + (1 Vn,
where V, Be(a, nt), B, DP Pr) and Q DP (a, Fx are independent on an appropriate
product probability space. Here, Be(a, () denotes the beta distribution with parameters a and B. The process can then be written as
vT((F= - PPn) = vVV, (Q - BB,) vn (BB Pn).
For the first term on the right, vnV, -> O in probability, and TB,, converges to a limit in Loe (F). As the mapping h Qh is in Loo (F), the process vnV, (Q B,,) converges in probability in Loo (F).
The second term vn (B, PP,) is well understood and converges conditionally in distribution given Z1, ... to G in (F) Po--a.s. -a.s., and applying Slutsky's 's theorem gives the result.
This Proposition A.I shows that (n(F- ...) converges weakly to G when + Now, we study FB with a similar process Proposition A.I, following Shin et (2021).
Proposition A.2. Let H. be class of functions that is Po-Donsker, with envelope function H(2) Sшрдеи [h(2)] which satisfies both IP [P] < oo and [+[H²] < co. Assume that - 0o as - Then the process vn (FB Pa), where FB is the blocked weighted posterior, converges conditionally in distribution given Z1, Zz.... ... in Lo (F) to G as n - 00 Po -a.s, where G is a Brownian bridge process.
Proof. Note that FB can be written as
FB= V2Q + (1- Bn
where Q ~ DP (a, Fx) as before, and
B,
82, 2=1
where (m....,, mL) Lx Dir(1,... .... 1). Finally, V, G/(G+ + H₂) where have independently G ~ Ga(o, 1), H, ~ Ga (L, L/n). As before, Va. Q and Bn are all independent on an appropriate
product probability space.
To see this, we note that FR can be written as
n
FB
Σ j=1
Yj oz
Tu(ii) 8z
1+
1)
Σ 24
7
7
Yj 8%
Tu(s) 8z
+
a
j=1
Ga(1, and > = > and 7 = 3y. Here, Ga(a, B3 denotes the Gamma distribution with parameters a and 3. This follows from the Gamma construction of the Dirichlet distribution. Following the proof of Proposition 10 of Ghosal & Van der Vaart (2017), define
where 71.r 1.d.
Ga(a/T, 1), 71.L
L.d. ~
L
7o(1)
V
B
Oz.-
7 + R
is clear that the above has the correct marginal distributions. From Proposition G.2(i) of Ghosal & Van der Vaart (2017), note that Q is independent of the normalizing constant 7 (and obviously of
17

---

Published as a conference paper at ICLR 2024
YiLL and y). Likewise, IB, is independent of (and of Y1:1: 7) so Vn. Q and B,, are all independent. Finally, we have
V₂Q 7+ T
Y
22
and similarly
(1 - 1B
Tu(3)02.
7 T.
which together gives FB ViQQ (1 Vn IB. The process under study can then be written as
Vn(FB P..) vnV, (Q- BB, vn B -P.)
The second term vn Bn P., is shown to converge conditionally in distribution given Z1, Z to G in Loo (F) Po- -a.s. -ass. in Theorem A.I of Shin et al. (2021) under our assumptions. This relies on the fact that [Mu(1):.... Mu(n) are exchangeable due the random allocation of data points into groups and the result of Præstgaard & Wellner (1993).
For the first term the right, we have that VVVn + in probability. To see this, note that
G
G H
BVV E
< E
G H
E
[G] a
E H
-.
As result, VnE(Vn] + O, and from Markov's inequality we have vVv, 2. 0. As before, this
gives VV @ Bn + O in probability in Los (F), which gives the desired result.
As FF and / FB P.)) converge to the same limit, we can also apply the rest of Theorem A.1 of Shin et al. (2021) under appropriate assumptions. This shows that the target distribution of a simplified version of the blocked posterior bootstrap is asymptotically equivalent to the target distribution non-blose blooklocked bootstrap.
B SUPPLEMENTARY MATERIAL ON EXPERIMENTS
Our code is constructed using the following libraries, which are available under the Apache-2.0 licence': JAX (Bradbury et al., 2018), Flax (Babuschkin et al., 2020), Optax (Babuschkin et al., 2020), TensorFlow Datasets (Abadi et al., 2015), and Transformers (Wolf al., 2020). We intend to make the code available to the public once the research has been published. All experiments were conducted on NVIDIA RTX 3090 GPU machines.
B. EXPERIMENTAL DETAILS
Datasets for ResNet-20x4 experiments. The ResNet-20x4 experiments make use of the following image classification datasets. The network processes input images with dimensions of 32 x 32 x 3, and all the images standardized through the subtraction of (0.481, 0.458, 0.408) and division by (0.269, 0.261, 0.276).
C1 0 and c1 00 CIFAR-10/100" (Krizhevsky al., 2009) consists of 10/100 classes sourced from 80 Million Tiny Images (Torralba et al., 2008), and every image in this dataset has sizes of 32 x 32, We allocated the 60,000 images publicly available into splits of 45,000 for training, 5,000 for validation, and 10,000 for testing.
18

---

Published as a conference paper at ICLR 2024
F101: Food-101 1010 (Bossard et al., 2014) originally consists of 101 food categories, and the images in this dataset have a maximum side length of 512 pixels. We center-cropped the images, resized them to 32 32, and divided the 101,000 publicly available images into .444 for training. 14,310 for validation, and 25,250 for testing.
D120: : Stanford Dogs" (Khosla et al., 2011) comprises pictures of 120 dogs breeds sourced from ImageNet (Russakovsky et al., 2015). We conducted center-cropping on these images, resized them to 32 x 32, and partitioned the publicly available set of 20,580 images into 10,240 for training, 1,760 for validation, and 8,580 for testing.
Datasets for ResNet-50 and ViT-B/16 experiments. The following image classification datasets are employed in the ResNet-50 and ViT-B/16 experiments. Both neural network architectures handle input images with dimensions of 224 x 224 3, and all these images are standardized by subtracting (0.481, 0.458, 0.408) and dividing by (0.269, 0.261, 0.276).
Caltech-UCSD Birds 200 (Wah et al., 2011) comprises pictures of 200 bird species, where some images overlap with images in ImageNet (Russakovsky et al., 2015). We allocated the 11,788 images publicly available into split of 5,394 for training, 300 for validation, and 5,994 for testing.
3200
C101: Caltech-101 (Li et al., 2022) comprises 101 categories, where the images generally have edge lengths ranging from 200 to 300 pixels. We divided the 9, 144 publicly available images into subsets of 2,754 for training, 306 for validation, and 6,084 for testing.
D47 Describable Textures Dataset (Cimpoi al., 2014) consists of 47 categories, and the image sizes range between 300 x 300 and 640 x 640. We utilized a total of 5,640 images, with 1,880 for training, 1,880 for validation, and 1,880 for testing.
F102 : Oxford Flowers 102 (Nilsback & Zisserman, 2008) comprises 102 distinct flower categories commonly found in the United Kingdom. We used a dataset consisting of a total of 8,189 images, splitted as 1,020 for training, 1,020 for validation, and 6,149 for testing.
P37 Oxford-IIIT Pet (Parkhi al., 2012) comprises 37 categories, where all images have an associated ground truth annotation of breed. We divided the 7,349 images publicly available into split of 3,312 for training, 368 for validation, and 3,669 for testing.
Datasets for RoBERTa-Base. The RoBERTa- Base experiments utilize the following subtasks from the GLUE benchmark. The network processes token sequences with a maximum length of 512 tokens, as per the default setup described in Liu al. (2019). In the main text, we reported the performance on the validation split.
cola Corpus of Linguistic Acceptability (Warstadt et al., 2018) comprises 8.55k sen- tences designated for training and 1.04k sentences designated for validation.
mrpc Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) includes 3.67k pairs of sentences for training and 408 pairs of sentences for validation.
xte: : Recognizing Textual Entailment (RTE) originated from a sequence of challenges for textual entailment, including RTEI (Dagan et al., 2006), RTE2 (Bar Haim et al., 2006), RTE3 (Giampiccolo et al., 2007), and RTES (Bentivogli et al., 2009). Consequently, comprises 2.49k pairs of sentences for training and 277 pairs of sentences for validation.
Architectural details ResNet-20x4. In line with Izmailov al. (2021), we endeavored in our ResNet- 20x4 experiments to establish a clear Bayesian interpretation through the following experi- mental configuration; 1) We opted not to utilize any data augmentation techniques. 2) We employed Swish activation (Hendrycks & Gimpel, 2016; Elfwing et al., 2018; 88; Ramachandran et al., 2017) to
iision.ee.ethz.cooo 3 'hups://dataa vision.ee. www.eeeeeet..eeeeeeeeeee www.eet.t......ee.e wtt...ch/ewl/ddaaaetss..o htwp:/sssssssetttrroooo-ooodd
"hup::/vision.staaa" stanford. wwunford.edu/adiig... helluddiyy.///mmggee/Deggg/anmnn
5 'htt:://www. www.wwwwwwwooo..oooow..........
wwwion.caltecc...... www.ech..deeeee www.echh.dddddattttssuubbbb 200 2/11
"https://data.. caltech. adtu/ccordds/mzzzj-6www0)
7 https::/wwwwroobll o...ossooo..cc..ooooooooooo uuk wgg//ataa//dd//iddee...n html
hhttps://www.. wowots.oo.ooooooooooo ox wx..cc.u// www.auuuk////gdaaaa http///g///aa//www//////
9 Phups://wwww wttp://tw.wwotttoooooooooooo ox uk/ wgg//tta/////t//////////////
19

---

Published as a conference paper at ICLR 2024
Table Hyperparameter setup. It summarizes the hyperparameter settings employed in the experimental results presented in the main text.
(a) ResNet-2004
B200
C101
D47
F1 02
L2SP
B 10-7
B 10-7
8 -00-
10
PTYL
? = 10
10
1016
10
NPTL
a 1.0
a 1.0
a 1.0
a 1.0
(b) ResNet-50 and ViT-B/16
B200
c101
D47
F102
P37
I.2SP
B = 10-3
8 -10-3
B == 10
B 3= 10-1
B 10-1
ResNet-50
PTYL
7 = 10
7 y= 11122
= 10
? = 10
7 = 10
NPTL
a 103
a 100
102
10a
a 10a
L2SP
B = 10-4
8 = 10
B = 10-4
B 10
B 10
VÝT-B/16
NPTL
1.0
1.0
1.0
1.0
1.0
(c) RoBERTa-Base
cola
mrpc
rte
L2SP
3 10 2
8 10 2
= 10 2
NPTL
a 10-2
a 10-2
10-2
attain smoother parameter posteriors. 3) We replaced batch normalization layers (loffe & Szegedy, 2015) with filter response normalization layers (Singh Krishnan, 2020). Although these setups might result in reduced performance compared to conventional experimental configurations (such as training the network with ReLU activations, batch normalization layers, and data augmentation strategies), we believe that these setups are likely to produce results that are clearer from a Bayesian perspective. It is worth noting that our ResNet-50 experiments address the conventional configura- tions with ReLU activation and batch normalization layers.
Optimization details on ResNet-20x4 and ResNet-50. We utilized an SGD optimizer with mo- mentum (Polyak, 1964). The momentum value kept constant 0.9, and experimented with different base learning rates using the cosine decay schedule, specifically testing values the range of 10.1, 0.03, 0.01). For the ResNet-20x4 experiments, training terminated after 10 epochs with a batch size of 80. For the ResNet-50 experiments, optimization halted after 5,000 training steps with batch size of 128. These experiments were conducted on single GPU machine. During the SGHMC experiments, we implemented posterior tempering strategy to mitigate the impact of the cold posterior effect (Wenzel al., 2020), employing posterior temperature setting of 0.0001,
Optimization details on ViT-B/16 and RoBERTa-Base. For the ViT-B/16 and RoBERTa- models, we employed an Adam optimizer (Kingma & Ba, 2015). The momentum hyperparameters
were set to their default value from the Optax library, which are 0.9 for the first moment and 0.999 for the second moment. configured the base learning rates at 0.0001 for ViT-B/16 and 0.00003 for RoBERTa-Base. In the case of ViT-B/16 experiments, for RoBERTa-BEE experiments, a linear decay schedule with warmup was used. The optimization terminated after 5,000 training steps with a batch size of 128 for ViT-B/16 experiments conducted on distributed training with two GPU machines. For RoBERTa-Baan experiments, the optimization terminated after 10 training epochs with batch size of 16 single GPU machine.
applied a cosine decay schedule, while
B.2 METHODS AND HYPERPARAMETERS
Table 7 summarizes comprehensive outline of the hyperparameter settings for each experiment.
20

---

Published as a conference paper at ICLR 2024
Table Calibration results for CIFAR-IO-C. Expected calibration error (ECE) of NPTL and baseline methods for BMA under five different levels of common corruptions. Results with standard deviations represent the average across 15 different corruption types. A boldfaced underline high- lights the best value, while underline denotes the top two values in each column. The last 'Avg.' overall results for each method across all intensity levels. supplements Table 4 presented in the main text.
column summarizes
Intensity levels
Metries
Methods
2
Avg.
SGHMC + L.2PP
0.039=0.20
0.050+n.043
0.060=oa71
0.092-0.093
0.138+0 134
0.076
SGHMC PTYL
0.036a0.027
0.048
0.063
0.098
0.149
0.079
ECE (4)
Ensemble L2SP
0.04620.057
0.07 10.008
0.096 102
0.135e0.123
0.191
0.108
Ensemble PTYL
0,051 20.067
0.078
0. 1.104
0.1
0.200
0.116
NPTL (ours)
0.025:0036
0.040
0.058:0076
0.084
0.123:0.10 tos
0.066
strength of prior belief in NPTL. NPTL includes hyperparameter a that signifies the strength of the prior belief. We summarize detailed hyperparameter configuration for each experiment:
In ResNet-20x4 and ResNet-50 experiments, we swept over the logarithmically spaces values of {10*}a.
In VTT-B/16 experiments, set constant value of 1.0 without any sweeping.
In RoBERTa-Bass experiments, ent values of a E {1.0, 0.1, 0.013.
to constant value of 0.01 after exploring differ-
prior variance L2SP and PTYL. Consider a Gaussian prior over neural network parameters & denoted p(Φ) = Ni μt, E), where μ represents the mean and E signifies the covariance. (1) The L.2SP prior corresponds to scenario where hyperparameter responsible for adjusting the prior
Φ and Σ = ,T. Here, & denotes
the pre-trained parameters, and / serves variance. (2) The PTYL prior corresponds to a scenario where = Φswa and Σ >>iswa Here, ΦSwA and ESSWAG are determined through the SWAG procedure using upstream data, while y acts as a hyperparameter controlling the scale of prior variance. Following the official implementation code of Shwartz-Z2 -ii et al. (2022)10, we introduced an additional hyperparameter & 0.1 to avoid any numerical issues by defining the covariance as Σ 7 + EI. We summarize detailed hyperparameter configuration for each experiment:
In ResNet-20x4 experiments, we swept over the logarithmically spaced values of 1/3 [2.5 x 2* b and E (2* hkk
In ResNet-50 experiments, we instead tuned 1/(8|D)) to align with the established prac- tical weight decay regularization convention, where [D] denotes the number of training examples. We swept over the logarithmically spaced values of D1(3)D)) E (10-*}x } and 7 E {10*)r-
We fixed 1/(3|D)) 0.0001 ViT-B/16 experiments and 1/(8|DI) 0.01 in RoBERTa- Base experiments.
B.3 SUPPLEMENTARY RESULTS FOR CIFAR-10- 10-C BENCHMARK
Measuring calibration error: Table 8 further provides the evaluation results using ECE. Again, NPTL surpasses other baselines, indicating provides well-calibrated predictions.
Results in box plots. Figure visually presents supplementary statistics regarding the results, including the median and the first and third quartiles, depicted through a box plot. Overall, NPTL achieves lower NLL. and ECE.
21

---

Published as a conference paper at ICLR 2024
SGHMC + L2SP
Ensemble + L2SP
NPTL (ours)
SGHMC + PTYL
Ensemble + PTYL
4.00
0.50
ECE
2.00
0.25
0.00
0.00
3
4
5
2
Intensity level
Intensity level
Figure 1: Box plots for CIFAR-IO-C. It supplements Table 4 presented in the main text.
Table Additional BMA results for ResNet--20x4. Evaluation results of NPTL and baseline method for BMA on four image classification datasets, including C1c, C100, F101, and D120. Results with standard deviations represent the average of three runs. Here, IG implies the zero- mean isotropic Gaussian prior.
Datasets
Metric
Methods
C1O
C100
F101
D120
Avg.
SGHMC
0.872+0.000
0.625s0.001
0.550s0.01
0.235
0.571
ACC (t)
NPTL (ours)
0.96420.001
0.8 0.8.8000000
0.64420.002
0.634.0.001
0.765
SCHMC IG
1..80000000
1
1.73320.008
3.07 12.002
1.627
NLL (L)
NPTL (ours)
0. 102..0000
0.606s0.002
1.34720.001
1.297
0.838
B.4 SUPPLEMENTARY RESULTS FOR RESNET-20X4
If the chosen prior distribution is not appropriate, it can have substantial impact on the performance of BMA with posterior samples. To exemplify this, we present the BMA performance using the zero-mean isotropic Gaussian prior, denoted as WN0, I). The empirical results in Table clearly demonstrate how an ill-suited prior distribution can significantly deteriorate performance.
B.5 SUPPLEMENTARY RESULTS FOR RESNET-50
Ablation study on Dirichlet concentration. Here, conducted an ablation study on concentration parameter a. As we already mentioned in $ 3.1, a indicates the amount of our belief in prior knowledge. Figure shows that when distribution shifts occur between the upstream dataset and the downstream dataset (s.a. C101, D4 7), the gradual increment of alpha leads to a decline in the model's performance. However, the upstream dataset and the downstream dataset are similar (s.a. F102, P37), increasing value leads to enhanced model performance.
Ablation study on block Dirichlet. When the number of downstream training data n increases, sampling accurate values from Dirichlet distribution becomes challenging due to the numerical is- sues. To empirically validate this point, we conducted an experiment for the image classification task. Specifically, confirmed a significant decline in ACC and rise in NLL when replaced block Dirichlet (where the number of blocks is 10) with non non-block Dirichlet the ResNet-50 ex- periments conducted on C101. The test ACC decreased from 0.901 0.880=D.01, while the NLL increased from 2.317 to a.075 Hinging on this observation, we fixed the number of blocks to 10 throughout the paper.
http:///gthha.commoo///aaaaaaaan ccaming
22

---

Published as a conference paper at ICLR 2024
8200
C101
D47
F102
P37
0.82
0.90
0.70
0.96
0.97
Valid ACC
0.81
0.89
0.69
0.95
0.96
loggo
lo d
logu:
1og
logn a
Figure 2: Ablations prior belief in NPTL. It highlights that the optimal value, representing the prior belief towards the pre-trained model, varies across diverse downstream datasets.
Table 10: Mean IoU results for PASCAL VOC 2012. BMA Mean IoU performance of SGLD with L2SP, SGLD with supervised pre-trained PTYL, SGLD with self-supervised pre-trained PTYL, and NPTL with ResNet-50 backbone.
Dataset
Backbone
L2SP
SL PTYL
SSLPTYL
NPTL
PASCAL VOC 2012
ResNet-50
73.16
73.72
74.15
74.72
B.6 SUPPLEMENTARY RESULTS FOR SEMANTIC SEGMENTATION EXPERIMENTS
In this subsection, extended experiments semantic segmentation using the PASCAL VOC 2012 dataset, following the approach of Shwartz-Za Ziv et al. (2022). Utilizing the pre-trained ResNet- 50 our backbone model, we employed the DeepLabv3+ (Chen et al., 2017) model our complete model. Our experimental setup, except for our model, remained consistent with that of Shwartz-ZZ Ziv al. (2022). Table 10 presents conclusive evidence that NPTL outperforms other baseline models.
C DISCUSSION ON THE LIMITATION OF THE NPTL APPROACH
The potential drawback of the suggested NPTL approach lies in the additional training cost. Specif- ically, implementing the proposed NPTL algorithm incurs extra computational expenses for obtain- ing frodeed through linear probing on the given downstream data. However, it is important to note that the computational cost associated with linear probing is relatively minimal compared to the training of the entire network.
D DISCUSSION ON ROBUSTNESS OF THE NPL POSTERIOR TO MODEL MISSPECIFICATION
In this section, we explore the robustness of the NPL posterior compared to the regular Bayesian posterior on 0 in the face of model misspecification. Both the parametric and NPL posteriors target the same parameter, denoted as 0* which minimizes the KL divergence between the true distribution F' and the model distribution Fo (Walker, 2013).
However, it is crucial to note that NPL utilizes nonparametric prior, leading to posterior distribu- tions on 0 with superior asymptotic properties compared to the regular Bayesian posterior when the model is misspecified. The key distinction lies in the fact that parametric Bayesian inference assumes the existence 0* such that Fo- = F*. On the other hand, NPI. does not make this as- sumption, acknowledging model misspecification, and updates the posterior distribution म, (P)) in a nonparametric manner. This results in more robust posterior inferences on 0" and asymptotically superior predictions, as observed practically (Fong al., 2019).
To elaborate on the superior asymptotic properties, Holmes & Walker (2019) demonstrate that the Bayesian bootstrap posterior (having the same limit NPL) asymptotically yields the sandwich covariance matrix, known for its robustness. Conversely, the parametric posterior does not achieve this asymptotically.
23

---

Published as a conference paper at ICLR 2024
In terms of prediction, our focus is the posterior predictive density (v( f fo T. (0), where Ta represents either the Bayesian or NPL posterior. Theorem 1 of Lyddon et al. (2018) shows that asymptotically, the KL divergence between F and P. is smaller for the NPL posterior compared to the Bayesian posterior. This asymptotic improvement attributed to the robust sandwich covariance matrix (Müller, 2013).
24


Reviews:
[{"review_id": "RD2MMLB7ry", "reviewer": "Reviewer_vw7S", "summary": "This paper introduces nonparametric transfer learning (NPTL), which uses a Dirichlet process prior with centering measure $F_{\\pi}$ defined by downstream samples and their outputs from a linear-probed model. Then each posterior sample defines a weighted loss of the downstream samples and pseudo samples from the linear-probed model. The weighted loss is optimized for each posterior sample. Then Bayesian model averaging is performed over $M$ posterior samples. Extensive experiments show the superior performance of the proposed method over traditional Bayesian sampling methods.", "questions": "Please see the questions in the weakness above. Some minor questions\n\n1. In section 2.1 paragraph 1. the zero mean isotropic Gaussian is called a non-informative prior. If I remember correctly, the non-informative prior usually refers to something else.\n\n2. For the Bayesian inference, one criterion of setting prior can be ensuring posterior concentration, e.g. in the sense of theorem 2.1 in [1]. Can the authors comment a bit on how the proposed prior related to those concepts?\n\n### Reference\n[1] Ghosal, Subhashis, Jayanta K. Ghosh, and Aad W. Van Der Vaart. \"Convergence rates of posterior distributions.\" Annals of Statistics (2000): 500-531.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using downstream samples and a linear prob model to construct prior for transfer learning is an interesting idea, and with Dirichlet Processes, the posterior gives a simple form. The paper provides extensive numerical experiments to support the superior performance of the proposed method. The limitation of Bayesian model averaging: the computational cost is also properly discussed.", "weaknesses": "I found some details of the proposed algorithm a bit confusing, especially in the context of transfer learning. I want to clarify the following points\n\n1. For the linear prob model, my understanding is that $\\phi^*$ is from the pre-trained model, how do we get $W^*$, is it obtained by only fitting the last fully connected layer on the downstream task?\n\n2. In step 7 of algorithm 1, how is $\\theta$ initialized, randomly or by $(\\phi^*, W^*)$, i.e. is the pre-trained model only used to create pseudo samples for the prior, or is it also used to initialize parameter as well. Looking at the training setting, e.g. ResNet-50, with a cosine learning rate decay schedule, it looks like the parameters are trained from scratch, otherwise, I imagine the large initial learning rate would drive parameters away from the initialization. \n\n3. For the SGHMC baseline, how is the prior specified and how does the pre-trained model help the SGHMC method? And how does the pre-trained model help the ensemble baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3uyNRM4aFV", "reviewer": "Reviewer_bi4C", "summary": "The paper studies the applicability of nonparametric transfer learning (NPTL) in the context of Bayesian NNs and transfer learning. The general idea is that having pre-trained models based on NNs, once can obtain good performance on Bayesian model averaging (BMA) prediction on different tasks. For instance, pre-training on Imagenet and testing performance on CIFAR or similar vision datasets.", "questions": "see my prev. comments", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, I think it is a good paper with positive ideas and contributions. I particularly see the interest behind the application of NPTL in this context for transfer learning. To me, the paper is clear in the details concerning the sampling methodology, and perhaps not that much in the problems related to scalability or computational cost (see my comments below).", "weaknesses": "In my opinion, I think there are several points that are not clear enough while reading the manuscript and they could be also potential weaknesses of the method.\n\n[w1] --- Access to the posterior distribution given pre-trained models. In general, I see the idea, but it is not entirely clear to me how we can get posterior samples from any model that has been pre-trained without a particular prior before. Are we in the MAP solution? similarly to the Laplace approximation. Are there conditions or requirements for pre-training the models?\n\n[w2] --- I appreciate the details and the sincere comments on the heuristics used and so on. However, I do not get a good feeling on the scalability and the computational cost. Is really the methods playing a role given huge models with large number of parameters? or is it just bc the pre-trained models are doing still well on similar vision tasks. In that regard, I'm not entirely convinced by the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nyWr6KSeIO", "reviewer": "Reviewer_izux", "summary": "In these transfer learning scenarios, the prior distribution for downstream data becomes crucial in Bayesian model averaging (BMA). While previous works proposed the prior over the neural network parameters centered around the pre-trained solution, such strategies have limitations when dealing with distribution shifts between upstream and downstream data. This paper introduces nonparametric transfer learning (NPTL), a flexible posterior sampling method to address the distribution shift issue within the context of nonparametric learning. Through extensive empirical validations, the authors demonstrate that the approach surpasses other baselines in BMA performance.", "questions": "(1) Following the experiments of [1], could the proposed method also uses self-supervised pretrained models and include experiments about segmentation?\n\nreferences:\n\n[1] Shwartz-Ziv, Ravid, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G. Wilson. \"Pre-train your loss: Easy bayesian transfer learning with informative priors.\" Advances in Neural Information Processing Systems 35 (2022): 27706-27715.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The authors maintain high quality of presentation. The motivation and algorithms are clearly explained. \n\n(2) Extensive experiments and ablation studies are conducted in this paper. The authors considered different model architectures, datasets, evaluation metrics, tasks.", "weaknesses": "(1) A detailed discussion of limitations is lacking. A possible aspect could be the computation cost. A careful discussion of pros and cons could be very helpful to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xBsovmBrPN", "reviewer": "Reviewer_ZDrq", "summary": "In the context of transfer learning, the choice of the prior distribution for downstream data is crucial when employing Bayesian model averaging (BMA). Prior strategies have limitations in handling distribution shifts between upstream and downstream data. This paper introduces nonparametric transfer learning (NPTL) that addresses distribution shift issues within the framework of nonparametric learning. The nonparametric learning (NPL) method, which uses a nonparametric prior for posterior sampling, efficiently deals with model misspecification scenarios.", "questions": "I feel confused about the authors' claim that \"the NPL posterior is robust to model misspecification through the adoption of a nonparametric model and gives asymptotically superior predictions to the regular Bayesian posterior on $\\theta$\".\n\nIn my understanding, NPL is just another way to compute the posterior for $\\theta$. In parametric Bayesian methods, a prior $p(\\theta)$ is placed on $\\theta$, and various methods are employed to compute the corresponding posterior $p(\\theta|\\mathcal{D})$. NPL, on the other hand, establishes a deterministic mapping between the data distribution $F$ and $\\theta$ using MLE (as shown in Equation 2). Then, a prior $p(F|\\alpha, F_\\pi)$ is placed on the data distribution $F$ to analytically derive the posterior $p(F|\\mathcal{D})$, which subsequently leads to the posterior $p(\\theta|\\mathcal{D}$).\n\nBoth of the above methods assume a parameterized model with parameters $\\theta$. In other words, the assumed parameterized model is still susceptible to potential discrepancies with the true data distribution, which could result in model misspecification. Therefore, if my understanding is correct, NPL may not inherently address the issue of model misspecification.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes the use of nonparametric learning techniques to address the issue of distribution shift in transfer learning. The writing in the article is of high quality and is easy to understand. I partially reviewed the mathematical derivations in the paper, and the technical aspects appear to be sound.", "weaknesses": "I did not find any significant drawbacks in the paper, except for some confusion in the nonparametric learning section. Please see below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)