Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported
Authors: Ethan Harvey
Mikhail Petrov
Michael C Hughes
Venue: Accepted by TMLR ()

Abstract: We pursue transfer learning to improve classifier accuracy on a target task with few labeled examples available for training. Recent work suggests that using a source task to learn a prior distribution over neural net weights, not just an initialization, can boost target task performance. In this study, we carefully compare transfer learning with and without source task informed priors across 5 datasets. We find that standard transfer learning informed by an initialization only performs far better than reported in previous comparisons. The relative gains of methods using informative priors over standard transfer learning vary in magnitude across datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on 2 datasets, modest gains (between 1.5-3 points of accuracy) on 2 other datasets, and substantial gains (>8 points) on one dataset. Among methods using informative priors, we find that an isotropic covariance appears competitive with learned low-rank covariance matrix while being substantially simpler to understand and tune. Further analysis suggests that the mechanistic justification for informed priors -- hypothesized improved alignment between train and test loss landscapes -- is not consistently supported due to high variability in empirical landscapes. We release code to allow independent reproduction of all experiments.

Published in Transactions on Machine Learning Research (05/2024)
Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported
Ethan Harvey
Sthan.horuee Stufts. huf edu mikhail, mikhail.petroofee petronntufts edu michael hughes Stufts. edu
Mikhail Petrov2*
Michael C. Hughes
DDpprtment of Computer Science, Tufts University
Deparbment of Mechanical Engineering, Tufts University
Reviewed on OpenReview: https: // openreview. net/ pormn/ id- BbuSU02jIL
Abstract
We pursuc transfer lcarning to improve classifier accuracy on target task with few labeled examples available for training. Recent work suggests that using source lask to learn prior distribution over neural net weights, not just initialization, performance. In this study, carefully compare transfer learning with and without source task informed priors across 5 datasets. We find that standard transfer learning informed by an initialization only performs far better than reported in previous comparisons. The relative gains of methods using informative priors over standard transfer learning vary in magnitude across datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on datasets, modest gains (between 1.5-3 points of accuracy) on other datasets, and substantial gains (>8 points) on one datasct. Among methods using informative priors, find that an isotropic covariance appears competitive with learned low-rank covariance matrix while being substantially simpler to understand and tune. Further analysis suggests that the mechanistic justification for informed priors hypothesized improved alignment between train and test loss landscapes - is not consistently supported due to high variability in empirical landscapes. We release code to allow independent reproduction all experiments.
boost target task
1 Introduction
Dataset size is often the limiting factor defining the ceiling of performance in deep learning application. Without large dataset of labeled examples available for training, deep learning models face challenges like overfitting and poor generalization (Brigato & Tocchi, 2021). One motivating application with such challenges is medical image classification, especially when labels correspond to diagnoses or annotations obtained by manual review of expert clinicians. In such scenarios, obtaining labels can be prohibitively expensive, and thus often only small labeled dataset available (say a few dozen or few hundred examples). Finding other potential sources of information to guide learning can be critical to boost performance.
The field of transfer leornáng offers a promising remedy (Zhuang et al., 2020). The idea is that in addition to small labeled set for target task of interest, the user has access "source" task, such as the well-known ImageNet dataset (Deng et al., 2009). Alternatively, users may have access to a pre-trained model snapshot or distribution derived from such a source. In modern deep learning, the typical off-the-shelf way to benefit from the source task unfolds in two steps (Murphy, 2022, Sec. 19.2). First, obtain a point estimate of neural network weights via "pre-training" on source data. Next, use that point estimate as an initialization for supervised training of the same architecture on the target task, known "fine-tuning". Throughout, refer to this two-step process standard transfer learning.
much larger labeled dataset from a
Methods that can deliver further accuracy gains beyond standard transfer learning are naturally of interest to practitioners. Recent results from a new Bayesian transfer learning approach by Shwartz- Ziv et al. (2022) are
'Equal contribution
Codee: httpe://github. htttshhuub.coeeeeeee con/tufts-nl /bdl- maall-trasfe----- ferarninn-

---

Published in Transactions on Machine Learning Research (05/2024)
particularly exciting. Their approach claims to improve upon standard transfer learning by using the source task to learn a distribution over network weights, not just a point estimate for initialization. This distribution then serves as an informative prior for the target task. For the target task of classification on CIFAR-10 10 data given a training set of just 1000 samples, Shwartz-Ziv ct al. (2022) report that their maximum a-posteriori (MAP) objective yields an error rale of 27% compared to the 36% possible with standard transfer learning, a gain in accuracy of percentage points. Similar large gains from MAP with informative priors are reported on other datasets when the target training set size remains rather small (n S 1000). Beyond MAP point estimation, they report even further gains from Bayesian estimation of the posterior distribution over weights on the target task via stochastic gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al., 2014).
Excited by the reported gains in the simpler MAP setting, our team undertook an extensive investigation into MAP transfer learning with informative priors. We originally intended to quantify how well the gains that Shwartz-Ziv et al. report on natural image datasets transferred to medical imaging scenarios. However, to our surprise our experiments suggest that standard transfer learning performs better than previously reported by Shwartz-Ziv et al. (2022). In systematic experiments that vary the size of the larget dalaset across four natural image datasets and one medical image dataset from dermatology, we find the relative gains of MAP with informative priors over standard transfer learning vary in magnitude across datasets.
Overall, this paper contributes three findings to the study of transfer learning for point estimated weights:
First, standard transfer learning without any informative prior performs better than reported in Shwartz- Ziv et al. (2022). In the specific setting of CIFAR-10 with 1000 labeled examples for training and ResNet-50 architecture (He et al., 2016), we find this method hits test set accuracy between 78-82% across multiple replicate samples of the desired training set size, far better than the 64% accuracy (36% error rate) previously reported. This finding is further supported by third-party experiments in Kaplun et al. (2023), who like us reach accuracy near 80% with a similar architecture in the same setting. Large gains over previous reports are also seen at the n = 100 setting (see Fig. 1).
Second, we find that the relative gains of methods using informative priors over standard transfer learning vary in magnitude across datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on 2 datasets, modest gains (between 1.5-3 points of accuracy) on 2 other datasets, and substantial gains (>8 points) on one dataset. Among methods using informative priors, we find that Chelba & Acero (2006)'s 's isotropic covariance appears competitive with Shwartz-Ziv et al. (2022)'s 's low-rank covariance matrix while being substantially simpler to understand and tune.
Finally, we empirically assess a specific mechanistic hypothesis from Shwartz-Ziv et al. (2022), who suggest informative priors allow improved "alignment." " between the train and test loss landscape of the target fine-tuning task. For an idealized illustration, see their Figure 1. We find that on real datasots, informative priors do not often align train and test losses well; instead our Fig. 2 shows there is considerable variability in the alignment especially across samples of a small labeled set from the target task.
To take steps to make our present analysis reproducible (including all dataset splitting and hyperparameter tuning procedures), we release a codebase' that can re-run all experiments and reproduce all figures here. In the spirit of open science, we welcome any feedback or questions from users.
Our inability to reproduce previously reported results is part of larger trend across machine learning research (Pineau el al., 2021). We hope our work here helps the community move loward improved best practices. The first best practice we advocate is sharing scripts that enable all experiments, especially those related to hyperparameter tuning. The second best practice is allowing all basclines at minimum the same hyperparameter tuning as a presented method in every experimental setting.
2 Background
Bayesian transfer learning. Recent work by Shwartz- Ziv et al. (2022) proposed Bayesian transfer learning, where re-scaled posterior from the source task is used as the prior for the target task. This work is motivated
2

---

Published in Transactions on Machine Learning Research (05/2024)
by sequential Bayesian updating, where some source data Ds is acquired, a posterior p(w(Ds) over weights w is formed, and this posterior is used as an informed prior for target dataset DDr (Murphy, 2012)
p(w)Ds) x p(Ds w))(w)
(1)
p( Ds) x p(D (pppppppp((ss)
In sequential Bayesian updating, analysts often assume all data independent and identically distributed, at least. when conditioned on parameters w. However, in transfer learning the target data comes from a somewhat different distribution than the source data, beliefs about which weights are a-priori plausible may need adjustment. To rectify this problem, Shwartz-Ziv et al. (2022) introduce a scaling factor À 2 controlling influence of the source-informed prior over weights we Concretely, À rescales the covariance matrix of the posterior p[w[Ds) (which they assume is well-approximated by a Gaussian) to form the source-informed prior. When = 1, the model directly uses the source posterior as the source-informed prior. As À grows larger, the source-informed prior has inflated covariance. In the limit as X oo the influence of the source data disappears (the prior approaches uniform distribution) and the only influence of the source task on the target task would come from initialization, thus recovering standard transfer learning.
2.1 Common framework for MAP transfer learning.
Probabilistic model for target task. Following Shwartz-Ziv et al. (2022), we form a model that performs a desired target task of image classification with C' possible classes. The model has two parts: a backbone and classification head. Let f denote the backbone neural network with weights w Rd. For modern computer vision, d might be in the millions or even billions. Given an image z; of prespecified size, ʃ produces a hidden representation vector (z;) E R² We assume the first entry of this vector is always 1 to allow learning intercepts. We define the classifier head as parameterized by weights V E RC*H that can map the representation vector to logits over C classes. Then, given a target training set of size n, containing pairs {2x, ye of images 2; and categorical labels ys E (1,2..... ..,,, we have the following template for a probabilistie model:
p(w)= N(w | m, XS)
source-informed prior on backbone
(2)
P(V) = (Vvcc(VV 10,
prior on clf. head
(3)
p(yim Wiim w. V)- = II Cat Cat(y/lsoitmaa(VVSuu F=1
likelihood
(4)
Here, 0 and >>0 0 are key hyperparameters that require tuning, while the choice of m, is method specific; these values may be simple (e.g. an all-zero mean) informed by a source-task.
Target task MAP estimation. We can fit the above model to the target datasct via MAP point estimation strategy, finding values of weights w, V that minimize the objective
L(w, V) = - Σ logr(uslai, w. V) + log + logr(V))
(5)
Given a concrete target dataset, we pursue stochastic first-order gradient descent to obtain estimates of w, V. In transfer learning, it is common initialize backbone weights w at a known good value from the source task, denoted μ. Classifier head weights V can be initialized randomly.
As surveyed below, several approaches to transfer learning in the literature share both the common model template defined above and the pursuit of MAP estimation. They vary only in the setting of the backbone prior's ' mean vector m and covariance matrix S. Tab. 1 summarizes the specific possible methods.
2.2 Concrete methods for MAP transfer learning.
Below cover spectrum of possible choices for the prior over backbone weights, all of which can be integrated into the common MAP transfer learning framework above. They differ in whether and how the prior is informed by a source task, especially the source-informed point estimate μ of the backbone weights.
Standard non-informed isotropic prior. The standard transfer learning approach most common in the literature to set m the all zero vector and set S to the identity matrix. We refer to this as StdPrior throughout (read as "standard prior"). Here, the backbone prior not informed by the source task; the only source-informed knowledge comes from initializing MAP estimation at source-informed weights p..
3

---

Published in Transactions on Machine Learning Research (05/2024)
Table Possible methods for point estimation of neural network weights for target task. All can be seen as MAP estimation for model defined by common likelihood and method-specific prior. The MAP objective is then optimized by first-order stochastic gradient descent. Our work focuses specifically on methods informed by a source task (ImageNet classification), using values of weight vector j and low-rank (LR) covariance matrix E learned from this pre-training phase. We include columns for related works and mark the methods they investigate.
Method
also known
Prior
Init.
Shwartz- Ziv Spendl & et al. (2022) Pirc (2023)
ours
StdPrior fromScratch
SGD Non-Learned Prior In Shwartz-Ztv et sl.
O0 XT)
random
StdPrior fromImgNet.
SGD Transfer Tnit.
(O,
P
In Shwartz-Zlv et al.
LearnedPriorlso fromImgNet MAP adaptation
(1,,
P
X
X
In Chalba & Acera
LearnePrrirrLR fromImgNet SGD Learned Prior
((,,
ia Sloartle-Zi et al.
Learned prior with isotropic covariance. A natural way to let the source task inform the backbone prior is to set the mean such that m = μr, The prior covariance S is left as the identity matrix. This corresponds to an older method from Chelba & Accro (2006), known as MAP adaptation, which was recommended as the "standard baseline for solving transfer learning tasks and benchmarking new algorithms" in Xuhong et al. (2018). We refer to this LearnedPriorIso throughout.
Learned prior with low-rank (LR) covariance. Shwartz-Ziv et al. (2022) propose informing both the prior's mean and covariance matrix from the source task. They use Stochastic Weight Averaging-Gausin (SWAG) (Maddox et al., 2019) to approximate the posterior distribution psw)Ds) of the source data Ds with a Gaussian distribution as (u, M) where μ is the learned mean and = 1 (Eding + LLR) a representation of covariance matrix with both diagonal and low-rank components. For a d-parameter neural net with typical d in millions or billions, standard d x d covariance matrix is unaffordable to store. Thus, we select concrete desired rank k (where k < d), and form the LR covariance ER = --QQ" where Q e dg is learned parameter from the source task (the diagonal component Ediag is also learned). We refer to this prior as LearnedPriorLR throughout, and use k = 5 throughout as recommended by Shwartz- Ziv et al..
2.3 Previous replication efforts with informative priors.
Work on informative priors by Shwartz-Ziv et al. (2022) has generated considerable follow-up attention, including a previous replication study with different goals than ours. Spendl & Pire (2023) sought to compare "from scratch" training on the target task alone (without even a source-informed initialization) to the proposed informed prior MAP procedure. Spendl & Pire (2023) used Shwartz-Ziv et al. (2022)'s 's released code with fixed initial learning rate and weight decay on just one dataset (Oxford Flowers-102 dataset by Nilsback & Zisserman (2008)). They report that transfer learning with informed priors did outperform a simple from-scratch baseline that did not do any transfer from source to target (see Tab. 1 for a comparison of which methods all papers investigate). In contrast, we run experiments designed to compare standard initialization-onlyy transfer learning to informed prior transfer learning. Our experiments also perform soparate hyperparameter tuning for all methods at all dataset sizes.
3 Experimental Procedures
We seek to fairly compare three distinct methods for point estimating optimal neural network weights w, V for a target classification task. All three make use of pro-training on a source task, some via initialization only and some via an additional learned prior, as described in Sec. 2.2 and sunmarized in Table 1. We refer to these methods in all result tables and figures StdPrior for standard transfer learning with an uninformative standard Gaussian prior with zero mean and isotropic covariance S=XI, LearnedPriorIso for a source-informed prior with learned mean μ but isotropic covariance S=AI, and LearnedPriorLA. for the source-informed prior with mean j and learned low-rank covariance multiplied by scalar X.
Below we provide essential details of our experimental procedure. We intend that all experiments are fully reprodueible by others. See Tab. B.1 in App. B for a detailed comparison of Shwartz-Ziv Ziv et al. (2022)'s and our experimental procedures. For further details, see App. A as well our released codebase.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Datasets and classification tasks. We study targeted classification tasks from five distinct open-access datasets. For all tasks, we study how well variants of MAP estimation can transfer knowledge from source task on ImageNet.
The first three datasets were selected as common benchmarks that enable direct comparisons to previous experiments by Shwartz-7iv et al. (2022): the 10-way classification task of CTFAR-10 (Krizhevsky al., 2010), the 102-way classification task of Oxford Flowers (Nilsback & Zisserman, 2008), and the 37-way classification task of Oxford-IIIT Pets (Parkhi et al., 2012). Each of these contains natural images (RGB, resized to 224x224 pixels) that typically depict one object of the labeled class.
For the last two datasets, we select the 100-way classification task of -inenGrained Visual Classification of Aircraft (FGVC-Aircratt) (Maji et al., 2013) and HAM10000 (Tschandl et al., 2018; Codella et al., 2019), an open-source deidentified dataset, informally known as "Skin Cancer MNIST". HAM10000 was designed to support asscssment of automatic diagnosis of several possible skin Icsion types from RGB images resizod to 224x224 pixels of skin surface microscopy (dernatoscopy)) To avoid extreme imbalance, we form a 4-class classification task between 3 original classes with sufficient data (at least 10% prevalence) and an additional "other" class that combines all other original classes.
Common architecture and optimization. Across all experiments, we always set the backbone architecture of network f to a ResNet-50 (He et al., 2016). This directly matches the experiments in Shwartz-Ziv et al. codebase uses a common optimization procedure for training: SGD with batch
(2022). Our PyTorch-basee size of 128 and Nesterov momentum of 0.9 with oesine-annealing schedule for learning rate (Loshchilov & Hutter, 2016). Initial learning rate is tuned hyperparameter (see below).
Common source-task knowledge. For all methods, we take the learned weights μ that form the initialization (and if needed, the prior mean), from Shwartz-Ziv et al. (2022)'s 's released snapshots². For Le,reePrrirLLR, we also obtain the diagonal and low-rank ingredients for the covariance matrix directly from Shwartz-Ziv et al. (2022)'s released snapshots. We specifically use Shwartz-Ziv et al.'s ' snapshots pro-trained on ImageNet with SimCLR (Chen el al., 2020), following their recommendation that pre-training with self-supervised learning (as in SimCLR) leads to improved transfer learning compared with fully-supervised learning (e.g. using cross-entropy loss with ImageNet class labels).
Fixes to Shwartz-Ziv et al. (2022)'s code. Shwartz-Ziv el al. (2022)'s 's released code performs inconsistent scaling by À of the low-rank and diagonal components of the covariance matrix of their informed prior. Looking at their code (see line 40 of utils.py), py), they obtain ΣLR by rescaling the low-rank matrix Q by A before computing QQ", which yields overall covariance matrix AEdlag + A ELR. To remedy this, we rescale the low-rank matrix Q by vÀ instead of X, obtaining the intended 3/2 LEding + 2 ELR. We also remove the redundant "double prior" due to their implementation's use of weight decay on weights w in addition to the informative prior. Details for both of these changes can be found in Tab. B.1 in App. B.
Varying train set size n. For each dataset, we setup range of transfer learning tasks by artificially varying n, the number of available samples for training. In all cases, we ensure balanced class frequencies, ensuring the same number of images per class. For CIFAR-10, by selecting n E {10, 100, 1000, 10000, 50000), we cover exactly 1, 10, 100, 1000, and all available examples per class. For Oxford Flowers, we select n E {102, 510, 1020] to cover 1, 5, and 10 examples per class. For Oxford-IIIT Pets, we select n E (37,370, 3441) to cover 1, 10, and 93 examples per class (93 the maximum possible value that keeps exact balance). For FGCV-Aireraft, we select n E {100, 500, 1000,5000) to cover 1, 5, 10, and 50 examples per class. For HAM10000, we select. E (100, 10003. We emphasize that the 10-1000 examples per class regime is likely of most practical interest for many applications.
Repeated trials. At each chosen value of n, we build 3 separate "replicate" training sets by randomly sampling n image-label pairs (without replacement) from the predefined complete training set. We then separately perform all training (including hyperparameter tuning) on each replicate. We emphasize we are careful to ensure similar class-composttion for all replicates.
2Shwartz Ziv et. al. (2022)'s SimCLR snapshots can found at http:///ggtta..cccco///////ttatt
5

---

Published in Transactions on Machine Learning Research (05/2024)
Hyperparameter tuning. All methods need to tune three key hyperparameters: the covariance scaling factor X, the classification head weight decay т, and the initial learning rate. To give informative prior approaches the best possible settings, for LearnedPriorL we tune all three together. For simplicity, for both StdPrior and LearnedPriorIso we fix À A= T, tuning only 7 and learning rate. In some ways, this setup puts standard Transfer learning at a disadvantage, as LearnedPriorI.R is allowed to search larger hyperparameterr space.
Model fitting (including tuning) has two stages. In the first stage, we first randomly divide available data of size into train set and validation set with 4:1 ratio. We then tune all hyperparameters via grid search (ranges in App. A.2), selecting the best configuration based on the log likelihood of the labels of the validation set. After selecting an optimal hyperparameter configuration in stage one, we refit to the entire dataset of n examples in stage two. In all stages, the optimizer runs until reaching a specified maximum number of iterations.
Performance metrics test set. All datasets provide predefined train/lest split which our experiments respect. We report heldout performance metrics the complete available test set for each task. For CIFAR-10, Oxford Flowers, Oxford-IlIT Pets, and FGVC-Aireraft, we report classifier accuracy (higher is better) following Shwartz-Ziv et al. (2022). For HAM10000, we report average AUROC (higher is better), macro-averaged over all classes. Uliimatey, we report the average performance metric across all 3 replicate trials, as well as the min-max range. For all tasks, we also assess the negative log likelihood (NLL) of the test set.
4 Results
Examining the results of all experiments, we come three key findings. These are detailed in the subsections below.
SidPrir fromingMe Super (Kaplan et al., 2023)
SidPrior Somm (ours)
SidPrior faminggeeSSmmUA (Shwarte-Ziv al. LearnedRrooLR fromingNeSSinCL. (Shwariz-Ziv al., 2022)
LeamedFriorLR framingNesSSiCLRRRRR. (ours)
4.8
4.8
I
I
65
3.8
3.8
Test error
40
2.8
I
2.8
5 10
5 10
10
102
10a
10a
105
10
102
103
10
105
Number of training samples (log-scale)
Number training samples (log-scale)
Figure Error rate (lower better) vs. target train on CIFAR-10, for various MAP estimation methods for transfer learning from ImageNet. Left: Our results. Right: Results copied from Shwartz-Ziv et al. (2022) (their Tab. 10). Takeaway: our experiments, standard transfer learning (StdPrior) does better than previously reported. Setting details: The blue and purple lines across both pancls come from comparable settings: common ResNet-50 architecture and common learned values for mean and low-rank (LR)) covariance taken directly from the SimCLR pre-trained snapshots in Zivartz-Ziv et al. (2022)'s repository. Green line: The left panel's green line is third-party experiment copied from Kaplun et al. (2023), suggesting others can achieve similar performance as we do for standard transfer learning with ResNet-50. They fully-supervised pre- te-trriinngg not sule-supe SimCLR. Plotted mean and standard deviations confirmed via direct correspondence with Kaplun et al..

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: CIFAR-10 heldout accuracy (higher is better) as target train set size n increases. For each method and train set size ran our two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (independent samples of n examples from the provided full training set). We report mean accuracy on test set across these 3 trials, along with the min-max range. 10 classes, balanced. See App. A for details.
Method
n= 10 (1/cl.)
100 (10/cl.)
1000 (100/cl.)
10000 (1k/cl.)
50000 (5k/cl.)
StdPrior fromImgNet
22.0 (13.2-26.7)
56.2 (53.0-59.9)
80.9 (78.2-82.8)
93.4 (93.3-93.7)
96.8 (96.7-96.9)
LearnedPriorIso Neg
20.7 (17.7-23.4)
56.7 (54.8-59.8)
81.9 (81.6-82.1)
94.3 (94.1-94.5)
97.3 (97.2-97.4)
LearnedPriorT.R fromIngNet.
15.7 (11.3-18.8)
58.7 (56.4-60.6)
83.5 (83.3-83.9)
93.8 (93.4-94.1)
96.9 (96.7-97.0)
Table 3: Oxford Flowers heldout accuracy (higher is better) as target train set size n increases. For each method and train set size n, ran our two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (independent samples of n examples from the provided full training set). We report mean accuracy on lest set across these 3 trials, along with the min-max range. 102 classes, balanced. See App. A for details.
Method
102 (1/cl.
510 (5/cl.)
1020 (10/cl.)
StdPrior fromimgNet
31.1 (12.6-40.6)
78.9 (78.4-79.3)
87.9 (87.4-88.1)
LearnedPriorIso Nmgt
19.2 9.0-34.8)
88.6 (88.2-89.1) 77.9 (74.8-79.4)
79.1 (78.5-79.4
LearnedPriorI.R fromImgNet.
28.8 (10.6-39.2)
88.4 (88.2-88.7)
Table 4: Oxford-IIIT Pets heldout accuracy (higher is better) as target train set size n increases, For each method and train set size n, we ran our two-phase training pipeline (including hyperparameter tuning) on distinct training sets (independent samples of examples from the provided full training set). We report mean accuracy on test set. across these 3 trials, along with the min-max range. 37 classes, balanced. See App. A for details.
Method
n= 37 (1/cL)
370 (10/cL.)
3441(93/cl.)
StdPrior fromimgNet
17.3 (14.5-20.0
54.8
(53.2-57.5)
86.4 (86.0-86.6)
LearnedPriorfso fromImgNet
7.0 ( 5.4- 8.5)
55.5 (53.3-57.7)
86.4 (85.4-87.0)
LearnedPrion P. fromImgNNet
6.6 ( 6.2 7.3)
57.4 (56.2-58.2)
86.7 (85.0-87.8)
4.1 Finding 1: Standard transfer learning better than reported in Shwartz- Ziv et al. (2022).
Our Fig. provides a side-by-side comparison of results on CTFAR-10 for our experiments (left) compared those originally reported in Shwartz-Ziv et al. (2022) (right). Here, we report error rate (lower is better) and use color scheme that matches original plots from Shwartz-Zi Ziy et al.. We see that StdPrior (purple line) is consistently 10-15 percentage points better at dataset sizes E (100, 1000), essentially erasing the gap in performance previously attributed to informative priors.
Our results are further supported by other third-party experiments by Kaplun et al. (2023) that use the same architecture (ResNet-50)) and same source task (ImageNet) to classify CIFAR-10. Our results at n = 1000 show test error for StdPrior in the 17-22% range, matching results from Kaplun et al. (green line in our figure) well. We suggest this lends credibility to our claim that it is possible for others to do far better with standard transfer learning than has been previously reported. We note that Kaplun et al. do pre-training with supervised methods, not the self-supervised SimCLR; see App. D for a comparison with Shwartz-Ziv et al. (2022)'s 's fully-supervised pre-training results.
4.2 Finding 2: Relative gains of informed priors over standard transfer learning vary across datasets.
Ultimate test set performance for all methods across train set sizes n is reported for all datasets in Tables 2, 4, 3. 5, and respectively. Each table entry reports that method's mean test accuracy across the 3 replicate samples of training points, as well as the min-max range.
We find that the relative gains of methods using informative priors over standard transfer learning vary in magnitude across datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on Oxford Flowers and HAM10000; modest gains (between 1.5-3 points of accuracy) on CIFAR-10 and Oxford-IIIT Pets; and substantial gains (>8 points) on FGVC-Aireraft.
Among methods using informative priors, we find that Chelba & Accro (2006)'s isotropic covariance appears competitive with Shwartz-Zip et al. (2022)'s low-rank covariance matrix while being substantially simpler to understand and tune. There are few cases where Shwartz-Ziv et al. (2022)'s low-rank covariance
7

---

Published in Transactions on Machine Learning Research (05/2024)
Table 5: FGVC-Aircraft heldout accuracy (higher is better) as target train set size n increases. For each method and train set size n, ran our two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (independent samples of n examples from the provided full training set). We report mean accuracy on test set across these 3 trials, along with the min-max range. 100 classes, balanced. See App. A for details.
Method
= 100 (1/cl.)
500 (5/cl.)
1000 (10/cl.)
5000 (50/cl.)
StdPrior fromImgNet
2.6 (1.3-4.6
22.5 4-2.44)
40.6 (39.8-41.8) 8)
85.7 (85.4-86.0
LearnedPriorIsi Neg
3.5 (2.8-4.5)
25.9 (24.0-27.3)
51.3 (50.0-52.6)
85.9 (85.8-86.0)
LearnedPriorT.R fromImgNet.
3.8 (3.5-4.2)
24.5 (23.7-25.3)
50.9 (50.4-51.9)
84.2 (83.5-85.3)
Table 6: HAM10000 AUROC on heldout data (higher better, macro-averaged across classes) target train set size n increases. For each method and size n, we ran our two-phase training (including hyperparameter tuning) on 3 distinct training sets (indepeudent samples of size n from the provided full training set). We report mean AUROC on test set across 3 trials (min-max range). 4 classes, ranging from 11%-67% prevalance. See App. A for details.
Method
= 100
1000
StdPrior fromlngNet
78.1 (75.0-82.8)
85.6 (85.0-86.3)
LearnedPriorIso froNIt
78.0 (75.0-82.8)
86.5 (85.5-87.4)
LearnedPriorL.R fromImgNet
78.7 (74.9-8..7)
86.6 (85.0-87.5)
is clearly better than Chelba & Acero (2006)'s isotropic covariance: CIFAR-10 at n=100 has 58.7% for LearnedPriorLR vs. 56.7% for LearnedPriorlso; CIFAR-10 at n=1000 has 83.5% for LearnedPriorLR vs. 81.9% for LearnedPriorTso: and Oxford-TITT Pets at ==370 has 57.4% accuracy for LearnedPriorl.Rt vs. 55.5% accuracy for LearnedPriorlsse However, these wins are not consistent across datasets, and when there are substaintal gains from informed priors, Chelba & Acero (2006)'s isotropic covariance is competitive with Shwartz- Ziv et al. (2022)'s 's low-rank covariance (see FGVC-Aireraft at dataset sizes n E {5000, 1000)).
Further results in App. C compare methods using NLL as a performance metric. Again, the conclusion is that the relative gains of methods using informative priors over slandard transfer learning vary in magnilude across datasets.
4.3 Finding 3: Large variability in quality of alignment between training and test loss landscapes.
Shwartz-Ziv et al. (2022) suggest an underlying mechanism for why informed priors improve transfer learning: better alignment between the loss landscapes of the target classification task across the small available training set and the large test set. (Remember, test set performance is assumed reliable "gold-standard" here due to its size, but such data is not available for training in the proposed transfer learning setting). See an idealized illustration in their Figure 1, where the overall function shape and especially the locations of minima are far more similar across train and test with source-informed priors than standard methods.
To empirically study this proposed phenomenon, we examine how informed priors reshape the loss surface on the CIFAR-10 task when n = 1000. It is difficult to visualize the high-dimensional input space (the size of all weights w is over 23 million), so we look a ID slice that linearly interpolates between two values: the optima WSya found by StdPrior MAP estimation the optima win found by LearnedPrirL.R and the optima w found at the largest dataset size. (We interpolate between classifier heads V as well, but use just w symbols for simplicity). Each panel of Fig. 2 visualizes over this 1D interpolation how two functions behave: the train set loss of each MAP objective (red line, vary by columns as StdPrior or LearnedPriorL and the test set NLL (blue, vary by columns as different 1D slices). Each row comes from a different replicate sample of n = 1000 image-label pairs for training.
If informed priors lead to better alignment, we contend that this should be visible in these plots: the location of the minima for the blue test curve should be closer to the muure--nfforeet optima wiR than the standard method optima wrea- We label the Euclidean distance in each plot to help readers focus on which method reduces the gap between trained and idcal minimum. However, across 3 replicate train sets (rows) we find there is high variability in how methods rank in alignment. Sometimes, wi,n does have a smaller gap, indicating better alignment (bottom row). However, we also find the gap can be similar (middle row) or even prefer the standard method (top row). We thus suggest that there is considerable variability alignment quality, making the empirical evidence for the loss-alignment hypothesis of Shwartz-Zit et al. (2022) less decisive than their idealized illustration would suggest.
8

---

Published in Transactions on Machine Learning Research (05/2024)
random random_state= state=20011
Loss
Loss
Wtd
1111.1372
WIR
random stats= 1001
Loss
Loss
Wsid
1111.1556
w'
WER
1112.552
W
random random_state=
Test loss
Train loss
Loss
Loss
WSdd
1273.3645
WLR
1173.9144
W
(a) Standard transfer learning (StdPrior)
(b) LearnedPriorL.
Figure 2: Empiriaal aligument of loss landscapes for target task across train and test for CIFAR-10 = 1000. Compare to Shwartz-Z4 Ziv et 's Figure which is idealized illustration not au empirical result. Each panel: We assess ID slice of the high-dimensional landscape by linearly interpolating parameters between the optima waya found via minimizing standard MAP objective (left) or optima wiR found via minimizing objective (right) and the optima w" found at the largest dataset size. Red curve shows the indicated training loss (varies by column) on CIFAR-10 with n = 1000 samples; blue curve (varies by column) shows CIFAR-10 test set NLL. Each row shows results from different train set sample of size 1000. The gap between the optima found via training and the Lest set's ideal minimum shown as a double-sided gold arrow. Takeaway: Shwartz-Ziv et al. (2022)'s learned prior approach does not always reduce the gap between trained and ideal minimum.
LearnedPriorL.R MAP

---

Published in Transactions on Machine Learning Research (05/2024)
5 Discussion and Conclusion
Identifying the reason our current study leads different conclusions than prior work is complex. The code released by the original study authors was incredibly useful to us. We are very thankful to the authors for making it available, it was more usable than other code repositories we have seen. However, we found their released code does not allow exact replication of key experimental steps from Shwartz-Ziv et al. (2022), such as replicable sampling of the same n image-label pairs they used for training in their experiments, or how exactly to perform hyperparameter tuning (how to carve out a validation set from available data, etc.). It is thus difficult to isolate what setting in our present study lead to the different results seen here, though we attempt careful side-by-side comparison in App. B. Our current best guess is that our substantial effort. into fair hyperparameter tuning of all methods led to improved numbers for standard transfer learning.
Is standard transfer learning better than full Bayesian inference? We stress that all findings here are confined to the simpler MAP point estimation setting; we do not study the full Bayesian posterior estimation methods of Shwartz-Ziv et al. (2022), which require notable additional computational complexity across finettunig and testing phases of the model development life cycle. For example, the-tuning is usually at least 5-10x as time-consuming. Full Bayesian approaches also require substantive expertise from downstream practitioners when debugging needed.
That said, our standard transfer learning results on CIFAR-10 are good as or better than the full Bayesian inference results reported in Shwartz-Ziv et al. (2022) (for direct comparison, see Fig. F.1 in App. F). We expect deep ensembles to further improve on our results (Lakshminarayanan et al., 2017). We leave it to future work to investigate the relative performance gains from full Bayesian inference using source-informed priors compared to properly tuned deep ensemble baselines.
What transfer learning method is recommended? Based off our experiments, when point estimating neural network weights we recommend both StdPrior and LearnedPriorlso for their simplicity and performance. We do not see enough of large consistent boost in discriminative performance from Shwartz-Zit Zii et al. (2022)'s 's low-rank covariance to clearly justify its downstream use as "default" approach. However, there are a few cases at small dataset sizes where Shwartz-Ziv Ziv et al. (2022)'s 's low-rank covariance is clearly better than Chelba & Acero (2006)'s 's isotropic covariance and may be a method to reach for to boost target task performance by a couple percentage points.
Limitations. We only consider transfer learning from one fixed source task, ImageNet. Further work could investigate informative priors may be more useful if the source task is more directly related to a target task of interest. To define the priors, also only consider one fixed mean vector and learned low-rank (with rank k = 5) covariance matrix taken directly from released snapshots by others. Others could find different behavior with other pre-training procedures, especially with more tuning of the rank on the target task.
Outlook. We stress that our ultimate goal is to improve the community's 's shared understanding about how to best achieve the goals of transfer learning, not to cast doubt on a specific past study. We hope that moving forward, two best practices will become more widespread. First, sharing code that allows reproducing all experimental results (including dataset preparation and hyperparameter tuning). Sccond, allowing all methods (especially baselines) the same time and compute resources for hyperparameter tuning.
Acknowledgments
Effort by authors EH and MCH was supported in part by a grant from the Alzheimer's Drug Discovery Foundation. Computing infrastructure support provided in part by the U.S. National Science Foundation under award OAC CC* 2018149.
We thank Ravid Shwartz-Zir Ziv, Micah Goldblum, and Hosscin Souri for answers to our questions, well making their codebase available. We also thank anonymous reviewers for suggestions that substantially improved the content and presentation of our paper.
10

---

Published in Transactions on Machine Learning Research (05/2024)
References
Lorenzo Brigato and Luca locchi. A Close Look at Deep Learning with Small Data. In 2020 25th International Conference on Pattern Recognition (TCPR), 2021.
Ciprian Chelba and Alex Acero. Adaptation of Maximum Entropy Capitalizer: Little Data Can Help a Lot. Computer Speech & Language, 20(4):382-33 -998929999921 2006.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic Gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning (ICML), 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning (ICML), 2020.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin Lesion Analysis Toward Melanoma Detection 2018: Challenge Hosted by the International Skin Imaging Collaboration (ISIC). Xivi preprint arxiv:100s 03368, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Gal Kaplun, Andrey Gurevich, Tal Swisa, Mazor David, Shai Shalev-Shwartz, and Eran Malach. SubTuning: Efficient Finetuning for Multi-Task Learning. arXiv preprint arXte:2302.0635, 2023.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (Canadian Institute for Advanced Research). 2010. URL sttp://www.c.. cs. ca..oronte edu/-kriz/cifar tet.trrrto.eeed/////a//a.aa....a html.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems (NewrIPS), 2017.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A Simple Bascline for Bayesian Uncertainty in Deep Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-Grained Visual Classification of Aircraft. arXiv preprint arXiv:1306.51 2013.
Kevin P Murphy. Machine Learning: A Probabilistic Perspective, chapter 3.3.3 Posterior. MIT Press, 2012.
Kevin S. Murphy. Probabilistic Machine Learning: An Introduction, chapter 19.2 Transfer learning. MIT Press, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing (ICVGIP), 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats And Dogs. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recogrition (CVPR), 2012.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca. Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Sigh-Porformance Deep Learning Library. In Advonces in Neural Information Processing Systems (NewrIPS), 2019.
Joelle Pineau, Philippe Vincent-Tamarre Koustuv Sinha, Vincent Larivière, Alina Beygekimer, Florence d' Alché Bue, Emily Fox, and Hugo Larochelle. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). The Journal of Machine Learning Rescarch, 22(1):7459 7478, 2021.
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-Train Your Loss: Fasy Bayesian Transfer Learning with Informative Priors. In Advances in Neural Information Processing Systems (NewrIPS), 2022. URL httpss///prroceeddinn wew.eeeddddnnnn.nnis....... cc/ paper
pdf.
Martin Spendl and Klementina Pirc. [Re] Easy Bayesian Transfer Learning with Informative Priors. In ML Reproducibilily Challenge 2022, 2023. URL.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1 9, 2018. URL https://dataverrs harvard. hatu/datsse..... xhtml IporsisteetIdddoi 10.
Xuhong, Yves Grandvalet, and Franck Davoine. Explicit Inductive Bias for Transfer Learning with Convolutional Networks. In International Conference on Machine Learning (ICML), 2018.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing Hc. A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE, 119(1):43 76, 2020.
A Classification
A.1 Dataset Details
Our experiments include a replication of CIFAR-10 (Krizhevsky et al., 2010) experiments from Shwartz- Ziv et al. (2022), similar Oxford Flowers (Nilsback & Zisserman, 2008) experiments to Shwartz-Ziv et al. (2022), similar Oxford-IIIT Pets (Parkhi et al., 2012) experiments to Shwartz-Ziv et al. (2022), additional FGVC- Aircraft (Maji et al., 2013) experiments, and additional HAM10000 (Tschandl et al., 2018; Codella al., 2019) experiments.
For Oxford Flowers experiments, modify prior experiments to enforce classes uniformly distributed in the training data, that at least one sample from each class is included in all tested values of and that the validation set is proportional with the training set. Shwartz-Ziv et al. (2022) validate on the predefined validation set specified by the dataset creators. In our minds, it not realistic to train on a range of dataset sizes of 102 or 510 images but validate on 1020 images. Practitioners looking for guidance in how to use transfer learning on their datasct will be unlikely to put more images in validation than training. Instcad, we ignore the predefined validation and use 1/5 of the training set for our two-phase training pipeline.
For Oxford-IIIT Pets experiments, modify prior experiments to enforce classes are uniformly distributed in the training data, and that at least one sample from cach class is included in all tested values of n.
For FGVC-Aireraft experiments, we combine the training and validation sets specified by the creators to enforce that the validation set is proportional to the training set.
For HAM10000 experiments, we form a classification task using classes with sufficient data (at least 10% prevalence). One additional "other" class was formed from all other classes that did not have sufficient.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Table A.1: Distribution of labels in HAM10000 training set.
Label
Percent of training sct
Melanocytic
66.95%
Melanoma
11.11%
Benign kcratosis
10.97%
Other
10.97%
prevalance. For the validation set, ensure each patient's data belongs to exactly one split and stratify by class to ensure comparable class frequencies.
We use the same preprocessing steps for all five datasets. For each distinct training set size n, we compute the mean and standard deviation of each channel to normalize images. During fine-tuning we resize the images to 256x256 pixels, perform random cropping to 224x224, and perform horizontal flips. At test time, we resize the images to 256x256 pixels and center crop to 224x224.
A.2 Classifier Details
We use SGD with a Nesterov momentum parameter of 0.9 for optimization. We train for 6,000 steps using a cosine annealing learning rate (Toshchilov & Hutter, 2016). We select the initial learning rate from 4 logarithmically spaced values between 0.1 to 0.0001 and the weight decay from 5 logarithmically spaced values between 1e-2 to le-6, as well as without weight decay. For LearnedPrior we select the covariance scaling factor from 10 logarithmically spaced values between le to le
While tuning hyperparameters,
hold out 1/5 of the training set for validation, ensuring balanced class frequencies between sets. At the smallest tested values of n suggested by Shwartz-Ziv el al. (2022), the number of training samples is equal to the number of classes, and thus any images assigned to the validation set will have their corresponding class not represented in the training data.
A After selecting the optimal hyperparameters from the validation set NLL, we retrain the model using the selected hyperparameters on the combined sot of all n images (merging training and validation). All results report performance on the task in question's 's predefined test set.
StdPrior. Stocastic gradient descent (SGD) in PyTTrch (Paszke et al., 2019) implements weight decay via an efficient direct modification of the gradient vector. Given decay parameter or 0, ileration / the optimizer internally performs additive incremental update of the gradient with respect to backbone weights: V v= vw + aw;-1 where f(w) is the (unpenalized) loss function (e.g. cross entropy). The gradient with respect to classifier parameters V is updated similarly. This direct modification is equivalent to, but more efficient. than, automatic differentiation of a penalized loss that includes an L.2 penalty term for each w and V:
a L(w, V) = log logp(ys/z,, w. V) + - |w| ||| + /eeee
(6)
((w.V)
Weight decay in PyTorch has a Bayesian interpretation as maximum a-posteriori (MAP) estimation of parameters w, V, under the assumption that the weights w, V each follow Gaussian prior AO0 with scalar precision parameter T I no Rescaling by n ensures that minimizing the above loss equivalent to maximizing the per-example MAP objective: [[logp(yla w, V) + log V)].
We use Shwartz-Ziv et al. (2022)'s 's SimCLR Resnet-50 pre-trained initialization to initialize weights w.
LearnedPriorlso. For LearneePriorrss,, we minimize the following MAP objective
n L(w, V)
log Log w, V) + 2 f|lw- 1112 + _ [ec())) Σ
(7)
71
so that the weight decay search space the same as SGD in PyTorch We use Shwartz-Zii Ziv et al. (2022)'s SimCLR Resnet-50 pre-trained initialization the mean i and to initialize weights w.
13

---

Published in Transactions on Machine Learning Research (05/2024)
LearnedPriorL... For LearnedPriorL., we minimize the following MAP objective
Mw, V) = Σ logpluil, w, V) logN(wllu, AΣ) a vec[[)]
(8)
Although not discussed Shwartz-Ziv (2022); they add & to the prior variance (they the default. value of to 0.1). We include the addition of & to the prior variance by adding A E + EI. We note that adding e to the prior variance has significant effect. if the covariance scaling factor is small (e-g., X 1).
B Sideeb Experimental Settings
Table B.1: A detailed comparison of Shwartz-Ziv et al. (2022)'s 's and our experimental procedures.
Shwartz-Ziv et al. (2022)'s
Ours
Normalization
In their code, they use the mean and standard deviation of the en- tire training dataset.
We use the mean and standard deviation of the training dataset of size n.
Data augmentation
their paper, they say they use random cropping and random hor- izontal flips. In their code, they use random vertical flips for some datasels as well.
We follow their paper, we use ran- dom cropping and random hori- zontal flips.
Batch size
They use batch size of 128. They use SGD with a Nesterov momentum parameter of 0.9 for optimization.
We use batch size of 128.
Optimizer
We use SGD with a Nesterov mo- mentum parameter of 0.9 for op- timization.
Gradient clipping
their paper, they do not men- tion gradient clipping. In their code, gradients are clipped so that the L2 norm is 2.0 by default. For Bayesian inference they train for 30,000 steps using cyclic learning rate (Zhang et al., 2019) with 5 chains. They never men- tion the number of steps they use for SGD-based methods.
We do not use gradient clipping. In our experiments, gradient clip- ping had little impact on perfor- mance.
Number of steps
We train for 6,000 steps using cosine annealing learning rate (Loshchilov & Hutter, 2016) to re- produce their Bayesian inference settings for our SGD-based meth- ods.
Initial learning rate
They select the initial learning rate from logarithmically spaced learning rates between 0.1 and 0.0001.
We select the initial learning rate from logarithmically spaced learning rates between 0.1 and 0.0001. We use coarser search space to reduced computational complexity. We expect a finer search space to improve on our results.
Weight decay
They select weight decay from 7 logarithmically spaced values be- tween le 6 and le le- 2, as well as without weight decay. They also divide weight decay by learning rate.
We select weight decay from 5 logarithmically spaced values be- tween 1e-6 and le 1e-2, as well without weight decay. We do not divide weight decay by learning rate. We use a coarser search space to reduced computational complexity. We expect a finer search space to improve on our results.
Continued on next page.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Table B.1: Continued from previous page.
Shwartz-Ziv Ziv et al. (2022)'s their paper, they plot test er- ror on CIFAR-10 for 10 logarith- mically spaced covariance scaling factors between 1e" and le (see their Fig. 2d) but don't specify search space for the hyperparam- eter. We assume they don't 't tune the covariance scaling factor since no search space is mentioned.
Ours
Covariance scaling factor
We select the covariance scal- ing factor from 10 logarithmically spaced values between le to le".
Validation set size
their paper, they don't say. In their paper, they don't say. In their code, they use loss.
We use 1/5 of the training set. We use the validation set NLL.
Hyperparameter selection metric
Rank of low-rank covariance
In their paper, they plot test error on CIFAR-10 and CIFAR-100 for covariance ranks between 0 and 9 (see their Fig. 2b) but use rank of 5 for experiments.
We use rank of for experi- ments.
Prior epsilon (scc App. A.2)
In their paper, they do not men- tion prior epsilon. In their code, the default 0.1.
We use 0.1.
Covariance scaling
their paper, they say to scale the covariance by multiplying AE. In their code, they scale the co- variance diagonal by multiplying AEdiag and the covariance fac- tor by multiplying XQ which cre- ates a multivariate normal dis- tribution with covariance matrix AEding + >EER They absorb the 1 into A.
We scale the covariance factor by multiplying vXQ which creates a multivariate normal distribution with covariance matrix & Ediag >LLR.
Double prior over weights
In their paper, they say add zero-mean isotropic Gaussian prior added parameters (c.g., classification head) to solve the target task. In their code, they use weight decay over all parame- ters.
We do not use weight decay over all parameters.
C NLL Results
Table C.I. CIFAR-10 heldout NLL (lower is better) as target train size n increases. For each method and train
set size n, we ran our two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (indepeudent samples of examples from the provided full training set). report mean NLL on test set across these trials, along with the min-max range. 10 classes, balanced. See App. A for details.
Method
n 10 (1/cl.)
100 (10/cl.)
1000 (100/cl.)
10000 (k//ll))
50000 (5k/cl.)
Prir fromIngNet
2.47 (2.10-))
1.48 (1.27-1.75)
0.71 (0. 00.66)0.79)
0.23 0.223333))
0.11 (0.10-0.
LearnedPrion Prirlss IngNet
2.31 (2.24-2.35)
1.40 (1.30-1.50)
0.68 (0.62 62-078)778)
0.18 (0.18-0.19)
0.09 (0.09-0.09)
LearnedPrion PriorL. fromIngNet
2.69 (2.30-3.39)
1.31 (1.20-1.44)
0.58 (0.56-0.58)
0.22 (0.21-0.23)
0.10 (0.10-0.11)
15

---

Published in Transactions on Machine Learning Research (05/2024)
Table C.2: Oxford Flowers heldout NLL (lower is better) as target train set size increases. For each method and two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (independent samples of n examples from the provided full training set). report mean NLL on test set across these trials, along with the min-max range. 102 classes, balanced. See App. A for details.
train set size
Method
n= 102 (1/cl.)
510 (5/cl.)
1020 (10/el.)
StdPrior fromImgNet
3.69 (3.40-4.23)
1.00 (0.96-1.05)
0.59 (0.58-0.61)
LearnedPriorfst Net
4.43 (3.62-5.14)
1.00 (0.96-
0.58 (0.57-0.59)
LearnedPriorT.R fromImgNet.
4.00 (3.59-4.81)
1.04 (0.95-1.14)
0.54 (0.52-0.58)
Table C.3: Oxford-III' Pets heldout NLL (lower is better) target train set size n increases. For each method and train set size n, ran our two-phase training pipeline (including hyperparameter tuning) distinct training (independent samples of n examples from the provided full training set). We report mean NLL on test set across these trials, along with the min-max range, 37 classes, balanced. See App. A for details.
Method
(1/cl.
370 (10/cl.)
3441 (93/cl.)
StdPrior fromIngNeet
3.33 (3.25-3.40)
1.78 (1.72-1.83)
0.59 (0.55-0.66)
LearnedPriorIso fromImgNet
3.62 (3.56-3.69)
1.79 (1.71-1.87)
0.58 (0.62-0.66)
LearnedPriorL. fro
3.62 (3.59-3.63)
1.09 (1.64-1.72)
0.52 (0.52-0.53)
Table C.:: FGVC-Aireraft heldout NLL (lower is better) target train set size increases. For each method and train set size we ran our two-phase training pipeline (including hyperparameter tuning) on 3 distinct training sets (independent samples of examples from the provided full training set). We report mean NLL on test set across these 3 trials, along with the min-max range. 100 classes, balanced. See App. A for details.
Method
" = 100 (1/cl.)
500 (5/cl.)
1000 (10/cl.)
5000 (50/cl.)
StdPrior fromImgNet
4.88 (4.71-5.18)
3.52 (3.43-3.57)
2.46 (2.40-2.50)
0.61 (0.60-0.62)
LearnedPriorIso froNIt
4.67 (4.66-4.68)
3.37 (3.23-3.63)
2.01 (1.94-2.05)
0.57 (0.56-0.57)
LearnedPriorL. fromImgNet
4.68 (4.66-4.71)
3.37 (3.33-3.45)
2.00 (1.96-2.03)
0.64 (0.60-0.67)
Table C.5: HAM10000 NLL on heldout data (lower is better) as target train set size n increases. For each method two-phase training (including hyperparameter tuming) 3 distinct training sets (independent samples of size n from the provided full training set). We report mean NLL on test set across 3 trials (min-max range). classes, ranging from 11 %-67% prevalance. See App. A for details.
and size
Method
100
1000
StdPrior fromingNet
1.16 (0.91-1.33)
0.90 (0.79-1.00)
LearnedPriorIso fromimgNet
1.16 (0.91-1.32)
0.83 (0.79-0.91)
LearnedPriorLR fromImgNet
1.35 (0.89-1.90)
0.89 (0.78-1.09)
16

---

Published in Transactions on Machine Learning Research (05/2024)
D
Comparison of Fully-Supervised Pretraining
StoPrier fromi muNetSuper (Kapiun at al, 2023)
StoPrior fromi maasSer (Shwarte Ziv
90
4.8
3.8
error
2.8 1
104
10
102
10a
10a
10
Number of training samples (log-scale)
Figure D.1: Error rate (lower is better) vs. target train set size on CIFAR- 10, for standard transfer learning from ImageNet using fully-supervised pre-training. The orange line results copied from Shwartz-Ziv al. (2022) (their Tab. 2). The green line is Uhird-parly experiment copied from Kaplun et al. (2023). Plotted mean and standard deviations confirmed via direct correspondencce with Kaplun et al.. Takeaway: In third-party experiments, standard transfer learning (StdPrior) performs better at dataset sizes n E (1000, 10000, 50000} than reported in Shwartz- Ziv al. (2022).
17

---

Published in Transactions on Machine Learning Research (05/2024)
E
Parameter Interpolation
random_ state=2001
Loos
Losc
3123:3236
random_ 1001
Loxs
Lo03
1331.1552
1114.5059
1112
random_ state=3001
Lass
Win
(a) Standard transfer learning (StdPrior)
(b) LeernedPriorlso
(c) LearnedPriorLR
Figure E.1: Expanded version of Fig. 2, including third column for the LearnedPrioriso method. Each panel: We assess a 1D slice of the high-dimensional landscape by linearly interpolating parameters w between the optima w found via minimizing the slandard MAP objective (left), the optima wis. found via minimizing the LearnedPrioriso MAP objective (center), or the optima wi. found via minimizing the LearvedPriorLit MAP objective (right) and the optima w² found the largest dataset size. Red curve shows the indicated training loss (varies column) CIFAR-10 with n = 1000 samples; blue curve (varies by column) shows CIFAR- R-11 test set NL.L. Each row shows results from different train set sample of size n 1000. The gap between the optima found via training and the test set's ideal minimum is shown as double-sided gold arrow.
18

---

Published in Transactions on Machine Learning Research (05/2024)
random 1atom__satt==001
Loss
Loss
Wsdd
WER
W
Gap
WiR
random =ttpsstttte=111
Loss
Loss
Wstd
WCR
W
Gap
random_starn 3001
Test loss
Train loss
Loss
Loss
Word
Gap
WCR
W
(a) Standard transfer learning (StdPrior)
(b) LeurnedPriorlA.
Figure E.2: Alternative version of Fig. looking 1D slice that interpolates between the optima found by each method instead the optima found by each method and the optima found at the largest dataset size. Each panel: We assess 1D slice of the high-dimensional landscape by linearly interpolating parameters between the optima W found via minimizing the standard MAP objective and optima wis found via minimizing the LearnedPriorL. MAP objective. Red curve shows the indicated training loss (varies by column) CIFAR-10 with 1000 samples; blue curve (same across columns) shows CIFAR-10 test set NLL. Each row shows results from a different train set sample of is shown as double-sided gold arrow.
1000. The gap between the optima found via training and the test set's minimum on the 1D slice
19

---

Published in Transactions on Machine Learning Research (05/2024)
F
Comparison with Full Bayesian Inference
FulBayes LearnesProrrR fromimgvetSimc... (Shwartz-Ziv et al,
202 2)
MAP StdPrior fromi miNeetimeLR (ours)
90
4.8
65
3.8
error
40
2.8
5 10
15
10a
102
10a
10
10
Number of training samples scoges
Figure F.1: Error rate (lower is better) vs. target train set size on CIFAR-10, for standard transfer learning and full Bayesian inference with LearnedPriorL.t from ImageNet. The red line are results copied from Shwartz-Z4 et al. (2022) (their Tab. 10). Takeaway: Our standard transfer learning (StdPrior) results on CIFAR-10 are good as or better than full Bayesian inference results reported Shwartz- Ziv al. (2022).
G Hyperparameter Sensitivity
1.0
0.8
Accuracy
0.6
0.4
0.2
StdPrior fromingMetSim CLR LeameePriariso frominngettSmmcL LeamedPrtorLR fromimcNeetiincLR
0.0
Best
2nd best 3rd best 25th %ile 50 %ile
Figure G.1: Sensitivity of selected hyperparameters for CIFAR-10 1000. We report CIFAR-10 heldout accuracy (higher better) with various selected hyperparameters as validation NLL increases.
20


Reviews:
[{"review_id": "5eXjzFMwOy", "reviewer": "Reviewer_M6Gg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper makes a concerted effort to produce a clear and reproducible replication of the procedure of Schwartz-Ziv et al. I particularly appreciated the table in the appendix which included a description of key factors where the empirical choices in this work differed from those of the original paper proposing learned priors.\n- The claim that “ standard transfer learning without any informative prior performs better than reported in Shwartz-Ziv et al. (2022).” is well-supported by Figure 1, at least with respect to the CIFAR-10 dataset. \n- The paper performs some nice analysis which fleshes out the claimed mechanism by which learned priors help improve performance via alignment between train and test loss landscapes actually occurs when using learned priors. In my experience with the Bayesian Deep Learning literature, it is very common to claim a particular mechanism (usually motivated by some connection to Bayesian inference) is driving the success of a method without rigorously testing whether this is the case, and I appreciate the paper’s work aiming to falsify these claims empirically.\n- The paper is very well-written and clearly describes and justifies its methodology.\n\n\n**Weaknesses**\n\n- Schwartz-Ziv et al. evaluate their method on 4 benchmarks in their Figure 3. Of these, only two are evaluated in this paper. In order to be fully convinced of the claim that learned priors do not make a difference to performance, I would want to see the analysis of Figure 1 replicated for each of these datasets.\n- The claim that learned priors do not improve alignment of the loss landscape requires additional evidence. Figure 2 seems to show very good alignment, in the sense that the train and test losses roughly agree on local minima. Figure 1 of Schwartz-Ziv et al. shows examples where a local minimum of the train loss is far from a minimum of the test loss. \n  - I am also not sure if the choice of endpoints of the linear interpolation necessarily makes the most sense. If I understand correctly, the interpolation is performed between the minima found by the standard and learned prior MAP objectives. I think in the idealized figure 1 from Schwartz-Ziv et al. the visualization was intended to be from the initial parameters of the fine-tuning phase. \n    - Given that the authors already have the experimental aparatus for evaluating loss landscapes, I would suggest contrasting the loss landscapes of the learned prior with those obtained by regular L2 regression as well as a learned prior with isotropic covariance (and for good measure perhaps also using the low-rank covariance matrix with a zero-centered prior), so that we can not just see whether the loss landscapes are aligned but whether they are more aligned in one method compared to another.\n- One difference between this paper and that of Schwartz-Ziv is the use of SGD in lieu of HMC when training with learned priors. I would assume that a learned prior would shine more when used with a Bayesian training method, and so could believe that some of the effect that is observed in this paper might be attributed to the use of SGD as opposed to HMC. Can the authors comment on this choice?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a replication study of the findings of Schwartz-Ziv et al. which suggested that learned Bayesian priors can improve sample-efficiency in transfer learning over point estimate initializations. It finds that the purported benefits of learned priors over simpler pre-training baselines are largely a result of poor tuning of the baselines, and that with appropriate tuning these simple baselines exhibit minimal reductions in performance relative to, and in some cases even exceed the performance of, the method of Schwartz-Ziv et al. This analysis is extended to contrast low-rank and isotropic prior covariance matrices on a variety of tasks, including HAM10000, which was not considered by Schwartz-Ziv et al. Finally, the paper also studies the loss landscapes of networks trained with learned priors, visualizing the alignment between train and test losses in a one-dimensional region of parameter space obtained by interpolating between two local minima.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See \"weaknesses\" above. I also would like to see the following more minor questions addressed.\n\n- Does the pre-training procedure exactly match that of Schwartz-Ziv et al.?\n- How does the experimental protocol used to generate Figure 1 differ from Table 2? Or are they based on the same data with Table 2 incorporating a second baseline (LearnedPriorIso)? \n- Why are error bars not provided in Figure 1? Am I correct in interpreting this as only presenting two seeds per dataset size? If so, how does this correspond to the claim that three different randomly selected datasets were used in the evaluations?\n- It would be useful to indicate what the different rows correspond to in the plot titles in addition to the description in the legend of Figure 2.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nLJvn2jsDN", "reviewer": "Reviewer_uKdC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Machine Learning is going through a reproducibility crisis, due to the lack of reliable experimental setups and abundance of overclaiming, so paper is definitely of interest to someone in the TMLR community. Given the topic of transfer learning is currently very popular, and the method under consideration is growing in popularity, I think the paper is very well motivated.\n* The paper is well written. It provides a good introduction to the problem formulation of the class of methods considered and clearly explains the scope of the claims being made.\n* The paper does a good job of designing an experimental setup that mimics the setting considered by Shwartz-Ziv et al. (2022), with extensive effort to exam why the results are different given in the appendix.\n* Findings 1 and 2 are interesting and have practical significance to the research community.\n\nWeaknesses:\n* A limitation of the work in Shwartz-Ziv et al. (2022) is the lack of fine-tuning tasks considered in the evaluation. This weakness is carried to the present work, and means that the conclusions drawn in this paper can only be relied upon with low confidence.\n* The experiment investigating the alignment loss surfaces seems flawed. The one dimensional slice obtained by interpolating between two different optima is very unlikely to contain an optimum for the test data. Looking at the precise distance between the optima that is found and the optimum for the test data on this one dimensional slice will not be a good approximation of the actual distance between the train and test optima.\n\nSome questions:\n* For the repeated trials discussed in Section 3, are the replicate training sets sampled with or without replacement?\n* Regarding hyperparameter tuning, why is a lower dimensioned space of hyperparamters search for LearnedPriorLR?\n* Why is NLL used as the validation criterion? Typically this would be accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the performance of the LearnedPriorLR method of Shwartz-Ziv et al. (2022) in the context of transfer learning. The paper finds that the method does not have a performance difference with the baseline, as claimed in the original paper, and that the alignment of the loss surfaces of the train and test data is not as good as claimed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Adopting the broader range of benchmarks used in the (dense) fine-tuning literature (e.g., from Li et al. (2018) or Gouk et al. (2021)) would allow for a more reliable set of results.\n* The experiment investigating the alignment loss surfaces should be rethought. A more reliable way to investigate the alignment of the loss surfaces would be to consider the distance between the train and test optima in the full parameter space, rather than on a one dimensional slice.\n* It would be nice if the experiments could be re-run with a more uniform hyperparameter optimisation procedure across all methods considered, and using the validation accuracy rather than NLL.\n\nLi et al. Explicit inductive bias for transfer learning with convolutional networks. In ICML,2018.\n\nGouk et al. Distance-Based Regularisation of Deep Networks for Fine-Tuning. In ICLR, 2021.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "odUhjwx9om", "reviewer": "Reviewer_KTv2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**strengths**\n\n- The paper uses careful experimental design to independently verify the claims made by another paper and reaches more nuanced conclusion. This sort of work is very important and perhaps not as valued as they should be. \n- The paper is carefully written and very easy to follow and presents a balanced view of the issue of prior works.\n\n**weaknesses**\n\n- I do not see any major flaws in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the empirical performance of using informed prior in transfer learning. The primary method of focus is [1] who proposed that instead of learning a point estimate of a good initialization from pretraining data, one could learn a distribution of weights in the form of mean and a low-rank covariance and perform the finetuning is a Bayesian manner. [1] found that their proposed Bayesian finetuning method performed significantly better than the baseline which is simply initializing the backbone at $\\mu$ which is the weights of the pretrained model and apply standard $\\ell_2$ regularization during finetuning / transfering. However, this work compare the performance of [1] against a naive baselinea and a baseline that only use isotropic posterior without the low-rank covariance with all their hyperparamters properly tuened. The result shows that the naive baseline can perform much better if the hyperparameters are properly tuned. In fact, there is almost no gap between the naive baseline and that of [1] which raises the question about whether the added complexity is actually needed. Furthermore, the paper provides a counter example to the loss-alignment hypothesis.\n\n**Reference**\n\n[1] Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors. Shwartz-Ziv et al.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "If possible, I would like to see a visualization / analysis of how the performance varies as a function of the hyperparameters (e.g., some kind of sensitivity analysis) for all methods. I feel like something like that would be beneficial to further understanding the strength and weaknesses of the studied methods.", "broader_impact_concerns": "I do not see any concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers were leaning accept and two out of three reviewers recommended the paper for a reproducibility certification.
Decision: Accept (poster)