Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This submission shows how an approximate posterior distribution on DNN weights learned on a source task can be beneficial in transfer-learning to downstream tasks. 

A 3-step method is proposed: 1) Fit posterior over weights on source task using SWAG. 2) Re-scale the learned posterior using a single scalar coefficient. 3) Plug the re-scaled prior into a Beysian inference algorithm like SGLD or SGHMC.

Experimental results are provided to support the following claims:

- In terms of performance, learned priors > SGD transfer learning > non-learned priors.
- SGD with learned priors significantly outperforms standard transfer learning (i.e., learned priors lead to performance gains with or w/o Bayesian inference).
- Learned priors lead to more data-efficient performance on the downstream task.
- When using the same prior Bayesian inference often outperforms SGD training -- though Bayesian inference with a non-learned prior has inferior accuracy to SGD with pre-training.

strengths_and_weaknesses: # Strengths

**S1.** The method is well-motivated, presented with clarity and shown to lead to performance gains.

**S2.** The insights provided prompt further exploration of loss-surface alignment in transfer learning and other settings (e.g., few-shot learning).

**S3.** The method builds on prior art and adds a single hyperparameter. For this reason it should be easy for others to apply.

**S4.** The experimental validation includes classification tasks of different complexities and semantic segmentation. Thus, the method is shown to be beneficial in interesting problems.

# Weaknesses

**W1.** The proposed method claims to be a drop-in replacement for standard transfer learning.Because the additional computation is not expected to be trivial, it would have been beneficial to provide a proper study of the expected increase in computation w.r.t. standard transfer learning.

The submission includes some discussion about the additional training and inference cost but no discussion about the cost (in practice) of tuning hyperparameters. For example, there are a number of hyperparameters for the methods involved in the different steps of the proposed method. Setting or tuning these hyperparameters for a given transfer would be part of the cost of the method, in particular for steps 2 (scaling $\lambda$ of prior) and 3 (parameters of SGLD and SGHMC).

**W2.** The main technical novelty in the method is the scaling of the prior (steps 1 and 3 are applications of prior art) but this was not given much attention in the experiments (only Fig. 2c). What is the recommended procedure and expected cost for tuning the scaling parameter $\lambda$?

**W3.** Plots, legends and captions are not synchronized. For example, in Fig. 2c there is a purple plot but the legend does not include purple; in Fig. 7 the caption speaks of a red plot but there is no red in the plots.

questions: **Q1.** What would be expected increase in computation (tuning included) w.r.t. conventional transfer learning for applying the proposed method to common tasks?

**Q2.** The method is compared to deep ensembles with regards to accuracy. How does the method compare with regards to overconfidence?

limitations: The two limitations discussed are the additional computation and the inclusion of only vision applications.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a Bayesian learning method for transfer learning on target task. Specifically, a posterior approximation method SWAG is used to estimate the posterior distribution of a supervised or self-supervised pre-trained model. This distribution is used as the prior for downstream learning. Experiments and ablation study demonstrate the effectiveness of learning a prior for transfer.
strengths_and_weaknesses: Strengths

1) The problem of learning a prior for transfer with Bayesian learning is well motivated.

2) The proposed method with three steps is reasonable.

3) Experimental results of the proposed method on semantic segmentation are quite strong.

Weaknesses

1) This paper employs methods of posterior approximation and Bayesian learning and incorporates them into Bayesian transfer by learning a prior from source tasks. These existing methods have been well developed so the overall technical contribution is incremental upon these.

2) The method introduced in section 3.2 looks a bit arbitrary. In order to slightly smooth the prior distribution, this paper proposes to multiply a single scaling factor to the covariance matrix. This does not seem to be a principled way to adapt a prior distribution to a target task. Are there any other possible methods of adjusting the covariance matrix for transfer? The holdout validation method used to select this scaling factor also neglects other data information that is potentially useful for determining the scaling factor.

3) The background review on Bayesian transfer learning seems not very thorough, which makes the overall contribution of the proposed transfer learning with Bayesian learning unclear. The only relevant review on this direction is about continual learning.

4) The proposed method will introduce additional computational burden for transfer learning.



questions: Q1: Most of the evaluations in experiments focus on the performance of the learned prior when transferred to a new target task. I'm not sure if it is possible to use Bayesian model average can on source task itself, as it is not clear how good is the approximated posterior distribution. This study on the posterior approximation on the source task can probably show the effectiveness of estimation with close-form density as well as Bayesian learning.

Q2: The rescaling method proposed in section 3.2 is not well developed. Are there any related work for the proposed scaling method? Practically, from Fig. 2d, it seems that the scaling factor is very big. It seems that it is necessary to tune this scaling factor for every new target dataset. Is it possible to estimate this parameter without using holdout validation set?

Q3: Transfer learning results on target classification tasks are reported mainly using test error. A more common comparison is classification accuracies as error depends on the loss function. As seen from the tables the test error of several baseline methods are quite close. What are the classification accuracies for these results?


limitations: See weakness and questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a Bayesian perspective of transfer learning in deep neural networks, whereby a re-scaled Bayesian parameter posterior from a source task is used as a pre-trained prior for a target task. The proposed procedure effectively reshapes the training objective of the target task to more faithfully reflect knowledge learnt from the source task. The authors find that modifying the loss surface of target tasks through informative priors significantly improves performance and calibration, especially with Bayesian inference. 
strengths_and_weaknesses: **Strengths**:

- Highly important and relevant topic for the research community;
- Clear and organized exposition;
- Well motivated arguments and experiments.

**Weaknesses**:
- Proposed approach can incur significant additional computational costs;
- May require expert knowledge to use effectively.

Overall I like this paper. I find the arguments and experiments convincing enough and believe it would be valuable for the research community. One concern is whether the proposed Bayesian inference based transfer learning pipeline would be of limited utility to regular practitioners, as expert knowledge may be required to get these systems working stably as intended. There is also the incurred computational cost which may deter uptake if the performance gain does not justify it. 
questions: Since transformers (visual or otherwise) are the main subjects of transfer learning nowadays, it would make a lot of sense to evaluate them too. It is also known that transformers incur different loss landscapes than typical CNNs such as ResNets, it would be interesting to see how performance, robustness and calibration are effected by the proposed transfer learning policy. 
limitations: See above.
review_title: null

Meta-review: metareview: This work presents a Bayesian method for transfer learning using SWAG. Reviewers agree that this is well-motivated, it's novel and the proposed method is well done and works well. There are some concerns about the computational burden, but the authors claim that the proposed part adds about 1/7 total cost. I share some of the concerns with one of the reviewers regarding the fact that this method may be limited in usefulness to people who are experts, rather than the more general public. However, given that the method builds on prior art and adds a single hyperparameter, I feel it should be relatively easy for someone to actually use this, if they are interested in transfer learning.
confidence: Certain
Decision: Accept