Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents numerous empirical facts about the distribution of the weights after training a neural network. The authors considered one FCNN, one CNN, and one ResNet, and computed several statistics on their weights.

The authors also propose, in a Bayesian setup, to compare the performance of trained BNNs with different priors.
main_review: # Weaknesses

 * There is no clear take-at-home message.
 * The authors generalize their results on *one* FCNN to *all* FCNNs and on *one* CNN to *all* CNNs, on one task. The consistency across models and datasets should have been checked.
 * There is no heuristics or explanation of the observed results (except for the influence of the data augmentation on the "cold posterior effect".

# Strengths

The experimental report is interesting for researchers who want to have a better intuition of the influence of the prior.

# EDIT

I have taken note of the modification made by the authors in the Discussion section, which moderates a bit the main claims. But I do not change my recommendation, since the main claims (in the first sections) are still too affirmative regarding the experiments.

Anyway, as an impressive empirical work, this paper deserves to be published. 
summary_of_the_review: Weak accept.

Despite the numerous and contradictory experiments (heavy tailed priors are better for FCNNs, but worse for CNNs), and the difficulty the authors have to interpret their results, the empirical results are significant enough for helping other researchers to guide their research. 

For instance:
 * a heavy-tailed distribution of the weights is not necessarily a problem;
 * the distribution of weights after training a FCNN and a CNN may be structurally different;
 * the "cold posterior effect" can be partly explained by data augmentation;
 * the "cold posterior effect" can be removed when using a prior with heavier tail.

A major issue remains: the authors should have checked that their results about "FCNNs" and "CNNs" are consistent for *several* FCNNs/CNNs and several datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work studies prior distributions for Bayesian CNNs. The work report that conventionally used priors e.g., Gaussian poorly fit empirical distributions of the trined weights. The empirical distributions appear to be heavy-tailed and correlated. Thus the paper proposes to use "heavy-tailed priors" for FCNNs and correlated Gaussian priors for CNNs that tend to improve classification performance.
main_review: __Comments__:


- The results are kind of unstable, on the many plots (Figure 5, Figure A.19, Figure A.20) Gaussian prior behaves similarly or better than heavy-tailed analogs. 


- The results on the cold-posterior effect are also unstable between models, which is reported in the paper.


- SGD baselines are undertrained (Figure 5, Figure A.11), e.g., ResNet-20 (w/ data aug.) error should be approx 8%, while 10% is reported. That is a quite important issue that may significantly flip the results of the experiments.


- The work studies only a single inference technique, while the results will not necessarily generalize between inference techniques. So, it is hard to make any general conclusions.
summary_of_the_review: The work studies an interesting problem of prior selections for BNNs. At the same time, the results of the two main experiments are mixed and do not clearly demonstrate the importance of the heavy-tailed priors. The study also considers a single inference technique. Overall that makes me conclude that results are not generalizable and reliable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an analysis of priors other than standard isotropic Gaussians for Bayesian neural network. The authors first empirically estimate the posterior distributions of the weights of a BNN and then use these distributions as a prior for these BNNs. They show that those priors allow better performance of BNNs. The paper discusses this analysis in the context of the cold posterior effect.
main_review: *Update after rebuttal* I have read the authors’ response and other reviews. I would like to thank both the authors and other reviewers for providing an interesting discussion. Although the other reviews raise valid points on how to make the paper stronger, I believe it is worth publishing in its current form.

======= 




Strong features:
1. Very well written paper. I really enjoy reading this paper. One can rarely see this style nowadays with more textbook, discussion-like language. 
2. Very thorough experiments. Although it should be expected from a paper which main contribution is empirical evaluation of a phenomenon, but not always empirical papers hold this expectations. This submission does. Everything is considered from different angles, everything is tested - different activation functions, different variance, convergence, etc. 
3. The paper is well placed in the context of the current literature. 
4. The paper provides interesting insights into the default choice of the prior for BNNs as well as provides a recipe to explore this direction further.

Weak features:
I do not really see major weak points for this paper. Of course, there is always a room for improvement in anything, but in this case it would be "nice to have", rather that "should/must have". It would be interesting to see other types of priors, bigger datasets, other domains, but the current experiments have already taken 10,000 GPU hours as claimed by the authors. 
The actual insight from the paper, i.e. that isotropic Gaussians are not the best choice of prior, may be not very exciting and surprising, but the paper provides a thorough analysis to show this and a recipe how to choose a better prior.

Even with a non-surprising result, I like this paper and enjoy reading it. I believe it provides useful and interesting discussion to the community and therefore I recommend acceptance of this paper. There is no anything particular novel in terms of the algorithms and methods in the paper, but rather a very well done empirical analysis that addresses a relevant problem. 

Specific suggestions/comments (of different significance for assessment, but mostly for further improvement of the paper):
1. The acronym SGD in abstract is not introduced
2. The second paragraph in Introduction. "at lower temperatures" - a temperature has not been introduced yet
3. Eq. (1) - strictly speaking x and y should be defined separately
4. In the empirical analysis of weight distributions (Section 3), it is claimed that the posterior distribution reached by SGD would be a good choice for a prior. However, the posterior distribution family = the prior distribution family only when a prior is conjugate to a likelihood. This argument is therefore should be made more carefully
5. Figure 2a. It would be interesting to see some explanation/discussion of why the degree of freedom decreases between L14 and L19
6. Details on data augmentation is missing
7. Figure 3. It is claimed that the strength increases for later layers, but the max variance is less for them
8. The acronym SGLD is not introduced
9. Section 4. “To the best of our knowledge, this procedure constitutes the best SGLD-based inference approach” - before that it was stated that the authors used a combination of 3 different methods, therefore this claim looks unjustified
10. Figures 4-… What do shaded areas represent?
11. For the correlated Gaussian, it is not discussed why the Matern kernel in particular has been chosen
12. Sec A.1. “The covariances of the FCNN weights are shown in Figure A.5” – seems that it should be Figure A.1
13. Figure A.10 is not referred to in the text
14. Figure A.12 – not sure I can agree with the conclusions about heavy-tailed distributions made in the caption
summary_of_the_review: I vote for acceptance of this paper. I believe it provides a thorough empirical insights on the prior choice for BNNs and a recipe for future prior exploration.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use distributions different from isotropic gaussian for Bayesian neural network priors. For fully connected neural networks the heavy-tailed distributions such as laplacian and student-t are used, while for convolutional neural networks multivariate gaussian with spatial correlations are employed. Empirical results show that these alternative priors give better results than isotropic gaussian on several image classification tasks. Cold-posteriors versions of the isotropic gaussian prior and the proposed priors are also provided for comprehensive comparison. Overall this is an interesting paper with novel discoveries.
main_review: Strengths:
1) The paper took an empirical approach by first analyzing the weight distributions of the SGD trained model, then identifying the characteristics of these distributions, and finally proposing priors with the corresponding characteristics to facilitate the Bayesian learning. An expectation-maximization procedure is provided to justify the above approaches and empirical results support the choice of the priors.

2) The impact of these new priors on different type of networks are discussed, especially the cold-posterior effect which relates to the setting the the temperature in the posterior.

Weakness:
1) It would be great if the impact to the computational speed could be discussed further. Would the new priors drastically increase the computational demand, or the computation needed is similar to the isotropic gaussian prior?

2) As discussed in section 3 equation (2), SGD is used to sample from the posterior on w. However, it is unclear to me why SGD could be used to sample from the posterior weights of the networks. According to my understanding, a proper tool to sample from neural network posteriors is stochastic gradient Langevin dynamics (https://icml.cc/2011/papers/398_icmlpaper.pdf) rather than SGD. The "stochastic" part of SGD is with respect to the batch sampling from the data distribution, not with respect to the weight parameters. More clarification is needed regarding this choice of the weight sampler.

3) One motivation of the work from the introduction section is to impose proper prior distributions on the network weights, so that principled Bayesian inference can be carried out without temperature adjustment as in cold-posterior. However, according to my understanding, when the temperature is set to be 1, the results with the proposed priors do not always match or outperform the (non-Bayesian) SGD result (see Figure 4 and 5). This somewhat weakens the motivation and impact of the paper.
summary_of_the_review: This paper studied the important question of what would be a proper prior for Bayesian neural networks, and proposals were made based on empirical investigations of weight distributions of different neural networks. Experimental results on some image datasets validated the choice of the priors. In summary this is a nice paper with interesting findings. Still there are some technical details that need to be addressed (see my comments above).
review_title: null

Meta-review: This paper provides some empirical investigation of the choice of the prior distribution for the weights in Bayesian neural networks. It shows empirically that, when trained via SGD, weights in feedforward neural networks exhibit heavy-tails, while weights in convolutional neural networks are spatially correlated. From this observation they show that the use of such priors leads to some improved performances compared to the iid Gaussian prior in some experimental settings.

Reviewers have conflicting views on this paper, that have not been reconcilied after the author's response and the discussion.
On the plus side, the paper is very well written, the experimental part is carefully conducted, and provides some insights on the choice of the prior in Bayesian neural networks, which could lead to further developments. 
On the negative side, the claims made in the introduction are not fully supported by the experiments (the claims have been slightly amended in the revised version), and the take-home message is not so clear. In particular, Bayesian approaches with the proposed priors still underperform compared to SGD without tempering. The authors could also have considered a broader sets of experiments.

Overall, I think the contributions outweight the limitations of this paper, and I would recommend acceptance.
Decision: Accept (Poster)