Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims at better understanding of the sources of the cold posterior effect (CPE) in Bayesian deep learning. It investigates the effects of some of the hypotheses proposed in the literature for this phenomenon, in particular: dataset curation, data-augmentation and bad priors. It examines each hypothesis via carefully designed experiments and empirically shows how CPE can arise from each of these sources in isolation, suggesting that there is no single simple cause.
main_review: This paper does not provide a solution to the CPE problem. However, it aggregates some of the most prominent existing hypotheses and does a valuable effort to shed more light on the sources of the problem. The experiments are designed to disentangle different potential sources of the CPE problem.

Even though the paper does not propose a new hypothesis, its overview of the existing hypotheses and clarifications is valuable to the community.

The paper is very well-written and easy to follow and does a good job of covering the background and explaining its approach.
limitations_and_societal_impact: The only limitation I see is that the experiments of the paper are all classification tasks. I am wondering if any of the results would change for regression problems?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a series of experiments on the cold-posterior effect, and argues that it is broadly prior-dominated,  rather than arising from e.g. the data curation hypothesis of Aitchison (2021).  While it is an interesting series of experiments that clearly deserve to be published, I have a number of concerns about the interpretation of the experiments.  It should be possible to deal with these during the rebuttal phase, at which point I'd be happy to increase my score.
main_review: General comments:
The figures need significant reformatting.  The fonts are tiny (basically unreadable from my printer).  And Figures should appear at the top, not in line.

Sec 2
=====

Worth noting that for large-scale ResNets (i.e. the model class we care about), Fortuin et al. 2021 show that more complex priors improve performance at all temperatures so the CPE remains basically unchanged (Fortuin et al. 2021; Fig. 7 bottom (CIFAR-10))

Sec 3
=====

Why isn't Figure 4 referenced here?  Why is Figure 4 placed after the Figures for other sections?  Except in exceptional circumstances, figures should just come in the order they are discussed.

Important to note an alterantive hypothesis consistent with the data.  In particular, if you believe the CPE is "variability driven" and works by suppressing stochasticity in the distribution over parameters.  In that case, there will be less CPE for more data because with more data the posterior is already highly concentrated.  If nothing else, if you have lots of data, then the posterior converges to the ML value of the parameters, and taking a cold posterior will have no effect at all, and this holds for any choice of prior!


Sec 4
=====

No CPE without data augmentation
--------------------------------

Very clean results without augmentation.

Conclusion that inaccurate inference is not necessary is correct.

"We can also conclude that the curation of SVHN and CIFAR-10 does not give rise to a CPE".  Only if the true model does not include augmentation.  But it is quite possible  that the true model incorporates data augmentation in the prior, as introduced in arxiv.org/abs/1808.05563, and as discussed further in contemporaneous work arxiv.org/abs/  2106.05586).  In particular, Bayes theorem only tells us that the true model should perform best.  But it doesn't say anything about the relative performance of             misspecified models.  So it is quite possible that a model with curation + augmentation (i.e. priors capturing data augmentation, as in arxiv.org/abs/1808.05563) is best,   while under the wrong model without augmentation, no curation beats curation.

CPE and synthetic curation
--------------------------

Correct.  In essence, including curation in the model makes the posterior more certain everywhere, which hurts performance on the ambiguous examples which would otherwise   be curated away.

Do standard Gaussian priors give too much weight to complicated hypotheses?
---------------------------------------------------------------------------

Isn't the more complicated distribution at T=1 better than the straight-lines at T=0.01?  The decision boundary should be more uncertain far from the data.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the so-called cold-posterior effect (CPE), which refers to the poorly understood previous empirical observation that Bayesian posteriors sharpened by a temperature parameter T<1 perform better that the "correct" Bayesian posteriors corresponding to T=1. To me, the paper makes two nice contributions: first, a thorough discussion summarizing all previous explanations for the CPE from the literature; second, an insightful empirical analysis (or re-analysis) of the putative explanations and their impact on the CPE. The main finding was that the CPE does not have a single cause, but can be caused by any of: 1) dataset curation, 2) data augmentation, 3) bad priors. In particular, I think the paper implies that data augmentation and especially the prior have a larger role than previously appreciated, while the impact of dataset curation may not be that significant (although it's non-negligible in some setups, too).
main_review: PROS:
- A nice investigation of a relatively poorly understood phenomenon of the CPE, with novel insights that increase understanding, and I expect these results to be significant within the Bayesian DL community.
- Very clear presentation.

CONS:
- The paper identifies many causes for the CPE, but does not offer much practical guidance on how to avoid this issue in practice.
- The results are mainly empirical, without theory to back up the findings (I agree this might not be straightforward to come up with).

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: Another nice contribution to the ongoing community effort of understanding Bayesian deep learning.

All reviewers liked the paper.

Accept.

Decision: Accept (Poster)