Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Apologies for the late review!

## Summary of the Paper

The paper provides a potential theoretical explanation of the known empirical observation that cold (or tempered) posteriors improve predictive performance of deep Bayesian neural networks. The provided explanation is simple and it leads to additional predictions, which the authors check empirically as far as possible with existing data sets. The empirical results agree with the predictions.

## Main Strengths

I believe the main message of the paper is so relevant and seems so simple (at least in hindsight) that it has the potential of becoming a kind of "common knowledge" in Bayesian neural networks community (caveat: I can't judge if these findings had already been informally known to a larger group of researchers, but unless someone has explicitly written them down somewhere I wouldn't hold this against the paper).

Researchers have been trying to increase predictive performance of deep neural networks by applying scalable Bayesian methods to deep learning for a while, but even replicating the performance of point estimated models with Bayesian neural networks has proven surprisingly difficult. To my knowledge, it was found out only recently that Bayesian neural networks systematically outperform their point estimated counterparts if the prior is made artificially sharper than what probability theory would predict (i.e., by "lowering the temperature"). This empirical result has been puzzling from a theoretical perspective, but the present paper provides a simple potential explanation for this effect.

I believe the findings in this paper go beyond a theoretical justification of an empirically known fact. The findings may also have implications on model robustness: the authors argue that the "tempering effect" is a result of curation of the training set. Validations and tests sets are typically curated in the same way as the training set in the machine learning community. However, when models are deployed in the field, they typically see uncurated data points. I would be curious to know if explicitly modeling the curation process, as the authors do in this paper, would also address this issue.

## Potential Weaknesses

There's one caveat to my review: I am not an expert on Bayesian neural networks and, as stated above, the argument made in this paper seems so simple in hindsight that I cannot say with absolute certainty that it hasn't been made before. I personally haven't heard this argument before, but if other reviewers can point to a reference that already made this argument, then that would probably be the only thing that could convince me to lower my rating. Otherwise, I would consider the simplicity of the authors' argument a strength of the paper.

## Questions to the Authors

- What do the authors mean with the phrase "finite networks" in the first paragraph? Is it the same as networks with point estimated parameters (as opposed to Bayesian neural networks)?
- As mentioned above, the paper models the curation of the training set, but I didn't understand how or if curation of the test set is modeled. Could the authors clarify this? Specifically, what changes for a model trained on curated data when it is either (a) tested on an equally curated test set or (b) applied to uncurated data in the wild? Would the optimal $\lambda$ during training differ between cases (a) and (b) or would the posterior have to be changed after training?

## Minor Issues

- I like the short Section 2 (which compares "cold" and "tempered" posteriors) very much! I think it could even be improved by adding one reference each for "cold" and "tempered" posteriors, respectively. More importantly, as far as I understand, the two are really essentially the same if one uses, e.g., a Gaussian prior. Unless I'm mistaken, the missing factor of $\frac{1}{T}$ in Eq. 2 could then be absorbed into a rescaling of the prior covariance (unless the prior covariance itself is learned with expectation maximization). If this is correct then I'd add a corresponding statement at the end of Section 2 (it would make the paper's claims more widely applicable).
- I think Figure 2 is never discussed in the paper (but there should be enough space left to discuss it). The figure caption says "Schematic diagram". Could the authors clarify what this means? Do the points come from some toy model with a 2-d parameter space (or does the figure show a 2-d PCA of the parameter space) or were the points really just drawn manually to visualize the idea? I think both would be fine but I would be very curious to know how the figure looks with real data.
- In Figure 4, the last panel is labelled "F" but referred to as "E". Also, in the last two panels, the left dashed vertical bar is not discussed. Is it the theoretically expected optimal value of $\lambda$ (i.e., $\lambda=\frac{1}{4}$) or the empirically found optimal value?
review_title: What a beautiful paper!

final_score: rating: 7: Good paper, accept
review_content: This paper addresses the perplexing issue of cold posterior having better predictive performance than the ideal Bayesian posterior in Bayesian deep learning (Wenzel et al., 2020), and offers a possible explanation in terms of a mis-specified likelihood function that deviates from the true generative process of the data. By considering the data curation process and augmenting the likelihood model accordingly, the effect of cold posterior is shown to diminish significantly, and the ideal posterior is again optimal. Empirical results on both a toy problem and image classification support the theory.


------------------

Pros:
1. Given the prevalence of Bayesian deep learning and the issue of cold posterior, this paper offers a timely contribution that bridges theory and practice.
1. The paper is well written and motivated, the method appears sound (but see questions below), and concepts are explained in a clear and pedagogical manner.
2. The experiments are well thought out and offer clear empirical support of the proposed hypothesis.

------------------

Cons:
This might be due to my limited understanding of the paper, but I think there are still some limitations to the paper's proposed theory, e.g., it doesn't explain the observation that extremely cold posterior (λ -> 0) doesn't seem to hurt the performance of BNN (which should, according to the proposed theory, as there is only one optimal temperature λ = 1 / S, where S is the true number of underlying labelers), and more below.

------------------

Questions and Comments:
1. My biggest confusion is this: the paper argues that it's incorrect to assume a simple categorical likelihood p(y|x) as it doesn't take into account the data curation process; however, under the extended likelihood model as proposed, when conditioning on the event that y!=None and x!=None (as we do when training on standard datasets), and after marginalizing out the intermediate variables and renormalizing, isn't the conditional distribution p(y|x) still just a categorical distribution (except parameterized in a different way now)?  If so, then the difference between the two likelihoods is really just a different parameterization, and I'm no longer sure what to make of the suggested theory and the supporting results.  I find it very surprising that the more complex parameterization significantly reduced the tempering effect.  And if the we take the ground truth likelihood p(y|x) to be as in the standard (curated) dataset, which the paper argues is in some sense artificially "tempered", then why can't a well-spcified BNN just adapt to this (still categorical) likelihood and learn an optimal posterior under it?  I'm happy to raise my score if the authors can clarify these issues for me.
2. Since point estimation with SGD optimizes the same likelihood function, why don't we observe the tempering effect in SGD? Perhaps there is some effect but rather minimal; in any case, some experiments on SGD (with / without the corrected likelihood) would be interesting.
3. Related to my comment about "extremely cold posterior": in the GP experiment, when trained and tested on the corrected likelihood (considering curation), the test performance seems to really prefer the optimal λ = 1, whereas on the image experiment, more tempering (λ -> 0) doesn't seem to affect test performance. Is there an explanation for this?
4. Finally, does the proposed theory explain the observation that the cold posterior effect is more prominent in BNN with higher capacity (Wenzel et al. 2020)?


------------------

Possible typos and minor mistakes:
1. p.3, under eq (7), " This likelihood is equivalent to labeling each datapoint S times with the same label, and therefore has exactly the effect of setting λ = S in a tempered posterior". Should be "λ = 1/S" instead.
2. The right-most subfigure in Figure 4 should be labeled "E" instead of "F" to match the caption below.


------------------
Update:

I've raised my score in light of author response and new results. 
review_title: Sensible idea and very well executed, convincing results, but some questions remain.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work propose a theory suggesting that the cold posterior phenomena arises solely due the the curated nature of image benchmarks. A generative model is proposed where multiple annotators label datapoints, and only unanimously labeled datapoints are accepted into a dataset. This theory is studied under a toy-problem using VI and a relabelled version of the CIFAR-10 test set with SGLD. 

However, many questions remain unanswered and the proposed theory is not sufficiently studied.

- Q: The cold posterior problem was highlighted in the SGMCMC case, but this work's main toy problem only explores tempered posteriors as prevalent in VI. It would be beneficial if the work highlights why these results should extend to the cold posterior, or better yet, run experiments in this scenario.
- Q: 4.1 strongly suggests that there is a relationship between between \lambda and S in the toy problem. It should be an easy addition to study this connection for a range of values for S to explore if this holds.
- Q: The work claims that the consensus protocol for standard datasets is not available, but it would appear that this is a a simple manner of reaching out to the authors of the datasets.
- Q: The main experiment presented in figure 5 is missing some important ablations: what happens when the CIFAR-10 baseline is trained under the same conditions (learning rate) as CIFAR-10H?
- Q: Why is it acceptable to use the original CIFAR-10 training-set as the test-set for CIFAR-10H? This seems like a problematic shift in data distribution.
- Q: the theory of dataset curation is interesting but makes a broad claim. More datasets should be explored from varying modalities. The sole focus here on CIFAR-10 provides too little evidence. As a suggestion: curation processes are different for e.g. medical imaging datasets, and typically well documented. 
- Q: It's unclear to me why it is acceptable to increase the training set size by the number of annotators. An increase of factor 50 is effectively setting the temperature to ~ 1e-2. At this temperature the baseline performs just as well. Does this large gap still hold if just a much smaller subset of annotators is taken from the CIFAR-10H dataset?
 
The core idea proposed in this work is thought provoking and contributes to the discussion on this topic. The work is relatively short, which is not a problem in its own right, but the experiment section needs to provide more evidence and analysis. I vote for reject.

Nitpicks:
- F instead of E in figure 4.
- "As expected, we when"

Update: I've increased my score.
review_title: Interesting theory but lacking evidence.  

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose the idea that cold posteriors in Bayesian neural networks could be caused by the likelihood instead of the prior. They argue theoretically that the curation process of popular benchmark data sets would lead to a different weighting of the likelihood in the posterior. They show in some experiments that the cold posterior effect can be reduced when accounting for this.

Major comments:
- The paper title suggests that it is about cold posteriors, and it quite prominently references [1] in the introduction. However, in Sec. 2, it is then clarified that the paper is in fact not about cold posteriors, but about tempered ones. It is just briefly mentioned in passing that the results should transfer, but this is never tested. I think an experiment on actual cold posteriors, similar to the one in [1], would be warranted to support such a statement and the usage of the current paper title.
- The theory suggests that the optimal posterior performance should be achieved at lambda=S, that is, cooling down beyond that point should deteriorate performance again. This is an interesting prediction, since it does not seem to fit the observations in [1]. It would be nice to see this confirmed on an actual BNN experiment, similarly to what can be seen in the toy GP experiment.
- The related work section seems awfully short. It does not even mention [1] (although it is cited heavily elsewhere). Moreover, for a paper that is proposing a statistical theory of tempered posteriors, works such as [2] and [3] should probably be mentioned.

Minor comments:
- Sec. 4.2: "we when" -> "when"

Summary:
The idea that cold/tempered posterior effects can be caused by the data set curation instead of by misspecified priors is very interesting and definitely deserves a theoretical and empirical investigation. However, the investigation at hand seems a bit incomplete in some places, especially the related work and experiments sections. Also the title does not currently seem to fit the experiments. Given that the current manuscript is comfortably within the ICLR page limit, I'm hopeful that these points can be addressed in a revised version during the discussion phase. 

Update: I increased my score following the clarifications and addition of the BNN experiment during the discussion phase.

[1] Wenzel, F., Roth, K., Veeling, B. S., Świątkowski, J., Tran, L., Mandt, S., ... & Nowozin, S. (2020). How good is the bayes posterior in deep neural networks really?. arXiv preprint arXiv:2002.02405.
[2] Grünwald, P. (2012, October). The safe bayesian. In International Conference on Algorithmic Learning Theory (pp. 169-183). Springer, Berlin, Heidelberg.
[3] Grünwald, P., & Van Ommen, T. (2017). Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12(4), 1069-1103.
review_title: Interesting theory, but the experiments are not quite there yet

Meta-review: This is an interesting, controversial paper that contributes to an ongoing debate in Bayesian deep learning.

Bayesian inference with artificially “cooled” posteriors (e.g., trained with Langevin dynamics with down-weighted noise) was recently found to outperform over both point estimation and fully-Bayesian treatments (Wenzel et al., 2020). This paper proposes a new explanation for these observed phenomena in terms of a data curation mechanism that popular benchmark data sets such as CIFAR underwent. The analysis boils down to an evidence overcounting/undercounting argument and takes into account that curated data sets only contain data points for which all labelers agreed on a label. The authors claim that, when modeling the true generative process of the data, the cold posterior effect (partially) vanishes.

The paper is well-written and provides a consistent analysis by modeling the data curation mechanism in terms of an underlying probabilistic graphical model of the labeling mechanism. Unfortunately, several observed phenomena of (Wenzel et al., 2020) remain unexplained by the theoretical arguments, e.g., the fact that “very cold” (T --> 0) posteriors don’t hurt performance, or the observation that the optimal temperature seems to depend on the model capacity. While the proposed explanation doesn’t capture the full picture (upon which both authors and reviewers agree), the paper’s focus on the data curation process, supported extensive experiments, gives a partial explanation and provides an interesting perspective that will spur further discussion and should be of broad interest to the Bayesian deep learning community. 

Decision: Accept (Poster)