Cold Posteriors through PAC-Bayes
Authors: Konstantinos Pitas
Julyan Arbel
Venue: Submitted to ICLR 2023 (2023)
Keywords: cold posteriors
Bayesian
Bayesian Neural Networks
PAC-Bayes
Laplace approximation

Abstract: We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds. We argue that in the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance on out-of-sample data. Instead, out-of-sample error is better described through a generalization bound. In this context, we explore the connections of the ELBO objective from variational inference and the PAC-Bayes objectives. We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain a temperature parameter $\lambda$ which is not restricted to be $\lambda=1$. For both simplified regression and realistic classification tasks, in the case of Laplace approximations to the posterior,  we show how this PAC-Bayesian interpretation of the temperature parameter captures important aspects of the cold posterior effect.

Under review as a conference paper at ICLR 2023
COLD POSTERIORS THROUGH PAC-BAYES
Anonymous authors
Paper under double-blind review
ABSTRACT
We investigate the cold posterior effect through the lens of PAC-Bayes general- ization bounds. We argue that the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance error is better described through a generalization bound. In this context, we ex- plore the connections of the ELBO objective from variational inference and the PAC-Bayes objectives. We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain temperature parameter which is not restricted to be À = 1. For classification tasks, in the case of Laplace ap- proximations the posterior, we show how this PAC-Bayesian interpretation of the temperature parameter captures important aspects of the cold posterior effect.
out-of-sample data. Instead, out-of-sample
INTRODUCTION
Ab
'Test
LIFAR-3 CTPAR 3o
10
MAP
Prior
Beabai
(a) Classification tasks
(b) Laplace, different temperatures À
Figure 1: PAC-Bayes bounds correlate with the 0-1 Loss for different values of the temperature (quantities on both axes are normalized). (a) Classification tasks on CIFAR-10, CIFAR- 100, and SVHN datasets (o? 0.1, ResNet22) and FMNIST dataset (a 0.1, ConvNet). (b) Graphical representation of the Laplace approximation for different temperatures: for hot temperatures À & 1, the posterior variance becomes equal to the prior variance; for regularized according to the curvature h; for cold temperatures À > 1, the posterior becomes Dirac delta on the MAP estimate.
the posterior variance
In their influential paper, Wenzel et al. (2020) highlighted the observation that Bayesian neural networks typically exhibit better test time predictive performance if the posterior distribution is "sharpened" through tempering. Their work has been influential primary because it serves as well documented example of the potential drawbacks of the Bayesian approach deep learning. While other subfields of deep learning have seen rapid adoption, and have had impact on real world problems, Bayesian deep learning has, to date, seen relatively limited practical use (Izmailov al., 2021; Lotfi et al., 2022; Dusenberry et al., 2020; Wenzel et al., 2020). The "cold posterior effect"" as the authors of Wenzel al. (2020) named their observation, highlights an essential mismatch between Bayesian theory and practice. As the number of training samples increases, Bayesian theory tells states that the posterior distribution should be concentrating more and more on the true model parameters, in frequentist sense. At any time, the posterior our best guess at the true model parameters, without having to resort to heuristics. Since the original paper, a number of works (Noci et al., 2021; Zeno al., 2020: Adlam al., 2020; Nabarro al., 2022: Fortuin et al., 2021;

---

Under review as a conference paper at ICLR 2023
Aitchison, 2021) have attempted to explain the cold posterior effect, identify its origins, propose remedies and defend Bayesian deep learning in the process.
The experimental setups where the cold posterior effect arises have, however, been hard to pinpoint precisely. Noci et al. (2021) conducted detailed experiments testing various hypotheses. The cold posterior effect shown arise from augmenting the data during optimization (data augmenta- tion hypothesis), from selecting only the "easiest" data samples when constructing the dataset (data curation hypothesis), and from selecting a "bad"" prior (prior misspecification hypothesis). Nabarro et al. (2022) propose a principled log-likelihood that incorporates data augmentation, however they show that the cold-pon posterior persists. Bachmann al. (2022) also propose a mechanism by which data-aupe aatt--uugmentttion leads to mispecification and how the tempered posterior alleviates it. They prove their results for simplified settings, and acknowledge that there might be other potential sources of the cold-posterior effect. Data curation was first proposed as an explanation in Aitchison (2021), however the author shows that data curation can only explain a part of the cold posterior effect. Misspecified priors have also been explored as possible cause in several other works (Zeno al., 2020; Adlam et al., 2020; Fortuin et al., 2021). I). Again the results have been mixed. In smaller mod- els, data dependent priors seem to decrease the cold posterior effect while in larger models the effect increases (Fortuin et al., 2021).
We posit that discussions of the cold posterior effect should take into account that in the non- asymptotic setting (where the number of training data points is relatively small), Bayesian inference does not readily provide guarantee for performance on out-of-sample data. Existing theorems describe posterior contraction (Ghosal et al., 2000; Blackwell & Dubins, 1962), however in prac- tical settings, for finite number of training steps and for finite training data, is often difficult to precisely characterise how much the posterior concentrates. Furthermore, theorems on posterior contraction are somewhat unsatisfying in the supervised classification setting, in which the cold posterior effect is usually discussed. Ideally, one would want theoretical analysis that links the posterior distribution to the test error directly.
Here, we investigate PAC-Bayes generalization bounds (McAllester, 1999; Catoni, 2007: Alquier et al., 2016; Dziugaite & Roy, 2017) as the model that governs performance on out-of-sample data. PAC-Bayes bounds describe the performance on out-of-sample data, through an application of the convex duality relation between measurable functions and probability measures. The convex du- ality relationship naturally gives rise to the log-Laplace transform of special random variable (Catoni, 2007). Importantly the log-Laplace transform has a temperature parameter À which is not constrained be x = 1. We investigate the relationship of this temperature parameter cold posteriors.
In summary, contributions are the following:
Through detailed experiments for the Laplace approximation to the posterior, we show that PAC-Bayes bounds correlate with out-of-sample performance for different values of the temperature parameter X. This might indicate that the temperature in the cold-posterior lit- erature coincides with the temperature of the log-Laplace transform, and motivate Bayesian practitioners to use heuristics when targeting Frequentist metrics.
Contrary to Wenzel et al. (2020), we find that the coldest temperature (such that the pos- terior is a Dirac delta centered on MAP estimate of the weights) is empirically almost always optimal in terms of test accuracy. PAC-Bayes bounds track and predict this be- haviour. However, the negative log- likelihooo and the Expected Calibration Error (ECE) have more complex behaviour. Contrary to prior work (Wenzel et al., 2020; Noci et al., 2021), this highlights that the evaluation metric choice plays an important role when dis- cussing the cold-posterion effect. More importantly, show that to improve the test ECE or NLL one typically needs to reduce the test accuracy.
We derive PAC-Bayes bound for the case of the widely used generalized Gauss-Newtor Laplace approximations to the posterior. Contrary to prior work (Bachmann et al., 2022; Aitchison, 2021) our bound implies that À does not simply fix misspecified prior like- lihood. For a fixed target test risk, likelihood and prior, the required À varies due to the stochasticity of the inference procedure and the loss landscape shape.
We also include detailed FAQ section in the Appendix.
2

---

Under review as a conference paper at ICLR 2023
2 COLD POSTERIOR EFFECT: MISSPECIFIED AND NON-ASYMPTOTIC SETTING
We denote the learning sample (X = [(,,,,))); v:) E (A'x x 3)"" that contains n input-output pairs, Observations (X, Y) are assumed to be sampled randomly from a distribution D. Thus, we de- note (X, ) ~ D the i.dd observation of n elements. We consider loss functions E: F x X x x + R. where F is a set of predictors f X - V. We also denote the risk (f) E (( T.,
y) and the empirical risk ( = (1/n) Σ, L,, Vi). We consider two probability measures, the prior E M(F) and the approximate posterior ý E MMF). Here, MMF) denotes the set of all probability measures on F. We encounter cases where we make predictions using the approx- imate posterior predictive distribution E, E (lyl,, [)]. We will use two loss functions, the non- differentiable zero-one loss (or x. y) I(rgg max, f()) * y). and the negative log-likelihood, which is a commonly used differentiable surrogate (nl( xz, y) = log(p(ylæ,, f)). We assume that outputs of f form probability distribution p(y|e,, f) either through a Gaussian likelihood (in the case of regression) or using the softmax activation function (in the case of classification). Given the above, the Evidence Lower Bound (ELBO) has the following form
()() Es-pCiey An
(1)
where À 1. Note that our temperature parameter À is the inverse of the one typically used in cold posterior papers. In this form A has a clearer interpretation as the temperature of log-Laplace trans- form. Our setup is discussed in Wenzel et al. (2020), p3 Section 2.3, and used in Bachmann et al. (2022); Aitchison (2021). While Wenzel et al. (2020) use MCMC to conduct their experiments, we opt for the ELBO for analytical tractability. While Wenzel et al. (2020) temper by À both the like- lihood and the prior, as discussed in Aitchison (2021) and Wenzel et al. (2020) the relevant setting for the ELBO is the one of (Eq. 1). where only the KL tempered. One then typically models the posterior and prior distributions over weights using a parametric distribution (commonly a Gaus- sian) and optimizes the ELBO, using the reparametrization trick, to find the posterior distribution (Blundell et al., 2015; Khan et al., 2018). The cold posterior is the following observation:
Even though the ELBO has the form (1) with X practitioners have found that much larger values X >> typically result in better test time performance, for example a lower test misclassification rate and lower test negative log- log-likelihood
The starting point of our discussion will be thus to define the quantity that we care about in the context of Bayesian deep neural networks and cold posterior analyses. Concretely, in the setting of supervised prediction, what we often try to minimize is
PD (ylar) (r)
KT. (T)(p [>)): f))) E
E Ey-p(yyyle,
the conditional relative entropy (Cover, 1999) between the true conditional distribution PD (y)(²) and the posterior predictive distribution Ey-plp(y) æ. f)]. For example, this is implicitly the quantity that we minimize when optimizing classifiers using the cross-entropy loss (Masegosa, 2020; Morningstar et al., 2022). It is also on this and similar predictive metrics that the cold posterior appears. In the following we will outline the relationship between the ELBO, PAC-Bayes and (2).
2.1 ELBO
We assume a training sample (X, D" as before, denote p(w) X, Y) the true posterior prob- ability over predictors f parameterized by w (typically weights for neural networks), and π and ò respectively the prior and variational posterior distributions as before. The ELBO results from the following calculations
p(w) p(w|X, Y) dw = (((W) In p(w)p(Y|X) dw T( X, w)
(L(((()lp((XXx7)
((w) In
/ p(w)
In -Inp(Y IX, w) In p(W)
+ +Imp(((x)) dw
T T(w)
-Er (f) - kL
-
+Inp(Y1X).
ELBO
3

---

Under review as a conference paper at ICLR 2023
Thus, maximizing the ELBO can be seen as minimizing the KL divergence between the true poste- rior and the variational posterior over the weights ((l((())lp((/>>)) Y)). The true posterior distri- bution Y) gives more probability mass predictors which are more likely given the train- ing data, however these predictors do not necessarily minimize (P (y [+)] B [x. O)), the evaluation metric of choice (2) for supervised prediction. In the well-specified regime (where the true predictor f* is f E F) and when n - 30, the Blackwell-Dubins consistency theorem (Black- well & Dubins, 1962) implies that the posterior quickly concentrates on the true set of parameters. In such cases, more detailed analysis, such as a PAC-Bayesian one, is unnecessary as the posterior akin to a Dirac delta mass at the true parameters. However neural networks do not operate in this regime. The existence of multiple minima hints that neural networks are misspecified, and the number of samples is small relative to the number of parameters.
Operating in the regime where f É F and where n is (comparatively) small makes it important to derive a more precise certificate of generalization through generalization bound, which directly bounds the true risk. In the following we focus analyzing a PAC-Bayes bound in order to obtain insights into when the cold posterior effect occurs.
2.2 PAC-BAYES
We first look at the following bound denoted by BAlquerr It was considered by Alquier et al. (2016) (Theorem 1. see also Theorem I by Masegosa (2020) for a statement under the same conditions. Theorem 1 (Balquier: Alquier et al., 2016). Given distribution D over X V, hypothesis set F, a loss function f: F x X x y R. a prior distribution π over F, real numbers ō E [0, and X O, with probability least 1 8 over the choice (X,Y) ~ DD, we have for all p on F
1
1 <L(P|(T) + In 5 + Ve...(A..n)
E (ff S E ExpgExx (f) An
where V (e,,,D(A,n) In E Ex' exp [An -x,1))xxxxxx01))
There are three different terms in the above bound. The empirical risk term E, Es-a (f) is the empirical mean of the loss of the classifier over all training samples. The KL term 1/ KL( is the complexity of the model, which in this case is measured as the KL-divergence between the posterior and prior distributions. The Moment term L/(An)We,,,, (A, n²) is the log-Laplace trans- form of the difference between the risk and empirical risk for a reversal of the temperature. We will keep the name "Moment" in the following. Note that unless stated otherwise (for example in Section 4), we can make some assumption on the risk to ensure that the Moment term is bounded. Such common assumptions include that the risk sub-Gaussian under the prior и and the distribution D (see eg Germain et al., 2016) or more generally lub-Weibull (Vladimirova et al., 2019; 2020). Using PAC-Bayes bound together with Jensen's inequality, one can bound (2) directly as follows
sub-Gamme random variable
KT.(po x:) Bx-slp((llal..... f))) = EnyDD
In E B f)l
EnyxDl In (p x fill + Ex D pn
ExyxD E Ex In p(y x. f)ll + Eg g [n)l=))
Caa An
(l()) + In ((m,) + Ee..-llllpovlee))
PAC-Bayes
The last line holds under the conditions of Theorem I and in particular with probability least 1 - 5 over the choice (X, Y) ~ D Notice here the presence of the temperature parameter À 2 0, which need not be X= 1.
In particular it is easy to see that maximizing the ELBO is equivalent to min- imizing PAC-Bayes bound for X 1, which might not necessarily be opti- mal for a finite sample size. More specifically even for exact inference, where Ew (p(y) x w)ll w)]lo-po()x. = plyla, x2 X. Y), the Bayesian posterior predictive distribution does not necessarily minimize (L,(p(y) (z) (E |Eppolplya f)))
4

---

Under review as a conference paper at ICLR 2023
2.3 CLASSIFICATION TASKS
For classification tasks, we are typically mainly interested in achieving low expected zero-one risk Ex-iCD' (f). The ELBO objective is not directly related to this risk. However in the PAC-Bayesian literature there exist bounds specifically adapted it. In the following we will use one of the tightest and most commonly used bounds, the "Catoni" bound, denoted Battoni from Catoni (2007) Theorem 1.2.6.
Theorem 2 (Beatoni. Catoni, 2007). Given a distribution D over X x V. a hypothesis set F, the 0-1 loss function for : F x X' x > [0, 1), a prior distribution x over F, a real number ō E (0, 1]. and real number À > O, with probability at least 1 - over the choice (X,Y)
DD, we have
Vpon F: Ey (5) S Φ5 (BI (Br-phxrr()+ An
kT((PlTT) + In 1 8 ))
33)
where + (x) = 1-e
1-e-x-
Similarly to the Alquier bound, the empirical risk term is the empirical mean of the loss of the classifier over all training samples. The KL term is the complexity of the model, which in this case is measured as the KL-divergence between the posterior and prior distributions. The Moment term has been absorbed in this case in the function T (x: T-E-X
2.4 SAFE-BAYE AND OTHER RELEVANT WORK
After identifying two sources of misspecification in standard Bayesian inference, Grünwald & Lang- ford (2007) proposed a solution, through an approach which they named Safe-Bayes (Grünwald, 2012: Grünwald & Van Ommen, 2017). Safe-Bayes corresponds finding a temperature parameter A for generalized (tempered) posterior distribution with À possibly different than 1. The optimal value of À is found by taking a sequential view of Bayesian inference, and for a Cèsaro averaged posterior, which is an average of the posteriors at different optimization steps, and which doesn't 't coincide with the standard posterior. The analysis of Grünwald (2012); Grünwald & Van Ommen (2017) is also restricted to the case where À < 1. By contrast provide analytical expression of the bound on true risk, given X, and also numerically investigate the case of 1. Our analysis thus provides intuition regarding which parameters (for example the curvature) might result in cold posteriors. Catoni (2007) discusses the optimal value of the temperature À for PAC- PAC-Bayes bounds, for fixed priors and posteriors. By contrast we investigate the case where the posterior optimized for different X and which is the relevant one for the cold-posterior literature. Germain et al. (2016) find that minimizing a PAC-Bayesian generalization risk bound maximizes the Bayesian marginal likelihood. However they only investigate the case where À = 1.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
The ELBO (1) is minimized at the probability density pf(f) given by: p²(f)
T (ffe An (f) E AnE faul (r) (Catoni, 2007). We will use the Laplace approximation
to the posterior in our experiments. This is equivalent to approximating XnC'e (f) using a second
order Taylor expansion around a minimum Wa. such that An (fw) Re Xn + An(w
w ) 2 VV San (fw)lw=w,(l (w- wp). Assuming a Gaussian prior N(00,021 the Laplace approximation the posterior أ is again Gaussian
1 02
p N
AH +
where H is the network Hessian H VV Cfall ble to compute in practice for modern deep neural networks, such that many approaches employ the generalized Gauss-Newtor (GGN) approximation HGGN = =En I T T TAly (w:)),
(-xw ww This Hessian generally infeasi-
5

---

Under review as a conference paper at ICLR 2023
CIFAR-10
CIFAR-100
SVHN
FMNIST
1.0
Zuet Beanol
15
LG 0.75
7000
0.75
0.8
Caioni
0.50
-10
0.50
-0 0.50
5000
0.6
2500
0.25
0.25
0.25
0.4
0.00
10
10
L0P
10
10a
10
10
10
X
1.0
KBAC (0-1)
0.75
0.20
099
0.05
0.50
0.15
C.8
0,255
0.04
0.7
0.10
10
10
1:1
16
164
10
102
X
A
20
KFAC (NLL)
6-
1.0
25
0.8
0.5
0.4
10 X
10
100
10
10-3
10a
10-3
10
A
along with the mean Figure 2: Test 0-1 Loss (we denote À = 1 by - ---): BCatoni PAC-Bayes bound 0-1 loss generalization certificate (top). standard KFAC Laplace 0-1 loss (middle) and standard KFAC NLL (bottom). The Bcatoui PAC-Bayes bound closely tracks the test 0-1 loss. For the standard KFAC posteriors the test and validation 0-1 loss behave similar to the Catoni case, with a rapid improvement f followed by a plateau. Coldest posteriors X >>>>>> 1 are almost always best.
mean, as well as 10 MAP trials
where (2) is the network per-sample Jacobian [x) = wwfe wa), and A f) -V log p(y: f) is the per-input noise matrix (Kunstner et al., 2019). We will use two simpli- fied versions of the GGN
=
og where h Σi,, g(i, (V ( wp)r)² is the trace of the [2 approximation to the Hessian, with ( k) =
An isotropic approximation with variance (A) such that
* The Faatooiie- Approximate Curvature (KFAC) (Martens & Grosse, 2015) ap- proximation, which retains only block diagonal part of the GGN.
When making predictions, we use the posterior predictive distribution Ew-rlp(yla,, [w)] of the fall neural network model, meaning that samples from ` are inputted to the full neural network. Since the 0-1 loss is not differentiable, the posterior estimated with the cross entropy loss will be used for classification problems.
We have tested extensively in realistic classification tasks. We used the CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011) and FashionMnist (Xiao et al., 2017) datasets. In all experiments, we split the dataset into three sets. These three are the typical pre- diction tasks sets: training set Zerain; testing set Zusst, and validation set Zvalidation- We use Monte Carlo sampling to estimate the Empirical Risk term ( ~ (). For the isotropic Laplace approximation, and Gaussian isotropic prior, the KL divergence has a simple analytical expres- sion (T.( = (1
IW wx - d d' + dln dlno2) PAC-Bayes bounds require correct control of the prior mean as the l, distance between prior and posterior means in the KI. term is often the dominant term in the bound. To control this distance, follow variation of

---

Under review as a conference paper at ICLR 2023
CI
CIFAR-100
SVHN
FMNIST
0.300
0-1 Losss
0.275
0:90
0.0/10
0. 0.095
0.280
0.85
0.038
0.090
0.225
0.08
0:09
0.4
0.5
0.01
0.02
D.U3G
0.038
BCE
ECE
ECE
BCE
Figure 3: The Pareto front of the test ECE compared to test 0-1 loss. Apart from the case of FMNIST we see a clear trade-off between 0-1 loss and ECE. As such there seems to be no À that simultaneously doesn't hurt the test 0-1 loss while improving the ECE, which is the most pertinent task, in light of the different behaviours of the different metries with respect to A.
the approach in Dziugaite et al. (2021) to constructing our classifiers. We first use Zuaan to find a prior mean Wa. We then set the posterior mean equal to the prior mean Wa w, but evaluate the r.h.s of the bounds on 7validation- Note that in this way || [wa w 11 = 0, while the bound is still valid since the prior is independent from the evaluation set X, Y Zvalidation- For the CIFAR-IO, CIFAR-100, and SVHN datasets, we use a WideResNet22 (Zagoruyko & Komodakis, 2016), with Fixup initialization (Zhang et al., 2019). For the FashionMnist dataset, we use a convolutional ar- chitecture with three convolutional layers, followed by two fully connected non-lines layers. More details on the experimental setup can be found in the Appendix.
3.2 CLASSIFICATION EXPERIMENTS
We find ten MAP estimates for the neural network weights of the CIFAR-10, CIFAR- SVHN and FMNIST datasets by training on Zirain using SGD. We then fit an Isotropic Laplace approximation each MAP estimate using X, Y Z.adidation For different values of X we then estimate the Catoni bound (Theorem 2) using Zvalidation- We also estimate the test 0-1 Loss, negative log- likelihood (NLL) and the Expected Calibration Error (ECE) (Nacini al., 2015) of the posterior predictive on Zuest- We use the prior variance of 0.1, as optimizing the marginal likelihood leads to o 0 which not relevant for BNNs. We also test a standard KFAC Laplace setup. Specifically we fit the KFAC Laplace on Zerain and also choose the prior through the marginal likelihood. In this case, we estimate a standard validation set bound using the validation set Zvalidation (instead of PAC-Bayes bound) as from the literature we know that any PAC-Bayes bound will be vacuous (larger than 1) as we do not control ||w,- wx 1 We plot the results for all datasets in Figure 2. The Catoni bound correlates tightly with test 0-1 Loss for all datasets and plot this correlation in Figure (a). Contrary to Wenzel et al. (2020) in terms of test 0-1 Loss, the MAP estimate (obtained where À >> and the posterior is "coldest") is almost always optimal. This behaviour is replicated in the "KFAC" case. This result is more coherent than the one in Wenzel et al. (2020) (Figure I) where the coldest temperature 0-1 test loss and the MAP and the 0-1 test loss don't 't match. highlights that in a tightly controlled setting, often Bayesian approaches don't 't improve at all over deterministic ones in terms of test 0-1 loss,for for any temperature X.
We plot in Figure 2 (bottom row) the NLL for the KFAC case. Even without data augmentation and even when we optimize the prior variance using the marginal likelihood, we find that all three cases of temperatures (cold posterior, warm posterior, as well as posterior with X = 1) can be optimal, for varying datasets. This highlights the importance of the choice of the evaluation metric when discussing the cold posterior effect, as results can vary significantly depending our choice. Specifically one can see in Kapoor et al. (2022) p18 and Aitchison (2021) p9 that different metrics have different behaviours and/or minima with respect to X.
We then ask the more pertinent question: "If the Laplace approximation doesn't 't in general improve the test 0-1 loss, can it (for some value of X) retain the same test 0-1 loss while improving calibra- tion?". In Figure we see that in our experiments in most cases we couldn't 't find such a temperature X. Apart from FMNIST there seems to be a clear tradeoff between test 0-1 loss and calibration error.
7

---

Under review as a conference paper at ICLR 2023
10.0
CIEAR-10
10.0
10.0
10.0
W pi W
7.5
(X,Y
7.5
7.3
7.5
doelty
5.0
5.0
5.0
5.0
2.5
2.5
2.5
2.5
0.0
0.0
0.0
0.0
0.25
0.00
0.25
0.25
0.00
U.25
0.25
0.25
-0.25
0.00
0.25
0.. Loss (oontzre))
0.1 Loss (csstared)
0-1 Los (oncered)
6-1 Los (oanordd)
Figure 4: Intragroup variability of the test 0-1 loss for all Laplace approximations W ~ P(X,Y) Y), well as the test 0-1 loss variability for each Laplace approximation due to the MC approximation to the Laplace predictive w ~ p(w). We display the combined variability for all values of À [10-?, 10*). The intragroup test 0-1 variability cannot be explained by the vari- ability of each Laplace predictive due to the MC approximation. This large intragroup variability contradicts the assumption that À simply adjusts for the aleatoric uncertainty of the data. Intuitively, different MAP estimates result in different predictive functions, which yield different test 0-1 losses for fixed prior, X, and likelihood. Alternatively for fixed prior, likelihood, and target test 0-1 loss the required value of À depends on the MAP estimate. As MAP estimates are usually found using a stochastic algorithm (XX) finding a that consistently achieves the target test 0-1 loss might be difficult.
4 THE TEMPERATURE A, ALEATORIC UNCERTAINTY AND CURVATURE
In light of our empirical results, it would be interesting to derive an analytical form that clucidates the important variables that affect the bound. However, PAC-Bayes objectives are difficult to analyze theoretically for the non-convex case. Thus in the following we make number of simplifying assumptions. The Laplace approximation with the Generalized Gauss-Newton approximation the Hessian corresponds to linearization of the neural network around the MAP estimate wp E Rd (Immer et al., 2021) (2;; w) f wp) + Vw (w wp).
When analyzing minima of the loss landscape linearization is reasonable even without assuming infinite width Zancato et al. (2020); Maddox et al. (2021). For appropriate modelling choices, we aim at deriving bound for this linearized model. We adopt the aforementioned linear form to- gether with the Gaussian likelihood, yielding fun(ww Lan(w,,, x; y) In((2TT) + 2 (U f(z; wp)
fw; wp) T (w W We also make the following modeling choices: Prior over weights: w w~ ~NWW,, v: I).. 2) Gradients as Gaussian mixture: Vwf(<z; wp) ~ E 0,N(H, 22 3) Labeling function:
wp) + e, where e ~ NO0 o?). We also f(: p)) + / w;) (w. ww..
assume that have deterministic estimate of the posterior weights W, which keep fixed, and model the posterior as N(w,, of (A)I), similarly our experimental section. Therefore estimating the posterior corresponds to estimating the variance (A))).
Proposition 1 (Bapproximatte) With the above modeling choices, and given a distribution D over X V, real numbers (0, 1] and X E (0, 2) with e 2no with probability at least 1 over the choice (X, ~ D", we have
Ew-a Chr
Chall (w)
yy wX)
A h
h
1 In
0 (o +
W 113)
(2
0²
o²
2n o²
do²2
2n
0²
2
2
no oz
Empiiriaal Risk
Monent
d
1
+ +ln 1
Ww
w
1
d
dln
+ d
2
2
h
+ o
o²2
77
2
KL
8

---

Under review as a conference paper at ICLR 2023
is the curvature parameter, o²2 = Ei= H5 is the posterior gradient variance, and 2 is the variance of the likelihood function.
where h = (WW,,
We now make number of observations regarding Proposition 1. Here, h is the trace of the Hes- sian under the Gauss-Newton approximation (without a scaling factor n)). Under the PAC-Bayesian modeling of the risk, cold posteriors are the result of complex interaction between various param- eters resulting from 1) our model such as the prior variance of and prior mean Wa. and 2) our data o and w. (the curvature of the minimum and the MAP estimate wa depend on the deep neural network architecture, the optimization procedure and the data), Contrary to prior work (Bachmann et al., 2022; Nabarro et al., 2022; Aitchison, 2021) our bound suggests that À cannot be seen as simply fixing mispecified likelihood variance o² or prior variance o². In particular it does not simply rescale the aforementioned quantities. Furthermore, our bound implies that even for a fixed prior and likelihood the same À can imply different test risk based on the properties of each MAP estimate such as the curvature and the distance from initialization [wa -
We can observe this in our empirical data. We fit a Laplace approximation with similar proce- dure as the KFAC case of Figure/ 2 but using the Isotropic approximation to the posterior, for the CIFAR-10. 10, CIFAR- 100, SVHN and FMNIST datasets. We keep the prior variance fixed o 0.1. For each value of X E [10-1, 104] we compute the test Q-1 loss of all the Laplace approximations at the different MAP estimates E)~A(s) S Year ( E [I....,M M] where M is the number of
MAP estimates, and then the mean of these test 0-1 losses (1/M) Σ Ex Yame, Yines (f). We then compute the residuals of the test 0-1 loss of each Laplace approximation with respect to the aforementioned mean. This gives approximation deviates from the mean test 0-1 loss of all Laplace approximations. We plot the in- ferred distribution of residuals for all À combined in Figure 4. For the Laplace approximation at each MAP we are using 100 Monte Carlo samples to approximate the predictive. As such, it is im- portant plot sanity check the variability we would expect simply from the MC approximation of each predictive for each E))p() S Yiuse (f). Specifically we model this as a Gaussian dis- tribution with a 0.005 such that IP( (ES-aEX Xaess (f) (1/Naac EES Ólo: 0.01) 2 0.95 where fi X (though we stress that this modelling comes from past experience and we didn't 't have time to precisely estimate this error). In Figure 4 we see significant deviations from (1/M) ΣM _ Your (f) which can't can't be explained from the MC approximation of each predictive. Most of the variability comes from moderate values of À as for large values the Laplace approximations degenerate to Dirac masses on the MAP estimates, and all our MAP estimates have approximately the same test 0-1 loss. Alternatively for small values of À all Laplace approximations have 90% test 0-1 loss.
measure of how much the test 0-1 loss of each Laplace
Xxcen (ft) S
Assume that different MAP estimates W generated from some randomized algorithm Wp (XX, () where Y are the training data. Our results imply that even for the same dataset, a fixed prior and likelihood, it might be difficult find a single value of X that gives target test loss, simply because of the stochasticity w (X.Y)
5 DISCUSSION
We argued that Bayesian inference does not readily provide high probability guarantees on out- of-sample performance, leading to inconsistencies such as the cold-posterior effect. We hope that this will motivate Bayesian practitioners to use heuristics "guilt-free" when targeting Frequentist performance metrics, or target contraction to the true posterior. Furthermore, our empirical results on the ECE point towards the need for the use of Pareto curves when evaluating Bayesian approaches. Finally, it would be interesting to see how our results from Section 4 translate the MCMC setting.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ben Adlam, Jasper Snoek, and Samuel L Smith. Cold posteriors and aleatoric uncertainty. ICML. Workshop on Uncertainty and Robustness Deep Learning. 2020.
Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. In International Conference on Learning Representations, 2021.
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. The Journal of Machine Learning Research, (7(1):8374-4114 2016.
Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. How Tempering Fixes Data Augmentation in Bayesian Neural Networks. International Conference on Machine Learning. 2022.
David Blackwell and Lester Dubins. Merging of opinions with increasing information. The Annals of Mathematical Statistics, 33(3):882-888 1962.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference Machine Learning, pp. 1613-1622. PMLR, 2015.
Olivier Catoni. PAC-Bayesian supervised classification: the thermodynamics of statistical learning, volume 56 of Monograph Series. Institute of Mathematical Statistics Lecture Notes, 2007.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In International Conference of Machine Learning, pp. 2782-2792. PMLR, 2020.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Uncertuinty in Artificial Intelligence, 2017.
Gintare Karolina Dziugaite, Kyle Hsu, Waseem Gharbieh, Gabriel Arpino, and Daniel Roy. On the role of data in PAC-Bayes. International Conference on Artificial Intelligence and Statistics, pp. 604-612, PMLR, 2021.
Vincent Fortuin, Adrià Garriga-Alonso, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. In Interna- tional Conference on Learning Representations, 2021.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-Bayesian theory meets Bayesian inference. Advances in Neural Information Processing Systems, 29, 2016.
Subhashis Ghosal, Jayanta K Ghosh, and Aad W Van Der Vaart. Convergence rates of posterior distributions. Annals of Statistics, pp. 500-531, 2000.
Peter Grünwald. The safe Bayesian. In International Conference on Algorithmic Learning Theory, pp. 169-183. Springer, 2012.
Peter Grünwald and John Langford. Suboptimal behavior of Bayes and MDL in classification under misspecification. Machine Learning, 66(2): 19-149, 2007.
Peter Grünwald and Thijs Van Ommen. Inconsistency of Bayesian inference for misspecified linear models, and proposal for repairing it. Bayesian Analysis, 12(4): 1069-1103, 2017.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions Bayesian neural nets via local linearization. In International Conference on Artificial Intelligence and Statistics, pp. 703-711, PMLR, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are Bayesian neural network posteriors really like? In International Conference on Machine Learning, pp. 4629 PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Sanyam Kapoor, Wesley J Maddox, Pavel Izmailov, and Andrew Gordon Wilson. On uncertainty, tempering, and data augmentation in bayesian classification. arXiv preprint arXiv:2203. 16481, 2022.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weight-perturbation in ADAM. In International Conference on Machine Learning, pp. 2611-2620. PMLR, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Cite- seer, 2009.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Sanae Lotfi, Pavel Izmailov, Gregory Benton, Micah Goldblum, and Andrew Gordon Wilson. Bayesian model selection, the marginal likelihood, and generalization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14223-14247. PMLR, 17-23 Jul 2022.
Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. International Conference on Artificial Intelligence and Statistics, pp. 2737-2745. PMLR, 2021.
James Martens and Roger Grosse. Optimizing neural networks with fronecker-factored approxi- mate curvature. In International conference on machine learning, pp. 2408-2411 PMLR, 2015.
Andres Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. Advances in Neural Information Processing Systems, 33:5479-5491, 2020.
David A McAllester. Some PAC-Bayesian theorems. Machine Learning, 37()):35--33,, 1999.
Warren R Morningstar, Alex Alemi, and Joshua Dillon. PAC PACT-BBays: Narrowing the empirical risk gap in the misspecified Bayesian regime. In International Conference Artificial Intelli- gence and Statistics, pp. 8270-8298. PMLR, 2022.
Seth Nabarro, Stoil Ganev, Adrià Garriga-Alons Vincent Fortuin, Mark van der Wilk, and Lau- rence Aitchison. Data augmentation in Bayesian neural networks and the cold posterior effect. In Uncertainty in Artificial Intelligene, pp. 1434-1444, PMLR, 2022.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated prob- abilities using Bayesian binning. Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. arxiv, 2011.
Lorenzo Noci, Kevin Roth, Gregor Bachmann, Sebastian Nowozin, and Thomas Hofmann. Dis- entangling the roles of curation, data-augmennation and the prior in the cold posterior effect. Advances in Neural Information Processing Systems, 34, 2021.
Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding Priors in Bayesian Neural Networks the Unit Level. ICML, 2019.
Mariia Vladimirova, Stéphane Girard, Hien D Nguyen, and Julyan Arbel. Sub-Weii Sub-Weibull distributions: generalizing sub-Gaussian and sub-Exponential properties to heavier-tailed distributions. Stat, 2020. doi: 10.1002/sta4.38.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snock. Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? International Conference on Machine Learning, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms. arxiv, 2017.

---

Under review as a conference paper at ICLR 2023
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Pre- dicting training time without training. Advances in Neural Information Processing Systems, 33: 6136-6146, 2020.
Chen Zeno, Itay Golan, Ari Pakman, and Daniel Soudry. Why cold posteriors? on the suboptimal generalization of optimal Bayes estimates. Third Symposiun on Advances Approximate Bayesian Inference, 2020.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization, In International Conference on Learning Representations, 2019.
12


Reviews:
[{"review_id": "MGUJq3_7S-u", "reviewer": "Reviewer_oDKT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the application of PAC-Bayes theory to study the \"cold posterior effect\" previously observed in Bayesian neural networks. They rely on the aesthetic relationship between cold posteriors and certain PAC-Bayes bounds on the negative log likelihood to carry-out their analysis. They consider the negative log-likelihood loss for regression and the $0-1$ loss for classification. The authors rely on data-dependent priors selected with a validation set, then apply a Laplace approximation to determine the posterior (in the PAC-Bayes sense). From the experiments, the authors conclude that cold posteriors often lead to better generalization bounds, and suggest this as a mechanism for explaining the cold posterior effect.", "summary_of_the_review": "I think this paper is slightly below the acceptance threshold. The writing is good, experiments are thorough and well-thought out. However, I think some details of the experiments (e.g. Monte Carlo estimation of the moment term) lead to technical issues; and it is somewhat problematic that the generalization bounds reported may not be reflective of the actual values due to sampling error. More importantly, I think it isn't totally clear what the take-away of the paper is, in terms of insight that sheds more light on deep learning/Bayesian deep learning. The writing should make this evident. The discussion section points to more things that could be done, but really I think this space would be better spent clarifying how what was done in this paper relates to the bigger picture. It is quite possible I have missed something regarding this, and I hope the authors can clarify this in their response.", "strength_and_weaknesses": "## Strengths\n\n- The writing is clear; figures are high quality.\n- Experiments seem reasonably thorough.\n\n## Weaknesses\n\n- I found it hard to pin down exactly what insight about (Bayesian) deep learning is gained. The authors show that lower temperatures correspond correlate with better generalization and tighter generalization bounds. I would like the authors to clarify what this correlation teaches us about deep learning above and beyond what is known. The claim that a tighter generalization bound correlates with better generalization does not seem inherently interesting, and I think needs to be tied more clearly to concrete/actionable insight.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FOfT4uT9VU5", "reviewer": "Reviewer_Pea3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors do a PAC-Bayesian analysis for the cold-posterior effect, i.e. an observation that sharp posterior typically lead to better predictive performance. Main contribution of the paper is obtaining a PAC-Bayesian bound for Laplace approximation to the posterior. Since Laplace approximation relies on the Hessian, authors study versions of generalized Gaussian—Newton approximations to the posterior. Under these approximations, authors experimentally show that “colder” posterior yields better smaller PAC-Bayesian bound. Authors also derive PAC-Bayesian bound for stochastic linearlized network. ", "summary_of_the_review": "I recommend a weak reject for the paper. Main reasons for my recommendation is that the problem being studied is not very well motivated.  And the fact the technical developments are only incremental progress.", "strength_and_weaknesses": "## Strengths\n\nThe paper is generally well organized.\n\nPaper tries to study Bayesian phenomenon of cold-posterior from a frequentist point of view of PAC-Bayesian bounds. deriving performance guarantees of an approximation of the Bayesian posterior. \n\n## Weaknesses\n\n- In my opinion, the problem of studying the cold-posterior on itself is not well motivated as Bayesian analysis is more useful for uncertainty quantification rather than pure prediction. Of course, one can improve prediction quality if the posteriors are sharper.\n- For the linearized network, authors merely suggest a connection to the cold-posterior without attempting to tease apart details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D8HAW4AO1R", "reviewer": "Reviewer_qVvM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies what happens in Bayesian deep learning if the likelihood is scaled down by a factor of $1/\\lambda$. This is related to the \"cold posterior effect\". The authors empirically show that two different PAC-Bayesian bounds, which also include a scaling of $1/\\lambda$, correlate well with this tempered posterior's performance. Finally, the authors derive a PAC-Bayesian bound for a simplified model neural network and show that it has a complex dependence on the temperature $\\lambda$.", "summary_of_the_review": "Overall, I think the paper needs more connection with the literature, and also a deeper explanation of the presented results. As-is, it's not clear to me what the new high-level takeaways are from this paper.", "strength_and_weaknesses": "I have a few issues with the paper as-is; I've broken these down into categories below.\n\n**Discussion of previous work.** I think the discussion of previous work is flawed in a few places.\n1. The problem this paper studies is mathematically equivalent to tempering the likelihood in Bayesian inference. Maybe the literature has advanced and I've missed something, but I don't think this is the setting people talk about when they discuss the \"cold posterior effect,\" which is what the current paper says it is about. For example, the authors write \"As discussed in ... Wenzel et al. (2020) the relevant setting for the ELBO is the one we consider...\" I don't think Wenzel et al. (2020) says this. That paper notes that people have considered tempering just the likelihood in the context in variational inference, but they differentiate this likelihood-tempered setting from the cold posterior setting they consider in Appendix E.\n1. Citations of theorems from other papers (Theorems 1 and 2) should explicitly cite the theorem from that work. This is especially important for something like Catoni, 2007, which is 175 pages long; as a reader, there's no practical way for me to find the relevant theorem from that book. Also, have all the conditions for each theorem been recalled here? Theorem 1 looks like Theorem 4.1 from Alquier et al. 2016, which has an assumption on it that isn't recorded in the current paper.\n5. The paper is written as if it is about Bayesian inference. But in the experiments, the authors note \"We first use $Z_{train}$ [the training data] to find a prior mean $w_\\pi$.\" This means the experiments are about an *empirical Bayesian* method; Empirical Bayes is not the same thing as standard Bayesian inference.\n2. \"...the Blackwell-Dubins consistency theroem...\" I think Doob's theorem is a much more common reference for the contraction of Bayesian posteriors.  But it's important to note that posterior asymptotics are going to be different when using regular Bayesian inference than when using an Empirical Bayesian procedure (which is what this paper seems to be about), and results for one method may not be valid for the other.\n3. \"Germain et al. (2016) were the first to find connections between PAC-Bayes and Bayesian inference.\" I don't think this is quite true -- e.g. Grunwald (2012) cited here discusses PAC-Bayes. Is there some more specific context in which this is true?\n\n**Theoretical development (Section 4).** I'm not sure what the takeaways from the authors' theory is, given some issues with the assumptions going into it and the discussion of their results.\n1. Why is taking $\\sigma = 1$ a reasonable model? Is there anything lost by making this assumption?\n2. It is assumed that the gradients of a neural network come from a Gaussian mixture, and the fact that previous work has shown neural network gradients to be clusterable (Zancato et al. 2020) is cited as justification. I don't see why the results of Zancato et al. (2020) imply that the gradients should come from a normal distribution -- data can be clusterable and not come from a mixture of Gaussians. For example, a mixture of uniform distributions will be very different from a mixture of Gaussians.\n3. The proved PAC-Bayesian bound requires the temperature $\\lambda$ to be $O(1/n)$, where $n$ is the number of training datapoints. But \"cold posterior\" effects will show up for large $\\lambda$. So what interesting behavior can we extract from this bound? E.g. for $\\lambda \\in (0, 0.0001)$ (which this bound is restricted to for even moderate $n$), all of the plots in the paper show the same behavior.\n4. There isn't much discussion of the proved bound except that it displays \"complex interactions\" between the model and data. Some more discussion seems needed here. Can we learn anything from these complex interactions? What do the interactions tell us about the cold posterior effect? I think it's clear that the assumptions going into the bound are unrealistic. But if the bound for this toy model matches empirical results in interesting ways that previous bounds do not, this would still be a great contribution to the literature.\n\n**Empirical results.** I wasn't entirely clear on what the takeaway was supposed to be from the experiments. These seem to be evaluating existing PAC-Bayesian bounds; overall, what does each experiment tell us about understanding the cold posterior effect? It seems to me that most of the experiments just validate the cold posterior effect by showing larger $\\lambda$ gives better performance. Is there something that I'm missing here? Also a few more specific points:\n1. \"We might think these results are due to the poor (Isotropic) approximation to the posterior, however as we will see in the next section this behaviour carries over to other approximations to the posterior.\" This next section changes both the model (regression -> classification), the bound used (Theorem 1 -> Theorem 2), and the posterior approximation. Too much is different here to validate that changing the approximation isn't changing things; how do we know that the effect of changing the approximation isn't being canceled out by one of the other changes?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8eA67H2F4K", "reviewer": "Reviewer_jFkt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "        This paper analyzes the cold posterior effect in Bayesian neural networks. This effect is simply that these models perform better at test time if the posterior is sharpened artificially. The authors argue that there are several works form the literature analyzing the causes of this, with mixed results. In this paper the authors use generalization PAC-Bayes bounds to study the cold posterior effect. They show that these bounds correlate well with the generalization error in terms of the parameter lambda, controlling the amount of sharpness. Then, they analyze the influence of these bounds on such parameter, in the case of linearized models. The results show that the bounds incorporates complex interactions, which may explain the mixed results observed in the literature.\n", "summary_of_the_review": "  I believe that this is a nice paper with nice theoretical results and good empirical observations. However, I am a bit disappointed in the sense that it does not provide any improvement over the current methods that are provided. This questions the practical utility of the theoretical results obtained and the observations made.\n", "strength_and_weaknesses": "Strength:\n\n        - Nice theoretical results and experimental evaluation.\n\nWeaknesses:\n\n        - The results obtained may explain the wide variety of explanations observed in the literature for the cold posterior effect, but the paper does not provide insights about to address this problem or improve current models. This is a bit disappointing in this regard.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject