The Cold Posterior Effect Indicates Underfitting, and Cold Posteriors Represent a Fully Bayesian Method to Mitigate It
Authors: Yijie Zhang
Yi-Shan Wu
Luis A. Ortega
Andres R Masegosa
Venue: Accepted by TMLR ()

Abstract: The cold posterior effect (CPE) (Wenzel et al., 2020) in Bayesian deep learning shows that, for posteriors with a temperature $T<1$, the resulting posterior predictive could have better performance than the Bayesian posterior ($T=1$). As the Bayesian posterior is known to be optimal under perfect model specification, many recent works have studied the presence of CPE as a model misspecification problem, arising from the prior and/or from the likelihood. In this work, we provide a more nuanced understanding of CPE as we show that misspecification leads to CPE only when the resulting Bayesian posterior underfits. In fact, we theoretically show that if there is no underfitting, there is no CPE. Furthermore, we show that these tempered posteriors with $T < 1$ are indeed proper Bayesian posteriors with a different combination of likelihoods and priors parameterized by $T$. This observation validates the adjustment of the temperature hyperparameter $T$ as a straightforward approach to mitigate underfitting in the Bayesian posterior. In essence, we show that by fine-tuning the temperature $T$ we implicitly utilize alternative Bayesian posteriors, albeit with less misspecified likelihood and prior distributions. The code for replicating the experiments can be found at https://github.com/pyijiezhang/cpe-underfit.

Published in Transactions on Machine Learning Research (08/2024)
The Cold Posterior Effect Indicates Underfitting, and Cold Posteriors Represent a Fully Bayesian Method to Mitigate It
Yijie Zhang University of Copenhagen Novo Nordisk A/S
yiah@di ku, dk
Yi-Shan Wu University of Southern Denmark
ysun&ömada.sdu.c dk
Luis A. Ortega
tuis. ciiisortega@uamm
Autonomous University of Madrid
Andrés R. Mascgosa
arma awu.
Aalborg University
Reviewed on OpenReview: https: // opwneevie.. net/ net/ forum? id- GZORXG=HHT
Abstract
The cold posterior effect (CPE) (Wenzel et al., 2020) in Bayesian deep learning shows that, for posteriors with a lemperature T < 1, the resulting posterior predictive could have better performance than the Bayesian posterior (T 1). As the Bayesian posterior is known to be optimal under perfect model specification, many recent works have studied the presence of CPE as a model misspecification problem, arising from the prior and/or from the likelihood. In this work, we provide a more nuanced understanding of CPE as we show that misspeciffcattion leads to CPE only when the resulting Bayesian posterior underfits. In fact, we thcorctically show that if there no underfitting, there no CPE. Furthermore, show that these lempered posteriors with T with different combination of likelihoods and priors parameterized by T. This observation validates the adjustment of the temperature hyperparameter T as straightforward approach to mitigate underfitting in the Bayesian posterior. In essence, we show that by fine-tuning the temperature we implicitly utilize alternative Bayesian posteriors, albeit with less misspecified likelihood and prior distributions. The code for replicating the experiments can be found at /ttps://githin com/pyij wwwezann/epppppeeeeeeeeeee cp-/epderfit.....
indeed proper Bayesian posteriors
1 Introduction
In Bayesian deep learning, the cold posterior effect (CPF) (Wenzel al., 2020) refers to the phenomenon which we artificially "temper" the postcrior by either D) (p()))) or p(0 x p(D)0) with a lemperature T < the resulting posterior enjoys better predictive performance than the standard Bayesian posterior (with T 1). The discovery of the CPE has sparked debates in the community about its potential contributing factors.
If the prior and likelihood are properly specified, the Bayesian solution (i.e., T = 1) should be optimal (Gelman et al., 2013), assuming approximate inference is properly working. Hence, the presence the CPE implics either the prior (Wenzel al., 2020; Fortuin et al., 2022), the likelihood (Aitchison, 2021; Kapoor et al., 2022), or both are misspecified. This has been, so far, the main argument of many works trying explain the CPE.

---

Published in Transactions on Machine Learning Research (08/2024)
One line of rescarch examines the impact of the prior misspecification the CPE (Wenzel et al., 2020; Fortuin et al., 2022). The priors of modern Bayesian neural networks are often selected for tractability. Consequently, the quality of the selected priors in relation to the CPE is a natural concern. Previous research has revealed that adjusting priors can help alleviate the CPE in certain cases (Adlam et al., 2020; Zeno et al., 2021), there are instances where the ellect persists despite such adjustments (Fortuin et al., 2022). Some studics even show that the role of priors may not be critical (Izmailov et al., 2021). Therefore, the impact of priors on the CPE remains an open question.
Furthermore, the influence of likelihood misspecification on CPE has also been investigated (Aitchison, 2021; Noci et al., 2021; Kapoor et al., 2022; Fortuin et al., 2022), and has been identified to be particularly relevant in curated datasets (Aitchison, 2021; Kapoor et al., 2022). Several studies have proposed alternative likelihood functions to address this issue and successfully mitigate the CPE (Nabarro et al., 2022; Kapoor et al., 2022). However, the underlying relation between the likelihood and CPE remains a partially unresolved question. Notably, the CPE usually emerges when data augmentation (DA) techniques are used (Wenzel et al., 2020; Tamailov et al., 2021; Fortuin el. al., 2022; Noci et al., 2021; Nabarro et al., 2022; Kapoor el. al., 2022). A popular hypothesis is that using DA implies the introduction of a randomly perturbed lig-liielhhood, which lacks a clear interpretation as a valid likelihood function (Wenzel ct al., 2020; Izmailov et al., 2021). However, Nabarro et al. (2022) demonstrates that the CPE persists even when a proper likelihood function incorporating DA is defined. Therefore, further investigation is needed to fully understand their relationship.
Other works argued that CPE could mainly be an artifact of inaccurate approximate inference methods, especially in the context of neural networks, where the posteriors are extremely high dimensional and complex (Izmailov et al., 2021). However, many of the previously mentioned works have also found setups where the CPE either disappears or is significantly alleviated through the adoption of better priors and/or better likelihoods with approximate inference methods. In these studies, the same approximate inference methods were used to illustrate, for example, how using a standard likelihood function leads to the observation of CPE and how using an alternative likelihood function removes it (Aitchison, 2021; Noci et al., 2021; Kapoor ct al., 2022). In other instances, under the same approximate inference schemi,, CPE is observed when using certain types of priors but it is strongly alleviated when an alternative class of priors is utilized (Wenzel et al., 2020; Fortuin al., 2022). Therefore, there is compelling evidence suggesting that approximate methods are not, at least, a necessary condition for the CPE.
This study, both theoretically and empirically, demonstrates that the presence of the cold posterior effect (CPE) implies the existence of underfitting; in other words, if there is no underfitting, there is no CPE. Integrating this perspective with previous findings suggesting that CPE indicates misspecified likelihood, prior, or both (Gelman et al., 2013), we conclude that CPE implies both misspecification and underfitting. Consequently, mitigating CPE necessitates addressing both aspects. Notably, simplifying the issue by solely focusing on misspecification is insufficient, as misspecification can lead Bayesian methods to both underfitting and overfitting (Domingos, 2000; Immer et al., 2021; Kapoor et al., 2022); CPE only arises when underfitting occurs. This study thus offers a nuanced perspective on the factors contributing to CPE. Additionally, by building on Zeno et al. (2021), we show how tempered posteriors represent proper Bayesian posteriors under different likelihood and prior distributions, jointly parameterized by the temperature parameter T. Consequently, by adjusting T,, we effectively identify Bayesian posteriors with less misspecified likelihood and prior distributions, leading to a more accurate representation of the training data and improved generalization performance. Furthermore, we delve into the relationship between prior/likelihun misspecification, data augmentation, approximate inference, and CPE, offering insights into potential strategies for addressing these issucs.
Contributions (i) We theoretically demonstrate that the presence of the CPE implies the Bayesian posterior is underfitting in Section 3. (ii) We show in more general case that any tempered posterior is a proper Bayesian posterior with an alternative likelihood and prior distribution in Section 4, exlending Zeno et al. (2021) in the case of classification. (iii) We show in Section that likelihood misspecification and prior misspecification result in CPE only if they also induce underfitting. Furthermore, the tempered posteriors offer an effective and well-founded Bayesian mechanism to address the underfitting problem. (iv) Finally, we show that data augmentation results in stronger CPE because it induces stronger underfitting of the
2

---

Published in Transactions on Machine Learning Research (08/2024)
Bayesian posterior in Section 6. In conclusion, our theoretical analysis reveals that the occurrence of the CPE signifies underfitting of the Bayesian posterior. Also, fine-tuning the temperature in tempered posteriors offers a well-founded and effective Bayesian approach to mitigate the issue. Furthermore, our work aims to settle the debate surrounding CPE and its implications for Bayesian principles, specifically within the context of deep learning.
2 Background
2.1 Notation and assumptions
Let us start by introducing basic notation. Consider a supervised learning problem with the sample space yx x X. In this work, we consider two cases: when y a finite set, corresponding to supervised classification problem, and when ) is a subset of R. corresponding to regression problem. For simplicity, we also assume that X' is subset of RRd Let the training set D (Y, X),, where Y denotes the set of output entries and X denotes the set of input entries. If D consists pairs of samples, we denote D = (;)),
We assume a family of probabilistic models parameterized by O where each E θ defines a conditional probability distribution for a sample x)), denoted by p(y|æ, 0). The standard metric to measure the quality of a probabilistic model 0 on a sample (y. x) the (negative) log-loss - In p(y x. 0). The expected (or population) loss of probabilistic model 0 is defined as L(0) = Egg.))=l-- In p - 0)], where denotes the unknown data-generating distribution v on y X. The empirical loss of the model O the data D is defined as O(D,0) - lnp(( X 0) -1 - Eien Inp(ys/a,, 0). In this work, we assume that the likelihood function fully factorizes, i.e., p(Y| for p( X 0) in the presentation when the roles of caput/outpin in the samples are not important in the context. Also, if it induces no ambiguity, we use [.] as a shorthand for [gya))=ll
(D p(y x 0). We might use the notation p(D(0)
2.2 (Generalized) Bayesian learning
In Bayesian learning, we learn a probability distribution D), often called a posterior, over the parameter space & from the training data D. Given a new input æ, the posterior P makes the prediction about y through (an approximation of) Bayesian model averaging (BMA) p(y]z.,) posterior p is used to combine the predictions of the models. Again, if it induces no ambiguity, we use B,]] shorthand for Eo-p[t) The predictive performance of such BMA is usually measured by the Bayes loss, defined by
Ee~olp(y)z 0)), where the
B(p) = [-,[-,nnii (xppyl/eppp((p
(1)
For some > 0 and a prior p(0), the so-called tempered posteriors (or the generalized Bayes postcrior) (Barron & Cover, 1991; Zhang, 2006; Bissiri et al., 2016; Grünwald & van Ommen, 2017), are defined as a probability distribution
D) P(
(2)
Note that when X + 1. JM OY might not be 1 in general. An implicit assumption is that (0 D) is proper distribution, meaning the normalization constant is finite. In supervised classification problems, this is always the case because p(Y|X, 0) < 1. Consequently, for any X > O, we have 1 = (() d0 > f (| p() de. Thus, the tempered posteriors are always a proper distribution in supervised classification problems.
Even though many works on CPE use the parameter T -= 1/> instead, we adopt À in the rest of the work for the convenience of derivations. Therefore, the CPE (T 1) corresponds to when À > 1. We also note that while some works study CPE with a full-tempering posterior, where the prior is also tempered, many works also find CPE for likelihood-tempering posterior (see (Wenzel et al., 2020) and the references therein). Also, with some widely chosen priors (e.g, zero-centered Gaussian priors), the likelihood-teemprinng posteriors are equivalent to full-tempering posteriors with rescaled prior variances (Aitchison, 2021; Bachmann et al., 2022).
When À the tempered posterior equals the (standard) Bayesian posterior. The tempered posterior can be obtained by optimizing generalization of the so-called (generalized) ELBO objective (Alquier et al., 2016;
3

---

Published in Transactions on Machine Learning Research (08/2024)
Higgins et al., 2017), which, for convenience, write as follows:
PA () = arg min En _ KL(p(01D),((O))
The first term is known as the (un-normalized) reconstruction error or the empirical Gibbs loss of the posterior P on the data D, denoted as G(p, D) F pD) which further equals to E,,Lf((,,)) 0)]- Therefore, it is often used as the training loss in Bayesian learning (Morningstar al., 2022). The second term is Kullback-Leibler divergence between the posterior D) and the prior p(0) scaled by hyper-parameter A.
As it induces no ambiguity, we will use PA as shorthand for px D). So, for example, B(xx) would refer to the expected Bayes loss of the sempero--posterior px D) In the rest of this work, we will interpret the CPE as how changes in the parameter affect the lest error and the training error of px or, equivalently, the Bayes loss B(px) and the empirical Gibbs loss G(px. D).
3
The presence of the CPE implies underfitting
Section Overview
We present a definition of the Cold Posterior Effect (CPE) (Definition 1) and show that the presence of CPE indicates the Bayesian posterior is underfitting, where both the testing loss (Definition 1) and training loss (Proposition 1) can be improved at the same time by decreasing the temperature T (increasing A). We also present the necessary condition of the CPE in Proposition 3 and the case when Baycsian postcrior is optimal in Theorem 4.
A standard understanding for underfitting refers to situation when the trained model cannot properly capture the relationship between imput and output in the data-generating process, resulting in high errors on both the training data and lesting data. In the context of highly flexible model classes such neural networks, underfitting refers to a scenario where the trained model exhibits (much) higher training and testing losses compared to what is achievable. Essentially, it mcans that there exists another model in the model class that achieves lower training and testing losses simultaneously. In the context of Bayesian inference, we argue that the Bayesian posterior underfitting if there exists another posterior distribution with lower empirical Gibbs and Bayes losses at the same time. In fact, we will show later in Section 4 that such a posterior is essentially another Bayesian posterior but with different prior and likelihood function. Before delving into that, we focus on characterizing the cold posterior effect. (CPR) and its connection to underfitting.
As previously discussed, the CPE describes the phenomenon of getting better predictive performance when we make the parameter of the tempered posterior, X, higher than 1. The next definition introduces a formal characterization. We do not claim this the best possible formal characterization. However, through the rest of the paper, we will show that this simple charactorization is enough understand the relationship between CPE and underfitting
Definition 1. We say there is a CPE for Bayes loss if and only if the derivalive of the Bayes loss of the posterior px, B(px), evaluated at X negative. That is,
d B(pa)/=-1 <0.
(4)
dA
where the magnitude of the derivalive a B(px) defines the strength of the CPE.
According to the above definition, a (relatively large) negative derivative d B(px))== implics that by making A slightly greater than 1, we will have a (relatively large) reduction in the Bayes loss with respect to the Bayesian posterior. Note that the derivative d
da
ax B(x))x== is not relatively large and negative, then we can not expect a relatively large reduction in the Bayes loss and, consoquence, the CPE will not be significant. Obviously, this formal definition could also be extended to other specific À values different from 1, or even consider some aggregation over different values. We will stick this definition because it is simpler, and
4

---

Published in Transactions on Machine Learning Research (08/2024)
the insights and conclusions extracted here can be easily extrapolated to other similar definitions involving the derivative of the Bayes loss.
Next, we present another critical observation. We postpone the proofs in this section to Appendix A.
Proposition 2. The derivative of the empirical Gibbs loss of the tempered posterior Px satisfies
d D)= = - (Ipp((D)))) 10,
YAD0 0
(5)
dA
where V() denotes the voriance.
As shown in Proposition 7 in Appendix A, to achieve pp, (Inp(D)0)) = 0, we need px = p(0), implying that the data has no influence on the posterior. In consequence, in practical scenarios, Vp, (In ((D0)) will always be greater than zero. Thus, increasing will monotonically reduce the empirical Gibbs loss G(p., D) (i.e., the train error) of px- The next result also shows that the empirical Gibbs loss of the Bayesian posterior G(px=1) camnot reach its minimum to observe the CPE.
Proposition 3. A necessary condition for the presence of the CPE, as defined in Definition 1, is that
G(px=1, D) > min - min inin-(np(D(0))
Insight 1. Desinilion in combination with Proposition shows if the CPE is present, by making X >> 1, the test loss (px) and the empirical Gibbs loss G(px, D) will be reduced at the same time. Purthermore, Proposition 3 states that the Bayesian posterior still has room to fit the training data further (e.g., by placing more probabilily mass on the mazimum likelihood eslimalor). We hence deduce that the presence of CPE implies that the original Bayesian posterior (A = 1) underfits. This conclusion arises becouse there erists another Bayesian posterior (i.e, px(O)D) with >> 1) that has lower training (Proposition 3) and testing (Definition 1) loss at the same time. Further elaboration on the nature of D)) as another Bayesion posterior will be provided later in Section 4- In short, if there is CPE, the original Bayesian posterior is underfitting. Or, equivalently, if the original Bayesion posterior does not underfit, there is CPE.
However, final question arises: when is A = 1 (the original Bayesian posterior of interest) optimal? More precisely, when does the derivative of the Bayes loss with respect to X evaluated at X = 1 become zero (# ((p)))==1 = 0)? This would imply that neither (infinitesimally) increasing nor decreasing À changes the predictive performance. We will see that this condition is closely related to the situation that updating such a Bayesian posterior with more data does not enhance its fit to the original training data better. In other words, the extra information about the data-generation process does not provide the Bayesian posterior with better performance the originally provided training data.
We start by denoting (0) (y. x)) as the distribution obtained by updating the posterior PA (0)D) with one new sample (y, i.e., fx(0 D, (y, æ)) x p(yla, DD
And we also denote Px as the distribution resulting from averaging PA (0 D. (y, x)) over different unseen samples from the generatnng distribution (y. x) ~ v(y,
() = E, (g,
(6)
In this sense, Pa represents how the posterior Px would be, on average, after being updated with a new sample from the dota-generating distribution. This updated posterior contains a bit more information about the nata-generating distribution, compared to Px- Using the updated posterior Px, the following result introduces characterization of the optimality of the original Bayesian posterior.
Theorem 4. The derivative of the Bayes loss at X = 1 is null, i.c., a B(p)) 0, if and only if,
G(p D) G(
The original Bayesian posterior of interest is optimal if after updating it using the procedure described in Equation 6, or in other words, after exposing the Bayesian posterior to more data from the neta-ggenngaiing distribution, the empirical Gibbs loss over the initial training data remains unchanged.
Insight 2.
We will give examples that empirically illustrate Theorem 4 and the induced insight later in Section 5.5.
5

---

Published in Transactions on Machine Learning Research (08/2024)
4 Tempered posteriors are Bayesian posteriors
Section Overview
By extending Zeno et al. (2021) on classification only, show in general that tempered posteriors are proper Bayesian posteriors with an alternative combination of likelihood and prior functions parameterized by A. Thus, the occurrence of CPE can be explained within the Bayesian framework.
We provide two examples to show how influences the new likelihoods in Section 4.1 and two examples to show how À influences the new priors in Section 4.2.
We show in Section 4.3 that the generalized ELBOs are also proper ELBOs.
We expand the discussion of the implications in Section 4.4.
As previously discussed, the CPE phenomenon involves achieving improved predictive accuracy by employing a tempered posterior. A potential criticism is that this tempered posterior does not strictly adhere to the principles of proper Bayesian posterior because the tempered likelihood, P(D fails to meet the criteria of proper likelihood function when À + 1 (i.e., P(DIO) # when À + 1). However, as previously discussed by Zeno et al. (2021), this tempered posterior effectively serves a proper Bayesian posterior with combination of new likelihood and prior functions. We extend this result beyond classification to our Proposition 5, proved in Appendix B.1.
Before delving into the description the new likelihood and prior functions, it is essential to acknowledge a fundamental aspect. Given a labeled dataset D D= (X, Y) and the conditional likelihood associated to a classification model, the application of Bayes' theorem naturally results in the following Bayesian posterior:
X, Y) (0)((
where the prior over 0 is a conditional prior(Marek et al., 2024; Zeno et al., 2021) that depends on the unlabelled training data X. However, specifying X) for a complex model, like a deep neural network, poses a significant challenge. Therefore, for practical purposes, nearly all existing works (Wenzel al., 2020; Fortuin et al., 2022) assume O to be independent of X, resulting in the simplified expression X() Y) & (0)(X,))((0) where the prior over O is now an unconditional prior.
Proposition 5. For any given dalaset D (X,Y) such that the likelihood fully factories p(Y)X, TyyD plylax 0), and À > 0, the tempered posterior defined in Equation 2 can be expressed as Bayesian posterior with new prior and likelihood function follows:
(x (0)XX A) II qyyl. 0 A),
(D.E)ED
where the new prior distribution q(0|X, A) and likelihood function g x, 0 A) are defined as:
plyla, 0)*
(0 x (()
II #EX
p ply|,, 0y
q(1|,, 0, = ] p Oy
(8)
Note that the new conditional likelihood q ar, 0, A) and the new prior (0 A) are both parametrized by the same 0, and note that the prior only depends on the unlabelled training data X.
Adlam et al. (2020) shows that in the specific scenario of Gaussian process regression (essentially Bayesian linear regression example in Figure 2), any positive temperature aligns with a legitimate posterior under an adjusted unconditional prior. This can be seen as special case of our argument.
In the rest of the section, we will discuss Section 4.1 how the new likelihoods change with respect to X, and Soction 4.2 how the new priors change with respect to X. Additionally, besides demonstrating that tempered posteriors are proper Bayesian posteriors, we also show in Section 4.3 that the generalized ELBOs are also proper ELBOs. Lastly, we discuss the implications of the results in Section 4.4.

---

Published in Transactions on Machine Learning Research (08/2024)
g ze, 0)
q(0) X, A)
q(0 X ;)
2-1
A=1
A=1
0.8
A 10
==3
==3
A=
10-
A=
0.6
1,5
+10(X
0.2
0.0
1.0 0.0
0.2
0.6
8(x) 1.0
0.00 0.25 0.75
1.00 1.25 (r(()
1.50
1.75
2.00
800)
Figure 1: Illustration of the new likelihood (x A) and priors g00 A). In the left and middle figures, the original likelihood the form the Bernoulli distribution. The left figure demonstrates the transformation from ((x)
In the middle figure, we display a Beta-Binomiaa example, where the prior, initialized as Beta distribution, is updated with a single Bernoull- sample. In the right figure, we display the new prior, initialized as an inverse inverse-gamma prior and updated with a Gaussian likelihood with single observation.
0" (x; A) == 01a)>(1--0aa)
*(x)*
4.1 How À influences the new likelihoods
The next result, proved in Appendix B.2, shows that in supervised classification settings, higher À values induce new likelihood distributions with lower aleatoric uncertainty or, equivalently, lower Shannon entropy, denoted as H (g(y) 0, A)) == -Evey a 0, X) In (yyl,, O.
Proposition 6. For any 0 e ④ any x E X, and any finite output set V, the entropy of the conditional likelihood q A) monotonically decreases with X > 0, i.e.,
d d Higlyl 0, A)) 5O AY
(9)
This result also holds for regression settings, where y Rd, under the differential entropy, assuming the Leibniz rule holds. See the proof for detailed discussion on the matter.
We give two concrete examples, one in regression and one in classification, to illustrate the proposition.
Regression Example Consider the case where the original likelihood is Gaussian, defined as p(y) x. 0) N(u(æ, o2))), where the variance is inpul-independent, as typically seen in many regression problems. Then, following Equation 8, the new likelihood corresponds to scaling in the voriance, given by 4(3) ar 0, A) N(n(x, 0), <?(0)). Thus, as X increases, the tempered likelihood q(yl2x, 0, X) induces a proper Gaussian likelihood with reduced variance, i.e., a new likelihood with lower aleatoric uncertainty.
Classification Example Consider the case of a binary classification problem where the original con- ditional likelihood Bernoulli, defined as plyla 0) (()) (()))- with y E {0,1] and the input- dependent parameter function f(x=) [0, 1], which is usually implemented by neural network with softmax activation function in the last layer. Then, following Equation 8, the new conditional likelihood q3y/2x, A) = (x,, - 8" (x, (zx A)) also follows Bernoulli distribution with different parameter B(a)*
function 8t (xx (1=P))( E []. The function 6" (xx X) is displayed in Figure 1 (left). When X increnses, the parameter function that defines the new Bernoulli likelihood becomes more extreme, resulting in new likelihood with lower aleatoric uncertainty-
both cases, we see that as suggested by Proposition 6, as À increases, the new conditional likelihoods ( 0, A) have lower entropy, i.c., lower alcatoric uncortainty. In Sections 5.2 and 5.3, we will further explore the implications of this finding and its connection to existing literature.
4.2 How À influences the new priors
On the other hand, according to Proposition 5, using the tempered posteriors implics implicitly using the prior (()) x,))))). Such prior depends the unlabelled training data X. On top of that, the functional form
7

---

Published in Transactions on Machine Learning Research (08/2024)
of the likelihood function is defined by the probabilistic model family through the term p(y x. dy for a E X. Hence, models 0 that yield large value for this term across most of the training data E X will be assigned larger probability mass by the new prior. We will showcase this effect in both regression and binary favors those models within the model class that yield likelihoods with lower alealoric uncertainty on the training dala X.
classification problems. Moreover, we will see how the new prior g(0 X) with
Regression Example Consider the case where the original likelihood is Gaussian, defined as ply x., 0) M(p(x,0) o where the variance is input-independent, as typically seen in many regression problems. A common parumetrization involves O = (w, where w refer the weights of the neural network deftning the funclion plæ, 0) and ? > O is a parameter encoding the varionce of the Gaussian likelihood such that (0) %. The prior p(0) is then defined as p(0) p(w))p(),, where p(w) is usually Gaussion distribution with diagonal coveriance matrix, and p(1) is usually defined in terms of inverse-gamma distribution. Following Equation 8, the new prior would be expressed as g(0 q(w (xw,X, ()))() [)l())XX))) where each term
((w) X A) = pp(n)
1(11x,) & p(7)\/-(A-1))
Figure/ 1 (right) plots the density of x( when only one data is observed, with various À >>> values when p(y) is an inwerse- inverse-gamma prior. For larger X values, this new prior will assign more probability mass to models defining a likelihood with smaller variance or, equivalently, smaller aleatoric uncertainty.
Classification Example Consider another case where the original conditional likelihood Bernoulli, defined as ply),, x 0) ((x))((- (x)))²-v with y {0,1} and 0(x) E [0, 1], as commonly used in binary classification problems. Also, take any prior p(0). Then, following Equation 8, the new prior is expressed
(0) x O()) II (o(a)) + (1- (1=0()))) #EX
Figure 1 (middle) illustrates the transformation of the prior for Belu- Binnomall model with single training somple. Triitiall,, the prior p(0) is taken as a Beta distribution, while the likelãhood of this single data is Bernoulli. As X 2 1 increases, the new prior assigns more probability mass to models where 0(x) is close to either or 0. In other words, this new prior assigns more probobility mass to models that assign more extreme probabilities to the training data (i.e,, models with lower aleatoric uncertainty)- Note that the prior does not consider how accurately these models classify the training data, but only the extremity of the probabilities assigned to the training dala.
We will discuss in Section 5.4 further implications of this finding and how it relates to the literature.
4.3 Generalized ELBOs are also proper ELBOs
Generalized ELBOs, characterizod by scaling the KL divergence term using a hyper-parameter X, have found widespread application in many studies (Wenzel et al., 2020). This popularity stems from the demonstrated ability to adjust to improve the predictive accuracy of variational approximations:
x == arg min In + KI(r(().,(()). r
(10)
where II defines the variational family. Critics have pointed out flaw in the above generalized ELBO when À deviates from 1, as it no longer functions
true lower bound for the marginal likelihood. However, Proposition 5 can be used to justify that such a variational posterior z still emerges from minimizing valid ELBO. Specifically, it is rucstd based on the revised likelihood and prior functions as follows:
9 = arg min E, In q X, 0, X)] + (L(r()). X,))
(11)
rell
Consequently, this analysis shows that using generalized ELBOs as Equation perfectly adheres to variational and Bayesian principles.
8

---

Published in Transactions on Machine Learning Research (08/2024)
4.4 Insights and implications from the section
In this section, we show that employing tempered posteriors seamlessly fits within a Bayesian framework, which strcamlines and enriches the use of diverse likelihood and prior functions.
With the characterization in Section 3, observing CPE implies that the lempered posterior, which implicitly employs the new likelihood and priors defined in Equation 8 is better specified in comparison to the original Bayesian posterior. The alignment with the underlying data-generating distribution is casily achieved by tempering. Consequently, tempered posteriors offer a simple, computationally efficient, and theoretically sound approach mitigate the underfitting problem often encountered contemporary Bayesian deep learning methods.
Furthermore, discussed in Section 4.1 and Section 4.2, increasing À results in likelihoods with lower aleatoric uncertainty and priors that favor models yielding such likelihoods on the training data X. Therefore, the occurrence of CPE in contemporary Bayesian deep learning indicates that the models currently employed in the field often underfit the data by assuming models with too high alcatoric uncertainty. This strengthens our understanding the CPE as a consequence of underfitting, resulting from poorly specified likelihood and prior functions.
We will further expand on and discuss how these implications relate to the literature in Section 5.
5 Likelihood misspecification, prior misspecification and the CPE
Scction Overview
We relate our analysis in previous sections to the main arguments of CPE from the literature.
Section 5.1: bring CPE, showing CPE is not merely side effect of approximate inference in NNs.
demonstrate with Bayesian lincar regression that exact inference can also
using the same regression examples, we show that model misspecification can lead to underfitting or overfitting. CPE arises specifically when misspecified likelihoods priors cause underfitting, not just from misspecification alone.
Section 5.2:
likelihood misspecification often identified as a source CPE in practice. We show it because the standard softmax likelihood (high aleatoric uncertainty) misspecified and underfits the data-generating process (curated data with low alcatoric uncertainty).
Section 5.3:
Section 5.4: we show prior misspecification leads to CPE if it induces underfitting. Using tempered posteriors implicitly defines better-specified conditional priors that alleviate it
Section 5.5: we show that larger models have more flexibility to fit data, thereby mitigating underfitting and CPE. Conversely, with small models and abundant data, the Bayesian posterior may already fit the data optimally, thereby exhibiting minimal underfitting (by our definition) and CPE.
5.1 CPE, approximate inference, and NNs
As mentioned in the introduction, several works have discussed that CPE is an artifact of inappropriate approximate inference methods, especially in the context of the highly complex posterior that emerge from neural networks (Wenzel et al., 2020). There are occasions suggesting that if the approximate inference method is accurate enough, the CPE disappears (Izmailov et. al., 2021). However, Proposition shows that when is made larger than 1, the training loss the exact Bayesian posterior decreases; if the test loss decreases too, the exact Bayesian posterior underfits. It means that even if the inference method is accurate, we can still observe the CPE due to underfitting. In fact, Figure 2 shows examples of a Bayesian linear regression model learned synthetic data. Here, the exact Bayesian posterior can be computed,

---

Published in Transactions on Machine Learning Research (08/2024)
and it is clear from Figures 2c and 2d that the CPE can occur in Bayesian linear regression with exact inference. Although simple, the setting is articulated specifically to mimic the classification tasks using BNNs where CPE was observed. In particular, the linear model has more parameters than observations (i.e., it' overparameterized)) We also note that Adlam et al. (2020) presents similar findings and observations for Gaussian process regression.
good 91
overft
uncorft
uncorft
good
2-10
4-3.5
Tut fandorr
-1.
055
0.
-12
0.0
-12
05
0.
-18
055
0.5
-18
-35
05
1.0
OPE
CPE
with CPE
with decreased
2.0
1.5
225
20
1.5
Gipu,
1.4
BiDs)
1.5
1.2
12
1.0
25
05
15
25
5.0
05
10
20
3.0
08
1.0
15
25
35
0.
15
25
355
(a) No likeltbood or prior misspecifeattion
(b)
Misspecified
(c)
Misspecifted
(d) Only misspecifted prior
(e) Same as (c) but with
likelihood
likelhhodd II
50 data points
Figure 2: The CPE occurs in Bayesian linear regression with exact inference. 2. Model misspecification can lead to overfitting and to a "warm" posterior effect (WPE). Every column displays a specific selting, indicated in the caption. The first row shows exact Bayesian posterior predictive fits for three different values of the tempering parameter x. The second row shows the Gibbs loss G(px, D) (aka training loss) and the Bayes loss B(px) (aka testing loss) with respect to A. The experimental details are given Appendix D.
5.2 Model misspecification, CPE, and underfitting
Prior and/or likelihood misspccification can lead Bayesian methods to both underfitting and overfitting, widely discussed in the literature (Domingos, 2000; Immer et al., 2021; Kapoor el al., 2022). We illustrate this using a Bayesian linear regression model: Figures 2c and 2d show how the Bayesian posterior underfits due to likelihood and prior misspocification, respectively. On the other hand, Figure 2b showcasos a scenario where likelihood misspecification can perfectly lead to overfitting as well, giving rise to what we term a "warm" posterior effect (WPE), i.e., there exist other posteriors (px with 1) with lower testing loss, which, at the same time, have higher training loss due to Proposition 2. As a result, describe CPE merely as model misspecification issue without acknowledging underfitting offers narrow interpretation of the problem.
The examples presented in Figure 2 help illustrate the results of Proposition 5 and provide concrete demonstrations of the theoretical insights discussed: when CPE shows up, tuning À is akin to finding another Bayesian posterior with a less misspecified likelihood and prior. However, we note that in this particular Bayesian linear regression setup, the new prior (()) X, A) is always equal to initial prior p(0) because the variance of the likelihood is assumod to be constant. Therefore, the analysis of the regression case Scction 4.2 does not directly apply here.
In the discussion regarding likelihood, we refer to the regression example in Section 4.1. Let's first have a look at Figure 2c, where the Gaussian likelihood model has larger variance than the true data-generating process. By increasing X, we obtain a likelihood model with a smaller variance (divided by X², as shown in the regression example in Section 4.1), i.e. we induce a new likelihood with lower aleatoric uncertainty (Proposition 6). Such a new model is closer to the true data-generating distribution and less misspecified, thus enjoying better performance. The opposite can be seen in Figure 2b, where the Gaussian likelihood model has a lower variance than the true data-generating distríbution and the WPE occurs.
10

---

Published in Transactions on Machine Learning Research (08/2024)
5.3 The likelihood misspecification argument
Likelihood misspecification has also been identified as cause of CPE, especially in cases where the datasel. has been curated (Aitchison, 2021; Kapoor et al., 2022). Data curation often involves carefully selecting samples and labels to improve the quality of the datasct. As a result, the curated data-generating distribution typically presents very low aleatoric uncertainty, meaning that r(y/z) usually takes values very close to either or However, as previously discussed in (Aitchison, 2021; Kapoor al., 2022), the standard likelihoods used in deep learning for image classification, like softmax or sigmoid, tend allocate more spread-out probabilities to the outcomes, implicitly reflecting higher level of alcatoric uncertainty. Therefore, their use in curated datasets that exhibit. low uncertainty made them misspecified (Kapoor et al., 2022; Fortuin et al., 2022). To address this issue, alternative likelihood functions like the Noisy-Dirichlet model (Kapoor et al., 2022, Section 4) have been proposed, which better align with the characteristics of the curated data. On the other hand, introducing noise labels also alleviates the CPE, as demonstrated in Aitchison (2021, Figure 7). By introducing noise labels, we intentionally increase alcatoric uncertainty the data-generating distribution, which aligns better with the high aleatoric uncertainty assumed by the standard Bayesian deep networks (Kapoor et al., 2022). Consequently, according to these works, the CPE be strongly alleviated when the likelihood misspecification is addressed.
Our theoretical analysis aligns with these findings in Sections 4.1 and 4.4: fitting low alcatoric uncertainty data-generating distributions, e.g., ((yla)) E {0.01, 0.99), with high aleatoric uncertainty likelihood functions ions e.g., p(yl æ 0) E [0.2, 0.8], induces underfitting, and thus, CPE. The presence of underfitting is not mentioned at all by any of these previous works (Aitchison, 2021; Kapoor et al., 2022). On top of that, using Propositions and 6, our work explains why the likelihood implicitly used by the tempered posterior with À provides better generalizaton performance. Because, in this case, we are using a likelihood x(0)X, A) (Equation 8) with lower aleatoric uncertainty, which better aligns with the low aleatoric uncertainty data-generating distribution induced by curated datasets, thus reducing the degree of model misspeciffcation.
0.0005, y 1.0, no aug.
0.0005, y 2.0, aug.
0
1.0, y 1.0, aug.
1.5
1.5
NLL (lower the better)
G(pa.
NLL (lower the better)
NLL (lower the better)
1.0
B(px)
1.0
0.5
0.5
0.0
0.0
1.0
1.5
3.0
3.5
1.5 2.0
2.5
0.5
1.0
2.0
2.5
3.0
3.5
A
(a) Narrow prior and standard softmax
(b) Narrow prior and tempered softmax
(c)
Standard prior standard softmax
Figure 3: Experimental illustrations for the arguments in Section 5 using small CNN via SGLD on MNIST. We show similar results on MaSHIO with small CNN and CIFAR-10(0) with ResNet-18 in Appendix E. Figures 3a to 3c illustrate the arguments in Section 5. Figure 3c uses the standard prior (a 1) and the standard softmax (r 1) for the likelihood without applying DA. Figure follows similar setup except for using narrow prior. Figure 3b uses a narrow prior in Figure 3a but with tempered softmax that results in lower alcatoric uncertainty. We report the training loss G(px, D) and the testing losses, ((px) and G(px), from samples of the small Convolutional neural network (CNN) via Stochastic Gradient Langevin Dynamics (SGLD). We show the mean and standard error across three different seeds. For additional experimental details, please refer to Appendix E.
Figures 3a and 3b, along with Figures 4a and 4b, illustrate this point through regular multi-class classification task on curated benchmark datasct. Both secnarios utilize the same narrow prior. The distinction in Figure 3b lies in the adoption of a tempered softmax likelihood, defined as p(y|æ,0) 0) = (1 + exp (-> logits(x,, 0))) -1 with y = 2, compared to y = in Figure 3a. This tempered softmax likelihood, more closely aligned with the datasct's low alcatoric uncertainty as outlined by (Guo et al., 2017), leads to a reduced incidence of CPE in Figure 3b compared Figure 3a. From the perspective of Proposition 5 and specifically Proposition 6, the intrinsic lower aleatoric uncertainty of the likelihood used in Figure 3b (softmax with 7 2) makes
11

---

Published in Transactions on Machine Learning Research (08/2024)
0 0= 0.0005, y 1.0, no aug.
0= 0.0005, y= 2.0, no aug.
0= 1.0, y 1.0, no aug.
1.5
1.5
1.5
NLL (lower the better)
G(p.. D)
NLL (lower the better)
NLL (lower the better)
1.0
Blpx)
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
À
A
(a) Narrow prior and standard softmax
(b) Narrow prior and tempered softmax
(c)
Standard prior and standard softmax
Figure Experimental illustrations for the arguments in Section using large CNN via SGLD on MNIST. We show similar results on Fashion-MNIST with large CNN and CIFAR-10(0) with ResNet-50 Appendix The experiment setup similar to the setups Figure but with large CNN. Please refer Appendix F. for further details on the model.
the potential for improvement through increasing À somewhat limited, resulting in a less pronouneed CPE compared to Figure 3a. It is, however, important to highlight the critical interaction between the likelihood and the prior, as we dicuss next.
5.4 The prior misspecification argument
As highlighted in previous works, such as in Wenzel et al. (2020): Fortuin et al. (2022), isotropic Gaussian priors are commonly chosen in modern Bayesian neural networks for the sake of tractability in approximate Bayesian inference rather than chosen based their alignment with our actual beliefs. Given that the presence of the CPE implics that either the likelihood and/or the prior are misspecified, and given that neural networks define highly flexible likelihood functions, there are strong reasons for thinking these commonly used priors are misspecified. Notably, the experiments conducted by Fortuin et al. (2022) demonstrate that the CPE can be mitigated in fully connected neural networks when using heavy-tailed prior distributions that better capture the weight characteristics typically observed in such networks, However, such priors were found to be ineffective in addressing the CPE in convolutional neural networks (Fortuin et al., 2022), indicating the challenges involved in designing effective Bayesian priors within this context.
Our theoretical amalysis provides a deeper insight into these observations. As discussed in Section 3, the absence of underfitting means the absence of CPE. This suggests flexible likelihood functions may still result in posteriors that underfit due to the prior's tendency to overly regularize. This may incur CPE when the strong prior fails to allocate enough probability to models that both fit the training data well and exhibit good generalization capabilities. As detailed in Section 4.2, employing tempered posteriors with X effectively defines a conditional prior {0| A) that favors modcls with lower alcatoric uncertainty. If such models align better with the training data compared to the original prior, then may observe CPE. Also, the conditional prior q(0) X A) with À > 1 can be considered better specified than the original prior p(0).
Figures 3a and 3c exemplify this situation. The prior in the case of Figures 3a is very narrow (a = 0.0005), inducing strong regularization. Such a narrow prior results in a posterior that severely underfits the training data, evident from the high empirical Gibbs loss that deviates significantly from zero. Additionally, we observe strong CPE. On the other hand, with flatter prior in the case of Figure 3c, the CPE is considerably diminished. According to the discussion above and Sections 4.2 and 4.4, we know that the flatter prior allocates more probability mass to preferred models. Also, such preferred models have lower aleatoric uncertainty than the ones assigned initially by the narrow prior in the former case. To claborate further, in the former case, the new prior q(0) X, A) with À >> 1 would place much more probability mass to models with lower aleatoric uncertainty than the narrow prior and strongly alleviating underfitting, In the second case, since the flatter prior already distributes probability mass more broadly across the model class, the room shift probability mass to models with lower alcatoric uncertainty is more limited than that from a narrower prior, resulting in milder CPE.
12

---

Published in Transactions on Machine Learning Research (08/2024)
5.5 Model size, sample size in relation to CPE and underfitting
Larger models have the capacity to fit. data more elfectively, while smaller models are more likely to underfit. As we have argued that if there is no underfitting, there is no CPE, we expect that the size of the model has an impact on the strength of CPE as well. We demonstrate in Figure 3 and Figure 4. Specifically, in our experiments presented in Figure 3, we use a relatively small convolutional neural network (CNN), which has a more pronounced underfitting behavior, and this indeed corresponds to a stronger CPF. On the other hand, we employ a larger CNN in Figure 4, which has less underfitting, and we see the CPE is strongly alleviated. Actually, this effect can be directly inferred from Theorem 4. For an extremely flexible model capable of perfectly fitting both the original training samples and new samples, this theorem suggests that a CPE should not be expected, as the model's fit on the original data remains perfect, even when new examples are introduced.
Theorem 4 can also be used to understand why small models in the presence of large training data sets do not exhibit CPE. We empirically illustrate this point in Figure 2. In particular, Figure 2c and Figure 2c use the same (small) regression models and settings where the only difference is that Figure 2c uses data points while Figure 2e uses 50 data points. In situations where model possesses limited flexibilityy and the training set is large, including additional examples should barely affect the fit of the original training data because the Bayesian posterior is highly concentrated and will be barely affected by single extra sample. Then, as predicted by Theorem 4, CPE in Figure 2e is much less significant. than in Figure 2c.
Finally, it's worth noting that Figure 11 in Wenzel et al. (2020) shows the opposite effect, where larger models exhibit much stronger CPE compared shallower or narrower versions of the same architectures. However, it's important to recognize that Wenzel et al. (2020) studied full-tempering, whereas our work focuses on lkeeihhoddtempeeree For full-tempering, Proposition 2 does not necessarily hold. Intuitively, since À operates both the likelihood (data) and the prior (regularization) simultaneously, the elfect of increasing À is mixed, not necessarily improving the fit on the training data. Consequently, the CPE brought by full-tempering as À increases does not necessarily coincide with better training loss, as the training loss may not be improvable. As a result, the CPE observed with full-tempering cannot be interpreted solely as underfitting. Therefore, for full- fulltempperngg increasing model capacity may not achieve a lower degree of CPE, unlike the behavior we observed in our case focusing on likelihood-temperingg.
6
Data augmentation (DA) and the CPE
Section Overview
We show conditions under which data augmentation exacerbates the CPE.
Section 6.1: starting with the Gibbs loss for clarity, we show that data augmentation induces a stronger CPE on the Gibbs loss if the augmented data provides more information about the data-gencrating process, incrcasing the correlation between the expected and empirical losses.
Section 6.2: extending the above idea, we show analogous conditions where data augmentation exacerbates CPE on the Bayes loss.
Machine learning is applied to many different fields and problems. In many of them, the data-generating distribution is known to have properties that can be exploited to generate new data samples (Shorten & Khoshgoftaar, 2019) artificially. This is commonly known as dato angmentation (DA) and relies the the data-generating distribution satisfies "(y|=)) v (z)(((x)) for all h E H. In practice not all the transformations are applied to every single data. Instead, a probability distribution (usually uniform) pn is defined over H, and augmented samples are drawn accordingly. As argued in Nabarro et al. (2022), the use of data augmentation when training Bayesian neural networks implicitly targets the following (pscudo) log- liceliio,oo,,, denoted Loa (D, and defined as
property that for given set of transformations H = [h : x -x+x,;, X}.
i.D (D,0)
Σ BA-AR |- In (())))((2))))))))) se[n
(12)
13

---

Published in Transactions on Machine Learning Research (08/2024)
where data augmentation provides unbiased estimates of the expectation under the set of transformations using Monte Carlo somples (i.e., random data augmentations).
Although some argue that this data-augmented (pseudo) log-likelihoee "does not have a clean interpretation as a valid likelihood function" " (Wenzel et al., 2020; Izmailov et al., 2021), we do not need to enter into this discussion to understand why the CPF emerges when using the generalized Bayes posterior (Bissiri et al., 2016) associated to this (pseudo) log- liketihoodd which is the main goal of this section. We call this posterior the DA-tempered posterior and is denoted by (0)D). The DA-tempered posterior can be expressed as the global minimizer of the following learning objective,
1 (T(((()))),
Px (0ID) arg min EpluIma ( +
(13)
This is similar to Equation 3 but now using Lo (D.0) instead of L(D,0), 0), where we recall the nota- tion O(D,0) 0) = - pn)(O)) Hence, the resulting DA-tempered posterior is given by (0 x e e-nALaa (D.0) p(0). In comparison, the tempered posterior () in Equation can be similarly expressed as e (-nAL(D.0)p(O))
There is large empirical evidence that DA induces a stronger CPE (Wenzel et al., 2020; Izmailov et al., 2021; Fortuin et al., 2022). Indeed, many of these studies show that if CPE is not present in our Bayesian learning scttings, using DA makes it appear. According to our previous analysis, this means that the use of DA induces a stronger underfitting. To understand why this is case, we will take step back and begin analyzing the impact of DA in the so-called Gibbs loss of the DA-Bayesian posterior pdd rather than the Bayes loss, as this will help us in understanding this puzzling phenomenon.
6.1 Data augmentation and CPE on the Gibbs loss
The expected Gibbs loss of given posterior p, denoted G(p), is a commonly used metric in the theoretical analysis of the generalization performance of Bayesian methods (Germain el al., 2016; Masegosa, 2020). The Gibbs loss represents the average of the expected log-loss of individual models under the posterior p, that is,
G(p) = ( [L()) = E, E [E_ lnalIIInppppu In(p(y| x, 0)]]
In fact, Jensen's inequality confirms that the expected Gibbs loss serves as an upper bound for the Bayes loss, i.e., G(p) 2 B(p) This properly supports the expected Gibbs loss to act as a proxy of the Bayes loss, which justifies its usage in gaining insights into how DA impacts the CPE.
We will now study whether data augmentation can cause a CPE the Gibb loss. In other words, we will examine whether increasing the parameter À of the DA-tempered posterior leads to reduction in the Gibbs loss. This can be formalized by extending Definition to the expected Gibbs loss by considering its derivative ax G(px) at x = 1, which can be represented as follows:
d GleNN== = pCOVpa,, (11) TA
(14)
Where COV(X, Y) denotes the covariance of X and Y. Again, due to the page limit, we postpone the necessary proofs in this section to Appendix C.
With this extended definition, if Equation 14 is negative, can infer the presence of CPE for the Gibbs loss as well. Based on this, we say that DA induces a stronger CPE if the derivative of the expected Gibbs loss for
the DA-tempered posterior exhibits more negative trend at A = 1, i... if d G(PR*)))=)))))))) < d G(px)lA=1 This dA
ax
condition can be equivalently stated as
COV P2
(), > COV 2OVp,.,(1e (ni 0), L())) > 0.
(15)
The incquality presented above helps characterize and understand the occurrence of a stronger CPE when using DA. A stronger CPE arises if the expected Gibbs loss of model L(0) is more correlated with the empirical Gibbs loss of this model on the augmented training dataset (D,,D,,) 0) than the non-augmented
14

---

Published in Transactions on Machine Learning Research (08/2024)
dataset O)..0). This observation suggests that, if we empirically observe that the CPE is stronger when using an augmented dataset, the set of transformations H used to generate the augmented dataset are introducing valuable information about the data-generating process.
1.0, y= 1.0, aug.
1.0, y 1.0, reg. aug.
0= = 1.0, y= 1.0, rand. shuffle
1.5
(lower better)
Glpa.
NLL (lower the better)
NLL (lower the better)
15
1.0
B(px)
G(px)
0.5
0.0
1.0
1.5
3.0
3.5
1.5 2.0 2.5
1.5
2.0
2.5
3.0
3.5
A
A
(a) Standard prior and standard softmax
(b) Random crop and horizontal flip
(c) Pixels randonly shuffled
Figure 5: Experimental illustrations for the arguments in Section 6 using small CNN via SGLD MNIST. We show similar results on Pashion-MNIST with small CNN and CIFAR-10(0) with ResNet-11 in Appendix E. Figures 5a to 5e illustrate the arguments in Section 6. Figure 5a uses the standard prior (o = 1) and the standard softmax (y = 1) for the likelihood without applying DA. Figure follows the setup as Figure 5a but with standard DA applied, while Figure 5e uses fabricated DA. We report the training loss C(px. D) and the testing losses (px)) and G(px) from 10 samples of the small Convolutional neural network (CNN) via Stochastic Gradient Langevin Dynamics (SGLD). We show the mean and standard error across three different seeds. For additional experimental details, please refer to Appendix E.
1.0, == 1.0, aug.
1.0, y 1.0. reg. aug.
0 1.0, y= 1.0, rand. shuffle
30
NLL (lower the better)
G(pa. D)
NLL (lower the better)
NLL (lower the better)
1.0
B(px)
20
G(px)
0.5
10
0.0
3.5
2.0
2.0
2.5
3.0
3.5
(a) Standard prior standard softmax
(b) Random crop horizontal flip
Pixels randomly shuffled
Figure 6: Experimental illustrations for the arguments in Section 6 using large CNN via SGLD on MNIST. We show similar results Fashion-MNIST with large CNN and CIFAR-10(0) with ResNet-50 in Appendix E. The experiment setup is similar to the setups in Figure 5 but with a large CNN. Please refer to Appendix for further details on the model.
Figure 5 clearly illustrates such situations. Figure 5b shows that, compared to Figure 5a, the standard DA, which makes use of the invariances inherent in the data-generating distribution, induces CPE on the Gibbs loss. Thus, the condition in Equation 15 holds by definition. On the other hand, Figure 5c uses a fabricated DA, where the same permutation is applied to the pixels of the images in the training dataset, which destroys low-level features present the iata-generating distribution. In this case, the derivative of the Gibb loss is positive, and Equation holds in the opposite direction. These findings align perfectly with the explanations provided above, showing that DA induces stronger underfitting.
15

---

Published in Transactions on Machine Learning Research (08/2024)
6.2 Data augmentation and CPE on the Bayes loss
Now, we step aside of the Gibbs loss and focus back to the Bayes loss. The derivative of the Bayes loss at.
À = 1 can also be written as,
dd B(px))==1= - pa-1 (nL.D))
(16)
where for any posterior p. (0) is a (negative) performance measure defined as
p(y|2, 0)
Sp(0) = - E,
(17)
Ep x 0)1
This function measures the relative performance of a model parameterized by 0 compared to the average performance of the models weighted by p. Such measure is conducted on samples from the data-generating distribution v(y, æ). Specifically, if the model & outperforms the average, we have (,) < -1, and if the model performs worse than the average, we have (p()) > -1 (i.e., the lower the better). The derivations of the above equations are given in Appendix C.
By Definition 1 and Equation 16, DA will induce stronger CPE if and only if the following condition is satisfied:
COV, 8- Po (D,
(0) COVPA (ni(D,,), (0)
(18)
The previous analysis on the Gibbs loss remains applicable in this context, with the use of Sp(O) as metric for the expected performance on the true data-genorating distribution instead of L(0). While these metries are slightly different, it is reasonable to assume that the same arguments we presented to explain the CPE under data augmentation for the Gibbs loss also apply here. The theoretical analysis aligns with the behavior of the Bayes loss as depicted in Figure 5.
Finally, comparing Figure 5b with Figure 6b, also notice that using larger neural network enables us to mitigate the CPE because we reduce the underfitting introduced by DA.
Related work of the data augmentation argument. The relation between data augmentation and CPE is an active topic of discussion (Wenzel et al., 2020; Ixmailov et al., 2021; Noci et al., 2021; Nabarro et al., 2022). Some studies suggest that CPE is an artifact of DA because turning off data augmentation is enough to climinate the CPE (Izmailov et al., 2021; Fortuin et al., 2022). Our study shows that this is much more than an artifact, as also argued in Nabarro el al. (2022). As discussed, the (pseudo) log-likelioood induced by standard DA is better proxy of the expected log-loss, as precisely defined by Equation 15 and Equation 18.
Some argue that when using DA, we are not using a proper likelihood function (Iamailov el al., 2021), which could be a problem. Recent works (Nabarro et al., 2022) have developed principle likelihood functions that integrate DA-based approaches, hoping to remove CPE. However, they find that CPE still persists. Another widely accepted viewpoint regarding the interplay between the CPE and DA is that DA increases the effective sample size (Tamailov et al., 2021; Noci et al., 2021): "intuitively, data augmentation increases the amount of data observed by the model, and should lead to higher posterior contraction" (Izmailov et al., 2021).
Our analysis provides a more nuance understanding of this interplay between CPE and DA. First, we show that, when the augmented data provide extra information about the data-generating process, there is a stronger CPE, as shown in Equations 15 and 18. This, in turn, leads higher postcrior concentration. But, we also show that higher posterior concentration in the context of non-meaningful DA does not improve performance; as discussed before, Figure 5c illustrates this situation. Using the analysis given in Section 4, we can also add that tempering the posterior under DA is again way to define alternative Bayesian posteriors that addresses this stronger underfitting, i.e., they better fit the training data and improve generalization.
7 Conclusions and limitations
Our research contributes to understanding the cold posterior effect (CPE) and its implications for Bayesian deep learning in several ways. Firstly, we theoretically demonstrate that the presence of the CPE implies that
16

---

Published in Transactions on Machine Learning Research (08/2024)
the Bayesian posterior is underfitting Secondly, by building on Zeno et al. (2021), show that, in general, any tempered posterior can be considered as proper Bayesian posterior with an alternative likelihood and prior distribution jointly parametrized by T, beyond merely the case of classification. Hence, fine-tuning the temperature parameter T serves as an effective and theoretically sound approach to addressing the underfitting of the Bayesian posterior. Furthermore, we comprehensively discuss the interplay between several factors and CPE, including the use of approximate versus exact inference, model misspecification, and the size of the model and samples. Finally, our analysis in Section reveals that data augmentation exacerbates the CPE by intensifying underfitting. This occurs because augmented data provides richer and more reliable information, enhancing the capacity for fitting.
Overall, our theoretical analysis underscores the significance of the CPE as an indicator of underfitting within the Bayesian framework and promotes the fine-tuning of the temperature T in tempered posteriors as a principled approach mitigate this issue. Furthermore, by dissecting the nature of the CPE and its effect on the Bayesian principle, our work aims resolve ongoing debates and clarify the role of cold postcriors enhancing the predictive performance of Bayesian deep learning models.
As limitation of this work, want to highlight that the characterization of CPE proposed here is defined only as the local change of Bayes loss at À = 1. This approach does not account for scenarios where significant decreases in Bayes loss at other values might also indicate the presence of CPE. We believe that our theoretical analysis could be expanded to include these cases as well.
Acknowledgments
YW acknowledges support from Independent Research Fund Denmark, grant number 0135-00259B and Novo Nordisk Foundation, grant number NNF210C0070621. The main part of the work was done when YW worked at University of Copenhagen. YZ acknowledge Ph.D. funding from Novo Nordisk A/S. AM acknowledges funding for cloud computing from Google Cloud for Researchers program and from the Junta de Andalucia, grant P20-00091, and UAL-FEDER, grant FAA-2000--MM-1190000 project PID2022- 139856NB-100 funded by MCIN/ AEI/ / 10.13039/500 /00 100011033 FEDER, UE and project VID2019-106227GB-J / AEI 0000999501000110333 and from the Autonomous Community of Madrid (ELLIS Unit Madrid).
L,O acknowledges financial support from
17

---

Published in Transactions on Machine Learning Research (08/2024)
References
Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: attps://www. tensorflow orrg///wwwwoowo.oooooooooo Software available from tensorflow.org.
Ben Adlam, Jasper Snock, and Samuel L. Smith. Cold posteriors and alcatoric uncertainty, 2020.
Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. The Journal of Machine Learning Research, 17(1):8374-4114, 2016.
Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. How tempering fixes data augmentation in bayesian neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Andrew Barron and Thomas Cover. Minimum complexity density estimation. IEEE Trunsactions on Information Theory, 1991.
Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methooloogy),, 1805):1103-1130, 2016.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859- 2017.
François Chollet et al. Keras. https://kera: 1o, 2015.
Joshua Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif Saurous. Tensorflow distributions. arXio preprint 2017.
Pedro Domingos. Bayesian averaging of classifiers and the overfitting problem. In ICML, 2000.
Vincent Fortuin, Adrià Carriga-Alonse Sebastian W Ober, Florian Wenzel, Gunnar Ratsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. In Proceedings of the International Conference on Learning Representations (TCLR), 2022.
Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-Bayesian theory meets Bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884-1892, 2016.
Peter Grünwald and Thijs van Ommen. Inconsistency bayesian inference for misspecified linear models, and proposal for repairing it. Bayesian Analysis, 12(4):1069-1105 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Tn ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1321 1330. PMLR, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Proceedings of the International Conference Learning Represendations (ICLR), 2017.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization. In International conference arlificial intelligence ond stalistics, pp. 703-711, PMLR, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In Proccedings of the International Conference Machine Learning (ICML), 2021.
Sanyam Kapoor, Wesley Maddox, Pavel Izmailov, and Andrew G Wilson. On uncertainty, tempering, and data augmentation in bayesian classification. Aduances in Newral Informalion Processing Systems (NewrIPS), 2022.
Martin Marek, Brooks Paige, and Pavel Izmailov. Can a confident prior replace a cold posterior? arXiv preprint ar arXiv:2433.02722 2024.
Andrés R Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. Advances in Newral Information Processing Systems, 2020.
Warren R. Morningstar, Alex Alemi, and Joshua V. Dillon. Pacm-bayes: Narrowing the empirical risk gap in the misspecified bayesian regime. In AISTATS, volume 151 of Proceedings of Machine Learning Research, PMLR, 2022.
pp.
Seth Nabarro, Stoil Ganev, Adrià Garriga-Alonso, Vincent Fortuin, Mark van der Wilk, and Laurence Aitchison. Data augmentation in bayesian neural networks and the cold posterior effect. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2022.
Lorenzo Noci, Kevin Roth, Gregor Bachmann, Sebastian Nowozin, and Thomas Hofmann. Disentangling the roles of curation, data-augmentation and the prior in the cold posterior effect. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tojani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NewrIPS, pp. 8024 8035, 2019.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, pp. 681-688. Omnipress, 2011.
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Chen Zeno, Itay Golan, Ari Pakman, and Daniel Soudry. Why cold posteriors? on the suboptimal generalization of optimal bayes estimates. In Third Symposium Advances in Approzimate Bayesion Inference, 2021. URL http:////pepaeeeeeeeeeeeeetttttr
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations, 2019.
Tong Zhang. From e-entropy to kl-entropy: Analysis of minimum information complexity density estimation. Annals of Statistics, 2006.
Yijie Zhang and Eric Nalisnick. On the inconsistency of bayesian inference for misspecified neural networks. In Third Symposium on Advances in Approximate Bayesion Inference, 2021. URL https://openrevier net/forum?id:vvfoen
19

---

Published in Transactions on Machine Learning Research (08/2024)
A Proofs for Section
In this scction, we provide the proofs for Section in the following order. We first prove the derivative of the empirical Gibbs loss in Proposition 2. Then, show in Proposilion 7 that for meaningful posteriors (depends on training data), the derivative won't be zero. Before proving Proposition 3 and Theorem 4, we first provide Proposition 8, stating an alternative expression of the derivative of the Bayes loss. The proofs of Proposition 3 and Theorem then follow from that.
A.1 Proof of Proposition 2
We first show slightly more general result of _ Epx [f(O]] for any function f(0) that is independent of Recall that the posterior P) x p(D))))p(0). With the fact that aA (p(DI0)^p(0)) In(x((D))))p(((0)>>00)) the derivative
dA Px [f(0) Ep, [In (D(()))) Ep, [I (n)(()))) (l
(19)
where we denote COV(X,Y) Y) as the covariance of X and Y. Hence, the derivative of the empirical Gibbs loss = COv, G(p,, D) = dA Ep 1 dA
= - (Inp())))).
A.2 Proposition 7
Proposition 7. For any À > 0 and D + 0, if the tempered posterior (x p(D(0))p(0 satisfies Vn (In P(DIO)) = 0, then, D) p(0).
Proof. First of all, note that the tempered posterior is defined as
p(D)O)p((O) (aP(D(0))(())
Then,
Pp (Inp(D)0)) = / (x (In((D)) - Ep, [I (n)())))
Thus, for any 0 e supp(px), it verifies that
In In Ep, [In (D()))]
That is, In (D(() is constant the support of px- Let e denote such constant, then
e p())
p
(x
Jo ecc Œod Sor(0) v()).
A.3 Proof of Proposition 3 and Theorem
In order to prove Proposition 3 and Theorem 4, we first show in Proposition 8 that the derivative of the Bayes loss of the tempered posterior px can be expressed by the difference between the empirical Gibbs loss of px and the empirical Gibbs loss px-
Proposition 8. The derivative of the Bayes loss of the tempered posterior Px can be expressed by
d B(pa) = G(px, D) G(p,, D).
(20)
dX
20

---

Published in Transactions on Machine Learning Research (08/2024)
Proof. By definition,
d
d B(p)) = dx
Exl- Bpp plyl,, 0)]] = -EE,
In Ep, (p(y)z,
Yd
where
dA Ep [ply| x, 0)] = di Ep (p x 0)
COV px (In p (O))))).pp x. 0))
Ex ( x 0)]
Ep, (pyy 0)
due to Equation 19. By expanding the covariance, the above formula. further equals to
pplyl Epp In p(D)))(( x 0) Ep, pD) (p(y), 0) = Ep, (mp(())) - Ep, [mp((
Ep. pp 0)
where the probability distribution px (x D, (y, x)) x (x 0). Put everything together, have
B(px) = Pp, (ppp(pp( E. F (D)O)) = Ep, [np(D()) E,, [Dp(())))
(21)
dA
where
Px p2 D) ()pyy) Ep. 1p( x 0)]
= - = (g. E,
The last equality is because
-1 f v(y,2) px(0) (1) de d(y,
E, (nlp((((D)))))))))
23 D. x))
do
(1)a)
Ev 1px D. (y +)) (np)
Ep. (D
The last expression in Equation 21 further equals to G D) - G(px, D) by definition.
A.3.1 Proof of Proposition
Note that for any distribution A, we have G(p, D) = En 2 ming - ((D(O).. On the other hand, Proposition 8 together with Definition give that the CPE takes place if and only if
dA B(px) = G(px-1, D) - G(px=1, D) < <.
Therefore, it is not possible to have G(px=1, D) ming Inngg-in((D1) and, the same time, G(p==1, D) G(pxz because G(pA=1, D) ming In (D(0))
A.3.2 Proof of Theorem 4
It's easy to see from Proposition that
B(p))A==1 = G(px-1, D) - G(px=1,))
dA
if and only if G(Tx=1, D) D()
21

---

Published in Transactions on Machine Learning Research (08/2024)
B Proofs for Section 4
B.1 Proof of Proposition 5
First of all, by the definition in Equation 2, and assuming a data-independent prior p(0 = p(0), the tempered posterior given by
px Y) x p('(0,
where the tempered likelihood fully factorizes as pY||,, 0)* = (X P(yla, 0)* Let similar but g-independent. function k(O, X, X) IL / p(y) a., 0)^ dy-
Therefore, x p(YiX.0)) (A:0 X, A)p(O) where we can the new prior xA)
q(0)x x X) = p(0) II / plyl dy, dy
m
and the new posterior
Igg.ae x) P(ylar, 0)^ a ( 100,X,4 IL p(y x, 0)^ dy
p(Y|X,00 0)^
II (3.))E(Y,X)
(lylar, 0)
B.2 Proof of Proposition 6
The proof is made using differential entropy, i.c. assuming continuous target values y. The only assumption is that Leibniz integral rule holds for dyylæ, 0, X) gyylo, 0, A)), verifying that
dd / (a(v) 0, A) A) In 0, A)) dy / d dd (4(y) æ 0, In 0, A)) dy.
d
In the case of supervised classification problems, we adopt the Shanon entropy, where equality holds naturally A))
d
Σ (glylz,0.A) 0,
Σ da (1 DEY
yx
From the definition of differential entropy, we got that
H(q(y 0, = 6,) 0.A1) dy
Thus, taking derivative w.r.t. À and exchanging derivative and integral leads to the following expression
d xx H 0,1) = =- - / d _ (1(y)z 0, in 0)) dy = ( (In A) / 0,
d
1)
dx 18r 0, A) dy.
Using that S d 1 x 0, A)dy d A (tyla A)dy 0, simplifies the expression
d / ing 0, da qlyla, 0, dy
H H(g 0, )))
dA
Let us consider now the second term inside the integral. Using the derivative of the quotient rule leads to the following:
d d plylx, 0)
0 In
plylæ, 0) p(y|æ, 0)* Inp(y/ 0) dy
aA 11 0,
dd p 0)* dy J p 0)^ dy
(J p(y x 0) dy)²
Where, using the definition of q( x., 0, A), we got that
p x. 0) ()) a(yla x, 0, pn 0)
p y x 0) dy
22

---

Published in Transactions on Machine Learning Research (08/2024)
and
plyla 0)^ p(y) x, 0)^ In p(ula, 0) dy
o 0, 2) / ((3/22 0, in 0) dy - 0, (n
( ppyl 0)^ dy)²
As result, we got that / Inq x, 0, A) Yd aiy 0, dy = e 0) - E, 0, (m 0)
Using (lyl,, X) definition again:
/ Ing 0, A) dA
alyl2 0) dy = E, 0) In ply x. 0)^ - E,I ply ax 0)^ J p(y x. 0)^ E (In p(y/2, 0)
p ar, 0)
Where, expanding the logarithms the denominators cancel each other, leading to
f Invlyia 0, A)
d
ai Alll, 0) dy = (lu 0) Inplya 0)*1 - E, (, (0mpppy/a 0) p(y az, 0))
XV(nn(g)) 0))
As a result, the entropy is negative.
C Proofs for Section
C.1 Proof of Equation 14
Note that
G(ps) = dA Ep, Ep. [[()) COV, (lnp(D)0), L(O)) = cOV, (-
where the second equality is by applying Equation 19. By (aking 1, we obtain the desired derivative.
C.2 Proof of Equation 16
Recall from the proof of Theorem 8 that
d
B(px)
pp [In (([(0] Ep, (In pl (D Rp, (L((D,)) Ep, (L(D,)))
dx
where (0 D) = B p.x ( DD, (y, xz)] (Equation 6), and px D, (y. x px D) 0) is the distribution obtained by updating the posterior px with one new sample x).
Therefore,
plyla a. 0) Epa [p( lx 0)] T((D,O)
E,, [L(D))) Ep [L(D) 0)] = Ey
Epa
By Fubini's theorem, the above formula further equals to p plylæ, 0) E (1 0)
Epx
Ep
L(D, 0)
Eps
E
p x
L(D,0) 0)
= Eps SSp (0) L(D,0)
Epp,, (p y x 0)]
On the other hand, since
p(y 0)
p x 0)
Spx Epx
E,
E, Epx
1
Eps (p() cc 0)
Eps
pP c 0)]
we have
Ep, (L(,,)) -Ep, (L(D.0)EEn S-Sp (0))
By putting them altogether,
d B(px) = Ep, SSp (()))(()()) 0) dx
- -Ep, [D 0) Sp, (0) = -co (p.
23

---

Published in Transactions on Machine Learning Research (08/2024)
D Experiment details for Bayesian linear regression on synthetic data with exact inference
In this section we detail the settings of the loy experiment using synthetic data and exact Bayesian linear regression in Figure 2. We also show extra results of the derivative of Gibbs loss and Bayes loss w.r.t to À approximated by samples.
To begin, we will outline the data-generating process for the synthetic data used in the experiment shown in Figure 2 and Figure 7. We sample z uniformly from the [-1, 1] interval and pass it through a Fourier transformation to construct the input of the data. That is, for a sampled x, the input æ is constructed by 10-dimensional Fourier basis function ((x)) [g (2), ...)....9 9K (x)] for K 10, where the basis functions and for other odd values of k. 9a(x)) = Vx
are defined as follows: 91 (x) = 2
sin (kx), whereas for even
values of k (x(r) 1
V cos (kz).. The distribution of the output y E R given an imput æ, denoted as v ax),
follows Normal distribution with mean 1ex and variance 1.0, where 1 is all-ones vector. That is, ((yl)) =N 1.0).
In our experiment, the likelihood model and the prior model are defined differently for the four settings in Figure 2. To enable exact inference, both the likelihood and the prior are Gaussian, which gives a closed-form solution for the posterior predictive. This choice also provides convenience when studying the CPE: different values of Ả on the likelihood term can be naturally absorbed into the Gaussian densities by adjusting the variance (dividing by X) without hindering the exact inference step. We describe them in detail in the following.
No misspecification: likelihood p(ylæ, 0) = N/00a. x. 1.0), prior p(0) NO 2). This is the baseline for comparison.
2. Misspecified likelihood I: likelihood p(y]æ,, 0) -N0 x 0.15) (the order of Fourier transformation is K 20, however note that it still contains the K = 5 Mata-generating process in its solution space), prior p(0)- (0, In this case, the model is misspecified in a way that it has smaller variance than the data- generating process.
3. Misspecified likelihood II: likelihood p(ylæ, 0) = NV x, 3.0), prior p(0) = (0, In this case, the model is misspecified in a way that it has a larger variance than the dlaa-ggeneaate process. This is similar to one of the scenarios where CPE was found: the curated data has a lower aleatoric uncertainty than the model (Aitchison, 2021).
4. Misspecified prior: likelihood plyl a 0) NN x; 1.0), prior p(0) NO0, 0.5). The prior is poorly specified in way that it is tightly centered at 0 while the best 0 should be 1.
In all the experiments, every training set consists of only samples. Since there are more parameters than the number of training data points, our setting falls within the "overparamellrizeed" regime where CPE has been observed in Bayesian deep learning (Wenzel et al., 2020).
Continuing from Figure 2, where we show the Gibbs loss G(px, D) (training) and the Bayes loss B(px) (Lesting) with respect to X, we now show their derivatives d Zx G(x, (Equation 5) and d B(px) (Equation 2x 20) respectively in Figure! 7. Here the losses are included for clearer depiction of the derivatives. To approximate the Bayes loss for generating the plot, we use 10000 data points sampled from the data-generating distribution. Also, the derivatives are approximated using 10000 samples from the exact posteriors. From Figure 7, could clearly see that the derivatives perfectly characterize the losses in all four settings.
E Experiment details for Bayesian neural networks on image data with approximate inference
In this section, we first present in Appendix E.1 the architectures of the small and large CNNs used in this paper. As promised in the main text, then provide results on additional image datasets trained with
24

---

Published in Transactions on Machine Learning Research (08/2024)
no CPE
with CPE
1.8
2.0
2.25
20
2.00
P(b))
1.4
18
1.30
0.5
05
2.
0.5
10
15
25
3.0
0.
25
GPE
CPE
with GPE
with CPE
with dcoreased GPE
0.0
0
-50
-15
0.5
1.5
20
36
-100
05
76
1.5
25
(a) No likelihood prior (b) Misspecified likellbood
(c) Misspecified likelihood I1
(d) Ouly misspecified prior
(e) Same (e) but with
misspecification
50 data points
Figure The derivatives 4pxx, D) (Equation 5) and B(s) (Equation 20) characterize the Gibbs loss G(px, D) and the Bayes loss B(px) perfectly.
Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011) in Appendix E.2. Lastly, we provide additional results using mean-field variational inference (MFVI) (Blei et al., 2017) on MNIST, where we observe that the results of MFVI align with the ones with SGLD.
E.1
Architectures of small/large CNN
Small CNN
The small CNN is similar to LeNet-5, but with 107786 parameters in total:
Convolutional layer 1. Input channels: 1 (assuming grayscale images), output channels: 6, kernel
size: 5x5, padding: 2, activation: ReLU.
Average pooling layer 1. Kernel size: 2x2, stride:
3. Convolutional layer 2. Input channels: 6, output channels: 16, kernel size: 5x5, padding: 2, activation: ReLU.
Average pooling layer 2. Kernel size: 2x2, stride:
Flattening layer. Flattens the output from the previous layers.
6. Fully connected layer 1. Input features: 784 (16 channels * * 7), output features: 120, activation: ReLU.
Fully connected layer Input features: 120, output features: 84, activation: ReLU.
8. Fully connected layer 3 (output layer). Imput features: 84, output features: num classes (specified during instantiation).
Large CNN The large CNN similar to the small CNN, but with 545546 parameters in total:
1. Convolutional layer 1. Input channels: 1 (assuming grayscale images), output channels: 6, kernel size: 5x5, padding: 2, activation: ReLU.
Average pooling layer 1. Kernel size: 2x2, stride:
Convolutional layer 2. Input channels: 6, output channels: 16, kernel size: 5x5, padding: 2, activation: ReI.U.
25

---

Published in Transactions on Machine Learning Research (08/2024)
4. Average pooling layer 2. Kernel size: 2x2, stride: 2.
Convolutional layer 3. Input channels: 16, output channels: 120, kernel size: 5x5, padding: 2, activation: ReLU.
6. Flattening layer. Flattens the output from the previous layers.
Fully connected layer 1. Input features: 5880 (120 channels ReLU.
7), output features: 84, activation:
Fully connected layer (output layer). Input features: 84, output features: num classes (specified during instantiation).
In all the convolutional layers, no stride = 1 and padding is set to same.
E.2 Stochastic Gradient Langevin Dynamics (SGLD)
Our experiments using SGLD are categorized into 4 groups:
Bayesian CNNs (small and large) on MNIST (Figures 3 - 6 in the main text)
Bayesian CNNs (small and large) on Fashion-MNIST (Appendix E.2.1)
3. Bayesian ResNets (18 and 50) CIFAR-10 (Appendix E.2.2)
Bayesian ResNets (18 and 50) on CIFAR-100 (Appendix E.2.3)
where each group evaluates the effect of underfitting on small model and large model. Note that follow the standard ResNet-18 and ResNet-50, the details of the architectures are omitted. They have around 11 million and 23 million parameters, respectively. We implement with PyTorch (Paszke et al., 2019) and train the model using cyclical lcarning rate SGLD (cSGLD) (Zhang et al., 2019) for 1000 epochs. We set the learning rate to 1e-6 with a momentum term of 0.99. We run cSGLD for 10 trials and collect 10 samples for each trial. Experiments were conducted on NVIDIA A100 GPU, with each trial taking around 30 hours.
26

---

Published in Transactions on Machine Learning Research (08/2024)
E.2.1 Small and Large CNNs via SGLD Fashion- MNISTT
Fashion-MNIST
0 0.0005, y = 1.0, no aug.
0 = 0.0005, y = 2.0, no aug.
0= 1.0, y = 1.0, no aug.
0.8
0.8
0.8
NLL (lower better)
G(p,. D)
NLL (lower better)
NLL (lower the better)
B(pa)
0.6
0.6
0.6
0.4
0.4
0.5
1.0
1.5
2.0
2,5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
A
A
À
(a) Narrow prior and standard softmax
(b) Narrow prior and tempered softmax
(c)
Standard prior and standard softmax
Figure 8: Extended results of Figure 3 using small CNN via SGLD Fashion-MNIST.
0.0005, y 1.0, no aug.
0= 0.0005, 2.0, aug.
0= 1.0, 1.0, aug.
0.8
0.8
NLL (lower the better)
Ō(ps.
NLL (lower the better)
NLL (lower the better)
B(px)
0.6
0.6
0.4
0.4
0.4
0.5
1.0
1.5
2.5
3.0
3.5
A
prior standard
(b)
prior tempored
(c)
Standard prior standard
Figure 9: Extended results of Figure 4 using large CNN via SGLD on Fashion-MNIST.
27

---

Published in Transactions on Machine Learning Research (08/2024)
1.0, y= 1.0, aug.
1.0, y 1.0, reg. aug.
1.0, y= 1.0, rand. shuffle 0=
0.8
NLL (lower better)
Glpa.
NLL (lower the better)
NLL (lower the better)
15
Blpx)
0.8
0.8
G(px)
0.4
0.4
1.0
1.5
3.0
3.5
1.5
2.0 À
2.5
2.0
2.5 3.0
(a) Standard prior and standard softmax
(b)
Random crop horizontal flip
(c) Pixela randonly shuffled
Figure 10: Extended results of Figure using small CNN via SGLD on Fashion-M MNIST.
0= 1.0, y = 1.0, no aug.
0= 1.0, y = 1.0, reg. aug.
1.0, 1.0, rand. shuffle
0.8
0.8
NLL (lower better)
G(p,. D)
NLL (lower the better)
NLL (lower the better)
15
B(pa)
0.6
G(pa)
10
0.4
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
A
A
A
(a) Standard prior and standard softmax
(b) Random crop and horizontal flip
(c) Pixels randomly shuffled
Figure 11: Extended results of Figure using large CNN via SGLD Fashion-MNIST.
28

---

Published in Transactions on Machine Learning Research (08/2024)
E.2.2 ResNet- 18 and ResNet-50 via SGLD CIFAR-10
0 = 0.0005, y = 1.0, no aug.
0
0.0005, y = 2.0, no aug.
0
1.0, y 1.0, no aug.
NLL (lower better)
0.8
G D)
(lover better)
0.8
NLL (lower the better)
0.8
Blpa)
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2,5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
A
À
A
(a) Narrow prior and standard softmax
(b) Narrow prior and tempered softmax
(c)
Standard prior and standard softmax
Figure 12: Extended results of Figure 3 using ResNet-18 via SGLD CIFAR-10.
0.0005, y 1.0, no aug.
0 0.0005, 2.0, aug.
0= 1.0,
1.0, no aug.
(lower the better)
0.8
E(ps.
NLL (lower the better)
NLL (lower the batter)
0.8
0.6
B(px)
0.6
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
3.0
3.5
A
prior standard
(b)
prior tempored
(c) Standard prior standard
Figure 13: Extended results Figure 4 using ResNet-50 via SGLD on CIFAR-10.
29

---

Published in Transactions on Machine Learning Research (08/2024)
o= 1.0, = 1.0, rand. shuffle
1.0, y = 1.0, aug.
1.0, y 1.0, reg. aug.
NLL (lower better)
0.8
NLL (lower the better)
NLL (lower the better)
0.6
0.6
0.4
G D)
0.4
B(p))
0.2
G(ox)
0.0
0.0
1.0
1.5
3.0
3.5
2.0
2.5
A
(a) Standard prior and standard softmax
(b) Random crop and horizontal fii
(c)
Pixels randonly shuffled
Figure 14: Extended results of Figure 5 using ResNet-18 via SGLD CIFAR-10.
1.0, y= 1.0, rand. shuffle
== 1.0, y = 1.0, no aug.
0 = 1.0, y = 1.0. reg. aug.
NLL (lower batter)
0.8
NLL (lower the better)
0.8
NLL (lower the better)
0.4
Glor.D) D)
0.4
B(p
0.2
0.2
G(o))
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
À
À
(a) Standard prior and standard softmax
(b) Random crop and horizontal flip
(c) Pixels randomly shuffled
Figure 15: Extended results of Figure using ResNet-50 via SGLD on CIFAR-10.
30

---

Published in Transactions on Machine Learning Research (08/2024)
E.2.3 ResNet- 18 and ResNet-50 via SGLD CIFAR-100
0 == 0.0005, y= 1.0, no aug.
0 0.0005, y= 2.0, no aug.
0= 1.0, y = 1.0, no aug.
NLL better)
Glp,. D)
NLL (lower the better)
NLL (lower the better)
B(ox)
2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Narrow prior softmax
(b) Narrow prior and tempered softmax
(c)
Standard prior standard softmax
Figure 16: Extended results of Figure 3 using ResNet-18 18 via SGLD on CIFAR-100.
0 0.0005, y 1.0, no aug.
0.0005, = 2.0, no aug.
1.0, y 1.0, no aug.
the better)
G(p,. D)
NLL (lower the better)
NLL (lower the better)
B(px)
2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
X
À
A
(a) Narrow prior and standard softmax
(b) Narrow and tempered softmax
(c)
Standard prior and staudard softmax
Figure
Extended results of Figure 4 using ResNet-50 via SGLD on CIFAR-100.
31

---

Published in Transactions on Machine Learning Research (08/2024)
0 1.0, y = 1.0, no aug.
0= = 1.0, y 1.0, reg. aug.
o= 1.0, = 1.0, rand. shuffle
NLL (lower better)
NILL (lower better)
3
NLL (lower the better)
6
G(pa.
4
B(px)
2
G(px)
0.5
À
(a) Standard prior and standard softmax
(b)
Random crop and horizontal flip
(c)
Pixels randonly shuffled
Figure 18: Extended results of Figure 5 using ResNet-18 via SGLD CIFAR-100.
1.0, == 1.0, aug.
1.0, y 1.0, reg. aug.
1.0, y= 1.0, rand. shuffle
NLL (lower better)
3
NLL (lower the better)
3
NLL (lower the better)
2
2
Glpx 9)
4
B(p))
G(pa)
0.5
1.0
1.5
2.0
25
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
A
A
(a) Standard prior and standard softmax
(b) Random crop and horizontal flip
(c) Pixels randomly shuffled
Figure 19: Extended results of Figure 6 using ResNet-50 via SGLD on CIFAR-100.
32

---

Published in Transactions on Machine Learning Research (08/2024)
E.3 Mean-Field Variational Inference (MFVI)
Experimental Settings: These experiments were run using Tensorflow (Abadi et al., 2015), Tensorflow Probability (Dillon et al., 2017) and Keras (Chollet et al., 2015). By default, we use zero-center Normal distributions, o), as priors with different standard deviations, i.e., o values. For the variational approximation, we use fully factorized Normal distributions, where both the mean and the standard deviation of each of them were the paramcters to be learned by the variational algorithm. Although using an over- simplified family to approximate the true posterior, MFVI also achieves competitive results (Zhang & Nalisnick, 2021) compared SGLD.
The convolutional neural network used for this experiment is a variational implementation of the network described above. This variational model uses total of 1091092 parameters, double the number of parameters of the original model.
We use an Adam optimizer with a default learning rate 0.001, batch size 100, and run during 100 epochs, which in our case, is enough achieve convergence. The Keras global seed was to 15. Other seeds were set, but similar results were obtained. Experiments were performed Google Colab NVIDIA T4 GPU. The computation time was in the order of few hours.
Prior Misspecification, Likelihood Misspecification and the CPE:
We run a similar experiment to the one reported in Figure but using MFVI (Blei et al., 2017) as an approximate inference technique. The results of this experiment are reported Figure 20. The conclusions are completely similar the ones already discussed in Section 5.
CNN [a= y=11
CNN [0 0.01.y=101
CNN [0= [0=1101-==1
0.6
0.6
0.6
D)
Gip!
0.5
0.5
B(o')
0.3
0.5
2.0
25
0.5
1.0
(a) Baseline: "narrow" prior + standard (b) "Narrow" prior + tempered softmax like- like-(c) (c) Standard prior + standard softmax like- softmax likellhood
libood
lihood
Figure 20: CPE can be mitigated by a less misspecifled model (Figure 20b) or imposing a less regularizing prior (Figure 20c). We plot the training loss G(px, D) and the testing loss ((px) with different priors and likelihood models. The parameter a the standard deviation of the isotropic Gaussian prior centered at zero, while the parameter 7
smoothing parameter the logits. All metrics approximated using 10 samples drawn from the MFVI posterior.
Data Augmentation (DA) and the CPE:
As in the previous case, we ran a similar experiment the one reported in Figure but using MFVI (Blei al., 2017) as an approximate inference technique. The results of this experiment are reported in Figure 21. The conclusions very similar to the ones already discussed in Section 6.
33

---

Published in Transactions on Machine Learning Research (08/2024)
0.20
0.20
Gip
Gip".D)
Gip".
B(r")
Big")
B(e')
Gip")
Gip'!
Gip")
0.05
0.05
0.00
0.5
1.0
1.5
2.0
2.5 3.0
3.5
0.00
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
(a) No augmentation
(b) Rand. crop and horis. flip
(c) Pixels randomly shuffled
Figure 21: CPE only occurs with "meaningful" augmentation (Figure 21b). We plot the training loss G(px. D) and the testing losses B(px) and G(px) with different augmentation methods. While Figure 20 shows augmentation, Figure 21b and 21c show standard augmentation and an artificially designed "harmful" augmentation, where the pixels are shuffled randomly. All metrics approximated using 10 samples drawn from the MFVI posterior.
34


Reviews:
[{"review_id": "H6WcaL4P4z", "reviewer": "Reviewer_WSjK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nI must note that I am not intimately familiar with the literature on the CPE, so I cannot evaluate if some of the claimed contributions have appeared elsewhere.\n\nThat said, I think the authors' work represents a reasonable step towards better understanding the CPE. The central premise of the authors' work rests on the identity in Proposition 2, which shows that tempering the likelihood always decreases the training error. While this is certainly not surprising, it is valuable to connect it to underfitting. \n\nSimilarly, while the observation that the tempered posterior can be written as a posterior arising from performing Bayesian inference using a different prior and likelihood function is fairly trivial, it is valuable to draw attention to this fact. I particularly liked the observation that tempering can make the prior depend on the training input locations.\n\nFurthermore, I found most of the paper to be reasonably well-written.\n\nI have also carefully checked the supplementary material for the proofs and found all claims to be correct under reasonable assumptions (see weaknesses for more on this).\n\n# Weaknesses\nOverall, I have three main worries regarding the paper:\n 1. The originality of the theoretical claims\n 2. The strength of the theoretical analysis\n 3. The exposition in Sections 4-6\n\n## Originality\n\nWhile I found the authors' theoretical claims correct, they all follow from direct and fairly elementary calculation; hence, I wonder if these identities have been derived elsewhere in the machine learning or statistical physics literature before. While I don't know a source that contains the exact identities the authors derive in this paper, all of their results follow mutatis mutandis from the derivations in [1]. I have the same worry regarding the contents of section 4.\n\nI am not suggesting that the paper be rejected if these results are not original. However, I think the authors should at least note that very similar results already exist in the literature (although in a different context, perhaps) and reference these.\n\n## Strength of the theoretical analysis\n\nThere are a few points that I think, if clarified, would significantly strengthen the paper:\n - The authors should state under what assumptions their results hold. While it's clear that the results should apply to \"nice enough\" models, it is not clear what \"nice enough\" means. For example, it is not even clear under what assumptions the tempered posterior would exist as the tempered posterior might not be normalizable.\n - The theory's limitations are unclear, and the analysis could be sharper. For example:\nwhat are the limitations of Definition 1? \n   - Are saddle points in lambda possible? \n   - Under what assumptions can we find an optimal lambda? \n   - Is the Bayes loss convex in lambda? \n   - Is it possible that under some circumstances, the Bayes loss monotonically decreases in lambda? \n   - Currently, the authors only look at the behavior of the loss for a fixed set of training data; what happens, e.g., in the limit as we collect more and more data? \n - It is unclear what assumptions about the data-generating process are needed where. For example, I presumed that all arguments went through for non-iid data, but Proposition 5 seems to assume that the data is, in fact, iid. It is particularly important to clarify the context of section 6, which deals with data augmentation. The CPE has been linked to data augmentation, creating correlated training data [2], which the authors do not discuss in Section 6. Could the authors comment on this?\n\n## Sections 4 - 6\nI think the writing in these sections could be improved. They feel longer than needed and contain many references that make them hard to follow.\n\nThe examples in Section 4 take up a lot of space, and it is unclear what purpose they serve; if the authors meant them as a set-up for the experiments in Sections 5 and 6, then they should merge them appropriately instead.\n\nI found Section 5 especially difficult to follow. This is in part because the paragraphs' focus is unclear: they combine some high-level technical discussion with a literature review and interpret the authors' experimental results. To be blunt, I couldn't really figure out the authors' take-home messages in this section.\n\nFinally, in Section 6, besides my worry about the lack of discussion on the iid assumption on the training data, it also needs to be clarified how Sections 6.1 and 6.2 relate to each other. Much of the content in these sections seems almost duplicated, and I'm wondering what insight they are supposed to provide.\n\n# Questions\n - Figs 3 and 4: where is the green curve for $G(p)$?\n - Figs 5 & 6: why are the augmented curves worse than the non-augmented ones?\n\n# References\n\n - [1] Masrani, V., Le, T. A., & Wood, F. (2019). The thermodynamic variational objective. Advances in Neural Information Processing Systems, 32.\n - [2] Bachmann, G., Noci, L., & Hofmann, T. (2022). How tempering fixes data augmentation in bayesian neural networks. arXiv preprint arXiv:2205.13900.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the cold posterior effect (CPE) and argues that underfitting is a necessary condition for the CPE to occur. The authors study how tempering the likelihood changes a parametric model's training and test losses. They show that more tempering always reduces the training error; hence, if the CPE occurs and tempering reduces the test error too, the model must have underfitted the data. In light of this finding, the authors investigate and reinterpret some proposed causes of CPE, such as data augmentation, model misspecification, and errors with the approximate inference procedure.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In order for me to recommend the paper's acceptance, the authors need to significantly improve their writing and address the issues I outlined in the weaknesses section.\n\nBesides this, I have a few more minor requested changes:\n - The notation $[n]$ is undefined; I believe it means the integers from $1$ to $n$.\n - The reconstruction term in eq 3 is missing a $1/n$ term.\n - Throughout the paper, the authors refer to the derivative of many quantities with respect to $\\lambda$ as the \"gradient.\" While this is technically correct, I think it is confusing as $\\lambda$ is always a scalar. Hence, I would prefer if the authors just called it a derivative instead.\n - The authors use $p^\\lambda$ to denote the tempered posterior. I think this needs to be clarified; I recommend using $p_\\lambda$ to avoid confusion with raising to a power.\n - Above prop 3: I have not heard the minimum be called the floor before; I think it would make more sense to stick with the standard terminology.\n - Proof of proposition 7: The fourth-to-last line has an extra $\\alpha$\n - Proof of prop 7: $\\bar{p}$ should be defined in the theorem statement\n - I think it is confusing to call $p^\\lambda$ the **Bayes** posterior (e.g. in insight 1), before the authors explain precisely what they mean in section 4.\n - \"In other words, when such a Bayesian posterior contains more information about the data-generating distribution, it continues to fit the originally provided training data in a similar manner\" - I'm not sure what this sentence means.\n - $d\\theta$ is missing in the appendix in many integrals.\n - Prop 5: inconsistent notation: $q$ in the main text, $p'$ in the appendix; proof in an appendix not quite right because $p'$ is not a valid prior since it doesn't integrate to $1$. Also, it does not prove the same statement since $p'$ doesn't depend on the input location in the appendix.\n - \"However, the standard likelihoods used in deep learning, like softmax or sigmoid, implicitly assume a higher level of aleatoric uncertainty in the data.\" - I'm not sure what this statement means.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YLLwfjYqfj", "reviewer": "Reviewer_hcmZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe main strength of the paper stems from the theoretical analysis, which is later supported by experimental evidence. The theoretical analysis is sound, and the experiments align with the research objectives.\n\nThe overall writing style is appropriate (even though occasionally relaxed) and the experiments are clearly presented. Additionally, a significant effort was made to connect existing, somewhat disjointed explanations and approaches in CPE research, enhancing the work's value to researchers in the field and knowledge in general.\n\nWeaknesses:\n\nThe main weakness of the paper is that, despite the in-depth discussions, it is quite hard to follow the lines of reasoning. I highly suggest emphasizing your hypothesis (as in Wenzel et al., 2020) or the main argument (in bold or italics), providing the discussion (like in sections 4,5 and 6), and then the conclusion within the same section or even within subsections (likelihoods, priors, model misspecification). The use of lists and tables with the results of the discussion is encouraged. Parts of discussions could be trimmed down and put into the Appendix section.\n\nThe same goes for images. For instance, figures 3a and 3b are primarily related to the discussion in section 5 on likelihood misspecification, but figure 3c is relevant to the discussion on priors instead. It was not clear how it related to the discussion at first, but it becomes clearer later in the text.\n\nDefinition 1, despite being very relevant to the analysis, would benefit from mentioning similar ways of defining CPE or similar quantities in the literature. The way it is defined here, and the authors acknowledge that it’s not “[the authors] do not claim this is the best possible formal characterization,” would benefit from referring to existing cases of definitions of such quantities.\n\nIn section 5, “Model size, CPE, and underfitting,” the experiments demonstrate and authors argue that larger CNN models exhibit less CPE. However, Wenzel et al., 2020 in figure 11 showed that larger models tend to possess much stronger CPE compared to either shallower or narrower version of the same architectures. Some discussion on that would be particularly relevant and interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a theoretical and empirical study of the cold posterior effect (CPE). The authors offer both theoretical and practical evidence supporting the main argument of the work: that CPE originates from underfitting. Additionally, the authors review their connection to existing arguments regarding the effects and origins present in the field. For instance, they demonstrate that likelihood and prior misspecifications are relevant to CPE only in cases showing underfitting. Furthermore, the paper argues that data augmentation is a cause of CPE for the same reason—underfitting of the models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A Critical suggestion:\nFigures 3 and 4 have G(p^{\\lambda}), which is not shown anywhere and is clearly a typo/mistake.\n\nMinor Suggestions / Recommendations:\n\nOverall, captions need more descriptions, with more focus on how they relate to discussions in the corresponding sections (especially figures 3, 4, and 5). \n\nSince this paper provides an extensive discussion and comparisons between existing methods and authors' theoretical framework, it would be convenient to have images on the same page as sections discussing them for convenience's sake.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dLP8YyDVLX", "reviewer": "Reviewer_GSTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The central argument (that underfitting may be the main cause of the CPE) is a compelling one which is valuable to the ongoing discussion around the CPE.\n\n2. The paper is relatively rigorous. The authors' attempt to formalise the notion of the CPE allows them to argue their case from a mathematical perspective as well as an empirical one. The latter has been the basis of most previous CPE-related works.\n\n3. The paper is well-structured and, in general, well written. The narrative is coherent throughout.\n\n4. Toy experiments (Fig 2) neatly demonstrate the authors' argument, in addition to the standard image classification experiments (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100).\n\n5. For the most part, the authors do a good job of contextualising their work within the existing CPE-related literature. Particularly in Sections 5 and 6, the relation between their hypothesis and those proposed previously is clear.\n\n\n# Weaknesses\n1. While the majority of relevant work is properly cited, two important references are missing — Adlam et al., 2020 and Zeno et al., 2020 (see full references below). Adlam et al., 2020 show that i) cold posteriors can be caused by priors which overestimate aleatoric uncertainty, ii) for Gaussian processes the tempered posterior corresponds to choosing a different prior, which can be viewed as empirical Bayes. They do not link these observations to underfitting, but clearly this is related work — their Fig 3 illustrates a similar finding to Fig 2 in the authors' work.  \n\n    I believe Zeno et al., 2020 is yet more pertinent. They show that tempered posteriors can be viewed as the Bayes posteriors for different, but valid models, and derive the forms of the modified prior and likelihood. They also discuss how these modified priors encourage confident predictions at the training inputs. As far as I can tell, this negates the novelty of Contribution ii) of this paper and much of Section 4. Again, Zeno et al., 2020 do not view this through the lens of underfitting, so I believe the submitted work is still valuable, but needs to be properly qualified relative to Zeno et al., 2020.\n\n2. The paper recognises that $T$ may be viewed as a model hyperparameter and tuned via Empirical Bayes (EB). I agree, and believe this is a worthwhile research direction. However, it is not clear to me that the method of tuning $T$ proposed in the paper is a form of EB. In particular, EB describes the hyperparameters wrt the _marginal likelihood_ of the _training data_ given the model (after parameters have been marginalised out). I therefore do not see how choosing $T$ based on the performance of a held-out validation set can be viewed as EB. I am happy to be proven wrong on this point.\n\n## Minor Weaknesses\n1. I definitely agree that eqn (4) is not the best possible characterisation of the CPE. In particular it only reflects the _local_ gradient of performance at $T=1$, where one may find e.g. performance is flat wrt $T$ at $T=1$, but perfomance improves as $T$ is reduced from $0.1$ to $0.01$. I believe this would still be intuitively viewed as a CPE.\n\n2. The argument in Section 3 claims that $T=1$ should be optimal when additional examples from the data generating distribution do not improve the model fit on the original training data. This is an interesting result, but the implications should be discussed in more detail.\n    \n    i) Are the following predictions correct? If so I believe they should be discussed in the paper.\n\n        A. For a very flexible model which can perfectly fit the original training set and new examples, we should not expect a CPE as the training fit on the original data will still be perfect even with the new examples. \n\n        B. For a model with little flexibility and a small training set, the additional examples should reduce the impact of noise in the original training data and therefore improve the model fit on the original training data. We should expect a CPE in this case.\n\n        C. For a model with little flexibility but infinite training data, the additional examples should not affect the model fit on the original data. We should not expect a CPE.\n\n    ii) I believe A is demonstrated experimentally by comparing Figs 3 and 4. However, I would like to see B and C verified empirically, i.e. show that the CPE is more pronounced for smaller training sets with the same model.\n\n\n## Clarification\nI would be grateful if the authors could clarify the following point in relation to their central argument. The gist of the argument is:\n\n1. Underfitting is is defined as the scenario in which \"the trained model exhibits (much) higher training and testing losses compared to what is achievable\".\n\n2. Due to how tempering is defined (applied to the likelihood only), increasing $\\lambda$ is guaranteed to reduce the training loss of a BNN.\n\n3. The CPE is defined to be present if the gradient of the test loss wrt $\\lambda$ is negative at $\\lambda=1$.\n\nThus, given 2, if criterion 3 is satisfied then so is 1, i.e. the presence of a CPE implies underfitting. My confusion as a reader is that 2 is a given, and therefore a CPE seems synonymous with underfitting. So for a BNN, is a CPE the same as underfitting _by definition_ or does underfitting _cause_ as CPE? I'm not sure I've posed this question clearly, but would appreciate any insight from the authors on the subject.\n\n\n    \n### References\n(Adlam et al., 2020) Adlam, B., Snoek, J. and Smith, S.L., 2020. Cold posteriors and aleatoric uncertainty. arXiv preprint arXiv:2008.00029.\n\n(Zeno et al., 2020) Zeno, C., Golan, I., Pakman, A. and Soudry, D., 2020, November. Why cold posteriors? on the suboptimal generalization of optimal bayes estimates. In Third Symposium on Advances in Approximate Bayesian Inference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an explanation for the Cold Posterior Effect (CPE). Their theory states that the CPE can be understood in terms of the Bayes posterior ($T=1$) underfitting, which can be alleviated by likelihood tempering ($T<1$). They show that certain misspecification of the prior and/or likelihood can result in such underfitting (at $T=1$) and mitigated by tempering, and discuss in depth how this relates to previous hypotheses of the CPE (Aitchison, 2021; Fortuin et al., 2022; Noci et al., 2021; Kapoor et al., 2022; Wenzel et al., 2020). Further, they demonstrate their theory predicts a more pronounced CPE when training with data augmentation (DA) whose transforms are consistent with invariances in the data. This prediction is consistent with the findings of many previous works which have investigated the relation between DA and the CPE (Izmailov et al., 2021; Nabarro et al., 2022; Kapoor et al., 2022).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical\n1. Include references to Adlam et al., 2020 and Zeno et al., 2020, and revise the claims made in the current submission relative to these previous works.\n\n2. Either justify why the current approach can be viewed as Empicial Bayes, or remove reference to EB.\n\n# Minor\n1. For completeness, include test accuracies in the image classification results.\n2. Experimentally validate Theorem 4 by characterising how the strength of the CPE varies with the size of training set.\n3. Show results for the standard prior + tempered softmax ($\\sigma=1,\\gamma=2$) case in Figs 3 and 4.\n\n## Very Minor\n- $p^\\lambda(\\mathbf{\\theta}|D)$ would be clearer as $p_\\lambda (\\mathbf{\\theta}|D)$ so as not to be confused with \"to the power of $\\lambda$​\"\n- Section 2: \"If it induces no ambiguity\" -> \"as it induces no ambiguity\"\n- Section 3: Should eqn (5) be \"Proposition **1**\" rather than \"Proposition 2\"? I do not see a Proposition 1 anywhere in the paper.\n- Section 3: \"We will then show that the tempered posterior...\" -> \"We will **now** show that the tempered posterior...\"\n- Square brackets for outer expectation over $\\nu$ missing in proof pof proposition 4 (bottom line of pg 17) and mid way down pg 18\n- Section 4 repetition: \"We also show that higher $\\lambda$ values result in likelihood distributions with lower aleatoric uncertainty\". This is already mentioned before eqn (9).\n- Section 4: \"...using the tempered posteriors implies implicitly using the prior...\" -> \"...using the tempered posteriors implies use of the prior...\"\n- Section 6: Do not denote DA transformations with $T$ which has already been defined as temperature.\n- The mathematical results in Section 6 are satisfying and intuitive: there will be a larger CPE if the loss function used for inferring parameters is more strongly correlated with the expected Gibbs loss (or $S(\\theta)$). The authors show that (for some datasets) the loss which incorporates DA is more strongly correlated to $L(\\theta)$ than that which does not, therefore predicting a stronger CPE when using DA. However, I believe the reader would benefit from an intuitive explanation as to how this relates to the overall argument that underfitting causes a CPE.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper should be accepted. However, it has also been suggested that the description of the theoretical assumptions of the work could be improved, so I would recommend the authors take that into account for the camera-ready version.
Decision: Accept (poster)