Excess risk analysis for epistemic uncertainty with application to variational inference
Authors: Futoshi Futami
Tomoharu Iwata
Naonori Ueda
Issei Sato
Masashi Sugiyama
Venue: Submitted to ICLR 2023 (2023)
Keywords: Uncertainty
variational inference
Bayesian inference

Abstract: Bayesian deep learning plays an important role especially for its ability evaluating epistemic uncertainty (EU). Due to computational complexity issues, approximation methods such as variational inference (VI) have been used in practice to obtain posterior distributions and their generalization abilities have been analyzed extensively, for example, by PAC-Bayesian theory; however, little analysis exists on EU, although many numerical experiments have been conducted on it. 
In this study, we analyze the EU of supervised learning in approximate Bayesian inference by focusing on its excess risk. First, we theoretically show the novel relations between generalization error and the widely used EU measurements, such as the variance and mutual information of predictive distribution, and derive their convergence behaviors. Next, we clarify how the objective function of VI regularizes the EU. With this analysis, we propose a new objective function for VI that directly controls the prediction performance and the EU based on the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing VI methods.

Under review as a conference paper at ICLR 2023
EXCESS RISK ANALYSIS FOR EPISTEMIC UNCERTAINTY WITH APPLICATION TO VARIATIONAL INFERENCE
Anonymous authors Paper under double-bling review
ABSTRACT
Bayesian deep learning plays an important role especially for its ability evaluating epistemic uncertainty (EU). Due to computational complexity issues, approxi- mation methods such as variational inference (VI) have been used in practice to obtain posterior distributions and their generalization abilities have been analyzed extensively, for example, by PAC-Bayesian theory; however, little analysis exists on EU, although many numerical experiments have been conducted on it. In this study, we analyze the EU of supervised learning in approximate Bayesian inference by focusing on excess risk. First, we theoretically show the novel relations between generalization error and the widely used EU measurements, such as the variance and mutual information of predictive distribution, and derive their conver- gence behaviors. Next, we clarifyy how the objective function of VI regularizes the EU. With this analysis, we propose a new objective function for VI that directly controls the prediction performance and the EU based the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing VI methods.
INTRODUCTION
As machine learning applications spread, understanding the uncertainty of predictions is becoming more important increase our confidence in machine learning algorithms (Bhatt et al.] (021)) Uncertainty refers to the variability of a prediction caused by missing information. For example, in regression problems, it corresponds to the error bars in predictions; and classification problems, is often expressed as the class posterior probability, entropy, and mutual information (Hüllermeier &
Waegeman 2021 Gawlikowski et al. 2022). There are two types uncertainty (Bhatt et al. 2021): I) Aleatoric uncertainty (AU), which caused by noise in the data itself, and 2) Epistemic uncertainty (EU), which is caused by a lack of training data. In particular, since EU can tell us where in the input space is yet to be learned, integrated with deep learning methods, it is used in such applications as dataset shift (Ovadia et al.] (2019), adversarial data detection (Ye & Zhu 2018), active learning al. 2014), and reinforcement
(Houlsby al. [011] 1), Bayesian optimization ([Heráánezz-Lobaal) learning (Janz et al. 2019).
Mathematically. AU is defined Bayes risk, which expresses the fundamental difficulty of learning problems (Depeweg et al.) 2018 Jain et al.) 2021; Xu 2020). For EU, Bayesian inference is useful because posterior distribution updated from prior distribution can represent a lack of data (Hüllermeier & Waegeman 2021). In practice, measurements like the variance of the posterior predictive distribution, and associated conditional mutual information represented EU in practice 2017 Depeweg et al. 2018).
(Kendall & Gall
In Bayesian inference, since posterior distribution is characterized by the training data and the model using Bayes' formula, its prediction performance and EU are determined automatically. However, due to computational issues, such exact Bayesian inference difficult to implement; we often use approximation methods, such variational inference (VI) (Bishop) 2006), especially for deep Bayesian models. Since the derived posterior distribution also depends on the properties of approximation methods, the prediction performance and EU of deep Bayesian learning automatically guaranteed through Bayes' formula. The prediction performance has been analyzed as generalization error, for example, by PAC-Bayesian theory (Alquier 2021). Since EU is also essential
no longer

---

Under review as a conference paper at ICLR 2023
in practice, we must obtain theoretical guarantee of the algorithm- and the sample- dependent non-asymptotic theory for EU, similarly to generalization error analysis.
Unfortunately, study has been limited in that direction. Traditional EU analysis has focused on the properties of the exact Bayesian posterior and predictive distributions (Fiedler et al. 202T: Lederer et al. 2019) well as large sample behaviors (Clarke & Barron 1990). Since Bayesian deep learning uses approximate posterior distributions, we cannot apply such traditional EU analysis based on Bayes' formula to Bayesian deep learning. The asymptotic theory of sufficiently large sample may overlook an important property of the EU that is due to the lack of training data.
Recently, analysis of EU focusing on loss functions was proposed for supervised learning (Xu & Raginsky 2020; Jain et al. 2021). EU was defined as the excess risk obtained by subtracting the Bayes risk corresponding to the AU from the total risk. Thus, excess risk implies the loss due to insufficient data when the model is well specified. Although this approach successfully defines EU with loss functions, the following limitation still remains. [Xu & Raginsky (2020) assume that the data generating mechanism is already known and that we can precisely evaluate Bayesian posterior and predictive distribution, correct model is not necessarily a realistic assumption, and the assumption about an exact Bayesian posterior hampers understanding EU in approximation methods.
To address these limitations, it appears reasonable to analyze excess risk under a similar setting as Bay-saa theory and apply it to EU of approximate Bayesian inference. However, shown in Sec. 2 analyzing excess risk in such a way leads to impractical theoretical results, and the relations between excess risk and the widely used EU measurements remain unclear. This greatly complicates EU analysis. Because of this difficulty, to the best of our knowledge, no research exists on excess risk for EU for approximation methods.
In this paper, we propose a new theoretical analysis for EU that addresses the above limitations of these existing settings. Our contributions are the followings:
We show non-asymptotic analysis for widely used EU measurements (Theorems 2 and 3) We propose computing the Bayesian excess risk (BER) (Eq, (9) and show that this excess risk equals to widely used EU mcasurements. Then we theoretically show the convergence behavior of BER using PAC-Bayesian theory (Eqs. (13) and (8)).
Based theoretical analysis, give a new interpretation of the cxisting VI that clarifies how the EU is regularized (Eqs. (19) and (20). Then we propose a novel algorithm that directly controls the prediction and the EU estimation performance simultaneously based on PAC-Bar BaC-Bayesian theory (Eq. (2I)). Numerical experiments suggest that our algorithm significantly improves EU evaluation over the existing VI.
2 BACKGROUND OF PAC- THEORY AND EPISTEMIC UNCERTAINTY
Here we introduce preliminaries. Such capital letters as X represent random variables, and such lowercase letters as I represent deterministic values. All the notations are summarized in Appendix A In Appendix B we show summary of the settings.
2.1 PAC-BAYESIAN THEORY
We consider a supervised setting and denote input-output pairs by 2 (x,x = X x y. E
We assume that all the data are i.i.d. from some unknown data-generating distribution 2(2)
(YIX)r(XX) Learners can access N training data, z == (2,, ZN) with 2, == (X,,, Yn), which are generated by N distribution as x = x)) as [z| for simplicity. We introduce loss function y x A + R where A is an action space. We express loss of action a E and target variable y is written as I(y, a). We introduce model fo X' - A. parameterized by 0 E θ C R.d. When we put prior p(0) over 6, the PAC-Bayesian theory (Alquier 2021 Germain et al. 2016 guarantees the prediction performance by focusing the average of the loss with respect to posterior distribution (* E Q. Q is family of distributions and q()) Z N) ) is not restricted to Bayesian posterior distribution. In this work consider the log loss and the squared loss. For the log loss, we consider model p2y f). and the loss is given as (y,, p(yl 0)) = In inp(yle x.. 8). where A is probability distributions. For the squared loss, we use model fo(x) and I(y, fo(x)) y - [ where y A R.
v(Z)N. We express v(Z) as I(Z²). We express conditional
2

---

Under review as a conference paper at ICLR 2023
PAC- BAC-Bayesian theory provides
guarantee for the generalization of test error R (' ) = BuizN) RoCO (2))) (x()) and training error Eu (Z
EA(Z E (( En= (( (X,)). A typical PAC-Bayesian error bound takes form (Y ) I Buzzarr (Z*) ) Gen' (Z*). Gen't (Z*) is called generalization error. Under suitable assumptions (Alquier 2021). ) Gen (Z*) is upper-bounded by (1//Na) for a E (1/2, 1]. In many cases, it depends the complexity of the posterior distribution, such as Kullback-Leibler (KL) divergence ((l(((/())(((())) When Gen (Z
(((g(((/Z>(((())) + c, where À and are positive constants, given training data Z = zN we get posterior distribution for the prediction by
(L(((0 (()))
(01lz*) arg min r(z") ((0
A
When the log loss and N is used, this minimization is closely related to variational inference (VI) in Bayesian inference. See (Germain et al. 2016) for details.
Under additional moderate assumptions, using ((01z>) for the test error, we can derive the following excess risk (ER) bound from the PAC-Bayesiaa generalization bound (Alquier 2021
In N R (Y 2 R'(,,X, 0*) S S1
ER' (Y Z. 8*) == R
Na
where (Y pends only on the problem. Since we aim to analyze EU, we do not further discuss the details of the PAC-Bayesian bound. See Appendix C.2 for the explicit conditions of this bound.
0*) = E2( (Y, fo- (X)) and 0" = argmingEer ( fo(X)). Constant Ci de-
Although PAC-Bayesian theory focuses on the average test predictive distribution for predictions in Bayesian inference. Thus we define Prediction Risk (PR):
posterior distribution,
PR' Z M) == Eu(zz (X, fo()))
(3)
When the loss is log loss, PR 2 (yll,, z²i := ( p(ylz,, 0) is the approximate predictive distribution. Thus, ( z corresponds to the log loss of the predictive distribution, which is commonly used in the analysis of Bayesian inference (Watanabe 2009 2018).
(ZN B Ez)z) log p" (Y|X, ZN) where
2.2 EPISTEMIC UNCERTAINTY MEASUREMENTS
Here, we introduce widely used EU measurements in approximate Bayesian inference. For the log loss, conditioned on (X, ZN)= (z. z x²). the approximate mutual information has been widely used for uncertainty estimation (Depeweg al. 2018):
(0; Y [r, x H pp (Y x. z E v (p(Y (1|((1),)))) 0)].
where (Y =
(y -Bp tive distribution and (0)=2)))) H [p((La lx, 0) is the conditional entropy. (0; Y|z, z²x has been used in Bayesian experimental design (Foster et al. 2019) and reinforcement learning (Depeweg et al. 2018).
log (Y(()(() x)) is the entropy of the approximate predic-
Note that by taking the expectation, we have I, (0: Y X, ZN Bu(x=x) T =N) (0: z
In the case of squared loss, the variance of the model often used for EU. This is a common practice in VI. Monte Carlo (MC) dropout (Kendall & Gal 2017). and deep ensemble methods
llakshminarayanan et al. 2017). Conditioned on (X, 2n) c z ), written as
Vargja fo(x) (0)22 (fo(z) E fo
(5)
Although Eqs.
and 5 are widely used in application, there have been limited theoretical study for them discussed in Sec. 1
2.3 EXCESS RISK ANALYSIS AND EPISTEMIC UNCERTAINTY
Recently, the analysis of EU based excess risk proposed (u & Raginsky] 2020). The key idea of this analysis is to assume that our statistical model p |z, 8) is correct and address the average performance of this model by assuming prior distribution over 0 with distribution p(0).
3

---

Under review as a conference paper at ICLR 2023
Specifically, the joint distribution of the training data, the test data, and parameter of the model is given PB PB(Z*, 2,0)
== ((0)II () (),(((X)))()XX))))) (X)( Under this setting, the goal of learning is to infer decision rule 1b z X + A that minimizes expected loss (pe( 2,,9 [[Y, (xX, ))] They refer to this setting as Bayesian learning since we marginalize out parameter 9. With this notation, Xu & Raginsky (2020) defined minimum excess risk as MER' (Y)
inf
Ere(2* 2,0 (XX, z }>] X
inf
Bpg(2" 22,,,, 6(0, x))], (6)
N- XX-
where the first term is the minimum achievable risk using the training data and the second term is the Bayes risk since uses learning rule Φ θ X + A. which takes true parameter 0 instead of the training data. Thus, the second term is the aleatroic uncertainty (AU) since it expresses the task's 's fundamental difficulty. Then MER can be regarded as the EU since is the difference between the total risk and the AU (Xu & Raginsky 2020; Hafez-Kolaan Kolahi et al.. 2021).
For the log loss, the first term is H [p > and the second term is pp [p XX 0)] Thus, MER log (Y X, ) T(0; Y|x, Z which is the conditional mutual information. Other than the log loss, the loss function satisfies the o² sub- sub-Gaussian property conditioned on (X, Z- = (a, N. MER' (Y ) I I/20II(B) | Z holds (Xu & Raginsky 2020). In many practical settings X
T(0: X, Z is upper-bounded by (In N/N). Thus, EU converges with O(nn N/N) under this settings. See Appendix for more details.
Although this analysis successfully defined EU with rigorous theoretical analysis, the assumptions are clearly impractical since assume that the correct model, exact Bayesian posterior, and predictive distributions are available. To extend this analysis into approximate Bayesian inference, is tempting to combine the theory of MER with PAC-B -ay-Baaesian theory where the data are generated i.i.d from v(2 (Z). For that extension, here we introduce the Prediction Excess Risk (PER) using Eq. (3):
PER (Y)
PR (Y - inf Bu(z) ((Y e(X)),
where the second term corresponds to the Bayes risk. Although we introduced this definition inspired by Eq. (6. it is impractical for evaluating EU. In practice, we are interested in evaluating EU using only imput æ. as shown in Eqs. (4) and (5). However, we cannot use Eq. (7) for that purpose since we do not know both v (x) and the Bayes risk in the second term. Despite less practical definition, as shown in Sec,, PER plays a fundamental role in understanding the algorithm-dependent behavior of the widely used EU measurements in Eqs. and (5)
3 ANALYSIS OF EPISTEMIC UNCERTAINTY BASED ON EXCESS RISK
In this section, we develop theories for analyzing the widely used EU measurements introduced in Sec 2.2 We focus on the following questions. (Q1) The convergence behaviors of those measurements are not apparent. As the number of training data points increases, we expect these measurements to converge to zero. (Q2) The relationship between these measurements and the generalization error is unclear. Since these measurements depend on the training data and the algorithm, we expect some meaningful relationships must exist, All the proofs in this section are shown Appendix D
3.1
RELATION BETWEEN EPISTEMIC UNCERTAINTY AND BAYESIAN EXCESS RISK
First, to connect the practical EU evaluation methods in Sec 2.2 with the excess risk analysis in we introduce the approximate joint distribution of test data, training data, and parameters:
Sec 2.3
v(Z^ ) (z) = (0, N Z) = v(Z N' (0 Z ())(X (Y)))))))) #).
(8)
When the log loss is used, we employ model for p(y) ax, 0) Eq. ( When the squared loss is used, we assume Gaussian distribution p(y|z, /z, 0) = N (y) fo( 2 for some v E IR.
When a model is well specified, that is, u(yl=r) = p(yl 0*) holds for some 0* E E we expect that the predictive distribution converges to p(ylar, 0²) and the approximation of Eq. (8) becomes accurate
as N increases. We discuss the quality of this approximation in See 3.2 Under this setting, we define Bayesian Excess risk (BER):
BERR (Y X, N = BPR (Y|X,
inf E Sy0ez,,2)l( +OxX-A
(9)
4

---

Under review as a conference paper at ICLR 2023
where BPR is the Bayesian Prediction Risk defined as
BPR z^)= : 22,((, Betor aNy (X))
(10)
and the second term is the Bayes risk under the approximate joint distribution of Eq. (8) Note that
(Y ZM ) is always larger than O; see Appendix D.2 for details. We also show the formal
definitions of BER and BPR conditioned on (X,, (2) I in Appendix D.1
BER and BPR are defined, motivated by PER, PR, and MER. The difference is the mechanism of through which the test data are generated. In BER, assume that model p(y|ær, 0) is correct, and the parameters follow the approximate posterior distribution q( 2 N. Thus, the data-generating mechanism resembles the setting in Sec 2.3 Therefore, similar MER, BER implies the loss due to insufficient data under the assumption that our current model q( z ppyll, 0) is correct.
The next theorem elaborate this intuition and connects BER widely used EU measurements:
Theorem 1. Conditioned
2,, express BER (Y Jr x for the log loss and BER (2) (Ylz, [xx z for the squared loss. Under the definition of Eq. (9) we have
BER (Y|2, (2,zz"z"> = (0; )x
BER (Y [x, x²) = Vargzzx fo(z).
(11)
Thus, by studying BER, we can analyze the widely used EU measurements. We point out that BPR (Y|r,, [r, xa H
(Y |ær x²)]. and the Bayes risk is given as E (x((,, 0)]- Remark 1. BER captures EU when our model is well specified. On the other hand, PER (Eq. 7
represents the prediction performance, which has the relation to the quality of approximation of Eq. (8) under the given loss function. This intuition leads to our new VI in SecB.3
3.2 ANALYSIS OF EXCESS RISK BASED ON PAC-BAYESIAN THEORY
Based on the definitions introduced in Sec. 3.1 here we develop a novel relation between BER and the generalization. First, we show the results of the squared loss. For simplicity, assume y R. See Appendix D.6 for 3 Rd.
Theorem 2. Conditioned on (z, ),, assume that a regression function is well specified, that is. Ey x) ||x] = fo (x:) holds. Then we have
PER (2) (x|x, z²) (ER (Y x² (2) 20,0"" S (Y
(12)
Furthermore assume that the PAC- bound Eq. (2) holds, and then have
C In
PFR (YXX + BER (2) 2 E X NN 07)
(13)
No
Proof. We use the following relation about the Jensen gap and BER:
Lemma 1. For any (x, 1) and any posterior distribution conditioned on N N,, we have
ly - E BN + Vargz fo(z) = EE900/2>1y -aieawwyy--far fo(2) folyll/--o)(
(14)
From this lemma, the theorem follows directly.
Remark 2. When we use a flexible model, such as deep neural network for fe(x), assumption Ev(Y 2) [Y|2] = fe- (x)) holds even when v(Ylz) # p(y|,, 0)), which means we misspecify the noise function in regression tasks.
From Eq. 12 Vare " fo(=;) clearly is lower bound of excess risk and test error, consistent with the well-known result that the variance of the predictor often underestimates EU (Labshminnrryyaaan et al. 2017). From Eq. (13). BER (Y|X, z converges to with the same order as the PAC-Bayesiae
bound. Finally, we remark that from Lemma]] ! we have
(15) This indicates that the test error is decomposed into PR and BER. As pointed out in Remark T BER EU under the approximation of Eq. (8). and PER represents the quality of that approximation,
(X|x, N (2) (Y XX ZN +
Eq. [55) suggests that the test error simultaneously regularizes those BER and PER.
Next we show the log loss result. Our analysis requires additional assumption about model p(y|a, 2z 0).
We define log density ratio L(y, xr, 0 0*) In p(y|2, 0) + In p(y|x, 0*).
5

---

Under review as a conference paper at ICLR 2023
Assumption 1. Conditioned on (x.0.2²), there exists convex function h(p) for [0, b) such that cumulant functionn L(y, x, 8, 8*) is upper-bounded by h(p).ie. the following inequality holds:
In Ep pY -.p0))"
= h(p)
(16)
For example, if h(pp = (2, 0)/2 and b= oc, this assumption resembles the o² Sau-Gaussian property given (z,0, 8,2%),,2.. When considering Gaussian likelihood p(y|æ, 0) NN fx(xx) v²), we have h(p) 2cth fo(x) - (x= Thus, o² (x,, 0) depends
and 0, and we refer this o² (2 as a Lub-Gaussian property. Other than the Gaussian likelihood, when the log loss is bounded, it satisfies the sub-Gaussian property. In this paper, we focus on this sub- Sab-Gassiin setting for Assumption Too clarify the presentation. We show an example of the logistic regression in Appendix D.11
When the model is well specified, that is, v(yla) = p(y|z, (*) holds and o² (x, 0) sub- Gaussian property is satisfied for Lyy, 2, 0,0*), as discussed above. Assume that ye (2 j (z, (2
Theorem 3.
of < 00. Conditioned on (zz, zN) ). we have
PER log (Y 2 BY
2 (*)).
(17)
Moreover, assume PAC-Bayesian bound Eq. then we have
and Bu )o(0 (xx) og hold, and
C IN
PER lox ( + BER oo (Y|X, E
20 D ZN, 8*)
(18)
No
we apply change-of-measile inequalities (Ohnishi Honorio 2021 to control the approximation error of Eq. (8).
Proof. Using Assumption] T
Remark 3. When we use a generalized linear model, the assumption of a well-specified model is relaxed so that (x))) Y ai] is well specified, similar Remark 2 See Appendix D.9 for details.
From Eq.
17
(0 x is a lower bound of the excess risk and test error. ( L ZN) converges in the order of O//IN/NN=) if we can (xx "
From Eq.
18
upper-bound EZ 20
of < co. For the Gaussian likelihood, we have s 2F (Y| zN 50) 2 No == o² See Appendix D.8 for a detail.
Eu "110)))))))))"))))""))"
In a similar way, we can derive the convergence rate of the entropy of the predictive distribution, which shows H (Y )>)] = XY 0²) + O( N/ See Appendix D.10 for formal statement. We show similar results for Theorem 3 under sub-exponential property in Appendix E
In summary, we obtained the convergence of widely used EU measurements and the entropy in the approximate Bayesian inference for the first time. They converges faster than excess risks. Moreover we obtained two messages from Theorems 2 and First, the widely used EU measurements are the lower bounds of the test error and excess risks. This consistent with the experimental fact that these EU measurements often underestimate EU. Second, the sum of PER and BER is upper-bounded by excess risk (test error). Thus, when minimizing the test error, we also simultaneously minimize PER and BER. This interpretation extends the intuition of Remark[Tand leads to new VI in Sec. 3.3
3.3 NOVEL EU REGULARIZATION METHOD FOR VARIATIONAL INFERENCE
As seen in Sec. 3.2 minimizing the test error leads to minimizing BER and PER. In this section, we discuss this relation using the objective function of VI. As explicit example, consider regression problem using N Nyylf((z) (x)x), v²). Then from Lemma 1 the loss function of the standard VI (eliminating (L(((022>))
can be written:
Y E fx = + Vara NN fo())
In 2
-Ev(z) Eg( In N(ylff(()) EV(Z)
+
2v²
2
PR (Y X + BER (2) (Y
In 2
(19)
20
2
Eq. 19 implies that the standard VI tries to fit the mean of the predictive distribution to target variable y with the regularization term about the variance of the predictor. These terms corresponds

---

Under review as a conference paper at ICLR 2023
2 and BER ( (Y],, Note that there is relation BER log (Y Varguy fo(r)/e² for the Gaussian likelihood. This interpretation is consistent with Remark
to PR ( (Y|X,
1
It has been numerically reported that the standard VI often underestimates EU. Alternative objective functions have been proposed to address this issue. For example, the entropic loss defined as Ent,, (y. 2) == -1 In (e (allfooo) for a > 0, which is used in the a-divergence dropout (a-DO) (Li & Gal 2017) and the second order PAC- PAC-Bayesian methods (2"d-PAC) Masegosa 2020 Futami et al. 2021). p. can capture EU better than the standard VI. Note that when a and the log
loss is used, the entropic risk corresponds to the log loss using the predictive distribution. For the
Gaussian likelihood, we can upper-bound the entropic risk:
Y y( (M (X)) Varo fo(X) BER (Y|X, x²) ) +I 2mv?, (20)
BZZ Ent Eo a=1 (e,,,,,Y,,, X) S FZ(z)
y2
where we used Eq. 17). See Appendix 1.12 for the derivation. Compared to Eq. (19) the entropic risk implicitly introduces smaller regularization term for BER. This explains why a-DO and 2nd_ PAC showed larger EU than the standard VI. We show a similar result for the entropic risk of the general log loss other than the Gaussian likelihood in Appendix D.12
From these relations, balancing BER and PR appropriately leads to solution that better evaluates the EU. Motivated by the decomposition in Eqs. (19) and (20), we directly control the prediction performance and the Bayesian excess risk for the Gaussian likelihood:
Vi- E E0 fo(=;))
Varaz* fo(as)
2TO2
(BER(A)=
KT(g(0/>*) (21)
2xa
2
where < x
SASI S is the coefficient of the BER regularizer. (L((())
a regularization term
motivated by PPC-Bayesiag theory. We select À by cross-validation and it should be smaller than 1 since À = corresponds to the standard VI from Eq. [9) and the standard VI often underestimates the EU. We call Eq. (2I) the regularized Bayesian Excess Risk VI (rBER) and show the PAC-Bayesian generalization guarantee for our rBER in Appendix In Sec. 5.2
we numerically evaluated this
objective function.
rBER can also be seen as an extension of the standard VI. In the standard VI, the test loss is lower- bounded by the sum of the PR and BER with equal weights (Eq. (15)). rBER has the flexible weights between PR and BER. See Appendix a detailed comparison.
4 RELATION TO EXISTING WORK
The existing theoretical analysis of uncertainty focused on the calibration performance and clarified when a model over- and underestimates uncertainty (Nixon et al., Bai et al.) [02T] Naeini et al. 2015: Guo et al., 2017). We did not focus on analyzing the calibration property this work because considering only the EU is not sufficient to deal with the calibration. We need to study both the AU and EU simultaneously. This is the limitation of our work. Other than calibration, the analysis of Gaussian processes (GP) has been gaining attention since GP's s posterior predictive distribution can be expressed analytically (Fiedler et al.. 2021 Lederer al. 2019). Some research focused on the distance or geometry between the test and training data points to derive HU (Liu et al.] 2020 Tian al. 20221) Other approaches connect the randomness of the posterior distribution to predictions by the delta method (Nilsen et al. 2022). Differently, the information-theerrtie approach (Xu & Raginsky 2020) focused on the loss function of the problem and defined the excess risk as the EU. Loss function- analysis was proposed in the deterministic learning algorithm (Jan et al. 202Tb. Our theory, which can be regarded as an extension of the information-theont & Raginsky 2020) to approximate Bayesian inference, derived the convergence properties of the variance and the entropy of the posterior predictive distributions.
approach (Xu
Although the excess risk bound in Eq. ( has been discussed by PAC-Bayesian theory (Alquier 202I), its relation to the EU has not been investigated. The relationship between BAC-Bayesian theory and Bayesian inference has been investigated in terms of marginal likelihood (Germain et al.) 2016; Rothfuss et al. (021). Our work established new relationships that connect the uncertainty of the Bayesian predictive distribution and the PAC-Bayesian generalization bound. The information- theoretic approach (Xu & Raginsky 2020) clarified that EU can be expressed by conditional mutual information. This relation was extended to meta-learning (Jose al. 2021). However, the researchers
7

---

Under review as a conference paper at ICLR 2023
BBB
Dropout
Deep ensemble
0.125
0.150
0.030
X 24
PERPXY] 2)
0.125
PER [x 291
0.025
PER 2
WoNF5r (X,2%)
Beaeehy | 2
0.100
0.020
0.075
0.075
0.015
0.050
0.050
0.010
0.025
0.025
0.005
0.000
0.000
0.000
0
5000
10000
0
5000
10000
0
5000
10000
Figure I: Rresult of toy data experiments: N represents number of training data points, and vertical line is value of each excess risk.
Dropout
Deep ensemble
(Y"X,
0.00
DU03
W IX, 2
0.005
Dropost
PR (X. 20%)
PAPMMXX 2
0.004
Deeu ersemble
Rile
0.00
RRR"" 2%))
0.02
BER
0.003
0.002
0,001
0.001
0.00
0.000
20000
10000
20000
Figure 2: Real data experiments of depth estimation: Vertical line is the value of each risk.
assumed that correct models and exact posterior distributions are available. Our proposed analysis relaxes these assumptions.
NUMERICAL EXPERIMENTS
In this section, we numerically confirm the theoretical findings in Sec. 3 and our proposed rBER in Eq. (21). We show the detailed experimental settings and additional results in Appendix G
5.1 NUMERICAL. EVALUATION OF THEOREM 2
We numerically confirm the statement of Theorem First, we consider toy data experiments where the true model y 0.5æ* + E, e ( x N 1). We consider Bayesian neural network (BNN) for fo(z) as a 4 layer neural network model with ReLU activation. We approximate the posterior distribution of the parameters of the neural network by Bayes by back-
propagation (HBrnn(Eenndde & Adams) 2015), dropout (Kendall & Gall 2017). and
deep ensemble [akshminarayaaan et al. 2017]. We evaluate PER (Y Z). BER
22 (Y X, Z)
(:= (x) Vargzz (X)), and (Y XX, Z) (test error). The results are shown in Fig, I Our
numerical results satisfy Eq. (13 in Theorem that is, (2E(() (Y|X, Z) and BER (2) (Y X Z) are
upper-bounded by R! (YIX, Z) and converge to zero as the number of samples increases. We
calculated the Spearman Rank Correlation (SRC) among PFR ( (Y|X, Z), BER 2). and R(2) (Y|X, Z) and showed at least 0.97 suggesting high correlation relation between them.
Jsiig real-world dataset. Following the setting of existing work (Amini et al. 2020), we trained U-Net style network (Ronneberger et al., 2015)) with the data of the NYU Depth v2 dataset (Silberman et al., 2012), which consists of RGB-to-depth. We applied dropout and deep ensemble methods. Since we cannot evaluate (ER (Y| Z). we instead evaluated Eq. (15 which only requires (2((2) (YIX, Z), BER (2) (Y|X, 2), and (Y))) Z). The result is shown in Fig. We found that PR (2) (Y]X, Z). (ER (Y|X, Z) are upper-bounded by ((()( (Y|X, Z) for
Next, we confirm Theorem 2
real dataset experiments. We calculated the SRC among PR (Y Zy) BER (2) (Y|X, Z), and
8

---

Under review as a conference paper at ICLR 2023
Table 1: Benchmark results on test RMSE, PICP, and MPIW.
Dataset
Aug.
Tost RMSE
Aug. Test PICP and MPTW in parenhesis
1-SVG
VAR
(BER(D)
(BE8/0.06)
F-SVG
VAR
(BE320)
(BE
Cuscress
4.33:30.5
4,30:0.7
4,470066
4.4530777
0.220003
(0.13::000))
0.57:0.04 (0.16:0.00)
09P:002 (0 00500:044)
0.95:0.02
0..25-0002))
Rosten
2.54200.00
2.57=:000
2.5360.50
2
06540077
(0 10:000)
0.76:0.05 (1, 14:0001)
097::001 (o (1:5).04))
002-0000::2::0000
061:00.44
06140.04
0642004
063:0002
0.79:0003
(0.22:0.5))
0.8350.02 -0.00:0.000
0.9::000 (1 (160)0000)
0.55:0.33
(0:32-0.15)
Prwser
1.8800.14
175:20111
16620.15
1692011
04420001
(107220000)
0.82:0.00 (h: 15:30.00)
09:20001 (0 (0:0000013
0:06:00011 () (0:7-0001
06666028
0600::25
0.75:041
0.78:045
0.22::044
(0.02:000)
0:90:0.00 (0.06:0.01)
0.96:
1060.0))
0:94:0.04 (0.06=0.01)
Prozin
A9819.54
39720005
3.320010
3852005
055:001 (0 (0244:000))
0.8320.00 (o S8.0.01
10 10000000 0420001)
0.900..01 (0:30-00000)
Table 2: Cumulative regret relative to that of the uniform sampling.
Dataset
MAP
PACE
f-SVGD
VAR
rBER(0)
rBER(0.01)
rBER(0.05)
Mushroom
0.129:4:0.09
0.037=0.012
0.043::0.09
0.029::0.100
0.075=00005
0.024:100009
0.021=:0.004
Financial
0.791=:0.219
0.189=0.025
0.15440.017
0.15540.024
0.351=0.030
0.075:1:0024
0.075=+00311
Statlog
0.675 +:0.877
0.032:00.03
0.010::0.000
0.006::00000
0.145=0.223
0.005:::.001
0.005:00.000
CoverType
0.610:10.051
0.396=00.006
0.372:00.007
0.291:10.004
0.610=0.051
0.351::0.003
0.290::00002
R (Y|X, Z) and showed at least 0.98, suggesting high correlation relation between them. We also evaluated the convergence behaviors of BER and show the result on the right in Fig. 2
BER
converges with ((1/N1/2), which is consistent with Eq. 13
5.2 REAL DATA EXPERIMENTS OF REGULARIZED BAYESIAN EXCESS RISK VI
We numerically compared the prediction and EU evaluation performances of our proposed method shown in Eq. (2T) in regression and contextual bandit tasks. Motivated by the success of the entropic risk in particle VI (PVI) (Masegosa, 2020 Futami et al., 202I). which approximates the posterior distribution by the ensemble of models, we also applied our rBER to the PVI setting. Thus, the
posterior distribution is expressed as q(0)= == N/ Σi= 50, (0), where 5g, (0) is the Dirac distribution that has a mass 0,. See Appendix Gor details about PVI. We refer (BER(0) when X 0 in
Eq. (11) We compared our method with the existing PVI methods, f-SVGD (Wang et al. 2019).
PACE E MMasegosa 2020), and VAR Futami et al. 2021).
We used the UCI dataset (Dheeru & Karra Taniskidou 2017) for regression tasks. The model is single-layer network with ReLU activation, and we used 20 ensembles. The results of 20 repetitions shown in T We evaluated the fitting performance by RMSE and the uncertainty estimation performance by the prediction interval coverage probability (PICP), which shows the number of test observations inside the estimated prediction interval where the interval was to 0.95. PICP is best when it is close to 0.95. We evaluated the mean prediction interval width (MPIW), which shows an average width of prediction interval. A smaller MPIW is a better uncertainty estimate when PICP is near the best. Due to space limitations, the results of PACE and the other As and the negative log-likelihood are shown in Appendix G We found the existing PVIs show small PICP and MPIW, indicating that the existing methods underestimate the uncertainty. rBER(() shows large PICP and MPIW since the Bayesian excess risk is not regularized. rBER(0005) shows moderate MPIW with better PICP and almost identical prediction performance in RMSE. Thus, rBER successfully controlled the prediction and the uncertainty evaluation performances.
Next we evaluated the rBER using contextual bandit problems (Riquelme et al. 2018). We need to balance the trade-off between exploitation and exploration to achieve small cumulative regret. For that purpose, our algorithms must appropriately control the prediction and uncertainty evaluation performance. We used the Thompson sampling algorithm with BNN and two hidden layers. We used 20 ensembles for approximating the posterior distribution. The results of 10 repetitions are shown in Table 2 Our approach outperformed other methods, which means proposed method showed better prediction and uncertainty control than the existing methods.
6 CONCLUSION
We theoretically and numerically analyzed the epistemic uncertainty of approximate inference. We clarified the novel relations among excess risk, epistemic uncertainty, and generalization error. We then showed the convergence rate of the widely used uncertainty measures for the first time. Motivated by theoretical analysis, we proposed a novel variational inference (VI) and applied it to the particle VI. In future work, it would be interesting to explore the relation between BER and evidential learning.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2 10. 1216,
2021.
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of gibbs posteriors. The Journal of Machine Learning Research, 17(1):8374-8414 2016.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Don't just blame wee-ppraametrization for over-confidence: Theoretical analysis of calibration in binary classification. In International Conference on Machine Learning, pp. 566-576. PMLR, 2021.
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as form of transparency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 401-413, 2021.
Christopher M Bishop. Pattern recognition and machine learning. springer. 2006.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Lawrence D Brown and Roger Purves. Measurable selections of extrema. The annals of statistics, pp. 902-912, 1973.
B.S. Clarke and A.R. Barron. information--theordd asymptotics of bayes methods. IEEE Transactions on Information Theory, 36(3):453 471, 1990. doi: 109/188788
Stefan Depeweg, Joge-Miguel Hernandez-Lobato, Finale Doshi-Veled. and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1184-1193, PMLR, 2018
Dua Dheeru and Efi Karra Taniskidou. 7/archive. ics. uci wcw.mdummillmlm....cc.d
UCI machine learning repository, 2017. URL http:
Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Practical and rigorous uncertainty bounds for gaussian process regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7439-7447, 2021.
Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational bayesian optimal experimental design. Advances in Neural Information Processing Systems, 32, 2019.
Futoshi Futami, Tomoharu Iwata, Issei Sato, Masashi Sugiyama, et al. Loss function based second- order jensen inequality and application to particle variational inference. Advances in Neural Information Processing Systems, 34, 2021.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. A survey of uncertainty in deep neural networks, 2022.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-bayesian the- ory meets bayesian inference. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 1892, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.
Hassan Kolahi,, Behrad Moniri, Shohrch Kasaei, and Mahdieh Soleymani Baghshah. Rate- distortion analysis minimum excess risk in bayesian learning. International Conference on Machine Learning, pp. 3998-9900 PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
David Haussler and Manfred Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics, 25(6):2451-244) 1997.
Rianne Heide, Alisa Kirichenko, Peter Grunwald, and Nishant Mehta. Safe-bayesian generalized linear regression. International Conference on Artificial Intelligence and Statistics, pp. 2623- 2633. PMLR, 2020.
José Miguel Hemánddez and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning, pp. 1861-1866 1869. PMLR, 2015.
José Miguel Hernández- Lobato, Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances neural information processing systems, 27, 2014.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457-55) 5065 2021.
Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint arXiv:2102.08501, 2021.
David Janz, Jiri Hron, Przemyslaw Mazur, Katja Hofmann, José Miguel Hernández-Lobato. and Sebastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning. Advances in Neural Information Processing Systems, 32, 2019.
Sharu Theresa Jose, Sangwoo Park, and Osvaldo Simeone. Information theoretic analysis of epistemic uncertainty in bayesian meta-learning. arXiv preprint arXiv:2106.00233 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Armin Lederer, Jonas Umlauft, and Sandra Hirche. Posterior variance analysis of gaussian processes with application to average learning curves, 2019.
Yingzhen Li and Yarin Gal. Dropout inference in bayesian neural networks with alpha- digha-divergences.. In International conference on machine learning, pp. 2052-2061. PMLR, 2017.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. Advances in Neural Information Processing Systems, 33:7498-7573 -7112, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. In Advances In Neural Information Processing Systems, pp. 2378-2228 2386, 2016.
Andres Masegosa. Learning under model misspecification: Applications variational and ensemble methods. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5479-5491. Curran Associates, Inc., 2020.
Andres Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order pac-bayesian bounds for the weighted majority vote. Advances in Neural Information Processing Systems, 33, 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ning AAAI Conference on Artificial Intelligence, 2015.
11

---

Under review as a conference paper at ICLR 2023
Geir K Nilsen, Antonella Z Munthe-Kart Hans J Skaug, and Morten Brun. Epistemic uncertainty quantification in deep learning classification by the delta method. Neural Networks, 145:164-176, 2022.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning.
Yuki Ohnishi and Jean Honorio. Novel change of measure inequalities with applications to pac- bayesian bounds and monte carlo estimation. In International Conference on Artificial Intelligence and Statistics, pp. 1711-1719. PMLR, 2021.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta-learingg with pac-guarantees. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9116-91222 PMLR, 18 24 Jul 2021.
Nathan Silberman, Derek Hoiem, Pushmect Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference computer vision, pp. 746-760. Springer, 2012.
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008
Junjiao Tian, Dylan Yung, Yen-Chang Hsu, and Zsolt Kira. geometric perspective towards neural calibration via sensitivity decomposition. Advances in Neural Information Processing Systems, 34, 2021.
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https //openreview www.:ett..fo.... www.:neet.oooofdeeeomo d--.ormtmmmmd-kKktttsccCCC
Sumio Watanabe. Algebraic geometry and statistical learning theory. Number 25. Cambridge university press, 2009.
Sumio Watanabe. Mathematical theory of Bayesian statistics. Chapman and Hall/CRC, 2018.
Aolin Xu. Continuity of generalized entropy and statistical learning. arXiv preprint arXiv:2012.15829, 2020.
Aolin Xu and Maxim Raginsky. Minimum excess risk in bayesian learning. arXiv preprint arXiv:2012. 14868, 2020.
Nanyang Ye and Zhanxing Zhu. Bayesian adversarial learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 31. Curran Associates, Inc., 2018
12


Reviews:
[{"review_id": "r4NMna7lfS", "reviewer": "Reviewer_15vK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper derives PAC-Bayes bounds for epistemic uncertainty in terms of excess risk. A new quantity\nis defined called Bayesian Excess Risk which captures the epistemic uncertainty when the true\nmodel is known.\n", "summary_of_the_review": "This is interesting and useful theoretical analysis of bounding epistemic uncertainty.\n", "strength_and_weaknesses": "This paper solves a pressing problem in understanding epistemic uncertainty (EU) and connects it very\nwell to the more traditional methods of practically estimating EU. The paper offers novel and useful\ntheoretical contributions which are then supported by demonstratng a regularization strategy and several\nnumeric experiments. The work though would be valuable even without the additional validation.\n\nThe main weakness is some of the contributions could be better signposted in the writing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JOPpf5foh4b", "reviewer": "Reviewer_1SWx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explores the relationship among excess risk, widely used epistemic uncertainty and generalization error with convergence analysis in the setting of approximate Bayesian inference. Empirical evaluations on synthetic datasets are carried out to confirm the theoretical results. Further evaluations on UCI datasets and contextual bandit tasks are presented to compare the proposed approach with existing baselines.", "summary_of_the_review": "Despite some minor confusion and some potential improvement for the empirical evaluations, the contributions in this work seems solid and novel to me.", "strength_and_weaknesses": "This work focus on excess risk analysis under approximate Bayesian inference instead of an exact setting as in existing work, which is a more practical setting and seems novel to me. The main theoretical contributions are Thm 1, Thm 2 and Thm 3 where Thm 1 connects Bayesian excessive risk under log loss with approximate mutual information and the one under squared loss with variance and Thm 2&3 upper bounds the summation of prediction excess risk and Bayesian excess risk with convergence analysis.\nare nicely explained and seem technically sound.\n\nStill, here are a few concerns:\n- p^q is not defined in Eq 4.\n- I'm confused by Remark 2. What do you mean by \"misspecify noise function\"? Can you explain it in a more formal way?\n- In Fig 1 and Fig 2, I wonder which N are chosen? From the curves, it seems that the N are not uniformly chosen and also the curves are highly non-smooth. The author might what to try with more N to make some smooth curves that clearly shows the relationship among the three quantities.\n- The empirical evaluation would benefit from including more baselines other than particle VI. For example, to include Bayesian Dropout methods including MC dropout, variational dropout, ensembles, Langevin MCMC etc...", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "F9qXFW6yO9", "reviewer": "Reviewer_6WHt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This works studies the frequentist validity of popular epistemic uncertainty estimates, showing that they equal certain \"Bayes excess risk\" measures (Thm. 1) and are *lower bound* for the unobservable excess risk of a stochastic predictor derived from the (approximate) posterior.  The authors argue that such behaviors are undesirable and consistent with the fact that epistemic uncertainty are often underestimated in practice.  Based on these observations, the authors proposed a modified variational inference objective for regression problems, where a conditional variance term is down-weighted (Eq. 21).  Empirically the new objective demonstrates improved performance on regression and contextual bandit tasks.", "summary_of_the_review": "The authors studied a relevant problem, but appeared to have misinterpreted their results.\n\n**Post-rebuttal update**. \n\nI have read the authors' response but did not find them addressing my concerns.  My concerns were twofold:\n\n1. **The messages in Sec 3.2 were misleading.**\n\nFor the square loss case, the manuscript appears to draw connections between their Theorem 2, which shows that the posterior variance ($Var_{\\theta|z^N} f_\\theta(x)$) *lower bounds* their version of excess risk ($ER^{(2)}$), and the experimental fact that the posterior variance underestimates the epistemic uncertainty (EU); as shown in the following quotes:\n\n> From Eq. (12), Varθ|zN fθ(x) clearly is a lower bound of excess risk and test error, consistent with the well-known result that the variance of the predictor often underestimates EU (Lakshminarayanan et al., 2017).\n\n> First, the widely used EU measurements are the lower bounds of the test error and excess risks. This is consistent with the experimental fact that these EU measurements often underestimate EU.\n\nAs noted in my review and response, this is only because the authors looked at the wrong definition of ER (for this purpose): for squared loss, it equals \n$$\nER^{(2)} = E_{Z^n} E_{q(\\theta|Z^n)} E_{Z=(X,Y)} (f_0(X)-f_\\theta(X))^2,\n$$\nas opposed to \n$$\nE_{Z^n} E_{Z=(X,Y)} (f_0(X) - E_{q(\\theta|Z^n)} f_\\theta(X))^2.\n$$\n(The last tuple $(X,Y)$ denote the test input.) The latter is what we want to compare the posterior variance with.\n\nThe authors misunderstood my definition of \"coverage problem\" in the first response, which I have clarified; in their second response, they noted that my concerned relation about $E_{(X,Y)}(Y-E_{q(\\theta)} f_\\theta(X))^2$ and $E_X E_{q(\\theta)} (f_\\theta(X) - E_{q(\\theta)} f_\\theta(X))^2$ (posterior variance) was not analyzed in this work.  But this is precisely my point: what Sec 3.2 analysed was not useful (and misleading), and to explain the experimental fact ofunder-coverage, one should analyze this relation as opposed to the relation studied in Theorem 2.\n\n2. **The motivation of the algorithm was unsatisfactory.**\n\nAs the authors noted in their second response, the algorithm is motivated by the fact that \"minimizing the standard VI leads to minimizing PR and BER simultaneously, as shown in Eq.~19\". (I think they were referring to the standard ELBO.)  Without further explanation, or restriction of the scope of discussion, it is very unclear why this is a problem that needs to be addressed: if we do not restrict the variational family of choice, the true posterior is the minimizer for the ELBO, and clearly it doesn't always suffer from coverage problems.  The remaining of that subsection (3.3) did not offer any convincing explanation.\n\nIn summary, while the manuscript contains interesting materials (the algorithm and experiments, for example), it appears that the theoretical discussions need to be thoroughly revised.  Therefore, my rating remains unchanged.", "strength_and_weaknesses": "This work studies a very relevant problem: whether (approximate) Bayesian epistemic uncertainty estimates are valid upper bounds for the excess risk, in which case they can be viewed as valid in a frequentist sense.  The authors showed that posterior variance-like quantities can be *lower bounds* for the excess risk, which would be a serious issue and worth addressing.\n\nHowever, it appears to me that the authors studied the wrong excess risk: they considered the excess risk of the stochastic predictor derived from the Gibbs posterior,\n$$\nR^l(Y | X, \\mathbf{Z}^N) := E_{Z^n} E_{\\color{blue}q(\\theta\\mid Z^n)} E_{Z=(X,Y)} \\ell(Y, f_\\theta(X)),\n$$ \nas opposed to the deterministic predictor which approximates the true Bayes predictor.  The former predictor only achieves order-optimal excess risk, which is reasonable for studying contraction properties such as predictive error, but usually unreasonable as a target for uncertainty quantification.\n\nAs a concrete example, for homoscedastic regression problems with a square loss, our point estimate is typically based on the posterior mean estimate $\\hat f_n(x) := \\mathbb{E}_{q(\\theta\\mid Z^n)} f_\\theta(x)$, and we are most interested in whether posterior variance quantities such as $\\mathbb{E}\\_{q(\\theta\\mid Z^n)}\\big(f\\_\\theta(x) - \\hat f_n(x)\\big)^2$ upper bounds the excess risk-type quantities for the posterior mean estimator, $\\mathbb{E}\\_{\\nu(Y|X=x)} (\\hat f\\_n(x) - Y)^2 - \\text{(Bayes Risk at $x$)} = (\\hat f_n(x) - f_0(x))^2$.  This corresponds to the scenarios where the credible intervals may have correct coverage.  However, the authors in their Theorem 2 compared this posterior variance with the quantity \n$$\nR^l(Y | x, \\mathbf{Z}^N) - \\text{(Bayes Risk)} = \\underbrace{\\mathbb{E}\\_{q(\\theta\\mid Z^n)}\\big(f\\_\\theta(x) - \\hat f_n(x)\\big)^2}\\_{\\mathrm{BER}(Y|x,\\mathbf{z}^N)} +  \\underbrace{((\\hat f_n(x) - f_0(x))^2}\\_{\\mathrm{PER}(Y | x, \\mathbf{z}^N)}.\n$$\nClearly, the fact that the posterior variance fails to upper bound the above says nothing about its frequentist validity.\n\nAs another sanity check, it is also unconvincing that theoretical explanations for the under-coverage issue can be provided in such general settings, as for the results in Section 3: the theorems apply to the true Bayesian posterior and correctly specified (low-dimensional) parametric models.  Clearly, the right conclusion shouldn't be that under-coverage happens across all such settings.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject