On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
Authors: Prateek Jaiswal
Harsha Honnappa
Vinayak Rao
Venue: NeurIPS 2023 poster (2023)
Keywords: Variational Bayes
Loss Calibration
Bayesian Statistics
Variational Inference
Statistical Theory

Abstract: We study data-driven decision-making problems in the Bayesian framework, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure with respect to the posterior distribution. We focus on problems where calculating the posterior distribution is intractable, a typical situation in modern applications with large datasets and complex data generating models. We leverage a dual representation of the entropic risk measure to introduce a novel risk-sensitive variational Bayesian (RSVB) framework for jointly computing a risk-sensitive posterior approximation and the corresponding decision rule. Our general framework includes \textit{loss-calibrated} VB (Lacoste-Julien et al. [2011] ) as a special case. We also study the impact of these computational approximations on the predictive performance of the inferred decision rules. We compute the convergence rates of the RSVB approximate posterior and the corresponding optimal value. We illustrate our theoretical findings in parametric and nonparametric settings with the help of three examples.

On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
Prateek Jaiswal Department of Statistics Texas A&M University College Station, TX 77843 jaiswalpßstat. tamu. edu
Harsha Honnappa
Vinayak Rao Department of Statistics Purdue University West Lafayette, IN, 47907 varao@purdue. edu
School of Industrial Engineering Purdue University West Lafayette, IN, 47906 honnappa@purdue. edu
Abstract
We study data-driven decision-making problems in the Bayesian framework, where the expectation in the Bayes risk is replaced by risk-sensitive entropic risk measure with respect to the posterior distribution. We focus on problems where calculating the posterior distribution is intractable, typical situation in modern applications with large datasets and complex data generating models. We leverage dual representation of the entropic risk measure to introduce novel risk-sensitive variational Bayesian (RSVB) framework for jointly computing a risk-sensitive pos- terior approximation and the corresponding decision rule. Our general framework includes loss- calibrated VB [16] as a special case. We also study the impact these computational approximations
the predictive performance of the inferred decision rules. We compute the convergence rates of the RSVB approximate poste- rior and the corresponding optimal value. We illustrate our theoretical findings parametric and nonparametric settings with the help of three examples.
Introduction
This paper focuses on risk-sensitive Bayesian decision decision-making, considering objective functions of the form
min eñ, (R(a,,) 0)) == 7 log Er,, [exp(>R(a, 0))]- aeA
(SO)
Here A is the decision/action space, 0 is random model parameter lying in arbitrary measurable space (9, T), and R(a, 0) A θ + IR is problem-specifte model risk function. For any B C O II, (B) : (B) is the Bayesian posterior distribution over the parameters given observations X, from the true model Ph (= Pit) with parameter Op E θ. The scalar IR is user-specified and characterizes the sensitivity of the decision-maker (DM) to the distribution IT Recall that
the posterior distribution is obtained by updating prior probability distribution I1(-), capturing subjective beliefs of the decision maker over O according to the Bayes rule
xTT())..) x (T(O)dP((X..)
where d.P. (X,.. is the likelihood of observing X,, We denote the corresponding densitics (if they exist) for the model, prior, and posterior as p (-), (T)), and क (-I[x,,, respectively.
The functional o in (SO) is also known the entropic risk measure, and models range of risk- averse or risk-seeking behaviors in a succinet manner through the parameter 7y Consider only strictly positive 7, and observe that lim 7 log En, (exp(1 (exp(>R(a,))))) (((a0))) 0)) En, (R(a, 0)); that is, there is sensitivity to potential risks due to large tail effects and the decision-mant maker is risk neutral.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

On the other hand, lim Pi1, (R(a, = ess sus,,, (R(a, 0)), the essential supremum of the model risk Ri(a, 0). In other words, decision maker is completely risk averse and anticipates the worst possible realization -ammsst surely). Observe that (SO) strictly generalizes the standard Bayesian pocision-theorett formulation of decision-making problem, where the goal is to solve mil.e.A En,,. [R(a, 0)].
The risk-sensitive formulation (SO) is very general and can be used to model a wide variety of decision- making problems in machine learning [3] operations research/management science 24 20
simulation optimization 6 [2). and finance [7 [ 4]. However, solving (SO) to compute an optimal decision over A is challenging. The difficulty mainly stems from the fact that, with the exception of conjugate models, the posterior distribution in ( is an intractable quantity. Canonically, posterior intractability is addressed using either sampling- or optimization- approaches, such as Markov chain Monte Carlo (MCMC), offer a tractable way to compute the integrals and theoretical guarantees of exact inference in the large computational budget limit.
approach. Sampling-basee
aptimization-basel methods such as variational Bayes (VB) or variational inference (VI) have emerged as popular alternative 3.. The VB approximation of the true posterior is tractable distribution chosen from a 'simpler' family of distributions known as a variational family by min- imizing the discrepancy between the true posterior and members of that family. Kullback-Liebler (KL) divergence is the most often used measure of the approximation discrepancy, although other divergences (such as the a-Rényi divergence [9 28
have been used. The minimizing member, termed the VB approximate posterior, can be used as a proxy for the true posterior. Empirical studies have shown that VB methods are computationally faster and far more scalable to higher-diment dimensoonll problems and large datasets. Theoretical guarantees, such as large sample statistical inference, have been a topic of recent interest in the theoretical statistics community, with asymptotic properties such as convergence rate and asymptotic normality of the VB approximate posterior recently established respectively.
in [34 21 and
14
Our ultimate goal is not to merely approximate the posterior distribution but also to make decisions when that posterior is intractable. näive approach would be plug the VB approximation in place of the true posterior in (SO) and compute the optimal decision. However, it has been noted in [16] that such a naive loss-unaware approach can be suboptimal. In particular, [16] demonstrated. through an example, that a naive posterior approximation only captures the most dominant mode of the true posterior, which may not relevant from a decision-makin perspective. Consequently, they proposed a coss-calibrated variational Bayesian (LCVB) algorithm for solving Bayesian decision- making problems where the underlying risk function is discrete. [15] extended their approach to continuous risk functions. Despite these algorithmic advances in developing decision-centrie VB methods, their statistical properties, such as asymptotic consistency and convergence rates of the loss-aware posterior approximation and the associated decision rule are not well understood. With an aim to address these gaps, we summarize our contribution in this paper below:
1. In Section 2 we introduce a minimax optimization framework titled 'risk-sensitive varia- tional Bayes' (RSVB), extracted from the dual representation of (SO) using the so-called Donsker- Varadhan variational free-energy principle [9]. The decision-maker computes a risk-sensitive approximation to the true posterior (termed as RSVB posterior) and the decision rule simultaneously by solving a minimax optimization problem. Moreover, we recover the LCVB approach [16] as a special case of RSVB (with y = 1).
2. In Section 3 we identify verifiable regularity conditions on the prior, likelihood model and the risk function under which the RSVB posterior enjoys the same rate of convergence as the true posterior to a Dirac delta distribution concentrated at the true model parameter fp. as the sample size increases. Using this result, we also prove the rate of convergence of the RSVB decision rule when the decision space compact. Our theoretical results also imply the asymptotic properties of the LCVB posterior and the associated decision rule.
3. In Section 4 we demonstrate our theoretical results with the help of three widely studied decision-making making problems, including Gaussian process classification and a newsvendor problem. For each example, we show that the rate of convergence of the respective RSVB approximate posterior matches that of the corresponding true posterior.
4. In Section we also present some simulation results with the single product (1-d) newsven- dor problem, which is summarized here in Figure T The figures demonstrate the effect of changing 7 on the optimality gap in values (see Definition 2.1 and the variance of the
2

---

RSVB posterior for given n. In particular, we observe that for smaller n, increasing (after certain value) results in significantly more risk-avers decision; however, the effect of increasing ? risk-averse decision-making reduces as increases. This observation demonstrates the fact that when there enough certainty the parameter estimation, the need to be risk-sensitive to parametric uncertainty diminishes. The Figure[]aaso compares the optimality gap in values evaluated at RSVB decision rules and variance of the RSVB posterior for various values of against the näive VB approach and against the true posterior when the conjugate prior is used.
10a
Ralo 101
20
R(a .o) Ria .06l
10a
10
10-1
Variance
10
10
10
10
10-1
10-1
40
80
100
20
40
60
100
number of samples (n)
number of samples (n)
(a) Optimality gap in Values
(b) Variance
Figure I: (a) Optimality gap in values and (b) variance the RSVB posterior (mean over 100 sample
paths) against the number of samples (11) for various values of
2 Risk-Sensitive Variational Bayes
Our approach exploits the dual representation of the log-exponential risk measure (SO), which convex (or extended coherent) risk measure [23 10]. From the Lonsker-Varadhan variational free energy principle PTT, (a) max [Eo 0)] (L(Q)))))) QEM
(9] observe that,
0,
(DV)
where M is the set of all distribution functions that are absolutely continuous with respect to the posterior distribution II, and 'KL' represents the Kullback-Leibler divergence. Formally, for any two distributions P and Q defined on measurable space (9, T), the KL divergence is defined as dQ(0)
KL(Q fo dQ(() log aP(0) when measure Q is absolutely continuous with respect to / and co otherwise. Notice that this dual formulation exposes the reason we choose to use the log-exponential risk the 'free-energy' objective on the RHS provides combined assessment of the risk associated with model estimation (the KI. divergence (Q) and the decision risk under the estimated posterior Q (Eq|R((a,)) 0))).
As stated above, the reformulation presented in (DV) offers no computational gains. However, restricting ourselves to an appropriately chosen subset @ C .M, that consists of distributions where the integral [R(, f)) and (II can be tractably computed and optimized, we immediately obtain a risk-sensitive variational Bayesian (RSVB) formulation from (DV):
7 log ET,, (().,)))
max [Ea[R(a, 0)] - - "KLK-- =: =: F(a;
(RSVB)
QEO
RSVB our framework for data-driven risk-sensitive decision- maaiiin..-maiiig..... The family of distributions Q is popularly known as the variational family. Our analysis in Section reveals general guidelines on how to choose Q that ensures a small optimality gap (defined below) with high probability.
With an appropriate choice of Q, the optimization on the RHS can yield a good approximation to the log-exponential risk measurement the left hand side (LHS). For brevity, for given a E AA define the RSVB approximation to the true posterior II(0) X,)
Qa, argmax (Q Q F(a; Q(), X,.
3

---

and the RSVB optimal decision as
ags argmin F(a; (0))); Xa, 7) argmin aEA max F(a; Q(),, Q(),, 7).
QEO
Observe that (0 and ais are random quantities, conditional on the data Xn..
Examples of Q include the family of Gaussian distributions, delta functions, or the family of factorized mean--iell'' distributions that discard correlations between components of 9. The choice of Q is decisive in determining the performance of the algorithm. Part of the analysis in this paper is to articulate sufficient conditions on Q that ensure small optimality gap (defined below) for the optimal decision, ags- This establishes the statistical "goodness" of the procedure as number of samples increase. In this paper, we analyze the efficacy of the decision rules obtained using the RSVB approximation, by providing high-probability bounds on the optimality gap. We define the optimality gap for any a E A with value = R[a, 0o) as,
Definition 2.1 (Optimality Gap). Let Vø == minae R(a, Oo) be the optimal value for the true model parameter 0g- Then, the optimality gap in the value is the difference V Vi-
A similar performance measure was used in [1], to measure the effectiveness of loss-calibrated VB (LCVB) approach, which can be obtained by setting ? 1. We provide definitions of some standard terminologies that we use in the subsequent sections, such as covering number, test functions, I Tecnnergencc,, and primal feasibility the Appendix [ofor ready reference. the following section, we lay down important assumptions used throughout the paper to establish our theoretical results.
2.1 Assumptions
In order to bound the optimality gap, we require some control over how quickly the posterior distribution concentrates at the true parameter On. Our next assumption in terms of a verifiable test condition on the model (sub-)space is one of the conditions required to quantify this rate. Let
Ln e x θ + [0, oo) be a distance metric on the model space.
Assumption 2.1 (Model indentifiabilityy. Fix n. 2 1. Then, for any > such that en + 0 as
oo and n 21 1, there exists measurable sequence of test functions On. -n 1 [0, 1] and sieve set O (E) C θ such that (i) [onc] S Co exp(-Cne") and (ii)
sup (0
Ep [1-
Gn,el S exp(-Cne²).
Observe that Assumption 2.1 i) quantifies the rate at which a Type I error diminishes with the sample size, while the condition in Assumption 2.1 ii) quantifies that of a Type II error. Notice that both of these are stated through test functions; indeed, what is required are consistent test functions. 11 Theorem 7.1] (stated in Appendix as Lemma C.7 for completeness) roughly implies that an appropriately bounded model subspace {P 0 E O guarantees the existence of consistent test functions, to test the null hypothesis that the true parameter is 0, against an alternate hypothesis the alternate being defined using the distance function L,, Subsequently, we will use a specific distance function obtain finite sample bounds for the optimality gap decisions and values. Note that in some cases, it is also possible to construct consistent test functions directly without recourse to Lemma C.7 We demonstrate this in Section below. Next, we assume a condition on the prior distribution that ensures that it provides sufficient mass to the sieve set θ, (c) C O, as defined above in Assumption 2.1
Assumption 2.2. Fix 2 1. Then, for any En such that e,, + 0 as n - oc and ne? 2 1. the prior distribution satisfies (T(O,, S exp(-Cne")
Notice that Assumption 2.2 is trivially satisfied if ,c, (c) = θ. The next assumption ensures that the prior distribution places sufficient mass around neighborhood of the distribution P
Assumption 2.3 (Prior thickness). Fix n 2 1 and constant À > O. Let An := {0 E
θ DI+A
(PTI)P)) < Csnea), where DI+A (PT)) == log
dP is the Rényi divergence between Pe and Pj. assuming Pe is absolutely continuous with respect to Pp. The prior distribution satisfies 11(An) 2 exp(-nCze)).
aPR
This assumption guarantees that the prior distribution covers the neighborhood with positive mass. The above three assumptions are adopted from [I] and has also been used in [34] to prove convergence

---

rates of variational posteriors. Interested readers may refer to 1 and [4] to read more about the above assumptions.
is apparent by the first term in (RSVB) that in addition to Assumption 2.1 2.2 and 2.3 we also
require regularity conditions the risk function R(a,-). The next assumption restricts the prior distribution with respect to R(a, 0).
Assumption 2.4. Fix 2 1 and y > O. For any E E> en. a E A, ( exp(-Ca(7))e where Ca(7) and Cs(t)) are scalar positive functions of %
S
Note that the set {rlB(a, > C=(r)me²} represents the subset of the model space where the risk R(a, f) (for a fixed decision a) is large, and the prior is assumed to place suf- ficiently small mass over such sets. Moreover, using the Cauchy-Schward inequality ob- 1/2 (Enie 1/2
S (, --Ca(r)ne En Iele 1)2 which implies that if the risk function is bounded in (a, 0), then the as- sumption trivially satisfied. Finally, we also require the following condition lower bounding the risk function R.
serve that En [1 ((()a))))
Assumption 2.5. R(a,0) is assumed satisfy W infoge inf R(a.8) > 0.
Note that any risk function which is bounded from below in both the arguments satisfies this condition. In the next section, we establish sigh-probability bounds on the optimality gap in values and decision rules computed using RSVB approach for sufficiently large n.
3 Asymptotic Analysis of the Optimality Gaps
Our first result, establishes an upper bound on the expected deviation from the true model Po measured using distance function (.,(-,0)) under the RSVB approximate posterior. We also note that the following result generalizes Theorem 2.1 of [34], which is exclusively for the case when
7 0+ Our proof techniques follows that of Theorem 2.1 in [34).
Theorem 3.1. Fix a' E A and y > O. For any L,, 0g) 2 under Assumptions 2.1 2.2 2.3
2.4 and 2.5 and for min(C, Ca(r) + Os(7)) > C2 + +C + Calr) + and The (7)
π QeQ Epp (I(((())I((OXX.))) - iinfac (e( 01] for sufficiently large the RSVB
approximator of the true posterior Q2 i (0) (n) satisfies,
Erg J. I, (0, do (01x..) S n (M))) + MT)(
for a positive number M(7) = 2 (C1 + MC)(7)), where M min(C.),1)
2C,
Proof sketch. For brevity we denote the likelihood ratio as CR. (0,0o) = Ml X (8) We prove this
p(X., [0,)
result using series of lemma. The first Lemma C.I separates the term " (r). which is later analyzed
using Assumption 3.J in Section 3.1 Lemma establish that
Pp V/ (0, (o) dQ=, (01X))
er .0 CR, 0o d(l() (0.00)
< log Ep
e CR, (0, O (T)
|/ eill .) CR, (R,(0,00))11 do)
+ pl
nu 1 (7).
Jo (O,, (o))
The proof of Lemma C.I uses simple arguments that follow easily from Jensen's 's inequality and the definition of the posterior distribution.
To analyze the first term in the above display, we define the set K, (0 e (0, > Cine }. with aim control the exponential moment of T., (0,, fo) by characterizing its tails. Also, notice that set K, is the set of alternate hypotheses as defined in Assumption 2. through Lemma C.2 For
brevity, define I (K (x,) := Jx., e Riee .0) CR. (()))(I())
Next, we divide the expected calibrated
Lo eri(aa (R..1..0))dl
posterior probability of the set Kn as follows:
TT (Ie(K,, ( S Erpon + Epp [1 Epg [1 (n,),, Mn, (n(Kn (n(K.....)))
E
(4)
5

---

where recall that { is the sequence of test function from Assumption 2.1 and set B, =
with set A, is defined in Assumption 2.3 [Xn Se (0, (O,,(, 2 e e=(++C))mmII(a
Set B, introduced to separately control (by lower bounding) the denominator in the denni-
tion of the posterior distribution in the last term of ( In addition, the Assumption 2.3 is
used to show that Po (B.) & e->mee_ The first term can be controlled by Assumption 2.1 (i).
Now, it remains to analyze the last term in 4 Using Assumption [23 and 2.5 observe that on set Bu Sg E² R(a',) (0, (o) 2 W"e=(++Caaaamee This observation, with the application of Fubini's theorem, enables us to bifureate further the last term in (4) on the set sy = [e 0) > eC(tt)ne)) as
Epp (1 (1-0n,e) )..))B, (R S -W
Uxins,1 Epy [[1 )n.))) ((n-d))11 d1
(,)
+ te Cu(imm?
e en 001(()) (5)
(0)
021+C2+00 + C3 Observe that the last term above can be controlled using Assumption 2.4
The first integral in (5) can be further bounded by (k.ne..()) Ep [[1 @n.c)] )] dlI(0) + II(O,.
where ,,, is defined in Assumption 2.1 and both the terms can be exponentially bounded using Assumption 2.1 (ii) and 2.2 respectivvll. The last term in 3) can be bounded using similar set of arguments and techniques.
Remark: We note that while deriving the bound in Lemma C.I we can interchange the order of expectation and infimum and compute tighter bound. However, this bring us back to the RSVB optimizer, the very objective of the whole analysis. Taking expectation before infimum enables us to derive more interpretable and meaningful bound. This is mainly because it is easier to control the expectation of the RSVB objective than its infimum through Assumption 3.1, as presented in Section 3.1
The detailed proof of Theorem 3.1 is provided in the Appendix C.2 Now recall that en is the
convergence rate of the true posterior 11 Theorem 7.3]. Notice that the additional term R (7) emerges from the posterior approximation and depends on the choice of the variational family Q. risk function R(-,).. and the parameter 7. The appearance of ()(()) in the bound also signifies that to minimize the expected gap (under the RSVB posterior) between the true model and any other model (defined using n >>-'L,,,,, (0, 0o)) the expected RSVB objective has to be maximized. Later in this section, we specify the conditions that ensure R (7) -> as n - oc. Moreover, we also identify mild regularity conditions on Q to show that n,, (7) is O(2.) and show that ? increases n, (γ) decreases. We discuss this result and the bound therein later in the next subsection. Before that, we establish our main result (the bounds on the optimality gap) using the theorem above. We now fix
L (0, 0o) = n sup [((a, 0) - R(a, H(a,Bo))) AaEA
(6)
Notice that for a given 0, n 1/2 / Tun (0, 0o) the uniform distance between the R(a, 0) and R(a, fo). Intuitively, Theorem 3.1 implies that the expected uniform difference 1 L., (0, 0o) with respect to the RSVB approximate posterior is O ( + (x)), and if (7) + MT (7) - 0 as n + oo then it converges to zero at that rate.
Also, note that in order to use (6) we must demonstrate that it satisfies Assumption 2.1 This can be achieved by constructing bespoke test functions for given R(a, 0). We demonstrate this approach by an example in Section B.2 We also provide sufficient conditions for the existence of the test functions in the appendix. These conditions are typically easy to verify when the loss function I((-,-) is bounded, for instance. Next, we bound the optimality gap between R(ass, 0b) and Vo--
Theorem 3.2. Fix > 0. Suppose the set A is compact. Then, under Assumptions 2.7 2.2 2.4 and 2.5 for min (7()) + Cs(r)) > C2 + C3 + +C(7)) + 2 and any T >> 0, the p probability
of X,, : R(ass: 0n) infaE R(a, 0o) S 2r [M()] + MT)(()]]
is at least - r for a
positive mapping M(7) = (Ci + MC1(7)) where M = min(C.A.II for sufficiently large n.
20,

---

3.1 Properties of nn (7)
Evidently, the bounds obtained in both the results that we have proved so far depend on n (r). Consequently, we establish some important properties of (t)) with respect to n and % under additional regularity conditions. In order to characterize 7h2 (7). we specify conditions on variational family Q such that (7) = O(r)), for some <, 2 v and en - O. We impose the following condition on the variational family @ that lets us obtain a bound on (n(()) in terms of n and 7. Assumption 3.1. There exists a sequence of distributions (4a()) {-) in the variational family such for positive constant Cg.
1 KL (Qx(O))II(())) + EQ, KL (X < Coc
(7)
If the observations in Xx are i.i.d, then observe that 1 EE (0) n
KL dd =
(Q) [KL (dPolldPo)] Intuitively, this assumption implies that the variational family must contain sequence of distributions weakly convergent to Dirac delta distribution concentrated at the true parameter Og otherwise the second term in the LHS of 7 will non-zero. Also, note that the above assumption does not imply that the minimizing sequence Q² (0 (0)_,) (automatically) converges weakly to a Dirac-deltt distribution at the true parameter 8g. Furthermore, unlike Theorem 2.3 of 34], our condition on Q in Assumption 3.1 to obtain a bound on ny, y), does not require the support of the distributions in Q to shrink to the true parameter Og at some appropriate rate, as the numbers of samples increases.
The condition that the variational family contains Dirac delta distributions at each point in the param- eter space is a mild and reasonable requirement for consistency. Further, Assumption 3.1 requires that contains sequences of distributions that weakly converge to each Dirac delta distribution at a certain rate. This is easily satisfied if Q has no "holes", e.g. if it is the family of Gaussians with all means and variances, then we can always construct sequences converging to any Dirac delta at any rate. A similar assumption has also been made in [31), and is true for most exponential family distributions. For instance, in the newsvendor application, we fix the variational family to a class of Saftted-Gama distributions and choose sequence of distributions with parameter sequence a = n and 3 = n n//0. This implies that the sequence of distributions has mean On and variance 0g/ /,, and, therefore, it converges to a Dirac delta distribution at Og- Next, we show that
Proposition 3.1. Under Assumption 3.1 and for a constant Cs = ---ffee inf Eq [R(a, 0) and Cg > 0, ny. (7) I 7 CCs + C
In Section 4 we present example where the likelihood is exponentially distributed, the prior is evverse-gamma (non-conjugate), and the variational family the class of gamma distributions, where we construct sequence of distributions in the variational family that satisfies Assumption [3.]] We also provide another example where the likelihood is multivariate Gaussian with unknown mean and variational family is uncorrelated Gaussian restricted to compact subset of gd with uniform prior on the same compact set satisfy Assumption 3.1
By definition e2 + 0 and é, + 0 as + oc, and therefore it follows from Proposition 3.1 that ( + .M respect to 7, when C3 < O. To see this, we prove the following result.
'n (7) + O. However, the bound obtained in the last proposition might be loose with
Proposition 3.2. If the solution to the optimization problem in n (7) is primal feasible then 1 ( (7) decreases as y increases.
4 Applications
In the examples below, we use three examples to study the interplay between sample size n and the risk parameter and their effect on the optimality gap in values. Additionally, we consider a multi-product newsvendor example in the Appendix
4.1 Single-product Newsvendor Model
We start with a canonical data-driven decision-making problem with 'well-behaved' risk function R(a, 0), the data-driven newsvendor model. This problem has received extensive study in the literature and remains cornerstone of inventory management 2 [8]. Recall that the newsvendor
7

---

loss function is defined as ((a, E) := h b a)* where h (underage cost) and b (overage cost) are given positive constants, 5 E [0, oo) the random demand, and a the inventory or decision variable, typically assumed to take values in a compact decision space A with := min(a and a == max {a a E A), and a > 0. The distribution over the random demand, Po is assumed to be exponential with unknown rate parameter 0 E (0, oo). The model risk R(a, 0) := Ep, (e(a, E))
E A}
ha - う/ + (b+ h) 0 which is convex in a. We assume that X,, == (Li, fg. ........ ... } be n. observations of the random demand, assumed to be i.i.d random samples drawn from Po-
=
We fix the model space θ [T' 20) for some T 0 and assume that 0g lies in the interior of O. We now assume a non-conjugate truncated inverse-gamma (Inv I`) prior distribution restricted to O, with shape and rate parameter a and / respectively, that is for set A S O, we define II (I)
Inv - -e (A: a, .8) = Inv - (Ano); a. 8) I (). We verify Assumptions 2.2 2.1
2.3 2.5
and 2.4 (in that order) in this newsvendor setting and provide the proofs in the Appendix B. Next,
we bound the optimality gap in values for the single product newsvendor model risk.
Theorem 4.1. Fix ? > 0. Suppose that the set A is compact. Then, for the newsvendor model with exponentially distributed demand with rate 0 E += [T, oo), prior distribution II(.) hy - [e(:;a, --v-I I(Ano; a, /m) (n)) at 3)), and the variational family fixed to shifted (by T > 0) gamma distributions, and for any > 0, the Pa probability of the following event
1/2
Xa= R(ais Oo) - inf Ro),00) S <TT log
is at least 1 - -- for sufficiently
large n and for some known mapping M': R - R., where R(-, 0) is the newsvendor model risk.
The proof of the theorem above is a direct consequence of Theorem 3.2 and Proposition 3.2, and Lemmas B. B.2 B.3 B.4 B.5 stated in the Appendix. We also extend the analysis above to a multi-produet newsvendor problem. The details are provided in Appendix B..2]
Next, we demonstrate the effect of varying the risk risk-senst parameter 7/- We fix 0g = 0.1,
h = 5, a and _ 4.1. We run RSVB algorithm with y {0( naive ), 1, 4.5, 5, 6} and repeat the experiment over 100 sample paths. We plot the results in Figure ! In Figure Ta) we plot the optimality gap in values that is R(ass (7), 0u) R(ag: 0a) for various values of 7. We observe that the gap decreases when n increases. This observation supports our results in Propositions 3.] and 3.2 that establishes the properties of n (7) as n increases. Lastly, in Figure (b), we plot the variance of the RSVB posterior as n increases for various values of y as anticipated, the variance reduces as increases. To observe the effect 7, first recall that y increases the decision maker becomes more risk averse, and so is our algorithmic framework RSVB. Indeed, from the rightmost variance plot in Figure 1 it is evident that for larger value of 7 (> 4), the RSVB posterior is more concentrated on the subset of O, where risk is more and consequently we observe large optimality gaps in values. Moreover, as increases, the effect of larger 7 reduces, since as n increases, the incentive to deviate from the posterior reduces (due to increased KL divergence dominance for larger in RSVB). We also observe that the decision rule learned using the conjugate prior (Gamma distribution for exponential models) has similar performance as the näive approach. However, the variance of the true conjugate posterior is higher than those computed through the RSVB and VI approach, corresponding to the well known fact that the variational approximations underestimate the true posterior variance ((8 19)).
4.2 Gaussian process classification
Consider problem of classifying an input pattern or features Y lying in measure space ([O, 1]d y, (with sigma algebra y and probability measure v/) into one of two classes 1). Let Y + <(Y) E 1 denote the class of Y. For a given Y, we model the classifier using a Bernoulli distribution p(E)Y, 0) Ve (OY)), where 0 [0, 1]d + R is a non-parametric model parame- ter in separable Banach space (G, II) and measurable functions & (x(x) = (1 + +e- and V (x) = 1 V (x). Note that (-)- is a logistic function. We denote vb(-) as the derivative of (1). Assume that the features y are generated independently of &. Thus the sequence of inde- pendent observations { ( X,
E1), (22,22,, (Yn, {n)} are assumed be generated from the model dPo(a, u) Po = a, Y E dy) = p(E = alY = y, ()(0))))) In the above binary classification problem, the objective is to estimate 0(-) using the observation vector {Ý,,. X,
We posit Gaussian process (GP) prior IT(.) on 0(-) E θ defined W() =
J. =1= 1= My (, where {my} } is sequence that decreases with j. (Z,,)) } are i.i.d. standard
8

---

Gaussian random variables and { form a double-indexed orthonormal basis (with respect to mea-
sure v), that is E, (. = (j=(,k==) Ja is the smallest integer satisfying 2 (2/+++)
for a given a > 0. This prior construction is motivated from the work in 30J. We also as- sume that v.) is known, and we do not place any prior on it. The posterior distribution over O())
given observations (Yn, X can be defined as dTI n. xx,)) = /T:: [I? Ve, (P(.)))))) IT 2e, (P(T))
TIl Ve, (0(0 ((0))dT100)) Consider the loss function E(a, E) = {0, if a = 5: c,, = +1, =
-1; c_, if a = -1, = +1), where 04 and c. are known positive constants. For a given feature
Y, the model risk is given by
R(a, 8) EE, Er, ((,,(())))) = c. ( (((Y))) a a=+1,
[V (cc,
(8)
a a=
We fix the variational family Qap is class of Gaussian distributions on + defined (m,, Cg). my belongs to O and C. is the covariance operator defined as Cg CT/2 S) for any S which is a symmetric and Hilhert-Schmidt (HS) operator on θ (eigenvalues of HS operator are square summable). Note that S and my span the distributions in Qar. We can show, using the technical lemmas derived in Section B.3 in Appendix, that the optimality gap in values of the binary GP classification problem converges zero at a minimax optimal rate (upto logarithmic factors).
Theorem 4.2. Fix > O and for given J E N. For the binary GP classification problem with U3Z where p4j = 2-/24/2-j- for some a >> O, {Zzr} are ii.d. standard Gaussian random variables and {0 form double-indexed orthonormal basis (with respect to measure v), and |1001l < oc, and 0g (v) lie in the Cameron-Marlin space lm(C1/2), the variational family QGP, and for any T > 0, the P probability of (X,. R(aas: Oo) - infzEA R(2, 0o) (x)))))) } is at least 1 r for sufficiently large and for some mapping M' IR+ - R+, where R) is defined in and
GP prior induced by W= EE==
(20+4) log n vB E la, n (Botd) log Va E [a, Ba; (2+4)) (logn) n) (B,+4) Va [a, B]: n (2:+d) (log n) (Bx+4)
6n
(9)
(o,a).
similar Gaussian process classification problem was studied empirically in 16
4.3 Eight-schools model
We consider the eight-schools problem [33 15]. where the objective is to study the effectiveness of the special coaching program for SAT exams, offered in 8 schools. Each school reported the treatment effect yJ and its standard deviation σ;, where i E {1, 2,,,...,88) 88.
The observations {yi, az are
modeled using the following hierarchical Bayesian model: 1) Prior distributions: p p~ N [N 5),
half-Cauchy (-10, -00, 5), 2) 0 ~ N(,l,, -²) and ys (V af) for each i E {1. 3. 8), where {ar)s_1 are assumed to be known sequence of covariates. The posterior distribution defined as TT(H4 T, {0 SessHKe Hys, o, = x (- Jaaff-CCy (-10, 5) WV (V(0,u,, T") (v.lo, a?)) For a
given treatment effect for eight schools y E R.S, the loss function is defined as:
(0.2 la: Vil. Yi 2 0g
a(y,a)= = Σ Uy a.), where ((y), a4)
(10)
{11 0.2) la; Vil. yi < az.
where the decision variable
{ai, (di,ar..... ag...., Jz} denotes the effectiveness level. We de- fine R(a,0) 0) = log I p(y10, {o,))_, (5m,) a) a)dy. f E R considering {ar] is a given se- quence of covariates. Now for any 7 > 0. RSVB joint-optimizationn problem is defined as min maxqee [E, 0)] (()l
We fix the variational family Q to be the mean-
field variational family, that is qA (p, r, 0, = NNu(-\A) (-1A1o) No No. (lxx), where {AI.2..... ....... }.o}. X; is the mean and variance parameter for each Gaussian distribution. Follow- ing [15], we measure the performance of the RSVB method using the metric called empirical risk re- duction (ERR). I &RVB ERRSVB()) where & = Ny Σnn ((y) aatg), where Ralg de- note the empirical risk evaluated at the decision rule a obtained using method alg (VB.RSVB(7))

---

for various values of y, and Ny is the size of the test data {v]**) Note that ERR is the empirical approximation of the difference of optimality gap between the two-step näive VB approach and RSVB(7) approach. Recall, in the näive VB method, we first compute the KI. minimizer of the true posterior and then compute the optimal decision using this approximate posterior. Following [[S), due to small size of the dataset in this example, the ERR is evaluated on the training data itself.
We modified the experiments in [5] by introducing 7 {5, 2.5, 1,0,1, and obtain the results as summarized in Figure 2 a) and (b). In Figure [a), we observe that as increases, the ERR also increases, which implies that the decisions are more optimistic as empirical loss of the RSVB decision rule decreases as increases. Also, observe from Figure 2 that, as increases, the RSVB posterior (joint marginal posterior distritbution for (03. r)) approaches the true posterior, and the variance of the approximate posterior also reduces.
(0.2-) 10.2--1000--,,111 2-1
10.0
Empirical Risk Reduction (in
-02
Posterior
7.5-
7
=0.4
RSVB-y"
RSVB-y"
5.0
=1
RSVB-y
RSVB-Y =2
0-
VIVB
-2.5
10
10000
20000
Epoch
(a) ERR
(b) [0s.T)
Figure 2: a) Empirical risk reduction plot for different level of risk sensitivity b) Joint RSVB posterior distribution of {03, 7) plotted for different level of risk sensitivity.
Notice that is not obvious that the variance of the RSVB posterior will reduce as y increases. We believe that it depends on the landscape of the expected risk function and the choice of the variational family. Intuitively, a possible explanation of this phenomenon can provided using the equation (RSVB). Consider the RSVB formulation and note that KL > 0, therefore
increases, there is more incentive to deviate from the true posterior and choose Q E Q that maximizes expected risk for given E .A. Note that Q that places more mass near the 0 that maximizes the risk will be preferred over the one with more spread.
5 Discussion and Future Work
The RSVB formulation as stated in this paper requires us to solve stochastic minimax optimization problem compute decision rule. This is often difficult to solve, particularly the nonconvex- nonconcave setting of RSVB, and indeed, we are not aware of any computationally efficient methods for solving such problems in all generality [. To circumvent this, the authors in the LCVB literature maximize utility instead of minimizing risk, and thus convert the whole problem to much simpler max-max optimization problem. If we replace risk with utility in the RSVB derivation, we will also get max-max problem. In this formulation, as you increase 7, the incentive to deviate from the posterior and maximize the maximum utility increases. Therefore, increasing 7 corresponds to being more optimistic in making decisions than being more risk averse in the risk setting, as observed in the empirical results for the eight schools model in Section 4.3
We note that we assumed the lis-ffuctionn is lower-bounded, this is a natural assumption to place max-max problem would further require the risk
on risk functions. Converting our problem to be upper bounded, which is also assumed for the methodologies presented (without theory) in [5]. We emphasize that our theoretical results will continue to hold in this setting, but since our emphasis was on the statistical aspects of this problem, we chose to present our results with minimum assumptions. Developing an algorithm for solving general minimax optimization in RSVB without transforming it into a max-max problem is open and a part of our future work. Also an interesting topic for future work identifying minimum additional assumptions to theoretically characterize computational aspects of this problem.
10

---

6 Acknowledgement
We thank the anonymous reviewers for their helpful comments and discussions. This work is supported by the National Science Foundation under grant DMS-1812197.
References
[1] D. Bauder, Bodnar, N. Parolya, and W. Schmid. Bayesian mean-variame analysis: optimal portfolio selection under parameter uncertainty. Quant. Financ., 21 242 May 2020. doi: 10. 10..1000/110 202.666..200.1177771
URL httpp:://aoi..rg///// 108.1....00000116888880 2020
1748214
[2] D. Bertsimas and A. Thiele. data-driven approach to newsvendor problems. Working Paper, Massachusetts Institute of Technology, 2005.
[3] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians.
J. Am. Stat. Assoc. 12((51))8599 877, Feb 2017.
[4] T. Bodnar, S. Mazur, and Y. Okhrin. Bayesian estimation of the global minimum variance portfolio. Eur. Oper. Res., 256( (1:66))-:22777 Jan 2017. doi: 20.10166.eeoo:2216650.00 URL /ttpp::/ddi.orr/10...
org/10. 101110011.]111 eejor 2016.05.00
[5] Boucheron, G. Lugosi, and Massart. Concentration incqualities: nonasymptotic theory of independence. Oxford university press, 2013.
[6] F. Chick. Chapter subjective probability and Bayesian methodology. In Simulation, pages 225-257, Elsevier, 2006. doi: 10. (00100660092--0500000)))00000000 1016/s0927-000 (06) 13009-1
URL httpt:ttttsptooooooeeoo org/
[7] L. Y. Chu, J. Shanthikumar, and Z.-J. M. Shen. Solving operational statistics via a Bayesian analysis. Oper. Res, Lett., 36:06661110011100 Jan 2008. doi: 10.10166).011.200..0...00 URL
19t///00.116//.oorr https://ao..org/1..01116..
2007 04.00...0 010
[8] J. Diakonikolas, C. Daskalakis, and M. Jordan. Efficient methods for structured nonconvex- nonconcave min-max optimization. In International Conference Artificial Intelligence and Statistics, pages 2746-2754. PMLR, 2021.
[9] M. D. Donsker and S. S. Varadhan. Asymptotic evaluation of certain markov process expecta- tions for large time. iv. Commun. Pur. Appl. Math., 36(2):183-212, 1983.
[10] H. Föllmer and T. Knispel. Entropic risk measures: Coherence vs. convexity, model ambiguity and robust large deviations. Stoch. Dynam., 1(02003):333 351, 2011.
[11] Ghosal, K. Ghosh, and A. W. van der Vaart. Convergence rates of posterior distributions. Ann. Stat., 28(2):500-531, 2000. ISSN 00905364. URL httt::/tpp::..t: jstor org stable/ 2674039
[12] A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics. Int. Stat. Rev., 70(3): 419-435, 2002.
[13] M. Gil, F. Alajaji, and T. Linder. Rényi divergence measures for commonly used univariate continuous distributions. Inform. Sciences, 249: 124-131, 2013.
[14] Jaiswal, V. Rao, and H. Hounappa. Asymptotic consistency of rréény--apprxximale posteriors. J Mach. Learn. Res., 21(156): 2020.
[15] Kušmierezyk. J. Sakaya, and A. Klami. Variational Bayesian decision decision-making for continuous utilities. In Adv. Neur. In., pages 6395-6405, 2019.
[16] Lacoste-Julien, F. Huszár, and Z.. Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Int. Conf. Artif. Intell. Stat., pages 416-424, 201

---

[17] T. L. Lai, H. Xing, and Z. Chen. Mean-variance portfolio optimization when means and covariances are unknown. Ann. Appl. Stat., 5(2A), Jun 2011. doi: 10.1214/10-aous422 URL 1214/10
https://doi.... fttps:///oi.org/100. 1tt.://..rrr.....11111111110110. aoas422
[18] R. Levi, G. Perakis, and J. Uichanco. The data-driven newsvendor problem: new bounds and insights. Oper. Res., 63(6):1294 (11)::294-1306) 1306, 2015.
[19] Y. Li and R. E. Turner. Rényi divergence variational inference. In Adv. Neur. In., pages 1073 1073-1081, 1081, 2016.
[20] M. Lu, J. G. Shanthikumar, and Z.-J. M. Shen. Technical note-operational statisties: Properties and the risk-averse case. Nav. Res. Log., 62(3))206-214, 2015.
[21] D. Pati, A. Bhattacharya, and Y. Yang. On statistical optimality of variational Bayes. In A. Storkey and Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proc. of Mach. Learn. Res., pages 1579-
1588. PMLR, 09-11 Apr 2018. URL wtt.bttt//prrcceeinggeee//////// mlr press catess/v84/patti1 html
[22] M. H. Quang. Regularized divergences between covariance operators and gaussian measures hilbert spaces, 2019.
[23] R. T. Rockafellar. Coherent approaches to risk in optimization under uncertainty. In OR Tools and Applications: Glimpses of Future Technologies, pages 38-61. Informs, 2007.
[24] H. Scarf. Bayes solutions of the statistical inventory problem. Ann. Math. Stat., 30(2):490-500 1959.
[25] H. E. Scarf. Some remarks on Bayes solutions to the inventory problem. Nav. Res. Log., 7(4): 591-596, 1960.
[26] L. Schwartz. On Bayes procedures. Z. Wahrscheinlichkeit, 4(1):10-26, Mar 1965, ISSN 1432-2064. doi: 10. N0.1007/BF00535479 https:////// 10. 1007 10tp:///0000007//055547777
[27] A. M. Stuart. Inverse problems: A Bayesian perspective. Acta Numer., 19:451-559, May 2010. doi: 1010177/9962492110000061
URL. https://doo org/10..0017
80962492910000061
[28] R. E. Turner and M. Sahani. Two problems with variational expectation maximisation for time-series models. Cambridge University Press, 2011.
[29] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[30] A. W. der Vaart and J. H. van Zanten. Rates of contraction of posterior distribu-
tions based on Gaussian process priors. Ann. Stat., 36(3): - 1463, 2008 doi:
10. 001214//00053666000000006
https:/t:://dd org/ 10. 1214 /0900/00000000000001133
[31] Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. J. Am. Stat. Assoc., pages 1-15, Jun 2018.
[32] D. Wu, H. Zhu, and E. Zhou. A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM J. Optimiz., 28(2):1588118 1612, 2018
[33] Y. Yao, A. Vehtari, D. Simpson, and A. Gelman. Yes, but did work?: Evaluating vari- ational inference. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5581-5590. PMLR, 10-15 Jul 2018. URLAttps://proceeiing. mlr wtt.ppesss/..//////m yao18a html
[34] F. Zhang and C. Gao. Convergence rates of variational posterior distributioons. Ann. Stat., 48(4), Aug 2020. doi: 10.12 10.1214/19-aos 883. URL httpps//aaii..ogg//////////////
12

---

A Additional definitions
We provide the definitions of important terms used throughout the paper. First, recall the definition of covering numbers:
Definition A.1 (Covering numbers). Let P = (Po, E O) be a parametric family of distributions and d P x P + [0, oc) be a metric. An cover of a subset Px == {Po E K c }} of the parametric family of distributions is set K' K such that, for each 0 K there exists O' K' that satisfies d(Ps, Pp.) S The e-covering number of Pk is Nee, Pk, d) min(card( K' is an e-cover of K}, where card(-) represents the cardinality of the set.
Next, recall the definition test function [26]:
Definition A.2 (Test function). Let X., be sequence of random variables measurable space (8,, N. ). Then any Sh-measurable sequence of functions {Øn), On Xn [0, 1] Vn E N, is test of a hypothesis that a probability measure on St belongs to a given set against the hypothesis that it belongs to an alternative set. The test &n is consistent for hypothesis Po against the alternative pm E {P : E O%{00}} if Epm [0n] 7 [(0E0)100)) (0), 0,,0 θ as n + OC, where 1 1- is an indicator function.
classic example of test function is o 1 (KS. k. (0) that is constructed using the Kolmogorov- Smirnov statistic KS,, sup, (t) Fo(()), where (t) and (() the empirical and true distribution respectively, and Ku is the confidence level. If the null hypothesis is true, the Glivenko- Cantelli theorem 22] Theorem 19.1] shows that the KS statistic converges zero as the number of samples increases to infinity.
Furthermore, define the Hellinger distance h(01, 02) between the two probability distributions Po. and Poş is defined as dz(01 02) (J (VdPo, vdPo.) 1/2 We define the one-sided Hausdorff distance H (AAlB) between sets A and B in a metric space D with distance function is defined as:
B) sup (x, B), where da B) inf d(r,y). yEB
zеλ
Next, define an arbitrary loss function La θ x θ R that measures the distance between 02} E O. At the outset, we assume that Lu (01, 02) is always positive. We models (P%, Po, [v(0,, (V)))
define {<a} as sequence such that -
0 as
and ne 2 1.
We also define
Definition A.3 (T conveerrrg.. A sequence of functions Fn U + RR for each n E N, T- T-converges to F :4 + Rf
for every u E U and every {un, n E N} such that un - u, F(x) S lim inf Fn(un);
for every u E U there exists some (un,n n. E N} such that un - u. F(x) 2
lim supp--ooc Fu (un))
In addition, we define Definition A.4 (Primal feasibility). For any two functions f u + R and : U + R, a point u" E l is primal feasible to the following constraint optimization problem
inf fu) subject to b(u)= S c,
well
if (fb(")) S for a given e R.
B
Applications
B.1
Single product newsvendor problem (cont.)
First, we fix the sieve set θ,, (c) e which clearly implies that the restricted inverse- inverse-gamma gamma prior
I[)), places no mass the complement of this set and therefore satisfies Assumption 2.2
Second, under the condition that the true demand distribution is exponential with parameter Og (and
Po = Po.). demonstrate the existence of test functions satisfying Assumption 2.1
13

---

Lemma B.1. Fix n 2 5. Then, for any e > Cn := 1 with en + O, and ne 2 1, there exists
test function On (depending t) such that L (0, 0o) = (suPOEA |R(a, 0) R(a, 0o)l)²
satisfies Assumption w/th Co 20 and C = C1
(K) ) for constant C1 > 0 and KNV
du (T,6u)- (0) +)
(b h)² c-- 2- 2 1/2
80
The proof of the above result follows by showing that dy = nn n (8, 0b) can be bounded above by the Hellinger distance between two exponential distributions on (under which a test function exists) in Lemma C.10 in the appendix.
Third, we show that there exist appropriate constants such that the inverse-gamma prior satisfies Assumption 2.3 when the demand distribution is exponential.
Lemma B.2. Fix n2 2 2 and any X > 1. Ler A,, == {0 E θ e +1+a
S Cyne where D1+> (Po || is the Rényi divergence between Po and Pg. Then for e log and any C3 > such C2 = aC's 2 2, the truncated inverse- inverse-gamma gamma prior Inv - T a, B) satisfies II(A)) Z
exp(-nC2c,) 2,,, vn 2 n2.
Fourth, it is straightforward to see that the newsvendor model risk R(a 0) is bounded below for given a E A.
Lemma B.3. For any E A and positive constants h and b, the newsvendor model risk R(a, 0) h ha ① + (b 11++)==- --00))))) 2 (144207)) where a == minsa E A} and 0* satisfies h (b
h)e-a0"
a0" (1 a1+) O.
This implies that R(a, 0) satisfies Assumption 2.5 Finally, we also show that the newsvendor model risk satisfies Assumption 2.4
Lemma B.4. Fix n 2 1 and > For any E > en and any a E A. R(a, 0) satis- fies En (a(aa [B800000 S expl (xpp-Co(7)ne"), for any Ca(r) > 24 (ha + +) and
Cs(7) = C1(r) - 27 (hā + )), where ā == max(a E A}.
Note that Lemma B.1 implies that C Zk for any constant C₁ > O. Fixing a and
using Lemma B. 2 we can choose C2 C3 = 2. Now, Ci can be chosen large enough such that
C > C4(7) + Cs (7) for a given risk sensitivity ? > 0. Therefore, the condition on constants
in Theorem 3.1 reduces to C5(7) > 2 + C2 + C3- 5, and it can be satisfied easily by fixing
C5(7)) = (say).
These lemmas show that when the demand distribution is exponential and with a non-conjuse non-conjugate truncated inverse-gamma prior, our result Theorem (32 can be used for RSVB method to bound the optimality gap in decisions and values for various values of the risk-sensitivity parameter y. Recall that the bound obtained in Theorem 3.2 depends on </ and 17, (P (7).
Lemma B.2 implies that e log n but in order to get the complete bound we further need to characterize 7/ (7)). Recall that, consequence of Assumption 3.1 in Proposition 3.1 for a given Cs infqee infae EEq [M(a, 0)] that C. > 0 and (7) S 'n Cs + Cge!""
Therefore, in our next result, show that in the newsvendor setting, we can construct a sequence (0) C Q that satisfies Assumption 3.1, and thus identify é', and the constant Cg. We fix Q to be the family of shifted gamma distributions with support [T, oc).
Lemma B.5. Let (Q, be a sequence of shifted gamma distributions with shape parameter a = n and rate parameter b = Ho' then for truncated inverse gamma prior and exponentially distributed likelihood model
1 KL (Q)())(1())) + BQ,, KL (dP"((X))) dPp dP (X.))] < C2
(X.)))
n2
where c2 logn and C. max (0,2+ 28 parameters are chosen such that C. > 0. An
log /2T log (M(,)) + a log 6b) and prior
14

---

B.2 Multi-product newsvendor problem
Analogous to the one-dimensional newsvendor loss function, the loss function in its multi proti-product version is defined as
( bf a)+
where h and are given vectors of underage and overage costs respectively for each product and mapping (+)+ is defined component-wise. We assume that there are d items or products and & E IR denotes the random vector of demands. Let a E A C R4 be the inventory or decision variable, typically assumed to take values in a compact decision space A with a == {(minfa ag : a, E ) and ā := {{max{a; : a, E A.) and a > O, where A, is the marginal set of 1th component of A. The random demand is assumed to be multivariate Gaussian, with unknown mean parameter E IR' but with known covariance matrix E. We also assume that Σ is a symmetric positive Q" AQ, where Q is an orthogonal matrix and A is diagonal matrix consisting of respective eigenvalues of Σ. We also define A m Ai and A min; Au. The model risk
definite matrix and can be decomposed
R(a, 0) = Ep [[((a)))]] = Σ
Eps [h,(a; - + b.(E: a;)+1
i=1
(as- (a)- 0s)
Σ
(h; + b;)a,
b:0; + 0, h₂)
Tíi
(a: (a.-0.)
(as (a)-
Φ
+ Oz
h
+
(11)
+b
(a: 1 0z)
1 + (a;-0s)
Onl
which convex a. Here Po; is the marginal distribution of & for 7th product, x(-) and (.) probability and cumulative distribution function of the standard Normal distribution. We also assume that the true mean parameter 0g lies in a compact subspace C IRd, We fix the prior to be uniformly distributed on θ with no correlation across its components, that is T(A) = mi m(Ane)
m(
II m(A, where m (B) is the Lebesgue measure (or volume) of B C Rd As in the previous m(O.)
example, we fix the sieve set On (e) = O, which clearly implies that II(0) places no mass on the complement of this set and therefore satisfies Assumption 2.2
Then under the condition that the true demand distribution has a multivariate Gaussian distribution (with known Σ) and mean 0u (Po Pe.)). we demonstrate the existence of test functions satisfying Assumption by constructing a test function unlike the Lingl-produs newsvendor problem with exponential demand.
Lemma B.6. Fix n 2 1. Then, for any E > En == t/ with en + 0, and ne, 2 and test function On.e == x., 8..:10.- - 00 0o1l= L (0,00) = n (SuPOEA R[(a, 0) R(a, 0o)1)
satisfies Assumption 2.1 with Co 1, C1 4K2C and C 1/8 @ -1) for sufficiently large Ö dA 1) such that C and A m1,,,( Ass where K s,e 10, R)a
In the following result, we show that there exist appropriate constants such that prior distribution satisfies Assumption 2.3wnen the demand distribution is a multivariate Gaussian with unknown mean.
Lemma B.7. Fix n2 2 2 and any X > 1. Les A,, == {0 E O DI+A (Po) P S Czne ). where D1+A (Po || PP)) is the Rényi Divergence between Pe and Pt. Then for en log and any Ca O such that C2 M(+1))[IL_ m(e,)) C3 2 and for large enough n, the uncorrelated uniform prior restricted to satisfies (An)) 2 exp(-nC2e))
n
4d
Next, it is straightforward to see that the multi-produce newsvendor model risk R(a, 0) is bounded below for given a E A on a compact set ② and thus it satisfies Assumption 2.5 Finally, we also show that the newsvendor model risk satisfies Assumption 2.4
15

---

Lemma B.8. Fix n 2 and y > 0. For any > cn and E A. R(a, satisfies (Cia.)))aaaC G(a00) I exp(-Cs(r)nne) any Ca(r) > 2ysuppa G(a, and Cb(r) = C4(1) 27 sup (a.0)a))AAs C(a,0).
Similar to single product example, in our next result, we show that in the multi-product newsvendor setting, we can construct a sequence {Qn (0)) E Q that satisfies Assumption 3.1 and thus identify <, and constant Cg. We fix Q to be the family of uncorrelated Gaussian distributions restricted to e
Lemma B.9. Let (0) be sequence of product of d univariate Gaussian distribution defined as 20 me,
(0) O 2 E .1..,..., d}. Then for uncorrelated uniform distribution restricted O and multivariate normal likelihood model KL (Q)(())(I(()) Bo,,,0)) KL
Le. N. and fix oin = 1/v/ and 0; = for all N( (i....95)))
]] Cych. where max (0. 0 Σ [log( 2Te) log(m(O.))) | +
E Log and Cg 2
Now, using the result established in lemmas above, we bound the optimality gap in values for the multi-product newsvendor model risk.
Theorem B.1. Fix > O. Suppose that the set A compact. Then, for the mulli-product newsvendor model with multivariate Gaussian distributed demand with known covariance matrix E and unknown mean vector lying in a compact subset θ C. IR prior 11(-) =II m..))r and the variational
.m(3.)
family fixed to uncorrelated Gaussian distribution restricted to O and for any T > 0, the Po
probability of the following event Xnn R(ags, 0o) - in R(zz, 00) S (r) n 1/2
is
at least T for sufficiently large and for some mapping M' R IRt,, where R(-,0) the newsvendor model risk.
Proof. The proof a direct consequence of Theorem 3.2 Lemmas B.6 B.7 B.8 B.9 tion
and Proposi-
3.2
B.3
Gaussian process classification (cont.)
We define the distance function as _. 0g) (supae.A [R(a,)) R(a, 0o))) In anticipation of demonstrating that the binary classification model with GP prior and distance function LGP satisfy the desired set of assumptions, we recall the following result, from 30). which will be central in
establishing Assumptions 2.1 2.2 and 2.3 2.3
301 Let 0(-) be a Borel measurable, zero-mean Gaussian random element in a separable Banach space (0, - Il) with reproducing kernel Hilbert space (RKHS) (H. (lx) and let Og be contained in the closure of H in θ. For any > en satisfying yo, (e) S ne² where
Lemma B.10. [Theorem 2.1
(c)
inf KEB: [BolEE
|M log I[[[@]] < <)
(12)
and any C10 1 with e- -Cionc < 1/2, there exists measurable set On(e) θ such that
log N(3e, Or,(), II) s 6Cigne",
(13)
II(A (c) < Cione?
(14)
I Boll < 4c,) 2 2
(15)
The proof of the lemma above can be easily adapted from the proof of [30 Theorem 2.1]. which is specifically for Cn. Notice that the result above is true for any norm || on the Banach space if that satisfies Pn, (c) S ne?. Moreover, if PP, (en) S ne? is true, then it also holds for any e 6n. since by definition spa, (t) a decreasing function of
All the results in the previous lemma depend on (e,(() (e) being less than ne? In particular, observe that the second term in the definition of spe (E) depends on the prior distribution on θ. Therefore,
16

---

30 Theorem 4.5] show that (xx) I ne, ( (with || as supremum norm and for as defined
later in 9 is satisfied by the Gaussian prior of type
J. 2
W(-)
ΣE j=
My Mg
(16)
where {1y) } is sequence that decreases with j. (Z4 are i.i.d. standard Gaussian random vari- ables and } form a double-indexed orthonormal basis (with respect to measure v), that is E - [()=))==)) Ja is the smallest integer satisfying 2 = n/d((a++d) for given > 0. particular, the GP above is constructed using the function class that is supported on [0, 1] and has wavelet expansion, w() -i=1 Ek=1
00 2/ (z.(0z((.). The wavelet function space is
equipped with the Lo-norm: ||æl||2 Ej=1 Σ Ek= |,, p) 1/2 : the supremum norm: || oc = -5=1 2dd maxi Skg2i0 |w,,a|: and the Besov (3, oc, oo) -nomm: || supıs 21 max1gk522 |vtyal Note that W induces a measure over the RKHS H. defined as a collection of trun- cated wavelet functions w(.) =
Σ/=
2/d j= k= (.))) with norm induced by the inner-product on 11 + can be easily derived
H as 1 -
23d km
w R. 15
The RKHS kernel K [0, 1]dt [0,
as
Ja
Ja 2d
}
=(z.))=E[<<>)<> =
ΣΣ
Mg 3 sA(0))
Σ
u 5 9. (z)
j=1 k=1
2
f (1000000
Indeed, by the definition of this kernel and inner product, observe that (K(z, -), w()) = Σ k= Wy 2 = w(z). Moreover, (K(z,)) Kv, ))) =
2/d E/H ,
23 K(x, v). is clear from its definition that W is centered
Gaussian random field on the RKHS.
Next, using the definition of the kernel, we derive the covariance operator of the Gaussian random field W. Recall that Y ~ v. which enables us to define the covariance operator C, following (27 (6.19)
(cr) Joo.1" K (xx, (t) (y)d Also, observe that (v3: [9j,a] } is the eigenvalue and eigen function pair of the covariance operator C. Consequently, using Karhunen Loéve expansion [27 Theorem 6.] the prior induced by W on H is a Gaussian distribution denoted as (0, C). We also recall the Cameron-Martin space denoted as lm(C1/2) associated with Gaussian measure (N0, C) on HH to be the intersection of all linear spaces of full measure under (0,CC) C) (7 (page 530).. In particular, (2 is the Hilbert space with inner product (,, :)c = (C 1/2, (( C//22).
Next, we show the existence of test functions in the following result.
Lemma B.11. For any > On with 6, -> ne? 2 2 2og 2, and po, (c) S nc², there exists a test function on (depending on E) such that GP (0, 0b) = n (supae ae with C' = 1/6, Co = and C₁ = (max(c+,cc.) c- ))
[R(a, 0) R(a, (o)) satisfies
Assumption 2.1
Assumption 2.2 is direct consequence of (14) in Lemma B. 10 Next, we prove that prior distribution and the likelihood model satisfy Assumption 2.3 using (15 of Lemma B.10
Lemma B.12. For any X 1, let A,
{0 ( D1+a PP)) S Czne)) where D1+A (PT () is the Rényi Divergence between Pa and Pp. Then for any > 6n satisfying
(o, (c) < ne² and C3 = 16(A - 1) and C2 = 1, the GP prior satisfies II(A,) 2 exp(-nC2e))
Assumption 2.4 and 2.5 are straightforward satisfy since the model risk function R(a, 0) is bounded from above and below.
Now, suppose the variational family Ocp is class of Gaussian distributions θ defined as N (m,, Cg), m2 belongs to θ and Cg is the covariance operator defined as Cy C1/2 (I- 1/2
17

---

for any S which is symmetric and Hilbert-Schmidt (HS) operator on è (eigenvalues of HS operator are square summable), Note that S and my span the distributions in Qar.
The following lemma verifies Assumption 3.1 for specific sequence of distributions in Q.
Lemma B.13. For a given J E N, let {Qn} be sequence variational distribution such that Qn is the measure induced by GP. Wo(.) = 0g (v) Ej= 27
( where
()) E Ex 00:100 and S} 1
1 Then for GP prior induced by W
Σj=1 Ek=114 (i:=11i72j,0 Uz. and j4j = =2-30/2-ja for some a > 0. 110011::0,,0 < oc, and fg (y) lie in the Cameron-Martin space lm(C2/2), we have Coea where cn is defined in and Cg = max |10013,0c.00 2- 2-2ja-20
Cg)ll C)) n LQ (0' 0 PT) S
2 /22 - 1), 1),C*)
1-2-a
where C" is a positive constant satisfying [[0o(y) - 0g (1) C²2 2JB.
Using the result above together with Proposition 3.2 implies that the RSVB posterior converges at the same rate the true posterior, where the convergence rate of the true posterior is derived in [30 Theorem 4.5] for the binary GP classification problem with truncated wavelet GP prior. Finally, we use the results above to obtain bound the optimality gap in values of the binary GP classification problem.
C Proofs
C.1 Alternative derivation of LCVB
We present the alternative derivation of LCVB. Consider the logarithm of the Bayes posterior risk,
log _TT (cxp(R(a,)) = log
())
dQ(0) e dQ(()
log
exp( ((a, ())
dQ(0)
2 dQ(0) log exp(I((a, ())x)())X =: F(a; (17)
where the inequality follows from an application of Jensen's 's inequality (since, without loss of generality, exp(R(a,,)) 0)) > 0 for all a E A and E 9), and Q E Q. Then, it follows that
[exp( min log E (1 (exp( (R(a, 0)) 2 min max F(a: (.)),
aeAA
aeA
min max KL (00))]II((XXx))) AEA TEΩ
R(a, R(a.0)dQ(O)) (18)
C.2 Proof of Theorem 3.1
We prove our main result after a series of important lemmas. For brevity we denote CR, (0) p(8_10)
p(X..00))
Lemma C.1. For any d' A, 7 O, and <> 0.
( ( (0, (0)xa)
Erg
Ri(C'00) LR,, (0,00,,0lT))) (o)
< <log Erg
+ inf Epn Qee
(I((0()
Jo ea CR, (0, (TT
EolR(a, 0)]
+ log Epg [/ e RBae'
LR, (0, 0p (TT
-y inf a
(19)
Je CR,, (0, dlT((
18

---

Proof. For any fixed a' E 7 > O, and C > 0, and using the fact that KL is non-negative, observe that the integral the LHS of equation satisfies,
(T ooo (018.) (0,00,) s 20,,,,118, log (l..(00.0))
eCh,, dII(0).nn)
KL
dQ (012))
So 2ch,10000)e dlI(o)
o (01X.) [log (l.n
+ +og (Ch,.(1.00)) 2
dQ's (01x))
+ B B,r (81xa)
log c00,,10,00) Ra 0) dII dII())),,)
dQa (01xxn)
log ETT (1 + +Ro t0; (0
log
en (I1(0)
Next, using the definition of Q- (0) Xx) the second term of last equality, for any other Q(-) E Q
(cLLn(t.en) R((a' ,0] EQ log
dQ(0)
(012,,) (0, 06) < logiEn,
en 9) dII X
Finally, it follows from the definition of the posterior distribution that
(o (..) (0, 0n)]
S log (SLo(0.80)
(0, (O) R(d' e
EQ 1og
dQ(0)
Se CR,,, (0, Go ())
en ddI(0)
( (Shn(0.80) Ri .)) (0, 0n dII
dQ(()
log
+ Eq log Rr (I
Tgenila' 8) LR,, (0, ()))((()) + log ciR(d'00) .0) CR, (, Go) (T)
(20)
CR, 8 (I)
where the last equality follows from adding and subtracting log En [e .8) LR,, (0, Bn) Now
taking expectation on either side of equation (20) and using Jensen's inequality on the first and the last term in the RHS yields
Erg
(o (1p8.) (018.)) (0,(0,0)) (0,00)]
< log pr I cstunossss e)R(d LR, (0, 0o) (o)
+ inf Epg (L(QIIT,,) QEQ
So eTR(al00) () LR (0,000,,0000) (o)
[ Rid' .)
CR, 0 (T) So CR, (0, G (dI(()
inf EQ [R(a, 0]] aEA
+ log
(21)
where in the second term in RHS of (2).. we first take infimum over all a E A which upper bounds the second term in 20) and then take infimum over all Q E Q, since the LHS does not depend Q.
Next, state technical result that is important in proving our next lemma. Lemma C.2 (Lemma 6.4 of [34). Suppose random variable X satisfies
P(X 2 t) S c1 exp(-c2t),
for all t 2 to > 0. Then for any 0 < 3 S c2/2.
E3xx)(3XX] & cxp( + Ct
Proof. Refer Lemma 6.4 of 34
19

---

In the following result, we bound the first term on the RHS of equation 19). proof are essentially similar Lemma 6.3 in 34
The arguments in the
Lemma C.3. Under Assumptions 2.7 2.2
2.3
2.4
and
2.5 and for (ai + Cs(n)) >
C2 C3 + C4(7) + 2 and any E 2 Cn.
Ery 1 (SLul(ORe) CR (8, 8 (I)
e. LR, (0, (o)
S eCCine" + (1+Co+3W-) (22)
for 0 < < C10/2, where Cl0 =
for any X >0. 0.
Proof. First define the set
Bn = Xn, / CR, (0, (000)) 2 e e II(AA))
(23)
where set A, is defined Assumption 2.3 We demonstrate that, under Assumption 2.3 PB (B is bounded above by an exponentially decreasing(in n) term. Note that for A, as defined in Assump- tion 2.3
1
CR (0, (0) S - -11--5)mm))
P6
[I(.An)
CO, (d) S e- e-(t-Canee)
s P
(24)
(I())) JenA.
Let diT(0)= = 1 I(ena, (0)
(f), and use this in for any X to obtain,
I((A,)
(1/ |CR, (0, 6o (f) s < ) =PP Pö (1 (
2 ((+tC))dne)
Po
Then, using the Markov's inequality in the last equality above, we have
po (L, (0, d() se- <e-(++Ca)me))
< e (1+CGa)) Epp
Epp (1. (1,c.100000))000000 "
S (1+Ca)Ame
Epy (ICR >) diio)
=e-(1+G))mm exp(ADx*1 (P
=e+(Caa) ACanch -nn
(25)
where the second inequality follows from first applying Jensen's inequality (on the term inside
(-)
and then using Fubini's theorem, and the penultimate inequality follows from Assumption 2.3 and the definition of (I))).
Next, define the set K,, :- (0 e L..(0,0)) > Cine"). Notice that set K₂ is the set of alternate
hypothesis as defined Assumption 2.1 We bound the calibrated posterior probability of this set
Kn, to get bound on the first term in the RHS of equation (19). Recall the sequence of test function {dn,e} from Assumption 2.1
Observe that
SK. R(a 8) (0, (O)
Epg
So eniiial0) (0,00))
(Gn.e + 1- 1- SKC, Ridd LR, (0, Go (o)
= Epp
(Ønne
So er LR, P (IT()
< Epg [dn,e] + Epp [11 - [BC]
JK. eiR(a' .) LR,, (0, 6o) (oT
+ Erg
(1-
(1 ( [n,e) 1 Bn So e)Ria'
LR, (8, 0,) (TT
SK. ei CR, (0, 6o) (0))
< <Epron. Ep [[ Epp
(I - 1 B,
(26)
So (Raii () LR (0, 0 (TT(
20

---

where in the second inequality, we first divide the second term over set B, and its complement and then use the fact that Jx. e.iiiee,0) ;0; C.R, (9 (
Jo e o) CR., (0.06 (6 I 1. The third inequality is due the fact that
On.e E [0, 1]. Next, using Assumption 2.3 and 2.5 observe that on set B,
[ (ila' .0)
/ (R,
2 W
2 W (s) 2 Wee
Substituting the equation above in the third term of equation (6). we obtain
Epy (1 - Φn,e) 1. JK. e .0) CR, (0, to (0)
Io er CR. 80 dll
(1 2n.c) Br / eiRla .) CR,, (o)d1((0)
< W Epg
Kn
< e Epg
(1 - @nce / eiB(dd LR, (0, 0o) do)
Kn
Now using Fubini's theorem observe that,
( = - erRid
{
exRla'D)BEg Bpg [(1 - @n,e)l dII
Kn
< ( U UKunternie.osssccomo Epy [11 dl1())
+e Calaailee
eRlid' Ma1((o)
where in the last inequality, we first divide the integral over set (0 E θ Ri(a'00) s Cals)mee) and its complement and then use the upper bound on e .0) in the first integral. Now, it follows that
(+) < W (
Epg [(1 mn,e)) dII
Kn
+e -Ci(r)ne
eriidd ertido0 (di((o)
[erA(u (0 (
Epp [(1 - xn,)) d(T(((((((
Kan0a(r)
Ep [(1 [n,c)] dIT(() + -cbbine (e:1Baa00)sc
Ridl' (TTT(())
k.noa(e)
U
JK.neate) Epp [[1 - 6n.c)l dII(() + (c))()))
+ Ce-limme (onRear Case
Ridd R aa10))
where the second equality is obtained by dividing the first integral (c) and its complement, and the second inequality is due the fact that Φn.e E [0, 1]. Now, using the equation above and Assumption 2.1
2.2 and 2.4 observe that
Sx_ (.R(dd CR, 0 011 e CR, (0, G (
Epp (1 dn B,,
w (w 2e Cnee + -e- (cM))Ccc))
Hence, choosing C, C2 Ca, C7(7) and C5(7) such that m+C=++C++Ca()mmiin(CC Css Cs(7))) implics
21

---

SK. eridd CR,, (.000,00000 (0)
Epg (1 @n,,) Br
3W - ne?
(27)
Jo eriiaar 0) CR, (0, Bo ddll()()))))))))))))){))
By Assumption 2.1 we have
Dn. S Coe One" -nne
(28)
Therefore, substituting equation (25 equation 27 and
(28 into (26). we obtain
JK.
en 8) (O, Oo (dT(()))
Epg
< Co + 3w " -CoCCnn"
(29)
Jo er (0, 0o) dT)()))
where Ciu = min(d, C,13/C,. Using Fubini's theorem, observe that the LHS in the equation (29 be expressed j((n),, where
(R, 0b)
du(0) Epø
II(0)e 2 R(a' M) d0.
[o ex 0) (0, (1)
Next, recall that the set K = {0 + : Ln (0,0)) > Cine), ). Applying Lemma C.2 above with
X (0, 0o), c1 (1 Co + 3W -) C2 C10 to Cinen and for 0 5 C10/2, obtain
Epp
/ (
erilla 0) CR, (0, 0o (1)
d0
< efCinet + Co+ (30)
So er CR (0, G (lT
Further, we have another technical lemma, that will be crucial in proving the subsequent lemma that upper bounds the last term in the equation (I9)
Lemma C.4. Suppose a positive random variable X satisfies
P(X 2 e²) S a exp(- (C2 + 1)t).
for all t to 0, O, and 02 0. Then,
e E[ s exp(to)
C2
Proof. For any Zu 1,
000
E[ < Zo +
P(X 2 x)dr
Zo
1 exp(-cau)
=oo +
P(X > dy S Zo +
Jn
In 2o
Therefore, choosing Zo exp((u).
E < exp(to) + c
exp((ccto) s exp(to) + 4 C2
02
Next, we establish the following bound on the last term equation 19
Lemma C.5. Under Assumptions 2.1 2.2 2.3
2.4 2.5 and Cs(7) C2 C2+2,
e2 8) LR, (0,00)(IT (I)
Ep pm
Ca(t)me? + 2C)()))
(31)
SS (0,0%, dl))
for any X 121+C1())
22

---

Proof. Define the set
M, {0 ④ : BB > calt)ne }.
(32)
Using the set Bn in equation distribution satisfies,
23 observe that the measure of set M,, under the posterior
JM, C0, (0)
JM, CR, (0)
E
Se CR,(0, 8 ())
- Epg
Bg
+
Epy
LB.
Je (0, 8o dll
(33)
Now, the second term of equation (33) can be bounded as follows: recall Assumption 2.3 and the definition of set B,, both together imply that,
SMN (0, 0o (o)d)(()) 1 Bn
E P
(1 (++22+0s)ne
Epp
1 J. CR, (0,00 8 80)
So CR, (0, dd)((())
Mr
S (1 Cl Epp Uhe Mn.
(0,(0)0)))
(**)
Then, using Fubini's Theorem (**) e(1+0++0s)me (TMM. Next, using the definition of set M and then Assumption 2.4 we obtain
SM, CR. 0n (0)
3 el+tsstCs)mee C- f ex .0) 0d1((())
EE pm
1B.
Calt)me S (14 e -cll)mme -ooltimee
So CR, (0, (n)
Mn.
Hence, choosing the constants implies
1+ C2 C3 C1(7) and C5(7) such that -1 -1> C2 C3 - C3()
SM. LR, (0, Gu dT)
S -1+CC((()))
E Pp
1B
(34)
So CR, (0, O do)))
Therefore, substituting
25 and
(34
into (3)
SM. CR, (0,000((1(0)
Epg
S 2e 2e- (()))
(35)
Jo (O, 0o dd1(((
where (7) min Ca(()))CC(() 1. Using Fubini's theorem, observe that the RHS in (35) can be expressed as v(MM), where the measure
(R,(0,00))
du(() = Epr [T CR, (0,00) (TT
dTT()).
Applying Lemma C.4 for X llal' (0) .0),cc we obtain
2, C11(7) to Ca(7)net and 1+
I P.
erl(o'0) LR, (0, On )d1(())
Cila)ne? + Cn(r)
2
S Cs(t)nec + +C44()). (36)
Is LR. (0, 8 (II
Proof. Proof of Theorem
3.1 Finally, recall 19
<Erg Vc [n 0o) x,)
1 (chuleeww
edala 0) LR, (0, 0o (T))
log Ern
+ inf Egg QEO
KL(Q|
Jo er LR. (0, ( (TI(
[ Rlld'
LR, (0, dTT
- inf Eo(R(a,0) 8)]
+ log En
(37)
aeA
N LR, (0, 6 dTT(
23

---

Substituting 31 and
22
into the equation above and then using the definition of nn (n), we get
Bpg
(0, aQa ( (0)Xx)
log(esCimei + (1 CD+ +3W + log (e (c(()))))(((()))) + (1))
(1+
1 (alt)) nez s nn (7)
(1+ Co + 3W 3el + (2)(()))
where the last inequality the fact that log S 1. Choosing < C10/2 2 = mint C.A..))
201
Erg 1 I. 00) dQa (0
$ (M())((()) + M' M'nn + 2(1+ + Co + 3W- S ne + 4 Ccat)mm
(38)
C10
where (() Ci C2()) and M' & depend on C, C1 C1(7), W and X. Since the last two terms in (38) decrease and the first term increases as increases, we can choose M' large enough, such that for all n 2 1
2(1 + Co + 3)
4Cr(1)
M'mn (7)
Cio
Cio
and therefore for M 2M',
Epg Vo L.((0.00) 00) (0
(M())n((() + Mun (7).
(39)
Also, observe that the LHS in the above equation is always positive, therefore M(7) + M1) 2 0 vn 2 and y O.
C.3 Proof of Theorem 3.2
Lemma C.6. Given d' A for constant M, as defined in Theorem 3.1
R(a, 0n)| < [M(()) Mn
Epg sup LaEA
(018.) (R(,
(40)
Proof. First, observe that
E R (1.,0) R(a, 01) R(a,0))
2
sup R(a,, 0) R(a, 0o)
sup a
(018.))
aEd
27
<B (81x.)
sup (aEA
[R(a,))
M0l-R((a00) R(a, 0o)1
where the last inequality follows from Jensen's 's inequality. Now, using the Jensen's 's inequality again
[R(a, 0)] -R(a, 0b) 8b)
E
sup Lee
(
( E B (018.) LaEA
R(a, 0n))
< Ep
(01,) (M(a,
Now, using Theorem 3.1 the result follows immediately.
24

---

Proof of Theorem 3.2
Observe that
R(ass: 0o) inf R(2, 0o) zEA R[ (p)
inf R(2, 0n)
SEA
R(añs: On)
Boitt (01x.) [((ars:)) 0)] + R o (018.)
(018.) PR(aps: 03] inf R(z, 06)
sEA
s Ri R(aags (o)
Bo (0)2..)) [R(ass: 07] + B (018.) (R(añs: 01] inf R(a, 0n)
/
S 2 sup acA
()dQ% ( R((,
(41)
Given a E A and for constant M (defined in Theorem 3.1 have from Lemma C.6 for a' ajs
/
Ep
sup
R(a, (00, - Ri(a, 0n)
< [M(7) + Mn (x))
(42)
LaEA
It follows from above that the P probability of the following event is at least - --1
{
Xn R(aas: Ou) inf R(2, 0o) s 27 [M()) + Mn)(1)]] f
(43)
C.4 Proofs in Section 3.1
Proof of Proposition 3.1 Using the definition of n (7) and the posterior distribution 11(0) X...
observe that
mm (7)
inf E KL(QIII,,) - inf REAA
Eq R(a, 01]
QEO
KL(QIIII)+ + / dQ(()
[dI1())(XN1)))
inf Epg QEQ
- inf EQ (a(a 0)] aEd
1(
[ [dTT())((x10))
inf QEQ
KL(QIII) Y inf (aa 0)] + Epp EQ
log
aa
p(X,100)
Now, using Fubini's in the last term of the equation above, we obtain
()) inf QEQ
(L(Q())|[)(()) - inf EolR(a, 0)]
aEA
KL (dP(llo(X.100) KL (apa)) [ [
+ Eq
(44)
Observe that, fe... (1x00)p(x,10)akX Since, KL is always non-negative, follows from the equation above that
nn (())
1 inf
<
(L((())(IT(8)) - inf Eq Ra, 0)] EQ KL II(
n QEQ
aEA
1 inf QEQ
(L(()())IIT))) + Eq KL 1pp X,, 10)) ]]
inf inf Eo(A(a, 0)} (45)
QEeaEA
where the last inequality follows from the following fact, for any functions f(-) and g(.),
inf(f g) < inf f inf g.
25

---

Recall c 2 1
Now, using Assumption 3.1 it is straightforward to observe that the first term
v
in 45
1 inf KL (L(()))))))) BQ KL (dPp (el]ll(())))))))))))) I Cgc n QEQ
(46)
Now consider the last term in (45) Notice that the coefficient of independent of and is bounded from below. Therefore, there exist constant Css infqee an lBq R(a, 0)], such that with equation (46) it follows that 1/1 (7) I - Coe and the result follows.
Proof of Proposition 3.2 First recall that
non (1)
(1)
inf pp
KL(Q(0) - inf Eq R(a, 0)]
QEΩ
aEA
inf E Eg
[L(Q(0) ((())
- inf EQ [R(a,)) aEA
(47)
QEΩ
Observe that the optimization problem is equivalent to solving
Epg QEQ
min
s.t. inf aEAA
Eq Ra(a, ()] S O.
(48)
Now for any > 0, (0) E Q that minimizes the objective in (47) is primal feasible if
/
inf aEA
(Q) 0) 0.
Therefore, is straightforward to observe that as y increases (*)) decreases that is
Epp /
(0) log dQ, inf
aEA aEA / (Q;(0))((a)0)
dl1(0
C.5 Sufficient conditions on R(a, 0) for existence of tests
To show the existence of test functions, as required in Assumption 2.1 we will use the following result from [T] Theorem 7.1], that is applicable only to distance measures that are bounded above by the Hellinger distance.
Lemma C.7 (Theorem 7.1 III.. Suppose that for some non-increasing function D(e), some en > 0 and for every En.
N : 0,) s 2e} m) S D(E),
where m(.,.) is any distance measure bounded above by Hellinger distance. Then for every c >> Cu. there exists test on (depending on 0) such that, for every
(-2ne²
Egg [+,] < D(E) exp
exp (- and
- n -
sup (c)
Eppl [1 @nl S exp
Proof of Lemma C.7 Refer Theorem 7.1 of
For the remaining part of this subsection we assume that C IRd. In the subsequent paragraph, we state further assumptions on the risk function to show Ln(-,)) as defined in O satisfies Assump-
tion 2.1
For brevity we denote n -1/2 1L/22/L,, Ln,( 0o) by di (0, 0o), that is
26

---

(6(01.02) == sup Ri 81) - Ría, 82)1, 8(01, E θ aEd
(49)
and the covering number of the set T(e) := (Po dr. (0, 0o) < c) as (ó. T(e), dr.,, where 8 >> 0 is the radius each ball in the cover. We assume that the risk function R(a,) .) satisfies the following bound.
Assumption C.I.
The model risk satisfies
dr. dr.001,2)) S Kid (0, 0o);
where du (01,0))) 02) is the Hellinger distance between two models Po, and Po.-
For instance, suppose the definition of model risk is R(a, 8) - [ ((x; ())(y)0)dr, where {(x a) an underlying loss function. Then, observe that Assumption C.1 is trivially satisfied if ((x, a) is bounded in x for given AA and AA is compact, since dz (01 02) can be bounded by the total [dPa, (x) - dlo (a)) and total variation distance is bounded
variation distance (0 02) above by the Hellinger distance (12). Under the assumption above it also follows that
apply Lemma C.7 to the metric (- :) defined in 49 Now, we will also assume an additional regularity condition on the risk function.
Assumption C.2. For every (01, 02) E O, there exists constant K₂ > 0 such that
(01,000) - K2 02ll,
We now show that the covering number of the set T(c) satisfies Lemma C.8. Given e > 8 > O. and under Assumption
C.2
(2c +2
(2
N(6, T()), <
(50)
For any positive & and e, let E 100 - ke, 0o + ke] c θ C Rd. Now consider 0.+1} and H = 8 Hi with J = |2 where 0f = 0o ke + id' for j {0, 1, ........u J} and 0/+11 0o + ke. Observe that for any 0 100 ke, Oo + ke]s, there exists a 8 E H such that ||0 As |||| < 8'. Hence, union of the 8' balls for each element in set H covers
Proof of Lemma C.8
a set H, = (00 ............... 0g
10g ke, 0g + kejd, therefore (66', 10u - ke, 0u + ke]d , || (1) (J + +2
Now, due to Assumption C.2 for any 0 E (0p ke, 0n + ke]d
dr.(0,00 0o) S K2| - 0'1 S K28',
For brevity, we denote n L,, n-'L, 0u) by dp (0, 0n). that is
d2 02) == sup [(aa, (a. 01) R(a, 02),, v(01,0) e
(51)
a
and the covering number the set T(e) == Poo : (0,00) <) as N(6, T(4), dr.), where the radius of each ball in the cover.
Hence, coovr of set [0u ke, Oy ke]d is K 8 cover of set T(e) with k: 1/Kg. Finally,
2ke
d =
2c
((K))'T((),dd) dz) s 2J+ <
+
2
+
8
K28
which implies for 5 K28'
(2e
N(8, ((), <
+
8
Observe that the RHS in 50
is decreasing function of 5, infact for 5 = e/2, it is a constant in
Therefore, using Lemmas C.7 and C.8 we show in the following result that T.,, (0, 0o) in (6) satisfies Assumption 2.1
27

---

Lemma C.9.
Fix n 2 1. For given > and every > en such that ne, 2 1. Under
Assumption
C.I
and
C.2
Ln 00) (suPaEA [i(a, 0) Ría, 0o)l)² satisfies
E Ep [pn] Co Cxexp( exex(pCCnee)
(52)
sup (0E0:L.(0000)) (0,6
Epp [] @n - exp( exp(-Cne²),
(53)
where Co = 10t and C'
for a constant C1 > 0. 2K?
Ct
Proof of Lemma C.9 Recall (0, 0o) (supae [R(a,)) R(a, 0o)l) and T(e) (Po
(0,0, 0n) < cc.. Using Lemma C.8 observe that for every en > 0.
N 10:c S (0,00,)) S 26), de s N ( (0: (0, 0o) S 26)). < <10º.
Next, using Assumption C.1
we have
dp (0, 00) (0,
It follows from the above two observations and Lemma 2 that, for every > > 0. there exist tests ( such that
Eg [on,el S 10a exp(-C'ncc)
(54)
e-ex(((('ne))
sup (0E@id, (0,)))2e)
Epg dn,el & exp(-C'ne²),
(55)
where C' = 2K? Since the above two conditions hold for every c > Ons we can choose a constant K such that for every en
Epy (on.e S 10d exp(-C'K2ne²)
1 1-exp(-C2 (oo(-C"Kii C'K2ne2) S ((100))-C''k
(56)
sup (9.0%) k3
Epp [1 @n,el
sup
Epy [1 Φn,el S -O'K
(57)
(0,001
where the second inequality in (56) holds Vn 2 n10, where ng := == min(n 11: log(2))
Hence, the result follows for C K2 and C K2
Since Ln (0.0g) = df satisfies Assumption 2.1 Theorem 3.1
implies the following bound.
Corollary C.1. Fix a' E A And 7 > 0. Let en be sequence such that + 0 as n + cc, ne 2 1 and
(0,
sup R[(a, 0) R(a, ((laBoo)) aaE
Then under the Assumptions of Theorem 3.1 and Lemma C.9 : for C
2k7 Co = 2 10", C1 >
C2
such that min(C, Ca(r)+ > C2 CC + C4(()) +2, and for fl (1) as defined Theorem 3.1 the RSVB approximator of the true posterior Q (x)) satisfies,
E E J
L.. (2.0000) (0),, d0
(0)
I ((M(()) + Mn,(7)),
(58)
for sufficiently large n and for a function =(7)=22) + .C4(1)) where M min mC..C.
2C
Proof of Corollary C.1 Using Lemma
C.9
observe that for any (x(e)) S θ, (0, 0o) satisfies
Assumption 2.1 with C. = 2 10a C 2K?
and for any C₁ > 0, since
sup (O.0))
Epp Φn.el S
sup (..0
Epp [1 - dn,el S -Cr
Hence, applying Theorem 3.1 the proof follows.
28

---

C.6 Newsvendor Problem
We fix n nentially distributed model Pe satisfies Assumption 2.1 for distance function L 80). ) To show this, in the next result we first prove that dL' (0, 0o) - ni 1/2 / (0, 0o) satisty Assumption C.1 Also, recall that the square of Hellinger distance between two exponential distributions with rale
1/2 1/2 LL (0, Bb) ( R(a, 0) R(a. 80)1). Next, we aim show that the expo-
parameter O and 8g is d (0, Bu) Lemma C.10. For any 0 E θ [T, o0), and a A.
2 y 8460
2 1+86/0'
80/0
27 1/2
h
4 :)
+ h)²
6-
--000
( (0,
8g
T
d (6, Bo)
dh (T, 8a)
where a == min E A} and a > 0 and 0g lies in the interior of O
Proof.
Observe that for any a A,
RR(a, 0) Ría, 0o)
h
h
a-a
(b + h))
00
h
h
2
c
-00
2
h
h
(c-08
c-a80
(b + hh)²
+
2
(b+ + h)
0o
0
00
80
0
0o
h
h
2
c c-
(b + h)²
(59)
0o
0
00
where the last inequality follows since for 0 > 00. versa if < 0g that together makes the last term in the penultimate equality negative for all 0 E e. Moreover, the first derivative of the upperbound with respect is
-
2 0 and
(2 タ
480
0 and vice
h2 2
h
h
E a0 e a0
2(b h)2 02 0z
08
0o
00000
On
0
and it is negative when 0 S 0, and positive when 0 for all 0, 0, and E A. Therefore, the upperbound in (59) above is decreasing function of for all 0 0o and increasing function of for all f > 6g. The upperbound is tight at 0 = fy.
Now recall that the squared Hellinger distance between two exponential distributions with rate parameter 8 and 0 is
(0, = 1 1-2 1 = 1 12-2
V80
Vo0/0 (1- /B/002 1 00/0 1+ 0o/0)2
00+00 0n
Note that for 0 < Og. d (0, 0o) is a decreasing function of 0 and for all 0 > Op it is an increasing function of 0. Also, note that as - co, the squared Hellinger distance as well as the upperbound computed in (59) converges to constant for given h. b, Gg and a. However, as f + 0. the de (0, 0g) I but the upperbound computed in (59) diverges.
Since, θ = [T. oo) for some > 0 and T S Gg observe that if we scale (0, 0o) by factor by which the upperbound computed in 55 is greater than du at 0 T, then
h
2
e a0
-00
(b+1 h))²
0g
0
0
+ (b + h)2
-a0
2
<
80
T
T
8o
dz (0)
dy (T, (T,00)
h
2
+ (6 + h)²
-07
-00
2
8g
T
00
<
Lz (0, (0,00).
dy (T,0g)
29

---

where a ia E A} and in the last inequality we used the fact that (-/²)
u60
is
decreasing function of a for any b, h. T, and 0o Since, the RHS in the equation above does not depend a, it follows from the result in (59 and the definition of L (6, Bo) that
1/2
(
hb 0o +
#
+
(b + h1?
(--2" e-20
)
d. (0)
110
(0, 00).
d3p (T, 0o)
Lemma C.11. For any 0 E e = [T, 00), for sufficiently small T >> 0, and du lying in the interior of O we have
(0,00) - v000
80 (T + 00)
T
V T
Gol.
0o
Proof. Observe that
(0,00) 0n)
(0 + 0o) vou 2v0
V00
0o (0 + 80)2
(Vo
Op
00
(0 0o)
0o
Observe that + 0,
ad (0.0g)
+ 0o. Since,0 E θ [T, oo), therefore the supace
Odt, (0.0o) 50
<
00
00. In fact, for sufficiently small T > 0, supage
(8 68
(T+
(VT.
V
Pg
(V - Vo 0o ) Now the result follows immediately since the derivative of d3 (0, 9b)
Ou (T+00)²
(V--v/
bounded on +, which implies that dzz (0, 0o) Lipschitz on O.
Lemma C.12. For any 8 E @ = [T, oo), and a E A. h 0 Pol- V (0, 0o) S T2
Proof. Recall,
h
+ h) a0
R(a, 0) ha
(b
First, observe that for any a E A,
ÖR(a, abb
a8 (b+
08
1
+1())))) s h
(h - (b + h)e
-a0(1 h)0--)(111
(60)
00
g2
02
02
The result follows immediately, since supage
a a6
<
h
T
Proof. Proof of Lemma B.I
follows from Lemma satisfies Assumption
C.
10 that d2V(( (0,00) 0o) for any E θ = [T, 2o) and 0g lying the interior of O
with
1/2
h 8u
4
+ 0b+
-
K1
T
80
KNN
dy (T,0o)
Similarly, it follows from Lemma and C.12 that for sufficiently small T > d NNV (0, 0b) satisfies Assumption C.2 with K₂ h/T² K2" Now using similar arguments used in Lemma C.8 and Lemma 2.1 for given én > 0 and every > en, such that ne 2 1, it can be shown that LNV (0, 0o) (supaeA [R(a,0) 0) R(a, 0)) satisfies
30

---

E Epg [on] S Co exp(-
(61)
sup ( (9.0a) 20m))
E Pg [] onl (n
(62)
where Co = 20 and C' = V for a constant C1 > 0.
Proof. Proof of Lemma B.2
First, we write the Rényi divergence between Po and Pg.
log /
dPo
1
dPo
D1ta (P) = A
dPp
dPo
x log
dP
dPo
0u 0
0u
log
log
(A 1)00 X0
when + 1)0g 20) > 0 and D1+1 (P))) = oo otherwise. Also, observe that,
D1+A (Po P(') is non-decreasing in À (this also follows from non-decreasing property of the Rényi divergence with respect to A). Therefore, observe that
0o (P - C)ne,) 2 o (Po' (P)) & Cyne,) = II 0 S log
S C3cn
0
II Pue
Car < < 8o)
Now, recall that for set A e [T, oo). we define (A) Inv ((Ane)/In - r(@). Now, observe that for sufficiently small and large enough n,, we have
II (Oge Ca 0 0o) Inv I (ooe-Corh Co 0 00)
The cumulative distribution function of inverse inverse-gamma distribution Inv I[(0 E 3 {}) = [) F(a) where a(> 0) is the shape parameter, ((> 0) is the scale parameter, [-)) is the Gamma function, and ( -) is the incomplete Gamma function. Therefore, it follows for a >> 1 that
Inv - (0ee - Cse S S 60)
I (a. B/0a) T aa, B/bc
S0j00
B/000 e-2za- 1
T(a)
T(a)
3/0ge o +aCse
3
-0
-
aT(a)
1o
-8/00e
B
e CaCane
aT(a)
05p)
where the penultimate inequality folows since 0 < 2 and the last inequality follows from the fact that, e c can't hold true for e 1/n. However, 2 log it holds for any n 2 when aC3 > Therefore, for inverse- inverse-Gamma prior restricted to θ, C2 aC's and any X > 1 the result follows for sufficiently large n.
2 2 e aCCyee, for large enough n. Also note that, - -aCseh aCCyne
Proof. Proof of Lemma B.3
Recall,
h
+ (b h) e-
R(a,0) ha
0
0
First, observe that for any E A,
aR(a, 6)
e 00
08
00 02 ab + h)
(b + h)
(h (b + h)e a0 af))
(63)
0
02
02
31

---

Using the above equation the (finite) critical point 0 must satisfy, h - (b+ h)e 2a00 (1 a0*) = 0. Therefore.
aal
R(a. 0) R(a, ((a,0))=1 h (at
ha20-
8r + (a(1 atr)
(1 + aB*)
Since h b > 0 and a0" > 0, hence
ha²0r
R(a, 0) 2
(1 + aB*)²
where minfa A} and 0.
Proof. Proof of Lemma B.4 First, observe that R(a, is bounded above in 0 for given
E A
h
h) e a0
R(a,0) 0) ha
0 + (b
0
$ ha
Using the above fact and the Cauchy-Schward Schwarz inequality, we obtain
( (eRcAM)
(())dd
(f (2
1/2
(
1/2
(f (n((att)
( (c0((rat x00)
1/2
Se-Celrime? (f
(64)
where the last inequality follows from using the Chebyshev's inequality.
Now using the definition of the prior distribution, which is an inverse gamma prior restricted to O = [T' o0), we have
{ (emansssesmm (aaa ( (0)) s CC(t))nc (l
S -e-Ci(e (oot)))eeg2((a++ 21(+++
where = max(a E A} and > 0. Since ne 2 1, we must fix Ca(r) such that cC(l)) > 21(1++ that is Ca(r) > 27 (ha + +) and Cs(7) = Ca(7) - 27 (ha + 4).
Proof. Proof of Lemma B.5 Since family Q contains all shifted- thifted-gamma distributions, observe that (0-2)
(-n()) 2) 2 1. By definition, gn (0) Tp (0 -e Ni
Now consider the first
term; using the definition of the KL divergence it follows that
KL (0) = f.
q. (0) log(g,, (0))d0 f 4n (0) log( (0))dO.
(65)
Substituting go (0) the first term of the equation above and expanding the logarithm term, we obtain
In (0) (0) log(an (0))d0
(n (nn-1) 1 log(0- -
n2
(0 1 -7) - I
n2"
do
log
T (n)
[
- -log 0u + (n 1n--13 1) 1 log
0 T n"
m 6-r
n2
(0 - T)
80
d0
n - log
(66)
0u
0 [
[(m)
32

---

Now consider the second term in the equation above. Substitute 0 = t8o + T into the integral, we have
log
0
71"
( T)m=
e
P-T
30
d0
00
log
t
t" 1 ee'
T
80
(
0
[(n)
/
1 -1
E e = 0.
(67)
I(n))
Substituting the above result into 666 we get
l
n²"
9n (0) log(gn - log 0o + log
[(n)
nt"
- log 0o + log
22mmm -lm
- log 2 + 2
logn n,
(68)
where the second inequality uses the fact that V2wnn" (n)... Recall x(0)
8
0-
[(a)
Now consider the second term in (65). Using the definition of inverse- gamma prior and expanding the logarithm function, we have
(0) log (0)))d
log
8
+
(ax
log 0 n²
T (0 - -7)-1 -n
d0 + 3 n
T(a)
T
(n 1 1)0
3
n"
- log
+ JT
log
(0- 1-
-)- e
40
d0
T(a))
Oo (
+ 8
+ (a + 1) log Op
(n 1)00
$ log
3
+ JT
0 T
n2
(0 - T) 1- 1n
%0 d8
[(a)
0u
r
+ (n 1)00
n1
+ (a
+ logog 9o
g
n
lo
+ (n 1)00 + (a + l0
(69)
T(aa)
where the first inequality is due to fact that Eg., [3/0] < Eq [3/(0 T)] for any 0 > T and the
penultimate inequality follows from the observation in (67) and the fact that log 0 S 0o for any 0 > T. Substituting
0 -1 < 0 T
669 and 68) into 65 and dividing either sides by we obtain
KL (K))((0)
< (llog log v 22000 + log log
1
Aa [(a)) ) +
n (n (1)0
+ + + 1) log 0u
logn
1
1 (- log 2T log
Ba
+ 3
+ n
+ (a) log 0o
(70)
2
(n 1)00
r(a)
Now consider the second term in the assertion of the lemma. Since E,, E (1, (1,2...9) n) are independent and identically distributed, we obtain
1
Egn,(o) KL
= Eg.. (0) [KL (dPollp(E)0))
Now using the expression for KI. divergence between the two exponential distributions, we have
1 E Ey,,(
KL
(PPlllp(Xx
log
00
+
0
n"
(6 - T) 80 d0
R-
8
(g
T
71
1
<
(71)
71
11
33

---

where second inequality uses the fact that log & xx-15 x and (70) for > implies that
I for Bu > T. Combined together 71
KL (4m [T()) + (0)
KL
(P IIpp (Il((x,,10))
1 log
1
2
g [(aa]
log
2 +
log V2T log
+ a log 0o
Cg
(72)
2 n
0g
where Cg : 2 + max (0, 2 +
28 log V2T - log
8 [(a]
+ a log 0o) and the result follows.
Proof. Proof of Lemma B.5 Since family Q contains all gamma distributions, observe that (-n(-) n
@)tn 2 1. By definition, 9n (0)
(i)(n) g e %0 Now consider the first term; using the
definition of the KL. divergence it follows that
(L(q((0)|=(()) /
4n (0) (og()n(0)) (0) log(T(0))dO.
(73)
Substituting 9n in the first term of the equation above and expanding the logarithm term, we obtain
f gn (0) log(gn (0)) (n-1) 1)
/ log 0
n2t
n²
0
d0
log
OgI
- log Co + (n 1011 1) / log 0 n
n²a [(n)
0
d0 + log To
(74)
Oo (g))
Now consider the second term in the equation above. Substitute 0 = tlo into the integral, we have
f
n¹ gr- d0
/ log t" tn yn-le-tal dt
log Oo (3)(m)
l'nn)
/
1 (" n
-
0.
(75)
I(n)
Substituting the above result into
74 we get
f qn(()) (0))d) S log n log
n
[(n)
n
log n + log s
-2
log 22000 + logn.
(76)
where the second inequality uses the fact that /2rnn" S nl nn) Recall x(0)
[(a]
Now consider the second term in (73). Using the definition of the logarithm function, have
prior and expanding
f on (0) log (T (O))do
g
+ (a +1)+1140 /
log 0 n
log
0
n To d0 3
T(a)
0
(n 1)0
+ (a 1) /
0 n"
log
log Oo (g
0 1 n 90 d0
r(a)
+ B
n
+ (a - 1) lo 0o
(n 1)00
< 5 log
3a
+ 3 n
+ (a + 1) log to.
(77)
T(a)
(n 1)0b
34

---

where the last inequality follows from the observation in and dividing either sides by n, we obtain
75
Substituting
77 and 76 into 73
KLL (LL(((0)]
1 log v 2200 + log log
1
+
(n (n00 + (a + 1) log 0o)
n
I(a)
log
1 (- log 2 - log
gm
+ 8
+ (a) log log 8o)
(78)
2
(n 1)80
[(a)
Now, consider the second term in the assertion of the lemma. Since, Ei, E {1, n2 are indepen- dent and identically distributed, we obtain
Boy KL (dPoll(Xxa)) 0)
(o) [KL (dPollp((())))
Now using the expression for KL divergence between the two exponential distributions, we have
E
Ey(
KL
(dPo Ip [o(X,100)]
/ log 0g
--1)
n2"
0n 1e
d0
n
0o
og (g
n1
1 2
(79)
n 1
where second inequality uses the fact that logæ: S x 1. Combined together 79 and 78) for 2 implies that
2
KL (dP.))lp( }]]]]
KL (()) + By(
1
log
1
2
g [(a)
log
2
log 2 log
+ log 0o
C9
(80)
2
71
n
0n
n
where Cg
1 max 0,
2
28
log 2n - log
[[a]
8" a log 00) and the result follows.
C.7 Multi-product Newsvendor problem
In the multi-dimennional newsvendor problem, fix n 11/2 LMNV (0, (supaed R(a, 0) R(a, (0)l), where R(a, El (h, b.)a, b.a; 0, h4)
+oB h o((a;-9:)/a 9))/aa) + b I 0.)/ouc)
"i(aa ())/0a)
1/2 For brevity, we denote d NV (0, 0)) n 1/2 / LLLNVVVV (0) First, we show that
Lemma C.13. For any compact decision space A and compact model space Q d (8,6g) K| Bull.
for a constant K depending on compact sets A and + and given b, h and E.
Proof. Observe that
0n R(a, 8)
h
(b; h4) (a; 0.) 0.)/n - (1)00//)(((((0))))a))) 0.)/ox)
+ 1 ( 0.)/on)
Φ((a: 01)/oa)) bol(a; -1,)/an)
(a: (Qu-0.) hol(a; - ol))/on)
+ +on
Tis
0r 0.)/
ori(1 - $((a: 0,)/oz))32 h Φ((a: 0:) 0z)/ bob(aa - 0))/0
(b:
h4) (a: 0,) / - 0.)/ozz)
+ 1 Φ 0,)/oz)
(as 0.)
hol(a - 01))/z0)
+ o
(81)
Tsi
Φ((a: 0,)/on)2 (1 Φ(aa: 01)/o=))2
35

---

Since, A and @ are compact sets, therefore {(a: - ox)/ lie in a compact set. Consequently, (())o)))) and ((( also lie bounded subset of R and thus sup 1|0b, R(a, 8)1 v K for a given b, h and Σ. Since, the norm of the derivative of R(a, 0) is bounded on @ for any E A, therefore, N (0, 0o) is uniformly Lipschitz in A with Lipschitz constant K, that is
d (0,00) - K|10 0oll
Next, we show that the Pg satisfies Assumption 2.1 for distance function LM (6, 6u).
Proof. Proof of Lemma B.6
First consider the following test function, constructed using Xx {Ei, Ea....,
Φn.e 1l ( 10.110.-00||3GGa
where 0, = Ei. Note that 0, - Ou ~ (-10, 1 1 where y is a symmetric positive definite matrix. Therefore it can be decomposed as Σ = Q AQ, where Q is orthogonal matrix and A is daigonal matrix consisting of respective eigen values and consequently 0, 0, QN((10, A.).. So, we have |lO, - loll ~ [[V((-0, A))l Notice that [IV((-), A)]l is a linear combination of x(1) random variable weighted by elements of the diagonal matrix 1 A. Using this observation, first verify that dn,ee satisfies condition (i) of the Lemma. Observe that
n
Epp [6n] = Po" 8n 10..
0o > Cē² cee== =
Po (Xm. IIV(--0, > Cne))
Note that X21 is T distributed with shape 1/2 and scale 2, which implies xi1, 1 is a tuub-gamma ran- dom variable with scale factor 2 and variance factor 2. Now observe that for À = maxie Ans
Po (X,: (VV(10, Cne² S Po
8n : x²i) > Cnne
dA
s P
Xn x(I) > Cnez
dd
1
P
Xn : X(1)
- >>>
_ -nne -1)
dA
Se e 2(2 di ame-
Se -188 One" +1/8
- < -1/8 (1111111111) -1)ne,
(82)
where the third inequality we used the well known tail bound for sub-gamma random variable (Lemma 3.12 (51) assuming that C' is sufficiently large such that dA
1 Cne² - 1) 1 and in the last
inequality follows from the assumption that nc² > ne? 1.
Now, we fix the alternate set to be (0 E Rdd: : ||0 foll 2 2v Ce2. Next, we verify that Φn,e satisfies condition (ii) of the lemma. First, observe that
Epy [1 [1-ón] Po [x 10, eall Ce2
S Pp (Xn 10, - 0 110 0oll v vCe²
(83)
where in the last inequality, we used the fact that ||A Boll S 110, - -811 ||0, 80
Now on
alternate set {0 E Rd: : ||0 Poll 2 2 Ce
E p [1 dnl - PM
Xn 110, - 01| 2 100 Poll - VC
< P
X, |10, - l z || Boll - VC2²
< PM
8n 110, 01l 2 VC.2)
(84)
36

---

Now, it follows from
82 and θ C R that
Bpy [n) S e -1/8(
23 -1)me?
sup (0EB:]0-B6||22VGG
Bpy [1 0nl x
sup
E Bpy [1 - 2nl S 1/88(
ds -l)me
Using Lemma C.13 {0 :@:n LMNV (0, 0,) > 2K VK = 10 E θ N (0,0g) 2
2KVC2) 2 (0 e θ 100 Boll 2 2V Cc2, which implies that
sup (0E0: (0.00) Ck2
Epy dml
sup (
Epg [1 Φn)
Therefore, Pe for 0 θ satisifes Assumptions 2.1 for (0, LNNV (0, 0o) for Co 1,
C =4K3C and C = 1/8 dA
C
-1)
Proof. Proof Lemma B.7
First, we write the Rényi divergence between two multivariate Gaussian distribution with known Σ as
D DI=x(N((O0))W((110) = X+ (0 -0 -0)
(85)
and D (N1)) < oc if and only if Σ-1 positive definite 13
Since, we assumed that the sequence of models are iid, therefore, DI+A (Po (P)))
log /
dP TPF
dPp = n log
dPn
dPa = 2 0o) 0o)). when E-1 posi-
dPb
tive definite and D1+A (Po P. otherwise. Now observe that
I1 (Po < nC = - (10((00)))00 -(0-)))))))))))))))))))))))))))) S 7
2
Cae
2
=I
- (000-)0)0A\((( 0n)) 1 X+1 Ca
2 II
((0-))))TTQ(( 0n)) 1 Á(^ + 1)
Ca
= (((0-60) - 06)1) 5 Á(*+1)
Cs
(86)
where Å = (1.2..) As and in the second equality we used eigen value decomposition of E AQQ.Q. Next, observe that,
(PP11P)))) = II ([(0 (((0-9))]] 0b)) Bb)l) - A(A+1)
C2
2
III
[10 - [()0- S
C3e2
V
A(A + 1)
2
2 II
[(00 80)lmm -
Cze2
Á(A +1)
II 1I:
[(00 -0:) 5
Cae
A(X+1)
where in the last equality we used the fact that the prior distribution is uncorrelated. Now, the result follows immediately for sufficiently large n, if the prior distribution is uncorrelated and uniformly
37

---

distributed on the compact set O for each i E f1,2..... df In particular observe that for large enough n, we have
B +
Car f
2
C3C2
M(+1))
A(++1)
II(D++a (P PP) nC
II
m(O+)
iml
2dd A(++1) C2e
4/2
d m).))
-2/d
4/2
8
C3
I m(;))
(A(*+ + 1))
where m(A) is the Lebesgue measure (volume) of any set C R. Now if c log
then for
n
C3 2, A m(e.)) C3 2 Ma++((M m(e,)) Cane
A(+1) m(e.)) 2 2, therefore.
for all
M m(0,) ) Canen
(P < nCCeen) 26
Proof. Proof of Lemma B.9 Since family Q contains all uncorrelated Gaussian distribu- tions restricted to θ, observe that {-)(+) 2) 2 1. By definition, (, oc E
Mo.m) 1e. = N/ (...)) and fix Jin = 1/vn and 0; = f for all i E 21, d}. Now consider the first term; using the definition of the KL divergence it fol- lows that
V e
(0) log( (0))d0 - -f /am(()) log(T(0))do.
(L (0)
(87)
Substituting qr (0) in the first term of the equation above and expanding the logarithm term, we obtain
f on (0) log(an (0))d0
Σ
f d. (0.)
d
N oz log or
1=1
[ogl 22re) log oi,n);
(88)
where in the last equality, we used the well known expression for the differential entropy of Gaussian distributions. Recall (0)= = II m(..] Now consider the second term (87). is straightforward to observe that,
(0) log(T = Σ log(m(O.)).
(89)
Substituting
89 and 88 into 8
and dividing either sides by and substituting den, obtain
[L(xx(0)|==00) <
log( v 2Tre - log(m((O))) -2 log n)
n
1=1
log
1
log( (2T)) - -lo(mm( (t)))]
(90)
n
Now, consider the second term the assertion of the lemma. Since E,, (1,2... n. are independent and identically distributed, we obtain
Bqn.(0) [KL (dPo (I(((((()))))))))))))))))))))))
= E (0) [KL (dPollp(E(0))
38

---

Now using the expression for KL divergence between the two multivariate Gaussian distributions, we have
(0)
KL (dPCll((xx10))
1
g. (0) [[0- 110--00 (00 0o)]
2
E (o)( 100 (0--0) 00) 0o)
2
d A-
(91)
n
where Å = (1.....) Aa, and =-1==- - = where Q is an orthogonal matrix and A is
daigonal matrix consisting of the respective eigen values of Σ. Combined together (I and (90)
implies that
KL (gn (0)) + g.) (0)
KL ( [XXx0)0]]]
d log( v 2xee log(m(0,)) + d A-
d log n
log n
< Cp
(92)
2
2
==1
where Cg == max (0,-E llog (org(v/2e) 2Te)) log(m(0.))] - and the result follows.
C.8
Gaussian process classification
Proof of Lemma B.11 In view of Theorem 7.1 in [T] it suffices to show that
N O(,) S Cne
for some C> O. Now, first observe that
drv(Pe(w)) Poo(s))
E. (1V,, (0(y)) vI(((())) + ( v-1(06()))
= E, S Ey (10(y) (v)l)) I (10(y) - Go lo))llec
- v (0o(()))
(93)
where the second equality uses the definition of V-1(-). Since, total-variation distance above bounded above by supremum norm, there exists constant 0 < d' < 1/2, such that
N (e, (c))), drv) S N (c'e, (c), 1100) S Cione"
(94)
where the last inequality follows from (13 in Lemma B. Then follows from Theorem 7.1 in [[] that for every > En, there exists test on (depending on e >> 0) such that, for every j 2 1,
py [ón] I 3 "f Orane" e mm?
1
1- 1-exp (-3ne2) and
E Epy [1 - dnl S exp (-me²))
sup (0E (Po. )oje)
Now for all such that no2 ne? > 2 2oog 2 and C1o c 0-2/42 1 and j = 1, we have
Epp [øn] < 2c nn² and
(95)
Epple [1 @n) S e lno²
S e Jaac
sup (o
(96)
Now observe that
sup [up/G((a 0) - G(a, 0o)1 aeA
max (c+, Er(VV- 1.1_
max (c4 ( - [e,
(V.
max(c+, | (V [
max((4, e- (VV,
(0()))]
< max(c4, c_ ddv Po.)
(97)
39

---

where the second equality uses the fact that V V-1(-)- 1 - -v)(-). Consequently,
{0 On(e): : sup [G(a,
Go)) > max -C(,,)))
S 10 On(e) drv (Po, Pool > E)
a
Therefore, it follows from (95) and (96) and the definition of (0, 0o) that
Bpg [p]] I 2e } and
(98)
Epgl [1 - Φn) S c n
S e tmme
sup (0),..))) (max(e)) oo ))mm2))
(99)
Finally, the result follows for C = 1/3. Co = 2 and C (max c-))
Proof of Lemma B.12
The Rényi divergence
D1+a(PR PP)
="> In /
(4660))0H(((00000110000000000010 (dyy
in ʃ h100::/0000000000000444444400000
(100)
Note that the derivative of the exponent in the integrand above with respect to ((y) is
(- -A (Z (4NA6H0))))NXX(((0)00H00HHHHH110
(1) (116000))00000000044444440666044
=)((O(T))
((()))
V (O())) -1 (0())
(110000))))011000000)--000000000 (0(y))
V1 (0b())) -- (1000)))00000000000-000000000000
=
(T) (10
AO(O + (x+1)80(0) (1))((111) (1+ ((v)
(1+ (10)---))
AB(M) + e (x+1))8000)) (1 + tev()) P(v))
(A+ ()())) - 0 (v))
< X (e (A-181))((+)))66 1) + -
< A(A + 1)) (v)))
(101)
where in the fourth equality we used definition of the logistic function and the penultimate inequality follows from the well known inequality that 1 - e < œ. Consequently, using Taylor's theorem it follows that the exponent in the integrand of the Rényi divergence in (T0) is bounded above by X(A + 1)) - Bo())) and thus by A(A + 1))(((n) - lo Therefore,
D1+a PP))
= n In /
- (000)))))000 v(dy)
S n x (
In /
= n(( 1)l(0(1)-((0))))) - -..
40

---

Now using the inequality for C3 = 16(A + 1) above observe that
(IA))) [I (P)))))(P)))))) 2 Csnen) 1) 0o(v) S Cane
[T(]((v)= - ~o S 4cn) 2 e -ne
(102)
and the result follows from 15p of Lemma B.10
Proof of Lemma B.13 Let us first analyze the KL divergence between the prior distribution and variational family. Recall that two Gaussian measures on infinite dimensional spaces are either equivalent singular. 277 Theorem 6.13] specify the condition required for the two Gaussian measures to be equivalent. In particular, note that 0% (-) E Im(CI/2). Now observe that the covariance operator of Qn has eigenvalues {cp therefore operator S in the definition of Cg has eigen- 2nd
values {1 - -G//)))
2
For r] = 2-2ja-jd for any a > 0. Σラ= 2
ne2 25a Sd
2+
ΣA=1 2-jd
ne.2 1+
2ia
xo, therefore S is an HS operator.
For any integer J S Ja define 0/- = / ()) where ()(+) = E/=1 E (0.,)03(() Since, of (-) E lm(C1/2) and S is a symmetric and HS operator, we invoke Theorem 5 in (22. to write
1 _
log det(1 -S)+ + (r((-
23
J 2
2 /=1
D:k.j
1
II (1 x3)
log
123
2
/=1 ==11
23d
02
D:k.j
1
log
- (1-x3)2tt x3)224
2 2
2
2
j=1
5=1
2d
02
J
J Σ
Dzk.j
1
2d log (1 - -x3)
2d4
2
1ag.
2
3
2
2
2 j=1
Now for 2,22d/2 = =2-ja and using the definition of Besov norm of 9, denoted as |10 11001l 200,00 and
denoting -x. I we have
KL(N (O),
22 23+d) 1001|3.....
1
Σ
23d log(1 - x-)) 1 Σ
2 Kij
2
j=l
1=1
J
22
23 1100
,0,000000000
1
Σ
2 (log(1 - x23) + x2t)
2
j=l
Σ
22 28+d) 1
100011..0 00,00
+
1
Σ
J
23d
llog(1 + nn223))
neh
2
+ net
3=1
j=1
J
$
Σ
2222-22++ 1 1011 00,
+ Σ
1
2)d (nc)1))
2
2
j=1
41

---

where the last inequality follows from the fact that, log(1 + x) Substituting of 22jja-id, we have
S
x²
S
for
> 0.
I+
It
KL(NCCCGNNOOCC))
1
20120 10020-22+001110
3,0,00
-
2
22ja
2n j=1
2 j=1
10001,,00.00
2 2a
2
2Ja
2n
2
1
2-20
2a
The summation the first term above is bounded by < as derived in (30 Theorem 4.5]. Therefore,
KLLN(CCC))WOO & max (180113.00.00 π
2-2a 2-2 20
(103)
1 1 1-2-20 2-20
Now consider the second term
1 BQn (L(P)\IP.))
n
/ (V((00(W) log Vi(0o(y)) (0(y)) + (--(0o(V)) log (00(0))))) v (0(0)(
- -Een
v(dy)
/ (((G) Oo(v), 0 (o()))((dy)
< EEq.
/ [10(y) 0f (v) - (0o(g) (B)))l)))(
= Eq..
/ (10(y) (1)0) (10(()) (1))1 2(0(v) (o(. (v)))((d)))
EQn.
S Eq.. / (10(g) - 05 (v) + -66(v)) (1)))
J 2
BBn /
3, + (1)(v)) 0g (v))
< EQQ, Σ 2/
23 /
(00()) 0f (3)))
j=1mm
J 2d
GE 12 + (10(()) 0/ (2)
k=
2d
(3(1) + 100(v) (2))
j=1
1
2dd
+ (10o(y) 0g 12)
1 1 n
1=1
1
2
+ (10)(u)) 05 (2))12
n62 j=1
1
2dd + |10o(2) -od()))
2
2d 2dJ - 1
nez 2d 1 + (10(()) og (2))
2d 122 -1)
()0 2
42

---

where in the second inequality, we used the second assertion of Lemma 3.2 [3) for logistic function, the fifth inequality uses the fact that ((y) 0g (t) is orthogonal fo(y) 0g (y). For any a S fix
J = Ja otherwise J = Ja. and then it is straightforward to check from the definition of en given in the assertion of the theorem that (2dJ-1/ne?)
(log n) -2, The term [10((y) )l3) is also bounded
by C² as shown in the proof of Theorem 4.5 in [30]. Consequently, the term B. ( | P8) is
bounded above by ex (upto constant) for sufficiently large since (logn) -2 E and the result follows.
Proof of Theorem 4.2 The proof is direct consequence of Theorem
3.2 Lemmas
B.11
B.
B.13
and Proposition
43


Reviews:
[{"review_id": "WSqa8TKowN", "reviewer": "Reviewer_FLVD", "summary": "This paper studies risk-sensitive Bayesian decision-making, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure with respect to the posterior distribution. In particular, this work focuses on problems where the posterior distribution is intractable, and leverages a dual representation of the entropic risk measure to introduce a new framework, namely, RSVB. The goal of the paper is not to simply approximate the posterior distribution, but to also make decisions when that posterior is intractable. Simulations are performed to validate the effectiveness of the proposed framework.", "questions": "Do you forsee the applicability of the proposed framework for sequential decision-making problems (e.g. bandits and reinforcement learning)? And whether the convergence rate analysis / decision rules derived using the proposed algorithm could benefit the further understanding of Thompson sampling based algorithms in the case where posterior distributions are intractable?", "limitations": "This is a theory-based work. No potential negative societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a novel risk-sensitive variational Bayesian (RSVB) framework to jointly compute a risk-sensitive posterior approximation and the corresponding decision rule by solving a minimax optimization problem, where the true posterior distribution is intractable. The objective being considered is a general functional that can be frequently encountered in various decision-making problems.\n\nIn particular, the authors derive the posterior convergence rate for RSVB, as well as the convergence rate for decision rules, and identify the regularity conditions on the prior, likelihood model and the risk function. The authors also study the asymptotic consistency and provides a theoretical understanding of the statistical properties LCVB posterior and the associated decision rule.\n\nDifferent applications are discussed by applying the proposed framework.\n\nThis paper is well-motivated and technically solid, and makes a considerable contribution by providing a general framework with theoretical guarantees for a wide class of problems. The convergence rate could be of potential interest for future research in the community.", "weaknesses": "1. The current discussion of simulations and applications does not include a comparison with other risk-based methods or methods that solve non-risk-based objectives. Further discussion will be beneficial.\n\n2. As a benchmark comparison, it is desirable to see the comparison with the exact decision rule in the case where the true posteriors are available.\n\n3. The current method requires solving a stochastic minimax optimization to compute a decision rule, which can be computationally intractable in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback. Please find our response to your questions below:\n\n1. The current ..... be beneficial.\n\nR. Thank you for pointing this out. In our simulation and applications, we do include the existing approaches for risk-sensitive Bayesian decision-making, namely the standard approach VB/VI [1, 2] and LCVI [2,3].  We would also like to bring to your attention that our RSVB formulation includes LCVI (i.e. for $\\gamma=1$). We will make these points clear in the revised manuscript.\n\n2. As a benchmark comparison, ..... posteriors are available.\n\nR. Thank you for your comment. In our applications, the optimality gap itself is defined relative to the true decision (which is available in the synthetic experiment). Therefore, our synthetic experiment compares the RSVB decision rule with the ground truth. In neither of our current experiments, the true posterior is available as the priors are non-conjugate, making the posterior computation intractable. Having said that, we agree that comparison with a conjugate prior benchmark would be useful, therefore, in the newsvendor experiment, we have added a comparison with a conjugate prior (where the posterior is tractable) in Figure 1. We also uploaded it to the one-page rebuttal document. We note that this is a slightly different model.  With this caveat we observed that the decision rule learned using the conjugate prior has a similar performance as the VI (or VB) approach. However, the variance of the true conjugate posterior is higher than those computed through the RSVB and VI approach, corresponding to the well-known fact that  the variational approximations underestimate the true posterior variance [4,5].\n\n3. The current method requires solving a ..... computationally intractable in practice.\n\nR. Thank you for noting this limitation. As we note in our discussion section, we agree that our RSVB formulation requires solving a stochastic minimax optimization problem, which can be difficult in high-dimension without further assumptions. To circumvent this issue [2,3] convert the problem to max-max by switching the risk function with gain, which is a much easier problem to solve computationally. Note that to convert risk to gain, one requires the risk to be bounded above. We emphasize that our theoretical results will continue to hold in the max-max setting, but since our emphasis was on the statistical aspects of this problem, we chose to present our results with minimum assumptions.\n\n4. Do you foresee the applicability of the ........ posterior distributions are intractable?\n\nR. Yes, we agree that our concentration analysis can be useful to study Thompson/ Posterior sampling techniques in the bandit and reinforcement learning literature. However, note that we are in an offline setting where we use the observed data and then make a decision, whereas, in these settings decisions are taken online while sequentially adding more data observed at the chosen decision points.  This will require extensions to our work and we agree that this is an interesting future direction.\n\n\n[1] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. JASA (2017) \\\n[2] S. Lacoste-Julien, F. Huszár, and Z. Ghahramani. Approximate inference for the loss-calibrated Bayesian. AISTATS (2011)\\\n[3] T. Ku ́smierczyk, J. Sakaya, and A. Klami. Variational Bayesian decision-making for continuous utilities. NeurIPS (2019) \\\n[4] Y. Li and R. E. Turner. Rényi divergence variational inference. NeurIPS (2016) \\\n[5] R. E. Turner and M. Sahani. Two problems with variational expectation maximisation for time-series models. Camb. Univ. P (2011)"}, {"review_id": "OBWS8wgqzH", "reviewer": "Reviewer_3wrT", "summary": "This paper introduces risk-sensitivity parameter $\\gamma$ to a decision-making problem where latent model parameter $\\theta$ is considered via variational Bayesian inference. \nThe authors analyze how close the approximate posterior is to the true parameter (Theorem 3.1) and how good the chosen action is (Theorem 3.2). \nThe theory is also verified with experimental results. ", "questions": "* What is the purpose of Eight-schools experiment in Section 4.3? \nIs it meant to confirm any theoretical aspects? Or is this experiment designed to check how $\\gamma$ affect a complex hierarchical model?\nTo show the actual loss as a function of data points (or epochs) with various $\\gamma$ values may be informative to see the effect of $\\gamma$.\n\n* Figure 2 (b) is unclear visualization with many colors overlaid, as well as gives unclear message. \nIs it obvious the variance of posterior can change with $\\gamma$?\n\n* Is Figure 1 (c) missing? This appears in the text (line 288), but I could not find it. ", "limitations": "While the authors do not put a distinct Limitation section, the scope of method and its theory is given as assumptions. \nI do not find potential societal impact regarding this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This approach extends loss-calibrated variational Bayesian (LCVB) in that the risk sensitivity is controlled. \nThe objective of risk-sensitive variational Bayes (RSVB) has risk expection term and KL divergence between approximate and true posteiors. \nThis combination may distort the approximate posterior, but Theorem 3.1 proves the posterior approaches the true parameter and Theorem 3.2 claims the gap of risk is small (with accurately specified model parameter $\\theta_0$). ", "weaknesses": "It is nice to provide assumptions on variational family like Assumption 3.1, but how to check if it is met seems unclear in general. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and questions. We address each of them below:\n\n1. It is nice to provide ....... unclear in general.\n\nR. Thank you for asking this question. We actually believe that Assumption 3.1 is quite easy to verify. The condition that the variational family contains Dirac delta distribution at each point in the parameter space is a mild and reasonable requirement for consistency. Further, the assumption requires that the variational family contains sequences of distribution that weakly converge to each Dirac delta distribution at a certain rate. This is easily satisfied if the variational family has no \"holes\". E.g. if it is the family of Gaussians with all means and variances, then we can always construct sequences converging to any Dirac delta at any rate. A similar assumption has also been made in [6,7], and is true for most exponential family distributions. For instance, in the newsvendor application,  we fix the variational family to a class of shifted-Gamma distribution and choose a sequence of distribution with parameter sequence $\\alpha=n$ and $\\beta=\\frac{n}{\\theta_0}$ which implies that the sequence of distribution has mean $\\theta_0$ and variance $\\frac{\\theta_0^2}{n}$, and, therefore, it converges to Dirac delta distribution at $\\theta_0$.\n\n2. What is the purpose of Eight-schools ......... $\\gamma$.\n\nR. Thank you for your comment. The primary purpose of the eight-schools experiment is to empirically study a real-world example and see the effect of $\\gamma$ on the RSVB decision compared to the VI/VB approach.  In the eight-schools experiment, we do not have the 'true' decision, so we compare the performance of the RSVB approach to VB. We assume that by 'actual loss', you mean the average loss function evaluated at the actions generated by the algorithm, as opposed to the regret that we consider. The latter is just the 'actual loss' shifted by a constant with respect to $\\gamma$, so that plotting the `actual loss' as a function of epochs is not going to change our conclusions.\n\n3. Figure 2 (b) is unclear visualization with ..... $\\gamma$?\n\nR. Thank you for point this out. We will make Figure 2(b) look better and clearer in the revised version (also uploaded in the rebuttal document). However, we would like to note that it is not obvious that the variance of the RSVB posterior will reduce as $\\gamma$ increases. We believe that it depends a lot on the landscape of the expected risk function and the choice of the variational family. However, since we observed this phenomenon in our experiments, we thought it was worth reporting.  Intuitively, a possible explanation of this phenomenon can be provided using the equation (RSVB). Consider the RSVB formulation and note that $\\text{KL}>0$,  therefore as $\\gamma$ increases, there is more incentive to deviate from the true posterior and choose $Q \\in \\mathcal{Q}$ that maximizes expected risk for a given $a\\in \\mathcal{A}$. Note that to maximize the expected risk, a $Q$ that places more mass near the $\\theta$ that maximizes the risk will be preferred over the one with more spread. We will include this clarification in the revised manuscript.\n\n4. Is Figure 1 (c) missing.....\n\nR. Thank you for pointing out this typo. It should be 1(b). We corrected this in the revised manuscript.\n\n\n[6] Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. JASA (2018)\\\n[7] F. Zhang and C. Gao. Convergence rates of variational posterior distributions. Ann. Stat. (2020)"}, {"review_id": "02SS4CVznQ", "reviewer": "Reviewer_r5cb", "summary": "In machine learning we often want to infer an optimal course of action in an uncertain scenario from observations we've made. Bayesian decision theory provides a framework to do just that: observations are used to build a posterior, which is then used to build an objective to optimize to decide a course of action. Unfortunately, both building a posterior and optimizing the objective can be challenging. To address these issues, previous methods had relied on variational inference of the posterior: they built an objective to jointly optimize the action and the variational approximation. This paper sets out to generalize previous methods with a temperature parameter ($\\gamma$ in eq (SO)), and theoretically assess the convergence of this procedure, in particular assessing the effect of the expressivity of the variational family. They establish a convergence rate (Thm 3.1 and Prop 3.1) that depends on the contraction rate of the posterior as well as the expressivity of the variational family; they also address the effect of their temperature parameter. Finally they apply their theory to derive convergence rates in 3 examples.", "questions": "$\\eta_n$ seems to simply be a rescaled version of eq (RSVB) with an expectation. Could the authors elaborate on why the expectation must be taken before the infimum; it was not clear why this had to be the case in lemma 6.14.\n\nAssumption 3.1 is also not a standard assumption; I understand that they are able to prove it for the distributions given in the examples, but could they discuss why that is the case, and what the assumption means in simple terms?", "limitations": "In the Lacoste-Julien et al paper, they switched the loss for a \"gain\" so their variational objective became a joint maximization problem rather than a min-max objective, the later of which are in principle more difficult to optimize. This seems to be the major limitation and it is addressed but he authors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theory the authors establish is based on a number of standard assumptions that are readily verified for most Bayesian settings, including nonparametric settings. The authors also demonstrate their theory by applying it in 3 examples. Theory, like that of the authors, that accounts for the expressivity of the variational class is also much needed in the variational inference literature.", "weaknesses": "The assumptions the authors make (2.1-2.4) are generalizations of the standard Bayesian non-parametric convergence rate assumptions from Ghosal et al 2000. It is not immediately clear the significance of their differences, and where each of the assumptions are used in the following theorems. It is a weakness that it requires one to read the proofs in the supplement carefully to understand why each assumption is made. A sketch of the proofs, statements of lemmas, such as establishing which assumptions are necessary for the convergence rate of the posterior, would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and for asking insightful questions. We respond to each of them below:\n\n1. The assumptions the authors ....  the paper.\n\nR. Thank you for pointing this out.  We would also like to note that our assumptions 2.1-2.3 are the same as that in [7], which are motivated by the seminal work in [8]. [7] prove concentration rates for variational posterior, which are not risk-sensitive. Since we are interested in studying risk-sensitive posterior approximation, we introduce two additional assumptions (2.4 and 2.5) that involve the risk function. In Assumption 2.4, the prior is assumed to place sufficiently less mass in the region of the parameter space where risk is large, while Assumption 2.5 just requires the risk to be lower bounded.  We also add a proof sketch of our main Theorem 3.1 in the Appendix of the revised manuscript, highlighting the necessity of the assumptions and where they are invoked. \n\n2. $\\eta_n$ seems to simply be a rescaled ...... lemma 6.14.\n\nR. Thank you for noting this. Yes, your observation is correct. We also agree that we can very well interchange the order of expectation and infimum. However, note that if we do so, we will be back to analyzing the RSVB optimizer, the very objective of the whole analysis.  Taking expectation before infimum enables us to derive a more interpretable and meaningful bound. This is mainly because it is easier to control the expectation of the RSVB objective than its infimum through Assumption 3.1.\n\n3. Assumption 3.1 is also not a ..... means in simple terms?\n\nR. Thank you for your comment. We actually believe that Assumption 3.1 is quite easy to verify. The condition that the variational family contains Dirac delta distribution at each point in the parameter space is a mild and reasonable requirement for consistency. Further, the assumption requires that the variational family contains sequences of distribution that weakly converge to each Dirac delta distribution at a certain rate. This is easily satisfied if the variational family has no \"holes\". E.g. if it is the family of Gaussians with all means and variances, then we can always construct sequences converging to any Dirac delta at any rate. A similar assumption has also been made in [6,7], and is true for most exponential family distributions. For instance, in the newsvendor application, we fix the variational family to a class of shifted-Gamma distribution and choose a sequence of distribution with parameter sequence $\\alpha=n$ and $\\beta=\\frac{n}{\\theta_0}$ which implies that the sequence of distribution has mean $\\theta_0$ and variance $\\frac{\\theta_0^2}{n}$, and, therefore, it converges to Dirac delta distribution at $\\theta_0$.\n\n4. In the Lacoste-Julien et al paper, they ..... limitation and it is addressed but he authors.\n\nR. Thank you for pointing this out. As we note in our discussion section, we agree that our RSVB formulation requires solving a stochastic minimax optimization problem, which can be difficult in high-dimension without further assumptions. To circumvent this issue [2,3] convert the problem to max-max by switching the risk function with gain, which is a much easier problem to solve computationally. Note that to convert risk to gain, one requires the risk to be bounded above. We emphasize that our theoretical results will continue to hold in the max-max setting, but since our emphasis was on the statistical aspects of this problem, we chose to present our results with minimum assumptions. \n\n[2] S. Lacoste-Julien, F. Huszár, and Z. Ghahramani. Approximate inference for the loss-calibrated Bayesian. AISTATS (2011)\\\n[3] T. Ku ́smierczyk, J. Sakaya, and A. Klami. Variational Bayesian decision-making for continuous utilities. NeurIPS (2019) \\\n[6] Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. JASA (2018)\\\n[7] F. Zhang and C. Gao. Convergence rates of variational posterior distributions. Ann. Stat. (2020)\\\n[8] S. Ghosal, J. K. Ghosh, and A. W. van der Vaart. Convergence rates of posterior distributions. Ann. Stat. (2000)"}, {"review_id": "cZ9mMqfRz3", "reviewer": "Reviewer_j4gK", "summary": "The paper presents a study on data-driven decision-making problems within the Bayesian framework. It addresses situations where calculating the posterior distribution becomes intractable due to the data-generating models’ complexity or the datasets’ size. The authors propose a novel Risk-Sensitive Variational Bayesian (RSVB) framework to overcome these challenges. The RSVB framework leverages a dual representation of the entropic risk measure to approximate the risk-sensitive posterior distribution and derive the corresponding decision rule. The paper also investigates the impact of these computational approximations on the predictive performance of the inferred decision rules. Practical examples in parametric and nonparametric settings support the theoretical findings.", "questions": "NA", "limitations": "Yes / NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovative Approach: Introducing the Risk-Sensitive Variational Bayesian (RSVB) framework significantly contributes to the field. By replacing the expectation in the Bayes risk with a risk-sensitive entropic risk measure, the authors provide a novel solution for decision-making problems in the Bayesian framework where calculating the posterior distribution is intractable.\n\t\t\nPractical Relevance: The paper addresses a common challenge faced in modern applications with large datasets and complex data-generating models. The proposed RSVB framework offers a promising approach to handling such situations, making it relevant and valuable for researchers and practitioners in various domains.\n\t\t\nTheoretical Analysis: The authors provide a comprehensive theoretical analysis of the RSVB framework, including convergence rates of the approximate posterior and the corresponding optimal value. This analysis enhances the understanding of the proposed approach and its theoretical foundations.\n\t\t\nIllustrative Examples: The inclusion of three examples, both parametric and nonparametric, helps to demonstrate the practical applicability of the RSVB framework. By applying the proposed approach to real-world scenarios, the authors strengthen their theoretical findings and showcase the potential benefits of the RSVB framework in different contexts.\n", "weaknesses": "Some areas for improvement are as follows:\n\nClarity of Exposition: While the paper touches upon important concepts and theoretical aspects, the overall clarity of the exposition could be improved. Some technical details may require further explanation or examples to enhance reader comprehension.\n\t\t\nComparison with Existing Methods: It would be beneficial to include a more detailed comparison with existing approaches in the field. Specifically, comparing the proposed RSVB framework with other methods for handling intractable posterior distributions would provide a better understanding of its advantages and limitations.\n\t\t\nEmpirical Evaluation: Although the paper presents illustrative examples, a more extensive empirical evaluation could further support the practical effectiveness of the RSVB framework. Comparing its performance against alternative methods on broader datasets and problem domains would add more robustness to the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for reading through our manuscript and providing insightful comments. Please find our response to your questions below:\n\n1. Clarity of Exposition....\n\nR. To enhance the clarity and exposition of our theoretical contributions, we have added a proof sketch to the main theorems and provided more explanation to Assumption 3.1 (as provided in our response to the first comment of reviewer 3wrT). We have also added more discussion on the other assumptions in the revised manuscript. We would be happy to include any additional clarification in the revised manuscript. \n\n2. Comparison with Existing Methods...\n\nR. Thank you for pointing this out. In our simulation and applications, we do include the existing approaches for risk-sensitive Bayesian decision-making, namely the standard approach VB/VI [1,2] and LCVI [2,3]. We would also like to bring to your attention that our RSVB formulation includes LCVI (i.e. for $\\gamma=1$). We will make these points clear in the revised manuscript. \n\n3. Empirical Evaluation....\n\nR. Thank you for your comment. We would like to emphasize that the main focus of the current manuscript is to introduce the RSVB methodology and derive its theoretical properties. Nonetheless, our theoretical results establish statistical guarantees for a well-known risk-sensitive variational posterior approximation technique LCVI (or LCVB) [2,3], which is a special case of RSVB for $\\gamma=1$. \nWe also demonstrate the efficacy of RSVB compared to VI and LCVI with the help of some simple examples. As extracting the main message from more complicated problem domains with broader datasets is harder,  we present our methodology with these simple examples. We agree with you that evaluating its performance on broader datasets and problem domains would add more robustness to the findings; therefore, we are currently looking at more extensive applications of the proposed approach. However, with an objective to clearly convey the paper's main theoretical findings and be within the page limit, we will present a more detailed empirical evaluation of RSVB elsewhere.\n\n\n[1] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. JASA (2017) \\\n[2] S. Lacoste-Julien, F. Huszár, and Z. Ghahramani. Approximate inference for the loss-calibrated Bayesian. AISTATS (2011)\\\n[3] T. Ku ́smierczyk, J. Sakaya, and A. Klami. Variational Bayesian decision-making for continuous utilities. NeurIPS (2019)"}]

Author rebuttal:
We thank all the reviewers for their insightful and constructive comments. We greatly appreciate the time and effort they have
dedicated to reviewing our work. We have addressed each of them below and accordingly revised the manuscript. We are also attaching a 1-page pdf containing updated figures. 

Meta-review: The paper presents a novel Risk-Sensitive Variational Bayesian (RSVB) framework for addressing data-driven decision-making problems where calculating the posterior distribution is intractable. The paper provides a comprehensive theoretical analysis of the RSVB framework, including convergence rates of the approximate posterior and the corresponding optimal value. It also includes illustrative examples to demonstrate practical applicability. There were some questions and concerns raised by reviewers including a more detailed comparison with existing approaches for handling intractable posterior distributions on broader datasets and problem domains. The paper would benefit from addressing these comments.
Decision: Accept (poster)