Title: On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
Venue: NeurIPS 2023
Accepted: yes
Keywords: Variational Bayes

Loss Calibration

Bayesian Statistics

Variational Inference

Statistical Theory
Abstract: We study data-driven decision-making problems in the Bayesian framework, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure with respect to the posterior distribution. We focus on problems where calculating the posterior distribution is intractable, a typical situation in modern applications with large datasets and complex data generating models. We leverage a dual representation of the entropic risk measure to introduce a novel risk-sensitive variational Bayesian (RSVB) framework for jointly computing a risk-sensitive posterior approximation and the corresponding decision rule. Our general framework includes \textit{loss-calibrated} VB (Lacoste-Julien et al. [2011] ) as a special case. We also study the impact of these computational approximations on the predictive performance of the inferred decision rules. We compute the convergence rates of the RSVB approximate posterior and the corresponding optimal value. We illustrate our theoretical findings in parametric and nonparametric settings with the help of three examples.

On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making

Abstract

We study data-driven decision-making problems in the Bayesian framework, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure with respect to the posterior distribution. We focus on problems where calculating the posterior distribution is intractable, a typical situation in modern applications with large datasets and complex data generating models. We leverage a dual representation of the entropic risk measure to introduce a novel risk-sensitive variational Bayesian (RSVB) framework for jointly computing a risk-sensitive posterior approximation and the corresponding decision rule. Our general framework includes loss-calibrated VB [16] as a special case. We also study the impact of these computational approximations on the predictive performance of the inferred decision rules. We compute the convergence rates of the RSVB approximate posterior and the corresponding optimal value. We illustrate our theoretical findings in parametric and nonparametric settings with the help of three examples.

1 Introduction

This paper focuses on risk-sensitive Bayesian decision-making, considering objective functions of the form

Here A is the decision/action space, θ is a random model parameter lying in an arbitrary measurable space (Θ , T ) , and R ( a, θ ) : A× Θ ↦→ R is a problem-specific model risk function. For any B ⊆ Θ , Π n ( B ) := Π( B | ˜ X n ) is the Bayesian posterior distribution over the parameters given observations ˜ X n from the true model P n θ 0 ( ≡ P n 0 ) with parameter θ 0 ∈ Θ . The scalar γ ∈ R is user-specified and characterizes the sensitivity of the decision-maker (DM) to the distribution Π n . Recall that the posterior distribution is obtained by updating a prior probability distribution Π( · ) , capturing subjective beliefs of the decision maker over Θ , according to the Bayes rule

where dP n θ ( ˜ X n ) is the likelihood of observing ˜ X n . We denote the corresponding densities (if they exist) for the model, prior, and posterior as p n θ ( · ) , π ( · ) , and π ( ·| ˜ X n ) respectively.

The functional ϱ γ · in (SO) is also known as the entropic risk measure , and models a range of riskaverse or risk-seeking behaviors in a succinct manner through the parameter γ . Consider only strictly positive γ , and observe that lim γ ↓ 0 1 γ log E Π n [exp( γR ( a, θ ))] = E Π n ( R ( a, θ )) ; that is, there is no sensitivity to potential risks due to large tail effects and the decision-maker is risk neutral.

Vinayak A. Rao Department of Statistics Purdue University West Lafayette, IN, 47907 varao@purdue.edu

On the other hand, lim γ → + ∞ ϱ γ Π n ( R ( a, θ )) = ess sup Π n ( R ( a, θ )) , the essential supremum of the model risk R ( a, θ ) . In other words, a decision maker is completely risk averse and anticipates the worst possible realization ( Π n -almost surely). Observe that (SO) strictly generalizes the standard Bayesian decision-theoretic formulation of a decision-making problem, where the goal is to solve min a ∈A E Π n [ R ( a, θ )] .

The risk-sensitive formulation (SO) is very general and can be used to model a wide variety of decisionmaking problems in machine learning [16, 15], operations research/management science [24, 7, 20], simulation optimization [6, 32], and finance [17, 1, 4]. However, solving (SO) to compute an optimal decision over A is challenging. The difficulty mainly stems from the fact that, with the exception of conjugate models, the posterior distribution in (1) is an intractable quantity. Canonically, posterior intractability is addressed using either a sampling- or optimization-based approach. Sampling-based approaches, such as Markov chain Monte Carlo (MCMC), offer a tractable way to compute the integrals and theoretical guarantees of exact inference in the large computational budget limit.

Optimization-based methods such as variational Bayes (VB) or variational inference (VI) have emerged as a popular alternative [3]. The VB approximation of the true posterior is a tractable distribution chosen from a 'simpler' family of distributions known as a variational family by minimizing the discrepancy between the true posterior and members of that family. Kullback-Liebler (KL) divergence is the most often used measure of the approximation discrepancy, although other divergences (such as the α -Rényi divergence [19, 28, 14]) have been used. The minimizing member, termed the VB approximate posterior, can be used as a proxy for the true posterior. Empirical studies have shown that VB methods are computationally faster and far more scalable to higher-dimensional problems and large datasets. Theoretical guarantees, such as large sample statistical inference, have been a topic of recent interest in the theoretical statistics community, with asymptotic properties such as convergence rate and asymptotic normality of the VB approximate posterior recently established in [34, 21] and [31, 14] respectively.

Our ultimate goal is not to merely approximate the posterior distribution but also to make decisions when that posterior is intractable. A näive approach would be to plug in the VB approximation in place of the true posterior in (SO) and compute the optimal decision. However, it has been noted in [16] that such a naive loss-unaware approach can be suboptimal. In particular, [16] demonstrated, through an example, that a naive posterior approximation only captures the most dominant mode of the true posterior, which may not be relevant from a decision-making perspective. Consequently, they proposed a loss-calibrated variational Bayesian (LCVB) algorithm for solving Bayesian decisionmaking problems where the underlying risk function is discrete. [15] extended their approach to continuous risk functions. Despite these algorithmic advances in developing decision-centric VB methods, their statistical properties, such as asymptotic consistency and convergence rates of the loss-aware posterior approximation and the associated decision rule are not well understood. With an aim to address these gaps, we summarize our contribution in this paper below:

In Section 2, we introduce a minimax optimization framework titled 'risk-sensitive variational Bayes' (RSVB), extracted from the dual representation of (SO) using the so-called Donsker-Varadhan variational free-energy principle [9]. The decision-maker computes a risk-sensitive approximation to the true posterior (termed as RSVB posterior) and the decision rule simultaneously by solving a minimax optimization problem. Moreover, we recover the LCVB approach [16] as a special case of RSVB (with γ = 1 ).

In Section 3, we identify verifiable regularity conditions on the prior, likelihood model and the risk function under which the RSVB posterior enjoys the same rate of convergence as the true posterior to a Dirac delta distribution concentrated at the true model parameter θ 0 , as the sample size increases. Using this result, we also prove the rate of convergence of the RSVB decision rule when the decision space A is compact. Our theoretical results also imply the asymptotic properties of the LCVB posterior and the associated decision rule.

In Section 4, we demonstrate our theoretical results with the help of three widely studied decision-making problems, including Gaussian process classification and a newsvendor problem. For each example, we show that the rate of convergence of the respective RSVB approximate posterior matches that of the corresponding true posterior.

In Section 4, we also present some simulation results with the single product (1-d) newsvendor problem, which is summarized here in Figure 1. The figures demonstrate the effect of changing γ on the optimality gap in values (see Definition 2.1) and the variance of the

RSVB posterior for a given n . In particular, we observe that for smaller n , increasing γ (after a certain value) results in a significantly more risk-averse decision; however, the effect of increasing γ on risk-averse decision-making reduces as n increases. This observation demonstrates the fact that when there is enough certainty in the parameter estimation, the need to be risk-sensitive to parametric uncertainty diminishes. The Figure 1 also compares the optimality gap in values evaluated at RSVB decision rules and variance of the RSVB posterior for various values of γ against the näive VB approach and against the true posterior when the conjugate prior is used.

20

40

60

80

100

number of samples (n)

10

4

10

3

10

2

10

1

10 0

10 1

R

(

a

,

0)

R

(

a

*

,

0)

(a) Optimality gap in Values

20

40

60

80

100

number of samples (n)

10

6

10

5

10

4

10

3

10

2

10

1

Naive VB/VI

Conjugate

1.0

2.0

4.0

4.5

5.0

6.0

(b) Variance

2 Risk-Sensitive Variational Bayes

Our approach exploits the dual representation of the log-exponential risk measure in (SO), which is convex (or extended coherent) risk measure [23, 10]. From the Donsker-Varadhan variational free energy principle [9] observe that,

where M is the set of all distribution functions that are absolutely continuous with respect to the posterior distribution Π n and 'KL' represents the Kullback-Leibler divergence. Formally, for any two distributions P and Q defined on measurable space (Θ , T ) , the KL divergence is defined as KL ( Q ∥ P ) = ∫ Θ dQ ( θ ) log dQ ( θ ) dP ( θ ) , when measure Q is absolutely continuous with respect to P and ∞ otherwise. Notice that this dual formulation exposes the reason we choose to use the log-exponential risk - the 'free-energy' objective on the RHS provides a combined assessment of the risk associated with model estimation (the KL divergence KL ( Q ∥ Π n ) ) and the decision risk under the estimated posterior Q ( E Q [ R ( a, θ )] ).

As stated above, the reformulation presented in (DV) offers no computational gains. However, restricting ourselves to an appropriately chosen subset Q ⊂ M , that consists of distributions where the integral E q [ R ( a, θ )] and KL ( Q || Π n ) can be tractably computed and optimized, we immediately obtain a risk-sensitive variational Bayesian (RSVB) formulation from (DV):

RSVB is our framework for data-driven risk-sensitive decision-making. The family of distributions Q is popularly known as the variational family . Our analysis in Section 3.1 reveals general guidelines on how to choose Q that ensures a small optimality gap (defined below) with high probability.

With an appropriate choice of Q , the optimization on the RHS can yield a good approximation to the log-exponential risk measurement on the left hand side (LHS). For brevity, for a given a ∈ A we define the RSVB approximation to the true posterior Π( θ | ˜ X n ) as

and the RSVB optimal decision as

Observe that Q ∗ a,γ ( θ | ˜ X n ) and a ∗ RS are random quantities, conditional on the data ˜ X n .

Definition 2.1 (Optimality Gap) . Let V ∗ 0 := min a ∈A R ( a, θ 0 ) be the optimal value for the true model parameter θ 0 . Then, the optimality gap in the value is the difference V -V ∗ 0 .

Examples of Q include the family of Gaussian distributions, delta functions, or the family of factorized 'mean-field' distributions that discard correlations between components of θ . The choice of Q is decisive in determining the performance of the algorithm. Part of the analysis in this paper is to articulate sufficient conditions on Q that ensure small optimality gap (defined below) for the optimal decision, a ∗ RS . This establishes the statistical 'goodness' of the procedure as number of samples increase. In this paper, we analyze the efficacy of the decision rules obtained using the RSVB approximation, by providing high-probability bounds on the optimality gap. We define the optimality gap for any a ∈ A with value V = R ( a , θ 0 ) as,

A similar performance measure was used in [15], to measure the effectiveness of loss-calibrated VB (LCVB) approach, which can be obtained by setting γ = 1 . We provide definitions of some standard terminologies that we use in the subsequent sections, such as covering number, test functions, Γ -convergence, and primal feasibility in the Appendix A for ready reference. In the following section, we lay down important assumptions used throughout the paper to establish our theoretical results.

2.1 Assumptions

In order to bound the optimality gap, we require some control over how quickly the posterior distribution concentrates at the true parameter θ 0 . Our next assumption in terms of a verifiable test condition on the model (sub-)space is one of the conditions required to quantify this rate. Let L n : Θ × Θ ↦→ [0 , ∞ ) be a distance metric on the model space.

Assumption 2.1 (Model indentifiability) . Fix n ≥ 1 . Then, for any ϵ > ϵ n such that ϵ n → 0 as n → ∞ and nϵ 2 n ≥ 1 , there exists a measurable sequence of test functions ϕ n,ϵ : ˜ X n ↦→ [0 , 1] and sieve set Θ n ( ϵ ) ⊆ Θ such that (i) E P n 0 [ ϕ n,ϵ ] ≤ C 0 exp( -Cnϵ 2 ) , and (ii) sup { θ ∈ Θ n ( ϵ ): L n ( θ,θ 0 ) ≥ C 1 nϵ 2 } E P n θ [1 -

Observe that Assumption 2.1 ( i ) quantifies the rate at which a Type I error diminishes with the sample size, while the condition in Assumption 2.1 ( ii ) quantifies that of a Type II error. Notice that both of these are stated through test functions; indeed, what is required are consistent test functions. [11, Theorem 7.1] (stated in Appendix as Lemma C.7 for completeness) roughly implies that an appropriately bounded model subspace { P n θ , θ ∈ Θ } guarantees the existence of consistent test functions, to test the null hypothesis that the true parameter is θ 0 against an alternate hypothesis - the alternate being defined using the distance function L n . Subsequently, we will use a specific distance function to obtain finite sample bounds for the optimality gap in decisions and values. Note that in some cases, it is also possible to construct consistent test functions directly without recourse to Lemma C.7. We demonstrate this in Section 4.1 below. Next, we assume a condition on the prior distribution that ensures that it provides sufficient mass to the sieve set Θ n ( ϵ ) ⊆ Θ , as defined above in Assumption 2.1.

Assumption 2.2. Fix n ≥ 1 . Then, for any ϵ > ϵ n such that ϵ n → 0 as n →∞ and nϵ 2 n ≥ 1 , the prior distribution satisfies Π(Θ c n ( ϵ )) ≤ exp( -Cnϵ 2 ) .

Notice that Assumption 2.2 is trivially satisfied if Θ n ( ϵ ) = Θ . The next assumption ensures that the prior distribution places sufficient mass around a neighborhood of the distribution P n 0 .

Assumption 2.3 (Prior thickness) . Fix n ≥ 1 and a constant λ > 0 . Let A n := { θ ∈ Θ : D 1+ λ

( P n 0 ∥ P n θ ) ≤ C 3 nϵ 2 n } , where D 1+ λ ( P n 0 ∥ P n θ ) := 1 λ log ∫ ( dP n 0 dP n θ ) λ dP n 0 is the Rényi divergence between P n 0 and P n θ , assuming P n 0 is absolutely continuous with respect to P n θ . The prior distribution satisfies Π( A n ) ≥ exp( -nC 2 ϵ 2 n ) .

This assumption guarantees that the prior distribution covers the neighborhood with positive mass. The above three assumptions are adopted from [11] and has also been used in [34] to prove convergence

rates of variational posteriors. Interested readers may refer to [11] and [34] to read more about the above assumptions.

It is apparent by the first term in (RSVB) that in addition to Assumption 2.1, 2.2, and 2.3, we also require regularity conditions on the risk function R ( a, · ) . The next assumption restricts the prior distribution with respect to R ( a, θ ) .

Assumption 2.4. Fix n ≥ 1 and γ > 0 . For any ϵ > ϵ n , a ∈ A , E Π [ ✶ { γR ( a,θ ) >C 4 ( γ ) nϵ 2 } e γR ( a,θ ) ] ≤ exp( -C 5 ( γ ) nϵ 2 ) , where C 4 ( γ ) and C 5 ( γ ) are scalar positive functions of γ .

Note that the set { γR ( a, θ ) > C 4 ( γ ) nϵ 2 } represents the subset of the model space where the risk R ( a, θ ) (for a fixed decision a ) is large, and the prior is assumed to place sufficiently small mass over such sets. Moreover, using the Cauchy-Schwarz inequality observe that E Π [ ✶ { γR ( a,θ ) >C 4 ( γ ) nϵ 2 } e γR ( a,θ ) ] ≤ ( E Π [ ✶ { γR ( a,θ ) >C 4 ( γ ) nϵ 2 } ] ) 1 / 2 ( E Π [ e 2 γR ( a,θ ) ] ) 1 / 2 ≤ e -C 4 ( γ ) nϵ 2 E Π [ e 2 γR ( a,θ ) ] , which implies that if the risk function is bounded in ( a, θ ) , then the assumption is trivially satisfied. Finally, we also require the following condition lower bounding the risk function R .

Note that any risk function which is bounded from below in both the arguments satisfies this condition. In the next section, we establish high-probability bounds on the optimality gap in values and decision rules computed using RSVB approach for sufficiently large n .

3 Asymptotic Analysis of the Optimality Gaps

Our first result, establishes an upper bound on the expected deviation from the true model P n 0 , measured using distance function L n ( · , θ 0 ) , under the RSVB approximate posterior. We also note that the following result generalizes Theorem 2.1 of [34], which is exclusively for the case when γ → 0 + . Our proof techniques follows that of Theorem 2.1 in [34].

Theorem 3.1. Fix a ′ ∈ A and γ > 0 . For any L n ( θ, θ 0 ) ≥ 0 , under Assumptions 2.1, 2.2, 2.3, 2.4, and 2.5, and for min( C, C 4 ( γ ) + C 5 ( γ )) > C 2 + C 3 + C 4 ( γ ) + 2 and η R n ( γ ) := 1 n inf Q ∈Q E P n 0 [ KL ( Q ( θ ) ∥ Π( θ | ˜ X n )) -γ inf a ∈A E Q [ R ( a, θ )] ] , for sufficiently large n the RSVB approximator of the true posterior Q ∗ a ′ ,γ ( θ | ˜ X n ) satisfies, for a positive number M ( γ ) = 2 ( C 1 + MC 4 ( γ )) , where M = 2 C 1 min( C,λ, 1) .

Proof sketch. For brevity we denote the likelihood ratio as LR n ( θ, θ 0 ) = p ( ˜ X n | θ ) p ( ˜ X n | θ 0 ) . We prove this result using a series of lemma. The first Lemma C.1 separates the term η R n ( γ ) , which is later analyzed using Assumption 3.1 in Section 3.1. Lemma C.1 establish that

The proof of Lemma C.1 uses simple arguments that follow easily from Jensen's inequality and the definition of the posterior distribution.

To analyze the first term in the above display, we define the set K n := { θ ∈ Θ : L n ( θ, θ 0 ) > C 1 nϵ 2 } , with an aim to control the exponential moment of L n ( θ, θ 0 ) by characterizing its tails. Also, notice that set K n is the set of alternate hypotheses as defined in Assumption 2.1 through Lemma C.2. For brevity, define Π R ( K n | ˜ X n ) := ∫ Kn e γR ( a ′ ,θ ) LR n ( θ,θ 0 ) d Π( θ ) ∫ Θ e γR ( a ′ ,θ ) LR n ( θ,θ 0 ) d Π( θ ) . Next, we divide the expected calibrated posterior probability of the set K n as follows:

where recall that { ϕ n,ϵ } is the sequence of test function from Assumption 2.1 and set B n = { ˜ X n : ∫ Θ LR n ( θ, θ 0 ) d Π( θ ) ≥ e -(1+ C 3 ) nϵ 2 Π( A n ) } with set A n is defined in Assumption 2.3. Set B n is introduced to separately control (by lower bounding) the denominator in the definition of the posterior distribution in the last term of (4). In addition, the Assumption 2.3 is used to show that P n 0 ( B C n ) ≤ e -λnϵ 2 . The first term can be controlled by Assumption 2.1 (i). Now, it remains to analyze the last term in (4). Using Assumption 2.3 and 2.5 observe that on set B n , ∫ Θ e γR ( a ′ ,θ ) LR n ( θ, θ 0 ) d Π( θ ) ≥ W γ e -(1+ C 2 + C 3 ) nϵ 2 . This observation, with the application of Fubini's theorem, enables us to bifurcate further the last term in (4) on the set S γ ( θ ) := { e γR ( a ′ ,θ ) > e C 4 ( γ ) nϵ 2 } as

˜ C = 1+ C 2 + C 3 + C 4 ( γ ) . Observe that the last term above can be controlled using Assumption 2.4. The first integral in (5) can be further bounded by ∫ K n ∩ Θ n ( ϵ ) E P n θ [(1 -ϕ n,ϵ )] d Π( θ ) + Π(Θ n ( ϵ ) c ) , where Θ n ( ϵ ) is defined in Assumption 2.1 and both the terms can be exponentially bounded using Assumption 2.1 (ii) and 2.2 respectively. The last term in (3) can be bounded using a similar set of arguments and techniques.

Remark: We note that while deriving the bound in Lemma C.1, we can interchange the order of expectation and infimum and compute a tighter bound. However, this bring us back to the RSVB optimizer, the very objective of the whole analysis. Taking expectation before infimum enables us to derive a more interpretable and meaningful bound. This is mainly because it is easier to control the expectation of the RSVB objective than its infimum through Assumption 3.1, as presented in Section 3.1.

The detailed proof of Theorem 3.1 is provided in the Appendix C.2. Now recall that ϵ n is the convergence rate of the true posterior [11, Theorem 7.3]. Notice that the additional term η R n ( γ ) emerges from the posterior approximation and depends on the choice of the variational family Q , risk function R ( · , · ) , and the parameter γ . The appearance of η R n ( γ ) in the bound also signifies that to minimize the expected gap (under the RSVB posterior) between the true model and any other model (defined using n -1 L n ( θ, θ 0 ) ) the expected RSVB objective has to be maximized. Later in this section, we specify the conditions that ensure η R n ( γ ) → 0 as n →∞ . Moreover, we also identify mild regularity conditions on Q to show that η R n ( γ ) is O ( ϵ 2 n ) and show that as γ increases η R n ( γ ) decreases. We discuss this result and the bound therein later in the next subsection. Before that, we establish our main result (the bounds on the optimality gap) using the theorem above. We now fix

Notice that for a given θ , n -1 / 2 √ L n ( θ, θ 0 ) is the uniform distance between the R ( a, θ ) and R ( a, θ 0 ) . Intuitively, Theorem 3.1 implies that the expected uniform difference 1 n L n ( θ, θ 0 ) with respect to the RSVB approximate posterior is O ( M ( γ ) ϵ 2 n + Mη R n ( γ )) , and if M ( γ ) ϵ 2 n + Mη R n ( γ ) → 0 as n →∞ then it converges to zero at that rate.

Also, note that in order to use (6) we must demonstrate that it satisfies Assumption 2.1. This can be achieved by constructing bespoke test functions for a given R ( a, θ ) . We demonstrate this approach by an example in Section B.2. We also provide sufficient conditions for the existence of the test functions in the appendix. These conditions are typically easy to verify when the loss function R ( · , · ) is bounded, for instance. Next, we bound the optimality gap between R ( a ∗ RS , θ 0 ) and V ∗ 0 .

Theorem 3.2. Fix γ > 0 . Suppose the set A is compact. Then, under Assumptions 2.1, 2.2, 2.3, 2.4, and 2.5, for min( C, C 4 ( γ ) + C 5 ( γ )) > C 2 + C 3 + C 4 ( γ ) + 2 and any τ > 0 , the P n 0 -probability of { ˜ X n : R ( a ∗ RS , θ 0 ) -inf a ∈A R ( a, θ 0 ) ≤ 2 τ [ M ( γ ) ϵ 2 n + Mη R n ( γ ) ] 1 2 } is at least 1 -τ -1 , for a positive mapping M ( γ ) = 2 ( C 1 + MC 4 ( γ )) , where M = 2 C 1 min( C,λ, 1) for sufficiently large n .

3.1 Properties of η R n ( γ )

Evidently, the bounds obtained in both the results that we have proved so far depend on η R n ( γ ) . Consequently, we establish some important properties of η R n ( γ ) with respect to n and γ , under additional regularity conditions. In order to characterize η R n ( γ ) , we specify conditions on variational family Q such that η R n ( γ ) = O ( ϵ ′ 2 n ) , for some ϵ ′ n ≥ 1 √ n and ϵ ′ n → 0 . We impose the following condition on the variational family Q that lets us obtain a bound on η R n ( γ ) in terms of n and γ .

Assumption 3.1. There exists a sequence of distributions { q n ( · ) } in the variational family Q such that for a positive constant C 9 ,

If the observations in ˜ X n are i.i.d, then observe that 1 n E Q n ( θ ) [ KL ( dP n 0 ( ˜ X n ) ∥ dP n θ ( ˜ X n ) )] = E Q n ( θ ) [ KL ( dP 0 ∥ dP θ )] . Intuitively, this assumption implies that the variational family must contain a sequence of distributions weakly convergent to a Dirac delta distribution concentrated at the true parameter θ 0 otherwise the second term in the LHS of (7) will be non-zero. Also, note that the above assumption does not imply that the minimizing sequence Q ∗ a ′ ,γ ( θ | ˜ X n ) (automatically) converges weakly to a Dirac-delta distribution at the true parameter θ 0 . Furthermore, unlike Theorem 2.3 of [34], our condition on Q in Assumption 3.1, to obtain a bound on η R n ( γ ) , does not require the support of the distributions in Q to shrink to the true parameter θ 0 at some appropriate rate, as the numbers of samples increases.

The condition that the variational family contains Dirac delta distributions at each point in the parameter space is a mild and reasonable requirement for consistency. Further, Assumption 3.1 requires that Q contains sequences of distributions that weakly converge to each Dirac delta distribution at a certain rate. This is easily satisfied if Q has no 'holes', e.g. if it is the family of Gaussians with all means and variances, then we can always construct sequences converging to any Dirac delta at any rate. A similar assumption has also been made in [31], and is true for most exponential family distributions. For instance, in the newsvendor application, we fix the variational family to a class of shifted-Gamma distributions and choose a sequence of distributions with parameter sequence α = n and β = n/θ 0 . This implies that the sequence of distributions has mean θ 0 and variance θ 2 0 /n , and, therefore, it converges to a Dirac delta distribution at θ 0 . Next, we show that

Proposition 3.1. Under Assumption 3.1 and for a constant C 8 = -inf Q ∈Q inf a ∈A E Q [ R ( a, θ )] and C 9 > 0 , η R n ( γ ) ≤ γn -1 C 8 + C 9 ϵ ′ 2 n .

In Section 4, we present an example where the likelihood is exponentially distributed, the prior is inverse-gamma (non-conjugate), and the variational family is the class of gamma distributions, where we construct a sequence of distributions in the variational family that satisfies Assumption 3.1. We also provide another example where the likelihood is multivariate Gaussian with unknown mean and variational family is uncorrelated Gaussian restricted to a compact subset of R d with a uniform prior on the same compact set satisfy Assumption 3.1.

By definition ϵ 2 n → 0 and ϵ ′ n → 0 as n → ∞ , and therefore it follows from Proposition 3.1 that M ( γ ) ϵ 2 n + Mη R n ( γ ) → 0 . However, the bound obtained in the last proposition might be loose with respect to γ , when C 8 < 0 . To see this, we prove the following result.

Proposition 3.2. If the solution to the optimization problem in η R n ( γ ) is primal feasible then η R n ( γ ) decreases as γ increases.

4 Applications

In the examples below, we use three examples to study the interplay between sample size n and the risk parameter γ , and their effect on the optimality gap in values. Additionally, we consider a multi-product newsvendor example in the Appendix B.2.

4.1 Single-product Newsvendor Model

We start with a canonical data-driven decision-making problem with a 'well-behaved' risk function R ( a, θ ) , the data-driven newsvendor model. This problem has received extensive study in the literature and remains a cornerstone of inventory management [25, 2, 18]. Recall that the newsvendor

loss function is defined as ℓ ( a, ξ ) := h ( a -ξ ) + + b ( ξ -a ) + where h (underage cost) and b (overage cost) are given positive constants, ξ ∈ [0 , ∞ ) the random demand, and a the inventory or decision variable, typically assumed to take values in a compact decision space A with a := min { a : a ∈ A} and ¯ a := max { a : a ∈ A} , and a > 0 . The distribution over the random demand, P θ is assumed to be exponential with unknown rate parameter θ ∈ (0 , ∞ ) . The model risk R ( a, θ ) := E P θ [ ℓ ( a, ξ )] = ha -h θ +( b + h ) e -aθ θ , which is convex in a . We assume that ˜ X n := { ξ 1 , ξ 2 . . . ξ n } be n observations of the random demand, assumed to be i.i.d random samples drawn from P 0 .

We fix the model space Θ = [ T, ∞ ) for some T > 0 and assume that θ 0 lies in the interior of Θ . We now assume a non-conjugate truncated inverse-gamma (Inv -Γ ) prior distribution restricted to Θ , with shape and rate parameter α and β respectively, that is for a set A ⊆ Θ , we define Π( A ) = Inv -Γ Θ ( A ; α, β ) = Inv -Γ( A ∩ Θ; α, β ) / Inv -Γ(Θ; α, β ) . Weverify Assumptions 2.2, 2.1, 2.3, 2.5 and 2.4 (in that order) in this newsvendor setting and provide the proofs in the Appendix B.1. Next, we bound the optimality gap in values for the single product newsvendor model risk.

Theorem 4.1. Fix γ > 0 . Suppose that the set A is compact. Then, for the newsvendor model with exponentially distributed demand with rate θ ∈ Θ = [ T, ∞ ) , prior distribution Π( · ) = Inv -Γ Θ ( · ; α, β ) = Inv -Γ( A ∩ Θ; α, β ) / Inv -Γ(Θ; α, β ) , and the variational family fixed to shifted (by T > 0 ) gamma distributions, and for any τ > 0 , the P n 0 -probability of the following event { ˜ X n : R ( a ∗ RS , θ 0 ) -inf z ∈A R ( z, θ 0 ) ≤ 2 τM ′ ( γ ) ( log n n ) 1 / 2 } is at least 1 -τ -1 for sufficiently large n and for some known mapping M ′ R + R + , where R , θ is the newsvendor model risk.

: → ( · )

The proof of the theorem above is a direct consequence of Theorem 3.2 and Proposition 3.2, and Lemmas B.1, B.2, B.3, B.4, B.5 stated in the Appendix. We also extend the analysis above to a multi-product newsvendor problem. The details are provided in Appendix B.2.

Next, we demonstrate the effect of varying the risk-sensitivity parameter γ . We fix θ 0 = 0 . 1 , b = 1 , h = 5 , α = 1 , and β = 4 . 1 . We run RSVB algorithm with γ ∈ { 0( naive ) , 1 , 2 , 4 . 5 , 5 , 6 } and repeat the experiment over 100 sample paths. We plot the results in Figure 1. In Figure 1(a) we plot the optimality gap in values that is R ( a ∗ RS ( γ ) , θ 0 ) -R ( a ∗ 0 , θ 0 ) for various values of γ . We observe that the gap decreases when n increases. This observation supports our results in Propositions 3.1 and 3.2 that establishes the properties of η R n ( γ ) as n increases. Lastly, in Figure 1(b), we plot the variance of the RSVB posterior as n increases for various values of γ ; as anticipated, the variance reduces as n increases. To observe the effect of γ , first recall that as γ increases the decision maker becomes more risk averse, and so is our algorithmic framework RSVB. Indeed, from the rightmost variance plot in Figure 1, it is evident that for a larger value of γ ( > 4 ), the RSVB posterior is more concentrated on the subset of Θ , where risk is more and consequently we observe large optimality gaps in values. Moreover, as n increases, the effect of larger γ reduces, since as n increases, the incentive to deviate from the posterior reduces (due to increased KL divergence dominance for larger n in RSVB). We also observe that the decision rule learned using the conjugate prior (Gamma distribution for exponential models) has a similar performance as the näive approach. However, the variance of the true conjugate posterior is higher than those computed through the RSVB and VI approach, corresponding to the well known fact that the variational approximations underestimate the true posterior variance ([28, 19]).

4.2 Gaussian process classification

Consider a problem of classifying an input pattern or features Y lying in measure space ([0 , 1] d , Y , ν ) (with sigma algebra Y and probability measure ν ) into one of two classes {-1 , 1 } . Let Y ↦→ ξ ( Y ) ∈ {-1 , 1 } denote the class of Y . For a given Y , we model the classifier using a Bernoulli distribution p ( ξ | Y, θ ) = Ψ ξ ( θ ( Y )) , where θ : [0 , 1] d → R is a non-parametric model parameter in a separable Banach space (Θ , ∥ · ∥ ) and measurable functions Ψ 1 ( x ) = (1 + e -x ) -1 and Ψ -1 ( x ) = 1 -Ψ 1 ( x ) . Note that Ψ 1 ( · ) is a logistic function. We denote ψ ( · ) as the derivative of Ψ 1 ( · ) . Assume that the features Y are generated independently of ξ . Thus the sequence of independent observations { ˜ Y n , ˜ X n } = { ( Y 1 , ξ 1 ) , ( Y 2 , ξ 2 ) , . . . , ( Y n , ξ n ) } are assumed to be generated from the model dP θ ( a, y ) = P θ ( ξ = a, Y ∈ dy ) = p ( ξ = a | Y = y, θ ) ν ( dy ) . In the above binary classification problem, the objective is to estimate θ ( · ) using the observation vector { ˜ Y n , ˜ X n } .

We posit a Gaussian process (GP) prior Π( · ) on θ ( · ) ∈ Θ defined as W ( · ) = ∑ ¯ J α j =1 ∑ 2 jd k =1 µ j Z j,k ϑ j,k ( · ) , where { µ j } is a sequence that decreases with j , { Z i,j } are i.i.d. standard

Gaussian random variables and { ϑ j,k } form a double-indexed orthonormal basis (with respect to measure ν ), that is E ν [ ϑ j,k ϑ l,m ] = ✶ { j = l,k = m } ). ¯ J α is the smallest integer satisfying 2 ¯ J α d = n d/ (2 α + d ) for a given α > 0 . This prior construction is motivated from the work in [30]. We also assume that ν ( · ) is known, and we do not place any prior on it. The posterior distribution over θ ( · ) given observations { ˜ Y n , ˜ X n } can be defined as d Π( θ |{ ˜ Y n , ˜ X n } ) = d Π( θ ) ∏ n i =1 Ψ ξ i ( θ ( Y i )) ν ( Y i ) ∫ ∏ n i =1 Ψ ξ i ( θ ( Y i )) ν ( Y i ) d Π( θ ) = d Π( θ ) ∏ n i =1 Ψ ξ i ( θ ( Y i )) ∫ ∏ n i =1 Ψ ξ i ( θ ( Y i )) d Π( θ ) . Consider the loss function ℓ ( a, ξ ) = { 0 , if a = ξ ; c + , if a = +1 , ξ = -1; c -, if a = -1 , ξ = +1 } , where c + and c -are known positive constants. For a given feature Y , the model risk is given by

We fix the variational family Q GP is a class of Gaussian distributions on Θ , defined as N ( m q , C q ) , m q belongs to Θ and C q is the covariance operator defined as C q = C 1 / 2 ( I -S ) C 1 / 2 , for any S which is a symmetric and Hilbert-Schmidt (HS) operator on Θ (eigenvalues of HS operator are square summable). Note that S and m q span the distributions in Q GP . We can show, using the technical lemmas derived in Section B.3 in Appendix, that the optimality gap in values of the binary GP classification problem converges to zero at a minimax optimal rate (upto logarithmic factors).

Theorem 4.2. Fix γ > 0 and for a given J ∈ N . For the binary GP classification problem with GP prior induced by W = ∑ J j =1 ∑ 2 jd k =1 µ j Z j,k ϑ j,k , where µ j = 2 -jd/ 2 -j a for some a > 0 , { Z i,j } are i.i.d. standard Gaussian random variables and { ϑ j,k } form a double-indexed orthonormal basis (with respect to measure ν ), and ∥ θ 0 ∥ β ; ∞ , ∞ < ∞ , and θ J 0 ( y ) lie in the Cameron-Martin space Im ( C 1 / 2 ) , the variational family Q GP , and for any τ > 0 , the P n 0 -probability of { ˜ X n : R ( a ∗ RS , θ 0 ) -inf z ∈A R ( z, θ 0 ) ≤ 2 τM ′ ( γ ) ϵ n } is at least 1 -τ -1 for sufficiently large n and for some mapping M ′ : R + → R + , where R ( · , θ ) is defined in (8) and

A similar Gaussian process classification problem was studied empirically in [16].

4.3 Eight-schools model

We consider the eight-schools problem [33, 15], where the objective is to study the effectiveness of the special coaching program for SAT exams, offered in 8 schools. Each school reported the treatment effect y i and its standard deviation σ i , where i ∈ { 1 , 2 , 3 , . . . , 8 } . The observations { y i , σ i } 8 i =1 are modeled using the following hierarchical Bayesian model : 1) Prior distributions: µ ∼ N ( ·| 0 , 5) , τ ∼ half-Cauchy ( ·| 0 , 5) , 2) θ i ∼ N ( ·| µ, τ 2 ) and y i ∼ N ( ·| θ i , σ 2 i ) for each i ∈ { 1 , 2 , 3 , . . . , 8 } , where { σ i } 8 i =1 are assumed to be a known sequence of covariates. The posterior distribution is defined as π 8 ( µ, τ, { θ i } 8 i =1 |{ y i , σ i } 8 i =1 ) ∝ N ( ·| 0 , 5) half-Cauchy ( ·| 0 , 5) ∏ 8 i =1 N ( θ i | µ, τ 2 ) N ( y i | θ i , σ 2 i ) . For a given treatment effect for eight schools y ∈ R 8 , the loss function is defined as:

where the decision variable a = { a 1 , a 2 , . . . , a 8 } denotes the effectiveness level. We define R ( a, θ ) = log ∫ p ( y | θ, { σ i } 8 i =1 ) ℓ ( y, a ) dy, θ ∈ R 8 considering { σ i } 8 i =1 is a given sequence of covariates. Now for any γ > 0 , RSVB joint-optimization problem is defined as min a ∈A max Q ∈Q { E q [ R ( a, θ )] -1 γ KL ( q || π 8 ) } . We fix the variational family Q to be the meanfield variational family, that is q Λ ( µ, τ, θ 1 8 i =1 ) = N µ ( ·| λ 9 ) N τ ( ·| λ 10 ) ∏ 8 i =1 N θ i ( | λ i ) , where Λ = { λ 1 , λ 2 , . . . , λ 10 } , λ i is the mean and variance parameter for each Gaussian distribution. Following [15], we measure the performance of the RSVB method using the metric called empirical risk reduction (ERR) , I = ER VB -ER RSVB ( γ ) , where , ER alg = 1 N Y ∑ N Y j =1 l ( y test j , a alg ) , where ER alg denote the empirical risk evaluated at the decision rule a alg obtained using method alg ∈ { VB,RSVB( γ ) }

for various values of γ , and N Y is the size of the test data { y test j } . Note that ERR is the empirical approximation of the difference of optimality gap between the two-step näive VB approach and RSVB ( γ ) approach. Recall, in the näive VB method, we first compute the KL minimizer of the true posterior and then compute the optimal decision using this approximate posterior. Following [15], due to small size of the dataset in this example, the ERR is evaluated on the training data itself.

Wemodified the experiments in [15] by introducing γ = { 5 , 2 . 5 , 1 , 0 . 5 } = { 0 . 2 -1 , 0 . 4 -1 , 1 -1 , 2 -1 } and obtain the results as summarized in Figure 2(a) and (b). In Figure 2(a), we observe that as γ increases, the ERR also increases, which implies that the decisions are more optimistic as empirical loss of the RSVB decision rule decreases as γ increases. Also, observe from Figure 2(b) that, as γ increases, the RSVB posterior (joint marginal posterior distritbution for ( θ 3 , τ ) ) approaches the true posterior, and the variance of the approximate posterior also reduces.

(a) ERR

0

2

4

6

8

10

t

-

10

0

10

20

q

3

Posterior

RSVB-

g

-

1

=

0.2

RSVB-

g

-

1

=

0.4

RSVB-

g

-

1

=

1

RSVB-

g

-

1

=

2

VI/VB

(b)

{

θ

3

, τ

}

Notice that it is not obvious that the variance of the RSVB posterior will reduce as γ increases. We believe that it depends on the landscape of the expected risk function and the choice of the variational family. Intuitively, a possible explanation of this phenomenon can be provided using the equation (RSVB). Consider the RSVB formulation and note that KL > 0 , therefore as γ increases, there is more incentive to deviate from the true posterior and choose Q ∈ Q that maximizes expected risk for a given a ∈ A . Note that a Q that places more mass near the θ that maximizes the risk will be preferred over the one with more spread.

5 Discussion and Future Work

The RSVB formulation as stated in this paper requires us to solve a stochastic minimax optimization problem to compute a decision rule. This is often difficult to solve, particularly in the nonconvexnonconcave setting of RSVB, and indeed, we are not aware of any computationally efficient methods for solving such problems in all generality [8]. To circumvent this, the authors in the LCVB literature maximize utility instead of minimizing risk, and thus convert the whole problem to a much simpler max-max optimization problem. If we replace risk with utility in the RSVB derivation, we will also get a max-max problem. In this formulation, as you increase γ , the incentive to deviate from the posterior and maximize the maximum utility increases. Therefore, increasing γ corresponds to being more optimistic in making decisions than being more risk averse in the risk setting, as observed in the empirical results for the eight schools model in Section 4.3.

We note that we assumed the risk-function is lower-bounded, this is a natural assumption to place on risk functions. Converting our problem to a max-max problem would further require the risk to be upper bounded, which is also assumed for the methodologies presented (without theory) in [16, 15]. We emphasize that our theoretical results will continue to hold in this setting, but since our emphasis was on the statistical aspects of this problem, we chose to present our results with minimum assumptions. Developing an algorithm for solving general minimax optimization in RSVB without transforming it into a max-max problem is open and a part of our future work. Also an interesting topic for future work is identifying minimum additional assumptions to theoretically characterize computational aspects of this problem.

References

D. Bauder, T. Bodnar, N. Parolya, and W. Schmid. Bayesian mean-variance analysis: optimal portfolio selection under parameter uncertainty. Quant. Financ., 21(2):221-242, May 2020. doi: 10.1080/14697688.2020.1748214. URL https://doi.org/10.1080/14697688.2020. 1748214 .

D. Bertsimas and A. Thiele. A data-driven approach to newsvendor problems. Working Paper, Massachusetts Institute of Technology, 2005.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. J. Am. Stat. Assoc., 112(518):859-877, Feb 2017.

T. Bodnar, S. Mazur, and Y. Okhrin. Bayesian estimation of the global minimum variance portfolio. Eur. J. Oper. Res., 256(1):292-307, Jan 2017. doi: 10.1016/j.ejor.2016.05.044. URL https://doi.org/10.1016/j.ejor.2016.05.044 .

S. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.

S. E. Chick. Chapter 9 subjective probability and Bayesian methodology. In Simulation, pages 225-257. Elsevier, 2006. doi: 10.1016/s0927-0507(06)13009-1. URL https://doi.org/10. 1016/s0927-0507(06)13009-1 .

L. Y. Chu, J. Shanthikumar, and Z.-J. M. Shen. Solving operational statistics via a Bayesian analysis. Oper. Res. Lett., 36(1):110-116, Jan 2008. doi: 10.1016/j.orl.2007.04.010. URL https://doi.org/10.1016/j.orl.2007.04.010 .

J. Diakonikolas, C. Daskalakis, and M. I. Jordan. Efficient methods for structured nonconvexnonconcave min-max optimization. In International Conference on Artificial Intelligence and Statistics, pages 2746-2754. PMLR, 2021.

M. D. Donsker and S. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Commun. Pur. Appl. Math., 36(2):183-212, 1983.

H. Föllmer and T. Knispel. Entropic risk measures: Coherence vs. convexity, model ambiguity and robust large deviations. Stoch. Dynam., 11(02n03):333-351, 2011.

S. Ghosal, J. K. Ghosh, and A. W. van der Vaart. Convergence rates of posterior distributions. Ann. Stat., 28(2):500-531, 2000. ISSN 00905364. URL http://www.jstor.org/stable/ 2674039 .

A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics. Int. Stat. Rev., 70(3): 419-435, 2002.

M. Gil, F. Alajaji, and T. Linder. Rényi divergence measures for commonly used univariate continuous distributions. Inform. Sciences, 249:124-131, 2013.

P. Jaiswal, V . Rao, and H. Honnappa. Asymptotic consistency of α -rényi-approximate posteriors. J Mach. Learn. Res., 21(156):1-42, 2020.

T. Ku´ smierczyk, J. Sakaya, and A. Klami. Variational Bayesian decision-making for continuous utilities. In Adv. Neur. In., pages 6395-6405, 2019.

S. Lacoste-Julien, F. Huszár, and Z. Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Int. Conf. Artif. Intell. Stat., pages 416-424, 2011.

T. L. Lai, H. Xing, and Z. Chen. Mean-variance portfolio optimization when means and covariances are unknown. Ann. Appl. Stat., 5(2A), Jun 2011. doi: 10.1214/10-aoas422. URL https://doi.org/10.1214/10-aoas422 .

R. Levi, G. Perakis, and J. Uichanco. The data-driven newsvendor problem: new bounds and insights. Oper. Res., 63(6):1294-1306, 2015.

Y. Li and R. E. Turner. Rényi divergence variational inference. In Adv. Neur. In., pages 1073-1081, 2016.

M. Lu, J. G. Shanthikumar, and Z.-J. M. Shen. Technical note-operational statistics: Properties and the risk-averse case. Nav. Res. Log., 62(3):206-214, 2015.

D. Pati, A. Bhattacharya, and Y. Yang. On statistical optimality of variational Bayes. In A. Storkey and F. Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proc. of Mach. Learn. Res., pages 15791588. PMLR, 09-11 Apr 2018. URL http://proceedings.mlr.press/v84/pati18a. html .

M. H. Quang. Regularized divergences between covariance operators and gaussian measures on hilbert spaces, 2019.

R. T. Rockafellar. Coherent approaches to risk in optimization under uncertainty. In OR Tools and Applications: Glimpses of Future Technologies, pages 38-61. Informs, 2007.

H. Scarf. Bayes solutions of the statistical inventory problem. Ann. Math. Stat., 30(2):490-508, 1959.

H. E. Scarf. Some remarks on Bayes solutions to the inventory problem. Nav. Res. Log., 7(4): 591-596, 1960.

L. Schwartz. On Bayes procedures. Z. Wahrscheinlichkeit., 4(1):10-26, Mar 1965. ISSN 1432-2064. doi: 10.1007/BF00535479. URL https://doi.org/10.1007/BF00535479 .

A. M. Stuart. Inverse problems: A Bayesian perspective. Acta Numer., 19:451-559, May 2010. doi: 10.1017/s0962492910000061. URL https://doi.org/10.1017/ s0962492910000061 .

R. E. Turner and M. Sahani. Two problems with variational expectation maximisation for time-series models. Cambridge University Press, 2011.

A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.

A. W. van der Vaart and J. H. van Zanten. Rates of contraction of posterior distributions based on Gaussian process priors. Ann. Stat., 36(3):1435 -1463, 2008. doi: 10.1214/009053607000000613. URL https://doi.org/10.1214/009053607000000613 .

Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. J. Am. Stat. Assoc., pages 1-15, Jun 2018.

D. Wu, H. Zhu, and E. Zhou. A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM J. Optimiz., 28(2):1588-1612, 2018.

Y. Yao, A. Vehtari, D. Simpson, and A. Gelman. Yes, but did it work?: Evaluating variational inference. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5581-5590. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/ yao18a.html .

F. Zhang and C. Gao. Convergence rates of variational posterior distributions. Ann. Stat., 48(4), Aug 2020. doi: 10.1214/19-aos1883. URL https://doi.org/10.1214/19-aos1883 .

A Additional definitions

We provide the definitions of important terms used throughout the paper. First, recall the definition of covering numbers:

Definition A.1 (Covering numbers) . Let P := { P θ , θ ∈ Θ } be a parametric family of distributions and d : P × P ↦→ [0 , ∞ ) be a metric. An ϵ -cover of a subset P K := { P θ : θ ∈ K ⊂ Θ } of the parametric family of distributions is a set K ′ ⊂ K such that, for each θ ∈ K there exists a θ ′ ∈ K ′ that satisfies d ( P θ , P θ ′ ) ≤ ϵ . The ϵ -covering number of P K is N ( ϵ, P K , d ) = min { card ( K ′ ) : K ′ is an ϵ -cover of K } , where card ( · ) represents the cardinality of the set.

Next, recall the definition of a test function [26]:

Definition A.2 (Test function) . Let ˜ X n be a sequence of random variables on measurable space ( ⊗ n X , S n ) . Then any S n -measurable sequence of functions { ϕ n } , ϕ n : ˜ X n ↦→ [0 , 1] ∀ n ∈ N , is a test of a hypothesis that a probability measure on S n belongs to a given set against the hypothesis that it belongs to an alternative set. The test ϕ n is consistent for hypothesis P n 0 against the alternative P n ∈ { P n θ : θ ∈ Θ \{ θ 0 }} if E P n [ ϕ n ] → ✶ { θ ∈ Θ \{ θ 0 }} ( θ ) , ∀ θ ∈ Θ as n → ∞ , where ✶ {·} is an indicator function.

Aclassic example of a test function is ϕ KS n = ✶ { KS n >K ν } ( θ ) that is constructed using the KolmogorovSmirnov statistic KS n := sup t | F n ( t ) -F θ ( t ) | , where F n ( t ) and F θ ( t ) are the empirical and true distribution respectively, and K ν is the confidence level. If the null hypothesis is true, the GlivenkoCantelli theorem [29, Theorem 19.1] shows that the KS statistic converges to zero as the number of samples increases to infinity.

Furthermore, we define the Hellinger distance h ( θ 1 , θ 2 ) between the two probability distributions

1

/

2

P θ 1 and P θ 2 is defined as d H ( θ 1 , θ 2 ) = ( ∫ (√ dP θ 1 -√ dP θ 2 ) 2 ) . We define the one-sided Hausdorff distance H ( A ∥ B ) between sets A and B in a metric space D with distance function d is defined as:

Next, we define an arbitrary loss function L n : Θ × Θ ↦→ R that measures the distance between models ( P n θ 1 , P n θ 2 ) ∀{ θ 1 , θ 2 } ∈ Θ . At the outset, we assume that L n ( θ 1 , θ 2 ) is always positive. We define { ϵ n } as a sequence such that ϵ n → 0 as n →∞ and nϵ 2 n ≥ 1 .

We also define

Definition A.3 ( Γ -convergence) . A sequence of functions F n : U ↦→ R , for each n ∈ N , Γ -converges to F : U ↦→ R , if

for every u ∈ U and every { u n , n ∈ N } such that u n → u , F ( x ) ≤ lim inf n →∞ F n ( u n );

for every u ∈ U , there exists some { u n , n ∈ N } such that u n → u , F ( x ) ≥ lim sup n →∞ F n ( u n ) .

In addition, we define

Definition A.4 (Primal feasibility) . For any two functions f : U ↦→ R and b : U ↦→ R , a point u ∗ ∈ U is primal feasible to the following constraint optimization problem if b ( u ∗ ) ≤ c , for a given c ∈ R .

B Applications

B.1 Single product newsvendor problem (cont.)

First, we fix the sieve set Θ n ( ϵ ) = Θ , which clearly implies that the restricted inverse-gamma prior Π( θ ) , places no mass on the complement of this set and therefore satisfies Assumption 2.2.

Second, under the condition that the true demand distribution is exponential with parameter θ 0 (and P 0 ≡ P θ 0 ), we demonstrate the existence of test functions satisfying Assumption 2.1.

Lemma B.1. Fix n ≥ 5 . Then, for any ϵ > ϵ n := 1 √ n with ϵ n → 0 , and nϵ 2 n ≥ 1 , there exists a test function ϕ n (depending on ϵ ) such that L NV n ( θ, θ 0 ) = n (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) 2 satisfies Assumption 2.1 with C 0 = 20 and C = C 1 2 ( K NV 1 ) -2 for a constant C 1 > 0 and K NV 1 =

The proof of the above result follows by showing that d NV L = n -1 / 2 √ L NV n ( θ, θ 0 ) can be bounded above by the Hellinger distance between two exponential distributions on Θ (under which a test function exists) in Lemma C.10 in the appendix.

Third, we show that there exist appropriate constants such that the inverse-gamma prior satisfies Assumption 2.3 when the demand distribution is exponential.

Lemma B.2. Fix n 2 ≥ 2 and any λ > 1 . Let A n := { θ ∈ Θ : D 1+ λ ( P n 0 ∥ P n θ ) ≤ C 3 nϵ 2 n } , where D 1+ λ ( P n 0 ∥ P n θ ) is the Rényi divergence between P n 0 and P n θ . Then for ϵ 2 n = log n n and any C 3 > 0 such C 2 = αC 3 ≥ 2 , the truncated inverse-gamma prior Inv -Γ Θ ( A ; α, β ) satisfies Π( A n ) ≥ exp( -nC 2 ϵ 2 n ) , ∀ n ≥ n 2 .

Fourth, it is straightforward to see that the newsvendor model risk R ( a, θ ) is bounded below for a given a ∈ A .

Lemma B.3. For any a ∈ A and positive constants h and b , the newsvendor model risk R ( a, θ ) = ( ha -h θ +( b + h ) e -aθ θ ) ≥ ( ha 2 θ ∗ (1+ aθ ∗ ) ) , where a := min { a ∈ A} and θ ∗ satisfies h -( b + h ) e -aθ ∗ (1 + aθ ∗ ) = 0 .

This implies that R ( a, θ ) satisfies Assumption 2.5. Finally, we also show that the newsvendor model risk satisfies Assumption 2.4.

Lemma B.4. Fix n ≥ 1 and γ > 0 . For any ϵ > ϵ n and any a ∈ A , R ( a, θ ) satisfies E Π [ ✶ { R ( a,θ ) γ>C 4 ( γ ) nϵ 2 } e γR ( a,θ ) ] ≤ exp( -C 5 ( γ ) nϵ 2 ) , for any C 4 ( γ ) > 2 γ ( ha + b T ) and C 5 ( γ ) = C 4 ( γ ) -2 γ ( ha + b T ) , where a := max { a ∈ A} .

Note that Lemma B.1 implies that C = C 1 2( K NV 1 ) 2 for any constant C 1 > 0 . Fixing α = 1 and using Lemma B.2 we can choose C 2 = C 3 = 2 . Now, C 1 can be chosen large enough such that C > C 4 ( γ ) + C 5 ( γ ) for a given risk sensitivity γ > 0 . Therefore, the condition on constants in Theorem 3.1 reduces to C 5 ( γ ) > 2 + C 2 + C 3 = 5 , and it can be satisfied easily by fixing C 5 ( γ ) = 5 . 1 (say).

These lemmas show that when the demand distribution is exponential and with a non-conjugate truncated inverse-gamma prior, our result in Theorem 3.2 can be used for RSVB method to bound the optimality gap in decisions and values for various values of the risk-sensitivity parameter γ . Recall that the bound obtained in Theorem 3.2 depends on ϵ 2 n and η R n ( γ ) .

Lemma B.2 implies that ϵ 2 n = log n n , but in order to get the complete bound we further need to characterize η R n ( γ ) . Recall that, as a consequence of Assumption 3.1 in Proposition 3.1, for a given C 8 = -inf Q ∈Q inf a ∈A E Q [ R ( a, θ )] that C 9 > 0 and η R n ( γ ) ≤ γn -1 C 8 + C 9 ϵ ′ 2 n .

Lemma B.5. Let { Q n ( θ ) } be a sequence of shifted gamma distributions with shape parameter a = n and rate parameter b = n θ 0 , then for truncated inverse gamma prior and exponentially distributed likelihood model

Therefore, in our next result, we show that in the newsvendor setting, we can construct a sequence { Q n ( θ ) } ⊂ Q that satisfies Assumption 3.1, and thus identify ϵ ′ n and the constant C 9 . We fix Q to be the family of shifted gamma distributions with support [ T, ∞ ) .

where ϵ ′ 2 n = log n n and C 9 = 1 2 +max ( 0 , 2 + 2 β θ 0 -log √ 2 π -log ( β α Γ( α ) ) + α log θ 0 ) and prior parameters are chosen such that C 9 > 0 .

B.2 Multi-product newsvendor problem

Analogous to the one-dimensional newsvendor loss function, the loss function in its multi-product version is defined as where h and b are given vectors of underage and overage costs respectively for each product and mapping ( · ) + is defined component-wise. We assume that there are d items or products and ξ ∈ R d denotes the random vector of demands. Let a ∈ A ⊂ R d + be the inventory or decision variable, typically assumed to take values in a compact decision space A with a := {{ min { a i : a i ∈ A i }} d i =1 and ¯ a := {{ max { a i : a i ∈ A i }} d i =1 , and a > 0 , where A i is the marginal set of i th component of A . The random demand is assumed to be multivariate Gaussian, with unknown mean parameter θ ∈ R d but with known covariance matrix Σ . We also assume that Σ is a symmetric positive definite matrix and can be decomposed as Q T Λ Q , where Q is an orthogonal matrix and Λ is a diagonal matrix consisting of respective eigenvalues of Σ . We also define Λ = max i ∈{ 1 , 2 ,...d } Λ ii and Λ = min i ∈{ 1 , 2 ,...d } Λ ii . The model risk

which is convex in a . Here P θ i is the marginal distribution of ξ for i th product, ϕ ( · ) and Φ( · ) are probability and cumulative distribution function of the standard Normal distribution. We also assume that the true mean parameter θ 0 lies in a compact subspace Θ ⊂ R d . We fix the prior to be uniformly distributed on Θ with no correlation across its components, that is π ( A ) = m ( A ⋂ Θ) m (Θ) =

∏ d i =1 m ( A i ⋂ Θ i ) m (Θ i ) , where m ( B ) is the Lebesgue measure (or volume) of B ⊂ R d As in the previous example, we fix the sieve set Θ n ( ϵ ) = Θ , which clearly implies that Π( θ ) places no mass on the complement of this set and therefore satisfies Assumption 2.2.

Then under the condition that the true demand distribution has a multivariate Gaussian distribution (with known Σ ) and mean θ 0 ( P 0 ≡ P θ 0 ), we demonstrate the existence of test functions satisfying Assumption 2.1 by constructing a test function unlike the single-product newsvendor problem with exponential demand.

satisfies Assumption 2.1 with C 0 = 1 , C 1 = 4 K 2 C and C = 1 / 8 ( ˜ C d Λ -1 ) for sufficiently large ˜ C such that C > 1 and Λ = max i ∈{ 1 , 2 ,...d } Λ ii , where K = sup A , Θ ∥ ∂ θ R ( a, θ ) ∥ .

In the following result, we show that there exist appropriate constants such that prior distribution satisfies Assumption 2.3 when the demand distribution is a multivariate Gaussian with unknown mean.

Lemma B.7. Fix n 2 ≥ 2 and any λ > 1 . Let A n := { θ ∈ Θ : D 1+ λ ( P n 0 ∥ P n θ ) ≤ C 3 nϵ 2 n } , where D 1+ λ ( P n 0 ∥ P n θ ) is the Rényi Divergence between P n 0 and P n θ . Then for ϵ 2 n = log n n and any C 3 > 0 such that C 2 = 4 d Λ( λ +1) ( ∏ d i =1 m (Θ i ) ) 2 /d C 3 ≥ 2 and for large enough n , the uncorrelated uniform prior restricted to Θ satisfies Π( A n ) ≥ exp( -nC 2 ϵ 2 n ) .

Next, it is straightforward to see that the multi-product newsvendor model risk R ( a, θ ) is bounded below for a given a ∈ A on a compact set Θ and thus it satisfies Assumption 2.5. Finally, we also show that the newsvendor model risk satisfies Assumption 2.4.

Lemma B.8. Fix n ≥ 1 and γ > 0 . For any ϵ > ϵ n and a ∈ A , R ( a, θ ) satisfies E Π [ ✶ { G ( a,θ ) γ>C 4 ( γ ) nϵ 2 } e γG ( a,θ ) ] ≤ exp( -C 5 ( γ ) nϵ 2 n ) , for any C 4 ( γ ) > 2 γ sup { a,θ }∈A⊗ Θ G ( a, θ ) and C 5 ( γ ) = C 4 ( γ ) -2 γ sup { a,θ }∈A⊗ Θ G ( a, θ ) .

Lemma B.9. Let { Q n ( θ ) } be a sequence of product of d univariate Gaussian distribution defined as q i n ( θ ) ∝ 1 √ 2 πσ 2 i,n e -1 2 σ 2 i,n ( θ -µ i,n ) 2 ✶ Θ i = N ( θ i | µ i,n ,σ i,n ) ✶ Θ i N (Θ i | µ i,n ,σ i,n ) and fix σ i,n = 1 / √ n and θ i = θ i 0 for all i ∈ { 1 , 2 , . . . , d } . Then for uncorrelated uniform distribution restricted to Θ and multivariate normal likelihood model 1 n [ KL ( Q n ( θ ) ∥ Π( θ )) + E Q n ( θ ) [ KL ( dP n 0 ( ˜ X n )) ∥ dP n θ ( ˜ X n ) )]] ≤ C 9 ϵ ′ 2 n , where ϵ ′ 2 n = log n n and C 9 := d 2 +max ( 0 , -∑ d i =1 [log( √ 2 πe ) -log( m (Θ i ))] + d 2 Λ -1 ) .

Similar to single product example, in our next result, we show that in the multi-product newsvendor setting, we can construct a sequence { Q n ( θ ) } ∈ Q that satisfies Assumption 3.1, and thus identify ϵ ′ n and constant C 9 . We fix Q to be the family of uncorrelated Gaussian distributions restricted to Θ .

Now, using the result established in lemmas above, we bound the optimality gap in values for the multi-product newsvendor model risk.

Theorem B.1. Fix γ > 0 . Suppose that the set A is compact. Then, for the multi-product newsvendor model with multivariate Gaussian distributed demand with known covariance matrix Σ and unknown mean vector θ lying in a compact subset Θ ⊂ R d , prior Π( · ) = ∏ d i =1 m ( {·}∩ Θ i ) m (Θ i ) , and the variational family fixed to uncorrelated Gaussian distribution restricted to Θ , and for any τ > 0 , the P n 0 -probability of the following event { ˜ X n : R ( a ∗ RS , θ 0 ) -inf z ∈A R ( z, θ 0 ) ≤ 2 τM ′ ( γ ) ( log n n ) 1 / 2 } is at least 1 -τ -1 for sufficiently large n and for some mapping M ′ : R + → R + , where R ( · , θ ) is the multi-product newsvendor model risk.

Proof. The proof is a direct consequence of Theorem 3.2, Lemmas B.6, B.7, B.8, B.9, and Proposition 3.2.

B.3 Gaussian process classification (cont.)

We define the distance function as L GP n ( θ, θ 0 ) = n (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) 2 . In anticipation of demonstrating that the binary classification model with GP prior and distance function L GP n satisfy the desired set of assumptions, we recall the following result, from [30], which will be central in establishing Assumptions 2.1, 2.2, and 2.3.

Lemma B.10. [Theorem 2.1 [30]] Let θ ( · ) be a Borel measurable, zero-mean Gaussian random element in a separable Banach space (Θ , ∥ · ∥ ) with reproducing kernel Hilbert space (RKHS) ( H , ∥ · ∥ H ) and let θ 0 be contained in the closure of H in Θ . For any ϵ > ϵ n satisfying φ θ 0 ( ϵ ) ≤ nϵ 2 , where

and any C 10 > 1 with e -C 10 nϵ 2 n < 1 / 2 , there exists a measurable set Θ n ( ϵ ) ⊂ Θ such that

The proof of the lemma above can be easily adapted from the proof of [30, Theorem 2.1], which is specifically for ϵ = ϵ n . Notice that the result above is true for any norm ∥ · ∥ on the Banach space if that satisfies φ θ 0 ( ϵ ) ≤ nϵ 2 . Moreover, if φ θ 0 ( ϵ n ) ≤ nϵ 2 n is true, then it also holds for any ϵ > ϵ n , since by definition φ θ 0 ( ϵ ) is a decreasing function of ϵ .

All the results in the previous lemma depend on φ θ 0 ( ϵ ) being less than nϵ 2 . In particular, observe that the second term in the definition of φ θ 0 ( ϵ ) depends on the prior distribution on Θ . Therefore,

[30, Theorem 4.5] show that φ θ 0 ( ϵ n ) ≤ nϵ 2 n ( with ∥ · ∥ as supremum norm and for ϵ n as defined later in (9) ) is satisfied by the Gaussian prior of type

where { µ j } is a sequence that decreases with j , { Z i,j } are i.i.d. standard Gaussian random variables and { ϑ j,k } form a double-indexed orthonormal basis (with respect to measure ν ), that is E ν [ ϑ j,k ϑ l,m ] = ✶ { j = l,k = m } ). ¯ J α is the smallest integer satisfying 2 ¯ J α d = n d/ (2 α + d ) for a given α > 0 . In particular, the GP above is constructed using the function class that is supported on [0 , 1] d and has a wavelet expansion, w ( · ) = ∑ ∞ j =1 ∑ 2 jd k =1 w j,k ϑ j,k ( · ) . The wavelet function space is equipped with the L 2 -norm: ∥ w ∥ 2 = ∑ ∞ j =1 ( ∑ 2 jd k =1 | w j,k | 2 ) 1 / 2 ; the supremum norm: ∥ w ∥ ∞ = ∑ ∞ j =1 2 jd max 1 ≤ k ≤ 2 jd | w j,k | ; and the Besov ( β, ∞ , ∞ ) -norm: ∥ w ∥ β ; ∞ , ∞ = sup 1 ≤ j< ∞ 2 jβ 2 jd max 1 ≤ k ≤ 2 jd | w j,k | . Note that W induces a measure over the RKHS H , defined as a collection of truncated wavelet functions w ( · ) = ∑ ¯ J α j =1 ∑ 2 jd k =1 w j,k ϑ j,k ( · ) , with norm induced by the inner-product on H as ∥ w ∥ 2 H = ∑ ¯ J α j =1 ∑ 2 jd k =1 w 2 j,k µ 2 j . The RKHS kernel K : [0 , 1] d × [0 , 1] d ↦→ R can be easily derived as

Indeed, by the definition of this kernel and inner product, observe that

⟨ K ( x, · ) , w ( · ) ⟩ = ∑ ¯ J α j =1 ∑ 2 jd k =1 w j,k µ 2 j ϑ j,k ( x ) 1 µ 2 j = w ( x ) . Moreover, ⟨ K ( x, · ) , K ( y, · ) ⟩ = ∑ ¯ J α j =1 ∑ 2 jd k =1 µ 2 j ϑ j,k ( x ) µ 2 j ϑ j,k ( y ) 1 µ 2 j = K ( x, y ) . It is clear from its definition that W is a centered Gaussian random field on the RKHS.

Next, using the definition of the kernel, we derive the covariance operator of the Gaussian random field W . Recall that Y ∼ ν , which enables us to define the covariance operator C , following [27, (6.19)] as ( C h ν )( x ) = ∫ [0 , 1] d K ( x, y ) h ν ( y ) dν ( y ) . Also, observe that { µ 2 j , φ j,k } is the eigenvalue and eigen function pair of the covariance operator C . Consequently, using Karhunen Loéve expansion [27, Theorem 6.19] the prior induced by W on H is a Gaussian distribution denoted as N (0 , C ) . We also recall the Cameron-Martin space denoted as Im ( C 1 / 2 ) associated with a Gaussian measure N (0 , C ) on H to be the intersection of all linear spaces of full measure under N (0 , C ) [27, (page 530)]. In particular, Im ( C 1 / 2 ) is the Hilbert space with inner product ⟨· , ·⟩ C = ⟨C -1 / 2 · , C -1 / 2 ·⟩ .

Next, we show the existence of test functions in the following result.

Lemma B.11. For any ϵ > ϵ n with ϵ n → 0 , nϵ 2 n ≥ 2 log 2 , and φ θ 0 ( ϵ ) ≤ nϵ 2 , there exists a test function ϕ n (depending on ϵ ) such that L GP n ( θ, θ 0 ) = n (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) 2 satisfies Assumption 2.1 with C = 1 / 6 , C 0 = 2 and C 1 = (max( c + , c -)) 2 .

Assumption 2.2 is a direct consequence of (14) in Lemma B.10. Next, we prove that prior distribution and the likelihood model satisfy Assumption 2.3 using (15) of Lemma B.10.

Lemma B.12. For any λ > 1 , let A n := { θ ∈ Θ : D 1+ λ ( P n 0 ∥ P n θ ) ≤ C 3 nϵ 2 n } , where D 1+ λ ( P n 0 ∥ P n θ ) is the Rényi Divergence between P n 0 and P n θ . Then for any ϵ > ϵ n satisfying φ θ 0 ( ϵ ) ≤ nϵ 2 and C 3 = 16( λ +1) and C 2 = 1 , the GP prior satisfies Π( A n ) ≥ exp( -nC 2 ϵ 2 n ) .

Assumption 2.4 and 2.5 are straightforward to satisfy since the model risk function R ( a, θ ) is bounded from above and below.

Now, suppose the variational family Q GP is a class of Gaussian distributions on Θ , defined as N ( m q , C q ) , m q belongs to Θ and C q is the covariance operator defined as C q = C 1 / 2 ( I -S ) C 1 / 2 ,

for any S which is a symmetric and Hilbert-Schmidt (HS) operator on Θ (eigenvalues of HS operator are square summable). Note that S and m q span the distributions in Q GP .

Lemma B.13. For a given J ∈ N , let { Q n } be a sequence variational distribution such that Q n is the measure induced by a GP, W Q ( · ) = θ J 0 ( y ) + ∑ J j =1 ∑ 2 jd k =1 ζ 2 j Z j,k ϑ j,k ( · ) , where θ J 0 ( · ) = ∑ J j =1 ∑ 2 jd k =1 θ 0; j,k ϑ j,k ( · ) and ζ 2 j = µ 2 j 1+ nϵ 2 n τ 2 j . Then for GP prior induced by W = ∑ J j =1 ∑ 2 jd k =1 µ j Z j,k ϑ j,k and µ j = 2 -jd/ 2 -ja for some a > 0 , ∥ θ 0 ∥ β ; ∞ , ∞ < ∞ , and θ J 0 ( y ) lie in the Cameron-Martin space Im ( C 1 / 2 ) , we have 1 n KL ( N ( ¯ θ J 0 , C q ) ∥N (0 , C )) + 1 n E Q n KL ( P n 0 ∥ P n θ ) ≤ C 9 ϵ 2 n , where ϵ n is defined in 9 and C 9 := max ( ∥ θ 0 ∥ 2 β, ∞ , ∞ , 2 -2 a -2 -2 Ja -2 a 1 -2 -2 a , 2 d / (2 d -1) , C ′ ) , where C ′ is a positive constant satisfying ∥ θ 0 ( y ) -θ J 0 ( y )) ∥ 2 ∞ ≤ C ′ 2 -2 Jβ .

The following lemma verifies Assumption 3.1, for a specific sequence of distributions in Q .

Using the result above together with Proposition 3.2 implies that the RSVB posterior converges at the same rate as the true posterior, where the convergence rate of the true posterior is derived in [30, Theorem 4.5] for the binary GP classification problem with truncated wavelet GP prior. Finally, we use the results above to obtain bound on the optimality gap in values of the binary GP classification problem.

C Proofs

C.1 Alternative derivation of LCVB

We present the alternative derivation of LCVB. Consider the logarithm of the Bayes posterior risk,

where the inequality follows from an application of Jensen's inequality (since, without loss of generality, exp( R ( a, θ )) > 0 for all a ∈ A and θ ∈ Θ ), and Q ∈ Q . Then, it follows that

C.2 Proof of Theorem 3.1

We prove our main result after a series of important lemmas. For brevity we denote LR n ( θ, θ 0 ) = p ( ˜ X n | θ ) p ( ˜ X n | θ 0 ) .

Lemma C.1. For any a ′ ∈ A , γ > 0 , and ζ > 0 ,

Proof. For any fixed a ′ ∈ A , γ > 0 , and ζ > 0 , and using the fact that KL is non-negative, observe that the integral in the LHS of equation (19) satisfies,

Next, using the definition of Q ∗ a ′ ,γ ( θ | ˜ X n ) in the second term of last equality, for any other Q ( · ) ∈ Q

Finally, it follows from the definition of the posterior distribution that

where the last equality follows from adding and subtracting log E Π [ e γR ( a ′ ,θ ) LR n ( θ, θ 0 ) ] . Now taking expectation on either side of equation (20) and using Jensen's inequality on the first and the last term in the RHS yields

where in the second term in RHS of (20), we first take infimum over all a ∈ A which upper bounds the second term in (20) and then take infimum over all Q ∈ Q , since the LHS does not depend on Q .

Next, we state a technical result that is important in proving our next lemma.

Lemma C.2 (Lemma 6.4 of [34]) . Suppose random variable X satisfies

P ( X ≥ t ) ≤ c 1 exp( -c 2 t ) , for all t ≥ t 0 > 0 . Then for any 0 < β ≤ c 2 / 2 , E [exp( βX )] ≤ exp( βt 0 ) + c 1 .

In the following result, we bound the first term on the RHS of equation (19). The arguments in the proof are essentially similar to Lemma 6.3 in [34]

Lemma C.3. Under Assumptions 2.1, 2.2, 2.3, 2.4, and 2.5 and for min( C, C 4 ( γ ) + C 5 ( γ )) > C 2 + C 3 + C 4 ( γ ) + 2 and any ϵ ≥ ϵ n , for 0 ≤ 10 2 10 = min { 1 } 1 0

Proof. First define the set

where set A n is defined in Assumption 2.3. We demonstrate that, under Assumption 2.3, P n 0 ( B c n ) is bounded above by an exponentially decreasing(in n ) term. Note that for A n as defined in Assumption 2.3:

Let d ˜ Π( θ ) := ✶ { Θ ∩ An } ( θ ) Π( A n ) d Π( θ ) , and use this in (24) for any λ > 0 to obtain, P n 0 (∫ Θ LR n ( θ, θ 0 ) d ˜ Π( θ ) ≤ e -(1+ C 3 ) nϵ 2 ) = P n 0 ( [∫ Θ LR n ( θ, θ 0 ) d ˜ Π( θ ) ] -λ ≥ e (1+ C 3 ) λnϵ 2 ) .

Then, using the Markov's inequality in the last equality above, we have

where the second inequality follows from first applying Jensen's inequality (on the term inside [ · ] ) and then using Fubini's theorem, and the penultimate inequality follows from Assumption 2.3 and the definition of ˜ Π( θ ) .

Next, define the set K n := { θ ∈ Θ : L n ( θ, θ 0 ) > C 1 nϵ 2 } . Notice that set K n is the set of alternate hypothesis as defined in Assumption 2.1. We bound the calibrated posterior probability of this set K n to get a bound on the first term in the RHS of equation (19). Recall the sequence of test function { ϕ n,ϵ } from Assumption 2.1. Observe that

where in the second inequality, we first divide the second term over set B n and its complement and then use the fact that ∫ Kn e γR ( a ′ ,θ ) LR n ( θ,θ 0 ) d Π( θ ) ∫ Θ e γR ( a ′ ,θ ) LR n ( θ,θ 0 ) d Π( θ ) ≤ 1 . The third inequality is due the fact that ϕ n,ϵ ∈ [0 , 1] . Next, using Assumption 2.3 and 2.5 observe that on set B n

Substituting the equation above in the third term of equation (26), we obtain

Now using Fubini's theorem observe that,

where in the last inequality, we first divide the integral over set { θ ∈ Θ : e γR ( a ′ ,θ ) ≤ e C 4 ( γ ) nϵ 2 } and its complement and then use the upper bound on e γR ( a ′ ,θ ) in the first integral. Now, it follows that

where the second equality is obtained by dividing the first integral on set Θ n ( ϵ ) and its complement, and the second inequality is due the fact that ϕ n,ϵ ∈ [0 , 1] . Now, using the equation above and Assumption 2.1, 2.2, and 2.4 observe that

Hence, choosing C, C 2 , C 3 , C 4 ( γ ) and C 5 ( γ ) such that -1 > 1+ C 2 + C 3 + C 4 ( γ ) -min( C, ( C 4 ( γ )+ C 5 ( γ ))) implies

Therefore, substituting equation (25), equation (27), and (28) into (26), we obtain

where C 10 = min { λ, C, 1 } /C 1 . Using Fubini's theorem, observe that the LHS in the equation (29) can be expressed as µ ( K n ) , where

Next, recall that the set K n = { θ ∈ Θ : L n ( θ, θ 0 ) > C 1 nϵ 2 } . Applying Lemma C.2 above with X = L n ( θ, θ 0 ) , c 1 = (1 + C 0 +3 W -γ ) , c 2 = C 10 , t 0 = C 1 nϵ 2 n , and for 0 < ζ ≤ C 10 / 2 , we obtain

Further, we have another technical lemma, that will be crucial in proving the subsequent lemma that upper bounds the last term in the equation (19).

Lemma C.4. Suppose a positive random variable X satisfies

for all t ≥ t 0 > 0 , c 1 > 0 , and c 2 > 0 . Then,

Proof. For any Z 0 > 1 ,

Therefore, choosing Z 0 = exp( t 0 ) ,

Next, we establish the following bound on the last term in equation (19).

Lemma C.5. Under Assumptions 2.1, 2.2, 2.3, 2.4, 2.5, and for C 5 ( γ ) > C 2 + C 3 +2 ,

for any λ ≥ 1 + C 4 ( γ ) .

Proof. Define the set

Using the set B n in equation (23), observe that the measure of the set M n , under the posterior distribution satisfies,

Now, the second term of equation (33) can be bounded as follows: recall Assumption 2.3 and the definition of set B n , both together imply that,

Then, using Fubini's Theorem ( ⋆⋆ ) = e (1+ C 2 + C 3 ) nϵ 2 Π( M n ) . Next, using the definition of set M n and then Assumption 2.4, we obtain

Hence, choosing the constants C 2 , C 3 , C 4 ( γ ) and C 5 ( γ ) such that -1 > 1 + C 2 + C 3 -C 5 ( γ ) implies

Therefore, substituting (25) and (34) into (33)

where C 11 ( γ ) = min { λ, 1 + C 4 ( γ ) } /C 4 ( γ ) -1 . Using Fubini's theorem, observe that the RHS in (35) can be expressed as ν ( M n ) , where the measure

Applying Lemma C.4 for X = e γR ( a ′ ,θ ) , c 1 = 2 , c 2 = C 11 ( γ ) , t 0 = C 4 ( γ ) nϵ 2 n and λ ≥ 1 + C 4 ( γ ) , we obtain

Proof. Proof of Theorem 3.1: Finally, recall (19),

Substituting (31) and (22) into the equation above and then using the definition of η R n ( γ ) , we get

where the last inequality uses the fact that log x ≤ x -1 . Choosing ζ = C 10 / 2 = min( C,λ, 1) 2 C 1 ,

where M ( γ ) = C 1 + 1 ζ C 4 ( γ ) and M ′ = 1 ζ depend on C, C 1 , C 4 ( γ ) , W and λ . Since the last two terms in (38) decrease and the first term increases as n increases, we can choose M ′ large enough, such that for all n ≥ 1

and therefore for M = 2 M ′ ,

Also, observe that the LHS in the above equation is always positive, therefore M ( γ ) ϵ 2 n + Mη R n ( γ ) ≥ 0 ∀ n ≥ 1 and γ > 0 .

C.3 Proof of Theorem 3.2

Lemma C.6. Given a ′ ∈ A and for a constant M, as defined in Theorem 3.1

Proof. First, observe that

where the last inequality follows from Jensen's inequality. Now, using the Jensen's inequality again

Now, using Theorem 3.1 the result follows immediately.

Proof of Theorem 3.2. Observe that

Given a ∗ RS ∈ A and for a constant M (defined in Theorem 3.1), we have from Lemma C.6 for a ′ = a ∗ RS

It follows from above that the P n 0 -probability of the following event is at least 1 -τ -1 :

C.4 Proofs in Section 3.1

Proof of Proposition 3.1. Using the definition of η R n ( γ ) and the posterior distribution Π( θ | ˜ X n ) , observe that

Now, using Fubini's in the last term of the equation above, we obtain

Observe that, ∫ X n ∫ d Π( θ ) p ( ˜ X n | θ ) d ˜ X n = 1 . Since, KL is always non-negative, it follows from the equation above that

where the last inequality follows from the following fact, for any functions f ( · ) and g ( · ) ,

Recall ϵ ′ n ≥ 1 √ n . Now, using Assumption 3.1, it is straightforward to observe that the first term in (45),

Now consider the last term in (45). Notice that the coefficient of 1 n is independent of n and is bounded from below. Therefore, there exist a constant C 8 = -inf Q ∈Q inf a ∈A E Q [ R ( a, θ )] , such that with equation (46) it follows that η R n ( γ ) ≤ γn -1 C 8 + C 9 ϵ ′ 2 n and the result follows.

Proof of Proposition 3.2. First recall that

Observe that the optimization problem is equivalent to solving :

Now for any γ > 0 , Q ∗ γ ( θ ) ∈ Q that minimizes the objective in (47) is primal feasible if

Therefore, it is straightforward to observe that as γ increases nη R n ( γ ) decreases that is

C.5 Sufficient conditions on R ( a, θ ) for existence of tests

To show the existence of test functions, as required in Assumption 2.1, we will use the following result from [11, Theorem 7.1], that is applicable only to distance measures that are bounded above by the Hellinger distance.

Lemma C.7 (Theorem 7.1 of [11]) . Suppose that for some non-increasing function D ( ϵ ) , some ϵ n > 0 and for every ϵ > ϵ n ,

where m ( · , · ) is any distance measure bounded above by Hellinger distance. Then for every ϵ > ϵ n , there exists a test ϕ n (depending on ϵ > 0 ) such that, for every j ≥ 1 ,

Proof of Lemma C.7: Refer Theorem 7.1 of [11].

For the remaining part of this subsection we assume that Θ ⊆ R d . In the subsequent paragraph, we state further assumptions on the risk function to show L n ( · , · ) as defined in (6) satisfies Assumption 2.1. For brevity we denote n -1 / 2 √ L n ( θ, θ 0 ) by d L ( θ, θ 0 ) , that is

and the covering number of the set T ( ϵ ) := { P θ : d L ( θ, θ 0 ) < ϵ } as N ( δ, T ( ϵ ) , d L ) , where δ > 0 is the radius of each ball in the cover. We assume that the risk function R ( a, · ) satisfies the following bound.

Assumption C.1. The model risk satisfies

where d H ( θ 1 , θ 2 ) is the Hellinger distance between two models P θ 1 and P θ 2 .

For instance, suppose the definition of model risk is R ( a, θ ) = ∫ X ℓ ( x, a ) p ( y | θ ) dx , where ℓ ( x, a ) is an underlying loss function. Then, observe that Assumption C.1 is trivially satisfied if ℓ ( x, a ) is bounded in x for a given a ∈ A and A is compact, since d L ( θ 1 , θ 2 ) can be bounded by the total variation distance d TV ( θ 1 , θ 2 ) = 1 2 ∫ | dP θ 1 ( x ) -dP θ 2 ( x ) | and total variation distance is bounded above by the Hellinger distance [12]. Under the assumption above it also follows that we can apply Lemma C.7 to the metric d L ( · , · ) defined in (49). Now, we will also assume an additional regularity condition on the risk function.

Assumption C.2. For every { θ 1 , θ 2 } ∈ Θ , there exists a constant K 2 > 0 such that

We can now show that the covering number of the set T ( ϵ ) satisfies

Lemma C.8. Given ϵ > δ > 0 , and under Assumption C.2,

Proof of Lemma C.8: For any positive k and ϵ , let θ ∈ [ θ 0 -kϵ, θ 0 + kϵ ] d ⊂ Θ ⊂ R d . Now consider a set H i = { θ 0 i , θ 1 i , . . . θ J i , θ J +1 i } and H = ⊗ d H i with J = ⌊ 2 kϵ δ ′ ⌋ , where θ j i = θ 0 -kϵ + iδ ′ for j = { 0 , 1 , . . . , J } and θ J +1 i = θ 0 + kϵ . Observe that for any θ ∈ [ θ 0 -kϵ, θ 0 + kϵ ] d , there exists a θ j ∈ H such that ∥ θ -θ j ∥ < δ ′ . Hence, union of the δ ′ -balls for each element in set H covers [ θ 0 -kϵ, θ 0 + kϵ ] d , therefore N ( δ ′ , [ θ 0 -kϵ, θ 0 + kϵ ] d , ∥ · ∥ ) = ( J +2) d .

Now, due to Assumption C.2, for any θ ∈ [ θ 0 -kϵ, θ 0 + kϵ ] d

For brevity, we denote n -1 L n ( θ, θ 0 ) by d L ( θ, θ 0 ) , that is

and the covering number of the set T ( ϵ ) := { P θ : d L ( θ, θ 0 ) < ϵ } as N ( δ, T ( ϵ ) , d L ) , where δ > 0 is the radius of each ball in the cover.

Hence, δ ′ -cover of set [ θ 0 -kϵ, θ 0 + kϵ ] d is K 1 δ ′ cover of set T ( ϵ ) with k = 1 /K 2 . Finally,

which implies for δ = K 2 δ ′ ,

Observe that the RHS in (50) is a decreasing function of δ , infact for δ = ϵ/ 2 , it is a constant in ϵ . Therefore, using Lemmas C.7 and C.8, we show in the following result that L n ( θ, θ 0 ) in (6) satisfies Assumption 2.1.

Lemma C.9. Fix n ≥ 1 . For a given ϵ n > 0 and every ϵ > ϵ n , such that nϵ 2 n ≥ 1 . Under Assumption C.1 and C.2, L n ( θ, θ 0 ) = n (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) 2 satisfies

where C 0 = 2 ∗ 10 s and C = C 1 2 K 2 1 for a constant C 1 > 0 .

Proof of Lemma C.9: Recall d L ( θ, θ 0 ) = (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) and T ( ϵ ) = { P θ : d L ( θ, θ 0 ) < ϵ } . Using Lemma C.8, observe that for every ϵ > ϵ n > 0 ,

Next, using Assumption C.1 we have

It follows from the above two observations and Lemma 2 that, for every ϵ > ϵ n > 0 , there exist tests { ϕ n,ϵ } such that

where C ′ = 1 2 K 2 1 . Since the above two conditions hold for every ϵ > ϵ n , we can choose a constant K > 0 such that for every ϵ > ϵ n

where the second inequality in (56) holds ∀ n ≥ n 0 , where n 0 := min { n ≥ 1 : C ′ K 2 nϵ 2 ≥ log(2) } Hence, the result follows for C 1 = K 2 and C = C ′ K 2 .

Since L n ( θ, θ 0 ) = 1 n d 2 L satisfies Assumption 2.1, Theorem 3.1 implies the following bound.

Corollary C.1. Fix a ′ ∈ A and γ > 0 . Let ϵ n be a sequence such that ϵ n → 0 as n →∞ , nϵ 2 n ≥ 1 and

Then under the Assumptions of Theorem 3.1 and Lemma C.9 ; for C = C 1 2 K 2 1 , C 0 = 2 ∗ 10 s , C 1 > 0 such that min( C, C 4 ( γ )+ C 5 ( γ )) > C 2 + C 3 + C 4 ( γ )+2 , and for η R n ( γ ) as defined in Theorem 3.1, the RSVB approximator of the true posterior Q ∗ a ′ ,γ ( θ | ˜ X n ) satisfies,

for sufficiently large n and for a function M ( γ ) = 2 ( C 1 + MC 4 ( γ )) , where M = 2 C 1 min( C,λ, 1) .

Proof of Corollary C.1: Using Lemma C.9 observe that for any Θ n ( ϵ ) ⊆ Θ , L n ( θ, θ 0 ) satisfies Assumption 2.1 with C 0 = 2 ∗ 10 s , C = C 1 2 K 2 1 and for any C 1 > 0 , since

Hence, applying Theorem 3.1 the proof follows.

C.6 Newsvendor Problem

We fix n -1 / 2 √ L NV n ( θ, θ 0 ) = (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) . Next, we aim to show that the exponentially distributed model P θ satisfies Assumption 2.1, for distance function L NV n ( θ, θ 0 ) . To show this, in the next result we first prove that d NV L ( θ, θ 0 ) = n -1 / 2 √ L NV n ( θ, θ 0 ) satisfy Assumption C.1. Also, recall that the square of Hellinger distance between two exponential distributions with rate √ √

Lemma C.10. For any θ ∈ Θ = [ T, ∞ ) , and a ∈ A ,

where a := min { a ∈ A} and a > 0 and θ 0 lies in the interior of Θ .

Proof. Observe that for any a ∈ A ,

where the last inequality follows since for θ ≥ θ 0 , ( h θ 0 -h θ ) ≥ 0 and ( e -aθ θ -e -aθ 0 θ 0 ) < 0 and vice versa if θ < θ 0 that together makes the last term in the penultimate equality negative for all θ ∈ Θ . Moreover, the first derivative of the upperbound with respect to θ is

and it is negative when θ ≤ θ 0 and positive when θ > θ 0 for all b > 0 , h > 0 , and a ∈ A . Therefore, the upperbound in (59) above is decreasing function of θ for all θ ≤ θ 0 and increasing function of θ for all θ > θ 0 . The upperbound is tight at θ = θ 0 .

Now recall that the squared Hellinger distance between two exponential distributions with rate parameter θ and θ 0 is

Note that for θ ≤ θ 0 , d 2 H ( θ, θ 0 ) is a decreasing function of θ and for all θ > θ 0 it is an increasing function of θ . Also, note that as θ →∞ , the squared Hellinger distance as well as the upperbound computed in (59) converges to a constant for a given h, b, θ 0 and a . However, as θ → 0 , the d 2 H ( θ, θ 0 ) → 1 but the upperbound computed in (59) diverges.

Since, Θ = [ T, ∞ ) for some T > 0 and T ≤ θ 0 , observe that if we scale d 2 H ( θ, θ 0 ) by factor by which the upperbound computed in (59) is greater than d H at θ = T , then

where a = inf { a : a ∈ A} and in the last inequality we used the fact that ( e -aT T -e -aθ 0 θ 0 ) 2 is a decreasing function of a for any b, h, T, and θ 0 . Since, the RHS in the equation above does not depend on a , it follows from the result in (59) and the definition of L NV n ( θ, θ 0 ) that

Lemma C.11. For any θ ∈ Θ = [ T, ∞ ) , for sufficiently small T > 0 , and θ 0 lying in the interior of Θ , we have

Proof. Observe that

Observe that θ → 0 , ∂d 2 H ( θ,θ 0 ) ∂θ →∞ . Since, θ ∈ Θ = [ T, ∞ ) , therefore the sup θ ∈ Θ ∣ ∣ ∣ ∂d 2 H ( θ,θ 0 ) ∂θ ∣ ∣ ∣ < ∞ . In fact, for sufficiently small T > 0 , sup θ ∈ Θ ∣ ∣ ∣ ∂d 2 H ( θ,θ 0 ) ∂θ ∣ ∣ ∣ = ∣ ∣ ∣ ∣ θ 0 ( T + θ 0 ) 2 ( √ T θ 0 -√ θ 0 T )∣ ∣ ∣ ∣ =

( θ 0 ( T + θ 0 ) 2 (√ θ 0 T -√ T θ 0 )) . Now the result follows immediately since the derivative of d 2 H ( θ, θ 0 ) is bounded on Θ , which implies that d 2 H ( θ, θ 0 ) is Lipschitz on Θ .

Lemma C.12. For any θ ∈ Θ = [ T, ∞ ) , and a ∈ A ,

First, observe that for any a ∈ A ,

The result follows immediately, since sup θ ∈ Θ ∂R ( a,θ ) ∂θ ≤ h T 2 .

Proof. Proof of Lemma B.1

It follows from Lemma C.10 that d NV L ( θ, θ 0 ) for any θ ∈ Θ = [ T, ∞ ) and θ 0 lying the interior of Θ , satisfies Assumption C.1 with

. Similarly, it follows from Lemma and C.12 that for sufficiently small T > 0 , d NV L ( θ, θ 0 ) satisfies Assumption C.2 with K 2 = h/T 2 := K NV 2 . Now using similar arguments as used in Lemma C.8 and Lemma 2.1, for a given ϵ n > 0 and every ϵ > ϵ n , such that nϵ 2 n ≥ 1 , it can be shown that , L NV n ( θ, θ 0 ) = n (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) 2 satisfies

Proof. Recall,

where C 0 = 20 and C = C 1 2( K NV 1 ) 2 for a constant C 1 > 0 .

Proof. Proof of Lemma B.2:

First, we write the Rényi divergence between P n 0 and P n θ ,

when (( λ +1) θ 0 -λθ ) > 0 and D 1+ λ ( P n 0 ∥ P n θ ) = ∞ otherwise. Also, observe that, D 1+ λ ( P n 0 ∥ P n θ ) is non-decreasing in λ (this also follows from non-decreasing property of the Rényi divergence with respect to λ ). Therefore, observe that

Now, recall that for a set A ⊆ Θ = [ T, ∞ ) , we define Π( A ) = Inv -Γ( A ∩ Θ) / Inv -Γ(Θ) . Now, observe that for sufficiently small T and large enough n , we have

The cumulative distribution function of inverse-gamma distribution is Inv -Γ( { θ ∈ Θ : θ < t } ) := Γ ( α, β t ) Γ( α ) , where α ( > 0) is the shape parameter, β ( > 0) is the scale parameter, Γ( · ) is the Gamma function, and Γ( · , · ) is the incomplete Gamma function. Therefore, it follows for α > 1 that

where the penultimate inequality folows since 0 < ϵ 2 n < 1 and the last inequality follows from the fact that, 1 -e -αC 3 ϵ 2 n ≥ e -αC 3 nϵ 2 n , for large enough n . Also note that, 1 -e -αC 3 ϵ 2 n ≥ e -αC 3 nϵ 2 n can't hold true for ϵ 2 n = 1 /n . However, for ϵ 2 n = log n n it holds for any n ≥ 2 when αC 3 > 2 . Therefore, for inverse-Gamma prior restricted to Θ , C 2 = αC 3 and any λ > 1 the result follows for sufficiently large n .

Proof. Proof of Lemma B.3: Recall,

First, observe that for any a ∈ A ,

Using the above equation the (finite) critical point θ ∗ must satisfy, h -( b + h ) e -aθ ∗ (1 + aθ ∗ ) = 0 . Therefore,

Since h, b > 0 and aθ ∗ > 0 , hence

where a := min { a ∈ A} and a > 0 .

Proof. Proof of Lemma B.4:

First, observe that R ( a, θ ) is bounded above in θ for a given a ∈ A

Using the above fact and the Cauchy-Schwarz inequality, we obtain

where the last inequality follows from using the Chebyshev's inequality.

Now using the definition of the prior distribution, which is an inverse gamma prior restricted to Θ = [ T, ∞ ) , we have where a := max { a ∈ A} and a > 0 . Since nϵ 2 n ≥ 1 , we must fix C 4 ( γ ) such that e C 4 ( γ ) > e 2 γ ( ha + b T ) , that is C 4 ( γ ) > 2 γ ( ha + b T ) and C 5 ( γ ) = C 4 ( γ ) -2 γ ( ha + b T ) .

Proof. Proof of Lemma B.5: Since family Q contains all shifted-gamma distributions, observe that { q n ( · ) ∈ Q}∀ n ≥ 1 . By definition, q n ( θ ) = n n θ n 0 Γ( n ) ( θ -T ) n -1 e -n ( θ -T ) θ 0 . Now consider the first term; using the definition of the KL divergence it follows that

Substituting q n ( θ ) in the first term of the equation above and expanding the logarithm term, we obtain

Now consider the second term in the equation above. Substitute θ = tθ 0 n + T into the integral, we have

Substituting the above result into (66), we get

where the second inequality uses the fact that √ 2 πnn n e -n ≤ n Γ( n ) . Recall π ( θ ) = β α Γ( α ) θ -α -1 e -β θ . Now consider the second term in (65). Using the definition of inverse-gamma prior and expanding the logarithm function, we have

where the first inequality is due to fact that E q n [ β/θ ] ≤ E q n [ β/ ( θ -T )] for any θ > T and the penultimate inequality follows from the observation in (67) and the fact that log θ θ 0 ≤ θ θ 0 -1 ≤ θ θ 0 -T θ 0 for any θ 0 > T . Substituting (69) and (68) into (65) and dividing either sides by n , we obtain

Now consider the second term in the assertion of the lemma. Since ξ i , i ∈ { 1 , 2 . . . n } are independent and identically distributed, we obtain

Now using the expression for KL divergence between the two exponential distributions, we have

where second inequality uses the fact that log x ≤ x -1 ≤ x -T θ 0 for θ 0 > T . Combined together (71) and (70) for n ≥ 2 implies that

where C 9 := 1 2 +max ( 0 , 2 + 2 β θ 0 -log √ 2 π -log ( β α Γ( α ) ) + α log θ 0 ) and the result follows.

Proof. Proof of Lemma B.5: Since family Q contains all gamma distributions, observe that { q n ( · ) ∈ Q}∀ n ≥ 1 . By definition, q n ( θ ) = n n θ n 0 Γ( n ) θ n -1 e -n θ θ 0 . Now consider the first term; using the definition of the KL divergence it follows that

Substituting q n ( θ ) in the first term of the equation above and expanding the logarithm term, we obtain

Now consider the second term in the equation above. Substitute θ = tθ 0 n into the integral, we have

Substituting the above result into (74), we get

where the second inequality uses the fact that √ 2 πnn n e -n ≤ n Γ( n ) . Recall π ( θ ) = β α Γ( α ) θ -α -1 e -β θ . Now consider the second term in (73). Using the definition of inverse-gamma prior and expanding the logarithm function, we have

where the last inequality follows from the observation in (75). Substituting (77) and (76) into (73) and dividing either sides by n , we obtain

Now, consider the second term in the assertion of the lemma. Since, ξ i , i ∈ { 1 , 2 . . . n } are independent and identically distributed, we obtain

Now using the expression for KL divergence between the two exponential distributions, we have

where second inequality uses the fact that log x ≤ x -1 . Combined together (79) and (78) for n ≥ 2 implies that

where C 9 := 1 2 +max ( 0 , 2 + 2 β θ 0 -log √ 2 π -log ( β α Γ( α ) ) + α log θ 0 ) and the result follows.

C.7 Multi-product Newsvendor problem

In the multi-dimensional newsvendor problem, we fix n -1 / 2 √ L MNV n ( θ, θ 0 ) = (sup a ∈A | R ( a, θ ) -R ( a, θ 0 ) | ) , where R ( a, θ ) = ∑ d i =1 [ ( h i + b i ) a i Φ( a i ) -b i a i + θ i ( b i -h i ) + σ ii [ h ϕ (( a i -θ i ) /σ ii ) Φ(( a i -θ i ) /σ ii ) + b ϕ (( a i -θ i ) /σ ii ) 1 -Φ(( a i -θ i ) /σ ii ) ] ] .

For brevity, we denote d MNV L ( θ, θ 0 ) = n -1 / 2 √ L MNV n ( θ, θ 0 ) . First, we show that Lemma C.13. For any compact decision space A and compact model space Θ ,

for a constant K depending on compact sets A and Θ and given b, h and Σ .

Proof. Observe that

∂ θ i R ( a, θ )

Since, A and Θ are compact sets, therefore { ( a i -θ i ) /σ ii } d i =1 lie in a compact set. Consequently, ϕ (( a i -θ i ) /σ ii ) and Φ(( a i -θ i ) /σ ii ) also lie in bounded subset of R and thus sup A , Θ ∥ ∂ θ i R ( a, θ ) ∥ ≤ K for a given b , h and Σ . Since , the norm of the derivative of R ( a, θ ) is bounded on Θ for any a ∈ A , therefore, d MNV L ( θ, θ 0 ) is uniformly Lipschitz in A with Lipschitz constant K , that is

Next, we show that the P θ satisfies Assumption 2.1, for distance function L MNV n ( θ, θ 0 ) .

Proof. Proof of Lemma B.6:

First consider the following test function, constructed using ˜ X n = { ξ 1 , ξ 2 , . . . , ξ n } .

ϕ n,ϵ := ✶ { ˜ X n : ∥ ˆ θ n -θ 0 ∥ > √ Cϵ 2 } , where ˆ θ n = ∑ n i =1 ξ i n . Note that ˆ θ n -θ 0 ∼ N ( ·| 0 , 1 n Σ) , where 1 n Σ is a symmetric positive definite matrix. Therefore it can be decomposed as Σ = Q T Λ Q , where Q is an orthogonal matrix and Λ is a daigonal matrix consisting of respective eigen values and consequently ˆ θ n -θ 0 ∼ Q N ( ·| 0 , 1 n Λ) . So, we have ∥ ˆ θ n -θ 0 ∥ 2 ∼ ∥N ( ·| 0 , 1 n Λ) ∥ 2 . Notice that ∥N ( ·| 0 , 1 n Λ) ∥ 2 is a linear combination of d χ 2 (1) random variable weighted by elements of the diagonal matrix 1 n Λ . Using this observation, we first verify that ϕ n,ϵ satisfies condition ( i ) of the Lemma. Observe that

Note that χ 2 (1) is Γ distributed with shape 1 / 2 and scale 2 , which implies χ 2 (1) -1 is a sub-gamma random variable with scale factor 2 and variance factor 2 . Now observe that for ˆ Λ = max i ∈{ 1 , 2 ,...d } Λ ii ,

Now, we fix the alternate set to be { θ ∈ R d : ∥ θ -θ 0 ∥ ≥ 2 √ Cϵ 2 } . Next, we verify that ϕ n,ϵ satisfies condition ( ii ) of the lemma. First, observe that where in the third inequality we used the well known tail bound for sub-gamma random variable (Lemma 3.12 [5]) assuming that C is sufficiently large such that ( 1 d ˆ Λ Cnϵ 2 -1 ) > 1 and in the last inequality follows from the assumption that nϵ 2 > nϵ 2 n ≥ 1 .

where in the last inequality, we used the fact that ∥ θ -θ 0 ∥ ≤ ∥ ˆ θ n -θ ∥ + ∥ ∥ ∥ ˆ θ n -θ 0 ∥ ∥ ∥ . Now on alternate set { θ ∈ R d : ∥ θ -θ 0 ∥ ≥ 2 √ Cϵ 2 } ,

Now, it follows from (82) and Θ ⊂ R d that

Using Lemma C.13, { θ ∈ Θ : n -1 / 2 √ L MNV n ( θ, θ 0 ) ≥ 2 K √ Cϵ 2 } = { θ ∈ Θ : d MNV L ( θ, θ 0 ) ≥ 2 K √ Cϵ 2 } ⊆ { θ ∈ Θ : ∥ θ -θ 0 ∥ ≥ 2 √ Cϵ 2 } , which implies that

Therefore, P θ for θ ∈ Θ , satisifes Assumptions 2.1 for L n ( θ, θ 0 ) = L MNV n ( θ, θ 0 ) for C 0 = 1 , C 1 = 4 K 2 C and C = 1 / 8 ( C d ˆ Λ -1 ) .

Proof. Proof of Lemma B.7:

First, we write the Rényi divergence between two multivariate Gaussian distribution with known Σ as

and D 1+ λ ( N ( ·| θ ) ∥N ( ·| θ 0 )) < ∞ if and only if Σ -1 is positive definite [13]. Since, we assumed that the sequence of models are iid, therefore, D 1+ λ ( P n 0 ∥ P n θ ) = 1 λ log ∫ ( dP n 0 dP n θ ) λ dP n 0 = n 1 λ log ∫ ( dP 0 dP θ ) λ dP 0 = n ( λ +1 2 ( θ -θ 0 ) T Σ( θ -θ 0 ) ) , when Σ -1 is positive definite and D 1+ λ ( P n 0 ∥ P n θ ) = ∞ otherwise. Now observe that

where ˆ Λ = max i ∈{ 1 , 2 ,...d } Λ ii and in the second equality we used eigen value decomposition of Σ = Q T Λ Q . Next, observe that,

where in the last equality we used the fact that the prior distribution is uncorrelated. Now, the result follows immediately for sufficiently large n , if the prior distribution is uncorrelated and uniformly

distributed on the compact set Θ i , for each i ∈ { 1 , 2 , . . . , d } . In particular observe that for large enough n , we have

where m ( A ) is the Lebesgue measure (volume) of any set A ⊂ R . Now if ϵ 2 n = log n n , then for 8 ˆ Λ( λ +1) ( ∏ d i =1 m (Θ i ) ) 2 /d C 3 > 2 , 8 ˆ Λ( λ +1) ( ∏ d i =1 m (Θ i ) ) 2 /d C 3 ϵ 2 n ≥ e -8 ˆ Λ( λ +1) ( ∏ d i =1 m (Θ i ) ) 2 /d C 3 nϵ 2 n for all n ≥ 2 , therefore,

Proof. Proof of Lemma B.9: Since family Q contains all uncorrelated Gaussian distributions restricted to Θ , observe that { q n ( · ) ∈ Q}∀ n ≥ 1 . By definition, q i n ( θ ) ∝ 1 √ 2 πσ 2 i,n e -1 2 σ 2 i,n ( θ -µ i,n ) 2 ✶ Θ i = N ( θ i | µ i,n ,σ i,n ) ✶ Θ i N (Θ i | µ i,n ,σ i,n ) and fix σ i,n = 1 / √ n and θ i = θ i 0 for all i ∈ { 1 , 2 , . . . , d } . Now consider the first term; using the definition of the KL divergence it follows that

Substituting q n ( θ ) in the first term of the equation above and expanding the logarithm term, we obtain

where in the last equality, we used the well known expression for the differential entropy of Gaussian distributions. Recall π ( θ ) = ∏ d i =1 1 m (Θ i ) . Now consider the second term in (87). It is straightforward to observe that,

Substituting (89) and (88) into (87) and dividing either sides by n and substituting σ i,n , we obtain

Now, consider the second term in the assertion of the lemma. Since ξ i , i ∈ { 1 , 2 . . . n } are independent and identically distributed, we obtain

Now using the expression for KL divergence between the two multivariate Gaussian distributions, we have

where ˇ Λ = min i ∈{ 1 , 2 ,...d } Λ ii , and Σ -1 = Q T Λ -1 Q , where Q is an orthogonal matrix and Λ is a daigonal matrix consisting of the respective eigen values of Σ . Combined together (91) and (90) implies that

where C 9 := d 2 +max ( 0 , -∑ d i =1 [log( √ 2 πe ) -log( m (Θ i ))] + d 2 ˇ Λ -1 ) and the result follows.

C.8 Gaussian process classification

Proof of Lemma B.11. In view of Theorem 7.1 in [11], it suffices to show that

for some ¯ C > 0 . Now, first observe that

where the second equality uses the definition of Ψ -1 ( · ) . Since, total-variation distance above is bounded above by supremum norm, there exists a constant 0 < c ′ < 1 / 2 , such that

where the last inequality follows from (13) in Lemma B.10. Then if follows from Theorem 7.1 in [11] that for every ϵ > ϵ n , there exists a test ϕ n (depending on ϵ > 0 ) such that, for every j ≥ 1 ,

Now for all n such that nϵ 2 > nϵ 2 n > 2 log 2 and C 10 = c ′-2 / 4 > 1 and j = 1 , we have

Now observe that

where the second equality uses the fact that Ψ -1 ( · ) = 1 -Ψ 1 ( · ) . Consequently,

Therefore, it follows from (95) and (96) and the definition of L n ( θ, θ 0 ) that

Finally, the result follows for C = 1 / 3 , C 0 = 2 and C 1 = (max( c + , c -)) 2 .

Proof of Lemma B.12. The Rényi divergence

Note that the derivative of the exponent in the integrand above with respect to θ ( y ) is

where in the fourth equality we used definition of the logistic function and the penultimate inequality follows from the well known inequality that 1 -e -x ≤ x . Consequently, using Taylor's theorem it follows that the exponent in the integrand of the Rényi divergence in (100) is bounded above by λ ( λ +1) | θ ( y ) -θ 0 ( y ) | 2 and thus by λ ( λ +1) ∥ θ ( y ) -θ 0 ( y ) ∥ 2 ∞ . Therefore,

Now using the inequality for C 3 = 16( λ +1) above observe that

and the result follows from (15) of Lemma B.10.

Proof of Lemma B.13. Let us first analyze the KL divergence between the prior distribution and variational family. Recall that two Gaussian measures on infinite dimensional spaces are either equivalent or singular. [27, Theorem 6.13] specify the condition required for the two Gaussian measures to be equivalent. In particular, note that θ J 0 ( · ) ∈ Im ( C 1 / 2 ) . Now observe that the covariance operator of Q n has eigenvalues { ζ 2 j } J j =1 2 jd k =1 , therefore operator S in the definition of C q has eigenvalues { 1 -ζ 2 j /µ 2 j } J j =1 2 jd k =1 . For τ 2 j = 2 -2 ja -jd for any a > 0 , ∑ J j =1 2 jd ( nϵ 2 n 2 -2 ja -jd 1+ nϵ 2 n 2 -2 ja -jd ) 2 = ∑ J j =1 2 -jd ( nϵ 2 n 2 -2 ja 1+ nϵ 2 n 2 -2 ja -jd ) 2 < ∞ , therefore S is an HS operator.

For any integer J ≤ J α define ¯ θ J 0 = ∫ θ J 0 ( y ) ν ( dy ) , where θ J 0 ( · ) = ∑ J j =1 ∑ 2 jd k =1 θ 0; j,k ϑ j,k ( · ) . Since, θ J 0 ( · ) ∈ Im ( C 1 / 2 ) and S is a symmetric and HS operator, we invoke Theorem 5 in [22], to write

Now for µ j 2 jd/ 2 = 2 -ja , and using the definition of Besov norm of θ 0 denoted as ∥ θ 0 ∥ 2 β, ∞ , ∞ , and denoting 1 -κ 2 j = 1 1+ nϵ 2 n τ 2 j , we have

where the last inequality follows from the fact that, log(1 + x ) -x 1+ x ≤ x 2 1+ x ≤ x for x > 0 . Substituting τ 2 j = 2 -2 ja -jd , we have

The summation in the first term above is bounded by ϵ 2 n as derived in [30, Theorem 4.5]. Therefore,

Now consider the second term

where in the second inequality, we used the second assertion of Lemma 3.2 [30] for logistic function, the fifth inequality uses the fact that θ ( y ) -θ J 0 ( y ) is orthogonal to θ 0 ( y ) -θ J 0 ( y ) . For any a ≤ α fix J = J α otherwise J = J a , and then it is straightforward to check from the definition of ϵ n given in the assertion of the theorem that (2 dJ -1 /nϵ 2 n ) ≤ (log n ) -2 . The term ∥ θ 0 ( y ) -θ J 0 ( y )) ∥ 2 ∞ is also bounded by C ′ ϵ 2 n as shown in the proof of Theorem 4.5 in [30]. Consequently, the term 1 n E Q n KL ( P n 0 ∥ P n θ ) is bounded above by ϵ 2 n (upto a constant) for sufficiently large n since (log n ) -2 < ϵ 2 n and the result follows.

Proof of Theorem 4.2. The proof is a direct consequence of Theorem 3.2, Lemmas B.11, B.12, B.13, and Proposition 3.2.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study on data-driven decision-making problems within the Bayesian framework. It addresses situations where calculating the posterior distribution becomes intractable due to the data-generating models’ complexity or the datasets’ size. The authors propose a novel Risk-Sensitive Variational Bayesian (RSVB) framework to overcome these challenges. The RSVB framework leverages a dual representation of the entropic risk measure to approximate the risk-sensitive posterior distribution and derive the corresponding decision rule. The paper also investigates the impact of these computational approximations on the predictive performance of the inferred decision rules. Practical examples in parametric and nonparametric settings support the theoretical findings.\n\nSTRENGTHS:\nInnovative Approach: Introducing the Risk-Sensitive Variational Bayesian (RSVB) framework significantly contributes to the field. By replacing the expectation in the Bayes risk with a risk-sensitive entropic risk measure, the authors provide a novel solution for decision-making problems in the Bayesian framework where calculating the posterior distribution is intractable.\n\t\t\nPractical Relevance: The paper addresses a common challenge faced in modern applications with large datasets and complex data-generating models. The proposed RSVB framework offers a promising approach to handling such situations, making it relevant and valuable for researchers and practitioners in various domains.\n\t\t\nTheoretical Analysis: The authors provide a comprehensive theoretical analysis of the RSVB framework, including convergence rates of the approximate posterior and the corresponding optimal value. This analysis enhances the understanding of the proposed approach and its theoretical foundations.\n\t\t\nIllustrative Examples: The inclusion of three examples, both parametric and nonparametric, helps to demonstrate the practical applicability of the RSVB framework. By applying the proposed approach to real-world scenarios, the authors strengthen their theoretical findings and showcase the potential benefits of the RSVB framework in different contexts.\n\nWEAKNESSES:\nSome areas for improvement are as follows:\n\nClarity of Exposition: While the paper touches upon important concepts and theoretical aspects, the overall clarity of the exposition could be improved. Some technical details may require further explanation or examples to enhance reader comprehension.\n\t\t\nComparison with Existing Methods: It would be beneficial to include a more detailed comparison with existing approaches in the field. Specifically, comparing the proposed RSVB framework with other methods for handling intractable posterior distributions would provide a better understanding of its advantages and limitations.\n\t\t\nEmpirical Evaluation: Although the paper presents illustrative examples, a more extensive empirical evaluation could further support the practical effectiveness of the RSVB framework. Comparing its performance against alternative methods on broader datasets and problem domains would add more robustness to the findings.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nYes / NA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn machine learning we often want to infer an optimal course of action in an uncertain scenario from observations we've made. Bayesian decision theory provides a framework to do just that: observations are used to build a posterior, which is then used to build an objective to optimize to decide a course of action. Unfortunately, both building a posterior and optimizing the objective can be challenging. To address these issues, previous methods had relied on variational inference of the posterior: they built an objective to jointly optimize the action and the variational approximation. This paper sets out to generalize previous methods with a temperature parameter ($\\gamma$ in eq (SO)), and theoretically assess the convergence of this procedure, in particular assessing the effect of the expressivity of the variational family. They establish a convergence rate (Thm 3.1 and Prop 3.1) that depends on the contraction rate of the posterior as well as the expressivity of the variational family; they also address the effect of their temperature parameter. Finally they apply their theory to derive convergence rates in 3 examples.\n\nSTRENGTHS:\nThe theory the authors establish is based on a number of standard assumptions that are readily verified for most Bayesian settings, including nonparametric settings. The authors also demonstrate their theory by applying it in 3 examples. Theory, like that of the authors, that accounts for the expressivity of the variational class is also much needed in the variational inference literature.\n\nWEAKNESSES:\nThe assumptions the authors make (2.1-2.4) are generalizations of the standard Bayesian non-parametric convergence rate assumptions from Ghosal et al 2000. It is not immediately clear the significance of their differences, and where each of the assumptions are used in the following theorems. It is a weakness that it requires one to read the proofs in the supplement carefully to understand why each assumption is made. A sketch of the proofs, statements of lemmas, such as establishing which assumptions are necessary for the convergence rate of the posterior, would strengthen the paper.\n\nQUESTIONS:\n$\\eta_n$ seems to simply be a rescaled version of eq (RSVB) with an expectation. Could the authors elaborate on why the expectation must be taken before the infimum; it was not clear why this had to be the case in lemma 6.14.\n\nAssumption 3.1 is also not a standard assumption; I understand that they are able to prove it for the distributions given in the examples, but could they discuss why that is the case, and what the assumption means in simple terms?\n\nLIMITATIONS:\nIn the Lacoste-Julien et al paper, they switched the loss for a \"gain\" so their variational objective became a joint maximization problem rather than a min-max objective, the later of which are in principle more difficult to optimize. This seems to be the major limitation and it is addressed but he authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces risk-sensitivity parameter $\\gamma$ to a decision-making problem where latent model parameter $\\theta$ is considered via variational Bayesian inference. \nThe authors analyze how close the approximate posterior is to the true parameter (Theorem 3.1) and how good the chosen action is (Theorem 3.2). \nThe theory is also verified with experimental results.\n\nSTRENGTHS:\nThis approach extends loss-calibrated variational Bayesian (LCVB) in that the risk sensitivity is controlled. \nThe objective of risk-sensitive variational Bayes (RSVB) has risk expection term and KL divergence between approximate and true posteiors. \nThis combination may distort the approximate posterior, but Theorem 3.1 proves the posterior approaches the true parameter and Theorem 3.2 claims the gap of risk is small (with accurately specified model parameter $\\theta_0$).\n\nWEAKNESSES:\nIt is nice to provide assumptions on variational family like Assumption 3.1, but how to check if it is met seems unclear in general.\n\nQUESTIONS:\n* What is the purpose of Eight-schools experiment in Section 4.3? \nIs it meant to confirm any theoretical aspects? Or is this experiment designed to check how $\\gamma$ affect a complex hierarchical model?\nTo show the actual loss as a function of data points (or epochs) with various $\\gamma$ values may be informative to see the effect of $\\gamma$.\n\n* Figure 2 (b) is unclear visualization with many colors overlaid, as well as gives unclear message. \nIs it obvious the variance of posterior can change with $\\gamma$?\n\n* Is Figure 1 (c) missing? This appears in the text (line 288), but I could not find it.\n\nLIMITATIONS:\nWhile the authors do not put a distinct Limitation section, the scope of method and its theory is given as assumptions. \nI do not find potential societal impact regarding this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies risk-sensitive Bayesian decision-making, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure with respect to the posterior distribution. In particular, this work focuses on problems where the posterior distribution is intractable, and leverages a dual representation of the entropic risk measure to introduce a new framework, namely, RSVB. The goal of the paper is not to simply approximate the posterior distribution, but to also make decisions when that posterior is intractable. Simulations are performed to validate the effectiveness of the proposed framework.\n\nSTRENGTHS:\nThis paper introduces a novel risk-sensitive variational Bayesian (RSVB) framework to jointly compute a risk-sensitive posterior approximation and the corresponding decision rule by solving a minimax optimization problem, where the true posterior distribution is intractable. The objective being considered is a general functional that can be frequently encountered in various decision-making problems.\n\nIn particular, the authors derive the posterior convergence rate for RSVB, as well as the convergence rate for decision rules, and identify the regularity conditions on the prior, likelihood model and the risk function. The authors also study the asymptotic consistency and provides a theoretical understanding of the statistical properties LCVB posterior and the associated decision rule.\n\nDifferent applications are discussed by applying the proposed framework.\n\nThis paper is well-motivated and technically solid, and makes a considerable contribution by providing a general framework with theoretical guarantees for a wide class of problems. The convergence rate could be of potential interest for future research in the community.\n\nWEAKNESSES:\n1. The current discussion of simulations and applications does not include a comparison with other risk-based methods or methods that solve non-risk-based objectives. Further discussion will be beneficial.\n\n2. As a benchmark comparison, it is desirable to see the comparison with the exact decision rule in the case where the true posteriors are available.\n\n3. The current method requires solving a stochastic minimax optimization to compute a decision rule, which can be computationally intractable in practice.\n\nQUESTIONS:\nDo you forsee the applicability of the proposed framework for sequential decision-making problems (e.g. bandits and reinforcement learning)? And whether the convergence rate analysis / decision rules derived using the proposed algorithm could benefit the further understanding of Thompson sampling based algorithms in the case where posterior distributions are intractable?\n\nLIMITATIONS:\nThis is a theory-based work. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}