SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where recall that  $\{\phi_{n,\epsilon}\}$  is the sequence of test function from Assumption 2.1 and set  $B_{n} = \left\{\tilde{X}_{n}:\int_{\Theta}\mathcal{LR}_{n}(\theta ,\theta_{0})d\Pi (\theta)\geq e^{-(1 + C_{3})n\epsilon^{2}}\Pi (A_{n})\right\}$  with set  $A_{n}$  is defined in Assumption 2.3. Set  $B_{n}$  is introduced to separately control (by lower bounding) the denominator in the definition of the posterior distribution in the last term of (4). In addition, the Assumption 2.3 is used to show that  $P_0^n (B_n^C)\leq e^{-\lambda n\epsilon^2}$ . The first term can be controlled by Assumption 2.1 (i). Now, it remains to analyze the last term in (4). Using Assumption 2.3 and 2.5 observe that on set  $B_{n}$ ,  $\int_{\Theta}e^{\gamma R(a',\theta)}\mathcal{LR}_n(\theta ,\theta_0)d\Pi (\theta)\geq W^\gamma e^{-(1 + C_2 + C_3)n\epsilon^2}$ . This observation, with the application of Fubini's theorem, enables us to bifurcate further the last term in (4) on the set  $S_{\gamma}(\theta)\coloneqq \{e^{\gamma R(a',\theta)} > e^{C_4(\gamma)n\epsilon^2}\}$  as
$$
\begin{array}{l} \mathbb {E} _ {P _ {0} ^ {n}} \left[ (1 - \phi_ {n, \epsilon}) \mathbb {1} _ {B _ {n}} \Pi_ {R} (K _ {n} | \tilde {X} _ {n}) \right] \leq W ^ {- \gamma} e ^ {\tilde {C} n \epsilon^ {2}} \left[ \int_ {K _ {n} \cap S _ {\gamma} (\theta)} \mathbb {E} _ {P _ {\theta} ^ {n}} \left[ (1 - \phi_ {n, \epsilon}) \right] d \Pi (\theta) \right. \\ \left. + e ^ {- C _ {4} (\gamma) n \epsilon^ {2}} \int_ {K _ {n} \cap S _ {\gamma} (\theta)} e ^ {\gamma R \left(a ^ {\prime}, \theta\right)} d \Pi (\theta) \right], \tag {5} \\ \end{array}
$$
$\tilde{C} = 1 + C_2 + C_3 + C_4(\gamma)$ . Observe that the last term above can be controlled using Assumption 2.4. The first integral in (5) can be further bounded by  $\int_{K_n\cap \Theta_n(\epsilon)}\mathbb{E}_{P_\theta^n}\left[(1 - \phi_{n,\epsilon})\right]d\Pi (\theta) + \Pi (\Theta_n(\epsilon)^c)$ , where  $\Theta_{n}(\epsilon)$  is defined in Assumption 2.1 and both the terms can be exponentially bounded using Assumption 2.1 (ii) and 2.2 respectively. The last term in (3) can be bounded using a similar set of arguments and techniques.
Remark: We note that while deriving the bound in Lemma C.1, we can interchange the order of expectation and infimum and compute a tighter bound. However, this brings us back to the RSVB optimizer, the very objective of the whole analysis. Taking expectation before infimum enables us to derive a more interpretable and meaningful bound. This is mainly because it is easier to control the expectation of the RSVB objective than its infimum through Assumption 3.1, as presented in Section 3.1.
The detailed proof of Theorem 3.1 is provided in the Appendix C.2. Now recall that  $\epsilon_{n}$  is the convergence rate of the true posterior [11, Theorem 7.3]. Notice that the additional term  $\eta_n^R (\gamma)$  emerges from the posterior approximation and depends on the choice of the variational family  $\mathcal{Q}$ , risk function  $R(\cdot ,\cdot)$ , and the parameter  $\gamma$ . The appearance of  $\eta_n^R (\gamma)$  in the bound also signifies that to minimize the expected gap (under the RSVB posterior) between the true model and any other model (defined using  $n^{-1}L_{n}(\theta ,\theta_{0}))$  the expected RSVB objective has to be maximized. Later in this section, we specify the conditions that ensure  $\eta_n^R (\gamma)\to 0$  as  $n\to \infty$ . Moreover, we also identify mild regularity conditions on  $\mathcal{Q}$  to show that  $\eta_n^R (\gamma)$  is  $O(\epsilon_n^2)$  and show that as  $\gamma$  increases  $\eta_n^R (\gamma)$  decreases. We discuss this result and the bound therein later in the next subsection. Before that, we establish our main result (the bounds on the optimality gap) using the theorem above. We now fix
$$
L _ {n} (\theta , \theta_ {0}) = n \left(\sup  _ {a \in \mathcal {A}} | R (a, \theta) - R (a, \theta_ {0}) |\right) ^ {2}. \tag {6}
$$
Notice that for a given  $\theta$ ,  $n^{-1/2} \sqrt{L_n(\theta, \theta_0)}$  is the uniform distance between the  $R(a, \theta)$  and  $R(a, \theta_0)$ . Intuitively, Theorem 3.1 implies that the expected uniform difference  $\frac{1}{n} L_n(\theta, \theta_0)$  with respect to the RSVB approximate posterior is  $O(M(\gamma)\epsilon_n^2 + M\eta_n^R(\gamma))$ , and if  $M(\gamma)\epsilon_n^2 + M\eta_n^R(\gamma) \to 0$  as  $n \to \infty$  then it converges to zero at that rate.
Also, note that in order to use (6) we must demonstrate that it satisfies Assumption 2.1. This can be achieved by constructing bespoke test functions for a given  $R(a,\theta)$ . We demonstrate this approach by an example in Section B.2. We also provide sufficient conditions for the existence of the test functions in the appendix. These conditions are typically easy to verify when the loss function  $R(\cdot ,\cdot)$  is bounded, for instance. Next, we bound the optimality gap between  $R(\mathsf{a}_{\mathrm{RS}}^*,\theta_0)$  and  $V_{0}^{*}$ .
Theorem 3.2. Fix  $\gamma >0$ . Suppose the set  $\mathcal{A}$  is compact. Then, under Assumptions 2.1, 2.2, 2.3, 2.4, and 2.5, for  $\min(C,C_4(\gamma) + C_5(\gamma)) > C_2 + C_3 + C_4(\gamma) + 2$  and any  $\tau >0$ , the  $P_0^n - probability$  of  $\left\{\tilde{X}_n:R(\mathsf{a}_{\mathtt{RS}}^*,\theta_0) - \inf_{a\in \mathcal{A}}R(a,\theta_0)\leq 2\tau \left[M(\gamma)\epsilon_n^2 +M\eta_n^R (\gamma)\right]^{\frac{1}{2}}\right\}$  is at least  $1 - \tau^{-1}$ , for a positive mapping  $M(\gamma) = 2\left(C_1 + MC_4(\gamma)\right)$ , where  $M = \frac{2C_1}{\min(C,\lambda,1)}$  for sufficiently large  $n$ .
3.1 Properties of  $\eta_n^R (\gamma)$
Evidently, the bounds obtained in both the results that we have proved so far depend on  $\eta_n^R (\gamma)$ . Consequently, we establish some important properties of  $\eta_n^R (\gamma)$  with respect to  $n$  and  $\gamma$ , under additional regularity conditions. In order to characterize  $\eta_n^R (\gamma)$ , we specify conditions on variational family  $\mathcal{Q}$  such that  $\eta_n^R (\gamma) = O(\epsilon_n^{r_2})$ , for some  $\epsilon_n^\prime \geq \frac{1}{\sqrt{n}}$  and  $\epsilon_n^\prime \to 0$ . We impose the following condition on the variational family  $\mathcal{Q}$  that lets us obtain a bound on  $\eta_n^R (\gamma)$  in terms of  $n$  and  $\gamma$ .
Assumption 3.1. There exists a sequence of distributions  $\{q_n(\cdot)\}$  in the variational family  $\mathcal{Q}$  such that for a positive constant  $C_9$ ,
$$
\frac {1}{n} \left[ \mathrm {K L} \left(Q _ {n} (\theta) \| \Pi (\theta)\right) + \mathbb {E} _ {Q _ {n} (\theta)} \left[ \mathrm {K L} \left(d P _ {0} ^ {n} \left(\tilde {X} _ {n}\right) \| d P _ {\theta} ^ {n} \left(\tilde {X} _ {n}\right)\right) \right] \right] \leq C _ {9} \epsilon_ {n} ^ {\prime 2}. \tag {7}
$$
If the observations in  $\tilde{X}_n$  are i.i.d, then observe that  $\frac{1}{n}\mathbb{E}_{Q_n(\theta)}\left[\mathrm{KL}\left(dP_0^n (\tilde{X}_n)\| dP_\theta^n (\tilde{X}_n)\right)\right] = \mathbb{E}_{Q_n(\theta)}\left[\mathrm{KL}\left(dP_0\| dP_\theta\right)\right]$ . Intuitively, this assumption implies that the variational family must contain a sequence of distributions weakly convergent to a Dirac delta distribution concentrated at the true parameter  $\theta_0$  otherwise the second term in the LHS of (7) will be non-zero. Also, note that the above assumption does not imply that the minimizing sequence  $Q_{a',\gamma}^{*}(\theta |\tilde{X}_n)$  (automatically) converges weakly to a Dirac-delta distribution at the true parameter  $\theta_0$ . Furthermore, unlike Theorem 2.3 of [34], our condition on  $\mathcal{Q}$  in Assumption 3.1, to obtain a bound on  $\eta_n^R (\gamma)$ , does not require the support of the distributions in  $\mathcal{Q}$  to shrink to the true parameter  $\theta_0$  at some appropriate rate, as the numbers of samples increases.
The condition that the variational family contains Dirac delta distributions at each point in the parameter space is a mild and reasonable requirement for consistency. Further, Assumption 3.1 requires that  $\mathcal{Q}$  contains sequences of distributions that weakly converge to each Dirac delta distribution at a certain rate. This is easily satisfied if  $\mathcal{Q}$  has no "holes", e.g. if it is the family of Gaussians with all means and variances, then we can always construct sequences converging to any Dirac delta at any rate. A similar assumption has also been made in [31], and is true for most exponential family distributions. For instance, in the newsvendor application, we fix the variational family to a class of shifted-Gamma distributions and choose a sequence of distributions with parameter sequence  $\alpha = n$  and  $\beta = n / \theta_0$ . This implies that the sequence of distributions has mean  $\theta_0$  and variance  $\theta_0^2 /n$ , and, therefore, it converges to a Dirac delta distribution at  $\theta_0$ . Next, we show that
Proposition 3.1. Under Assumption 3.1 and for a constant  $C_8 = -\inf_{Q \in \mathcal{Q}} \inf_{a \in \mathcal{A}} \mathbb{E}_Q[R(a, \theta)]$  and  $C_9 > 0$ ,  $\eta_n^R(\gamma) \leq \gamma n^{-1} C_8 + C_9 \epsilon_n'^2$ .
In Section 4, we present an example where the likelihood is exponentially distributed, the prior is inverse-gamma (non-conjugate), and the variational family is the class of gamma distributions, where we construct a sequence of distributions in the variational family that satisfies Assumption 3.1. We also provide another example where the likelihood is multivariate Gaussian with unknown mean and variational family is uncorrelated Gaussian restricted to a compact subset of  $\mathbb{R}^d$  with a uniform prior on the same compact set satisfy Assumption 3.1.
By definition  $\epsilon_{n}^{2}\rightarrow 0$  and  $\epsilon_n^\prime \to 0$  as  $n\to \infty$  , and therefore it follows from Proposition 3.1 that  $M(\gamma)\epsilon_{n}^{2} + M\eta_{n}^{R}(\gamma)\rightarrow 0$  . However, the bound obtained in the last proposition might be loose with respect to  $\gamma$  , when  $C_8 <   0$  . To see this, we prove the following result.
Proposition 3.2. If the solution to the optimization problem in  $\eta_n^R (\gamma)$  is primal feasible then  $\eta_n^R (\gamma)$  decreases as  $\gamma$  increases.
In the examples below, we use three examples to study the interplay between sample size  $n$  and the risk parameter  $\gamma$ , and their effect on the optimality gap in values. Additionally, we consider a multi-product newsvendor example in the Appendix B.2.
4.1 Single-product Newsvendor Model
We start with a canonical data-driven decision-making problem with a 'well-behaved' risk function  $R(a, \theta)$ , the data-driven newsvendor model. This problem has received extensive study in the literature and remains a cornerstone of inventory management [25, 2, 18]. Recall that the newsvendor
loss function is defined as  $\ell(a, \xi) \coloneqq h(a - \xi)^{+} + b(\xi - a)^{+}$  where  $h$  (underage cost) and  $b$  (average cost) are given positive constants,  $\xi \in [0, \infty)$  the random demand, and  $a$  the inventory or decision variable, typically assumed to take values in a compact decision space  $\mathcal{A}$  with  $\underline{a} \coloneqq \min\{a : a \in \mathcal{A}\}$  and  $\bar{a} \coloneqq \max\{a : a \in \mathcal{A}\}$ , and  $\underline{a} > 0$ . The distribution over the random demand,  $P_{\theta}$  is assumed to be exponential with unknown rate parameter  $\theta \in (0, \infty)$ . The model risk  $R(a, \theta) \coloneqq \mathbb{E}_{P_{\theta}}[\ell(a, \xi)] = ha - \frac{h}{\theta} + (b + h)\frac{e^{-a\theta}}{\theta}$ , which is convex in  $a$ . We assume that  $\tilde{X}_n \coloneqq \{\xi_1, \xi_2 \dots \xi_n\}$  be  $n$  observations of the random demand, assumed to be i.i.d random samples drawn from  $P_0$ .
We fix the model space  $\Theta = [T,\infty)$  for some  $T > 0$  and assume that  $\theta_0$  lies in the interior of  $\Theta$ . We now assume a non-conjugate truncated inverse-gamma (Inv -  $\Gamma$ ) prior distribution restricted to  $\Theta$ , with shape and rate parameter  $\alpha$  and  $\beta$  respectively, that is for a set  $A \subseteq \Theta$ , we define  $\Pi(A) = \operatorname{Inv} - \Gamma_{\Theta}(A; \alpha, \beta) = \operatorname{Inv} - \Gamma(A \cap \Theta; \alpha, \beta) / \operatorname{Inv} - \Gamma(\Theta; \alpha, \beta)$ . We verify Assumptions 2.2, 2.1, 2.3, 2.5 and 2.4 (in that order) in this newsvendor setting and provide the proofs in the Appendix B.1. Next, we bound the optimality gap in values for the single product newsvendor model risk.
Theorem 4.1. Fix  $\gamma >0$ . Suppose that the set  $\mathcal{A}$  is compact. Then, for the newsvendor model with exponentially distributed demand with rate  $\theta \in \Theta = [T,\infty)$ , prior distribution  $\Pi (\cdot) = \text{Inv} - \Gamma_{\Theta}(\cdot ;\alpha ,\beta) = \text{Inv} - \Gamma (A\cap \Theta ;\alpha ,\beta) / \text{Inv} - \Gamma (\Theta ;\alpha ,\beta)$ , and the variational family fixed to shifted (by  $T > 0$ ) gamma distributions, and for any  $\tau >0$ , the  $P_0^n$  probability of the following event  $\left\{\tilde{X}_n:R(\mathsf{a}_{\mathtt{RS}}^*,\theta_0) - \inf_{z\in \mathcal{A}}R(z,\theta_0)\leq 2\tau M'(\gamma)\left(\frac{\log n}{n}\right)^{1 / 2}\right\}$  is at least  $1 - \tau^{-1}$  for sufficiently large  $n$  and for some known mapping  $M^{\prime}:\mathbb{R}^{+}\to \mathbb{R}^{+}$ , where  $R(\cdot ,\theta)$  is the newsvendor model risk.
The proof of the theorem above is a direct consequence of Theorem 3.2 and Proposition 3.2, and Lemmas B.1, B.2, B.3, B.4, B.5 stated in the Appendix. We also extend the analysis above to a multi-product newsvendor problem. The details are provided in Appendix B.2.
Next, we demonstrate the effect of varying the risk-sensitivity parameter  $\gamma$ . We fix  $\theta_0 = 0.1$ ,  $b = 1$ ,  $h = 5$ ,  $\alpha = 1$ , and  $\beta = 4.1$ . We run RSVB algorithm with  $\gamma \in \{0(\text{naive}), 1, 2, 4.5, 5, 6\}$  and repeat the experiment over 100 sample paths. We plot the results in Figure 1. In Figure 1(a) we plot the optimality gap in values that is  $R(\mathsf{a}_{\mathrm{RS}}^*(\gamma), \theta_0) - R(a_0^*, \theta_0)$  for various values of  $\gamma$ . We observe that the gap decreases when  $n$  increases. This observation supports our results in Propositions 3.1 and 3.2 that establishes the properties of  $\eta_n^R(\gamma)$  as  $n$  increases. Lastly, in Figure 1(b), we plot the variance of the RSVB posterior as  $n$  increases for various values of  $\gamma$ ; as anticipated, the variance reduces as  $n$  increases. To observe the effect of  $\gamma$ , first recall that as  $\gamma$  increases the decision maker becomes more risk averse, and so is our algorithmic framework RSVB. Indeed, from the rightmost variance plot in Figure 1, it is evident that for a larger value of  $\gamma (>4)$ , the RSVB posterior is more concentrated on the subset of  $\Theta$ , where risk is more and consequently we observe large optimality gaps in values. Moreover, as  $n$  increases, the effect of larger  $\gamma$  reduces, since as  $n$  increases, the incentive to deviate from the posterior reduces (due to increased KL divergence dominance for larger  $n$  in RSVB). We also observe that the decision rule learned using the conjugate prior (Gamma distribution for exponential models) has a similar performance as the naïve approach. However, the variance of the true conjugate posterior is higher than those computed through the RSVB and VI approach, corresponding to the well known fact that the variational approximations underestimate the true posterior variance ([28, 19]).
4.2 Gaussian process classification
Consider a problem of classifying an input pattern or features  $Y$  lying in measure space  $([0,1]^d,\mathcal{Y},\nu)$  (with sigma algebra  $\mathcal{V}$  and probability measure  $\nu$ ) into one of two classes  $\{-1,1\}$ . Let  $Y \mapsto \xi(Y) \in \{-1,1\}$  denote the class of  $Y$ . For a given  $Y$ , we model the classifier using a Bernoulli distribution  $p(\xi|Y,\theta) = \Psi_{\xi}(\theta(Y))$ , where  $\theta : [0,1]^d \to \mathbb{R}$  is a non-parametric model parameter in a separable Banach space  $(\Theta, \| \cdot \|)$  and measurable functions  $\Psi_1(x) = (1 + e^{-x})^{-1}$  and  $\Psi_{-1}(x) = 1 - \Psi_1(x)$ . Note that  $\Psi_1(\cdot)$  is a logistic function. We denote  $\psi(\cdot)$  as the derivative of  $\Psi_1(\cdot)$ . Assume that the features  $Y$  are generated independently of  $\xi$ . Thus the sequence of independent observations  $\{\tilde{Y}_n,\tilde{X}_n\} = \{(Y_1,\xi_1),(Y_2,\xi_2),\ldots,(Y_n,\xi_n)\}$  are assumed to be generated from the model  $dP_{\theta}(a,y) = P_{\theta}(\xi = a,Y \in dy) = p(\xi = a|Y = y,\theta)\nu(dy)$ . In the above binary classification problem, the objective is to estimate  $\theta(\cdot)$  using the observation vector  $\{\tilde{Y}_n,\tilde{X}_n\}$ .
We posit a Gaussian process (GP) prior  $\Pi (\cdot)$  on  $\theta (\cdot)\in \Theta$  defined as  $W(\cdot) = \sum_{j = 1}^{\bar{J}_{\alpha}}\sum_{k = 1}^{2^{j d}}\mu_{j}Z_{j,k}\vartheta_{j,k}(\cdot)$ , where  $\{\mu_j\}$  is a sequence that decreases with  $j$ ,  $\{Z_{i,j}\}$  are i.i.d. standard
Gaussian random variables and  $\{\vartheta_{j,k}\}$  form a double-indexed orthonormal basis (with respect to measure  $\nu$ ), that is  $\mathbb{E}_{\nu}[\vartheta_{j,k}\vartheta_{l,m}] = \mathbb{1}_{\{j = l,k = m\}}$ .  $\bar{J}_{\alpha}$  is the smallest integer satisfying  $2^{\bar{J}_{\alpha}d} = n^{d / (2\alpha +d)}$  for a given  $\alpha >0$ . This prior construction is motivated from the work in [30]. We also assume that  $\nu (\cdot)$  is known, and we do not place any prior on it. The posterior distribution over  $\theta (\cdot)$  given observations  $\{\tilde{Y}_n,\tilde{X}_n\}$  can be defined as  $d\Pi (\theta |\{\tilde{Y}_n,\tilde{X}_n\}) = \frac{d\Pi(\theta)\prod_{i = 1}^n\Psi_{\xi_i}(\theta(Y_i))\nu(Y_i)}{\int\prod_{i = 1}^n\Psi_{\xi_i}(\theta(Y_i))\nu(Y_i)d\Pi(\theta)} = \frac{d\Pi(\theta)\prod_{i = 1}^n\Psi_{\xi_i}(\theta(Y_i))}{\int\prod_{i = 1}^n\Psi_{\xi_i}(\theta(Y_i))d\Pi(\theta)}$ . Consider the loss function  $\ell (a,\xi) = \{0$  if  $a = \xi ; c_{+},$  if  $a = +1,\xi = -1;\quad c_{-},$  if  $a = -1,\xi = +1\}$ , where  $c_{+}$  and  $c_{-}$  are known positive constants. For a given feature  $Y$ , the model risk is given by
$$
R (a, \theta) = \mathbb {E} _ {P _ {\theta}} [ \ell (a, \xi) ] = \left\{ \begin{array}{l l} c _ {+} \mathbb {E} _ {\nu} [ \Psi_ {- 1} (\theta (Y)) ], & a = + 1, \\ c _ {-} \mathbb {E} _ {\nu} [ \Psi_ {1} (\theta (Y)) ], & a = - 1. \end{array} \right. \tag {8}
$$
We fix the variational family  $\mathcal{Q}_{GP}$  is a class of Gaussian distributions on  $\Theta$ , defined as  $\mathcal{N}(m_q,\mathcal{C}_q)$ ,  $m_{q}$  belongs to  $\Theta$  and  $\mathcal{C}_q$  is the covariance operator defined as  $\mathcal{C}_q = \mathcal{C}^{1 / 2}(I - S)\mathcal{C}^{1 / 2}$ , for any  $S$  which is a symmetric and Hilbert-Schmidt (HS) operator on  $\Theta$  (eigenvalues of HS operator are square summable). Note that  $S$  and  $m_{q}$  span the distributions in  $\mathcal{Q}_{GP}$ . We can show, using the technical lemmas derived in Section B.3 in Appendix, that the optimality gap in values of the binary GP classification problem converges to zero at a minimax optimal rate (upto logarithmic factors).
Theorem 4.2. Fix  $\gamma >0$  and for a given  $J\in \mathbb{N}$ . For the binary GP classification problem with GP prior induced by  $W = \sum_{j = 1}^{J}\sum_{k = 1}^{2^{jd}}\mu_{j}Z_{j,k}\vartheta_{j,k}$ , where  $\mu_{j} = 2^{-jd / 2 - ja}$  for some  $a > 0$ ,  $\{Z_{i,j}\}$  are i.i.d. standard Gaussian random variables and  $\{\vartheta_{j,k}\}$  form a double-indexed orthonormal basis (with respect to measure  $\nu$ ), and  $\| \theta_0\|_{\beta ;\infty ,\infty} < \infty$ , and  $\theta_0^J (y)$  lie in the Cameron-Martin space  $\operatorname {Im}(\mathcal{C}^{1 / 2})$ , the variational family  $\mathcal{Q}_{GP}$ , and for any  $\tau >0$ , the  $P_0^n$  - probability of  $\{\tilde{X}_n: R(\mathsf{a}_{\mathbb{R}S}^*,\theta_0) - \inf_{z\in A}R(z,\theta_0)\leq 2\tau M'(\gamma)\epsilon_n\}$  is at least  $1 - \tau^{-1}$  for sufficiently large  $n$  and for some mapping  $M^{\prime}:\mathbb{R}^{+}\to \mathbb{R}^{+}$ , where  $R(\cdot ,\theta)$  is defined in (8) and
$$
\epsilon_ {n} = \left\{ \begin{array}{l l} n ^ {- \frac {\beta}{(2 \alpha + d)}} \log n \quad \forall \beta \in [ \mathbf {a}, \alpha ]; & n ^ {- \frac {\alpha}{(2 \alpha + d)}} \log n \quad \forall \alpha \in [ \mathbf {a}, \beta ]; \\ n ^ {- \frac {\mathbf {a}}{(2 \mathrm {a} + d)}} (\log n) ^ {\frac {d}{(2 \mathrm {a} + d)}}, & \forall \mathbf {a} \in [ \alpha , \beta ]; \quad n ^ {- \frac {\beta}{(2 \mathrm {a} + d)}} (\log n) ^ {\frac {d}{(2 \mathrm {a} + d)}} \quad \forall \beta \in [ \alpha , \mathbf {a} ]. \end{array} \right. \tag {9}
$$
A similar Gaussian process classification problem was studied empirically in [16].
4.3 Eight-schools model
We consider the eight-schools problem [33, 15], where the objective is to study the effectiveness of the special coaching program for SAT exams, offered in 8 schools. Each school reported the treatment effect  $y_{i}$  and its standard deviation  $\sigma_{i}$ , where  $i \in \{1,2,3,\dots,8\}$ . The observations  $\{y_{i},\sigma_{i}\}_{i = 1}^{8}$  are modeled using the following hierarchical Bayesian model: 1) Prior distributions:  $\mu \sim \mathcal{N}(\cdot |0,5)$ ,  $\tau \sim$  half-Cauchy  $(\cdot |0,5)$ , 2)  $\theta_{i} \sim \mathcal{N}(\cdot |\mu ,\tau^{2})$  and  $y_{i} \sim \mathcal{N}(\cdot |\theta_{i},\sigma_{i}^{2})$  for each  $i \in \{1,2,3,\ldots ,8\}$ , where  $\{\sigma_i\}_{i = 1}^8$  are assumed to be a known sequence of covariates. The posterior distribution is defined as  $\pi_8(\mu ,\tau ,\{\theta_i\}_{i = 1}^8 |\{y_i,\sigma_i\}_{i = 1}^8)\propto \mathcal{N}(\cdot |0,5)$  half-Cauchy  $(\cdot |0,5) \prod_{i = 1}^{8}\mathcal{N}(\theta_i|\mu ,\tau^2)\mathcal{N}(y_i|\theta_i,\sigma_i^2)$ . For a given treatment effect for eight schools  $y \in \mathbb{R}^8$ , the loss function is defined as:
$$
\ell (y, a) = \sum_ {i = 1} ^ {8} l \left(y _ {i}, a _ {i}\right), \text {w h e r e} l \left(y _ {i}, a _ {i}\right) = \left\{ \begin{array}{l} 0. 2 \left| a _ {i} - y _ {i} \right|, y _ {i} \geq a _ {i} \\ (1 - 0. 2) \left| a _ {i} - y _ {i} \right|, y _ {i} <   a _ {i}, \end{array} \right. \tag {10}
$$
where the decision variable  $a = \{a_1, a_2, \ldots, a_8\}$  denotes the effectiveness level. We define  $R(a, \theta) = \log \int p(y|\theta, \{\sigma_i\}_{i=1}^8)\ell(y,a)dy$ ,  $\theta \in \mathbb{R}^8$  considering  $\{\sigma_i\}_{i=1}^8$  is a given sequence of covariates. Now for any  $\gamma > 0$ , RSVB joint-optimization problem is defined as  $\min_{a \in \mathcal{A}} \max_{Q \in \mathcal{Q}}\left\{\mathbb{E}_q[R(a,\theta)] - \frac{1}{\gamma}\mathrm{KL}(q||\pi_8)\right\}$ . We fix the variational family  $\mathcal{Q}$  to be the meanfield variational family, that is  $q_{\Lambda}(\mu,\tau,\theta_{1,i=1}^{8}) = \mathcal{N}_{\mu}(\cdot|\lambda_9)\mathcal{N}_{\tau}(\cdot|\lambda_{10})\prod_{i=1}^{8}\mathcal{N}_{\theta_i}(|\lambda_i)$ , where  $\Lambda = \{\lambda_1,\lambda_2,\dots,\lambda_{10}\}$ ,  $\lambda_i$  is the mean and variance parameter for each Gaussian distribution. Following [15], we measure the performance of the RSVB method using the metric called empirical risk reduction (ERR),  $\mathcal{I} = \mathcal{ER}_{\mathrm{VB}} - \mathcal{ER}_{\mathrm{RSVB}(\gamma)}$ , where,  $\mathcal{ER}_{\mathrm{alg}} = \frac{1}{N_Y}\sum_{j=1}^{N_Y}l(y_j^{\mathrm{test}},a_{\mathrm{alg}})$ , where  $\mathcal{ER}_{\mathrm{alg}}$  denote the empirical risk evaluated at the decision rule  $a^{\mathrm{alg}}$  obtained using method  $\mathrm{alg} \in \{\mathrm{VB},\mathrm{RSVB}(\gamma)\}$ .