SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We report the  $95\%$  confidence interval of the robust performance ( $\delta$ -percentile of expected returns) of the VaR framework on the test dataset with that of other baselines for different values of  $\delta$ .
Methods	Riverswim	Inventory	Population-Small	Population
VaR	68.54 ± 5.08	457.95 ± 0.74	-3102.48 ± 429.7	-4576.87 ± 147.3
VaRN	67.27 ± 0.0	452.78 ± 0.02	-4005.53 ± 8.76	-4570.17 ± 38.84
BCR l1	67.27 ± 0.0	369.67 ± 0.0	-5614.95 ± 80.28	-6013.21 ± 1177.94
BCR l∞	67.27 ± 0.0	199.41 ± 39.02	-7908.92 ± 41.6	-9033.7 ± 84.28
WBCR l1	67.9 ± 3.82	454.1 ± 4.16	-5290.38 ± 1084.26	-5408.01 ± 225.2
WBCR l∞	67.27 ± 0.0	199.4 ± 39.02	-7712.43 ± 55.96	-8377.64 ± 126.24
Soft-Robust	61.79 ± 1.46	460.6 ± 0.0	-3647.18 ± 94.62	-6932.86 ± 154.16
Naive Hoeffding	51.52 ± 6.06	-0.0 ± 0.0	-8647.7 ± 59.5	-9127.14 ± 140.98
Opt Hoeffding	50.76 ± 4.56	-0.0 ± 0.0	-8640.48 ± 2.34	-9163.64 ± 13.62
Table 1: shows the  $95\%$  confidence interval of the robust (percentile) returns achieved by VaR, VaRN, BCR  $\ell_1$ , BCR  $\ell_{\infty}$ , WBCR  $\ell_1$ , WBCR, Soft Robust, Worst RMDP, Naive Hoeffding and Opt Hoeffding agents at  $\delta = 0.05$  in Riverswim, Inventory, Population-Small, and Population domain.
Experimental Results Table 1 summarizes the performance of the VaR framework and the baselines for confidence level  $\delta = 0.05$  (Table 2 and Table 3 in the appendix summarizes the results for  $\delta = 0.15$  and  $\delta = 0.3$  respectively.). We observe that for confidence level  $\delta = 0.05$ , the VaR framework outperforms the baseline methods in terms of mean robust performance in most domains. On the other hand, for  $\delta = 0.15$ , the VaR framework outperforms baselines in the Population and Population-Small domains and has comparable performance to the Soft-Robust method in the Inventory domain. However, at  $\delta = 0.3$  we observe that the VaR framework has lower robust performance relative to the Soft-Robust method in Riverswim and Population domains. We conjecture that this is because the Soft-Robust method optimizes the policy to maximize the mean of expected returns and is therefore able to perform better in cases where a lower level of robustness is required. However, we note that in contrast to our method, the Soft-Robust method does not provide probabilistic guarantees against worst-case scenarios.
Furthermore, as expected, we find that in many cases, the robust performance of BCR Robust MDPs with span-optimized (weighted) ambiguity sets  $(WBCR\ell_{1}$ ,  $WBCR\ell_{\infty})$  is relatively higher than the robust performance of Robust MDPs with unweighted BCR ambiguity sets  $(BCR\ell_{1}$ ,  $BCR\ell_{\infty})$ . However, we find that even Robust MDPs with span-optimized BCR ambiguity sets are generally unable to outperform the robust performance of our  $\mathrm{VaR}_{\alpha}$  framework.
Figure 2 compares the robust performance of the VaR framework and the baselines on both train and test models. The trends in the robust performance of the VaR framework and the baselines are similar on both train and test models.
The main limitation of the VaR framework is that it does not consider the correlations in the uncertainty of transition probabilities across states and actions [19, 29, 30]. However, due to the non-convex nature of the percentile-criterion [3, 40], constructing a tractable  $\mathrm{VaR}_{\alpha}$  Bellman operator that considers these correlations is not feasible. One plausible solution is to use a Conditional Value at Risk Bellman operator [11, 27] which is convex and lower bounds the Value at Risk measure. We leave the analysis of this approach for future work. Empirical analysis of the  $\mathrm{VaR}_{\alpha}$  framework in domains with continuous state-action spaces is also an interesting avenue for future work.
In conclusion, we propose a novel dynamic programming algorithm that optimizes a tight lower-bound approximation on the percentile criterion without explicitly constructing ambiguity sets. We theoretically show that our algorithm implicitly constructs tight ambiguity sets whose asymptotic radius is smaller than that of any Bayesian credible region, and therefore, computes less conservative policies with the same confidence guarantees on returns. We also derive finite-sample and asymptotic bounds on the performance loss due to our approximation. Finally, our experimental results demonstrate the efficacy of our method in several domains.
The work in the paper was supported, in part, by NSF grants 2144601 and 1815275.
[1] Praveen Agarwal, Mohamed Jleli, and Bessem Samet. Banach contraction principle and applications, pages 1-23. Springer Singapore, Singapore, 2018.
[2] Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pages 511-520, 2021.
[3] Bahram Behzadian, Reazul Hasan Russel, Marek Petrik, and Chin Pang Ho. Optimizing percentile criterion using robust MDPs. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1009-1017, 2021.
[4] Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for  $L_{\infty}$ -constrained s-rectangular robust MDPs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[5] Aharon Ben-Tal, Dimitris Bertsimas, and David B. Brown. A soft robust model for optimization under ambiguity. Operations Research, 58(4):1220-1234, 2010.
[6] Dimitri P. Bertsekas. Abstract dynamic programming. Athena Scientific, 2013. ISBN 978-1-886529-42-7.
[7] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford University Press, 2013.
[8] G. P. Box and G. C. Tiao. Bayesian inference in statistical analysis. International Statistical Review, 43:242, 1973.
[9] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[10] Peter Buchholz and Dimitri Scheftelowitsch. Computation of weighted sums of rewards for concurrent MDPs. Mathematical Methods of Operations Research, 89, 2019.
[11] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in MDPs. In International Conference on Neural Information Processing Systems, pages 3509-3517. MIT Press, 2014.
[12] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in mdps. Neural Information Processing Systems NIPS'14, page 3509-3517, Cambridge, MA, USA, 2014. MIT Press.
[13] E. Delage and S. Mannor. Percentile optimization for Markov decision processes with parameter uncertainty. Operations Research, 58(1):203-213, 2010.
[14] Esther Derman, Daniel Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. In Annual Conference on Uncertainty in Artificial Intelligence, 2018.
[15] Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. A Bayesian approach to robust reinforcement learning. In Annual Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
[16] Javier García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015.
[17] Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2014.
[18] Charles Geyer and Glen Meeden. Asymptotics for constrained dirichlet distributions. Bayesian Analysis, 8:89-110, 03 2013.
Definition A.1 (Translation Subvariance). The operator  $\mathfrak{T}:\mathbb{R}^S\to \mathbb{R}^S$  satisfies the translation subvariance property if for all vector  $\pmb {v}\in \mathbb{R}^{S}$  and scalar  $c$ , there exists  $\gamma \in [0,1)$  that satisfies
$$
\mathfrak {T} (\boldsymbol {v} + c \mathbf {1}) = (\mathfrak {T} \boldsymbol {v}) + \gamma c \mathbf {1}.
$$
Definition A.2 (Monotonicity). The operator  $\mathfrak{T}:\mathbb{R}^S\to \mathbb{R}^S$  satisfies the monotonicity property if for all  $\pmb {v}\in \mathbb{R}^{S}$  and  $\pmb {u}\in \mathbb{R}^{S}$  such that  $\pmb {v}\preceq \pmb{u}$ ,  $\mathfrak{T}$  satisfies
$$
\mathfrak {T} \boldsymbol {v} \preceq \mathfrak {T} \boldsymbol {u}.
$$
Lemma A.3 (Contraction Mapping [6]). The operator  $\mathfrak{T}:\mathbb{R}^S\to \mathbb{R}^S$  is a contraction mapping if it satisfies monotonicity and translation subvariance properties, i.e., there exists  $\gamma \in [0,1)$  such that for all  $\pmb {u}\in \mathbb{R}^S$ $\pmb {v}\in \mathbb{R}^{S}$ ,  $\mathfrak{T}$  satisfies
$$
\left\| \mathfrak {T} \boldsymbol {u} - \mathfrak {T} \boldsymbol {v} \right\| _ {\infty} \leq \gamma \| \boldsymbol {u} - \boldsymbol {v} \| _ {\infty}.
$$
The proof of Lemma A.3 follows directly from Proposition 2.1.3 in [6]. We re-derive the proof for the sake of completeness.
$$
c = \max  _ {s \in S} \left| \boldsymbol {u} _ {s} - \boldsymbol {v} _ {s} \right|.
$$
Therefore for all  $s \in S$ ,
$$
\boldsymbol {u} _ {s} - c \leq \boldsymbol {v} _ {s} \leq \boldsymbol {u} _ {s} + c.
$$
Applying  $\mathfrak{T}$  to these inequalities and using the translation subvariance (Definition A.1) and monotonicity (Definition A.2) properties, we obtain that for all  $s\in S$
$$
\left(\mathfrak {T} \boldsymbol {u}\right) _ {s} - \gamma c \leq \left(\mathfrak {T} \boldsymbol {v}\right) _ {s} \leq \left(\mathfrak {T} \boldsymbol {u}\right) _ {s} + \gamma c.
$$
It follows that for all  $s\in S$
$$
\left| \left(\mathfrak {T} \boldsymbol {v}\right) _ {s} - \left(\mathfrak {T} \boldsymbol {u}\right) _ {s} \right| \leq \gamma c,
$$
and therefore  $\| \mathfrak{T}\pmb {u} - \mathfrak{T}\pmb {v}\|_{\infty}\leq \gamma c$  , proving the stated result.
Lemma A.4. For any policy  $\pi$ , let  $\hat{\pmb{v}}^{\pi}$  and  $\pmb{v}^{\pi}$  be the fixed point of the VaR policy evaluation operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  and Bellman policy evaluation operator  $\mathcal{T}_P^\pi$ . If the VaR Bellman policy evaluation operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  dominates the Bellman policy evaluation operator  $\mathcal{T}_P^\pi$  at  $\hat{\pmb{v}}^{\pi}$ , i.e.,  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}\hat{\pmb{v}}^{\pi} \preceq \mathcal{T}_P^\pi\hat{\pmb{v}}^{\pi}$ , then, the fixed point of the VaR Bellman evaluation operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  dominates the fixed point of the Bellman evaluation operator  $\mathcal{T}_P^\pi$ , i.e.,  $\hat{\pmb{v}}^{\pi} \preceq \pmb{v}^{\pi}$ .
We note that in contrast to the Bellman policy evaluation operator  $\mathcal{T}_P^\pi$ , the VaR Bellman policy evaluation operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  is a function of the random variable  $\tilde{P}$  and is not dependent on the transition probabilities  $\pmb{P}$  assumed in this setting.
Using the assumption  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\boldsymbol{v}}^{\pi}\preceq \mathcal{T}_{P}^{\pi}\hat{\boldsymbol{v}}^{\pi}$ , and from  $\hat{\boldsymbol{v}}^{\pi} = \mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}\hat{\boldsymbol{v}}^{\pi}$  and  $\boldsymbol{v}^{\pi} = \mathcal{T}_{P}^{\pi}\boldsymbol{v}^{\pi}$ , we get by algebraic manipulations:
$$
\hat {\boldsymbol {v}} ^ {\pi} - \boldsymbol {v} ^ {\pi} = \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} ^ {\pi} \hat {\boldsymbol {v}} ^ {\pi} - \mathcal {T} _ {P} ^ {\pi} \boldsymbol {v} ^ {\pi} \preceq \mathcal {T} _ {P} ^ {\pi} \hat {\boldsymbol {v}} ^ {\pi} - \mathcal {T} _ {P} ^ {\pi} \boldsymbol {v} ^ {\pi} \preceq \gamma \boldsymbol {P} _ {\pi} (\hat {\boldsymbol {v}} ^ {\pi} - \boldsymbol {v} ^ {\pi}) .
$$
Here  $P_{\pi}$  is the transition probability function corresponding to policy  $\pi$ . Subtracting  $\gamma P_{\pi}(\hat{\boldsymbol{v}}^{\pi} - \boldsymbol{v}^{\pi})$  from the above inequality gives,
$$
\left(\boldsymbol {I} - \gamma \boldsymbol {P} _ {\pi}\right) \left(\hat {\boldsymbol {v}} ^ {\pi} - \boldsymbol {v} ^ {\pi}\right) \preceq \mathbf {0}.
$$
where  $\pmb{I}$  is the identity matrix.  $(\pmb {I} - \gamma \pmb{P}_{\pi})^{-1}$  is monotone as can be seen from its Neumann series.
$$
\hat {\boldsymbol {v}} ^ {\pi} - \boldsymbol {v} ^ {\pi} \preceq (\boldsymbol {I} - \gamma \boldsymbol {P} _ {\pi}) ^ {- 1} \boldsymbol {0} = \boldsymbol {0}.
$$
Proof. From Lemma A.3, we know that an operator is a contraction mapping if it satisfies monotonicity and subvariance property.
In this proof, we will show that the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  satisfies monotonicity and subvariance property, and therefore, is a contraction mapping.
We will use shorthand  $\boldsymbol{r}_{s,a}$  to denote the reward vector corresponding to state  $s$  and action  $a$ , i.e.,  $\boldsymbol{r}_{s,a} = R(s, a, \cdot)$ .
First, we show that  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  satisfies translation subvariance property. Consider any  $c \in \mathbb{R}$  and state  $s$ . Then,
$$
\begin{array}{l} \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} (\boldsymbol {v} + c \mathbf {1})\right) _ {s} = \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} (\boldsymbol {r} _ {s, a} + \gamma (\boldsymbol {v} + c \mathbf {1})) ] \\ \stackrel {\mathrm {(a)}} {=} \max  _ {a \in A} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \left(\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v} + \gamma c \mathbf {1}\right) \right] \\ \stackrel {\mathrm {(b)}} {=} \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \left(\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}\right) + \gamma c \right] \\ \stackrel {\mathrm {(c)}} {=} \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) \right] + \gamma c \\ \stackrel {\mathrm {(d)}} {=} \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} + \gamma c. \\ \end{array}
$$
(a) follows from simple algebraic manipulations, (b) follows from  $\gamma c\bar{p}_{s,a}^{\top}\mathbf{1} = \gamma c$ , (c) follows from the translational invariance property of  $\mathrm{VaR}_{\alpha}$  measure [41], and (d) follows the definition of the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$ .
Next, we show that  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  satisfies the monotonicity property.
Let  $\mathbf{u}$  and  $\mathbf{v}$  be any two value functions such that  $\mathbf{v} \preceq \mathbf{u}$ . Consider any state  $s \in S$ . Then,
$$
\begin{array}{l} \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} - \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {u}\right) _ {s} \stackrel {{\mathrm {(a)}}} {=} \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \left(\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}\right) \right] - \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \left(\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {u}\right) \right] \\ \stackrel {(b)} {\leq} \max  _ {a \in A} \left(\operatorname {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) ] - \operatorname {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {u}) ]\right) \\ \stackrel {(c)} {\leq} 0 \\ \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} \leq \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {u}\right) _ {s}. \\ \end{array}
$$
$(a)$  follows from the definition of the  $\mathrm{VaR}_{\alpha}$  Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ ,  $(b)$  follows from the fact that  $\forall a' \in \mathcal{A}, -\max_{a \in \mathcal{A}} \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}(\pmb{r}_{s,a} + \gamma \pmb{v})] \leq -\mathrm{VaR}_{\alpha}[\pmb{p}_{s,a'}^{\mathsf{T}}(\pmb{r}_{s,a'} + \gamma \pmb{v})]$ ,  $(c)$  follows from the monotonicity property [41] of the VaR operator and  $\pmb{u} \succeq \pmb{v}$ .
Thus, we prove that  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is a  $\gamma$ -contraction mapping and a monotone operator. Since  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is a contraction operator on a Banach space and from the Banach fixed point theorem [1], it follows that the operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  has a unique solution  $\hat{v}$ , i.e.,  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{v} = \hat{v}$ .
Given a policy  $\pi$ , a state  $s \in S$ , and transition probabilities  $P$ , let
$$
\left(\mathcal {T} _ {\boldsymbol {P}} ^ {\pi} \boldsymbol {v}\right) _ {s} = \sum_ {a \in \mathcal {A}} \pi (s, a) \boldsymbol {p} _ {s, a} ^ {\intercal} \boldsymbol {w} _ {s, a} \text {a n d} \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} ^ {\pi} \boldsymbol {v}\right) _ {s} = \sum_ {a \in \mathcal {A}} \pi (s, a) \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\intercal} \boldsymbol {w} _ {s, a} \right],
$$
represent the Bellman evaluation operator corresponding to transition probabilities  $P$  and the  $\mathrm{VaR}_{\alpha}$  Bellman evaluation operator, respectively.
Proposition 3.2 (Lower Bound Percentile Criterion). For any  $\delta \in (0,1 / 2)$ , if we set the confidence level  $\alpha$  in the operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  to  $\delta /s$ , then for every policy  $\pi \in \Pi :\operatorname*{Pr}_{\tilde{\pmb{P}}}[\hat{\pmb{v}}^{\pi}\preceq \tilde{\pmb{v}}^{\pi}|\mathcal{D}]\geq 1 - \delta$
Proof. Let  $\alpha = \delta / s$ . Recall that for any policy  $\pi$ , state  $s \in S$ , transition probabilities  $\pmb{P}$ , the Bellman evaluation operator  $\mathcal{T}_{\pmb{P}}^{\pi}$  and the VaR Bellman evaluation operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  are defined as
Let  $\hat{\pmb{v}}^{\pi}$  be the fixed point of  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  conditioned on observed transitions  $\mathcal{D}$ , and let  $\tilde{\pmb{v}}^{\pi}$  be a random variable that represents the fixed point of  $\mathcal{T}_{\hat{P}}^{\pi}$  for a given realization of the transition probabilities  $\tilde{P}$ . Then, applying Lemma A.4 to  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  and  $\mathcal{T}_{\hat{P}}^{\pi}$ , we have,  $\hat{\pmb{v}}^{\pi} \preceq \pmb{v}^{\pi}$  implies
$$
\mathcal {T} _ {\mathrm {v a R} _ {\alpha}} ^ {\pi} \hat {v} ^ {\pi} \simeq \mathcal {T} _ {\bar {P}} ^ {\pi} \hat {v} ^ {\pi}.
$$
That is for each state  $s$
$$
\operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi} \right] \leq \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi}. \tag {10}
$$
Using the equation (10), we can bound the probability that the VaR value function lower bounds the true value.
$$
\Pr_ {\tilde {\boldsymbol {P}}} \left[ \hat {\boldsymbol {v}} ^ {\pi} \preceq \tilde {\boldsymbol {v}} ^ {\pi} | \mathcal {D} \right] = \Pr_ {\tilde {\boldsymbol {P}}} \left[ \forall s \in \mathcal {S}: \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi} \right] \leq \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi} \mid \mathcal {D} \right]. \tag {11}
$$
From the definition of VaR, we know that for any state  $s$  and action  $a$ ,
$$
\Pr_ {\tilde {\boldsymbol {P}}} \left[ \mathrm {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \hat {\boldsymbol {v}} ^ {\pi} \right] \leq \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \hat {\boldsymbol {v}} ^ {\pi} \mid \mathcal {D} \right] \geq 1 - \alpha , \tag {12}
$$
Therefore, using union bound and (11) in (12), we can write
$$
\operatorname * {P r} _ {\tilde {\boldsymbol {P}}} \left[ \hat {\boldsymbol {v}} ^ {\pi} \succ \tilde {\boldsymbol {v}} ^ {\pi} | \mathcal {D} \right] \leq \sum_ {s \in \mathcal {S}} \operatorname * {P r} _ {\tilde {\boldsymbol {P}}} \left[ \mathrm {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi} ] > \tilde {\boldsymbol {p}} _ {s, \pi (s)} ^ {\top} \hat {\boldsymbol {v}} ^ {\pi} \middle | \mathcal {D} \right].
$$
$$
\Pr \left[ \hat {\boldsymbol {v}} ^ {\pi} \succ \tilde {\boldsymbol {v}} ^ {\pi} | \mathcal {D} \right] = \sum_ {s \in \mathcal {S}} \frac {\delta}{S} = S \frac {\delta}{S} = \delta .
$$
B.3 Proof of Proposition 3.3
Proposition 3.3. Suppose that  $\tilde{\pmb{p}}_{s,a}$  for any state  $s$  and action  $a$ , is a multivariate sub-Gaussian with mean  $\bar{\pmb{p}}_{s,a}$  and covariance factor  $\pmb{\Sigma}_{s,a}$ , i.e.,  $\mathbb{E}\left[\exp \left(\lambda (\tilde{\pmb{p}}_{s,\pmb{a}} - \bar{\pmb{p}}_{s,\pmb{a}})^{\top}\pmb {w}\right)\right] \leq \exp \left(\lambda^{2}\pmb{w}^{\top}\pmb{\Sigma}_{s,a}\pmb {w} / 2\right), \forall \lambda \in \mathbb{R}, \forall \pmb {w} \in \mathbb{R}^{S}$ . Then, for any state  $s \in S$ ,  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  satisfies
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} \geq \max _ {a \in \mathcal {A}} \left(\bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} - \sqrt {2 \ln (1 / \alpha)} \sqrt {\boldsymbol {w} _ {s , a} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right).
$$
$$
\begin{array}{l} \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} ^ {\pi} \boldsymbol {v}\right) _ {s} = \max  _ {a \in \mathcal {A}} \mathrm {V a R} _ {\alpha} \left[ \bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \right] \\ \stackrel {\mathrm {(a)}} {=} \max  _ {a \in \mathcal {A}} \sup  \left\{t \mid \Pr \left(\tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \geq t\right) \geq 1 - \alpha \right\} \\ \stackrel {(b)} {=} \max  _ {a \in \mathcal {A}} \inf  \left\{t \mid \Pr \left(\left(\tilde {\boldsymbol {p}} _ {s, a} - \bar {\boldsymbol {p}} _ {s, a}\right) ^ {\top} \boldsymbol {w} _ {s, a} > \left(t - \bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a}\right)\right) <   1 - \alpha \right\} \\ \stackrel {\mathrm {(c)}} {=} \max  _ {a \in \mathcal {A}} \inf  \left\{t \mid \Pr \left(\exp ((\tilde {\boldsymbol {p}} _ {s, a} - \bar {\boldsymbol {p}} _ {s, a}) ^ {\top} \boldsymbol {w} _ {s, a}) > \exp (t - \bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a})\right) <   1 - \alpha \right\} \\ \stackrel {\mathrm {(d)}} {\geq} \max  _ {a \in \mathcal {A}} \inf  \left\{t \Bigg | 1 - \exp \left(\frac {- (t - \bar {\boldsymbol {p}} _ {s , a} ^ {\top} \boldsymbol {w} _ {s , a}) ^ {2}}{2 \boldsymbol {w} _ {s , a} ^ {\top} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right) <   1 - \alpha \right\} \\ \stackrel {\mathrm {(e)}} {=} \max  _ {a \in \mathcal {A}} \inf  \left\{t \Big | (t - \bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a}) ^ {2} <   - 2 \ln (\alpha) \boldsymbol {w} _ {s, a} ^ {\top} \boldsymbol {\Sigma} _ {s, a} \boldsymbol {w} _ {s, a} \right\} \\ \stackrel {\mathrm {f})} {=} \max  _ {a \in \mathcal {A}} \inf  _ {(t)} \left\{t \Big | (t - \bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a}) \in \left(- \sqrt {2 \ln (1 / \alpha)} \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}, \sqrt {2 \ln (1 / \alpha)} \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right) \right\} \\ \stackrel {\mathrm {(g)}} {=} \max  _ {a \in \mathcal {A}} \left(\bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} - \sqrt {2 \ln (1 / \alpha)} \sqrt {\boldsymbol {w} _ {s , a} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right) . \\ \end{array}
$$