SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Equality  $(a)$  follows from the definition of VaR,  $(b)$  follows from using the fact that  $\mathrm{VaR}_{\alpha}[\tilde{X}] = \inf \{t \in \mathbb{R} : \operatorname*{Pr}[\tilde{X} > t] < 1 - \alpha\}$  and subtracting  $\bar{\pmb{p}}_{s,a}^{\top} \pmb{w}_{s,a}$  on both sides,  $(c)$  follows from taking exponential on both sides,  $(d)$  follows from applying the upper-bound given by Chernoff bound for a sub-Gaussian distribution [7] i.e.,  $\operatorname*{Pr}\left(\exp \left(\left(\tilde{\pmb{p}}_{s,a} - \bar{\pmb{p}}_{s,a}\right)^{\top} \pmb{w}_{s,a}\right) \leq \exp \left(t - \bar{\pmb{p}}_{s,a}^{\top} \pmb{w}_{s,a}\right)\right) \leq \exp \left(\frac{-\left(t - \bar{\pmb{p}}_{s,a}^{\top} \pmb{w}_{s,a}\right)^2}{2 \pmb{w}_{s,a}^{\top} \pmb{\Sigma}_{s,a} \pmb{w}_{s,a}}\right)$ ,  $(e)$  follows from subtracting 1 on both sides and then, taking in on both sides,  $(f)$  follows from simple algebraic manipulations and  $(g)$  follows from taking the infimum of the solution interval of  $t$ .
Solving for  $t$  in step  $(g)$ , we get  $t = \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}] = \bar{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a} - \sqrt{2\ln(1 / \alpha)}\sqrt{\pmb{w}_{s,a}\pmb{\Sigma}_{s,a}\pmb{w}_{s,a}}$  which proves the stated result.
Next, we derive the VaR Bellman optimality update  $\mathcal{T}_{\mathrm{VaR}_\alpha}(\pmb{v})$  when  $\tilde{p}_{s,a}\forall s\in S,a\in \mathcal{A}$  is normally distributed.
Consider the VaR Bellman optimality operator defined for any state  $s$  and value function  $v$  as
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} = \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \right]. \tag {13}
$$
From the theory of multivariate Gaussian distributions [7], we know that, for any state  $s$  and action  $a$ , if  $\bar{\pmb{p}}_{s,a}$  is Gaussian distributed  $\mathcal{N}(\bar{\pmb{p}}_{s,a},\pmb{\Sigma}_{s,a})$ , then,  $\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}$  is also Gaussian distributed  $\mathcal{N}(\bar{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a},\pmb{w}_{s,a}^{\top}\pmb{\Sigma}_{s,a}\pmb{w}_{s,a})$ . To find the  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  for any state  $s$  and action  $a$ , it is sufficient
to find  $t$  such that  $\operatorname{Pr}(\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a} > t) = 1 - \alpha$ .
$$
\Pr \left(\frac {(\tilde {\boldsymbol {p}} - \bar {\boldsymbol {p}} _ {s , a}) ^ {\mathsf {T}} \boldsymbol {w} _ {s , a}}{\sqrt {\boldsymbol {w} _ {s , a} ^ {\mathsf {T}} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}} > \frac {t - \bar {\boldsymbol {p}} _ {s , a} ^ {\mathsf {T}} \boldsymbol {w} _ {s , a}}{\sqrt {\boldsymbol {w} _ {s , a} ^ {\mathsf {T}} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}}\right) = 1 - \alpha
$$
$$
1 - \Phi \left(\frac {t - \bar {\boldsymbol {p}} _ {s , a} ^ {\top} \boldsymbol {w} _ {s , a}}{\sqrt {\boldsymbol {w} _ {s , a} ^ {\top} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}}\right) = 1 - \alpha
$$
$$
\left(\frac {t - \bar {\boldsymbol {p}} _ {s , a} ^ {\top} \boldsymbol {w} _ {s , a}}{\sqrt {\boldsymbol {w} _ {s , a} ^ {\top} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}}\right) = \Phi^ {- 1} (\alpha)
$$
$$
t = \bar {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} + \Phi^ {- 1} (\alpha) \sqrt {\boldsymbol {w} _ {s , a} ^ {\mathsf {T}} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}
$$
$$
t = \bar {\pmb {p}} _ {s, a} ^ {\mathsf {T}} \pmb {w} _ {s, a} - \Phi^ {- 1} (1 - \alpha) \sqrt {\pmb {w} _ {s , a} ^ {\mathsf {T}} \pmb {\Sigma} _ {s , a} \pmb {w} _ {s , a}},
$$
The first equation follows from subtracting  $\vec{p}_{s,a}^{\mathsf{T}}\boldsymbol{w}_{s,a}$  and dividing by  $\sqrt{\boldsymbol{w}_{s,a}^{\mathsf{T}}\boldsymbol{\Sigma}_{s,a}\boldsymbol{w}_{s,a}}$  on both sides. The second equality follows from the definition of CDF of  $\mathcal{N}(0,1)$  and the third equality follows from simple algebraic manipulations.
Substituting the value of  $t = \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}] = \bar{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a} - \Phi^{-1}(1 - \alpha)\sqrt{\pmb{w}_{s,a}^{\mathsf{T}}\pmb{\Sigma}_{s,a}\pmb{w}_{s,a}}$  in (13), we obtain the stated results.
The normal form of the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is useful to analyze the asymptotic properties of the VaR Bellman operator.
B.4 Proof of Theorem 3.4
Theorem 3.4 (Performance). Let  $\hat{\pmb{v}}$  be the fixed point of the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  and  $\pi^{*}$  be the optimal policy in (1). Let  $\rho^{*} = \mathrm{VaR}_{\delta}[\rho (\pi^{*},\tilde{\pmb{P}})]$  denote the optimal percentile returns and  $\hat{\rho} = \pmb {p}_0^\top \hat{\pmb{v}}$  denote the lower bound on the percentile returns computed using the Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  with  $\alpha = \delta /s$ . Then for each  $\delta \in (0,1 / 2)$ :
$$
\rho^ {*} - \hat {\rho} \leq \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \left(\operatorname {V a R} _ {1 - \frac {1 - \delta}{\mathcal {S}}} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right] - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right]\right). \tag {6}
$$
Proof. We denote the optimal policy that optimizes the  $\delta$ -percentile criterion by  $\pi^{*}$ , i.e.,  $\pi^{*} \in \arg \max_{\pi \in \Pi} \mathrm{VaR}_{\delta}[\rho(\pi, \tilde{\pmb{P}})]$ .
Recall that  $\hat{\pmb{v}}\in \mathbb{R}^{S}$  is the fixed point of the  $\mathrm{VaR}_{\alpha}$  Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  , i.e.,  $\hat{\pmb{v}} =$ $\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\pmb{v}}$  and  $\hat{\rho} = p_0^\top \hat{\pmb{v}}$  is the returns of the corresponding policy. The operator  $\mathcal{T}_P^\pi$  represents the Bellman evaluation operator for a given policy  $\pi \in \Pi$  and a transition probability model  $P$  . The Bellman evaluation operator  $\mathcal{T}_P^\pi$  for any  $s\in S$  and value  $\pmb {v}\in \mathbb{R}^S$  is defined as
$$
\left(\mathcal {T} _ {P} ^ {\pi} \boldsymbol {v}\right) _ {s} = \boldsymbol {p} _ {s, \pi (s)} ^ {\intercal} \boldsymbol {w} _ {s, \pi (s)},
$$
where  $\pmb{w}_{s,\pi (s)} = \pmb{r}_{s,\pi (s)} + \gamma \cdot \pmb{v}$
It is known that the Bellman operator  $\mathcal{T}_P^\pi$  is a  $\gamma$ -contraction mapping, monotone, and has a unique fixed point. We will use  $\tilde{\boldsymbol{v}}^{\pi^{*}} = \mathcal{T}_{\tilde{P}}^{\pi^{*}}\tilde{\boldsymbol{v}}^{\pi^{*}}$  to represent the random unique fixed point of the Bellman operator  $\mathcal{T}_{\tilde{P}}^{\pi^{*}}$  defined for a random realization of transition probabilities  $\tilde{P}$ . Furthermore, we will use  $p_0^\top \tilde{v}^{\pi^*} = \rho (\pi^*,\tilde{P})$  to denote the random expected returns corresponding to policy  $\pi^{*}$  and random realization of the transition probabilities  $\tilde{P}$ .
Suppose that  $c \in \mathbb{R}_+$  upper-bounds the difference in performance of the  $\mathrm{VaR}_{\alpha}$  policy  $\hat{\pi}$  and optimal percentile-criterion policy  $\pi^*$ , i.e.,  $\rho^* - \hat{\rho} = c \iff \rho^* = \hat{\rho} - c$ .
From the definition of VaR and  $\pi^{*}$ , we know that
$$
\rho^ {*} = \operatorname {V a R} _ {\delta} [ \rho (\pi^ {*}, \tilde {\boldsymbol {P}}) ] = \sup  \left\{t: \Pr [ \rho (\pi^ {*}, \tilde {\boldsymbol {P}}) \geq t ] \geq 1 - \delta \right\}.
$$
Since  $\hat{\rho} + c$  upper bounds  $\rho^{*}$ , we can write
$$
\Pr [ \rho (\pi^ {*}, \tilde {\boldsymbol {P}}) \leq \hat {\rho} + c ] \geq \delta \Longleftrightarrow \Pr [ \rho (\pi^ {*}, \tilde {\boldsymbol {P}}) - \hat {\rho} \leq c ] \geq \delta . \tag {14}
$$
The above equation suggests that the error in the performance of the  $\mathrm{VaR}_{\alpha}$  policy is upper-bounded by  $c$  if  $\operatorname*{Pr}[\rho (\pi^{*},\hat{\boldsymbol{P}}) - \hat{\rho}\leq c]$  holds with at least  $\delta$  probability.
To derive a lower bound on  $c$ , we proceed as follows.
We begin by showing that  $\rho (\pi^{*},\tilde{\pmb{P}}) - \hat{\rho}\leq \| \tilde{\pmb{v}}^{\pi^{*}} - \hat{\pmb{v}}\|_{\infty}$
$$
\begin{array}{l} \rho \left(\pi^ {*}, \tilde {\boldsymbol {P}}\right) - \hat {\rho} = \rho \left(\pi^ {*}, \tilde {\boldsymbol {P}}\right) - \boldsymbol {p} _ {0} ^ {\top} \hat {\boldsymbol {v}} \quad \text {f r o m t h e d e f i n i t i o n o f} \hat {\rho} \text {a n d} \rho \left(\pi^ {*}, \tilde {\boldsymbol {P}}\right) \\ \leq \left| \boldsymbol {p} _ {0} ^ {\top} \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \boldsymbol {p} _ {0} ^ {\top} \hat {\boldsymbol {v}} \right| \quad | x | \geq x \tag {15} \\ \leq \| \boldsymbol {p} _ {0} \| _ {1} \| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \hat {\boldsymbol {v}} \| _ {\infty} \quad \text {f r o m H o l d e r ' s I n e q u a l i t y} \\ \leq \| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \hat {\boldsymbol {v}} \| _ {\infty} \quad \| \boldsymbol {p} _ {0} \| _ {1} = 1. \\ \end{array}
$$
Next, we bound  $\| \tilde{\pmb{v}}^{\pi^{*}} - \hat{\pmb{v}}\|_{\infty}$  to obtain a lower-bound on  $c$
$$
\begin{array}{l} \| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \hat {\boldsymbol {v}} \| _ {\infty} \stackrel {(a)} {=} \| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} + \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} - \hat {\boldsymbol {v}} \| _ {\infty} \\ \stackrel {(b)} {=} \left\| \mathcal {T} _ {\tilde {P}} ^ {\pi^ {*}} \tilde {v} ^ {\pi^ {*}} - \mathcal {T} _ {\tilde {P}} ^ {\pi^ {*}} \hat {v} + \mathcal {T} _ {\tilde {P}} ^ {\pi^ {*}} \hat {v} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {v} \right\| _ {\infty} \\ \stackrel {(c)} {\leq} \| \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} \| _ {\infty} + \| \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty} \\ \stackrel {(d)} {\leq} \gamma \| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \hat {\boldsymbol {v}} \| _ {\infty} + \| \mathcal {T} _ {\hat {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty}. \\ \end{array}
$$
(a) follows by simply adding and subtracting  $\mathcal{T}_{\tilde{P}}^{\pi^{*}}\hat{v}$ , (b) follows from the fact that  $\tilde{v}^{\pi^{*}}$  and  $\hat{v}$  are the fixed points of  $\mathcal{T}_{\tilde{P}}^{\pi^{*}}$  and  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  respectively, (c) follows from applying the triangle inequality to the R.H.S., and (d) follows from the contraction property of a Bellman operator.
Rearranging and re-normalizing the above terms, we get
$$
\left\| \tilde {\boldsymbol {v}} ^ {\pi^ {*}} - \hat {\boldsymbol {v}} \right\| _ {\infty} \leq \frac {1}{(1 - \gamma)} \left\| \mathcal {T} _ {\vec {P}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} \right\| _ {\infty}. \tag {16}
$$
Then, combining (15) and (16), we get
$$
\begin{array}{l} \rho \left(\pi^ {*}, \tilde {\boldsymbol {P}}\right) - \hat {\rho} \leq \frac {1}{1 - \gamma} \left\| \mathcal {T} _ {\tilde {\boldsymbol {P}}} ^ {\pi^ {*}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} \right\| _ {\infty} \\ \stackrel {\text {(a)}} {=} \frac {1}{1 - \gamma} \max  _ {s \in S} \left(\left(\mathcal {T} _ {\boldsymbol {P}} ^ {\pi^ {*}} \hat {\boldsymbol {v}}\right) _ {s} - \left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}}\right) _ {s}\right) \tag {17} \\ \stackrel {(b)} {=} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \left(\tilde {\boldsymbol {p}} _ {s, \pi^ {*} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \pi^ {*} (s)} - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, \hat {\pi} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \hat {\pi} (s)} \right]\right) \\ \stackrel {\mathrm {(c)}} {\leq} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \left(\tilde {\boldsymbol {p}} _ {s, \pi^ {*} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \pi^ {*} (s)} - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, \pi^ {*} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \pi^ {*} (s)} \right]\right) . \\ \end{array}
$$
(a) follows from the definition of  $l_{\infty}$  norm, (b) follows from the definitions of  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  and  $\mathcal{T}_{\hat{\pmb{P}}}^{\pi^{*}}$  Bellman operators, and (c) follows from the fact that  $\hat{\pi} (s) = \arg \max_{a\in \mathcal{A}}\mathrm{VaR}_{\alpha}\left[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}\right]$  and thus,  $\mathrm{VaR}_{\alpha}\left[\tilde{\pmb{p}}_{s,\hat{\pi}(s)}^{\top}\hat{\pmb{w}}_{s,\hat{\pi}(s)}\right] \geq \mathrm{VaR}_{\alpha}\left[\tilde{\pmb{p}}_{s,\pi^{*}(s)}^{\top}\hat{\pmb{w}}_{s,\pi^{*}(s)}\right]$ .
Therefore, for any  $\varepsilon > 0$ , we can write
$$
\begin{array}{l} \Pr \left[ \rho (\pi^ {*}, \tilde {\boldsymbol {P}}) - \hat {\rho} \leq \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \left(\mathrm {V a R} _ {1 - \frac {1 - \delta - \varepsilon}{S}} \left[ \tilde {\boldsymbol {p}} _ {s, \pi^ {*} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \pi^ {*} (s)} \right] - \mathrm {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, \pi^ {*} (s)} ^ {\mathsf {T}} \hat {\boldsymbol {w}} _ {s, \pi^ {*} (s)} \right]\right) \right] \geq \delta \\ \Pr \left[ \rho \left(\pi^ {*}, \tilde {\boldsymbol {P}}\right) - \hat {\rho} \leq \frac {1}{1 - \gamma} \max  _ {s \in S, a \in A} \left(\operatorname {V a R} _ {1 - \frac {1 - \delta - \varepsilon}{S}} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \hat {\boldsymbol {w}} _ {s, a} \right] - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \hat {\boldsymbol {w}} _ {s, a} \right]\right) \right] \geq \delta . \tag {18} \\ \end{array}
$$
The first equation in the above follows from  $\operatorname*{Pr}\left[\tilde{\pmb{p}}_{s,a}\pmb{w}_{s,a} > \mathrm{VaR}_{1 - \frac{1 - \delta - \varepsilon}{S}}\left[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}\right]\right] < \frac{1 - \delta}{S}$  for any state  $s$  and action  $a$  and applying union bound over all states yields  $\operatorname*{Pr}\left[\tilde{\pmb{p}}_{s,a}\pmb{w}_{s,a} > \mathrm{VaR}_{1 - \frac{1 - \delta - \varepsilon}{S}}\left[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}\right]\forall s\in S\right] < 1 - \delta$ . Thus, we get  $\operatorname*{Pr}\left[\tilde{\pmb{p}}_{s,a}\pmb{w}_{s,a}\leq \mathrm{VaR}_{1 - \frac{1 - \delta - \varepsilon}{S}}\left[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}\right],\forall s\in S\right]\geq \delta$ . The second equation follows by simply replacing  $\pi^{*}(s)$  with the worst-case action, i.e., action that maximizes the upper bound.
Comparing (14) and (18), we get
$$
\rho^ {*} - \hat {\rho} \leq \frac {1}{1 - \gamma} \max _ {s \in \mathcal {S}} \max _ {a \in \mathcal {A}} \left(\inf _ {\varepsilon > 0} \operatorname {V a R} _ {1 - \frac {1 - \delta - \varepsilon}{S}} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \right] - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \right]\right).
$$
Note that VaR of returns is upper semicontinuous and thus, this implies that the infimum in the above equation is achieved at  $\varepsilon = 0$ .
$$
\rho^ {*} - \hat {\rho} \leq \frac {1}{1 - \gamma} \max _ {s \in \mathcal {S}} \max _ {a \in \mathcal {A}} \left(\mathrm {V a R} _ {1 - \frac {1 - \delta}{\delta}} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\intercal} \boldsymbol {w} _ {s, a} \right] - \mathrm {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\intercal} \boldsymbol {w} _ {s, a} \right]\right).
$$
B.5 Proof of Theorem 3.5
Theorem 3.5 (Asymptotic Performance). Suppose that the normality assumptions on the posterior distribution  $\tilde{\pmb{P}}$  in Section 2 are satisfied. For any  $\delta \in (0,1 / 2)$ , set  $\alpha = \delta /s$  in  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ . For any states and action  $a$ , let  $I(\pmb{p}_{s,a}^{*})$  be the Fisher information matrix corresponding to the true transition probabilities  $\pmb{p}_{s,a}^{*}$ . Furthermore, let  $\sigma_{\max}^2 = \max_{s\in S,a\in A}\hat{\pmb{w}}_{s,a}^\top I(\pmb{p}_{s,a}^{*})^{-1}\hat{\pmb{w}}_{s,a}$  be the maximum over state-action pairs of the asymptotic variance of the returns estimate  $\hat{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}$ . Then the asymptotic performance of the VaR framework  $\hat{\rho}$  w.r.t. the optimal percentile returns  $\rho^{*}$  satisfies
$$
\lim  _ {N \rightarrow \infty} \sqrt {N} (\rho^ {*} - \hat {\rho}) \leq \frac {1}{1 - \gamma} \left(2 \Phi^ {- 1} (1 - \delta / s) \sigma_ {\max }\right) \leq \frac {1}{1 - \gamma} \sqrt {8 \ln (S / \delta)} \sigma_ {\max } .
$$
Proof. As noted in Section 2, we assume that as  $N \to \infty$ , the posterior distribution of transition probabilities at any state  $s$  and action  $a(\tilde{p}_{s,a})$  converges in the limit to a multivariate Gaussian distribution with mean  $p_{s,a}^*$  and covariance matrix  $I(p_{s,a}^*)^{-1}/N$ . Hence, we can write
$$
\lim _ {N \to \infty} \sqrt {N} (\tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*} \mathsf {T} \boldsymbol {w} _ {s, a}) \rightsquigarrow \mathcal {N} (0, \boldsymbol {w} _ {s, a} ^ {\mathsf {T}} I (\boldsymbol {p} _ {s, a} ^ {*}) ^ {- 1} \boldsymbol {w} _ {s, a}).
$$
We know from Proposition 3.3, that  $\mathrm{VaR}_{\alpha}$  of a univariate Gaussian random variable  $X \sim \mathcal{N}(\mu, \sigma^2)$  can be written as  $\mathrm{VaR}_{\alpha}[X] = \mu + \Phi^{-1}(\alpha)\sigma$ . Therefore, applying this result to the R.H.S. of Equation (6), we get
$$
\begin{array}{l} \lim  _ {N \rightarrow \infty} \sqrt {N} (\rho^ {*} - \hat {\rho}) \stackrel {(a)} {\leq} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \left(\sqrt {N} \boldsymbol {p} _ {s, a} ^ {*} ^ {\intercal} \boldsymbol {w} _ {s, a} + \Phi^ {- 1} \left(1 - \frac {1 - \delta}{S}\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\intercal} I (\boldsymbol {p} _ {s , a} ^ {*}) ^ {- 1} \boldsymbol {w} _ {s , a}} \right. \\ \left. - \left(\sqrt {N} \boldsymbol {p} _ {s, a} ^ {*} ^ {\text {T}} \boldsymbol {w} _ {s, a} + \Phi^ {- 1} \left(\frac {\delta}{S}\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\text {T}} I \left(\boldsymbol {p} _ {s , a} ^ {*}\right) ^ {- 1} \boldsymbol {w} _ {s , a}}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {=} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \left(\Phi^ {- 1} \left(1 - \frac {1 - \delta}{S}\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} I (\boldsymbol {p} _ {s , a} ^ {*}) ^ {- 1} \boldsymbol {w} _ {s , a}} \right. \\ - \Phi^ {- 1} \left(\frac {\delta}{S}\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} I \left(\boldsymbol {p} _ {s , a} ^ {*}\right) ^ {- 1} \boldsymbol {w} _ {s , a}}) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\mathrm {(c)}} {=} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \left(\left(\Phi^ {- 1} \left(1 - \frac {1 - \delta}{S}\right) - \Phi^ {- 1} \left(\frac {\delta}{S}\right)\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} I (\boldsymbol {p} _ {s , a} ^ {*}) ^ {- 1} \boldsymbol {w} _ {s , a}}\right) \\ \stackrel {(d)} {\leq} \frac {1}{1 - \gamma} \max  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \left(2 \Phi^ {- 1} \left(1 - \frac {\delta}{S}\right) \sqrt {\boldsymbol {w} _ {s , a} ^ {\top} I \left(\boldsymbol {p} _ {s , a} ^ {*}\right) ^ {- 1} \boldsymbol {w} _ {s , a}}\right). \tag {19} \\ \end{array}
$$
$(a)$  follows from the definition of  $\mathrm{VaR}_{\alpha}$  under Gaussian assumptions,  $(b)$  and  $(c)$  follow from simple algebraic manipulations, and  $(d)$  follows from the fact that  $\forall \delta \in (0,1 / 2), 0 \geq \Phi^{-1}\left(1 - \frac{1 - \delta}{S}\right) \geq \Phi^{-1}\left(\frac{\delta}{S}\right), -\Phi^{-1}\left(\frac{\delta}{S}\right) = \Phi^{-1}(1 - \frac{\delta}{S})$ , and assuming  $S > 1$ .
We prove the second inequality in Theorem 3.5, by leveraging the sub-Gaussian bounds for a standard normal distribution  $\mathcal{N}(0,\sigma^2)$  to show that  $\forall \alpha^{\prime}\in (0,1 / 2),\Phi^{-1}(1 - \alpha^{\prime})\leq \sqrt{2\ln(1 / \alpha^{\prime})}.$
We know that for a standard normal distribution  $X \sim \mathcal{N}(0, \sigma^2)$  where  $\sigma \in \mathbb{R}$ , the following sub-Gaussian bounds hold [7]:
$$
\Pr \left(- \sqrt {2 \ln \left(2 / \alpha^ {\prime}\right)} \sigma \leq X \leq \sqrt {2 \ln \left(2 / \alpha^ {\prime}\right)} \sigma\right) \geq 1 - \alpha^ {\prime}. \tag {20}
$$
By definition, for a standard normal distribution  $\mathcal{N}(0,\sigma^2)$ , it holds
$$
\Pr \left(- \Phi^ {- 1} \left(1 - \alpha^ {\prime} / 2\right) \sigma \leq X \leq \Phi^ {- 1} \left(1 - \alpha^ {\prime} / 2\right) \sigma\right) = 1 - \alpha^ {\prime}. \tag {21}
$$
Comparing equation (20) and (21), we get
$$
\Phi^ {- 1} \left(1 - \alpha^ {\prime} / 2\right) \leq \sqrt {2 \ln \left(^ {2} / \alpha^ {\prime}\right)} \Longrightarrow \quad \Phi^ {- 1} \left(1 - \alpha^ {\prime}\right) \leq \sqrt {2 \ln \left(^ {1} / \alpha^ {\prime}\right)}. \tag {22}
$$
Using the result in (22) in equation (19), proves the second inequality of the theorem.
B.6 Proof of Proposition 3.6
Proposition 3.6 (Empirical Error Bound). For any state  $s$ , action  $a$  and value function  $\mathbf{v}$ , let  $\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  represent the empirical estimate of  $\alpha$ -percentile of returns  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  and  $\Phi_f$  represent the cumulative density function (CDF) of the random estimate of returns  $\tilde{p}_{s,a}^{\top}\pmb{w}_{s,a}$ . Suppose that  $\Phi_f$  is differentiable at the point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  and let  $\eta = \Phi_f'(\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}])$  represents the density of estimate of returns at point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$ . Let  $M^{*}$  be the number of posterior samples required to obtain empirical error  $\varepsilon \in \mathbb{R}$ , with confidence  $1 - \zeta$ , where  $0 < \zeta < 1$ , i.e.,  $\operatorname*{Pr}\left[|\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}] - \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]| > \varepsilon \right] \leq \zeta$ . Then,  $\lim_{\varepsilon \to 0} M^{*}\varepsilon^{2} = \ln(2 / \zeta) / 2\eta^{2}$ .
Proof. To prove this theorem, we first compute the derivative of the inverse of the CDF  $\partial \Phi_f^{-1}(\alpha) / \partial \alpha$  as follows. From the definition of the cdf  $\Phi_f$  and VaR, we know that, for any  $\alpha \in (0, 1/2)$ ,  $\Phi_f^{-1}(\alpha) = \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}]$ .
From the inverse-function theorem, we get,
$$
\begin{array}{l} (\Phi_ {f} ^ {- 1} (\alpha)) ^ {\prime} = \frac {1}{\Phi_ {f} ^ {\prime} \left(\operatorname {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s , a} ^ {\mathsf {T}} \boldsymbol {w} _ {s , a} ]\right)} \\ = \frac {1}{\Phi_ {f} ^ {\prime} \left(\Phi_ {f} ^ {- 1} (\alpha)\right)} \\ = \frac {1}{\eta}. \\ \end{array}
$$
Equipped with the above result, we can now proceed to prove the main result.
To prove the result, we need to find  $M^{*}$  such that