SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr \left[ \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right] - \varepsilon \leq \widehat {\operatorname {V a R}} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right] \leq \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right] + \varepsilon \right] \geq 1 - \zeta \tag {23} \\ \stackrel {\mathrm {(a)}} {=} \Pr \left[ \Phi_ {f} ^ {- 1} (\alpha - \varepsilon \eta) \leq \widehat {\operatorname {V a R}} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} ] \leq \Phi_ {f} ^ {- 1} (\alpha + \varepsilon \eta) \right] \geq 1 - \zeta . \\ \end{array}
$$
Equation (a) follows from applying a first order Taylor expansion to  $\Phi_f^{-1}$  around the point  $\alpha$ . We apply the following results to obtain a bound on  $M^{*}$ .
Let  $\hat{F}$  and  $F$  represent the empirical CDF and the true CDF of a random variable  $\tilde{Z}$ . Suppose that the empirical estimate of the CDF  $\hat{F}$  is estimated using  $M^{*}$  samples from the true distribution of  $\tilde{Z}$  and
$0 < \zeta < 1$  represents the desired level of confidence guarantees, Then, from DWK inequality [31], we know that
$$
\Pr \left(\| \hat {F} - F \| _ {\infty} \geq \sqrt {\ln (2 / \zeta) / 2 M ^ {*}}\right) \leq \zeta .
$$
The above equation implies that
$$
\Pr \left(\exists p \in (0, 1): F ^ {- 1} (p) <   \hat {F} ^ {- 1} \left(p - z _ {t}\right) \text {o r} F ^ {- 1} (p) > \hat {F} ^ {- 1} \left(p + u _ {t}\right)\right) \leq \zeta , \tag {24}
$$
where  $z_{t} = u_{t} = \sqrt{\ln(2 / \zeta) / 2M^{*}}$
Thus, applying equation (24) to (23), i.e.,  $z_{t} = \sqrt{\frac{\ln(2 / \zeta)}{2M^{*}}} = \varepsilon \eta$  gives  $M^{*} = \frac{\ln(2 / \zeta)}{2\varepsilon^{2}}\eta^{2}$ .
B.7 Proof of Proposition 3.7
Proposition 3.7 (Value Iteration Error). Define the empirical VaR Bellman optimality operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  for any value  $\pmb{v} \in \mathbb{R}^{S}$  and state  $s \in S$  as  $(\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\pmb{v})_s = \max_{a \in \mathcal{A}} \widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$ . Let  $\hat{\pmb{v}} \in \mathbb{R}^{S}$  and  $\hat{\pmb{u}} \in \mathbb{R}^{S}$  represent the fixed points of  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  and  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  respectively. Suppose that Algorithm 3.1 returns policy  $\pi_k$  and value function  $\pmb{u}_k$  and  $\| \hat{\pmb{u}} - \pmb{u}_0 \|_{\infty} \leq r_{\max} / 1 - \gamma$ . Then,
$$
\| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {k} \| _ {\infty} \leq \varepsilon , a n d \| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {\pi_ {k}} \| _ {\infty} \leq \frac {2 \varepsilon \gamma}{1 - \gamma}.
$$
Proof. We begin by noting that similar to the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ , the empirical VaR Bellman operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  is also a contraction mapping because of the monotonicity of the  $\mathrm{VaR}_{\alpha}$  operator and since the empirical  $\mathrm{VaR}_{\alpha}$  in the algorithm is always computed from a fixed set of transition probabilities sampled from the posterior of  $\tilde{P}$ .
We can bound the error between the value function  $\pmb{u}_k$  returned by Algorithm 3.1 and the fixed point  $(\hat{\pmb{u}})$  of  $\widehat{\mathcal{T}_{\mathrm{VaR}_{\alpha}}}$  as
$$
\begin{array}{l} \left\| \boldsymbol {u} _ {k - 1} - \hat {\boldsymbol {u}} \right\| _ {\infty} = \left\| \boldsymbol {u} _ {k - 1} + \boldsymbol {u} _ {k} - \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \right\| _ {\infty} \\ \stackrel {(a)} {\leq} \| \boldsymbol {u} _ {k - 1} - \boldsymbol {u} _ {k} \| _ {\infty} + \| \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \| _ {\infty} \\ \frac {1}{\gamma} \| \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \| _ {\infty} \stackrel {(b)} {\leq} \frac {\varepsilon (\gamma - 1)}{\gamma} + \| \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \| _ {\infty} \\ \left(\frac {1}{\gamma} - 1\right) \| \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \| _ {\infty} \stackrel {(c)} {\leq} \frac {\varepsilon (1 - \gamma)}{\gamma} \\ \left\| \boldsymbol {u} _ {k} - \hat {\boldsymbol {u}} \right\| _ {\infty} ^ {(d)} \leq \varepsilon . \\ \end{array}
$$
$(a)$  follows from applying the triangle inequality to the  $l_{\infty}$  norm,  $(b)$  follows from the fact that  $\| \pmb{u}_{k - 1} - \pmb{u}_k\|_\infty \leq \varepsilon^{(1 - \gamma)} / \gamma$  and  $\| \pmb{u}_{k - 1} - \hat{\pmb{u}}\|_{\infty}\leq \frac{1}{\gamma}\| \pmb{u}_k - \hat{\pmb{u}}\|_{\infty}$  due to the contraction property of  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  operator, and finally,  $(c)$  and  $(d)$  follow from simply arranging the terms on both sides.
Next, we bound the approximation error  $\| \pmb{u}_k - \pmb{u}_{\pi_k}\|_{\infty}$ . From the above result, we can write
$$
- \varepsilon \mathbf {1} \leq \hat {\boldsymbol {u}} - \boldsymbol {u} _ {k} \leq \varepsilon \mathbf {1}. \tag {25}
$$
We will use the shorthand  $\pi$ ,  $\mathcal{T}$  and  $\mathcal{T}^{\pi_k}$  to represent the policy  $\pi_k$ , empirical VaR Bellman optimality  $\mathcal{T}_{\widehat{\mathrm{VaR}}_\alpha}$ , and VaR Bellman policy evaluation operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_\alpha}^\pi$  respectively. Suppose that  $\delta = \hat{\pmb{u}} - \pmb{u}_\pi$ .
$$
\delta = \hat {\boldsymbol {u}} - \boldsymbol {u} _ {\pi}
$$
$$
\stackrel {\text {三}} {=} \mathcal {T} \hat {\boldsymbol {u}} - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi}
$$
$$
\stackrel {(b)} {\leq} \mathcal {T} (\boldsymbol {u} _ {k} + \varepsilon \mathbf {1}) - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi}
$$
$$
\stackrel {\text {c e}} {=} \mathcal {T} \boldsymbol {u} _ {k} - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi} + \gamma \varepsilon \mathbf {1}
$$
$$
\stackrel {\mathrm {d e}} {=} \mathcal {T} ^ {\pi} \boldsymbol {u} _ {k} - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi} + \gamma \varepsilon \mathbf {1}
$$
$$
\leq \mathcal {T} ^ {\pi} (\hat {\boldsymbol {u}} + \varepsilon \mathbf {1}) - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi} + \gamma \varepsilon \mathbf {1} \tag {26}
$$
$$
\stackrel {(f)} {=} \mathcal {T} ^ {\pi} \hat {\boldsymbol {u}} - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi} + 2 \gamma \varepsilon \mathbf {1}
$$
$$
\| \delta \| _ {\infty} \stackrel {(\mathrm {g})} {=} \| \mathcal {T} ^ {\pi} \hat {\boldsymbol {u}} - \mathcal {T} ^ {\pi} \boldsymbol {u} _ {\pi} + 2 \gamma \varepsilon \mathbf {1} \| _ {\infty}
$$
$$
\stackrel {\mathrm {h})} {=} \gamma \| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {\pi} \| _ {\infty} + 2 \gamma \varepsilon \mathbf {1}
$$
$$
(1 - \gamma) \| \delta \| _ {\infty} \stackrel {(\mathrm {i})} {=} 2 \gamma \varepsilon \mathbf {1}
$$
$$
\| \delta \| _ {\infty} \stackrel {(\mathrm {j})} {\leq} \frac {2 \gamma \varepsilon}{(1 - \gamma)} .
$$
(a) follows from the fact that  $\hat{\pmb{u}}$  is the fixed point of  $\mathcal{T}\hat{\pmb{u}}$  and  $\pmb{u}_{\pi}$  is the fixed point of  $\mathcal{T}^{\pi}$ , (b) follows from (25), (c) follows from the  $\gamma$ -contraction property of  $\mathcal{T}$  operator, (d) follows from the definition of policy  $\pi = \pi_{k}$ , (e) follows from (25), (f) follows from simple algebraic manipulations, (g) follows from taking  $l_{\infty}$ -norm on both sides, (h) follows from applying triangle inequality, and (i) and (j) follow from simple algebraic manipulations.
Thus,  $\| \hat{\pmb{u}} -\pmb{u}_{\pi}\|_{\infty}\leq \frac{2\gamma\varepsilon}{(1 - \gamma)}$
Next we bound the error between the optimal value function  $\hat{\pmb{v}}$  and the fixed point of the empirical VaR Bellman optimality operator  $\hat{\pmb{u}}$ .
$$
\| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \| _ {\infty} \stackrel {(a)} {=} \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} + \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \widehat {\mathcal {T} _ {\mathrm {V a R} _ {\alpha}}} \hat {\boldsymbol {u}} \| _ {\infty}
$$
$$
\stackrel {(b)} {=} \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} + \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \widehat {\mathcal {T} _ {\mathrm {V a R} _ {\alpha}}} \hat {\boldsymbol {u}} \| _ {\infty}
$$
$$
\stackrel {(c)} {\leq} \left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} \right\| _ {\infty} + \left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} \right\| _ {\infty}
$$
$$
\stackrel {(d)} {\leq} \gamma \| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \| _ {\infty} + \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} \| _ {\infty}.
$$
(a) follows by simply adding and subtracting  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\pmb{u}}$  (b) follows from simply arranging the terms, (c) follows from applying triangle inequality, and (d) follows from the contraction property of the Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$
Using simple algebraic manipulations and rearranging the terms in the above equation, we can write
$$
\left\| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \right\| _ {\infty} \leq \frac {\left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} \right\| _ {\infty}}{1 - \gamma}. \tag {27}
$$
Similarly, we can bound  $\| \hat{\pmb{v}} -\hat{\pmb{u}}\|_{\infty}$  in the following manner.
$$
\begin{array}{l} \left\| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \right\| _ {\infty} \stackrel {{\mathrm {(a)}}} {=} \left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} + \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} \right\| _ {\infty} \\ \stackrel {(b)} {=} \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} + \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} \| _ {\infty} \\ \stackrel {(c)} {\leq} \left\| \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \widehat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \widehat {\boldsymbol {u}} \right\| _ {\infty} + \left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \widehat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \widehat {\boldsymbol {v}} \right\| _ {\infty} \\ \stackrel {(d)} {\leq} \gamma \| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \| _ {\infty} + \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty}. \\ \end{array}
$$
(a) follows by simply adding and subtracting  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\hat{\boldsymbol{v}}$ , (b) follows from simply arranging the terms, (c) follows from applying triangle inequality, and (d) follows from the contraction property of the Bellman operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$ .
$$
\left\| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \right\| _ {\infty} \leq \frac {\left\| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} \right\| _ {\infty}}{1 - \gamma}. \tag {28}
$$
Thus  $\| \hat{\boldsymbol{v}} -\hat{\boldsymbol{u}}\|_{\infty}\leq \frac{1}{1 - \gamma}\min \left(\| \mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\boldsymbol{v}} -\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\hat{\boldsymbol{v}}\|_{\infty},\| \mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\boldsymbol{u}} -\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\hat{\boldsymbol{u}}\|_{\infty}\right)$
B.8 Proof of Proposition 3.8
Proposition 3.8 (Time Complexity). Let  $r_{\max} = \max_{s,s' \in S, a \in A} |R(s,a,s')|$ . Then, Algorithm 3.1 terminates in  $k = \lceil \log_{1/\gamma}\left(\frac{r_{\max}}{\varepsilon(1 - \gamma)}\right) \rceil$  iterations with time complexity  $\mathcal{O}\left(S^2 AM\log_{1/\gamma}\left(\frac{r_{\max}}{\varepsilon(1 - \gamma)}\right)\right)$ .
Furthermore, for any failure probability  $\zeta \in (0,1)$ , suppose that the CDF  $(\Phi_f)$  of the random estimate of 1-step returns  $\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}$  is differentiable at the point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}]$ , and set  $\eta = \Phi'(\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}])$  and  $M = \mathcal{O}\left(\frac{\log(S / \zeta)}{\eta^{2}\varepsilon^{2}(1 - \gamma)^{2}}\right)$ . Then with probability at least  $1 - \zeta$ , it holds that  $\| \hat{\pmb{v}} - \pmb{u}_k \|_{\infty} \leq \mathcal{O}(\varepsilon)$ , and Algorithm 3.1 runs in  $\mathcal{O}\left(\frac{S^2A\log_1 / \gamma\left(\frac{r_{max}}{\varepsilon(1 - \gamma)}\right)\log\left(\frac{S}{\zeta}\right)}{\eta^2\varepsilon^2(1 - \gamma)^2}\right)$  time.
Proof. To bound the number of iterations  $(k)$  required to achieve value approximation error  $\| \pmb{u}_k - \hat{\pmb{u}}\|_{\infty} \leq \varepsilon$ , we first bound the value approximation error  $\| \pmb{u}_k - \hat{\pmb{u}}\|_{\infty}$  as follows.
$$
\begin{array}{l} \| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {k} \| _ {\infty} \stackrel {(a)} {\leq} \| \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \boldsymbol {u} _ {k - 1} \| _ {\infty} \\ \stackrel {(b)} {\leq} \gamma \| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {k - 1} \| _ {\infty} \\ \stackrel {(c)} {\leq} \gamma^ {2} \| \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \widehat {\boldsymbol {u}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \boldsymbol {u} _ {k - 2} \| _ {\infty} \\ \stackrel {\mathrm {(d)}} {\leq} \gamma^ {k} \frac {r _ {\operatorname* {m a x}}}{1 - \gamma} . \\ \end{array}
$$
(a) follows from the definition of the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ , (b) follows from the contraction property of the empirical VaR Bellman operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$ , (c) follows from applying the same procedure as in (a) to step (b), and (d) follows from unrolling (c) over  $k - 2$  time steps and using the fact that  $\| \hat{\boldsymbol{u}} - \boldsymbol{u}_0 \|_{\infty} \leq r_{\max} / 1 - \gamma$  for  $\boldsymbol{u}_0 = \mathbf{0}$ .
We find  $k$  such that  $\| \hat{\pmb{u}} -\pmb {u}_k\|_{\infty}\leq \varepsilon$
$$
\frac {\gamma^ {k}}{1 - \gamma} (r _ {\mathrm {m a x}}) = \varepsilon
$$
$$
k = \left\lceil \frac {\ln \left(\frac {r _ {\operatorname* {m a x}}}{\varepsilon (1 - \gamma)}\right)}{\ln (1 / \gamma)} \right\rceil \tag {29}
$$
$$
k = \left\lceil \log_ {\frac {1}{\gamma}} \left(\frac {r _ {\mathrm {m a x}}}{\varepsilon (1 - \gamma)}\right)\right\rceil .
$$
The time complexity follows from the fact that any quantile of an array of real values can be computed in linear time using the Quick Select algorithm [23], computing 1-step return requires  $\mathcal{O}(S)$  operations, and a single iteration of the loop in lines 3-12 of Algorithm 3.1 computes quantile of returns  $SA$  times.
Suppose that  $\Phi_f$  is differentiable at the point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}]$  and let  $\eta = \Phi_f'(\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}])$  represents the density of estimate of returns at point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}]$ .
Let for any state  $s$  and action  $a$ ,  $\hat{\pmb{w}}_{s,a} = \pmb{r}_{s,a} + \gamma \hat{\pmb{v}}$ . From Proposition 3.6, we know that to guarantee  $\operatorname*{Pr}\left[\left|\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\hat{\pmb{w}}_{s,a}] - \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\hat{\pmb{w}}_{s,a}]\right| > \varepsilon \right] \leq \zeta$ ,  $\mathbf{M}$  should satisfy  $\lim_{\varepsilon \to 0} M\varepsilon^{2} = \ln (2 / \zeta) / 2\eta^{2}$ .
Furthermore, in Proposition 3.7, we showed that  $\| \hat{\pmb{v}} -\hat{\pmb{u}} \|_{\infty}\leq \frac{\|\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\pmb{v}} - \mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\hat{\pmb{v}}\|_{\infty}}{1 - \gamma}$ .
$$
\begin{array}{l} \Pr \left[ \| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \| _ {\infty} \leq \varepsilon \right] \overset {(\mathrm {a})} {\geq} \Pr \left[ \frac {\| \mathcal {T} _ {\operatorname {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\operatorname {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty}}{1 - \gamma} \leq \varepsilon \right] \\ \stackrel {\mathrm {(b)}} {=} \Pr \left[ \| \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty} \leq \varepsilon (1 - \gamma) \right]. \\ \stackrel {\mathrm {(c)}} {=} 1 - \sum_ {s \in \mathcal {S}} \Pr \left[ | \max  _ {a \in \mathcal {A}} \widehat {\operatorname {V a R}} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {\hat {w}} _ {s, a} ] - \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} [ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {\hat {w}} _ {s, a} ] | > \varepsilon (1 - \gamma) \right] \\ \stackrel {\mathrm {(d)}} {=} 1 - \frac {\zeta}{S} S \\ = 1 - \zeta . \tag {30} \\ \end{array}
$$
(a) follows from the fact that  $\| \hat{\pmb{v}} -\hat{\pmb{u}} \|_{\infty}\leq \frac{\|\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\pmb{v}} - \mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\hat{\pmb{v}}\|_{\infty}}{1 - \gamma}$ , (b) follows from arranging terms, (c) follows from applying the union bound, and (d) follows from applying Proposition 3.6 and setting confidence to  $\zeta /s$
The final time complexity results follow from the triangle inequality  $\| \hat{\boldsymbol{v}} -\boldsymbol {u}_k\|_{\infty}\leq \| \hat{\boldsymbol{v}} -\hat{\boldsymbol{u}}\|_{\infty} + \| \boldsymbol {u}_k - \hat{\boldsymbol{u}}\|_{\infty}$  and combining results in Equation (30), Proposition 3.6 and the first part of Proposition 3.8.
B.9 Proof of Proposition 4.1
$$
\forall s \in \mathcal {S}: (\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}) _ {s} = \max  _ {a \in \mathcal {A}} \min  _ {\boldsymbol {p} \in \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \boldsymbol {v}}} \boldsymbol {p} ^ {\mathsf {T}} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}).
$$
Proof. The first part of the proposition simply follows from the definition of the VaR operator. For any state  $s$ , action  $a$  and value function  $\pmb{v}$ , we can write
$$
\min  _ {\boldsymbol {p} \in \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \boldsymbol {v}}} \boldsymbol {p} ^ {\top} \boldsymbol {w} _ {s, a} = \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \right]. \tag {31}
$$
The minimum above exists as it minimizes a linear function on a compact set. The direction “ $\geq$ ” in (31) follows immediately from the constraint in the construction of the set  $\mathcal{P}_{s,a}^{\mathrm{VaR},\pmb{v}}$ . The direction “ $\leq$ ” follows by linear program duality of the minimization over  $\pmb{p}$  and from the fact that  $\mathrm{VaR}_{\alpha}\left[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}\right] \geq \min_{s' \in S} \pmb{w}_{s,a,s'}$ .
Recall that the VaR Bellman optimality operator defined for each  $s \in S$  and  $\pmb{v} \in \mathbb{R}^S$  as
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} = \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathrm {T}} \boldsymbol {w} _ {s, a} \right]. \tag {32}
$$
Suppose that  $\hat{\pmb{v}}$  is the unique fixed point of  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  and  $\hat{\pmb{w}}_{s,a} = \pmb{r}_{s,a} + \gamma \cdot \hat{\pmb{v}}$  is the corresponding transition value for each  $s \in S$  and  $a \in \mathcal{A}$ . That is
$$
\hat {\boldsymbol {v}} = \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}}.
$$
Then, we define the following robust Bellman operator  $\tilde{\mathcal{T}}\colon \mathbb{R}^S\to \mathbb{R}^S$  for each  $s\in S$  and  $\pmb {v}\in \mathbb{R}^{S}$  as
$$
\left(\hat {\boldsymbol {\mathcal {T}}} \boldsymbol {v}\right) _ {s} = \max  _ {a \in \mathcal {A}} \min  _ {\boldsymbol {p} \in \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \bar {v}}} \boldsymbol {p} ^ {\top} \boldsymbol {w} _ {s, a}, \quad \text {w h e r e} \tag {33}
$$
$$
\mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \tilde {\boldsymbol {v}}} = \left\{\boldsymbol {p} \in \Delta^ {S} \mid \boldsymbol {p} ^ {\top} \hat {\boldsymbol {w}} _ {s, a} \geq \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \hat {\boldsymbol {w}} _ {s, a} \right] \right\}.
$$
We now show that  $\hat{v}$  is also the fixed point of the robust Bellman operator  $\hat{T}$ .
Using the equality in (31), we now get that
$$
\begin{array}{l} \hat {v} = \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {v} \\ = \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \hat {\boldsymbol {p}} _ {s, a} ^ {\mathrm {T}} \hat {\boldsymbol {w}} _ {s, a} \right] \\ = \max  _ {a \in \mathcal {A}} \min  _ {\boldsymbol {p} \in \mathcal {P} _ {s, a} ^ {\mathrm {V A R}, \hat {\boldsymbol {v}}}} \boldsymbol {p} ^ {\top} \hat {\boldsymbol {w}} _ {s, a} \\ = \hat {\mathcal {T}} \hat {\boldsymbol {v}}. \\ \end{array}
$$
Therefore,  $\hat{v}$  is the unique fixed point of the SA-rectangular robust Bellman operator [24] and  $\hat{\pi}$  is greedy with respect to the optimal robust value function of  $\hat{v}$  and, therefore, is an optimal policy that solves
$$
\max  _ {\pi \in \Pi^ {D}} \min  _ {\boldsymbol {P} \in \mathcal {P} ^ {\mathrm {V a R}, \hat {\nu}}} \rho (\pi , \boldsymbol {P}) ， \tag {34}
$$
where  $\hat{\pmb{v}}\in \mathbb{R}^{S}$  is the fixed point of the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_\alpha}$ , i.e.,  $\hat{\pmb{v}} = (\mathcal{T}_{\mathrm{VaR}_\alpha}\hat{\pmb{v}})$ .
B.10 Proof of Theorem 4.2
Theorem 4.2 (Asymptotic Radii of VaR Ambiguity Sets). For any state  $s$  and action  $a$ , let  $I(\{(p_{s,a}^*)_i\}_{i=1}^{S-1})$  be the Fisher information of the first  $S-1$  transition probabilities, i.e.,  $\{(p_{s,a}^*)_i\}_{i=1}^{S-1}$ . Define  $I'(p_{s,a}^*) = \begin{bmatrix} I(\{(p_{s,a}^*)_i\}_{i=1}^{S-1}) & 0 \\ 0^\top & 0 \end{bmatrix}$ . Suppose that the normality assumptions on the posterior distribution  $\tilde{\mathbf{P}}$  in Section 2 are satisfied. Then,
For any state  $s$  and action  $a$ , we define the 1-step Bellman update function  $f_{s,a}:\mathbb{R}^S\to \mathbb{R}$  such that
$$
f _ {s, a} (\boldsymbol {v}) = \mathrm {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) \right],
$$