SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As noted in (2), we assume that as  $N\to \infty$  , the posterior distribution of transition probabilities for any state  $s$  and action  $a$  ( $\tilde{\pmb{p}}_{s,a}$ ) converges in the limit to a multivariate Gaussian distribution with mean  $\pmb{p}_{s,a}^{*}$  and degenerate covariance matrix  $I(\pmb{p}_{s,a}^{*}) / N$ . Therefore, the 1-step returns  $\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a}$  is asymptotically a univariate Gaussian random variable, i.e.,  $\lim_{N\to \infty}\sqrt{N} (\tilde{\pmb{p}}_{s,a}^{\mathsf{T}}\pmb{w}_{s,a} - \pmb{p}_{s,a}^{*}\mathsf{T}\pmb{w}_{s,a})\sim \mathcal{N}\left(0,\pmb{w}_{s,a}^{\mathsf{T}}I^{-1}(\pmb{p}_{s,a}^{*})\pmb{w}_{s,a}\right)$ . Then, we can write
$$
\lim  _ {N \rightarrow \infty} \sqrt {N} \left(f _ {s, a} (\boldsymbol {v}) - \boldsymbol {p} _ {s, a} ^ {*} ^ {\top} \boldsymbol {w} _ {s, a}\right) = - \Phi^ {- 1} (1 - \alpha) \| \boldsymbol {w} _ {s, a} \| _ {I \left(\boldsymbol {p} _ {s, a} ^ {*}\right)}, \tag {35}
$$
where  $\Phi^{-1}$  represents the CDF inverse of a standard normal distribution. Equation (35) follows from the analytical form of VaR of a Gaussian random variable, i.e., for any Gaussian random variable  $\tilde{Y} \sim \mathcal{N}(\mu, \sigma^2)$  with mean  $\mu$ , variance  $\sigma^2$ , and confidence level  $\alpha$ ,  $\mathrm{VaR}_{\alpha}[\tilde{Y}] = \mu - \Phi^{-1}(1 - \alpha)\sigma$ .
Since  $f_{s,a}$  is convex in  $\pmb{v}$  when  $\tilde{\pmb{p}}_{s,a}$  is Multivariate Gaussian distributed, we can use the definition of support function of a closed convex set [9] to retrieve the unique ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR}}$ .
Note that, in this case the support function is  $f_{s,a}$  and  $\mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}}$  is the unknown closed convex set
$$
\forall \boldsymbol {v} \in \mathbb {R} ^ {S}: \min  _ {\boldsymbol {p} _ {s, a} \in \mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}}} \boldsymbol {p} _ {s, a} ^ {\top} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) = f _ {s, a} (\boldsymbol {v})
$$
$$
\forall \boldsymbol {v}\in \mathbb{R}^{S}:\min_{\boldsymbol{p}_{s,a}\in \mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}}}\sqrt{N}\boldsymbol{p}_{s,a}^{\top}\boldsymbol{w}_{s,a} = \sqrt{N} f_{s,a}(\boldsymbol {v})
$$
$$
\forall \boldsymbol {v}\in \mathbb{R}^{S}:\min_{\boldsymbol{p}_{s,a}\in \mathcal{P}_{s,a}^{\mathrm{V}}}\sqrt{N} (\boldsymbol{p}_{s,a}^{\intercal}\boldsymbol{w}_{s,a} - \boldsymbol{p}_{s,a}^{*}^{\intercal}\boldsymbol{w}_{s,a}) = \sqrt{N} (f_{s,a}(\boldsymbol {v}) - \boldsymbol{p}_{s,a}^{*}^{\intercal}\boldsymbol{w}_{s,a})
$$
$$
\forall \boldsymbol {v} \in \mathbb {R} ^ {S}, \boldsymbol {z} _ {s, a} \in \left(\sqrt {N} \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right)\right): \boldsymbol {z} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \geq - \Phi^ {- 1} (1 - \alpha) \| \boldsymbol {w} _ {s, a} \| _ {I (\boldsymbol {p} _ {s, a} ^ {*})},
$$
where the last statement follows from the positive homogeneity and translation property of support functions.
Rearranging the terms in the above and setting  $\pmb{z}_{s,a} = \pmb{p}_{s,a} - \pmb{p}_{s,a}^{*}$ , we get
$$
\lim  _ {N \rightarrow \infty} \sqrt {N} \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right) = \left\{\boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*} \mid \forall \boldsymbol {v} \in \mathbb {R} ^ {S}, \boldsymbol {p} _ {s, a} \in \Delta^ {S}, \boldsymbol {p} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \geq \boldsymbol {p} _ {s, a} ^ {*} ^ {\top} \boldsymbol {w} _ {s, a} - \Phi^ {- 1} (1 - \alpha) \| \boldsymbol {w} _ {s, a} \| _ {I (\boldsymbol {p} _ {s, a} ^ {*})} \right\}. \tag {36}
$$
Using basic algebraic manipulations as shown in Proposition B.1, we can write the asymptotic form of ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}}$  as an ellipsoid
$$
\lim  _ {N \rightarrow \infty} \sqrt {N} \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right) = \left\{\boldsymbol {p} _ {s, a} \in \Delta^ {S} \left| \| \boldsymbol {p} _ {s, a} ^ {*} - \boldsymbol {p} _ {s, a} \| _ {I ^ {\prime} \left(\boldsymbol {p} _ {s, a} ^ {*}\right) ^ {- 1}} \leq \Phi^ {- 1} (1 - \alpha) \right.\right\} - \boldsymbol {p} _ {s, a} ^ {*}. \tag {37}
$$
Proposition B.1. Consider the two ambiguity sets given in equations (36) and (37). These two representations are equivalent.
Proof. For any state  $s$  and action  $a$ , as  $N \to \infty$ ,  $(\pmb{p}_{s,a} - \pmb{p}_{s,a}^{*}) \in \sqrt{N} (\mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}} - \pmb{p}_{s,a}^{*})$  satisfies
$$
\boldsymbol {p} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} \geq \boldsymbol {p} _ {s, a} ^ {*} \top \boldsymbol {w} _ {s, a} - \Phi^ {- 1} (1 - \alpha) \| \boldsymbol {w} _ {s, a} \| _ {I (\boldsymbol {p} _ {s, a} ^ {*})}
$$
$$
\boldsymbol {p} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*} \top \boldsymbol {w} _ {s, a} \geq - \Phi^ {- 1} (1 - \alpha) \| \boldsymbol {w} _ {s, a} \| _ {I \left(\boldsymbol {p} _ {s, a} ^ {*}\right)} \tag {38}
$$
$$
\boldsymbol {w} _ {s, a} ^ {\top} (\boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}) (\boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}) ^ {\top} \boldsymbol {w} _ {s, a} \leq (\Phi^ {- 1} (1 - \alpha)) ^ {2} \boldsymbol {w} _ {s, a} ^ {\top} I (\boldsymbol {p} _ {s, a} ^ {*}) ^ {- 1} \boldsymbol {w} _ {s, a}.
$$
The first and second equations follow from the definition of  $\mathcal{P}_{s,a}^{\mathrm{VaR}}$  and simple algebraic manipulations. The third equation follows by multiplying by  $-1$  on both sides and squaring both sides.
Recall that  $\tilde{\pmb{p}}_{s,a}$  for any state  $s$  and action  $a$  is a categorical random variable. We assumed that the prior is a Dirichlet distribution and therefore, by the property of conjugate prior, the posterior distribution of  $\tilde{\pmb{p}}_{s,a}$  is also a Dirichlet distribution. We assumed in Section 2 that as  $N\to \infty$ ,  $\tilde{\pmb{p}}_{s,a}$  is Gaussian-distributed with mean  $\pmb{p}_{s,a}^{*}$  and degenerate covariance matrix  $I(\pmb{p}_{s,a}^{*})^{-1} / N$ , i.e., covariance matrix  $I(\pmb{p}_{s,a}^{*})^{-1} / N$  has  $S - 1$  independent rows and  $S - 1$  independent columns.
For any state  $s$  and action  $a$  and transition probabilities  $\tilde{\pmb{p}}_{s,a}$ , let  $\tilde{\pmb{q}}_{s,a} \in \mathbb{R}^{S-1}$  represent the first  $S-1$  elements of the random variable  $\tilde{\pmb{p}}_{s,a}$ , i.e.,  $\tilde{\pmb{q}}_{s,a} = \{(\tilde{\pmb{p}}_{s,a})_i\}_{i=1}^{S-1}$ . Then,  $\tilde{\pmb{q}}_{s,a}$  is Gaussian-distributed with the mean  $\pmb{q}_{s,a}^* = \{(p_{s,a}^*)_i\}_{i=1}^{S-1}$  and invertible covariance matrix  $I(\pmb{q}_{s,a}^*)^{-1}/N = I((\pmb{p}_{s,a}^*)_i)_{i=1}^{S-1})^{-1}/N$ .
Thus, if we sample  $\mathbf{q}_{s,a}$  from  $\mathcal{N}\left(\mathbf{q}_{s,a}^{*}, I(\mathbf{q}_{s,a}^{*})^{-1}/N\right)$  and  $\mathbf{q}_{s,a} \succeq \mathbf{0}$  and  $\sum_{i=1}^{S-1} (\mathbf{q}_{s,a})_i \leq 1$ , then we can easily compute the corresponding transition model as
$$
\{(\pmb {q} _ {s, a}) _ {i} \} _ {i = 1} ^ {S - 1} = \{(\pmb {p} _ {s, a}) _ {i} \} _ {i = 1} ^ {S - 1}
$$
$$
\left(\boldsymbol {q} _ {s, a}\right) _ {S} = 1 - \sum_ {i = 1} ^ {S - 1} \left(\boldsymbol {p} _ {s, a}\right) _ {i}. \tag {39}
$$
Equipped with the above notation, we can proceed to prove the result.
Using the definition of positive semi-definite matrices in Equation (38), we can write
$$
\left(\left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} I \left(\boldsymbol {p} _ {s, a} ^ {*}\right) ^ {- 1} - \left(\boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \left(\boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) ^ {\mathsf {T}}\right) \succeq 0. \tag {40}
$$
Sub-matrices of positive-semidefinite matrices are also positive-semidefinite. Thus, we can write,
$$
\left(\left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) ^ {- 1} - \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\mathsf {T}}\right) \overset {\mathrm {(a)}} {\succeq} 0
$$
$$
\left(I \left(\boldsymbol {q} _ {s, a} ^ {*}\right)\right) \left((\Phi^ {- 1} (1 - \alpha)) ^ {2} I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) ^ {- 1} - \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\top}\right) \left(I \left(\boldsymbol {q} _ {s, a} ^ {*}\right)\right) ^ {\top} \succeq^ {(b)} 0
$$
$$
\left(\left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) ^ {- 1} - \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\mathsf {T}}\right) I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\mathsf {T}} \succeq 0 ^ {(c)}
$$
$$
\left(\left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\top} I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right)\right) \left(\left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} \boldsymbol {I} \right.
$$
$$
\left. - \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) \left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\top}\right) I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\top} \left(\left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right) ^ {\top} I \left(\boldsymbol {q} _ {s, a} ^ {*}\right) \left(\left(\boldsymbol {q} _ {s, a} - \boldsymbol {q} _ {s, a} ^ {*}\right)\right) ^ {\top} \succeq^ {(d)} 0 \right.
$$
$$
\big (\Phi^ {- 1} (1 - \alpha) \big) ^ {2} \big (\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*} \big) ^ {\mathsf {T}} I (\pmb {q} _ {s, a} ^ {*}) \big (\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*} \big) - \big ((\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*}) ^ {\mathsf {T}} I (\pmb {q} _ {s, a} ^ {*}) (\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*}) \big) ^ {2} \overset {\mathrm {(e)}} {\succeq} 0 .
$$
$(a)$  holds because for any  $\mathbf{A} \in \mathbb{R}^{n \times n}$  if  $\pmb{x}^{\top} \pmb{A} \pmb{x} \succeq 0 \forall \pmb{x} \in \mathbb{R}^{n}$ , then  $\mathbf{A} \succeq 0$ ,  $(b)$  follows from  $\pmb{U}^{\top} \pmb{M} \pmb{U} \succeq 0 \forall \pmb{U}, \pmb{M} \succeq 0$ ,  $(c)$  follows from simple algebraic manipulations,  $(d)$  follows from  $(\pmb{q}_{s,a} - \pmb{q}_{s,a}^{*})^{\top} I(\pmb{q}_{s,a}^{*})(\pmb{q}_{s,a} - \pmb{q}_{s,a}^{*}) \succeq 0$  because  $I(\pmb{q}_{s,a}^{*})^{-1} \succeq 0$ , and  $(e)$  follows from simply rearranging all the terms in the above equation.
$$
\left(\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*}\right) ^ {\top} I (\pmb {p} _ {s, a} ^ {*}) \left(\pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*}\right) \leq \left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} \Rightarrow \| \pmb {q} _ {s, a} - \pmb {q} _ {s, a} ^ {*} \| _ {I \left(\pmb {q} _ {s, a} ^ {*}\right) ^ {- 1}} \leq \Phi^ {- 1} (1 - \alpha).
$$
Thus, it follows from construction that  $\| \pmb{p}_{s,a} - \pmb{p}_{s,a}^{*}\|_{I^{\prime}(\pmb{p}_{s,a}^{*})^{-1}}\leq \Phi^{-1}(1 - \alpha)$
The proof for the other direction is simply the reverse of this proof and therefore we omit it.
B.11 Proof of Theorem 4.3
Proof. For any state  $s$  and action  $a$  and transition probabilities  $\tilde{\pmb{p}}_{s,a}$ , let  $\tilde{\pmb{q}}_{s,a} \in \mathbb{R}^{S-1}$  represent the first  $S-1$  elements of the random variable  $\tilde{\pmb{p}}_{s,a}$ , i.e.,  $\tilde{\pmb{q}}_{s,a} = \{(\tilde{\pmb{p}}_{s,a})_i\}_{i=1}^{S-1}$ . Then,  $\tilde{\pmb{q}}_{s,a}$  is Gaussian-distributed with the mean  $\pmb{q}_{s,a}^* = \{(p_{s,a}^*)_i\}_{i=1}^{S-1}$  and invertible covariance matrix  $I(\pmb{q}_{s,a}^*)^{-1}/N = I((\pmb{p}_{s,a}^*)_i)^{S-1}_{i=1})^{-1}/N$ .
$$
\left\{\left(\boldsymbol {q} _ {s, a}\right) _ {i} \right\} _ {i = 1} ^ {S - 1} = \left\{\left(\boldsymbol {p} _ {s, a}\right) _ {i} \right\} _ {i = 1} ^ {S - 1}
$$
$$
\left(\boldsymbol {q} _ {s, a}\right) _ {S} = 1 - \sum_ {i = 1} ^ {S - 1} \left(\boldsymbol {p} _ {s, a}\right) _ {i}. \tag {41}
$$
Equipped with the above notation, we will now prove this theorem by contradiction.
For any state  $s$  and action  $a$ , suppose that  $\lim_{N\to \infty}\sqrt{N} (\mathcal{P}_{s,a}^{\mathrm{BCR}_{\alpha}} - \pmb{p}_{s,a}^{*})\subseteq \lim_{N\to \infty}\sqrt{N}\xi (\mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}} - \pmb{p}_{s,a}^{*})$ .
Since  $\mathcal{P}_{s,a}^{\mathrm{BCR}_{\alpha}}$  is a credible region, it satisfies that
$$
\begin{array}{l} 1 - \alpha \stackrel {\text {(a)}} {\leq} \Pr \left[ \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \in \mathcal {P} _ {s, a} ^ {\mathrm {B C R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*} \right] \\ \stackrel {(b)} {=} \Pr \left[ \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \in \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\mathcal {P} _ {s, a} ^ {\mathrm {B C R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right)\right] \\ \stackrel {\mathrm {(c)}} {=} \Pr \left[ \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \in \lim  _ {N \rightarrow \infty} \sqrt {N} \xi \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right)\right] \\ \stackrel {\mathrm {(d)}} {=} \operatorname * {P r} \left[ \lim  _ {N \rightarrow \infty} \sqrt {N} (\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}) ^ {\mathsf {T}} I ^ {\prime} (\boldsymbol {p} _ {s, a} ^ {*}) \sqrt {I ^ {\prime} (\boldsymbol {p} _ {s , a} ^ {*}) ^ {- 1} N} (\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}) \leq \xi^ {2} (\Phi^ {- 1} (1 - \alpha)) ^ {2} \right]. \\ \end{array}
$$
$(a)$  follows from the definition of Bayesian credible regions,  $(b)$  follows from multiplying by  $\sqrt{N}$  on both sides and taking limit  $N\to \infty$ ,  $(c)$  follows from the assumption  $\lim_{N\to \infty}\sqrt{N} (\mathcal{P}_{s,a}^{\mathrm{BCR}_{\alpha}} - \pmb{p}_{s,a}^{*})\subseteq$ $\lim_{N\to \infty}\sqrt{N}\xi (\mathcal{P}_{s,a}^{\mathrm{VaR}_{\alpha}} - \pmb{p}_{s,a}^{*})$ , and  $(d)$  follows from applying results in (37) and squaring both sides.
Since  $\xi < \frac{\sqrt{\chi_{S - 1,1 - \alpha}^2}}{\Phi^{-1}(1 - \alpha)}$ , there exists  $\beta > 0$ , such that  $\xi^2 (\Phi^{-1}(1 - \alpha))^2 \leq \chi_{S - 1,1 - \beta - \alpha}^2 - \beta$ . Fix such  $\beta$ . Then we can write
$$
\begin{array}{l} 1 - \alpha \leq \Pr \left[ \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) ^ {\top} I ^ {\prime} \left(\boldsymbol {p} _ {s, a} ^ {*}\right) \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \leq \xi^ {2} \left(\Phi^ {- 1} (1 - \alpha)\right) ^ {2} \right] \\ \stackrel {\mathrm {(a)}} {\leq} \Pr \left[ \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) ^ {\mathsf {T}} I ^ {\prime} \left(\boldsymbol {p} _ {s, a} ^ {*}\right) \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \leq \left(\chi_ {S - 1, 1 - \beta - \alpha} ^ {2} - \beta\right)\right] \\ \stackrel {(b)} {\leq} \Pr \left[ \lim  _ {N \rightarrow \infty} \| \sqrt {N} \left(\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}\right) \| _ {I ^ {\prime} \left(\boldsymbol {p} _ {s, a} ^ {*}\right) ^ {- 1}} ^ {2} \leq \left(\chi_ {S - 1, 1 - \beta - \alpha} ^ {2} - \beta\right)\right] \\ \stackrel {(c)} {\leq} \Pr \left[ \lim  _ {N \rightarrow \infty} \| \sqrt {N} (\tilde {\boldsymbol {p}} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*}) \| _ {I ^ {\prime} (\boldsymbol {p} _ {s, a} ^ {*}) ^ {- 1}} ^ {2} \leq \chi_ {S - 1, 1 - \beta - \alpha} ^ {2} \right] \\ \stackrel {(d)} {\leq} 1 - \beta - \alpha <   1 - \alpha \quad \text {C o n t r a d i c t i o n !}. \\ \end{array}
$$
(a) follows from choosing  $\beta > 0$  such that  $\xi^2 (\Phi^{-1}(1 - \alpha))^2 \leq \chi_{S - 1,1 - \beta -\alpha}^2 -\beta$ , (b) follows from simple algebraic manipulations,  $c$  follows because of the monotonic property of CDF, and (d) follows because  $\lim_{N\to \infty}\sqrt{N} (\tilde{\pmb{p}}_{s,a} - \pmb{p}_{s,a}^{*})\sim \mathcal{N}(\mathbf{0},I(\pmb{p}_{s,a}^{*}))$ , from construction,  $\lim_{N\to \infty}\| \sqrt{N} (\tilde{\pmb{p}}_{s,a} - \pmb{p}_{s,a}^{*})\|_{I^{\prime}(\pmb{p}_{s,a}^{*})^{-1}}^{2} = \lim_{N\to \infty}\| \sqrt{N} (\tilde{\pmb{q}}_{s,a} - \pmb{q}_{s,a}^{*})\|_{I(\pmb{q}_{s,a}^{*})^{-1}}^{2}$  and,  $\lim_{N\to \infty}\| \sqrt{N} (\tilde{\pmb{q}}_{s,a} - \pmb{q}_{s,a}^{*})\|_{I(\pmb{q}_{s,a}^{*})^{-1}}^{2}$  is a sum of squares of  $S - 1$  normal random variables, which in turn is a standard Chi-squared random variable with degrees of freedom  $S - 1$ . Therefore, the probability in (d) can be at most  $1 - \beta -\alpha$ .
Therefore,  $\xi \geq \frac{\sqrt{\lambda_{S - 1,1 - \alpha}^2}}{\Phi^{-1}(1 - \alpha)}$ .
Algorithm C.1: VaR Value Iteration Algorithm for Gaussian case
Input: Confidence  $\alpha \in (0,1 / 2)$  , Posterior distribution  $f$  , Value approximation error  $\varepsilon \geq 0$  Output: Robust policy  $\pi_{k}$  , Lower bound  $\pmb{u}_{k}$    
1 Initialize robust value-function  $\pmb {u}_0\in \mathbb{R}^S,k = 0$    
2 Sample M models  $\tilde{P} (\omega_1),\tilde{P} (\omega_2),\ldots ,\tilde{P} (\omega_M)$  from posterior  $f$    
3 repeat   
4 for  $s\gets 1$  to  $S$  do   
5 for  $a\gets 1$  to A do   
6  $\begin{array}{r}\pmb {w}_{s,a}\gets \pmb {r}_{s,a} + \gamma \pmb {u}_k\\ \pmb {q}_a\gets \bar{\pmb{p}}_{s,a}^{\mathrm{T}}\pmb {w}_{s,a} - \Phi^{-1}(1 - \alpha)\sqrt{\pmb{w}_{s,a}^{\mathrm{T}}\pmb{\Sigma}_{s,a}\pmb{w}_{s,a}}\quad /\bar{\pmb{p}}_{s,a},\pmb {\Sigma}_{s,a} \end{array}$  are mean and covariance of  $f$  at  $(s,a)$    
8 end   
9 end   
10  $k\gets k + 1$    
11  $\pmb {u}_k(s)\gets \max_{a\in \mathcal{A}}(\pmb {q}_a),\quad \pi_k(s)\gets \arg \max_{a\in \mathcal{A}}(\pmb {q}_a)$  // VaR Bellman optimality update   
12 until  $\| u_{k} - u_{k - 1}\|_{\infty}\leq \varepsilon^{(1 - \gamma)} / \gamma$    
13 return  $\pi_k,u_k$
D Scalability of the VaR Framework
We define the  $\mathrm{VaR}_{\alpha}$  q-value function (Q-function) for any policy  $\pi \in \Pi_{D}$  as  $\pmb{q}^{\pi}:S\times A\to \mathbb{R}$  such that for any state  $s$  and action  $a$ $\pmb {q}^{\pi}(s,a) = \mathrm{VaR}_{\alpha}[\pmb{p}_{s,a}^{\mathsf{T}}(\pmb{r}_{s,a} + \gamma \pmb{v}^{\pi})]$ , where  $\pmb{v}^{\pi} = \mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}\pmb{v}^{\pi}$  is the fixed point of the  $\mathrm{VaR}_{\alpha}$  Bellman evaluation operator for policy  $\pi$ . We will use  $\hat{\pmb{q}}$  to denote the q-value function corresponding to the optimal  $\mathrm{VaR}_{\alpha}$  policy  $\hat{\pi}$ , i.e.,  $\hat{\pi} (s) = \arg \max_{a\in A}\hat{\pmb{q}} (s,a)$ .
Let  $\hat{q}_{\theta}$  and  $\hat{q}_{\bar{\theta}}$  denote the parameterized Q-value and corresponding target Q-value networks with parameters  $\theta \in \Theta$  and  $\bar{\theta} \in \Theta$  respectively. Here  $\hat{q}_{\theta}$  and  $\hat{q}_{\bar{\theta}}$  may represent any function approximators with parameter space  $\Theta$ .
Methods	Riverswim	Inventory	Population-Small	Population
VaR	83.51 ± 11.76	474.98 ± 0.7	-1877.66 ± 104.64	-3338.26 ± 213.44
VaRN	92.32 ± 11.76	472.49 ± 2.24	-2271.13 ± 165.98	-3140.68 ± 122.72
BCR l1	82.04 ± 8.82	377.73 ± 0.0	-3731.29 ± 122.4	-4363.46 ± 240.62
BCR l∞	80.57 ± 0.0	199.82 ± 39.5	-6668.47 ± 43.1	-8118.68 ± 327.74
WBCR l1	82.04 ± 8.82	470.39 ± 5.82	-3286.26 ± 1115.22	-4257.33 ± 194.4
WBCR l∞	80.57 ± 0.0	199.82 ± 39.5	-6395.77 ± 88.1	-7547.52 ± 160.76
Soft-Robust	115.33 ± 1.16	477.81 ± 0.0	-2052.83 ± 78.2	-3869.46 ± 124.72
Naive Hoeffding	52.29 ± 9.18	-0.0 ± 0.0	-7778.21 ± 89.36	-8496.47 ± 205.54
Opt Hoeffding	51.15 ± 6.88	-0.0 ± 0.0	-7723.6 ± 4.82	-8583.83 ± 16.06
Table 2: shows the  $95\%$  confidence interval of the robust (percentile) returns achieved by VaR, VaRN, BCR  $\ell_1$ , BCR  $\ell_{\infty}$ , WBCR  $\ell_1$ , WBCR, Soft Robust, Naive Hoeffding and Opt Hoeffding agents at  $\delta = 0.15$  in Riverswim, Inventory, Population-Small, and Population domain.
Methods	Riverswim	Inventory	Population-Small	Population
VaR	100.27 ± 17.24	483.08 ± 0.4	-1117.57 ± 240.02	-1850.26 ± 191.08
BCR l1	108.9 ± 21.12	391.39 ± 34.28	-2578.25 ± 104.04	-3058.77 ± 972.58
BCR l∞	95.96 ± 0.0	254.43 ± 44.82	-5437.67 ± 46.26	-6428.03 ± 58.36
WBCR l1	108.9 ± 21.12	481.07 ± 4.58	-2251.96 ± 685.08	-2492.59 ± 228.48
WBCR l∞	95.96 ± 0.0	239.76 ± 0.0	-5133.85 ± 85.44	-5944.77 ± 192.38
VaRN	123.78 ± 15.34	482.92 ± 1.18	-1514.44 ± 24.62	-1785.27 ± 125.52
Soft-Robust	165.15 ± 0.54	484.53 ± 0.0	-692.08 ± 55.04	-1565.09 ± 76.24
Naive Hoeffding	53.31 ± 13.26	-0.0 ± 0.0	-7326.64 ± 77.52	-7560.94 ± 295.96
Opt Hoeffding	51.66 ± 9.94	-0.0 ± 0.0	-7020.42 ± 4.74	-7457.76 ± 12.44
The optimal Q-value network  $(\hat{q}_{\pmb{\theta}^{*}})$  can be learned by simply minimizing the empirical  $\mathrm{VaR}_{\alpha}$  Bellman residual error  $J_{\mathrm{VaR}_{\alpha}}(\pmb {\theta})$  , i.e.,
$$
\begin{array}{l} \hat{q}_{\boldsymbol{\theta}^{*}} = \operatorname *{arg  min}_{\boldsymbol {\theta}\in \Theta}J_{\mathrm{VaR}_{\boldsymbol{\alpha}}}(\boldsymbol {\theta}) \\ = \underset {\boldsymbol {\theta} \in \Theta} {\arg \min } \mathbb {E} _ {(s, a) \sim \mathcal {D}} \left[ \left(q _ {\boldsymbol {\theta}} (s, a) - \underset {a ^ {\prime} \in \mathcal {A}} {\max } \widehat {\operatorname {V a R} _ {\alpha}} [ \mathbb {E} _ {s ^ {\prime} \sim \tilde {\boldsymbol {p}} _ {s, a}} \left[ \boldsymbol {r} _ {s, a} + \gamma \bar {\boldsymbol {q}} _ {\boldsymbol {\theta}} \left(s ^ {\prime}, a ^ {\prime}\right) \right]\right) ^ {2} \right] ， \tag {42} \\ \end{array}
$$
where data  $\mathcal{D}$  is a list state-action (s, a) tuples that is collected using the current policy on the mean model  $\bar{P}$ . In this case, the target Q-value network parameters  $\bar{\theta}$  is periodically updated by overwriting the target q-value network parameter with the Q-value network parameters  $(\theta)$ .
We can also using any of the existing Actor-Critic methods [22, 42] to learn the optimal  $\mathrm{VaR}_{\alpha}$  policy. In this case, instead of optimizing the policy to minimize the Bellman residual error, the algorithm will have to optimize the policy to minimize the  $\mathrm{VaR}_{\alpha}$  Bellman residual error  $J_{\mathrm{VaR}_{\alpha}}(\pmb{\theta})$
Table 3: shows the  $95\%$  confidence interval of the robust (percentile) returns achieved by VaR, VaRN, BCR  $\ell_1$ , BCR  $\ell_{\infty}$ , WBCR  $\ell_1$ , WBCR, Soft Robust, Naive Hoeffding and Opt Hoeffding agents at  $\delta = 0.30$  in Riverswim, Inventory, Population-Small, and Population domain. Bolded text indicates the instances in which the VaR framework outperforms the other baselines in terms of the mean robust performance.
Hyperparameters for Riverswim Domain
Number of train models per dataset (M)	80
Number of test models (K)	700
Number of train datasets (L)	10
Hyperparameters for Inventory Domain
Number of train models per dataset (M)	80
Number of test models (K)	200
Number of train datasets (L)	10
(a) Inventory Domain
(b) Riverswim Domain
(c) Population Domain
(d) Population-Small Domain
Figure 2: Comparison of test and train robust returns achieved by VaR, VaRN, BCR  $\ell_1$ , BCR  $\ell_{\infty}$ , WBCR  $\ell_1$ , WBCR, Soft Robust, Naive Hoeffding and Opt Hoeffding agents at confidence level  $\delta = 0.05$  in Riverside, Inventory, Population-Small and Population domain. VaR framework achieves the highest mean robust returns in most of the domains on test and train datasets.
Hyperparameters for Population-Small Domain
Number of train models per dataset (M)	80
Number of test models (K)	100
Number of train datasets (L)	10
Hyperparameters for Population Domain
Number of train models per dataset (M)	800
Number of test models (K)	1000
Number of train datasets (L)	9
E.1 Code
The code and datasets are made available at https://github.com/elitalobo/VaRFramework.git.