SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Figure 1a (left) compares the asymptotic radius of  $\Sigma^{-1}$ -Minkowski norm BCR ambiguity sets to  $\mathrm{VaR}_{\alpha}$  ambiguity sets, where  $\Sigma$  is the covariance matrix. The size of the BCR ambiguity sets significantly grows with the number of states. Figure 1b and Figure 1c (right) compare the asymptotic forms of  $\mathrm{BCR}_{\alpha}$  and  $\mathrm{VaR}_{\alpha}$  ambiguity sets under high and low uncertainty in  $\tilde{P}$  at confidence level  $\alpha = 0.2$  respectively. The grey dots represent the transition probabilities samples from a Dirichlet distribution. The sizes of  $\mathrm{BCR}_{\alpha}$  and  $\mathrm{VaR}_{\alpha}$  ambiguity sets increase with an increase in the uncertainty in  $\tilde{P}$ , however, the  $\mathrm{VaR}_{\alpha}$  ambiguity sets are smaller than the  $\mathrm{BCR}_{\alpha}$  ambiguity sets.
(b) Dirichlet(5,5,5)
(c) Dirichlet(30,30,30)
We now show that the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  optimizes a lower bound on the percentile criterion. Given a policy  $\pi$ , a state  $s \in S$ , and transition probabilities  $P$ , let
$$
(\mathcal {T} _ {\boldsymbol {P}} ^ {\pi} \boldsymbol {v}) _ {s} = \sum_ {a \in \mathcal {A}} \pi (s, a) \boldsymbol {p} _ {s, a} ^ {\top} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}),
$$
represent the Bellman evaluation operator for transition probabilities  $P$ . Furthermore, let
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} ^ {\pi} \boldsymbol {v}\right) _ {s} = \sum_ {a \in \mathcal {A}} \pi (s, a) \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) \right],
$$
represent the VaR Bellman evaluation operator for random transition probabilities  $\tilde{P}$ . We use  $\hat{\boldsymbol{v}}^{\pi}$  to denote the fixed point of  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$ . Furthermore, we use  $\tilde{\boldsymbol{v}}^{\pi}$  to represent the random fixed point of  $\mathcal{T}_{\tilde{P}}^{\pi}$ , which is computed using a random realization of the transition probabilities  $\tilde{P}$  sampled from the posterior distribution conditioned on observed transitions  $\mathcal{D}$ .
Proposition 3.2 (Lower Bound Percentile Criterion). For any  $\delta \in (0,1 / 2)$ , if we set the confidence level  $\alpha$  in the operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}$  to  $\delta /s$ , then for every policy  $\pi \in \Pi : \operatorname*{Pr}_{\tilde{\pmb{P}}}[\hat{\pmb{v}}^{\pi} \preceq \tilde{\pmb{v}}^{\pi}|\mathcal{D}] \geq 1 - \delta$ .
Proposition 3.2 shows that for any policy  $\pi$  and state  $s$ , the  $\mathrm{VaR}_{\alpha}$  value at state  $s$ ,  $\hat{\pmb{v}}^{\pi}(s)$  lower bounds the true value  $\tilde{\pmb{v}}^{\pi}(s)$  with high confidence. Comparing Proposition 3.2 with the definition of the percentile-criterion in (1), we see that the percentile-criterion requires confidence guarantees only on the returns computed from the initial states, whereas the equation in Proposition 3.2 provides confidence guarantees on the value of every state. Therefore, for any policy  $\pi$ , the value  $\pmb{p}_0^\top \hat{\pmb{v}}^{\pi}$  is a lower bound on the percentile-criterion objective  $\mathrm{VaR}_{\delta}[\rho (\pi ,\tilde{\pmb{P}})]$ . Since  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  finds a policy  $\pi$  that maximizes the value  $\pmb{p}_0^\top \hat{\pmb{v}}^{\pi}$ , it follows [37] that  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  optimizes a lower bound on the percentile criterion in (1).
Proposition 3.3. Suppose that  $\tilde{\pmb{p}}_{s,a}$  for any state  $s$  and action  $a$ , is a multivariate sub-Gaussian with mean  $\bar{\pmb{p}}_{s,a}$  and covariance factor  $\pmb{\Sigma}_{s,a}$ , i.e.,  $\mathbb{E}\left[\exp \left(\lambda (\tilde{\pmb{p}}_{s,\pmb{a}} - \bar{\pmb{p}}_{s,a})^{\top}\pmb {w}\right)\right]\leq \exp \left(\lambda^{2}\pmb{w}^{\top}\pmb{\Sigma}_{s,a}\pmb {w} / 2\right),\forall \lambda \in \mathbb{R},\forall \pmb {w}\in \mathbb{R}^{S}$ . Then, for any state  $s\in S$ ,  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  satisfies
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} \geq \max  _ {a \in \mathcal {A}} \left(\bar {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {w} _ {s, a} - \sqrt {2 \ln (1 / \alpha)} \sqrt {\boldsymbol {w} _ {s , a} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right).
$$
As a special case, when  $\tilde{\pmb{p}}_{s,a}$  is normally distributed  $\tilde{\pmb{p}}_{s,a} \sim \mathcal{N}(\bar{\pmb{p}}_{s,a}, \pmb{\Sigma}_{s,a})$ , then  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  for any state  $s \in S$  can be expressed as
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} = \max _ {a \in \mathcal {A}} \left(\bar {\boldsymbol {p}} _ {s, a} ^ {\intercal} \boldsymbol {w} _ {s, a} - \Phi^ {- 1} (1 - \alpha) \sqrt {\boldsymbol {w} _ {s , a} \boldsymbol {\Sigma} _ {s , a} \boldsymbol {w} _ {s , a}}\right).
$$
Proposition 3.3 shows that by assuming that the transition probabilities are sub-Gaussian, we can easily compute a lower bound of the VaR Bellman update  $(\mathcal{T}_{\mathrm{VaR}_{\alpha}}\pmb {v})$  for a given value function using only the mean and the covariance matrix of  $\tilde{\pmb{P}}$ . In the special case where  $\tilde{\pmb{P}}$  is normally distributed, we can compute the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}(\pmb {v})$  exactly.
3.1 Performance Guarantees
We now derive finite-sample and asymptotic bounds on the loss of the VaR framework.
Theorem 3.4 (Performance). Let  $\hat{\pmb{v}}$  be the fixed point of the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  and  $\pi^{*}$  be the optimal policy in (1). Let  $\rho^{*} = \mathrm{VaR}_{\delta}[\rho (\pi^{*},\tilde{\pmb{P}})]$  denote the optimal percentile returns and  $\hat{\rho} = \pmb{p}_0^\top \hat{\pmb{v}}$  denote the lower bound on the percentile returns computed using the Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  with  $\alpha = \delta /s$ . Then for each  $\delta \in (0,1 / 2)$ :
$$
\rho^ {*} - \hat {\rho} \leq \frac {1}{1 - \gamma} \max  _ {s \in S} \max  _ {a \in A} \left(\operatorname {V a R} _ {1 - \frac {1 - \delta}{S}} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right] - \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\mathsf {T}} \boldsymbol {w} _ {s, a} \right]\right). \tag {6}
$$
Theorem 3.4 bounds the finite sample performance loss of the VaR framework. The loss varies proportionally to the maximum difference between the  $\delta / s$  and  $1 - (1 - \delta / s)$  percentile of the one-step Bellman update for the optimal robust value function  $\hat{\pmb{v}}$ . Furthermore, the VaR framework performs better when the uncertainty in the transition probabilities is small.
Theorem 3.5 (Asymptotic Performance). Suppose that the normality assumptions on the posterior distribution  $\hat{\pmb{P}}$  in Section 2 are satisfied. For any  $\delta \in (0,1/2)$ , set  $\alpha = \delta/s$  in  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ . For any states and action  $a$ , let  $I(\pmb{p}_{s,a}^{*})$  be the Fisher information matrix corresponding to the true transition probabilities  $\pmb{p}_{s,a}^{*}$ . Furthermore, let  $\sigma_{\max}^2 = \max_{s \in S, a \in A} \hat{\pmb{w}}_{s,a}^{\top} I(\pmb{p}_{s,a}^{*})^{-1} \hat{\pmb{w}}_{s,a}$  be the maximum over state-action pairs of the asymptotic variance of the returns estimate  $\hat{\pmb{p}}_{s,a}^{\top} \hat{\pmb{w}}_{s,a}$ . Then the asymptotic performance of the VaR framework  $\hat{\rho}$  w.r.t. the optimal percentile returns  $\rho^{*}$  satisfies
$$
\lim _ {N \to \infty} \sqrt {N} (\rho^ {*} - \hat {\rho}) \leq \frac {1}{1 - \gamma} \left(2 \Phi^ {- 1} (1 - \delta / s) \sigma_ {\max}\right) \leq \frac {1}{1 - \gamma} \sqrt {8 \ln (S / \delta)} \sigma_ {\max}.
$$
Theorem 3.5 shows that the asymptotic loss in performance of the VaR framework convergence to 0, i.e.,  $\lim_{N\to \infty}(\rho^{*} - \hat{\rho}) = 0$
3.2 Dynamic Programming Algorithm
We provide a detailed description of the VaR value iteration algorithm (Algorithm 3.1) below. We also bound the number of samples required to estimate a single VaR Bellman update  $(\mathcal{T}_{\mathrm{VaR}_{\alpha}}^{\pi}\pmb {v})_{s}$  for any given policy  $\pi$  and state  $s$  with high confidence  $1 - \zeta$ .
Algorithm 3.1: Generalized VaR Value Iteration Algorithm
Input: Confidence  $\alpha \in (0,1 / 2)$  , Value approximation error  $\varepsilon \geq 0$  , Transition models  $\tilde{P} (\omega_1),\tilde{P} (\omega_2),\ldots ,\tilde{P} (\omega_M)$  sampled from posterior distribution  $f$  Output: Robust policy  $\pi_{k}$  , Lower bound  $\pmb{u}_{k}$    
1 Initialize robust value-function  $\pmb {u}_0 = \pmb {0},k = 0$    
repeat   
for s<-1 to S do for a<-1 to A do  $\begin{array}{rlr}{\pmb{w}_{s,a}} & {\leftarrow} & {\pmb{r}_{s,a} + \gamma \pmb{u}_k}\\ {\pmb{q}_a\leftarrow \widehat{\mathrm{VaR}}_\alpha [\widehat{\pmb{p}}_{s,a}^\top \pmb{w}_{s,a}]} \end{array}$  // 1-step return from  $(s,a)$  // empirical VaR   
7 end   
8 end   
9  $k\gets k + 1$    
10  $u_{k}(s)\gets \max_{a\in \mathcal{A}}(q_{a}),\quad \pi_{k}(s)\gets \arg \max_{a\in \mathcal{A}}(q_{a})$  // VaR Bellman optimality update   
11 until  $\| u_k - u_{k - 1}\|_{\infty}\leq^{\varepsilon (1 - \gamma) / \gamma}$    
12 return  $\pi_{k},u_{k}$
In each iteration of Algorithm 3.1, we compute the one-step VaR Bellman update  $\mathcal{T}_{\mathrm{VaR}_\alpha}(\pmb{v})$  using the current value function  $\pmb{v}$ . When  $\tilde{\pmb{P}}$  is not normally distributed, we use the Quick Select algorithm [23] to efficiently compute the empirical estimate of the  $\alpha$ -percentile of 1-step returns for any state  $s$ , action  $a$ , and value function  $\pmb{v}$ , i.e.,  $\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}(\pmb{r}_{s,a} + \gamma \pmb{v})]$ . On the other hand, when  $\tilde{\pmb{P}}$  is normally distributed, we compute the VaR Bellman optimality update  $\mathcal{T}_{\mathrm{VaR}_\alpha}(\pmb{v})$  using the empirical estimate of mean  $(\bar{\pmb{p}}_{s,a})_{s\in S,a\in A}$  and covariance  $(\pmb{\Sigma}_{s,a})_{s\in S,a\in A}$  of the transition probabilities derived from the data  $\mathcal{D}$  (Proposition 3.3). We repeat these steps until convergence.
Proposition 3.6 (Empirical Error Bound). For any state  $s$ , action  $a$  and value function  $\mathbf{v}$ , let  $\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  represent the empirical estimate of  $\alpha$ -percentile of returns  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  and  $\Phi_f$  represent the cumulative density function (CDF) of the random estimate of returns  $\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}$ . Suppose that  $\Phi_f$  is differentiable at the point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$  and let  $\eta = \Phi_f'(\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}])$  represents the density of estimate of returns at point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$ . Let  $M^{*}$  be the number of posterior samples required to obtain empirical error  $\varepsilon \in \mathbb{R}$ , with confidence  $1 - \zeta$ , where  $0 < \zeta < 1$ , i.e.,  $\operatorname*{Pr}\left[|\widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}] - \mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]| > \varepsilon \right] \leq \zeta$ . Then,  $\lim_{\varepsilon \to 0} M^{*}\varepsilon^{2} = \ln(2 / \zeta) / 2\eta^{2}$ .
We now show that Algorithm 3.1 produces a policy  $\pi_{k}$  and value function  $\pmb{u}_{k}$  that approximates the optimal value function  $\hat{\pmb{v}}$ .
Proposition 3.7 (Value Iteration Error). Define the empirical VaR Bellman optimality operator  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  for any value  $\pmb{v} \in \mathbb{R}^{S}$  and state  $s \in S$  as  $(\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}\pmb{v})_s = \max_{a \in \mathcal{A}} \widehat{\mathrm{VaR}}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\pmb{w}_{s,a}]$ . Let
$\hat{\pmb{v}} \in \mathbb{R}^S$  and  $\hat{\pmb{u}} \in \mathbb{R}^S$  represent the fixed points of  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  and  $\mathcal{T}_{\widehat{\mathrm{VaR}}_{\alpha}}$  respectively. Suppose that Algorithm 3.1 returns policy  $\pi_k$  and value function  $\pmb{u}_k$  and  $\| \hat{\pmb{u}} - \pmb{u}_0 \|_{\infty} \leq r_{\max} / 1 - \gamma$ . Then,
$$
\left\| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {k} \right\| _ {\infty} \leq \varepsilon , a n d \left\| \hat {\boldsymbol {u}} - \boldsymbol {u} _ {\pi_ {k}} \right\| _ {\infty} \leq \frac {2 \varepsilon \gamma}{1 - \gamma}.
$$
Furthermore, the gap between the empirical and true value function is bounded by
$$
\| \hat {\boldsymbol {v}} - \hat {\boldsymbol {u}} \| _ {\infty} \leq \frac {1}{1 - \gamma} \min  \left(\| \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {u}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {u}} \| _ {\infty}, \| \mathcal {T} _ {\widehat {\mathrm {V a R}} _ {\alpha}} \hat {\boldsymbol {v}} - \mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \hat {\boldsymbol {v}} \| _ {\infty}\right).
$$
Proposition 3.8 (Time Complexity). Let  $r_{\max} = \max_{s,s' \in S, a \in \mathcal{A}} |R(s,a,s')|$ . Then, Algorithm 3.1 terminates in  $k = \lceil \log_{1/\gamma}\left(\frac{r_{\max}}{\varepsilon(1 - \gamma)}\right)\rceil$  iterations with time complexity  $\mathcal{O}\left(S^2 AM\log_{1/\gamma}\left(\frac{r_{\max}}{\varepsilon(1 - \gamma)}\right)\right)$ .
Furthermore, for any failure probability  $\zeta \in (0,1)$ , suppose that the CDF  $(\Phi_f)$  of the random estimate of 1-step returns  $\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}$  is differentiable at the point  $\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}]$ , and set  $\eta = \Phi'(\mathrm{VaR}_{\alpha}[\tilde{\pmb{p}}_{s,a}^{\top}\hat{\pmb{w}}_{s,a}])$  and  $M = \mathcal{O}\Big(\frac{\log(S / \zeta)}{\eta^{2}\varepsilon^{2}(1 - \gamma)^{2}}\Big)$ . Then with probability at least  $1 - \zeta$ , it holds that  $\| \hat{\pmb{v}} - \pmb{u}_k \|_\infty \leq \mathcal{O}(\varepsilon)$ , and Algorithm 3.1 runs in  $\mathcal{O}\left(\frac{S^2A\log_1 / \gamma\left(\frac{rmax}{\varepsilon(1 - \gamma)}\right)\log\left(\frac{S}{\zeta}\right)}{\eta^{2}\varepsilon^{2}(1 - \gamma)^{2}}\right)$  time.
4 Comparison with Bayesian Credible Regions
We are now ready to answer the question: Are Bayesian credible regions the optimal ambiguity sets for optimizing the percentile criterion? For this, we compare the VaR framework with BCR Robust MDPs. First, we derive the robust form of the VaR framework and show that in contrast to the BCR Bellman operator, the VaR Bellman optimality operator implicitly constructs value function dependent ambiguity sets, and thus, these sets tend to be smaller (Proposition 4.1). Then, we compare the asymptotic radii of the BCR ambiguity sets and the VaR ambiguity sets implicitly constructed by  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ . For any given confidence level  $\alpha$ , the radius of the VaR ambiguity sets are asymptotically smaller than that of BCR ambiguity sets (Theorem 4.3). Precisely, the ratio of the radii of VaR ambiguity sets to BCR ambiguity sets is at least  $\sqrt{\chi_{S-1,1-\alpha}^2 / \Phi^{-1}(1-\alpha)}$ , where  $\chi_{S-1,1-\alpha}^2$  is the CDF inverse of  $1 - \alpha$  percentile of Chi-squared distribution with degree of freedom  $S-1$  and  $\Phi^{-1}(1 - \alpha)$  is the  $1 - \alpha$  percentile of  $\mathcal{N}(0,1)$ . This implies that there exist directions in which the BCR ambiguity sets are at least  $\Omega(\sqrt{S})$  larger than VaR ambiguity sets. Thus, we prove that VaR framework is better suited for optimizing the percentile criterion than RMDPs with BCR ambiguity sets.
For any value function  $\pmb{v}$ , define the VaR ambiguity set  $\mathcal{P}^{\mathrm{VaR},\pmb{v}}$  as
$$
\mathcal {P} ^ {\mathrm {V a R}, \boldsymbol {v}} = \bigotimes_ {s \in \mathcal {S}, a \in \mathcal {A}} \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \boldsymbol {v}} \quad \text {w h e r e} \quad \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \boldsymbol {v}} = \left\{\boldsymbol {p} _ {s, a} \in \Delta^ {\mathcal {S}} \mid \boldsymbol {p} _ {s, a} ^ {\top} \boldsymbol {v} \geq \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \boldsymbol {v} \right] \right\}. \tag {7}
$$
Proposition 4.1 (Equivalence). The VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  can be expressed as
$$
\forall s \in \mathcal {S}: (\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}) _ {s} = \max  _ {a \in \mathcal {A}} \min  _ {\boldsymbol {p} \in \mathcal {P} _ {s, a} ^ {\mathrm {V a R}, \boldsymbol {v}}} \boldsymbol {p} ^ {\mathsf {T}} (\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}) .
$$
Furthermore, the optimal VaR policy  $\hat{\pi} \in \Pi^D$  solves  $\max_{\pi \in \Pi^D} \min_{\pmb{P} \in \mathcal{P}_{\mathrm{VaR},\hat{\nu}}} \rho(\pi, \pmb{P})$ , where  $\hat{\nu} \in \mathbb{R}^S$  is the fixed point of the VaR Bellman operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$ , i.e.,  $\hat{\nu} = (\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\nu})$ .
Proposition 4.1 shows that the VaR Bellman optimality operator optimizes a unique robust MDP whose ambiguity sets are SA-rectangular and value function dependent. Notice that for any state  $s$ , action  $a$  and value function  $\pmb{v}$ , the VaR ambiguity set is a half-space  $\{\pmb{p}_{s,a} \in \Delta^S : \pmb{p}_{s,a}^\top \pmb{v} \geq \mathrm{VaR}_\alpha[\tilde{\pmb{p}}_{s,a}^\top \pmb{v}]\}$  dependent on the value function  $\pmb{v}$ . In contrast, BCR ambiguity sets are independent of any policy or value function and are constructed such that they provide high-confidence guarantees on returns of all policies simultaneously. As a result, BCR ambiguity sets tend to be unnecessarily large.
We now compute the ratio of the asymptotic radii of BCR ambiguity sets and VaR ambiguity sets.
Theorem 4.2 (Asymptotic Radii of VaR Ambiguity Sets). For any state  $s$  and action  $a$ , let  $I(\{(p_{s,a}^*)_i\}_{i=1}^{S-1})$  be the Fisher information of the first  $S-1$  transition probabilities, i.e.,  $\{(p_{s,a}^*)_i\}_{i=1}^{S-1}$ .
Define  $I^{\prime}(\mathbf{p}_{s,a}^{*}) = \begin{bmatrix} I(\{(p_{s,a}^{*})_i\}_{i=1}^{S-1}) & \mathbf{0} \\ \mathbf{0}^{\mathrm{T}} & 0 \end{bmatrix}$ . Suppose that the normality assumptions on the posterior distribution  $\tilde{\mathbf{P}}$  in Section 2 are satisfied. Then,
$$
\lim  _ {N \rightarrow \infty} \sqrt {N} \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right) = \left\{\boldsymbol {p} _ {s, a} \in \Delta^ {\mathcal {S}} \mid \| \boldsymbol {p} _ {s, a} - \boldsymbol {p} _ {s, a} ^ {*} \| _ {I ^ {\prime} \left(\boldsymbol {p} _ {s, a} ^ {*}\right) ^ {- 1}} \leq \Phi^ {- 1} (1 - \alpha) \right\} - \boldsymbol {p} _ {s, a} ^ {*}. \tag {8}
$$
Theorem 4.2 shows that the asymptotic form of the VaR ambiguity set is an ellipsoid. Note that, in contrast to the finite-sample VaR ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR},\boldsymbol{v}}$  in problem (7), the asymptotic VaR ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR}}$  is independent of the value function  $\boldsymbol{v}$ . This is because  $\mathrm{VaR}_{\alpha}\left[\tilde{\boldsymbol{p}}_{s,a}^{\mathrm{T}}\boldsymbol{v}\right]$  is convex in the value function  $\boldsymbol{v}$  [39]. Therefore, the asymptotic ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR}}$  is simply the intersection of closed half-spaces in  $\mathcal{P}_{s,a}^{\mathrm{VaR},\boldsymbol{v}}$ , computed over all value functions  $\boldsymbol{v} \in \mathbb{R}^{S}$  [9]. It is also worth noting that the radius of the asymptotic VaR ambiguity set  $\mathcal{P}_{s,a}^{\mathrm{VaR}}$  is constant. In contrast, the asymptotic radius of the BCR ambiguity sets grows with the number of states in at least one direction, as we show in the following proposition.
Theorem 4.3 (Asymptotic Radius of Bayesian Credible Regions). For any state  $s$  and action  $a$ , let  $\mathcal{P}_{s,a}^{BCR_{\alpha}}$  represent any Bayesian credible region. Let  $\xi < \sqrt{\chi_{S - 1,1 - \alpha}^2} / \Phi^{-1}(1 - \alpha)$ . Suppose that the normality assumptions on the posterior distribution of  $\tilde{\pmb{P}}$  in Section 2 are satisfied. Then,
$$
\forall s \in \mathcal {S}, a \in \mathcal {A}: \lim  _ {N \rightarrow \infty} \sqrt {N} \left(\mathcal {P} _ {s, a} ^ {B C R _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right) \not \subseteq \lim  _ {N \rightarrow \infty} \sqrt {N} \xi \left(\mathcal {P} _ {s, a} ^ {\mathrm {V a R} _ {\alpha}} - \boldsymbol {p} _ {s, a} ^ {*}\right). \tag {9}
$$
We note that Theorem 4.3 is an adaptation of Theorem 10 in [21] in RL which proves that there exist directions in which the Bayesian credible regions is at least  $\xi = \sqrt{\chi_{S - 1,1 - \alpha}^2 / \Phi^{-1}(1 - \alpha)}$  larger than VaR ambiguity sets. Since the value of  $\xi$  only grows with the number of states, we conclude that BCR ambiguity sets are sub-optimal for optimizing the percentile criterion. Figure 1a shows the growth in the ratio of radius of BCR to VaR ambiguity sets with an increasing number of states.
We now empirically analyze the robustness of the VaR framework in three different domains.
Riverswim: The Riverswim MDP [46] consists of five states and two actions. The state represents the coordinates of the swimmer in the river and action represents the direction of the swim. The task of the agent is to learn a policy that would take the swimmer to the other end of the river.
Population Growth Model: The Population Growth MDP [25] models the population growth of pests and consists of 50 states and 5 actions. The states represent the pest population and actions represent the pest control measures. In our experiments, we use two different instantiations of the Population Growth Model: Population-Small and Population, which vary in the number of posterior samples.
Inventory Management: The Inventory Management MDP [56] models the classical inventory management problem and consists of 30 states and 30 actions. States represent the inventory level and actions represent the inventory to be purchased. The sale price  $s$ , holding cost  $c$  and purchase costs  $P$  are 3.99, 0.03, and 2.219. The demand is normally distributed with mean  $= s / 4$  and standard deviation  $s / 6$ .
Implementation details: For each domain in our experiments, we sample a dataset  $\mathcal{D}$  consisting of  $n$  tuples of the form  $\{s,a,r,s'\}$ , corresponding to the state  $s$ , the action taken  $a$ , the reward  $r$  and the next state  $s'$ . We construct a posterior distribution over the models using  $\mathcal{D}$ , assuming Dirichlet priors over the model parameters. Using MCMC sampling, we construct two datasets  $\mathcal{D}_1$  and  $\mathcal{D}_2$  containing  $M$  and  $K$  transition probability models, respectively.
We construct  $L$  train datasets by randomly sampling  $80\%$  of the models from  $\mathcal{D}_1$  each time. We use  $\mathcal{D}_2$  as our test dataset. For any confidence level  $\delta$ , we train one RL agent per train dataset and method.
For evaluation, we consider two instances of the VaR framework: one (denoted by  $VaRN$ ) that assumes that  $\tilde{P}$  is a multivariate normal, and another (denoted by  $VaR$ ) that does not assume any structure over  $\tilde{P}$ . We use seven baseline methods for evaluating the robustness of our framework. They are: Naive Hoeffding [35], Optimized Hoeffding (Opt Hoeffding) [3], Soft-Robust [5], and BCR Robust MDPs with weighted  $\ell_1$  ambiguity sets ( $WBCR\ \ell_1$ ) [3], weighted  $\ell_{\infty}$  ambiguity sets
$(WBCR\ell_{\infty})$  [3], unweighted  $\ell_1$  ambiguity sets  $(BCR\ell_1)$  [40] and unweighted  $\ell_{\infty}$  ambiguity sets  $(BCR\ell_1)$  ambiguity [40] sets. See Appendix C in the appendix for more details.