SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elita A. Lobo
Department of Computer Science  
University of Massachusetts Amherst elobo@umass.edu
Yair Zick
Department of Computer Science  
University of Massachusetts Amherst  
yzick@umass.edu
Cyrus Cousins
Department of Computer Science  
University of Massachusetts Amherst  
cwcousins@umass.edu
Department of Computer Science University of New Hampshire mpetrik@cs.unh.edu
In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion. The percentile criterion is approximately solved by constructing an ambiguity set that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses Bayesian credible regions as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.
Batch Reinforcement Learning (Batch RL) [26] is popularly used for solving sequential decision-making problems using limited data. These algorithms are crucial in high-stakes domains where exploration is either infeasible or expensive, and policies must be learned from limited data. In model-based Batch RL algorithms, transition probabilities are learned from the data as well. Due to insufficient data, these transition probabilities are often imprecise. Errors in transition probabilities can accumulate, resulting in low-performing policies that fail when deployed.
To account for the uncertainty in transition probabilities, prior work uses Bayesian models [10, 13, 27, 40, 45, 47] to model uncertainty and optimize the policy to maximize the returns corresponding to the worst  $\alpha$ -percentile transition probability model. These policies guarantee that the true expected returns will be at least as large as the optimal returns with high confidence. This technique is commonly referred to as the percentile-criterion optimization. Unfortunately, the percentile criterion is NP-hard to optimize. Thus, current work uses Robust Markov Decision Processes (RMDPs) to optimize a lower bound on the percentile criterion. An RMDP takes as input an ambiguity set (uncertainty set) that contains the true transition probability model with high confidence and finds a policy that maximizes the returns of the worst model in the ambiguity set.
Since the percentile criterion is non-convex, constructing ambiguity sets itself is a challenging problem. Existing work uses Bayesian credible regions (BCR) [40] as ambiguity sets. However, these ambiguity sets are often unnecessarily large [15, 40] and result in learning conservative robust policies.
Some recent works approximate ambiguity sets using various heuristics [3, 40], but we show that they remain too conservative. Thus, the question of the optimal ambiguity set, i.e., ambiguity sets that result in optimizing the tightest possible lower bound on the percentile criterion and less-conservative policies remains unanswered.
Our Contributions In this paper, we answer two important questions: a) Are Bayesian credible regions the optimal ambiguity sets for optimizing the percentile criterion? b) Can we obtain a less conservative solution to the percentile criterion than RMDPs with BCR ambiguity sets while retaining its percentile guarantees? Our theoretical findings show that Bayesian credible regions can grow significantly with the number of states and therefore, tend to be unnecessarily large, resulting in highly conservative policies. As our main contribution, we provide a dynamic programming framework (Section 3), which we name the VaR framework, for optimizing a lower bound on the percentile criterion without explicitly constructing ambiguity sets. Specifically, we propose a new robust Bellman operator, the Value at Risk (VaR) Bellman operator, for optimizing the percentile criterion. We show that it is a valid contraction mapping that optimizes a tighter lower bound on the percentile criterion, compared to RMDPs with BCR ambiguity sets (Section 3). We theoretically analyze and bound the performance loss of our framework (Section 3.1). We provide a Generalized VaR Value iteration algorithm and analyze its error bounds. We also show that there exist directions in which the Bayesian credible regions can grow unnecessarily large with the number of states in the MDP and possibly result in a conservative solution. On the other hand, the ambiguity sets implicitly optimized by the VaR Bellman operator tend to be smaller, i.e., they have a smaller asymptotic radius and are independent of the number of states (Section 4). Finally, we empirically demonstrate the efficacy of our framework in three domains (Section 5).
Several work[13, 34, 40] propose different methods for solving the percentile criterion, as well as other robust measures for handling uncertainty in the transition probabilities estimates. Russel and Petrik [40] and Behzadian et al. [3] propose various heuristics for minimizing the size of the ambiguity sets constructed for the percentile-criterion. Russel and Petrik [40] propose a method that interleaves robust value iteration with ambiguity set size optimization. Behzadian et al. [3] propose an iterative algorithm that optimizes the weights of  $\ell_1$  and  $\ell_{\infty}$  ambiguity sets while optimizing the robust policy. However, these methods still construct Bayesian credible sets which can be unnecessarily large and result in conservative policies, as we show in Section 5.
Other works consider partial correlations between uncertain transition probabilities to mitigate the conservativeness of learned policies [4, 15, 19, 29, 30]. These approaches mitigate the conservativeness of S- and SA-rectangular ambiguity sets by capturing correlations between the uncertainty and by limiting the number of times the uncertain parameters deviate from the mean parameters. Despite these heuristics, most of these works [2, 20, 40, 53] either rely on weak statistical concentration bounds to construct frequentist ambiguity sets, or use Bayesian credible regions as ambiguity sets. These sets still tend to be unnecessarily large [15, 40], resulting in conservative policies.
Finally, a large number of works [12, 16, 36, 44, 49, 50, 52] have proposed RL algorithms that use measures like Conditional Value at Risk, Entropic risk measure amongst other risk measures. However, we note that these works use risk measures to obtain robustness guarantees against aleatoric uncertainty (system uncertainty) and not epistemic uncertainty (model uncertainty), which is the focus of our work. Since these works optimize a completely different objective, we do not compare our framework against theirs. Robust RL work [2, 14, 20, 28, 32, 55] proposes other robust measures for handling uncertainty in transition probabilities; however, these approaches do not provide probabilistic guarantees on the expected returns, and compute overly conservative policies.
In the standard reinforcement learning setting, a sequential decision task is modeled as a Markov Decision Process (MDP) [37, 48]. An MDP is a tuple  $\langle S, \mathcal{A}, P, R, \pmb{p}_0, \gamma \rangle$  that consists of (a) a set of states  $S = \{1, 2, \dots, S\}$ , (b) a set of actions  $\mathcal{A} = \{1, 2, \dots, A\}$ , (c) a deterministic reward function  $R: S \times \mathcal{A} \times S \to \mathbb{R}$ , (d) a transition probability function  $P: S \times \mathcal{A} \to \Delta^S$ , (e) an initial state distribution  $\pmb{p}_0 \in \Delta^S$ , where  $\Delta^S$  represents the  $S$ -dimensional probability simplex, and (f) a
discount factor  $\gamma \in [0,1]$ . We use  $p_{s,a}$  to denote the vector of transition probabilities  $P(s,a,\cdot)$  corresponding to the state  $s$  and action  $a$ . Likewise, we use  $r_{s,a}$  to denote the vector of rewards  $R(s,a,\cdot)$  corresponding to state  $s$  and action  $a$ . A Markovian policy  $\pi \colon S \to \Delta^A$  maps each state  $s$  to a distribution over actions  $\mathcal{A}$ . In a general RL setting, the goal is to compute a policy  $\pi$  that maximizes the expected discounted return  $\rho(\pi, P)$  over an infinite horizon,
$$
\max  _ {\pi \in \Pi} \rho (\pi , P) = \max  _ {\pi \in \Pi} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}, s _ {t + 1}\right) \mid s _ {0} \sim \boldsymbol {p} _ {0}, a _ {t} \sim \pi \left(s _ {t}\right), s _ {t + 1} \sim P \left(s _ {t}, a _ {t}, \cdot\right) \right].
$$
The value of a policy  $\pi$  at any state  $s$  is the discounted sum of rewards received by an RL agent, if it starts from state  $s$ , i.e.,  $v^{\pi}(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} r(s_{t}, a_{t}, s_{t+1}) | s_{0} = s, a_{t} \sim \pi, s_{t+1} \sim P(s_{t}, a_{t}, \cdot)\right]$ .
We assume a batch reinforcement learning setting [26] where the reward function is known, but the true transition probabilities  $P^{*}$  are unknown. Following prior work on robust Bayesian RL [8, 13, 40, 54], we use parametric Bayesian models to represent uncertainty over the true transition probabilities  $P^{*}$ . We will use  $\tilde{\pmb{P}} = \{\tilde{p}_{s,a}\}_{s\in S,a\in \mathcal{A}}$  to denote the random transition probabilities. We assume we have a batch of data  $\mathcal{D} = \{s_i,a_i,s_i'\}_{i = 1}^N$ , which in conjunction with some prior distribution defines the posterior distribution of transition probabilities  $\tilde{\pmb{P}}$ .
The Fisher information measures the amount of information about the unknown true parameters  $\pmb{\theta}^{*}\in \mathbb{R}^{d}$  carried by an observable random variable  $\tilde{\pmb{X}}\in \mathbb{R}^m$ . If  $f(\pmb {x};\pmb{\theta}^{*})$  is the probability density of  $\tilde{\pmb{X}}$  conditioned on  $\pmb{\theta}^{*}$ , then the Fisher information is given by  $I(\pmb{\theta}^{*}) = \mathbb{E}\left[(\nabla \log f(\tilde{\pmb{X}};\pmb{\theta}^{*}))(\nabla \log f(\tilde{\pmb{X}};\pmb{\theta}^{*}))^{\top}\right]$ .
Then, in the Bayesian setting, the Bernstein von Mises theorem [51] states that under mild regularity conditions the posterior distribution of the parameters  $\tilde{\pmb{\theta}}$  converges in the limit to the distribution of the MLE of  $\pmb{\theta}^{*}$  which is asymptotically Gaussian, in particular,  $\lim_{N\to \infty}\sqrt{N} (\tilde{\pmb{\theta}} -\pmb{\theta}^{*})\rightsquigarrow \mathcal{N}(\mathbf{0},I(\pmb{\theta}^{*})^{-1})$  where  $\rightsquigarrow$  indicates convergence in distribution. In many cases, the above holds even though the conditions of the Bernstein-von Mises theorem are not met [17, 18]. Of particular interest in this setting is the Dirichlet distribution where the asymptotic MLE is a multivariate Gaussian under certain conditions on the prior distribution [18], although in this case, it is degenerate (i.e., the covariance matrix is not full-rank).
For asymptotic results, we assume that the data  $\mathcal{D}$  is sampled such that each state  $s$  and action  $a$  is observed infinitely many times as  $N\to \infty$ . Furthermore, we assume henceforth that the prior is asymptotically negligible, and the MLE is asymptotically Gaussian with covariance matrix  $I(P^{*})^{-1} / N$ . Therefore, the posterior distribution of  $\tilde{\pmb{P}}$  is also asymptotically Gaussian with covariance matrix  $I(P^{*})^{-1} / N$ . Under these conditions, we will estimate the Fisher information corresponding to  $\pmb{\theta}^{*} = \pmb{P}^{*}$  using the asymptotic covariance of the posterior distribution of  $\tilde{\pmb{P}}$ .
To avoid unnecessary computational technicalities, we will assume that  $\tilde{P}$  is a discrete random variable taking on values  $\tilde{P}(\omega), \omega \in \Omega$  for some  $\Omega = \{1, \dots, M\}$  with a distribution  $f$ . That is, the random variable  $\tilde{P}$  represents a discrete approximation of the true, possibly continuous posterior, as is common in methods like Sample Average Approximation (SAA) [43].
Percentile Criterion The  $\alpha$ -percentile criterion is popularly used to derive robust policies under model uncertainty [13]. It aims to compute a policy  $\pi$  that maximizes the returns corresponding to the worst  $\alpha$ -percentile model:
$$
\underset {\pi \in \Pi} {\arg \max } \left\{y \in \mathbb {R} \mid \Pr_ {\tilde {\boldsymbol {P}} \sim f} \left[ \rho (\pi , \tilde {\boldsymbol {P}}) \geq y \right] \geq 1 - \alpha \right\}. \tag {1}
$$
The value  $y$  lower-bounds the true expected discounted returns with confidence  $1 - \alpha$  where  $\alpha \in (0, 1/2)$ . Optimizing the percentile criterion is equivalent to optimizing the Value at Risk  $(\mathrm{VaR}_{\alpha})$  of expected discounted returns when there exists uncertainty in transition probabilities  $\hat{P}$  and the expected returns function  $\rho$  is lower-semicontinuous. The optimization in (1) is equivalent to
$$
\max  _ {\pi \in \Pi} \operatorname {V a R} _ {\alpha} [ \rho (\pi , \tilde {\boldsymbol {P}}) ] \quad , \tag {2}
$$
where  $\mathrm{VaR}_{\alpha}$  of a bounded random variable  $\tilde{X}$  with a CDF function  $F:\mathbb{R}\to [0,1]$  is defined as [38]
$$
\operatorname {V a R} _ {\alpha} [ \tilde {X} ] = \sup  \left\{t \in \mathbb {R}: \Pr [ \tilde {X} \geq t ] \geq 1 - \alpha \right\}. \tag {3}
$$
A lower value of  $\alpha$  in (1) indicates a higher confidence in the returns achieved in expectation. For example,  $\mathrm{VaR}_{0.05}[\rho (\pi ,\tilde{P})] = x$  indicates that the true returns will be at least equal to the robust returns  $x$  for  $95\%$  of the transition probability models. When clear from context, we use VaR to denote the Value at Risk at confidence level  $\alpha$ . Unfortunately, the optimization problem in (1) is NP-hard to optimize and is usually approximately solved using Robust MDPs.
Robust MDPs Robust MDPs (RMDPs) generalize MDPs to account for uncertainty, or ambiguity, in the transition probabilities. An ambiguity set for an RMDP is constructed such that it contains the true model with high confidence. The optimal policy of a Robust MDP  $\pi^{*}$  maximizes the returns of the worst model in the ambiguity set:  $\pi^{*} = \arg \max_{\pi \in \Pi} \min_{P \in \mathcal{P}} \rho(\pi, P)$ . General RMDPs are NP-hard to solve [53], but they are tractable for broad classes of ambiguity sets. The simplest such type is the SA-rectangular ambiguity set [33, 53], defined as
$$
\mathcal {P} = \left\{\boldsymbol {P} \in (\Delta^ {S}) ^ {S \times A} \mid \boldsymbol {p} _ {s, a} \in \mathcal {P} _ {s, a}, \forall s \in \mathcal {S}, \forall a \in \mathcal {A} \right\},
$$
for a given  $\mathcal{P}_{s,a} \subseteq \Delta^{\mathcal{S}}, s \in \mathcal{S}, a \in \mathcal{A}$ . SA-rectangular ambiguity sets [3, 40] assume that the transition probabilities corresponding to each state-action pair are independent. Similarly to MDPs, the optimal robust value function  $\pmb{v}^{*} \in \mathbb{R}^{S}$  for an SA-rectangular RMDP is the unique fixed point of the robust Bellman optimality operator  $\mathcal{T} \colon \mathbb{R}^{S} \to \mathbb{R}^{S}$  defined as  $(\mathcal{T}\pmb{v})_{s} = \max_{a \in \mathcal{A}} \min_{\pmb{p}_{s,a} \in \mathcal{P}_{s,a}} \pmb{p}_{s,a}^{\top} (\pmb{r}_{s,a} + \gamma \cdot \pmb{v})$ .
To optimize the percentile criterion, an SA-rectangular ambiguity set  $\mathcal{P}$  is constructed such that it contains the true model with high probability, and thus, the following equation holds.
$$
\Pr \left[ \rho (\pi , \tilde {\boldsymbol {P}}) \geq \min  _ {P \in \mathcal {P}} \rho (\pi , \boldsymbol {P}) \right] \geq 1 - \alpha .
$$
Although RMDPs have been used to solve the percentile criterion [3], the quality of the robust policies it computes depends mainly on the size of the ambiguity sets. The larger the ambiguity sets, the more conservative the robust policy [30]. SA-rectangular ambiguity sets are most commonly studied; thus we focus our attention on SA-rectangular Robust MDPs. We investigate whether Bayesian credible regions are optimal ambiguity sets for optimizing the percentile criterion. We refer to SA-rectangular RMDPs and SA-rectangular ambiguity sets as Robust MDPs and ambiguity sets respectively.
Our work focuses on Bayesian (rather than frequentist) ambiguity sets. Bayesian ambiguity sets are usually constructed from Bayesian credible regions (BCR) [3, 40]. Given a state  $s$  and an action  $a$ , let  $\psi_{s,a}$  represent the size of the BCR ambiguity sets;  $\mathcal{P}_{s,a}^{\mathrm{BCR}_{\alpha}}$  and  $\bar{\pmb{p}}_{s,a}$  represent the mean transition probabilities. The set  $\mathcal{P}_{s,a}^{\mathrm{BCR}_{\alpha}}$  is constructed as
$$
\mathcal {P} _ {s, a} ^ {\mathrm {B C R} _ {\alpha}} = \mathcal {P} _ {s, a} (\boldsymbol {b}, \psi , q) = \left\{\boldsymbol {p} _ {s, a} \in \Delta^ {S} \mid \| \boldsymbol {p} _ {s, a} - \bar {\boldsymbol {p}} _ {s, a} \| _ {q, b} \leq \psi_ {s, a} \right\}, \tag {4}
$$
where  $\pmb{q} \in \{1, \infty\}$  represents the norm of the weighted ball in (4) and  $\pmb{b} \in \mathbb{R}_{+}^{S}$  is a weight vector. Here,  $\pmb{b}$  is jointly optimized with  $\psi \in \mathbb{R}$  to minimize the span of the ambiguity sets such that the true model is contained in the ambiguity set with high confidence, i.e.,  $\operatorname*{Pr}\left(\tilde{\pmb{p}}_{s,a} \in \mathcal{P}_{s,a}(\pmb{b}, \psi, q)\right) \geq 1 - \alpha$ . We refer to BCR ambiguity sets with non-uniform weights as weighted BCR ambiguity sets. We refer to the Robust Bellman optimality operator with BCR ambiguity sets  $(\mathcal{T}_{\mathrm{BCR}_{\alpha}})$  as the  $\mathrm{BCR}_{\alpha}$  Bellman optimality operator, and to RMDPs with BCR ambiguity sets as BCR RMDPs. For any  $\delta \in (0, 1/2)$ , setting the confidence level  $\alpha$  in  $\mathcal{T}_{\mathrm{BCR}_{\alpha}}$  to  $\delta / SA$  for all state-action pairs yields  $1 - \delta$  confidence on the returns of the optimal robust policy [3]. However, we show that even span-optimized BCR RMDPs can be sub-optimal for optimizing the percentile criterion.
We use the shorthand  $\boldsymbol{w}_{s,a}$  for any  $s \in S$ ,  $a \in \mathcal{A}$  to denote the vector of values associated with value  $\boldsymbol{v} \in \mathbb{R}^S$  and the one-step return from state  $s$  and action  $a$ , i.e.,  $\boldsymbol{w}_{s,a} = \boldsymbol{r}_{s,a} + \gamma \boldsymbol{v}$ . We use  $\bar{\boldsymbol{p}}_{s,a} \in \mathbb{R}^s$  and  $\boldsymbol{\Sigma}_{s,a} \in \mathbb{R}^{S \times S}$  for any  $s \in S$ ,  $a \in \mathcal{A}$  to represent the empirical mean and covariance of transition probabilities  $\tilde{\boldsymbol{p}}_{s,a}$  estimated from  $\mathcal{D}$ . We use tilde to indicate that it is a random variable. We use  $\phi(\cdot)$  and  $\Phi(\cdot)$  to represent the probability distribution function (PDF) and cumulative distribution function (CDF) respectively of the normal distribution with mean 0 and variance 1. The  $Z$ -Minkowski norm  $\| \boldsymbol{x} \|_Z$  for a vector  $\boldsymbol{x}$  given some positive-definite matrix  $Z$  is defined as  $\| \boldsymbol{x} \|_Z = \sqrt{\boldsymbol{x}^\top Z^{-1}\boldsymbol{x}}$ .
We illustrate the conservativeness of  $\mathrm{BCR}_{\alpha}$  ambiguity sets with the following example.
Example 2.1. Consider an MDP with four states  $\{s_0, s_1, s_2, s_3\}$  and a single action  $\{a_0\}$ . The state  $s_0$  is the initial state and the states  $s_1, s_2, s_3$  are terminal states with zero rewards. For the
sake of simplicity, we assume that it is only possible to transition to state  $s_1, s_2$  and  $s_3$  from state  $s_0$ . The transition probability of  $\pmb{p}_{s_0,a_0}$  is uncertain and distributed as a Dirichlet distribution  $\tilde{\pmb{p}}_{s_0,a_0} \sim \mathrm{Dir}(10,10,1)$  with mean [0.48, 0.48, 0.04]. The rewards for transitions from state  $s_0$  are given by  $r_{s_0,a_0} = [0.25, 0.25, -1]$ .
We wish to optimize the percentile criterion with confidence level  $\delta = 0.2$ . Following the sampling procedure proposed by Russel and Petrik [40] to construct a uniformly weighted  $\mathrm{BCR}_{\alpha}$  ambiguity set for  $\tilde{\pmb{p}}_{s_0,a}$  with 100 posterior samples, yields an ambiguity set  $\mathcal{P}_{s_0,a_0}^{\mathrm{BCR}_{\alpha}} = \{\pmb {p}\in \Delta^S ||\pmb {p} - \tilde{\pmb{p}}_{s_0,a_0}||_1\leq 0.277\}$ . In this case, the reward estimate against the worst model in the ambiguity set  $\pmb {p} = [0.50,0.32,0.18]$  is  $\rho^{\mathrm{BCR}_{\alpha}} = 0.025$ . Since we have a single non-terminating state in the MDP, the percentile returns are given by  $\rho^{\mathrm{VaR}_{\alpha}} = \mathrm{VaR}_{0.2}[\tilde{\pmb{p}}_{s_0,a_0}^{\top}\pmb{r}_{s_0,a_0}]$ . Computing  $\rho^{\mathrm{VaR}_{\alpha}}$  for Dirichlet distribution  $Dir(10,10,1)$ , we get  $\rho^{\mathrm{VaR}_{\alpha}} = 0.17 > \rho^{\mathrm{BCR}_{\alpha}}$ . Thus, this example shows that BCR ambiguity sets can be unnecessarily large, and thereby result in conservative policies.
3 VaR Framework
We introduce the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  for approximately solving the percentile criterion. We show that  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  is a valid Bellman operator: it is a contraction mapping and lower bounds the percentile criterion. For any value function  $\pmb{v} \in \mathbb{R}^S$ , state  $s \in S$  and action  $a \in \mathcal{A}$ , we define the VaR Bellman optimality operator  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  as
$$
\left(\mathcal {T} _ {\mathrm {V a R} _ {\alpha}} \boldsymbol {v}\right) _ {s} = \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} \left[ \tilde {\boldsymbol {p}} _ {s, a} ^ {\top} \left(\boldsymbol {r} _ {s, a} + \gamma \boldsymbol {v}\right) \right]. \tag {5}
$$
For each state  $s$ ,  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  maximizes the value corresponding to the worst  $\alpha$ -percentile model. In contrast to the  $\mathrm{BCR}_{\alpha}$  Bellman optimality operator  $\mathcal{T}_{\mathrm{BCR}_{\alpha}}$ , computing  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  does not require constructing ambiguity sets from confidence regions; it can simply be estimated from samples of the model posterior distribution, as we later show.
Proposition 3.1 (Validity). The following properties of  $\mathcal{T}_{\mathrm{VaR}_\alpha}$  hold for all value functions  $\pmb{u},\pmb{v} \in \mathbb{R}^S$ .
1. The operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is contraction mapping on  $\mathbb{R}^S$ :  $\| \mathcal{T}_{\mathrm{VaR}_{\alpha}}\pmb {u} - \mathcal{T}_{\mathrm{VaR}_{\alpha}}\pmb {v}\|_{\infty}\leq \gamma \| \pmb {u} - \pmb {v}\|_{\infty}$ .
2. The operator  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is monotone:  $\pmb{u} \succeq \pmb{v} \Rightarrow \mathcal{T}_{\mathrm{VaR}_{\alpha}} \pmb{u} \succeq \mathcal{T}_{\mathrm{VaR}_{\alpha}} \pmb{v}$ .
3. The equality  $\mathcal{T}_{\mathrm{VaR}_\alpha}\hat{\boldsymbol{v}} = \hat{\boldsymbol{v}}$  has a unique solution.
Proposition 3.1 formally proves that  $\mathcal{T}_{\mathrm{VaR}_{\alpha}}$  is a valid Bellman operator, i.e., it is a contraction mapping, monotone, and has a unique fixed point  $\hat{\pmb{v}} = (\mathcal{T}_{\mathrm{VaR}_{\alpha}}\hat{\pmb{v}})$ . Here on, we will refer to the policy  $(\hat{\pi})$  corresponding to the fixed point value  $\hat{\pmb{v}}$  as the  $\mathrm{VaR}_{\alpha}$  policy.
(a) Asymptotic Radii