SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, 2005.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse Bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems, 34:1142-1154, 2021.
A. Shapiro, D. Dentcheva, and A. Ruszczyński. Lectures on Stochastic Programming: Modeling and Theory. SIAM, 2014.
Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic programming. Operations Research Letters, 45(4):377-381, 2017.
Vishnu D Sharma, Maymoonah Toubeh, Lifeng Zhou, and Pratap Tokekar. Risk-aware planning and assignment for ground vehicles using uncertain perception from aerial vehicles. In International Conference on Intelligent Robots and Systems (IROS), pages 11763-11769. IEEE, 2020.
Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A CVaR optimization approach. In International Joint Conference on Computational Intelligence, pages 412-423, 2019.
Congbin Wu and Yuanlie Lin. Minimizing risk models in Markov decision processes with policies depending on target values. Journal of mathematical analysis and applications, 231(1):47-67, 1999.
Huan Xu and Shie Mannor. Probabilistic goal Markov decision processes. In International Joint Conference on Artificial Intelligence, 2011.
Huaiyang Zhong. Decision Making for Disease Treatment: Operations Research and Data Analytic Modeling. Stanford University, 2020.
Figure 3: Rewards of the MDP  $M_{\mathrm{E}}$  used in the proof of Theorem 4.1. The dot indicates that the rewards are independent of the next state.
Suppose that  $\alpha > 0$ ; the decomposition for  $\alpha = 0$  holds readily because  $\mathrm{CVaR}_0[\hat{x}] = \mathrm{essinf}[\hat{x}]$ .
To streamline the notation, Define a random variable  $\tilde{x} = r(\tilde{s},\tilde{a},\tilde{s}^{\prime})$  over a random space  $\Omega = S\times \mathcal{A}\times S$  with a probability distribution  $q\in \Delta_{m}$  such that  $q_{s,a,s'} = \hat{p}_s\cdot \pi (s,a)\cdot p(s,a,s')$ . The value  $\pmb{x}$  is the vector representation of the random variable  $\tilde{x}$  and  $\pmb{\xi}_{s} = \pmb{\xi}_{s,\cdot ,\cdot}\in \mathbb{R}^{S\cdot A}$  for  $\pmb {\xi}\in \mathbb{R}^{m}$  is a vector that corresponds to the subset of the elements of  $\Omega$  in which the first element is some  $s\in S$ . The vectors  $\pmb {x}_s = \pmb {x}_{s,\cdot ,\cdot}\in \mathbb{R}^{S\cdot A}$  and  $\pmb {q}_s = \pmb {q}_{s,\cdot ,\cdot}\in \mathbb{R}^{S\cdot A}$  are defined analogously to  $\pmb{\xi}_{s}$ .
Starting with the CVaR definition in (2) and introducing an auxiliary variable  $\zeta$  we get that
$$
\begin{array}{l} \operatorname {C V a R} _ {\alpha} [ \tilde {x} ] = \min  _ {\boldsymbol {\xi} \in \Delta_ {m}} \left\{\boldsymbol {x} ^ {\top} \boldsymbol {\xi} \mid \alpha \boldsymbol {\xi} \leq \boldsymbol {q} \right\} = \min  _ {\boldsymbol {\xi} \in \Delta_ {m}, \boldsymbol {\zeta} \in \mathbb {R} ^ {S}} \left\{\boldsymbol {x} ^ {\top} \boldsymbol {\xi} \mid \alpha \boldsymbol {\xi} \leq \boldsymbol {q}, \zeta_ {s} = \boldsymbol {1} ^ {\top} \boldsymbol {\xi} _ {s}, \forall s \in S \right\} \\ = \min  _ {\boldsymbol {\xi} \in \Delta_ {m}, \boldsymbol {\zeta} \in \Delta_ {S}} \left\{\boldsymbol {x} ^ {\top} \boldsymbol {\xi} \mid \alpha \boldsymbol {\xi} \leq \boldsymbol {q}, \zeta_ {s} = \mathbf {1} ^ {\top} \boldsymbol {\xi} _ {s}, \alpha \zeta_ {s} \leq \hat {p} _ {s}, \forall s \in S \right\} \\ = \min _ {\boldsymbol {\xi} \in \mathbb {R} _ {+} ^ {\Omega}, \boldsymbol {\zeta} \in \mathcal {Z} _ {\mathrm {C}}} \left\{\boldsymbol {x} ^ {\top} \boldsymbol {\xi} \mid \alpha \boldsymbol {\xi} \leq \boldsymbol {q}, \zeta_ {s} = \mathbf {1} ^ {\top} \boldsymbol {\xi} _ {s}, \forall s \in \mathcal {S} \right\}. \\ \end{array}
$$
In the derivation above, we replaced the infimum by a minimum because  $\Omega$  is finite, introduced a new variable  $\zeta$ , derived implied constraints on  $\zeta$ , and then dropped superfluous constraints on  $\xi$ . Continuing with the derivation above and noticing that the constraints on each  $\xi_s$  are independent given  $\zeta$ , we get that
$$
\begin{array}{l} \mathrm {C V a R} _ {\alpha} [ \tilde {x} ] = \min  _ {\boldsymbol {\xi} \in \mathbb {R} _ {+} ^ {\Omega}, \boldsymbol {\zeta} \in \mathcal {Z} _ {\mathbb {C}}} \left\{\sum_ {s \in \mathcal {S}} \boldsymbol {x} _ {s} ^ {\top} \boldsymbol {\xi} _ {s} \mid \alpha \boldsymbol {\xi} \leq \boldsymbol {q}, \zeta_ {s} = \boldsymbol {1} ^ {\top} \boldsymbol {\xi} _ {s}, \forall s \in \mathcal {S} \right\} \\ \stackrel {\mathrm {(a)}} {=} \min  _ {\boldsymbol {\zeta} \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \inf  _ {\boldsymbol {\xi} _ {s} \in \mathbb {R} _ {+} ^ {\Omega_ {s}}} \left\{\boldsymbol {x} _ {s} ^ {\top} \boldsymbol {\xi} _ {s} \mid \alpha \boldsymbol {\xi} _ {s} \leq \boldsymbol {q} _ {s}, \zeta_ {s} = \boldsymbol {1} ^ {\top} \boldsymbol {\xi} _ {s} \right\} \\ \stackrel {(b)} {=} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \cdot \min  _ {\chi \in \Delta_ {S \cdot A}} \left\{\boldsymbol {x} _ {s} ^ {\top} \boldsymbol {\chi} \mid \alpha \hat {p} _ {s} ^ {- 1} \zeta_ {s} \chi_ {a, s ^ {\prime}} \leq \hat {p} _ {s} ^ {- 1} q _ {s, a, s ^ {\prime}}, \forall a \in \mathcal {A}, s ^ {\prime} \in \mathcal {S} \right\} \\ \stackrel {\mathrm {(c)}} {=} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \cdot \operatorname {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} [ \tilde {x} \mid \tilde {s} = s ]. \\ \end{array}
$$
The step (a) follows from the interchangeability principle (Shapiro et al., 2014, theorem 7.92), and the step (b) follows by substituting  $\xi_{s,a,s'} = \zeta_s\chi_{a,s'}$  taking care when  $\zeta_s = 0$  and multiplying both sides of the inequality by  $\hat{p}_s^{-1} > 0$ . Finally, in step (c), the random variable  $\tilde{x} = r(\tilde{s},\tilde{a},\tilde{s}')$  conditional on  $\tilde{s} = s$  is distributed according to  $q_{s,a,s'}\hat{p}_s^{-1}$  and the equality follows from the definition of CVaR in (2).
A.2 Proof of Theorem Theorem 4.1
Consider an MDP  $M_{\mathrm{E}}$  depicted in Figure 3 with  $\mathcal{S} = \{s_1,s_2\}$  and  $\mathcal{A} = \{a_1\}$  and a reward function  $r(s_{1},a_{1},\cdot) = 1$  and  $r(s_{2},a_{1},\cdot) = 0$ . We abbreviate the rewards to  $r(s_{1})$  and  $r(s_{2})$  because they only depend on the originating state. The initial distribution is  $\hat{p}_{s_1} = \hat{p}_{s_2} = 0.5$ . We finally let  $\alpha = 0.75$ .
Because  $\mathcal{Z}_{\mathrm{E}} \subseteq \mathcal{Z}_{\mathrm{C}}$ , the right-hand side of (13) can be lower-bounded by CVaR as
$$
\begin{array}{l} \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} [ r (s, a _ {1}, \tilde {s} ^ {\prime}) ] = \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} r (s) \tag {19} \\ \geq \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \xi_ {s} r (s) = \operatorname {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right]. \\ \end{array}
$$
The first equality holds from the positive homogeneity and cash invariance properties of EVaR, and the last equality follows from the dual representation of CVaR (Follmer and Schied, 2016).
Because  $\mathrm{EVaR}_{\alpha}[\tilde{x}]\leq \mathrm{CVaR}_{\alpha}[\tilde{x}]$  for each  $\alpha \in [0,1]$  and  $\tilde{x}\in \mathbb{X}$  (see (Ahmadi-Javid, 2012, proposition 3.2)), we can further lower-bound (19) as
$$
\operatorname {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] \leq \operatorname {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] \leq \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} \left[ r \left(s, a _ {1}, \tilde {s} ^ {\prime}\right) \right]. \tag {20}
$$
Therefore, (13) holds with an inequality.
To prove by contradiction that the inequality in (13) is strict, suppose that
$$
\operatorname {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] = \min  _ {\xi \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} \left[ r \left(s, a _ {1}, \tilde {s} ^ {\prime}\right) \right]. \tag {21}
$$
Equalities (21) and (20) imply that  $\mathrm{EVA}\mathrm{R}_{\alpha}[r(\tilde{s},a_1,\tilde{s} ')] = \mathrm{CVA}\mathrm{R}_{\alpha}[r(\tilde{s},a_1,\tilde{s} ')]$  which is false in general (Ahmadi-Javid, 2012).
We now show that EVaR does not equal CVaR even for the categorical distribution of  $\tilde{s}$ . The CVaR of the return in  $M_{\mathrm{E}}$  reduces from (2) to
$$
\mathrm {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] = \min  _ {\xi \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \xi_ {s} r (s) = \max  \left\{0, \frac {\hat {p} _ {s _ {1}} + \alpha - 1}{\alpha} \right\}. \tag {22}
$$
Since  $1 - \alpha = 0.25 < 0.5 = \hat{p}_{s_1}$ , then the optimal  $\xi^{\star}$  in (22) is
$$
\boldsymbol {\xi} ^ {\star} = \left( \begin{array}{c} \frac {\hat {p} _ {s _ {1}} + \alpha - 1}{\alpha} \\ \frac {1 - \hat {p} _ {s _ {1}}}{\alpha} \end{array} \right).
$$
Since  $\mathrm{KL}(\pmb{\xi}^{\star}\| \hat{\pmb{p}}) < -\log \alpha$ , we have that  $\pmb{\xi}^{\star}$  is in the relative interior of the EVaR feasible region in (3), and, therefore, there exists an  $\epsilon > 0$  such that
$$
\mathrm {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] = \mathrm {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] - \epsilon <   \mathrm {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right],
$$
which proves the desired inequality.
A.3 Proof of Corollary 4.2
We start by proposing a new decomposition for EVaR.
Proposition A.1. Given a random variable  $\tilde{x} \in \mathbb{X}$  and a discrete variable  $\tilde{y} \colon \Omega \to \mathcal{N} = \{1, \dots, N\}$ , with probabilities denoted as  $\{\hat{p}_i\}_{i=1}^N$ , for any  $\alpha \in (0, 1]$  we have that
$$
\mathrm {E V a R} _ {\alpha} [ \tilde {x} ] = \inf  _ {\boldsymbol {\zeta} \in (0, 1 ] ^ {N}} \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}} ^ {\prime} (\boldsymbol {\zeta})} \sum_ {i} \xi_ {i} \mathrm {E V a R} _ {\zeta_ {i}} [ \tilde {x} | \tilde {y} = i ],
$$
$$
\mathcal {Z} _ {\mathrm {E}} ^ {\prime} (\boldsymbol {\zeta}) = \left\{\boldsymbol {\xi} \in \Delta_ {N} \mid \boldsymbol {\xi} \ll \hat {\boldsymbol {p}}, \sum_ {i = 1} ^ {N} \xi_ {i} \left(\log \left(\xi_ {i} / \hat {p} _ {i}\right) - \log \left(\zeta_ {i}\right)\right) \leq - \log \alpha \right\}.
$$
Proof. Let  $\mathbf{q}$  denote the joint probability distribution of  $\tilde{x}$  and  $\tilde{y}$ . The proof exploits the chain rule of relative entropy (e.g., Cover and Thomas (2006, theorem 2.5.3)), which states that for any probability distributions  $\eta, \mathbf{q} \in \Delta_{\Omega}$  with  $\eta \ll \mathbf{q}$  that
$$
\operatorname {K L} (\boldsymbol {\eta} \| \boldsymbol {q}) = \operatorname {K L} (\boldsymbol {\eta} (\tilde {y}) \| \boldsymbol {q} (\tilde {y})) + \operatorname {K L} (\boldsymbol {\eta} (\tilde {x} | \tilde {y}) \| \boldsymbol {q} (\tilde {x} | \tilde {y})), \tag {23}
$$
where the conditional relative entropy is defined as
$$
\operatorname {K L} (\boldsymbol {\eta} (\tilde {x} | \tilde {y}) \| \boldsymbol {q} (\tilde {x} | \tilde {y})) = \mathbb {E} ^ {\boldsymbol {\eta}} \left[ \log \frac {\boldsymbol {\eta} (\tilde {x} | \tilde {y})}{\boldsymbol {q} (\tilde {x} | \tilde {y})} \right].
$$
with  $\mathbb{E}^{\eta}[f(\tilde{x},\tilde{y})]$  as a shorthand notation to indicate that  $(\tilde{x},\tilde{y})\sim \eta$ . We can now decompose EVaR from its definition in (3) as
$$
\begin{array}{l} \operatorname {E V a R} _ {\alpha} [ \tilde {x} ] = \inf  _ {\boldsymbol {\eta} \in \Delta_ {m, N}: \boldsymbol {\eta} \ll \boldsymbol {q}} \left\{\mathbb {E} ^ {\boldsymbol {\eta}} [ \tilde {x} ] \mid \operatorname {K L} (\boldsymbol {\eta} \mid \boldsymbol {q}) \leq - \log \alpha \right\} \\ \stackrel {\text {(a)}} {=} \inf  _ {\boldsymbol {\eta} \in \Delta_ {m \cdot N}: \boldsymbol {\eta} \ll \boldsymbol {q}} \left\{\mathbb {E} ^ {\boldsymbol {\eta}} [ \tilde {x} ] \mid \operatorname {K L} (\boldsymbol {\eta} (\tilde {y}) \| \boldsymbol {q} (\tilde {y})) + \operatorname {K L} (\boldsymbol {\eta} (\tilde {x} | \tilde {y}) \| \boldsymbol {q} (\tilde {x} | \tilde {y})) \leq - \log \alpha \right\} \\ = \inf  _ {\boldsymbol {\eta} \in \Delta_ {m \cdot N}: \boldsymbol {\eta} \ll \boldsymbol {q}} \left\{\mathbb {E} ^ {\boldsymbol {\eta}} [ \tilde {x} ] \mid \operatorname {K L} (\boldsymbol {\eta} (\tilde {y}) \| \boldsymbol {q} (\tilde {y})) + \mathbb {E} ^ {\boldsymbol {\eta}} \left[ \mathbb {E} ^ {\boldsymbol {\eta}} \left[ \log \frac {\boldsymbol {\eta} (\tilde {x} | \tilde {y})}{\boldsymbol {q} (\tilde {x} | \tilde {y})} \right] \mid \tilde {y} \right] \leq - \log \alpha \right\} \\ \stackrel {\text {(b)}} {=} \inf  _ {\boldsymbol {\eta} \in \Delta_ {m \cdot N}, \boldsymbol {\zeta} \in (0, 1 ] ^ {N}: \boldsymbol {\eta} \ll \boldsymbol {q}} \left\{\mathbb {E} ^ {\boldsymbol {\eta}} [ \mathbb {E} ^ {\boldsymbol {\eta}} [ \tilde {x} \mid \tilde {y} ] ] \mid \begin{array}{l} \operatorname {K L} (\boldsymbol {\eta} (\tilde {y}) \| \boldsymbol {q} (\tilde {y})) + \mathbb {E} ^ {\boldsymbol {\eta}} [ - \log (\zeta_ {\tilde {y}}) ] \leq - \log \alpha \\ \mathbb {P} _ {\boldsymbol {\eta}} [ \mathbb {E} ^ {\boldsymbol {\eta}} [ \log (\boldsymbol {\eta} (\tilde {x} | \tilde {y}) / \boldsymbol {q} (\tilde {x} | \tilde {y})) \mid \tilde {y} ] \leq - \log (\zeta_ {\tilde {y}}) ] = 1 \end{array} \right\} \\ \stackrel {\mathrm {(c)}} {=} \inf  _ {\boldsymbol {\xi} \in \Delta_ {N}, \boldsymbol {\zeta} \in (0, 1 ] ^ {N}: \boldsymbol {\xi} \ll \hat {\boldsymbol {p}}} \left\{\mathbb {E} ^ {\boldsymbol {\xi}} \left[ \operatorname {E V a R} _ {\zeta_ {\tilde {y}}} [ \tilde {x} | \tilde {y} ] \right] \mid \operatorname {K L} (\boldsymbol {\xi} \| \hat {\boldsymbol {p}}) + \mathbb {E} ^ {\boldsymbol {\xi}} \left[ - \log \left(\zeta_ {\tilde {y}}\right) \right] \leq - \log \alpha \right\} \\ = \inf  _ {\boldsymbol {\xi} \in \Delta_ {N}, \boldsymbol {\zeta} \in (0, 1 ] ^ {N}: \boldsymbol {\xi} \ll \hat {\boldsymbol {p}}} \left\{\sum_ {i} \xi_ {i} \operatorname {E V a R} _ {\zeta_ {i}} [ \tilde {x} \mid \tilde {y} = i ] \mid \sum_ {i = 1} ^ {N} \xi_ {i} \log (\xi_ {i} / \hat {p} _ {i}) - \sum_ {i = 1} ^ {N} \xi_ {i} \log (\zeta_ {i}) \leq - \log \alpha \right\}. \\ \end{array}
$$
Here, we decompose the relative entropy of  $\eta$  and  $q$  using (23) in step (a) and then use the tower property of the expectation operator in the next step. In step (b), we introduce a variable  $\zeta_{i}$  for each realization of  $\tilde{y} = i$  with  $i\in \mathcal{N}$  to decouple the influence of  $\eta (\tilde{x} |\tilde{y})$ , under each  $\tilde{y}$ , in the inequality constraint. Finally, we replace the conditional EVaR definition by solving for  $\eta (\tilde{x} |\tilde{y})$  for a given  $\zeta$  in step (c), and representing  $\eta (\tilde{y})$  using  $\xi$ .
The first part of our theorem follows directly from Proposition A.1. Suppose that  $\alpha > 0$ ; the result follows for  $\alpha = 0$  because  $\mathrm{EVaR}_0[\cdot]$  reduces to ess inf. Then, the second part of the corollary holds as
$$
\begin{array}{l} \operatorname {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}\right) \right] = \\ = \inf  _ {\zeta \in (0, 1 ] ^ {N}, \xi \in \Delta_ {N}} \left\{\sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\zeta_ {s}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right] \mid \sum_ {s \in \mathcal {S}} \xi_ {s} \log \frac {\xi_ {s}}{\zeta_ {s} \hat {p} _ {s}} \leq - \log \alpha \right\} \\ \leq \inf  _ {\boldsymbol {\zeta} \in (0, 1 ] ^ {N}, \boldsymbol {\xi} \in \Delta_ {N}} \left\{\sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\zeta_ {s}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ] \mid \sum_ {s \in \mathcal {S}} \xi_ {s} \log \frac {\xi_ {s}}{\zeta_ {s} \hat {p} _ {s}} \leq - \log \alpha , \boldsymbol {\xi} \leq \alpha^ {- 1} \hat {\boldsymbol {p}} \right\} \\ \leq \inf  _ {\boldsymbol {\xi} \in \Delta_ {N}} \left\{\sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right] \mid \boldsymbol {\xi} \leq \alpha^ {- 1} \hat {\boldsymbol {p}} \right\}. \\ \end{array}
$$
The first inequality follows from adding a constraint on the pairs on the  $\xi$  considered by the infimum. The second inequality follows by fixing  $\zeta_s = \hat{\zeta}_s$  with  $\hat{\zeta}_s = \alpha \xi_s \hat{p}_s^{-1}$  for each  $s \in S$ . This is an upper bound because  $\hat{\zeta}_s$  is feasible in the infimum:
$$
\sum_ {s \in \mathcal {S}} \xi_ {s} \log \frac {\xi_ {s}}{\hat {\zeta} _ {s} \hat {p} _ {s}} = - \log \alpha \leq - \log \alpha .
$$
The value  $\hat{\zeta}_s$  is well-defined since  $\hat{p}_s > 0$  and the constraint  $\xi \leq \alpha^{-1}\hat{p}$  ensures that  $\hat{\zeta}_s \leq 1$ . Also, we can relax the constraint  $\zeta_s > 0 \Rightarrow \xi_s > 0$  to  $\xi_s \geq 0$  because  $\mathrm{EVaR}_0[\tilde{x}] = \lim_{\alpha \to 0} \mathrm{EVaR}_{\alpha}[\tilde{x}]$ , and, therefore, the infimum is not affected. Finally, the inequality in the corollary follows immediately by further upper bounding the decomposition above by adding a constraint.
A.4 Proof of Theorem 5.2
The equality develops from Theorem 5.1 as
$$
\begin{array}{l} \max  _ {\pi \in \Pi} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] = \max  _ {\pi \in \Pi} \sup  _ {\zeta \in \Delta_ {\mathcal {S}}: \alpha \cdot \zeta \leq \hat {p}} \min  _ {s \in \mathcal {S}} \left(\operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \pi (s)} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ]\right) \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}: \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}}} \max  _ {\pi \in \Pi} \min  _ {s \in S} \left(\operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \pi (s)} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ]\right) \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}: \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}}} \min  _ {s \in \mathcal {S}} \left(\max  _ {\boldsymbol {d} \in \Delta_ {A}} \operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ]\right), \\ \end{array}
$$
where we first change the order of maximum and supremum, followed by changing the order of  $\max_{\pi}\min_s$  with  $\min_s\max_{\pi}$ . The latter is a direct consequence of the interchangeability property of the maximum operation (Shapiro, 2017, proposition 2.2).
A.5 Proof of Theorem 5.3
The proof mainly relies on correcting the decomposition of lower quantile proposed in (Li et al., 2022).
Proposition A.2. Given an  $\tilde{x} \in \mathbb{X}$ , suppose that a random variable  $\tilde{y} \colon \Omega \to \mathcal{N} = \{1, \dots, N\}$  is distributed as  $\hat{\boldsymbol{p}} = (\hat{p}_i)_{i=1}^N$  with  $\hat{p}_i > 0$ . Then:
$$
Q _ {\alpha} (\tilde {x}) = \sup  _ {\zeta \in [ 0, 1 ] ^ {N}} \left\{\min  _ {i \in \mathcal {N}: \zeta_ {i} <   1} Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i) \mid \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} <   \alpha \right\}, \tag {24}
$$
where we interpret the supremum to be minus infinity if its feasible set is empty, which only occurs if  $\alpha = 0$ .
Proof. First, we decompose the lower quantile using its definition as
$$
\begin{array}{l} Q _ {\alpha} (\tilde {x}) = \sup  \left\{z \mid \mathbb {P} [ \tilde {x} <   z ] <   \alpha \right\} \overset {(\mathrm {a})} {\underset {z \in \mathbb {R}} {=}} \left\{z \mid \sum_ {i = 1} ^ {N} \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] \hat {p} _ {i} <   \alpha \right\} \\ \stackrel {\text {(b)}} {=} \sup  _ {z \in \mathbb {R}, \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}} \left\{z \mid \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} <   \alpha , \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] <   \zeta_ {i}, \forall i \in \mathcal {N}: \zeta_ {i} <   1 \right\} \\ \stackrel {\mathrm {(c)}} {=} \sup  _ {z \in \mathbb {R}, \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}} \left\{z \mid z <   Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i), \forall i \in \mathcal {N}: \zeta_ {i} <   1, \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} <   \alpha \right\} \\ \stackrel {\text {(d)}} {=} \sup  _ {\zeta \in [ 0, 1 ] ^ {N}} \left\{\sup  _ {z \in \mathbb {R}} \left\{z \mid z <   Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i), \forall i \in \mathcal {N}: \zeta_ {i} <   1 \right\} \mid \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} <   \alpha \right\} \\ \stackrel {\text {(e)}} {=} \sup  _ {\zeta \in [ 0, 1 ] ^ {N}} \left\{\min  _ {i \in \mathcal {N}: \zeta_ {i} <   1} Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i) \mid \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} <   \alpha \right\}. \\ \end{array}
$$
We decompose the probability  $\mathbb{P}[\tilde{x} < z]$  in terms of the conditional probabilities  $\mathbb{P}[\tilde{x} < z\mid \tilde{y} = i]$  in step (a) and then lower-bound them by an auxiliary variable  $\zeta_{i}$  in step (b). In step (c), we exploit the following equivalence:
$$
\mathbb {P} \left[ \tilde {x} <   z \mid \tilde {y} = i \right] <   \zeta_ {i} \quad \Leftrightarrow \quad z <   Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i)
$$
The direction  $\Leftarrow$  in the equivalence follows from the definition of  $Q_{\zeta_i}(\tilde{x}\mid \tilde{y} = i)$
$$
z <   Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i) = \inf  \left\{z \mid \mathbb {P} [ \tilde {x} \leq z \mid \tilde {y} = i ] \geq \zeta_ {i} \right\} \Rightarrow \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] <   \zeta_ {i}.
$$
The direction  $\Rightarrow$  follows from the definition of VaR (see equation (1)), which implies that VaR upper-bounds any  $z$  that satisfies the left-hand condition:
$$
\mathbb {P} \left[ \tilde {x} <   z \mid \tilde {y} = i \right] <   \zeta_ {i} \Rightarrow Q _ {\zeta_ {i}} (\tilde {x} \mid \tilde {y} = i) = \sup  \left\{z \in \mathbb {R} \mid \mathbb {P} \left[ \tilde {x} <   z \mid \tilde {y} = i \right] <   \zeta_ {i} \right\} \geq z,
$$
yet  $Q_{\zeta_i}(\tilde{x} \mid \tilde{y} = i) \neq z$  otherwise since  $\mathbb{P}[\tilde{x} < z \mid \tilde{y} = i]$  is right continuous, there must exist some  $\epsilon > 0$  for which  $\mathbb{P}[\tilde{x} < z + \epsilon \mid \tilde{y} = i] < \zeta_i$  hence:
$$
z = \sup  \left\{z \in \mathbb {R} \mid \mathbb {P} \left[ \tilde {x} <   z \mid \tilde {y} = i \right] <   \zeta_ {i} \right\} \geq z + \epsilon > z,
$$
which leads to a contradiction. In step (e), we solve for  $z$ . Finally, we obtain the form in (24).
Figure 4: An example used to show the sub-optimality of a policy in Appendix B.
Figure 5: Return computed in Chow et al. (2015) vs optimal CVaR policy for the example in Figure 4 with  $p_{s_1} = p_{s_2} = 0.5$  and  $M = 600$ . The optimal CVaR policy for each  $\alpha$  is denoted by  $\pi^{\star}$ . The value function  $\hat{v}$  is computed according to the decomposition in the r.h.s. of (8) and the corresponding policy is  $\hat{\pi}$ .
B CVaR Suboptimal Policy