SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
r \left(s _ {1}, a _ {2}, s _ {1}\right) = r \left(s _ {1}, a _ {2}, s _ {2}\right) = 0, \quad r \left(s _ {2}, a _ {1}, s _ {1}\right) = r \left(s _ {2}, a _ {1}, s _ {2}\right) = 1 0.
$$
The transition probabilities in  $M_{\mathrm{C}}$  are
$$
p \left(s _ {1}, a _ {1}, s _ {1}\right) = 0. 4, \quad p \left(s _ {1}, a _ {1}, s _ {2}\right) = 0. 6,
$$
and the initial distribution is uniform:  $\hat{p}_{s_1} = \hat{p}_{s_2} = 0.5$
To simplify the notation, we define  $\theta_{\pi} \colon \mathcal{Z}_{\mathrm{C}} \to \mathbb{R}$  for each  $\pi \in \Pi$  and  $\zeta \in \mathcal{Z}_{\mathrm{C}}$  as
$$
\theta_ {\pi} (\zeta) = \sum_ {s \in \mathcal {S}} \zeta_ {s} \operatorname {C V a R} _ {\alpha \zeta_ {s} \tilde {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {\pi} (s)} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ].
$$
Because CVaR is convex in the distribution (Delage et al., 2019) and any distribution for  $r(\tilde{s},\tilde{a},\tilde{s}^{\prime})$  obtained from a policy  $\pi \in \Pi$  is a mixture of the distributions of  $r(\tilde{s},a_1,\tilde{s}^{\prime})$  and  $r(\tilde{s},a_2,\tilde{s}^{\prime})$ , it is sufficient to consider only deterministic policies (there exists an optimal deterministic policy). Thus, we can reformulate the left-hand side of (9) in terms of  $\theta_{\pi}(\zeta)$  as
$$
\begin{array}{l} \max  _ {\boldsymbol {\pi} \in \Pi} \operatorname {C V a R} _ {\alpha} ^ {\tilde {a} \sim \boldsymbol {\pi} (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] = \max  _ {\pi \in \{\pi_ {1}, \pi_ {2} \}} \operatorname {C V a R} _ {\alpha} ^ {\tilde {a} \sim \boldsymbol {\pi} (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] \\ = \max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \cdot \operatorname {C V a R} _ {\alpha \zeta_ {s} \tilde {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \pi (s)} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ] \\ = \max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \theta_ {\pi} (\zeta), \\ \end{array}
$$
with  $\pi_1(s,a_1) = 1 - \pi_1(s,a_2) = 1$  and  $\pi_2(s,a_2) = 1 - \pi_2(s,a_1) = 1$ , for all  $s \in S$ . The functions  $\theta_{\pi_1}(\cdot)$  and  $\theta_{\pi_2}(\cdot)$  are depicted in Figure 2. Similarly, the right-hand side of (9) can be expressed using the convexity of CVaR in the distribution by algebraic manipulation as
$$
\min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \max  _ {\boldsymbol {d} \in \Delta_ {A}} \operatorname {C V a R} _ {\alpha \zeta_ {s} \tilde {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ] = \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \max  _ {\pi \in \{\pi_ {1}, \pi_ {2} \}} \theta_ {\pi} (\zeta).
$$
Using the notation introduced above and the sufficiency of optimizing over deterministic policies only, the inequality in (9) becomes
$$
\max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \theta_ {\pi} (\zeta) <   \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \theta_ {\pi} (\zeta). \tag {10}
$$
Figure 2 demonstrates the inequality in (10) numerically, with the rectangle representing the left-hand side maximum and the pentagon representing the right-hand side minimum. The dashed line represents the function  $\zeta \mapsto \max_{\pi \in \{\pi_1, \pi_2\}} \theta_\pi(\zeta)$ .
To show the strict inequality in (10) formally, we evaluate the functions  $\theta_{\pi_1}(\cdot)$  and  $\theta_{\pi_2}(\cdot)$  for MDP  $M_{\mathrm{C}}$ . The function  $\theta_{\pi_2}(\cdot)$  is linear because the CVaR applies to a constant, and CVaR is translation invariant. The function  $\theta_{\pi_1}(\cdot)$  is piecewise-linear and convex, and its slope can be computed using the subgradient that for each  $s \in S$  and  $\hat{\zeta} \in \mathcal{Z}_{\mathrm{C}}$  satisfies (Chow et al., 2015)
$$
\partial_ {\zeta_ {s}} \hat {\zeta} _ {s} \operatorname {C V a R} _ {\alpha \hat {p} _ {s} ^ {- 1} \hat {\zeta} _ {s}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right] \ni \operatorname {V a R} _ {\alpha \hat {p} _ {s} ^ {- 1} \hat {\zeta} _ {s}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right].
$$
Simple algebraic manipulation then shows that
$$
\theta_ {\pi_ {1}} (\zeta) = \max  \left\{1 0 - 6 0 \zeta_ {s _ {1}}, 9 0 \zeta_ {s _ {1}} - 5 0 \right\}, \quad \theta_ {\pi_ {2}} (\zeta) = 1 0 - 1 0 \zeta_ {s _ {1}},
$$
and  $\mathcal{Z}_{\mathrm{C}} = \Delta_{S}$ , which implies that  $\zeta_{s_1} \in (0, 1)$ . Therefore, by algebraic manipulation, we get the desired strict inequality
$$
0 = \max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \theta_ {\pi} (\zeta) <   \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \max  _ {\pi \in \left\{\pi_ {1}, \pi_ {2} \right\}} \theta_ {\pi} (\zeta) = 4,
$$
where 0 and 4 are represented by the pentagon and rectangle in Figure 2, respectively.
In summary, the decomposition in Proposition 3.1 cannot be exploited in policy optimization because the inequality in the derivation above may not be tight:
$$
\begin{array}{l} \max  _ {\pi \in \Pi} \mathrm {C V a R} _ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ] = \max  _ {\pi \in \Pi} \min  _ {\zeta \in \mathcal {Z} _ {\mathbb {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \mathrm {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {\tilde {s}} ^ {- 1}} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ] \\ \leq \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \max  _ {\pi \in \Pi} \sum_ {s \in \mathcal {S}} \zeta_ {s} \operatorname {C V a R} _ {\alpha \zeta_ {s} \tilde {p} _ {\tilde {s}} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {\pi} (\tilde {s})} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ] \\ = \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \max  _ {\boldsymbol {d} \in \Delta_ {A}} \operatorname {C V a R} _ {\alpha \zeta_ {s} \tilde {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right], \\ \end{array}
$$
where the last equality follows from the interchangeability property of optimization and expected value (Shapiro et al., 2014, theorem 7.92).
Finally, we omit to comment on the validity of the CVaR decomposition in Li et al. (2022) given that it considers a different measure than the CVaR defined in Equation (1). Namely, their measure takes the form:
$$
\widetilde {\mathrm {C V a R}} _ {\alpha} [ \tilde {x} ] = \inf  _ {z \in \mathbb {R}} \left(z + (1 - \alpha) ^ {- 1} \mathbb {E} [ \tilde {x} - z ] _ {+}\right) = - \mathrm {C V a R} _ {1 - \alpha} [ - \tilde {x} ],
$$
which is not a coherent risk measure.
4 EVaR: Decomposition Fails for Policy Evaluation
In this section, we show that a decomposition for EVaR proposed in Ni and Lai (2022) is inexact even when considering the policy evaluation setting. Ni and Lai (2022) recently proposed a decomposition of EVaR for a fixed  $\pi \in \Pi$  with  $\tilde{a} \sim \pi(\tilde{s})$  and a risk level  $\alpha \in (0,1]$  as
$$
\mathrm {E V a R} _ {\alpha} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] \stackrel {{\ref {e q : 1}}} {{=}} \min  _ {\xi \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \mathrm {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ], \tag {11}
$$
where  $\tilde{s}\sim \hat{p},\tilde{s}^{\prime}\sim p(\tilde{s},\tilde{a},\cdot)$  , and
$$
\text {i m p l i c i t} \mathrm {N i} \text {a n d} \mathrm {L a i} (2 0 2 2)
$$
$$
\mathcal {Z} _ {\mathrm {E}} = \left\{\boldsymbol {\xi} \in \Delta_ {\mathcal {S}} \mid \sum_ {s \in \mathcal {S}} \xi_ {s} \log \left(\xi_ {s} / \hat {p} _ {s}\right) \leq - \log \alpha , \quad \overbrace {\alpha \cdot \boldsymbol {\xi} \leq \hat {\boldsymbol {p}}} ^ {\text {}} \right\}. \tag {12}
$$
Note that we use variables  $\xi_{s} = z_{s}\hat{p}_{s}$  in comparison with  $z_{s}$  in Ni and Lai (2022).
The constraint  $\alpha \cdot \pmb{\xi} \leq \hat{\pmb{p}}$  in (12) was not stated explicitly in Ni and Lai (2022) but is necessary because  $\mathrm{EVaR}_{\alpha'}[\cdot]$  is defined only for  $\alpha' \in [0,1]$ . When  $\alpha' = \alpha \xi_s \hat{p}_s^{-1}$  in (11) it must also satisfy for each  $s \in S$  that
$$
\alpha^ {\prime} \leq 1 \quad \Leftrightarrow \quad \alpha \xi_ {s} \hat {p} _ {s} ^ {- 1} \leq 1 \quad \Leftrightarrow \quad \alpha \cdot \xi_ {s} \leq \hat {p} _ {s}.
$$
This additional constraint on  $\xi$  implies that  $\mathcal{Z}_{\mathrm{E}} \subseteq \mathcal{Z}_{\mathrm{C}}$ , for the  $\mathcal{Z}_{\mathrm{C}}$  defined in (7).
We claim in the following theorem (see Appendix A.2 for the proof) that the equality in (11) does not hold even in the policy evaluation setting.
Theorem 4.1. There exists an MDP with a single action and  $\alpha \in (0,1]$  such that
$$
\operatorname {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, a _ {1}, \tilde {s} ^ {\prime}\right) \right] <   \min  _ {\boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \hat {p} _ {s} ^ {- 1}} \left[ r \left(s, a _ {1}, \tilde {s} ^ {\prime}\right) \mid \tilde {s} = s \right], \tag {13}
$$
the set  $\mathcal{Z}_{\mathrm{E}}$  defined by (12).
Theorem 4.1 demonstrates a stronger failure mode than the one in Theorem 3.2 (for CVaR policy optimization) since it applies to both policy evaluation and policy optimization settings.
We propose a correct decomposition of EVaR in the following theorem and employ it to establish that the decomposition in (11) overestimates the actual value of EVaR (see Appendix A.3 for a proof).
Theorem 4.2. Given any finite MDP with horizon  $T = 1$  and  $\alpha \in (0,1]$ , we have that
$$
\mathrm {E V a R} _ {\alpha} \left[ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) \right] = \inf _ {\boldsymbol {\zeta} \in (0, 1 ] ^ {S}, \boldsymbol {\xi} \in \mathcal {Z} _ {\mathrm {E}} ^ {\prime} (\boldsymbol {\zeta})} \sum_ {s \in \mathcal {S}} \xi_ {s} \mathrm {E V a R} _ {\zeta_ {s}} \left[ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s \right],
$$
$$
\mathcal {Z} _ {\mathrm {E}} ^ {\prime} (\boldsymbol {\zeta}) = \left\{\boldsymbol {\xi} \in \Delta_ {S} \mid \boldsymbol {\xi} \ll \hat {\boldsymbol {p}}, \sum_ {s \in \mathcal {S}} \xi_ {s} \left(\log \left(\xi_ {s} / \hat {p} _ {s}\right) - \log \left(\zeta_ {s}\right)\right) \leq - \log \alpha \right\}.
$$
Moreover, EVaR can be upper-bounded as
$$
\operatorname {E V a R} _ {\alpha} \left[ r \left(\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}\right) \right] \leq \min  _ {\xi \in \mathcal {Z} _ {\mathrm {E}}} \sum_ {s \in \mathcal {S}} \xi_ {s} \operatorname {E V a R} _ {\alpha \xi_ {s} \bar {p} _ {s} ^ {- 1}} \left[ r \left(s, \tilde {a}, \tilde {s} ^ {\prime}\right) \mid \tilde {s} = s \right]. \tag {14}
$$
5 VaR: Decomposition Holds for Policy Evaluation and Optimization
In this section, we discuss a dynamic program decomposition for VaR whose decomposition resembles those for CVaR and EVaR described in Sections 3 and 4. We provide a new proof of the VaR decomposition to elucidate the differences that make it optimal in contrast to CVaR and EVaR decompositions. Our VaR decomposition closely resembles the quantile MDP approach in Li et al. (2022) with a few technical modifications that can significantly impact the computed value.
To contrast the typical definition of VaR with the quantile definition in Li et al. (2022), it is helpful to summarize how VaR is related to the quantile of a random variable. Let  $\mathfrak{q} \in \mathbb{R}$  define as the  $\alpha$ -quantile of  $\tilde{x} \in \mathbb{X}$  when
$$
\mathbb {P} [ \tilde {x} \leq \mathfrak {q} ] \geq \alpha \quad \text {a n d} \quad \mathbb {P} [ \tilde {x} <   \mathfrak {q} ] \leq \alpha . \tag {15}
$$
In general, the set of quantiles is an interval  $\left[\mathfrak{q}_{\tilde{x}}^{-}(\alpha), \mathfrak{q}_{\tilde{x}}^{+}(\alpha)\right]$  with the bounds computed as (Follmer and Schied, 2016, appendix A.3)
$$
\mathfrak {q} _ {\tilde {x}} ^ {-} (\alpha) = \sup  \left\{z \mid \mathbb {P} [ \tilde {x} <   z ] <   \alpha \right\} = \inf  \left\{z \mid \mathbb {P} [ \tilde {x} \leq z ] \geq \alpha \right\},
$$
$$
\mathfrak {q} _ {\tilde {x}} ^ {+} (\alpha) = \inf  \left\{z \mid \mathbb {P} [ \tilde {x} \leq z ] > \alpha \right\} = \sup  \left\{z \mid \mathbb {P} [ \tilde {x} <   z ] \leq \alpha \right\}.
$$
Note that when the distribution of  $\tilde{x}$  is absolutely continuous (atomless), then  $\mathfrak{q}_{\tilde{x}}^{+}(\alpha) = \mathfrak{q}_{\tilde{x}}^{-}(\alpha)$  and the quantile is unique. The following example illustrates a simple setting in which the quantile is not unique.
Example 1 (Bernoulli random variable). Consider a Bernoulli random variable  $\tilde{e}$  such that  $\tilde{e} = 1$  and  $\tilde{e} = 0$  with equal  $(50\%)$  probabilities. Then, any value  $q \in [0,1]$  is a valid 0.5-quantile because
$$
\mathfrak {q} _ {\tilde {e}} ^ {-} (0. 5) = \inf  _ {z \in \mathbb {R}} \left\{z \mid \mathbb {P} [ \tilde {e} \leq z ] \geq 0. 5 \right\} = \inf  _ {z \in \mathbb {R}} \left\{z \mid z \geq 0 \right\} = 0
$$
$$
\mathfrak {q} _ {\tilde {e}} ^ {+} (0. 5) = \sup  _ {z \in \mathbb {R}} \left\{z \mid \mathbb {P} [ \tilde {e} \geq z ] \geq 0. 5 \right\} = \sup  _ {z \in \mathbb {R}} \left\{z \mid z \leq 1 \right\} = 1.
$$
The objective in Li et al. (2022) is to maximize the quantile operator  $Q_{\alpha} \colon \mathbb{X} \to \mathbb{R}$  defined for a reward random variable  $\tilde{x} \in \mathbb{X}$  and a risk level  $\alpha \in [0,1]$  as
$$
Q _ {\alpha} (\tilde {x}) = \inf  _ {z \in \mathbb {R}} \left\{z \mid \mathbb {P} [ \tilde {x} \leq z ] \geq \alpha \right\}. \tag {16}
$$
The quantile operator  $Q_{\alpha}$  and VaR differ in which quantile of the random variable they consider:
$$
Q _ {\alpha} (\tilde {x}) = \mathfrak {q} _ {\tilde {x}} ^ {-} (\alpha), \quad \text {b u t} \quad \operatorname {V a R} _ {\alpha} [ \tilde {x} ] = \mathfrak {q} _ {\tilde {x}} ^ {+} (\alpha). \tag {17}
$$
As a result, the quantile MDP objective in (16) coincides with the VaR value only when the quantile is unique, which is not always the case, as shown in Example 1.
Theorem 5.1. Let  $\tilde{y} \colon \Omega \to [N]$  be a random variable distributed as  $\hat{\pmb{p}} = (\hat{p}_i)_{i=1}^N$  with  $\hat{p}_i > 0$ . Then for any random variable  $\tilde{x} \in \mathbb{X}$ , we have
$$
\operatorname {V a R} _ {\alpha} [ \tilde {x} ] = \sup  _ {\zeta \in \Delta_ {N}} \left\{\min  _ {i} \operatorname {V a R} _ {\alpha \zeta_ {i} \hat {p} _ {i} ^ {- 1}} [ \tilde {x} | \tilde {y} = i ] \mid \alpha \cdot \zeta \leq \hat {\boldsymbol {p}} \right\}, \tag {18}
$$
where we interpret the minimum to evaluate to  $\infty$  if all terms are infinite, which only occurs if  $\alpha = 1$ .
Proof. We decompose VaR using the definition in (1) as
$$
\begin{array}{l} \operatorname {V a R} _ {\alpha} [ \tilde {x} ] = \sup  \left\{z \in \mathbb {R} \mid \mathbb {P} [ \tilde {x} <   z ] \leq \alpha \right\} \stackrel {{(\mathrm {a})}} {{=}} \sup  \left\{z \in \mathbb {R} \mid \sum_ {i = 1} ^ {N} \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] \cdot \hat {p} _ {i} \leq \alpha \right\} \\ \stackrel {\mathrm {(b)}} {=} \sup  \left\{z \in \mathbb {R} \mid \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}, \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] \leq \zeta_ {i}, \forall i \in [ N ], \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} \leq \alpha \right\} \\ \stackrel {\mathrm {(c)}} {=} \sup  \left\{\sup  \left\{z \in \mathbb {R} \mid \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] \leq \zeta_ {i}, \forall i \in [ N ] \right\} \mid \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}, \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} \leq \alpha \right\} \\ = \sup  \left\{\sup  \bigcap_ {i \in [ N ]} \left\{z \in \mathbb {R} \mid \mathbb {P} \left[ \tilde {x} <   z \mid \tilde {y} = i \right] \leq \zeta_ {i} \right\} \mid \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}, \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} \leq \alpha \right\} \\ \stackrel {\mathrm{(d)}}{=} \sup  \left\{\min  _ {i \in [ N ]} \sup  \left\{z \in \mathbb {R} \mid \mathbb {P} [ \tilde {x} <   z \mid \tilde {y} = i ] \leq \zeta_ {i} \right\} \mid \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}, \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} \leq \alpha \right\} \\ \stackrel {\mathrm{(e)}}{=} \sup  \left\{\min  _ {i \in [ N ]} \operatorname {V a R} _ {\zeta_ {i}} \left[ \tilde {x} \mid \tilde {y} = i \right] \mid \boldsymbol {\zeta} \in [ 0, 1 ] ^ {N}, \sum_ {i = 1} ^ {N} \zeta_ {i} \hat {p} _ {i} \leq \alpha \right\}. \\ \end{array}
$$
We decompose the probability  $\mathbb{P}[\tilde{x} < z]$  as a marginal of the conditional probabilities  $\mathbb{P}[\tilde{x} < z\mid \tilde{y} = i]$  and  $\hat{p}_i = \mathbb{P}[\tilde{y} = i]$  in step (a) and then lower-bound them by an auxiliary variable  $\zeta_{i}$  in step (b). In step (c) we replace the joint supremum over  $z$  and  $\zeta$  by sequential suprema, and then we replace the supremum of an intersection by the minimum of the suprema of sets in (d). The equality in (d) holds because  $\mathbb{P}[\tilde{x} < z]$  is monotone and, therefore, the sets  $\{z\in \mathbb{R}\mid \mathbb{P}[\tilde{x} < z\mid \tilde{y} = i]\leq \zeta_i\}$  are nested. Finally, step (e) follows from the definition of VaR in (1).
Focusing on the finite MDP with horizon  $T = 1$ , we can show that the decomposition proposed in Theorem 5.1 is amenable to policy optimization. The main difference between the VaR decomposition and CVaR is that the former VaR was expressed as a supremum instead of an infimum over quantile levels  $\zeta$ . For VaR, changing the order of maximum  $(\pi)$  and supremum  $(\zeta)$  does not suffer from a potential gap, but changing the order of maximum  $(\pi)$  and infimum/minimum  $(\zeta)$  in CVaR does suffer from such a gap as shown in Theorem 3.2.
The following theorem (proved in Appendix A.4) summarizes the decomposition for VaR.
Theorem 5.2. Given any finite MDP with horizon  $T = 1$  and  $\alpha \in [0,1]$ , we have
$$
\max  _ {\pi \in \Pi} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s \in S} \max  _ {\boldsymbol {d} \in \Delta_ {A}} \operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ] \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\}.
$$
For completeness, Appendix C extends Theorem 5.2 to a setting with horizon  $T > 1$ , providing dynamic programming equations and a definition of the optimal policy. These equations are analogous to the equations presented in (Li et al., 2022) yet we obtain them using the accurate definition and decomposition of  $\mathrm{VaR}_{\alpha}$ .
We finally present below a valid decomposition for the lower quantile MDP, used officially as the objective in (Li et al., 2022) (see Appendix A.5 for a proof).
Proposition 5.3. Given any finite MDP with horizon  $T = 1$  and some  $\alpha \in [0,1]$ , we have that:
$$
\max  _ {\pi \in \Pi} Q _ {\alpha} ^ {\tilde {a} \sim \boldsymbol {\pi} (\tilde {s})} (r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime})) = \sup  _ {\zeta \in [ 0, 1 ] ^ {S}} \left\{\min  _ {s \in \mathcal {S}: \zeta_ {s} <   1} \max  _ {\boldsymbol {d} \in \Delta_ {A}} Q _ {\zeta_ {s}} ^ {\tilde {a} \sim \boldsymbol {d}} (r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s) \mid \sum_ {s = 1} ^ {S} \zeta_ {s} \hat {p} _ {s} <   \alpha \right\}.
$$
We note that the difference with the result presented in (Li et al., 2022) resides in the constraint imposed on  $\zeta$  that replaces the weak inequality with a strict one. In fact, this strict versus weak inequality is the main distinguishing factor between the decompositions for the lower and upper quantiles.
This paper shows that a popular decomposition approach to solving MDPs with CVaR and EVaR objectives is suboptimal despite the claims to the contrary. This suboptimality arises from a saddle-point gap when optimizing policy. We also prove that a similar decomposition approach is optimal for policy optimization and evaluation when solving MDPs with the VaR objective. The decomposition is optimal because VaR does not involve the same saddle point problem as CVaR and EVaR.
Our findings are significant because practitioners who make risk-averse decisions in high-stakes scenarios need to have confidence in the correctness of the algorithms they use. Our work raises awareness that popular static CVaR and EVaR MDP algorithms are suboptimal, and their analyses are inaccurate. We hope the results we present in our paper will increase the scrutiny of dynamic programming methods for risk-averse MDPs and motivate research into alternative approaches, such as the parametric dynamic programs.
We thank Yinlam Chow for his insightful comments that inspired this paper's topic and results. The work in the paper was supported, in part, by NSF grants 2144601 and 2218063. In addition, this work was performed in part while Marek Petrik was a visiting researcher at Google Research. Finally, E. Delage acknowledges the support from the Canadian Natural Sciences and Engineering Research Council and the Canada Research Chair program [Grant RGPIN-2022-05261 and CRC-2018-00105].
Mohamadreza Ahmadi, Xiaobin Xiong, and Aaron D Ames. Risk-averse control via CVaR barrier functions: Application to bipedal robot locomotion. IEEE Control Systems Letters, 6:878-883, 2021.
A. Ahmadi-Javid. Entropic Value-at-Risk: A new coherent risk measure. Journal of Optimization Theory and Applications, 155(3):1105-1123, 2012.
Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical Finance, 9(3):203-228, 1999.
Basel Committee on Banking Supervision. Minimum capital requirements for market risk. In Basel III: international regulatory framework for banks, 2019.
Nicole Bäuerle and Jonathan Ott. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 74(3):361-379, 2011.
Kang Boda, Jerzy A Filar, Yuanlie Lin, and Lieneke Spanjers. Stochastic target hitting time and the problem of early retirement. IEEE Transactions on Automatic Control, 49(3):409-419, 2004.
Margaret P Chapman, Jonathan Lacotte, Aviv Tamar, Donggun Lee, Kevin M Smith, Victoria Cheng, Jaime F Fisac, Susmit Jha, Marco Pavone, and Claire J Tomlin. A risk-sensitive finite-time reachability approach for safety of stochastic dynamic systems. In American Control Conference (ACC), pages 2958-2963. IEEE, 2019.
Margaret P Chapman, Riccardo Bonalli, Kevin M Smith, Insoon Yang, Marco Pavone, and Claire J Tomlin. Risk-sensitive safety analysis using conditional value-at-risk. IEEE Transactions on Automatic Control, 67(12):6521-6536, 2022.
Arnav Choudhry, Brady Moon, Jay Patrikar, Constantine Samaras, and Sebastian Scherer. CVaR-based flight energy risk assessment for multirotor uavs using a deep energy model. In IEEE International Conference on Robotics and Automation (ICRA), pages 262-268, 2021.