SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jia Lin Hau
Durham, NH
jialin.hau@unh.edu
Erick Delage
HEC Montréal
Montréal (Québec)
erick.delage@hec.ca
Mohammad Ghavamzadeh*
Marek Petrik
mpetrik@cs.unh.edu
Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming decompositions that augment the state space with discrete risk levels have recently gained popularity in the RL community. Prior work has shown that these decompositions are optimal when the risk level is discretized sufficiently. However, we show that these popular decompositions for Conditional-Value-at-Risk (CVaR) and Entropic-Value-at-Risk (EVaR) are inherently suboptimal regardless of the discretization level. In particular, we show that a saddle point property assumed to hold in prior literature may be violated. However, a decomposition does hold for Value-at-Risk and our proof demonstrates how this risk measure differs from CVaR and EVaR. Our findings are significant because risk-averse algorithms are used in high-stakes environments, making their correctness much more critical.
Risk-averse reinforcement learning (RL) seeks to provide a risk-averse policy for high-stakes real-world decision problems. These high-stake domains include autonomous driving (Jin et al., 2019; Sharma et al., 2020), robot collision avoidance (Ahmadi et al., 2021; Hakobyan and Yang, 2021), liver transplant timing (Köse, 2016), HIV treatment (Keramati et al., 2020; Zhong, 2020), unmanned aerial vehicle (UAV) (Choudhry et al., 2021), and investment liquidation (Min et al., 2022), to name a few. Because these domains call for reliable solutions, risk-averse algorithms must be based on solid theoretical foundations. This is one reason why monetary risk measures, such as Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR), have become pervasive in risk-averse RL (Prashanth and Fu, 2022). Indeed, risk measures such as CVaR are known to be coherent (Artzner et al., 1999) with respect to a set of fundamental axioms that define how risk should be quantified and have been adopted as gold standards in banking regulations (Basel Committee on Banking Supervision, 2019).
Introducing risk-averse objectives in Markov decision processes (MDPs)—the primary model used in RL—is challenging. Dynamic programming, the linchpin of most RL algorithms, cannot be used directly to optimize a risk measure like VaR or CVaR in MDPs. One line of work tackles this challenge by exploiting the primal representation of risk measures and augmenting the state space
*The work was done prior to joining Amazon, while the author was at Google Research.
of their dynamic programs (DPs) with an additional parameter that typically represents the total cumulative reward up to the current point (Bäuerle and Ott, 2011; Boda et al., 2004; Chow and Ghavamzadeh, 2014; Filar et al., 1995; Hau et al., 2023; Lin et al., 2003; Wu and Lin, 1999; Xu and Mannor, 2011). Even when the original MDP is finite, this DP requires computing the value function for a continuous state space, and thus, has been considered inefficient in practice (Chapman et al., 2022; Chow et al., 2015; Li et al., 2022).
Another line of recent work leverages the dual representation to produce a risk-level decomposition of risk measures (Pflug and Pichler, 2016). Using this decomposition, numerous authors have derived DPs for common risk measures and integrated them within various RL algorithms (Chapman et al., 2019, 2022; Chow et al., 2015; Ding and Feinberg, 2022, 2023; Ni and Lai, 2022; Rigter et al., 2021; Stanko and Macek, 2019). Although this risk-level decomposition requires augmenting the state space with a continuous parameter, this parameter is naturally bounded between 0 and 1. It has been generally accepted, with several tentative proofs supporting this claim (Chow et al., 2015; Li et al., 2022), that these DPs recover the optimal policy if we can discretize the augmented state space sufficiently finely. Moreover, it is believed that one can use the optimal value function from this DP to recover the policies that are optimal for the full range of risk levels.
In this paper, we make a surprising discovery that numerous claims of optimality of risk-level decompositions published in the past several years are incorrect. Even when one discretizes the augmented state space arbitrarily finely, most risk-level DPs are not guaranteed to recover the optimal value function and policy. There are several reasons why existing arguments fail. As the most common reason, several papers assume that a certain saddle point property holds, either explicitly (Chow et al., 2015) or implicitly (Ding and Feinberg, 2022, 2023). We show that this property does not generally hold, invalidating the optimality of DPs, as hinted at in Chapman et al. (2019, 2022). This finding directly refutes the claimed or hypothesized optimality of algorithms proposed in many recent research papers and pre-prints, such as Chapman et al. (2019); Chow et al. (2015); Ding and Feinberg (2022, 2023); Rigter et al. (2021); Stanko and Macek (2019). Our results also affect applications of these algorithms, such as automated vehicle motion planning (Jin et al., 2019). We also identify gaps in related decompositions (Li et al., 2022; Ni and Lai, 2022) and propose how to fix them.
We make the following contributions in this paper. First, we show in Section 3 that the popular DP for optimizing CVaR in MDPs may not recover the optimal value function and policy regardless of how finely one discretizes the risk level in the augmented states. This method was first proposed in Chow et al. (2015) but adopted widely afterwards (Chapman et al., 2019; Ding and Feinberg, 2022, 2023; Rigter et al., 2021; Stanko and Macek, 2019). The simple counterexample in this section contradicts the optimality claims in Chow et al. (2015); Ding and Feinberg (2022). We hypothesize that prior work missed this issue because the CVaR DP works for policy evaluation and only fails when one uses it to optimize policies based on the "risk-to-go value function". Therefore, our results do not contradict the original decomposition in Pflug and Pichler (2016) that only applies to policy evaluation. We give a new independent and simple proof that the CVaR decomposition indeed works when evaluating a fixed policy.
Second, we show in Section 4 that the DP for optimizing the Entropic-Value-at-Risk in MDPs, proposed by Ni and Lai (2022), does not compute the correct value function even when the policy is fixed. Although EVaR has not been as popular as CVaR, it has been gaining attention in recent years (Hau et al., 2023). We give an example that contradicts the correctness claims of the risk-level decomposition for EVaR in Ni and Lai (2022). The gap that we identify with this objective applies to both policy evaluation and policy optimization. Furthermore, we prove a new, correct EVaR decomposition for policy evaluation. Unfortunately, the EVaR decomposition fails and is sub-optimal when applied to policy optimization, similar to CVaR.
Third, we propose an optimal dynamic program for policy optimization of VaR in Section 5. Our DP is based on a risk-level decomposition that closely resembles the quantile MDP decomposition in Li et al. (2022) but corrects for several technical inaccuracies. The derivation shows why VaR stands apart from coherent risk measures like CVaR and EVaR. VaR is unique in that the decomposition can be constructed directly from the primal formulation of the risk measure, which avoids the complications that arise in the robust formulations used in CVaR and EVaR decompositions.
It is important to note that the correctness of DPs that augment the state space with the accumulated rewards is unaffected by our results (Bäuerle and Ott, 2011; Chow and Ghavamzadeh, 2014; Chow et al., 2018; Hau et al., 2023). These DPs use the primal risk measure representation and do not suffer
from the same saddle point issue as the augmentation methods that use the dual representation of the risk measures, such as the one in Chow et al. (2015).
This section summarizes relevant properties of monetary risk measures and outlines how they are typically used in the context of solving MDPs.
**Monetary Risk Measures** We restrict our attention to probability spaces with a finite outcome space  $\Omega$  such that  $|\Omega| = m$  for some  $m \in \mathbb{N}$ . We use  $\mathbb{X} = \mathbb{R}^m$  to denote the space of real-valued random variables. To improve the clarity of probabilistic claims, we always adorn random variables with a tilde, such as  $\tilde{x} \in \mathbb{X}$ . In finite spaces, we can represent any random variable  $\tilde{x} \in \mathbb{X}$  as a vector  $\boldsymbol{x} \in \mathbb{R}^m$ . We also use  $\boldsymbol{q} \in \Delta_m$  to represent a probability distribution over  $\Omega$  where  $\Delta_m$  represents the  $m$ -dimensional probability simplex. Using this notation, we can write that  $\mathbb{E}[\tilde{x}] = \boldsymbol{q}^\top \boldsymbol{x}$ .
A monetary risk measure  $\psi \colon \mathbb{X} \to \mathbb{R}$  assigns a real value to each real-valued random variable in a way that it is monotone and cash-invariant (Follmer and Schied, 2016; Shapiro et al., 2014). A risk measure can be seen as a generalization of the expectation operator  $\mathbb{E}[\cdot]$  that also takes into account the uncertainty in the random variable. In this work, we define all risk measures for random variables  $\tilde{x}$  that represent rewards. Thus, the risk-averse decision-maker aims to choose actions that maximize the value of the risk measure, i.e., a higher value of risk measure represents a lower exposure to risk.
We consider three monetary risk measures common in RL. Perhaps the most well-known measure is Value-at-Risk (VaR), which is defined for a risk-level  $\alpha \in [0,1]$  and a random variable  $\tilde{x} \in \mathbb{X}$  in modern literature as (e.g., Follmer and Schied 2016; Shapiro et al. 2014)
$$
\operatorname {V a R} _ {\alpha} [ \tilde {x} ] = \sup  \left\{z \in \mathbb {R} \mid \mathbb {P} [ \tilde {x} <   z ] \leq \alpha \right\} = \inf  \left\{z \in \mathbb {R} \mid \mathbb {P} [ \tilde {x} \leq z ] > \alpha \right\}. \tag {1}
$$
Note that  $\mathrm{VaR}_1[\tilde{x}] = \infty$ . The equality between the two definitions holds, for example, by Follmer and Schied (2016, remark A.20).
Another popular risk measure is the Conditional-value-at-Risk (CVaR), which is defined for a risk level  $\alpha \in [0,1]$  and a random variable  $\tilde{x} \in \mathbb{X}$  distributed as  $\tilde{x} \sim q$  as (e.g., Follmer and Schied 2016, definition 11.8 and Shapiro et al. 2014, eq. 6.23)
$$
\operatorname {C V a R} _ {\alpha} [ \tilde {x} ] = \sup  _ {z \in \mathbb {R}} (z - \alpha^ {- 1} \mathbb {E} [ z - \tilde {x} ] _ {+}) = \inf  \left\{\boldsymbol {\xi} ^ {\top} \boldsymbol {x} \mid \boldsymbol {\xi} \in \Delta_ {m}, \alpha \cdot \boldsymbol {\xi} \leq \boldsymbol {q} \right\}, \tag {2}
$$
with  $\mathrm{CVaR}_0[\tilde{x}] = \operatorname*{ess\ inf}[\tilde{x}]$  and  $\mathrm{CVaR}_1[\tilde{x}] = \mathbb{E}[\tilde{x}]$ . The equality above follows from standard conjugacy arguments for finite probability spaces (Follmer and Schied, 2016). Note that our CVaR definition applies to  $\tilde{x}$  that represents rewards and assumes that a higher value of the risk measure is preferable to a lower value. Other CVaR formulations exist in the literature, but they induce identical preferences for appropriately chosen rewards and risk level  $\alpha$ .
Finally, the entropic value at risk (EVaR), with  $\mathrm{EVaR}_0[\tilde{x}] = \mathrm{ess\ inf}[\tilde{x}]$  and  $\mathrm{EVaR}_1[\tilde{x}] = \mathbb{E}[\tilde{x}]$ , is defined for  $\alpha \in (0,1]$  as (Ahmadi-Javid, 2012)
$$
\begin{array}{l} \operatorname {E V a R} _ {\alpha} [ \tilde {x} ] = \sup  _ {\beta > 0} - \frac {1}{\beta} \log (\alpha^ {- 1} \mathbb {E} [ \exp (- \beta \cdot \tilde {x}) ]) \tag {3} \\ = \inf  \left\{\boldsymbol {\xi} ^ {T} \boldsymbol {x} \mid \boldsymbol {\xi} \in \Delta_ {m}, \boldsymbol {\xi} \ll \boldsymbol {q}, \operatorname {K L} (\boldsymbol {\xi} \| \boldsymbol {q}) \leq - \log \alpha \right\}, \\ \end{array}
$$
where  $\mathrm{KL}$  is the standard KL-divergence defined for each  $\boldsymbol{x},\boldsymbol{y} \in \Delta_m$  as  $\mathrm{KL}(\boldsymbol{x}\| \boldsymbol{y}) = \sum_{\omega \in \Omega}x_{\omega}\log \left(x_{\omega} / y_{\omega}\right)$ . This definition is valid only when  $\boldsymbol{x}$  is absolutely continuous with respect to  $\boldsymbol{y}$ , which is denoted as  $\boldsymbol{x} \ll \boldsymbol{y}$  and corresponds to  $y_{\omega} = 0 \Rightarrow x_{\omega} = 0$  for each  $\omega \in \Omega$ .
Risk Averse MDPs A Markov decision process (MDP) is a sequential decision model that underlies most of RL (Puterman, 2005). We consider finite MDPs with states  $S = \{s_1, \dots, s_S\}$  and actions  $\mathcal{A} = \{a_1, \dots, a_A\}$ . After taking an action in a state, the agent transitions to the next state according to a transition probability function  $p \colon S \times \mathcal{A} \to \Delta_S$  such that  $p(s, a, s')$  represents the transition probability from  $s \in S$  to  $s' \in S$  after taking  $a \in \mathcal{A}$ . We use  $\pmb{p}_{s,a} = p(s, a, \cdot) \in \Delta_S$  to denote the vector of transition probabilities. The initial state  $\tilde{s}_0$  is distributed according to  $\hat{\pmb{p}} \in \Delta_S$ . To avoid divisions by 0 that are not central to our claims, we assume that  $\hat{p}_s > 0$  for each  $s \in S$ . Finally,
the reward function is  $r \colon S \times \mathcal{A} \times S \to \mathbb{R}$ , where  $r(s, a, s')$  represents the deterministic reward associated with the transition to  $s'$  from  $s$  after taking an action  $a$ .
The most general solution to an MDP is a history-dependent randomized policy  $\pi$  which maps a sequence of observed states and actions  $s^0, a^0, s^1, a^1, \ldots, s^t$  to a distribution over the next action  $a^t$ . It is well-known that with risk-neutral objectives, there always exists an optimal stationary—depends only on the last state—deterministic policy (Puterman, 2005). When the objective is risk-averse, like VaR, or CVaR, there may not exist an optimal stationary or deterministic policy. Hence, we use the symbol  $\Pi$  to denote the set of history-dependent randomized policies in the remainder of the paper.
This paper focuses on the finite-horizon objective in which the agent aims to compute policies that optimize the sum of rewards over a known horizon  $T$ . We further restrict our attention to the objective with horizon  $T = 1$ . It turns out that having a single time step is sufficient to derive our counterexamples to existing dynamic programs. Moreover, deriving the decompositions with  $T = 1$  makes it possible to avoid technicalities caused by history-dependent policies, which could distract us from the main ideas presented in this work. Our results can be extended to general horizons  $T > 1$  and the discounted infinite-horizon objectives using standard techniques (Chow et al., 2015).
With horizon  $T = 1$ , the set of randomized history-dependent policies is  $\Pi = \{\pi : S \to \Delta_A\}$ . The symbol  $\pi(s, a)$  denotes the probability of action  $a$  in a state  $s$ , and  $\pi(s) = \pi(s, \cdot) \in \Delta_A$  denotes the  $A$ -dimensional vector of action probabilities in a state  $s$ . Given a risk measure  $\psi$  with a risk level  $\alpha \in [0, 1]$ , the finite-horizon risk-averse value of a policy  $\pi \in \Pi$  is computed as
$$
v _ {0} ^ {\pi} (\alpha) = \psi_ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ], \qquad (4)
$$
where the superscript in  $\psi_{\alpha}^{\tilde{a}\sim \pi (\tilde{s})}$  specifies the distribution of the random action. Throughout the paper, we generally use  $\tilde{s}$  to denote the random state at time  $t = 0$  and  $\tilde{s}'$  to denote the random state at time  $t = 1$ . In risk-neutral objectives, when  $\psi = \mathbb{E}$ , one can use the tower property of the expectation operator and define a value function  $v_{t}$  for each time step  $t$  (Puterman, 2005), but this property does not hold in most static risk measures (Hau et al., 2023). The term policy evaluation in the remainder of the paper refers to computing the value in (4).
The goal in an MDP is to compute an optimal value function and a policy that attains it. In risk-averse MDPs, this goal is formalized as the following risk-averse optimization
$$
v _ {0} ^ {\star} (\alpha) = \max  _ {\pi \in \Pi} v _ {0} ^ {\pi} (\alpha) = \max  _ {\pi \in \Pi} \psi_ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ], \tag {5}
$$
with the optimal policy  $\pi^{\star}$  being any policy that attains the maximum in (5). As with policy evaluation, when  $\psi = \mathbb{E}$ , the optimal value function  $v_{t}^{\star}$  can be defined for each time-step  $t$  (Puterman, 2005), but this is impossible in general for common risk measures, like VaR and CVaR. The term policy optimization in the remainder of the paper refers to computing the value and the maximizer in (5).
In the remainder of the paper, we study dynamic programming algorithms proposed to solve the policy evaluation problem in (4) and policy optimization problem in (5). In general, these algorithms build on risk-level decomposition (Pflug and Pichler, 2016) of risk measures to define a value function  $v_{t}^{\pi}(s,\alpha)$  for each time step  $t\in [T]$ , state  $s\in S$ , and risk-level  $\alpha \in [0,1]$  (Chow et al., 2015). The value function represents the risk-adjusted sum of rewards that can be obtained if starting in a state  $s\in S$  at time  $t$  and a risk level  $\alpha$ . For example, one would define the value function as  $v_{1}^{\pi}(s,\alpha) = \psi_{\alpha}^{\tilde{a}\sim \pi (s)}[r(s,\tilde{a},\tilde{s}^{\prime})]$  and compute  $v_{0}^{\pi}$  using a Bellman operator  $T_{\alpha}^{\pi}$  as  $v_{0}^{\pi}(\alpha) = (T^{\pi}v_{1}^{\pi})(\alpha)$ . In risk-neutral objectives, the Bellman operator is defined as  $(T^{\pi}(v_{1}^{\pi}))(a) = \mathbb{E}[v_{1}^{\pi}(\tilde{s},\alpha)]$ , with  $\alpha \in \{1\}$ , but in risk-averse formulations the operator definition is more complex. The remainder of the paper discusses the decompositions and the operator for CVaR, EVaR, and VaR risk measures respectively.
3 CVaR: Decomposition Fails in Policy Optimization
In this section, we show that a common CVaR decomposition proposed in Chow et al. (2015) and used to optimize risk-averse policies is inherently sub-optimal regardless of how closely one discretizes the state space. The following proposition represents one of the key results used to decompose the risk measure in multi-stage decision-making.
Figure 1: Rewards of MDP  $M_{\mathrm{C}}$  used in the proof of Theorem 3.2. The dot indicates that the rewards are independent of the next state.
Proposition 3.1 (lemma 22 in Pflug and Pichler 2016). Suppose that  $\pi \in \Pi$  and  $\tilde{s} \sim \hat{p}$ ,  $\tilde{a} \sim \pi(\tilde{s})$ ,  $\tilde{s}' \sim p_{s,a}$ . Then,
$$
\mathrm {C V a R} _ {\alpha} \left[ r \left(\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}\right) \right] = \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \mathrm {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} \left[ r \left(s, \tilde {a}, \tilde {s} ^ {\prime}\right) \mid \tilde {s} = s \right], \tag {6}
$$
where the state  $s$  on the right-hand side is not random and
$$
\mathcal {Z} _ {\mathrm {C}} = \left\{\boldsymbol {\zeta} \in \Delta_ {S} \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\}. \tag {7}
$$
The notation in Proposition 3.1 differs superficially from lemma 22 in Pflug and Pichler (2016). Specifically, our CVaR is defined for rewards rather than costs, the meaning of our  $\alpha$  corresponds to  $1 - \alpha$  in Pflug and Pichler (2016), and we use  $\xi_s = z_s\hat{p}_s$  as the optimization variable. We include a simple proof of Proposition 3.1 for completeness in Appendix A.1.
The decomposition in Proposition 3.1 is important because it shows that the CVaR evaluation can be formulated as a dynamic program. The theorem shows that CVaR at time  $t = 0$  decomposes into a convex combination of CVaR values at  $t = 1$ . Recursively repeating this process, one can formulate a dynamic program for any finite time horizon  $T$ . Because the risk level at  $t = 1$  differs from the level at  $t = 0$  and depends on the optimal  $\zeta$ , one must compute CVaR values for all (or many) risk-levels  $\alpha \in [0,1]$  at time  $t = 1$ . As a result, the dynamic program includes an additional state variable that represents the current risk level.
Chow et al. (2015) proposed to adapt the decomposition in Proposition 3.1 to policy optimization as
$$
\begin{array}{l} \max  _ {\pi \in \Pi} \mathrm {C V a R} _ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] = \max  _ {\pi \in \Pi} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \left(\mathrm {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \pi (s)} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ]\right) \\ \stackrel {{\ref {e q : 1}}} {{=}} \min  _ {\zeta \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \left(\max  _ {\boldsymbol {d} \in \Delta_ {\mathcal {A}}} \operatorname {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) \mid \tilde {s} = s ]\right). \tag {8} \\ \end{array}
$$
They used the decomposition in (8) to formulate a dynamic program with the current risk level as an additional state variable. We prove in the following theorem that the second equality in (8) marked with question marks is false in general.
Theorem 3.2. There exists an MDP and a risk level  $\alpha \in [0,1]$  such that
$$
\max  _ {\pi \in \Pi} \operatorname {C V a R} _ {\alpha} ^ {\tilde {a} \sim \pi (\tilde {s})} [ r (\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime}) ] <   \min  _ {\boldsymbol {\zeta} \in \mathcal {Z} _ {\mathrm {C}}} \sum_ {s \in \mathcal {S}} \zeta_ {s} \left(\max  _ {\boldsymbol {d} \in \Delta_ {\mathcal {A}}} \operatorname {C V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} \sim \boldsymbol {d}} [ r (s, \tilde {a}, \tilde {s} ^ {\prime}) | \tilde {s} = s ]\right). \tag {9}
$$
Before proving Theorem 3.2, we discuss its implications. First, Theorem 3.2 contradicts theorems 5 and 7 in Chow et al. (2015) and shows that their algorithm is inherently sub-optimal regardless of the resolution of the discretization. Theorem 3.2 also contradicts the optimality of the accelerated dynamic program proposed in Stanko and Macek (2019). The result of (Chow et al., 2015) was exploited as is in Chapman et al. (2019), Ding and Feinberg (2022), and Jin et al. (2019) to propose DP reductions, and extended, without proof, in (Rigter et al., 2021) to the context of a Bayesian MDP.
Finally, it is important to emphasize that Theorem 3.2 only applies to the policy optimization setting and does not contradict Proposition 3.1, which holds for the evaluation of policies that assign the same action distribution to each history of states and actions (i.e., policies that are independent of the hypothesized values of  $\zeta$ ).
Figure 2: The functions  $\theta_{\pi_1}(\cdot)$  and  $\theta_{\pi_2}(\cdot)$  used in the CVaR counterexample in the proof of Theorem 3.2. The dashed line shows the function  $\zeta_{s_1}\mapsto \max_{\pi \in \{\pi_1,\pi_2\}}\theta_\pi ([\zeta_{s_1},1 - \zeta_{s_1}])$
Proof. Let  $\alpha = 0.5$  and consider the MDP  $M_{\mathrm{C}}$  in Figure 1. In state  $s_1$ , both actions  $a_1$  and  $a_2$  are available, and in state  $s_2$ , only action  $a_1$  is available. The MDP's rewards are
$$
r \left(s _ {1}, a _ {1}, s _ {1}\right) = - 5 0, \quad r \left(s _ {1}, a _ {1}, s _ {2}\right) = 1 0 0,
$$