Title: On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

markov decision processes

monetary risk measures
Abstract: Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming decompositions that augment the state space with discrete risk levels have recently gained popularity in the RL community. Prior work has shown that these decompositions are optimal when the risk level is discretized sufficiently. However, we show that these popular decompositions for Conditional-Value-at-Risk (CVaR) and Entropic-Value-at-Risk (EVaR) are inherently suboptimal regardless of the discretization level. In particular, we show that a saddle point property assumed to hold in prior literature may be violated. However, a decomposition does hold for Value-at-Risk and our proof demonstrates how this risk measure differs from CVaR and EVaR. Our findings are significant because risk-averse algorithms are used in high-stake environments, making their correctness much more critical.

On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes

Abstract

Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming decompositions that augment the state space with discrete risk levels have recently gained popularity in the RL community. Prior work has shown that these decompositions are optimal when the risk level is discretized sufficiently. However, we show that these popular decompositions for Conditional-Value-at-Risk (CVaR) and EntropicValue-at-Risk (EVaR) are inherently suboptimal regardless of the discretization level. In particular, we show that a saddle point property assumed to hold in prior literature may be violated. However, a decomposition does hold for Value-at-Risk and our proof demonstrates how this risk measure differs from CVaR and EVaR. Our findings are significant because risk-averse algorithms are used in high-stakes environments, making their correctness much more critical.

1 Introduction

Risk-averse reinforcement learning (RL) seeks to provide a risk-averse policy for high-stakes realworld decision problems. These high-stake domains include autonomous driving (Jin et al., 2019; Sharma et al., 2020), robot collision avoidance (Ahmadi et al., 2021; Hakobyan and Yang, 2021), liver transplant timing (Köse, 2016), HIV treatment (Keramati et al., 2020; Zhong, 2020), unmanned aerial vehicle (UA V) (Choudhry et al., 2021), and investment liquidation (Min et al., 2022), to name a few. Because these domains call for reliable solutions, risk-averse algorithms must be based on solid theoretical foundations. This is one reason why monetary risk measures, such as Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR), have become pervasive in risk-averse RL (Prashanth and Fu, 2022). Indeed, risk measures such as CVaR are known to be coherent (Artzner et al., 1999) with respect to a set of fundamental axioms that define how risk should be quantified and have been adopted as gold standards in banking regulations (Basel Committee on Banking Supervision, 2019).

Introducing risk-averse objectives in Markov decision processes (MDPs)-the primary model used in RL-is challenging. Dynamic programming, the linchpin of most RL algorithms, cannot be used directly to optimize a risk measure like VaR or CVaR in MDPs. One line of work tackles this challenge by exploiting the primal representation of risk measures and augmenting the state space

Mohammad Ghavamzadeh ∗

Amazon

of their dynamic programs (DPs) with an additional parameter that typically represents the total cumulative reward up to the current point (Bäuerle and Ott, 2011; Boda et al., 2004; Chow and Ghavamzadeh, 2014; Filar et al., 1995; Hau et al., 2023; Lin et al., 2003; Wu and Lin, 1999; Xu and Mannor, 2011). Even when the original MDP is finite, this DP requires computing the value function for a continuous state space, and thus, has been considered inefficient in practice (Chapman et al., 2022; Chow et al., 2015; Li et al., 2022).

Another line of recent work leverages the dual representation to produce a risk-level decomposition of risk measures (Pflug and Pichler, 2016). Using this decomposition, numerous authors have derived DPs for common risk measures and integrated them within various RL algorithms (Chapman et al., 2019, 2022; Chow et al., 2015; Ding and Feinberg, 2022, 2023; Ni and Lai, 2022; Rigter et al., 2021; Stanko and Macek, 2019). Although this risk-level decomposition requires augmenting the state space with a continuous parameter, this parameter is naturally bounded between 0 and 1. It has been generally accepted, with several tentative proofs supporting this claim (Chow et al., 2015; Li et al., 2022), that these DPs recover the optimal policy if we can discretize the augmented state space sufficiently finely. Moreover, it is believed that one can use the optimal value function from this DP to recover the policies that are optimal for the full range of risk levels.

In this paper, we make a surprising discovery that numerous claims of optimality of risk-level decompositions published in the past several years are incorrect. Even when one discretizes the augmented state space arbitrarily finely, most risk-level DPs are not guaranteed to recover the optimal value function and policy. There are several reasons why existing arguments fail. As the most common reason, several papers assume that a certain saddle point property holds, either explicitly (Chow et al., 2015) or implicitly (Ding and Feinberg, 2022, 2023). We show that this property does not generally hold, invalidating the optimality of DPs, as hinted at in Chapman et al. (2019, 2022). This finding directly refutes the claimed or hypothesized optimality of algorithms proposed in many recent research papers and pre-prints, such as Chapman et al. (2019); Chow et al. (2015); Ding and Feinberg (2022, 2023); Rigter et al. (2021); Stanko and Macek (2019). Our results also affect applications of these algorithms, such as automated vehicle motion planning (Jin et al., 2019). We also identify gaps in related decompositions (Li et al., 2022; Ni and Lai, 2022) and propose how to fix them.

We make the following contributions in this paper. First , we show in Section 3 that the popular DP for optimizing CVaR in MDPs may not recover the optimal value function and policy regardless of how finely one discretizes the risk level in the augmented states. This method was first proposed in Chow et al. (2015) but adopted widely afterwards (Chapman et al., 2019; Ding and Feinberg, 2022, 2023; Rigter et al., 2021; Stanko and Macek, 2019). The simple counterexample in this section contradicts the optimality claims in Chow et al. (2015); Ding and Feinberg (2022). We hypothesize that prior work missed this issue because the CVaR DP works for policy evaluation and only fails when one uses it to optimize policies based on the 'risk-to-go value function'. Therefore, our results do not contradict the original decomposition in Pflug and Pichler (2016) that only applies to policy evaluation. We give a new independent and simple proof that the CVaR decomposition indeed works when evaluating a fixed policy.

Second , we show in Section 4 that the DP for optimizing the Entropic-Value-at-Risk in MDPs, proposed by Ni and Lai (2022), does not compute the correct value function even when the policy is fixed. Although EVaR has not been as popular as CVaR, it has been gaining attention in recent years (Hau et al., 2023). We give an example that contradicts the correctness claims of the risk-level decomposition for EVaR in Ni and Lai (2022). The gap that we identify with this objective applies to both policy evaluation and policy optimization. Furthermore, we prove a new, correct EVaR decomposition for policy evaluation. Unfortunately, the EVaR decomposition fails and is sub-optimal when applied to policy optimization, similar to CVaR.

Third , we propose an optimal dynamic program for policy optimization of VaR in Section 5. Our DP is based on a risk-level decomposition that closely resembles the quantile MDP decomposition in Li et al. (2022) but corrects for several technical inaccuracies. The derivation shows why VaR stands apart from coherent risk measures like CVaR and EVaR. VaR is unique in that the decomposition can be constructed directly from the primal formulation of the risk measure, which avoids the complications that arise in the robust formulations used in CVaR and EVaR decompositions.

It is important to note that the correctness of DPs that augment the state space with the accumulated rewards is unaffected by our results (Bäuerle and Ott, 2011; Chow and Ghavamzadeh, 2014; Chow et al., 2018; Hau et al., 2023). These DPs use the primal risk measure representation and do not suffer

from the same saddle point issue as the augmentation methods that use the dual representation of the risk measures, such as the one in Chow et al. (2015).

2 Preliminaries

This section summarizes relevant properties of monetary risk measures and outlines how they are typically used in the context of solving MDPs.

Monetary Risk Measures We restrict our attention to probability spaces with a finite outcome space Ω such that | Ω | = m for some m ∈ N . We use X = R m to denote the space of real-valued random variables. To improve the clarity of probabilistic claims, we always adorn random variables with a tilde, such as ˜ x ∈ X . In finite spaces, we can represent any random variable ˜ x ∈ X as a vector x ∈ R m . We also use q ∈ ∆ m to represent a probability distribution over Ω where ∆ m represents the m -dimensional probability simplex. Using this notation, we can write that E [˜ x ] = q ⊤ x .

A monetary risk measure ψ : X → R assigns a real value to each real-valued random variable in a way that it is monotone and cash-invariant (Follmer and Schied, 2016; Shapiro et al., 2014). A risk measure can be seen as a generalization of the expectation operator E [ · ] that also takes into account the uncertainty in the random variable. In this work, we define all risk measures for random variables ˜ x that represent rewards . Thus, the risk-averse decision-maker aims to choose actions that maximize the value of the risk measure, i.e., a higher value of risk measure represents a lower exposure to risk.

We consider three monetary risk measures common in RL. Perhaps the most well-known measure is Value-at-Risk (VaR), which is defined for a risk-level α ∈ [0 , 1] and a random variable ˜ x ∈ X in modern literature as (e.g., Follmer and Schied 2016; Shapiro et al. 2014)

Note that VaR 1 [˜ x ] = ∞ . The equality between the two definitions holds, for example, by Follmer and Schied (2016, remark A.20).

Another popular risk measure is the Conditional-value-at-Risk (CVaR), which is defined for a risk level α ∈ [0 , 1] and a random variable ˜ x ∈ X distributed as ˜ x ∼ q as (e.g., Follmer and Schied 2016, definition 11.8 and Shapiro et al. 2014, eq. 6.23)

with CVaR 0 [˜ x ] = ess inf[˜ x ] and CVaR 1 [˜ x ] = E [˜ x ] . The equality above follows from standard conjugacy arguments for finite probability spaces (Follmer and Schied, 2016). Note that our CVaR definition applies to ˜ x that represents rewards and assumes that a higher value of the risk measure is preferable to a lower value. Other CVaR formulations exist in the literature, but they induce identical preferences for appropriately chosen rewards and risk level α .

Finally, the entropic value at risk (EVaR), with EVaR 0 [˜ x ] = ess inf[˜ x ] and EVaR 1 [˜ x ] = E [˜ x ] , is defined for α ∈ (0 , 1] as (Ahmadi-Javid, 2012)

where KL is the standard KL-divergence defined for each x , y ∈ ∆ m as KL( x ∥ y ) = ∑ ω ∈ Ω x ω log ( x ω / y ω ) . This definition is valid only when x is absolutely continuous with respect to y , which is denoted as x ≪ y and corresponds to y ω = 0 ⇒ x ω = 0 for each ω ∈ Ω .

Risk Averse MDPs AMarkov decision process (MDP) is a sequential decision model that underlies most of RL (Puterman, 2005). We consider finite MDPs with states S = { s 1 , . . . , s S } and actions A = { a 1 , . . . , a A } . After taking an action in a state, the agent transitions to the next state according to a transition probability function p : S × A → ∆ S such that p ( s, a, s ′ ) represents the transition probability from s ∈ S to s ′ ∈ S after taking a ∈ A . We use p s,a = p ( s, a, · ) ∈ ∆ S to denote the vector of transition probabilities. The initial state ˜ s 0 is distributed according to ˆ p ∈ ∆ S . To avoid divisions by 0 that are not central to our claims, we assume that ˆ p s > 0 for each s ∈ S . Finally,

the reward function is r : S × A × S → R , where r ( s, a, s ′ ) represents the deterministic reward associated with the transition to s ′ from s after taking an action a .

The most general solution to an MDP is a history-dependent randomized policy π which maps a sequence of observed states and actions s 0 , a 0 , s 1 , a 1 , . . . , s t to a distribution over the next action a t . It is well-known that with risk-neutral objectives, there always exists an optimal stationary-depends only on the last state-deterministic policy (Puterman, 2005). When the objective is risk-averse, like VaR, or CVaR, there may not exist an optimal stationary or deterministic policy. Hence, we use the symbol Π to denote the set of history-dependent randomized policies in the remainder of the paper.

This paper focuses on the finite-horizon objective in which the agent aims to compute policies that optimize the sum of rewards over a known horizon T . We further restrict our attention to the objective with horizon T = 1 . It turns out that having a single time step is sufficient to derive our counterexamples to existing dynamic programs. Moreover, deriving the decompositions with T = 1 makes it possible to avoid technicalities caused by history-dependent policies, which could distract us from the main ideas presented in this work. Our results an be extended to general horizons T > 1 and the discounted infinite-horizon objectives using standard techniques (Chow et al., 2015).

With horizon T = 1 , the set of randomized history-dependent policies is Π = { π : S → ∆ A } . The symbol π ( s, a ) denotes the probability of action a in a state s , and π ( s ) = π ( s, · ) ∈ ∆ A denotes the A -dimensional vector of action probabilities in a state s . Given a risk measure ψ with a risk level α ∈ [0 , 1] , the finite-horizon risk-averse value of a policy π ∈ Π is computed as

where the superscript in ψ ˜ a ∼ π (˜ s ) α specifies the distribution of the random action. Throughout the paper, we generally use ˜ s to denote the random state at time t = 0 and ˜ s ′ to denote the random state at time t = 1 . In risk-neutral objectives, when ψ = E , one can use the tower property of the expectation operator and define a value function v t for each time step t (Puterman, 2005), but this property does not hold in most static risk measures (Hau et al., 2023). The term policy evaluation in the remainder of the paper refers to computing the value in (4).

The goal in an MDP is to compute an optimal value function and a policy that attains it. In risk-averse MDPs, this goal is formalized as the following risk-averse optimization

with the optimal policy π ⋆ being any policy that attains the maximum in (5). As with policy evaluation, when ψ = E , the optimal value function v ⋆ t can be defined for each time-step t (Puterman, 2005), but this is impossible in general for common risk measures, like VaR and CVaR. The term policy optimization in the remainder of the paper refers to computing the value and the maximizer in (5).

In the remainder of the paper, we study dynamic programming algorithms proposed to solve the policy evaluation problem in (4) and policy optimization problem in (5). In general, these algorithms build on risk-level decomposition (Pflug and Pichler, 2016) of risk measures to define a value function v π t ( s, α ) for each time step t ∈ [ T ] , state s ∈ S , and risk-level α ∈ [0 , 1] (Chow et al., 2015). The value function represents the risk-adjusted sum of rewards that can be obtained if starting in a state s ∈ S at time t and a risk level α . For example, one would define the value function as v π 1 ( s, α ) = ψ ˜ a ∼ π ( s ) α [ r ( s, ˜ a, ˜ s ′ )] and compute v π 0 using a Bellman operator T π α as v π 0 ( α ) = ( T π v π 1 )( α ) . In riskneutral objectives, the Bellman operator is defined as ( T π ( v π 1 ))( α ) = E [ v π 1 (˜ s, α )] , with α ∈ { 1 } , but in risk-averse formulations the operator definition is more complex. The remainder of the paper discusses the decompositions and the operator for CVaR, EVaR, and VaR risk measures respectively.

3 CVaR: Decomposition Fails in Policy Optimization

In this section, we show that a common CVaR decomposition proposed in Chow et al. (2015) and used to optimize risk-averse policies is inherently sub-optimal regardless of how closely one discretizes the state space. The following proposition represents one of the key results used to decompose the risk measure in multi-stage decision-making.

start

s

1

s

1

, a

1

r

(

s

1

, a

1

, s

1

) =

-

50

r

(

s

1

, a

1

, s

2

) = 100

s

1

, a

2

r

(

s

1

, a

2

,

·

) = 0

s

2

s

2

, a

1

r

(

s

2

, a

1

,

·

) = 10

Proposition 3.1 (lemma 22 in Pflug and Pichler 2016) . Suppose that π ∈ Π and ˜ s ∼ ˆ p , ˜ a ∼ π (˜ s ) , ˜ s ′ ∼ p s,a . Then,

where the state s on the right-hand side is not random and

The notation in Proposition 3.1 differs superficially from lemma 22 in Pflug and Pichler (2016). Specifically, our CVaR is defined for rewards rather than costs, the meaning of our α corresponds to 1 -α in Pflug and Pichler (2016), and we use ξ s = z s ˆ p s as the optimization variable. We include a simple proof of Proposition 3.1 for completeness in Appendix A.1.

The decomposition in Proposition 3.1 is important because it shows that the CVaR evaluation can be formulated as a dynamic program. The theorem shows that CVaR at time t = 0 decomposes into a convex combination of CVaR values at t = 1 . Recursively repeating this process, one can formulate a dynamic program for any finite time horizon T . Because the risk level at t = 1 differs from the level at t = 0 and depends on the optimal ζ , one must compute CVaR values for all (or many) risk-levels α ∈ [0 , 1] at time t = 1 . As a result, the dynamic program includes an additional state variable that represents the current risk level.

Chow et al. (2015) proposed to adapt the decomposition in Proposition 3.1 to policy optimization as

They used the decomposition in (8) to formulate a dynamic program with the current risk level as an additional state variable. We prove in the following theorem that the second equality in (8) marked with question marks is false in general.

Theorem 3.2. There exists an MDP and a risk level α ∈ [0 , 1] such that

Before proving Theorem 3.2, we discuss its implications. First, Theorem 3.2 contradicts theorems 5 and 7 in Chow et al. (2015) and shows that their algorithm is inherently sub-optimal regardless of the resolution of the discretization. Theorem 3.2 also contradicts the optimality of the accelerated dynamic program proposed in Stanko and Macek (2019). The result of (Chow et al., 2015) was exploited as is in Chapman et al. (2019), Ding and Feinberg (2022), and Jin et al. (2019) to propose DP reductions, and extended, without proof, in (Rigter et al., 2021) to the context of a Bayesian MDP.

Finally, it is important to emphasize that Theorem 3.2 only applies to the policy optimization setting and does not contradict Proposition 3.1, which holds for the evaluation of policies that assign the same action distribution to each history of states and actions (i.e., policies that are independent of the hypothesized values of ζ ).

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

10

0

10

20

30

40

ζ

s

1

θ

π

(

ζ

s

1

)

π

1

π

2

Proof. Let α = 0 . 5 and consider the MDP M C in Figure 1. In state s 1 , both actions a 1 and a 2 are available, and in state s 2 , only action a 1 is available. The MDP's rewards are

The transition probabilities in M C are

and the initial distribution is uniform: ˆ p s 1 = ˆ p s 2 = 0 . 5 .

To simplify the notation, we define θ π : Z C → R for each π ∈ Π and ζ ∈ Z C as

Because CVaR is convex in the distribution (Delage et al., 2019) and any distribution for r (˜ s, ˜ a, ˜ s ′ ) obtained from a policy π ∈ Π is a mixture of the distributions of r (˜ s, a 1 , ˜ s ′ ) and r (˜ s, a 2 , ˜ s ′ ) , it is sufficient to consider only deterministic policies (there exists an optimal deterministic policy). Thus, we can reformulate the left-hand side of (9) in terms of θ π ( ζ ) as

with π 1 ( s, a 1 ) = 1 -π 1 ( s, a 2 ) = 1 and π 2 ( s, a 2 ) = 1 -π 2 ( s, a 1 ) = 1 , for all s ∈ S . The functions θ π 1 ( · ) and θ π 2 ( · ) are depicted in Figure 2. Similarly, the right-hand side of (9) can be expressed using the convexity of CVaR in the distribution by algebraic manipulation as

Using the notation introduced above and the sufficiency of optimizing over deterministic policies only, the inequality in (9) becomes

Figure 2 demonstrates the inequality in (10) numerically, with the rectangle representing the lefthand side maximum and the pentagon representing the right-hand side minimum. The dashed line represents the function ζ ↦→ max π ∈{ π 1 ,π 2 } θ π ( ζ ) .

To show the strict inequality in (10) formally, we evaluate the functions θ π 1 ( · ) and θ π 2 ( · ) for MDP M C . The function θ π 2 ( · ) is linear because the CVaR applies to a constant, and CVaR is translation invariant. The function θ π 1 ( · ) is piecewise-linear and convex, and its slope can be computed using the subgradient that for each s ∈ S and ˆ ζ ∈ Z C satisfies (Chow et al., 2015)

Simple algebraic manipulation then shows that

and Z C = ∆ S , which implies that ζ s 1 ∈ (0 , 1) . Therefore, by algebraic manipulation, we get the desired strict inequality

where 0 and 4 are represented by the pentagon and rectangle in Figure 2, respectively.

In summary, the decomposition in Proposition 3.1 cannot be exploited in policy optimization because the inequality in the derivation above may not be tight:

where the last equality follows from the interchangeability property of optimization and expected value (Shapiro et al., 2014, theorem 7.92).

Finally, we omit to comment on the validity of the CVaR decomposition in Li et al. (2022) given that it considers a different measure than the CVaR defined in Equation (1). Namely, their measure takes the form:

which is not a coherent risk measure.

4 EVaR: Decomposition Fails for Policy Evaluation

In this section. we show that a decomposition for EVaR proposed in Ni and Lai (2022) is inexact even when considering the policy evaluation setting. Ni and Lai (2022) recently proposed a decomposition of EVaR for a fixed π ∈ Π with ˜ a ∼ π (˜ s ) and a risk level α ∈ (0 , 1] as

where ˜ s ∼ ˆ p , ˜ s ′ ∼ p (˜ s, ˜ a, · ) , and implicit in Ni and Lai (2022)

Note that we use variables ξ s = z s ˆ p s in comparison with z s in Ni and Lai (2022).

The constraint α · ξ ≤ ˆ p in (12) was not stated explicitly in Ni and Lai (2022) but is necessary because EVaR α ′ [ · ] is defined only for α ′ ∈ [0 , 1] . When α ′ = αξ s ˆ p -1 s in (11) it must also satisfy for each s ∈ S that

This additional constraint on ξ implies that Z E ⊆ Z C , for the Z C defined in (7).

We claim in the following theorem (see Appendix A.2 for the proof) that the equality in (11) does not hold even in the policy evaluation setting.

Theorem 4.1. There exists an MDP with a single action and α ∈ (0 , 1] such that

the set Z E defined by (12) .

Theorem 4.1 demonstrates a stronger failure mode than the one in Theorem 3.2 (for CVaR policy optimization) since it applies to both policy evaluation and policy optimization settings.

We propose a correct decomposition of EVaR in the following theorem and employ it to establish that the decomposition in (11) overestimates the actual value of EVaR (see Appendix A.3 for a proof).

Theorem 4.2. Given any finite MDP with horizon T = 1 and α ∈ (0 , 1] , we have that

where

Moreover, EVaR can be upper-bounded as

5 VaR: Decomposition Holds for Policy Evaluation and Optimization

In this section, we discuss a dynamic program decomposition for VaR whose decomposition resembles those for CVaR and EVaR described in Sections 3 and 4. We provide a new proof of the VaR decomposition to elucidate the differences that make it optimal in contrast to CVaR and EVaR decompositions. Our VaR decomposition closely resembles the quantile MDP approach in Li et al. (2022) with a few technical modifications that can significantly impact the computed value.

To contrast the typical definition of VaR with the quantile definition in Li et al. (2022), it is helpful to summarize how VaR is related to the quantile of a random variable. Let q ∈ R define as the α -quantile of ˜ x ∈ X when

In general, the set of quantiles is an interval [ q -˜ x ( α ) , q + ˜ x ( α )] with the bounds computed as (Follmer and Schied, 2016, appendix A.3)

Note that when the distribution of ˜ x is absolutely continuous (atomless), then q + ˜ x ( α ) = q -˜ x ( α ) and the quantile is unique. The following example illustrates a simple setting in which the quantile is not unique.

Example 1 (Bernoulli random variable) . Consider a Bernoulli random variable ˜ e such that ˜ e = 1 and ˜ e = 0 with equal (50%) probabilities. Then, any value q ∈ [0 , 1] is a valid 0 . 5 -quantile because

The objective in Li et al. (2022) is to maximize the quantile operator Q α : X → R defined for a reward random variable ˜ x ∈ X and a risk level α ∈ [0 , 1] as

The quantile operator Q α and VaR differ in which quantile of the random variable they consider:

As a result, the quantile MDP objective in (16) coincides with the VaR value only when the quantile is unique, which is not always the case, as shown in Example 1.

Theorem 5.1. Let ˜ y : Ω → [ N ] be a random variable distributed as ˆ p = (ˆ p i ) N i =1 with ˆ p i > 0 . Then for any random variable ˜ x ∈ X , we have

where we interpret the minimum to evaluate to ∞ if all terms are infinite, which only occurs if α = 1 .

Proof. We decompose VaR using the definition in (1) as

We decompose the probability P [˜ x < z ] as a marginal of the conditional probabilities P [˜ x < z | ˜ y = i ] and ˆ p i = P [˜ y = i ] in step (a) and then lower-bound them by an auxiliary variable ζ i in step (b). In step (c) we replace the joint supremum over z and ζ by sequential suprema, and then we replace the supremum of an intersection by the minimum of the suprema of sets in (d). The equality in (d) holds because P [˜ x < z ] is monotone and, therefore, the sets { z ∈ R | P [˜ x < z | ˜ y = i ] ≤ ζ i } are nested. Finally, step (e) follows from the definition of VaR in (1).

Focusing on the finite MDP with horizon T = 1 , we can show that the decomposition proposed in Theorem 5.1 is amenable to policy optimization. The main difference between the VaR decomposition and CVaR is that the former VaR was expressed as a supremum instead of an infimum over quantile levels ζ . For VaR, changing the order of maximum ( π ) and supremum ( ζ ) does not suffer from a potential gap, but changing the order of maximum ( π ) and infimum/minimum ( ζ ) in CVaR does suffer from such a gap as shown in Theorem 3.2.

The following theorem (proved in Appendix A.4) summarizes the decomposition for VaR.

Theorem 5.2. Given any finite MDP with horizon T = 1 and α ∈ [0 , 1] , we have

For completeness, Appendix C extends Theorem 5.2 to a setting with horizon T > 1 , providing dynamic programming equations and a definition of the optimal policy. These equations are analogous to the equations presented in (Li et al., 2022) yet we obtain them using the accurate definition and decomposition of VaR α .

We finally present below a valid decomposition for the lower quantile MDP, used officially as the objective in (Li et al., 2022) (see Appendix A.5 for a proof).

Proposition 5.3. Given any finite MDP with horizon T = 1 and some α ∈ [0 , 1] , we have that:

We note that the difference with the result presented in (Li et al., 2022) resides in the constraint imposed on ζ that replaces the weak inequality with a strict one. In fact, this strict versus weak inequality is the main distinguishing factor between the decompositions for the lower and upper quantiles.

6 Conclusion

This paper shows that a popular decomposition approach to solving MDPs with CVaR and EVaR objectives is suboptimal despite the claims to the contrary. This suboptimality arises from a saddlepoint gap when optimizing policy . We also prove that a similar decomposition approach is optimal for policy optimization and evaluation when solving MDPs with the VaR objective. The decomposition is optimal because VaR does not involve the same saddle point problem as CVaR and EVaR.

Our findings are significant because practitioners who make risk-averse decisions in high-stakes scenarios need to have confidence in the correctness of the algorithms they use. Our work raises awareness that popular static CVaR and EVaR MDP algorithms are suboptimal, and their analyses are inaccurate. We hope the results we present in our paper will increase the scrutiny of dynamic programming methods for risk-averse MDPs and motivate research into alternative approaches, such as the parametric dynamic programs.

Acknowledgments

We thank Yinlam Chow for his insightful comments that inspired this paper's topic and results. The work in the paper was supported, in part, by NSF grants 2144601 and 2218063. In addition, this work was performed in part while Marek Petrik was a visiting researcher at Google Research. Finally, E. Delage acknowledges the support from the Canadian Natural Sciences and Engineering Research Council and the Canada Research Chair program [Grant RGPIN-2022-05261 and CRC-2018-00105].

References

Mohamadreza Ahmadi, Xiaobin Xiong, and Aaron D Ames. Risk-averse control via CVaR barrier functions: Application to bipedal robot locomotion. IEEE Control Systems Letters , 6:878-883, 2021.

Ahmadi-Javid. Entropic Value-at-Risk: A new coherent risk measure. Journal of Optimization Theory and Applications , 155(3):1105-1123, 2012.

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical Finance , 9(3):203-228, 1999.

Basel Committee on Banking Supervision. Minimum capital requirements for market risk. In Basel III: international regulatory framework for banks , 2019.

Nicole Bäuerle and Jonathan Ott. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research , 74(3):361-379, 2011.

Kang Boda, Jerzy A Filar, Yuanlie Lin, and Lieneke Spanjers. Stochastic target hitting time and the problem of early retirement. IEEE Transactions on Automatic Control , 49(3):409-419, 2004.

Margaret P Chapman, Jonathan Lacotte, Aviv Tamar, Donggun Lee, Kevin M Smith, Victoria Cheng, Jaime F Fisac, Susmit Jha, Marco Pavone, and Claire J Tomlin. A risk-sensitive finite-time reachability approach for safety of stochastic dynamic systems. In American Control Conference (ACC) , pages 2958-2963. IEEE, 2019.

Margaret P Chapman, Riccardo Bonalli, Kevin M Smith, Insoon Yang, Marco Pavone, and Claire J Tomlin. Risk-sensitive safety analysis using conditional value-at-risk. IEEE Transactions on Automatic Control , 67(12):6521-6536, 2022.

Arnav Choudhry, Brady Moon, Jay Patrikar, Constantine Samaras, and Sebastian Scherer. CVaRbased flight energy risk assessment for multirotor uavs using a deep energy model. In IEEE International Conference on Robotics and Automation (ICRA) , pages 262-268, 2021.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in MDPs. In Neural Information Processing Systems (NIPS) , 2014.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: A CVaR optimization approach. In Neural Information Processing Systems (NIPS) , 2015.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research , 18: 1-51, 2018.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . Wiley-Interscience, 2 nd edition, 2006.

Erick Delage, Daniel Kuhn, and Wolfram Wiesemann. 'Dice'-sion-making under uncertainty: When can a random decision reduce risk? Management Science , 65(7):3282-3301, 2019.

Rui Ding and Eugene Feinberg. CVaR optimization for MDPs: Existence and computation of optimal policies. ACM SIGMETRICS Performance Evaluation Review , 50(2):39-41, 2022.

Rui Ding and Eugene A. Feinberg. Sequential Optimization of CVaR, February 2023.

Jerzy A. Filar, Dmitry Krass, and Keith W. Ross. Percentile Performance Criteria For Limiting Average Markov Decision Processes. IEEE Transactions on Automatic Control , 40(1):2-10, 1995.

Hans Follmer and Alexander Schied. Stochastic Finance: Introduction in Discrete Time . De Gruyter Graduate, fourth edition, 2016.

Astghik Hakobyan and Insoon Yang. Wasserstein distributionally robust motion control for collision avoidance using conditional value-at-risk. IEEE Transactions on Robotics , 38(2):939-957, 2021.

Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted MDPs. In Artificial Intelligence and Statistics (AISTATS) , 2023.

I Ge Jin, Bastian Schürmann, Richard M Murray, and Matthias Althoff. Risk-aware motion planning for automated vehicle among human-driven cars. In American Control Conference (ACC) , pages 3987-3993, 2019.

Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a CVaR policy. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 4436-4443, 2020.

Ümit Emre Köse. Optimal timing of living-donor liver transplantation under risk-aversion . PhD thesis, Bilkent Universitesi (Turkey), 2016.

Xiaocheng Li, Huaiyang Zhong, and Margaret L. Brandeau. Quantile Markov decision processes. Operations Research , 70(3):1428-1447, 2022.

Yuanlie Lin, Congbin Wu, and Boda Kang. Optimal models with maximizing probability of first achieving target value in the preceding stages. Science in China Series A: Mathematics , 46: 396-414, 2003.

Seungki Min, Ciamac C Moallemi, and Costis Maglaras. Risk-sensitive optimal execution via a conditional value-at-risk objective. arXiv preprint arXiv:2201.11962 , 2022.

Xinyi Ni and Lifeng Lai. Risk-sensitive reinforcement learning via Entropic-VaR optimization. In Asilomar Conference on Signals, Systems, and Computers , pages 953-959. IEEE, 2022.

Georg Ch Pflug and Alois Pichler. Time-consistent decisions and temporal decomposition of coherent risk functionals. Mathematics of Operations Research , 41(2):682-699, 2016.

A Prashanth and Michael Fu. Risk-sensitive reinforcement learning via policy gradient search. Foundations and Trends in Machine Learning , 15(5):537-693, 2022.

Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley-Interscience, 2005.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse Bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems , 34:1142-1154, 2021.

Shapiro, D. Dentcheva, and A. Ruszczynski. Lectures on Stochastic Programming: Modeling and Theory . SIAM, 2014.

Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic programming. Operations Research Letters , 45(4):377-381, 2017.

Vishnu D Sharma, Maymoonah Toubeh, Lifeng Zhou, and Pratap Tokekar. Risk-aware planning and assignment for ground vehicles using uncertain perception from aerial vehicles. In International Conference on Intelligent Robots and Systems (IROS) , pages 11763-11769. IEEE, 2020.

Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A CVaR optimization approach. In International Joint Conference on Computational Intelligence , pages 412-423, 2019.

Congbin Wu and Yuanlie Lin. Minimizing risk models in Markov decision processes with policies depending on target values. Journal of mathematical analysis and applications , 231(1):47-67, 1999.

Huan Xu and Shie Mannor. Probabilistic goal Markov decision processes. In International Joint Conference on Artificial Intelligence , 2011.

Huaiyang Zhong. Decision Making for Disease Treatment: Operations Research and Data Analytic Modeling . Stanford University, 2020.

A Proofs

A.1 Proof of Proposition 3.1

Suppose that α > 0 ; the decomposition for α = 0 holds readily because CVaR 0 [˜ x ] = ess inf[˜ x ] .

To streamline the notation, Define a random variable ˜ x = r (˜ s, ˜ a, ˜ s ′ ) over a random space Ω = S × A × S with a probability distribution q ∈ ∆ m such that q s,a,s ′ = ˆ p s · π ( s, a ) · p ( s, a, s ′ ) . The value x is the vector representation of the random variable ˜ x and ξ s = ξ s, · , · ∈ R S · A for ξ ∈ R m is a vector that corresponds to the subset of the elements of Ω in which the first element is some s ∈ S . The vectors x s = x s, · , · ∈ R S · A and q s = q s, · , · ∈ R S · A are defined analogously to ξ s .

Starting with the CVaR definition in (2) and introducing an auxiliary variable ζ we get that

In the derivation above, we replaced the infimum by a minimum because Ω is finite, introduced a new variable ζ , derived implied constraints on ζ , and then dropped superfluous constraints on ξ . Continuing with the derivation above and noticing that the constraints on each ξ s are independent given ζ , we get that

The step (a) follows from the interchangeability principle (Shapiro et al., 2014, theorem 7.92), and the step (b) follows by substituting ξ s,a,s ′ = ζ s χ a,s ′ taking care when ζ s = 0 and multiplying both sides of the inequality by ˆ p -1 s > 0 . Finally, in step (c), the random variable ˜ x = r (˜ s, ˜ a, ˜ s ′ ) conditional on ˜ s = s is distributed according to q s,a,s ′ ˆ p -1 s and the equality follows from the definition of CVaR in (2).

A.2 Proof of Theorem Theorem 4.1

Consider an MDP M E depicted in Figure 3 with S = { s 1 , s 2 } and A = { a 1 } and a reward function r ( s 1 , a 1 , · ) = 1 and r ( s 2 , a 1 , · ) = 0 . We abbreviate the rewards to r ( s 1 ) and r ( s 2 ) because they only depend on the originating state. The initial distribution is ˆ p s 1 = ˆ p s 2 = 0 . 5 . We finally let α = 0 . 75 .

Because Z E ⊆ Z C , the right-hand side of (13) can be lower-bounded by CVaR as

The first equality holds from the positive homogeneity and cash invariance properties of EVaR, and the last equality follows from the dual representation of CVaR (Follmer and Schied, 2016).

Because EVaR α [˜ x ] ≤ CVaR α [˜ x ] for each α ∈ [0 , 1] and ˜ x ∈ X (see (Ahmadi-Javid, 2012, proposition 3.2)), we can further lower-bound (19) as

Therefore, (13) holds with an inequality.

To prove by contradiction that the inequality in (13) is strict, suppose that

Equalities (21) and (20) imply that EVaR α [ r (˜ s, a 1 , ˜ s ′ )] = CVaR α [ r (˜ s, a 1 , ˜ s ′ )] which is false in general (Ahmadi-Javid, 2012).

We now show that EVaR does not equal CVaR even for the categorical distribution of ˜ s . The CVaR of the return in M E reduces from (2) to

Since 1 -α = 0 . 25 < 0 . 5 = ˆ p s 1 , then the optimal ξ ⋆ in (22) is

Since KL( ξ ⋆ ∥ ˆ p ) < -log α. we have that ξ ⋆ is in the relative interior of the EVaR feasible region in (3), and, therefore, there exists an ϵ > 0 such that

which proves the desired inequality.

A.3 Proof of Corollary 4.2

We start by proposing a new decomposition for EVaR.

Proposition A.1. Given a random variable ˜ x ∈ X and a discrete variable ˜ y : Ω →N = { 1 , . . . , N } , with probabilities denoted as { ˆ p i } N i =1 , for any α ∈ (0 , 1] we have that

where

Proof. Let q denote the joint probability distribution of ˜ x and ˜ y . The proof exploits the chain rule of relative entropy (e.g., Cover and Thomas (2006, theorem 2.5.3)), which states that for any probability distributions η , q ∈ ∆ Ω with η ≪ q that

where the conditional relative entropy is defined as

with E η [ f (˜ x, ˜ y )] as a shorthand notation to indicate that (˜ x, ˜ y ) ∼ η . We can now decompose EVaR from its definition in (3) as

Here, we decompose the relative entropy of η and q using (23) in step (a) and then use the tower property of the expectation operator in the next step. In step (b), we introduce a variable ζ i for each realization of ˜ y = i with i ∈ N to decouple the influence of η (˜ x | ˜ y ) , under each ˜ y , in the inequality constraint. Finally, we replace the conditional EVaR definition by solving for η (˜ x | ˜ y ) for a given ζ in step (c), and representing η (˜ y ) using ξ .

The first part of our theorem follows directly from Proposition A.1. Suppose that α > 0 ; the result follows for α = 0 because EVaR 0 [ · ] reduces to ess inf . Then, the second part of the corollary holds as

The first inequality follows from adding a constraint on the pairs on the ξ considered by the infimum. The second inequality follows by fixing ζ s = ˆ ζ s with ˆ ζ s = αξ s ˆ p -1 s for each s ∈ S . This is an upper bound because ˆ ζ s is feasible in the infimum:

The value ˆ ζ s is well-defined since ˆ p s > 0 and the constraint ξ ≤ α -1 ˆ p ensures that ˆ ζ s ≤ 1 . Also, we can relax the constraint ζ s > 0 ⇒ ξ s > 0 to ξ s ≥ 0 because EVaR 0 [˜ x ] = lim α → 0 EVaR α [˜ x ] , and, therefore, the infimum is not affected. Finally, the inequality in the corollary follows immediately by further upper bounding the decomposition above by adding a constraint.

A.4 Proof of Theorem 5.2

The equality develops from Theorem 5.1 as

where we first change the order of maximum and supremum, followed by changing the order of max π min s with min s max π . The latter is a direct consequence of the interchangeability property of the maximum operation (Shapiro, 2017, proposition 2.2).

A.5 Proof of Theorem 5.3

The proof mainly relies on correcting the decomposition of lower quantile proposed in (Li et al., 2022).

Proposition A.2. Given an ˜ x ∈ X , suppose that a random variable ˜ y : Ω → N = { 1 , . . . , N } is distributed as ˆ p = (ˆ p i ) N i =1 with ˆ p i > 0 . Then:

where we interpret the supremum to be minus infinity if its feasible set is empty, which only occurs if α = 0 .

Proof. First, we decompose the lower quantile using its definition as

We decompose the probability P [˜ x < z ] in terms of the conditional probabilities P [˜ x < z | ˜ y = i ] in step (a) and then lower-bound them by an auxiliary variable ζ i in step (b). In step (c), we exploit the following equivalence:

The direction ⇐ in the equivalence follows from the definition of Q ζ i (˜ x | ˜ y = i ) :

The direction ⇒ follows from the definition of VaR (see equation (1)), which implies that VaR upper-bounds any z that satisfies the left-hand condition:

̸

yet Q ζ i (˜ x | ˜ y = i ) = z otherwise since P [˜ x < z | ˜ y = i ] is right continuous, there must exist some ϵ > 0 for which P [˜ x < z + ϵ | ˜ y = i ] < ζ i hence:

which leads to a contradiction. In step (e), we solve for z . Finally, we obtain the form in (24).

start

s

1

s

1

, a

1

-

M

0

.

75

M

0

.

25

s

1

, a

2

0

s

1

, a

3

-

100

0

.

5

400

0

.

5

p

s

1

s

2

s

2

, a

1

200

p

s

2

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

200

-

100

0

100

200

α

Value

CVaR

π

/star

α

[

r

(˜

s,

˜

a,

˜

s

′

)]

ˆ

v

α

(

s

0

)

CVaR

ˆ

π

α

[

r

(˜

s,

˜

a,

˜

s

′

)]

B CVaR Suboptimal Policy

In this section, we construct a simple MDP to demonstrate that the suboptimality of the CVaR decomposition discussed in Section 3 can also lead to computing a suboptimal policy. First, we give a particular example in which the policy computed according to Chow et al. (2015) is suboptimal. Then, we show that the sub-optimality gap can be arbitrarily large.

To demonstrate the suboptimality of the policy computed in Chow et al. (2015), consider an MDP with two states and three actions and a horizon T = 1 as shown in Figure 4. Also let M = 600 and

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

100

200

300

ζ

s

1

θ

π

(

ζ

s

1

)

a

1

a

2

a

3

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

200

-

100

0

100

200

M

= 600

, p

s

2

= 0

.

5

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1000

-

500

0

500

M

= 3000

, p

s

2

= 0

.

5

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

200

-

100

0

100

200

M

= 600

, p

s

2

= 0

.

75

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1000

-

500

0

500

M

= 3000

, p

s

2

= 0

.

75

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

200

-

100

0

100

200

M

= 600

, p

s

2

= 0

.

9

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1250

-

1000

-

750

-

500

-

250

0

250

M

= 3000

, p

s

2

= 0

.

9

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

100

0

100

200

M

= 600

, p

s

2

= 0

.

99

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

900

-

600

-

300

0

M

= 3000

, p

s

2

= 0

.

99

p s 1 = p s 2 = 0 . 5 . An optimal solution for the s 1 sub-problem in this example is

Figure 5 depicts the sub-optimality demonstrated by the CVaR decomposition. CVaR decomposition for policy optimization would over-approximate (black dash) the value function and commit to a sub-optimal solution (red solid). This simple example answers the following important concerns on the suboptimality.

First, CVaR policy optimization decomposition in Chow et al. (2015) can lead one to choose a suboptimal policy . In particular, Figure 5 shows that the CVaR policy optimization decomposition is an overestimate of the return and the decision maker commits to the worst action for CVaR objective α ∈ (0 . 25 , 625) . Furthermore, action a 3 is never chosen for (Chow et al., 2015) even though it is the only optimal action for α ∈ (0 . 375 , 0 . 6875) . We can see from Figure 6 when we inspect the ζ function for α = 0 . 5 that a 3 is optimal with CVaR return of 50 . However, the CVaR decomposition would select a 1 or a 2 and over-estimate the CVaR return to 100 , but the sub-optimal actions a 1 or a 2 would have the true CVaR return of 0 instead.

Second, there is no upper bound on the sub-optimality of the action chosen with respect CVaR return of the optimal action. The regret of the sub-optimal return is dependent on the distribution of the sub-optimal action and unbounded with respect to the true optimal return, as the performance of the sub-optimal return for α ∈ (0 . 25 , 0 . 5) is worse than the worst case scenario of the best worst-case policy. Increasing the magnitude ( M > 400 ) for the reward instead of the sub-optimal action a 1 in s 1 in Figure 4 increases the sub-optimality and can be arbitrarily large.

Third, the sub-optimality can occur for any α ∈ (0 , 1) . As above, we use the MDP in Figure 4 and perturb two values to demonstrate the suboptimality by (i) increasing the magnitude ( M > 400 ) for the reward of the sub-optimal action s 1 , a 1 , or (ii) increasing the initial state probability ( p s 2 ). That leads to increasing the range where the CVaR decomposition for policy optimization yields a sub-optimal policy as shown in Figure 7.

C Extension to Longer Horizon

For completeness, we include in this section an extension of the corrected decomposition presented in Theorem 5.2 to horizon T > 1 . Our demonstration establishes that the decomposition proposed in Theorem 1 and 3 of (Li et al., 2022) actually applies to the upper quantile (i.e. value-at-risk) of the cumulated reward rather than the lower one as claimed by the authors. In comparison with the proof found in (Li et al., 2022), we pay close attention to demonstrating that the supremum in (5.1) is attained when finite and that deterministic policies are optimal.

Given any finite MDP with horizon T > 0 and α ∈ [0 , 1] , we define the initial value as

The state-action value function q t : S ×A× [0 , 1] → R for the terminal step t = T +1 and α ∈ [0 , 1] and each s ∈ S and a ∈ A is defined as

For each t = 1 , . . . , T and α < 1 the state-action function is defined recursively as

For α = 1 , the state-action value function is defined as

To construct the optimal policy, we also need to define a history-dependent risk level ¯ α t : S t ×A t -1 → [0 , 1] that satisfies for each t = 1 , . . . , T and s 1 , . . . , s t and a 1 , . . . , a t -1 that

The appropriate values ¯ α t can be readily recovered from the optimal solution to (26). Finally, letting

we construct a deterministic history-dependent policy ¯ π t : S t ×A t -1 →A for each t = 1 , . . . , T and s 1 , . . . , s t and a 1 , . . . , a t -1 that puts all the probability mass on a ∗ t ( s t , ¯ α t ( s 1: t , a 1: t -1 ))) , i.e.

The following theorem states that the value function defined above represents the VaR of the returns of the optimal policy. Moreover, the history-dependent policy ¯ π above is optimal and attains the optimal VaR return.

Theorem C.1. For a finite horizon T and any α ∈ [0 , 1] we have that v 0 defined in (25) coincides with the optimal VaR return:

Moreover, this optimal return is attained by the policy (¯ π t ) T t =1 defined in (27) :

In proving Theorem C.1, we will make use of the upper semicontinuity property of VaR α [˜ x ] , which is presented in the following Lemma for completeness.

Lemma C.2. The function VaR α [˜ x ] is upper semicontinuous in α on the interval α ∈ [0 , 1] .

Proof. This follows from the fact that VaR α [˜ x ] is non-decreasing and right-continuous in terms of α (see Lemma A.19 in (Follmer and Schied, 2016)).

Proof of Theorem C.1. The proof consists of two steps addressing the two equalities stated by the theorem.

Step 1: For all t = 1 , . . . , T , we let

where Π t +1: T refers to the set of all history-dependent policies starting from time t + 1 , and where VaR α [˜ x | s t , a t ] is short for VaR α [˜ x | ˜ s t = s t , ˜ a t = a t ] . We will first show how ˆ q t ( s, a, α ) = q t ( s, a, α ) for all t = 1 , . . . , T , s ∈ S , a ∈ A , and α ∈ [0 , 1] . We follow with our conclusion about v 0 .

First, addressing the case α = 1 , one can easily see that for all t = 1 , . . . , T , s ∈ S , and a ∈ A , we have that

based on our definition of value-at-risk for α = 1 .

Next, for 0 ≤ α < 1 and using p short for p sa when can start looking at time t = T , where we have that for all s T ∈ S , and a T ∈ A :

where we first use the definition of ˆ q T ( s, a, α ) , then the decomposition of VaR in Theorem 5.1. We follow with confirming, based on extreme value theory, that the supremum over ζ must be achieved given that VaR α (˜ x ) of a random variable ˜ x is upper semi-continuous (usc) in α (see Lemma C.2), that the minimum of a finite set of usc functions is usc, and that ∆ S is compact. The other two steps follow from the translation invariance of VaR and the definition of q T +1 ( s, a, α ) .

Moreover, for all 1 ≤ t < T let ˜ h t : t ′ := (˜ s t : t ′ , ˜ a t : t ′ -1 ) as the history between t and t ′ . Using the short-hand ˜ r t ′ = r (˜ s t ′ , ˜ a t ′ , ˜ s t ′ +1 ) , and when 0 ≤ α t < 1 , we have that:

where, as before, we start by exploiting the decomposition Theorem 5.1 and explicating that the supremum is attained. We then change the order of the two maximums, followed by changing the order of max π min s ′ with min s ′ max π , which follows based on the interchangeability property of the minimum operation (Shapiro, 2017, proposition 2.2). We finally introduced a ( s, α ) -value function operator which can be reduced as follows:

where we exploited the fact that value-at-risk is a mixture quasi-convex function (Delage et al., 2019), meaning that it cannot be improved by a randomized policy.

Hence, replacing ˆ v back in equation (29), we get

Finalizing our conclusion, we get using the short-hand ˜ r t = r (˜ s t , ˜ a t , ˜ s t +1 ) that

where we employ the same steps as for reducing ˆ q t to q t .

Step 2: Regarding the optimality of the proposed policy, we will show inductively that

is an upper envelope for ˆ q t ( s t , a t , α t ) for all 1 ≤ t ≤ T . Conditioning on α t in the definition above is necessary because the policy ˆ π depends on the risk level α . This can then be exploited to obtain the following inequality:

where the first step comes from the decomposition of VaR , the second from the definition of ¯ π 1 . The third step follows from the feasibility of ζ s := ¯ α 1 ( s ) / ( α ˆ p -1 s ) . We then exploit the definition of ¯ ˆ q 1 , the fact that it upper bounds ˆ q 1 , the equivalence of ˆ q 1 and q 1 , and definition of a ∗ 1 . On the sixth line, we exploit the definition of ¯ α 1 ( s ) , followed with the definition of v 0 . These steps would therefore confirm that ¯ π is optimal.

We are left with showing that ¯ ˆ q t upper bounds ˆ q t for all t = 1 , . . . , T . Specifically, starting with t = T , we have:

Then, for t < T we focus on the case α t < 1 , since otherwise we have ∞≥∞ . Specifically, we first define

which captures the value-at-risk at level α t of the reward-to-go looking forward from time t when running ¯ π given that the history up to time t satisfies ¯ α t (˜ s 1: t , ˜ a 1: t -1 ) = α t .

We then take inductive steps starting from t = T down to t = 2 . At each t , we can verify that if ¯ ˆ q t is an upper envelope for ˆ q t , then :

and that for 0 ≤ α t < 1 :

where we first used the definition of ¯ π , then used the definition of ¯ ˆ q T , followed with the assumed property that ¯ ˆ q t is an upper envelope for ˆ q t . We then employ the equivalence of ˆ q t and q t twice, the definition of a ∗ t , and the relationship between ˆ v t and ˆ q t established in (30).

Next, we confirm that if ¯ v t is an upper envelope for ˆ v t , then taking one step back and using the short-hand ˜ r t ′ = r (˜ s t ′ , ˜ a t ′ , ˜ s t ′ +1 ) we get:

The first step comes from the decomposition of VaR , the second from the fact that ¯ α t is known given that ¯ α t -1 , ˜ s t , and ˜ a t -1 are fixed while ∑ T t ′ = t r (˜ s t ′ , ˜ a t ′ , ˜ s t ′ +1 ) only depends on ˜ s t and ¯ α t . The third step follows from replacing the supremum over ζ with a feasible member of ∆ S . The fourth step follows from the definition of ¯ v t , followed with the fact that it upper bounds ˆ v t . Finally, we exploit the definition of g ( · ) and of ˆ q t -1 ( · ) . This completes the inductive proof demonstrating that ¯ ˆ q t ( · ) is an upper envelope for ˆ q t ( · ) for all 1 ≤ t ≤ T .

Peer reviews:
comments: SUMMARY:
The paper shows that the dynamic programming decomposition methods used in risk-averse MDP are suboptimal for CVaR and EVaR, but optimal for VaR. These findings seem important as such decomposition methods become increasingly popular recently.

STRENGTHS:
The findings that the decomposition fails for CVaR and EVaR are of importance and interesting to see.

WEAKNESSES:
The paper presents negative outcomes arising from the application of the decomposition method in CVaR and EVaR MDP. However, it does not offer a solution or rigorous analysis to address or evaluate these shortcomings. Particularly, in Theorem 3.2, the authors demonstrate that there exists a risk level where the equalities in (8) do not hold, indicating that the decomposition is suboptimal. While this finding is valuable, it raises crucial questions. For instance, under which parameter values does the optimality of the decomposition hold? If we randomly select a risk level from an interval, what are the chances of achieving optimal decomposition? Can we establish an upper bound on the loss incurred by using the decomposition? Is there an alternative form of decomposition that yields smaller losses? These questions hold significant relevance and would provide insights into handling the limitations.

Similar questions can be asked regarding the EVaR. Therefore, although the findings are noteworthy, the paper is not yet ready for publication. Further work on addressing the shortcomings of the decomposition method would significantly enhance the paper's quality.

QUESTIONS:
- Please refer to the questions asked above.
- Is there any way to numerically evaluate the suboptimality of the decomposition?

LIMITATIONS:
I do not see any negative societal impact from the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper tackles the risk-level decomposition of risk measures in the finite Markov Decision Process (MDP). The work firstly points out the suboptimality of the decomposition proposed in prior works (Conditional-Value-at-Risk), by mathematically proving that the decomposition in CVaR does not maintain the equality to the optimal policy under policy optimization. 

Then the work mathematically proves the decomposition in Entropy-Value-at-Risk (EVaR) fails to find the minimum for the policy evaluation. Following that, the paper corrects the decomposition of EVaR and proposes a new dynamic program decomposition for Value-at-Risk (VaR). By avoiding the saddle-point gap in policy optimization, the new method remains optimal for both policy optimization and policy evaluation.

STRENGTHS:
The method contributes to identifying the suboptimality in policy optimization and policy evaluation in prior methods (CVaR and EVaR). The paper points out the reason causing the suboptimality and proposes a new method for improvement. As the paper improves the correctness of the policy learning in risk-averse methods, the theoretical result can also be significant when people apply the risk-averse approach in practice.

The figures are clear and help with understanding the counterexamples.

WEAKNESSES:
I understand that the paper mainly focuses on theoretical analysis, and it makes sense to me that there is no empirical evidence for the correctness of the new method. One small suggestion is to add a numerical experiment.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution of the paper is to demonstrate that a popular dynamic programming decomposition for well-known risk metrics only provides suboptimal policies. More precisely, the underlying idea of such decompositions is to operate on an extended state space that incorporates a continuous state variable between 0 and 1 capturing the system risk level at the current stage. The belief in these existing decompositions was that the risk state variable could be finely discretized to approximate the optimal value function arbitrarily. However, the authors show a counter-example for CVaR and EVaR. Finally, they show that VaR does admit such a decomposition with a new proof scheme.\n\nSTRENGTHS:\n+ The paper and proofs are very well-written and structured\n+ Contribution is impactful within the analysis of risk measures\n\nI particularly find that this is an outstanding paper and contribution. First, it closes an open theoretical question on a meaningful and growing body of literature composed of 10+ papers only in these last five years, specifically demonstrating via a relatively simple but ingenious example that discretization is necessarily sub-optimal. The results are intriguing because the discretization assumption was a rather intuitive result, justifying many of the attempted proofs in the most recent literature. The authors also provide corrected decompositions for EVaR and VaR.\n\nI also note that it is also a bit concerning that these results contradict the optimality of many recent works, which are thus worth revisiting in the future. (On that note, I verified the proofs and examples of this paper to the best of my abilities, but did not re-analyze the previous papers' proofs, except to re-check their statements).\n\nWEAKNESSES:\n- Much of the intuition is hidden in the proofs.\n- Lack of discussion of possible research avenues to address this issue.\n\nMy primary concern is that the presentation can be excessively technical in some parts. I believe the proofs are very well written and easy to follow, especially as they are mostly counter-examples, but it would be interesting if authors could bring to light what fails more intuitively (just the saddle point discussion was commented on briefly). In particular, my understanding from the examples is that the discretization violates a certain variant of Von Neummann's minimax theorem due to non-convexity/non-concavity of policy choice, here captured in the strict inequality of equation (10). Perhaps my understanding is not accurate, but nonetheless it would be interesting if authors could expand more on the saddle point argument, perhaps providing directions on conditions for other coherent metrics that do not suffer from the same discretization issue. \n\n\nMinor notes:\n\n- Please kindly use capital letters when naming lemmas and theorems, e.g., Lemma X as opposed to lemma X.\n- In the proof of Prop. 3.1, \"Define\" should be \"define\"; also please define \"ess inf\".\n- In the proof of Theorem 4.2, I am a bit confused with line l.457: \"the first inequality follows from adding a constraint on the pairs.\"\n\nQUESTIONS:\n1. The next natural question is how poor the discretized systems can be. Are there any asymptotic guarantees on its quality, even under some non-zero approximation ratio?\n2. Are there specific coherent metrics where the discretization could still be valid?\n\nLIMITATIONS:\nThe authors have not included any limitation subsection.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors demonstrate that dynamic programming decompositions for CVaR and EVaR can be suboptimal, and show that VaR may not be subject to the same problems in MDPs of horizon 1.\n\nSTRENGTHS:\nThe paper provides some interesting observations conflicting with previous papers on CVaR and EVaR optimization in RL.\n\nWEAKNESSES:\nWhile this paper makes some interesting observations about the decompositions of CVaR, EVaR, and VaR, it seems like an incomplete paper, without enough contributions meriting acceptance. \n\nThe “dynamic program” presented in Section 5 seems far from implementable, with weak (if any) guarantees. For contrast, [Chow 2015] shows that the CVaR Bellman operator is a contraction with a unique fixed point, and also discuss how to discretize the space of $\\alpha$ to make practical implementations possible. They also provide proof-of-concept experiments. The proposed VaR operator seems to face similar issues (e.g., with $\\alpha$ being continuous), but the submitted paper appears to handle none of these points.  It would be interesting to see the decomposition proposed in Theorem 5.2 translated into a policy learning algorithm, e.g., of the style of value iteration proposed in [Chow 2015], complete with theoretical analysis and empirical evaluation.  \n\nAdditionally, Theorems 5.1 and Proposition 5.3 appear to only hold for $T=1$ which is essentially a contextual bandit? While the authors state that results can be extended to $T > 1$ using “standard techniques,” I find that $T = 1$ is an unusual assumption in RL (for MDPs) and usually something that makes risk-sensitive RL much easier (because you don’t have to deal with stochasticity from transitions). In this light, it seems worth it to at least expand Section 5 for $T > 1$.\n\nQUESTIONS:\n- In policy evaluation, will the proposed algorithm need to be able to handle continuous $\\alpha$, and how can that be done? \n\n- Is it possible to learn a policy with $T > 1$, and what would the specific policy learning algorithm look like (in broad strokes)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effectiveness of decomposition approaches for solving risk-averse Markov Decision Processes (MDPs) with Conditional Value-at-Risk (CVaR), Expected Value-at-Risk (EVaR), and Value-at-Risk (VaR) objectives. The goal is to assess the validity and accuracy of these common decomposition techniques for risk-averse decision-making.\n\nMain contributions:\n\nSub-optimality of CVaR and EVaR decomposition: The paper demonstrates that the widely used decomposition approaches for CVaR and EVaR objectives are inherently suboptimal, invalidating previous works. These methods involve a saddle-point gap during policy optimization, leading to incorrect results. As a result, practitioners should exercise caution when using these decomposition techniques for risk-averse MDPs.\n\nOptimal VaR decomposition: In contrast to the suboptimal CVaR and EVaR decompositions, the paper identifies the VaR decomposition as an optimal approach for both policy evaluation and optimization in risk-averse MDPs. The VaR decomposition does not suffer from the same saddle-point problem, making it a reliable method for risk-averse decision-making.\n\nIncreased awareness and call for alternative approaches: The findings highlight the limitations of two traditional decomposition methods, CVaR and EVaR, and emphasize the need for scrutiny when using these decompositions. Researchers are encouraged to explore alternative approaches, such as parametric dynamic programs, to improve risk-averse decision-making in MDPs.\n\nSTRENGTHS:\nRigorous analysis: The main strength of the paper is its rigorous treatment of three decomposition approaches for risk-averse MDPs. \n\nSignificant and clearly stated contribution for the field: The paper shows the sub-optimality of CVaR and EVaR decomposition methods invalidating several previous works on the topic. This finding is valuable as it warns practitioners about unknown limitations of these techniques in risk-averse decision-making.\n\nOptimal VaR decomposition: By identifying the VaR decomposition as an optimal approach for policy evaluation and optimization in risk-averse MDPs, the paper provides a practical solution to the limitations of CVaR and EVaR. Thus this VaR decomposition is a reliable alternative to these two methods.\n\nEncouraging further research: In light of the limitations faced by CVaR and EVaR, the paper calls for exploring alternative approaches, such as parametric dynamic programs.\n\nClarity in presentation and methodology: The paper is well-written. It uses clear explanations, comprehensive mathematical analysis, and compelling illustrative examples.\n\nWEAKNESSES:\nLimited set of approaches: Perhaps the main weakness of the paper is the limited set of (three) decomposition approaches being scrutinized. While the paper highlights the sub-optimality of CVaR and Expected Value-at-Risk EVaR methods, it does not compare a broader range of state-of-the-art techniques. This limitation makes unclear the extent to which VaR decomp Google Mapsosition can be considered optimal. I appreciate that it would not be possible, given length constraints and scope of the paper, to evaluate other baselines comprehensively. However, it would be valuable if the authors could mention other state-of-the-art techniques in a related works section, and whether they can say anything about them in relation to the approaches they investigated here. For instance, control as inference and active inference use the KL divergence as a decision-making objective (i.e., a belief-based reward function) which entails risk-averse behaviour, and which might be related to VaR [1,2,3].\n\nUnclear scalability discussion: The paper does not extensively discuss the scalability of the proposed VaR decomposition concerning problem size or dimensionality. I appreciate that this might not be important for theoretically minded readers, however, since the paper has important implications for practitioners (i.e., widespread adoption of VaR) it would be nice to immediately know, via some discussion or suitable reference, how well this method scales as MDPs grow complex and large in applications.\n\nLimited generalization: The paper comprehensively addresses the setting of risk-averse MDPs. It would be nice to know whether the theoretical analyses presented here can offer some insights on more general problem classes like risk averse partially observable Markov decision processes (POMDPs). For example, has VaR been extended to risk averse POMDPs, and if so would you expected it to be optimal in this case?\n\nFrom the two points above, one of the main weaknesses of this paper is that it is not straightforward to infer to what extent VaR is a viable approach in a wide range of practical problems, and thus it is unclear to what extent the paper implies an optimal, promising in the near long-term approach to the risk-averse decision problem. In my understanding, this is the main point of broader impact in the machine learning community, so it should be addressed.\n\nTypos: l 136, l200\n\n[1]\tS. Levine, ‘Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review’, arXiv:1805.00909 [cs, stat], May 2018, Accessed: Dec. 29, 2021. [Online]. Available: http://arxiv.org/abs/1805.00909\n[2]\tL. Da Costa, N. Sajid, T. Parr, K. Friston, and R. Smith, ‘Reward Maximization Through Discrete Active Inference’, Neural Computation, vol. 35, no. 5, pp. 807–852, Apr. 2023, doi: 10.1162/neco_a_01574.\n[3]\tD. Hafner, P. A. Ortega, J. Ba, T. Parr, K. Friston, and N. Heess, ‘Action and Perception as Divergence Minimization’, arXiv:2009.01791 [cs, math, stat], Oct. 2020, Accessed: Nov. 07, 2020. [Online]. Available: http://arxiv.org/abs/2009.01791\n\nQUESTIONS:\nCan the authors elaborate on their selection of the specific decomposition approaches used for comparison? Are there any reasons why these three approaches were chosen over others?\n\nCould the authors provide more insights into the computational complexity and scalability of the VaR decomposition method concerning the size and dimensionality of the risk-averse MDPs, either in a short discussion or references? Are there any notable challenges when applying this approach to large and more complex decision problems?\n\nAs a broader point of interest, how significant is the choice of the risk level (alpha) in the performance and efficiency of the VaR decomposition method? Are there any guidelines on selecting appropriate alpha values based on a given problem characteristics? This is important to understand the robustness of this method in a new risky environment, a desideratum for any risk averse decision-making algorithm in high stakes applications, which relates to the broader impact of this work as hinted in the last point of the conclusion.\n\nGiven that the paper focuses on risk-averse MDPs, are there any indications or early results on how the VaR decomposition approach and its optimality might generalize to other types of decision-making frameworks, such as partially observed problems?\n\nLIMITATIONS:\nThe paper does a good job at addressing its limitations insofar as it focuses on three specific approaches to risk averse decision making.\n\nThe main limitation which has not been addressed, and which in my opinion should be addressed, is giving more context as to why this three approaches were chosen, or why is it is sensible to choose them, and mention the fact that there exist other state-of-the-art approaches, which need to be considered and comprehensively evaluated in the future, and which could serve as avenues for future research in addition to parametric dynamic programs, e.g., control as inference and active inference.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-20"}