Distributional Reinforcement Learning for Risk-Sensitive Policies
Authors: Shiau Hong Lim
ILYAS MALIK
Venue: NeurIPS 2022 Accept (2022)
Keywords: distributional reinforcement learning
cvar
risk-sensitive RL

Abstract: We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that the standard action-selection strategy when applying the distributional Bellman optimality operator can result in convergence to neither the dynamic, Markovian CVaR nor the static, non-Markovian CVaR. We propose modifications to the existing algorithms that include a new distributional Bellman operator and show that the proposed strategy greatly expands the utility of distributional RL in learning and representing CVaR-optimized policies. Our proposed approach is a simple extension of standard distributional RL algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empirically show that our proposed algorithm is able to learn better CVaR-optimized policies.

Distributional Reinforcement Learning for Risk-Sensitive Policies
Shiau Hong Lim IBM Research, Singapore
Ilyas Malik IBM Research, Singapore malikillyas1166@gma... com
shonglimesg. ibm. com
Abstract
We address the problem of learning a risk-sensitive policy based the CVaR risk measure using distributional reinforcement learning. In particular, we show that the standard action-selection strategy when applying the distributional Bellman optimality operator can result convergence to neither the dynamic, Markovian CVaR nor the static, non- Mon-Markovian CVaR. We propose modifications to the existing algorithms that include a new distributional Bellman operator and show that the proposed strategy greatly expands the utility of distributional RL in learning and representing CVaR-optimized policies. Our proposed approach a simple extension of standard distributional RI algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empirically show that our proposed algorithm is able to learn better CVaR- optimized policies.
Introduction
In standard reinforcement learning (RI.) (Sutton and Barto 2018), one seeks to learn a policy that maximizes an objective, usually the expected total discounted rewards or the long-term average rewards. stochastic domains, especially when the level of uncertainty involved is high, maximizing the expectation may not be the most desirable since the solution may have high variance and occasionally performs badly. such scenarios one may choose to learn a policy that more risk- averse and avoids bad outcomes, even though the long-ter average performance is slightly lower than the optimal.
In this work we consider optimizing the conditional value-at-risk (CVaR) (Rockatellar and Uryasev 2000), a popular risk measure, widely used in financial applications, and is increasingly being used in RI. The CVaR objective focuses on the lower tail of the return and is therefore more sensitive to rare but catastrophic outcomes. Various settings and RI. approaches have been proposed solve this problem (Petrik and Subramanian, 2012 Chow and Ghavamzadeh [0144] Chow and Pavone 2014] Tamar et al.) [2015] Tamar et al., 2017 Huang and Haskell 2020). Most of the proposed approaches, however, involve ve more complicated al algorithms than standard RL algorithms such as Q-learning 1992) and its deep variants, e.g. DQN (Mnih et al. 2015).
Watkinn and Dayan
Recently, the distributional approach to RL (Bellemare et al. 2017: Morimura et al. 2010) has received increased attention due its ability learn better policies than the standard approaches in many challenging tasks (Dabney et al.] [00 8a by Yang et al. 2019). Instead of learning value function that provides the expected return each state-action pair, the distributional approach learns the entire return distribution of each state-action pair. The approach itself is simple extension to standard RL and is therefore easy to implement and able to leverage many of the advances in deep RL.
Since the entire distribution is available, one naturally considers exploiting this information optimize for an objective other than the expectation. Dabney et al. (2018a) presented a simple way to
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

do so for family of risk measures including the CVaR. The theoretical properties of such approach, however, are not clear. In particular, is not clear whether the algorithm converges to any particular variant of CVaR-optimal policy. We address this issue in this work.
Our main contribution is to first show that the commonly-usee action-selection strategy for CVaR in distributional RL, proposed by (Dabney et al. 8a) and (Keramati et al. 2020), among others, converges neither the dynamic. Markovian CVaR nor the static CVaR even if the optimal CVaR policy is stationary and Markov. Secondly, we propose a new distributional Bellman operator and show that for the class of stationary CVaR --otimam policies, the optimal value distribution is fixed point, and one can extract this optimal policy through distributional RL. Empirically, we show that the proposed approach learns better policies in terms of the CVaR objective on both synthetic and real-world problems.
We close the introduction section with some references to related works. We formally present problem setup as well as our main analytical results in Section 2 Section describes our proposed algorithm while DectionApreeene our empirical results. Finally, we conclude with discussion on future works.
1.1 Related Works
The literature on distributional RL has been greatly expanded recently (Morimura et al. 2010
Bellemare et al. 2017; Barth et al. 2018:; Dabney al. [018a Yang et al. [211 Most
of these works focus on the modeling aspects, such as the choice of representations for the value
distributions. The approach has been used enhance exploration in RL (Mavrin al. 2019) and in
risk-sensitive applications (Bernhard et al. 2019).
Solving Markov decision processes (MDP) with risk- objectives have been addressed in many works (Howard and Matheson] 1972] Ruszezynski] (2010) Bäuerle and Out 201T). including RL approaches (Borkar] 2001]Tamar Tamar al. 2012 L.A. and Ghavamzadehi 2013). In particular, (Cho
and Ghavamzadeh (2014); Tamar et al. (2015) deal with the static CVaR objectives while Petrik and
Subramanian (2012); Chow and Pavone (2014) deal with the dynamic CVaR objectives. Tamar et al.
(2117) proposed policy-gradien approach that deals with both the static and the dynamic CVRR objectives. Closest to ours is the work by (Stanko and Macek (2019), whose proposed approach also makes use of distributional RL but their policy evaluation step uses Markov action-selection strategy. not clear whether their overall approach properly optimizes either the static or the dynamic CVaR.
2 Problem Setup and Main Results
We consider discrete-time MDP M with state space X' and action space A. For simplicity we assume that X' and A are finite, although our results and algorithm can be readily extended to more general state-action spaces. We assume that the rewards are bounded and drawn from a countable set R. C RR. Given states Zt, Ft+11 E X for any i. E {0, 1,..,,, .. the probability of receiving reward E R. and transitioning to F++1 after executing a E A in dt is given by p(r,, Jt+ [2,,, at). Without loss of generality we assume fixed initial state xog unless stated otherwise. Given a policy π : H - P(A). where H is the set of all histories so far fúg := (ro, ao, To, 3 01, Ti, "I,...,,7y xx.) E H. and P(A) the space of distributions over A, its expected total discounted reward over time is given by
where E
E (0, a, are drawn from (he). The subscript p indicates that the rewards and state transitions are induced by p.
1) is discount factor. The superscript π in the expectation indicates that the actions
In standard RL, we aim to find a policy that maximizes V*, It is well-known that there exists a deterministic stationary policy X + A whose decisions depend only on the current state, that gives optimal Va, and therefore one typically works in the space of stationary deterministic policies. Key to a Syynami--rogrammnin solution to the above problem is the use of value function

---

(²(I, = B Et= 7'r No = 0, = al, which satisfies the Bellman equation
ax,a, a=) - I I. +
(1)
The optimal value (2, == Q= (x, a) for any optimal policy x' satisfies the Bellman optimality equation
Vz,a a, Q a) = Σ p(r, 12,,
r + + max Q"
Furthermore, for any Q-function Q E {q X show that the operator 7 defined by ("Q == Er p(r, p(r,,1< + 70( (()))) is a 7-contraction in the sup-norm [ == maX (Q, a)l with fixed-point satisfying ① One can therefore start with an arbitrary Q-function and repeatedly apply T or its stochastic approximation, to learn Q". An analogous operator T can also be shown to be a -ccottraction with fixed- ixed-point satisfying
( 0o, Var a), can
2.1 Static and Dynamic CVaR
The expected return V is risk-neutral in the sense that it does not take into account the inherent variability of the return. In many application scenarios, one may prefer a policy that is more risk- averse, with better sensitivity to bad outcomes. In this work, we focus the conditional value-at-risk (CVaR), which is a popular risk measure that satisfies the properties of being coherent (Artzner et al.
1999). The o-level CVaR for real-valued random variable Z, for E (0, 1J. is given by (Rockafellar
and Uryasev 2000)
Ca
== max sER
[ Z)+1
a
where (x)+ max(x: 0}. Note that we are concerned with that represents returns (the higher, the better), so this particular version of CVaR focuses on the lower tail of the distribution. In particular, the function s + s [E((s z)+] concave s and the maximum is always attained at the a-level quantile, defined as
9a (Z) inf Pr(Z < 2).
For = 1, C. reduces to the standard expectation. In the case Z is absolutely continuous, we have the intuitive (Z) = E < q..()))
Our target random variable is the total discounted return 2" == Er= y'r of a policy x, and our objective is to find policy that maximizes (Z*). where the optimal CVaR given by
- -E =²
max max R
In the context where Z is accumulated over multiple time steps, the objective B corresponds to maximizing the so-called static CVaR. This objective is inmosincossittent in the sense that the optimal policy may be distory-dependert and therefore -on-Markov. This is, however, perfectly expected since the optimal behavior in the later time steps may depend on how much rewards have been accumulated thus far more risky actions can be taken if one has already collected sufficiently large total rewards, and vice versa. From the point of view of dynamic programming, an alternative, time- consistent or Markovian version of CVaR may be more convenient. A class of such risk measures was proposed by Ruszczyyskii (2010). and we shall refer to this version of CVaR as the dynamic CVaR, defined recursively as
(a = Calrela, = 2, = a). D,,((, = Ca(r:+ D T (Z44))) = a = Yn, x, 0, (T = lim D (x,
VIT, x, a,
Vn, x, a, T> 0,
T
We use slightly different definition from that in Ruszczynski 2010). but conceptually they are essentially the same.

---

It can be shown [uuszzyynkki 2010) that there exists stationary deterministic optimal policy x², maximizing D x, a) for all a, whose dynamic CVaR is given by Da := D In particular, the operator Ta defined by
To D(z, a) = Calre + max ( a )21 =
a]
(4)
of
for D Q is a a-cotraction in sup-norm with fixed-point satisfying Vr,a, a, (z, Ca(re + max D (1 d')( = x, a, a].
(5)
Despite its theoretical properties, the dynamic CVaR is hard to interpret. Moreover, from a practical point of view, it can be overly optimistic in certain cases and overly conservative in other cases. We illustrate with some examples in Section 2.2 In such cases it may be favorable to use the static CVaR, Bäuerle and Out (201 I) suggest iterative process that, in theory, can be used solve for the optimal static CVaR policy. The approach is based on (3)
1. For a fixed 8, one can solve for the optimal policy with respect max, El-(8 -*)*}.
2. For a fixed π,, the optimal is given by the a-level quantile of z².
3. Repeat until convergence.
Step one above can be done by solving an augmented MDP M with states (x. s) E R. where is moving threshold keeping track of the accumulated rewards so far. no rewards (except in terminal states) and state transitions are given by p(0, (x', " a),
I In particular, this MDP has
p(r, jz, a). Solving M directly using model-free RL, however, can result in poor sample efficiency since each example (Ir, a, r') may need to be experienced many times under different threshold Furthermore, there is a question of updating 8. In this work, we propose an alternative solution using the approach of distributional RL.
2.2 Distributional RL
In standard RL, one typically learns the Q" a) value for each (x, a) through some form of
learning (Sutton and Barto 2018). In distributional RL (Bellemare et al. (2077).
one instead tries to learn the entire distribution of possible future return 2 (x: a) for each (ar a). The
Q-value can then be extracted by simply taking the expectation Q" (z; a) EE 2 (x. a)).
The objects of learning are distribution functions U 2 {Z Xx
((R) E E[ a) <
20, Vz, q 2 1}. For any state-action pair (z, a), we use U (z( a) to denote random variable with the respective distribution. Let T be the distributional Bellman operator on Z such that
7"U(z, :2 R (UXX,
where 2 denotes equality in distribution, generated by the random variables R, X' induced by p(r, x' JI,aa) a). We use the notation T instead of T when referring to distributional operator, where U= a) is random variable. Bellemare et al. (2117) show that 7 is a y-contraction in Z in the following distance metric
d(U, V) supw(U(z,a) V(r,a)) z,a
where w is the I-Wasserstein distance between the distributions of U(z a) and V(z, a). Furthermore, the operator 7 defined by
2 R A' = arg E a')]
(6)
can be shown to be y-contraction in Q (not necessarily in 2) in sup-norm under element-wise expectation, i.c.,
|ETU - IT S <|IEU - EV
"The results by Bäuerle and Ou [00 apply generally to any Borel state-action space, with mild technical conditions. Our finite state-action setting special case where such conditions are trivially statisfied. Also, since we assume countable and bounded reward set, the resulting augmented MDP M has countable state space, which simplities the technical presentation of main ideas.

---

where ETU= Q such that In general, T is not expected to be contraction in the space of distributions 2 for the obvious reason that multiple optimal policies can have very different distributions of the total return even though they all have the same expected total return.
a)). and EU, EV, ETV all similarly defined.
Since one keeps the full distribution instead of just the expectation, natural way to exploit this is to extract more than just the expectation from each distribution. In particular, in (6). one can select the action based on C. [U(x', a')) instead of [x', .a)] for a aisk-averse strategy. This strategy is proposed by Dabney et al. (2018a) and Keramati et al. (2020), among others, which we now refer to as the Markov action cetion-sllecton strategy:
(r, _ R+ +UX"X A')
A' = arg max (U (a)))
One may guess that this converges to the optimal dynamic CVaR policy satisfying 5 We now show that in general this is not true.
Proposition 1. In general, the distributional Bellman operator with the Markov action-selection strategy, Te converges neither the optimal dynamic-CViin nor the optimal static-CVal policies. even if the optimal CVaR policy is stationary and Markov.
Proof. We prove by counterexamples. We use the notation {(PL:T1). (22; denote random variable that takes value T1 with probability Pl. with probability P2 and so on. For simplicity we assume 7 = 1, but the examples work for any y by simple scaling the rewards. Let p. c such that 0 < & p < 1 c. Assume that the chosen CVaR level a is such that p² + a We only consider deterministic policies since both dynamic and static CVaR admit deterministic optimal policies.
For the MDP in Fig (a), let Xı be the initial state. There only two possible policies: choosing either action A, or Ag in state X1. It is easy to see that for a < p. D (X2) = 0 and D = O. and therefore the optimal dynamic-CVaR policy is A2. However, TU for any U E Z converges to U* (X,,, AI) = ((²2;)) (1 - 1²:1)) so (XI, e for a p² + e and therefore A₁ will be selected policy.
For the MDP in Fig [ (c), again let X1 be the initial state. The only possible policies correspond to choosing either A or A2 in X2. It is easy to see that TUU for any U E Z converges to U' U"2,,, A1) {(p: 0), (1 p; 1)} and U (X2, A2) {(1;e)} respectively and therefore A will be the chosen action for X₂ whenever a < p. By (7). this in turn means that ('()), = {(r:))
p; })}, thus C. (XX(((((X))))
The optimal static-CVaR policy, however, is to always choose A in X2 (i.e. stationary and Markov), and will result in C > é for a > p² +
(c)
(a)
(b)
X2
X2
1
P:O
p:0
1:0
X1
ps
1:0
X
P. I
1:0
X1
1:
X3
p
X
1:0
1 p;
A
Ag
A,,Ag
Probability; Reward
Figure 1: MDPs with actions A1, Ag and initial state X1, and consider p2 < < p. (a) Example where D underestimates the true CVaR. (b) Example where D; overestimates the true CVaR. (c) Example where the Markov action-sclection strategy results in underestimating the true CVaR.
2.3 Distributional RL for CVaR
It is now natural to ask whether we can properly optimize for the static CVaR while still staying within the framework of distributional RI. Recall that it is possible to optimize for the static CVaR by solving an augmented MDP M as part of an iterative process. We recall some results regarding M. Let से be a policy in M and 8 (z, s, a) be the Q-value of ग in M. Let 2 (z, a) be the
5

---

random variable representing the total return Σi=0 7TT e in the original MDP M when following म after executing a at (assuming the threshold s). Let (= Z (a, s, 7 (2, s)). Note that स is
non-Markov policy in M and executing it in M requires keeping track of the reward history with S(+1
From (Bäuerle and Ott 2011), we have that:
c (z,, 8, a) (2,sa)) (18--2)3 (-s--))t)
(8)
The key observation here that we can extract the Q-value of M from the distribution = through (8) The problem, however, is that in general the distribution (2x s.a) for a fixed (x, a) may vary depending on s. This means that keeping only single value distribution for each (x, a) precludes the extraction of the optimal CVaR policy in general. However, for the special case where there exists a stationary Markov optimal CVaR policy, we now show that this is possible.
Let मt be an optimal o-CVaR policy that is stationary and Markov in M. By definition of being stationary and Markov, there exists x" such that # (Ir, 8) xt" (1r) for every (1r, 8) pair encountered
when running #* starting from (Io, s*) with s" = q. (2 (F).s))). Let (*))) be the set {s 3t 2 O, Pr [ = (ar, 8) (00 = (rr, 8*) s*]] > 0). In words, S (xx) is the set of all possible thresholds s that one could encounter at state when following #* from Since we assume that the set of rewards
R is countable, the set of possible s is also countable so the probability in the definition S* (x)
well-defined. follows that 2* (I, 2 z= (xz, ' (x)) for all (z, s) where s E S*(x) We call S (z) the active set of æ under #*
For any distribution function U Zetet
w 8, == BawU(maa)) =)+}.
Suppose that we are given U E Z such that U(z, a) 2 2- (I, for all (I, X x A. We then exccute a policy based on using Algorithm
Algorithm 1 Policy execution for static CVaR for one episode
Input: 7 E (0, 1), a E (0,1).UE Z
1. x + z0
2. a arg m Cz(U(x, d')l
3. + (U((zz)))
4. While not terminal state,
(a) Execute a in x, observe reward r and next state xf
(b) +- x'
(c)
(d) f arg max, W (z, s, a')
Proposition 2. Let x²" be a (stationary and Markov) a-CVaR optimal policy in M. Assume that n² is unique in M. Running Algorithm with U E Z such that U(z, a) 2 z= (z, a) for all (x,a) X A results in executing n..
Proof. We show by induction that for every state 2t, Algorithm @xeeutteseee n" (xx).. At 2g, action is selected by the aelevel CVaR of U(x a') 2 z" (x, a'). By the optimality and uniqueness of x*, (2o) will be selected.
Suppose that Algorithm executes क (xx) for 0, 1, .,.,, (T 1). In M. the state (IT, ST) is such that ST (*T)) # (IT, n² (XT) and n² (or, ST, # (er,s))) 2 z (IT, (xr)). We have that for each a E A.
WU (IT, sT,A),,))))) a) EZSU((TTaa (-( (ST-2) (a)
E
2]+1
W (IT, ST. a) S W (IT, #t (IT. (2) WU (IT, Tt" (xT))

---

where the inequality (a) is due to H being the optimal policy in M, and (c) is due to the observation that 8T S* (IT) above. By the uniqueness of π.. only π (XT) can achieve equality at both (a) and (b) and therefore it will be the selected action at T.
Proposition 2 establishes the fact that one can execute a CVaR-optimal policy from a distribution function E 2 by tracking the states in the augmented MDP .M. Our objective is to learn such distribution function using distributional RL. Figure [c] and Proposition [ show that even for an MDP with stationary Markov optimal-CVas policy, To as defined in 7 will not converge to the optimal CVaR policy. We now propose an alternative distributional Bellman operator for CVaR. Given a mapping vb
X - RR define
a) - R
A' arg max W XX' o²
(/)((( R dd)
(10)
where 1 (z, 8, a) is as defined in ( Note that A' is a function of random variables X' and R. induced by p(r, 1212,,11110 a). We have the following result on Te:
Proposition 3. Let H" be a unique, optimal a- a-CVaR policy that is stationary and Markov in M. Choose 1b : X + TK such that (xx) S* (x) for all x, where S* (x) is the active set of x under #². Let xt be the policy in M such that ft (x, 8) = xx)xx) for all (I, where E (x). Then
Ts 7" (x, T" (x)) 7" (x, x (x))
The action-selection strategy in Algorithm is exactly that of Typ Proposition shows that applying To on 2" (xz, n" (x)) will always select the same mixture of distributions since p(x))r (axx)) for all x, r, x' with p(r, x' al|a,, x² (x)) > 0.
Proof.
if T (z). We need a stronger condition for this. Proposition 4. Let #* and sp be as defined in Proposition De', a, p(r, x|=', a) > ,. If there exists T³ in M such that it (I, 8) = " (x) for all (x, 8) where 8 E (x), then 7" is a fixed-poin of T
Proposition says nothing about To ZF (I, (f
Let (**(x)) { (
Proof. With the same reasoning in the proof of Proposition 2 and it can be seen that Top T follows that T n - 2 z"
Even if z= is fixed-point of T... it is an open question whether T&U will converge to 2 for all U E Z. On simple MDPs with known optimal policies, we have observed that T not only converges, but can extract both stationary and even non-station -ta-staionnay CVaR-optimal policies. Some of these examples are included in the supplementary material.
Algorithm
Our proposed algorithm is based on distributional Q-learning using quantile regression (Dabney et al., 2018b). It can be readily adapted to other variants of distributional RL. In this approach, each distribution Uzz a) is approximated by N quantiles (z, a), 1...N .. each corresponds No The quantile function qa (U(r, a)) for any (x, a) can therefore be
to quantile level f; = 1-0.5
conveniently extracted from ((z, a). Similarly, W (z. s, a) can be computed by 1 I--I-(s- (x(x,a)) J. Given 0, one can then exccute Algorithm
For training, one typically represents / by a neural-netwoll that maps input to output 0, a) for each and a. The neural-network is updated based on gradients computed using sampled mini-batches from buffer that stores actual transition examples during training runs. In our proposed approach, each stored example includes Tke 8k, dke T% and v/. The gradients are computed with respect to a quantile-regressoon loss function (=(u) = u(( - du<co) where ò = 1 if u < 0 and 0 otherwise. Algorithm 2 shows the main algorithm for computing the loss over a mini-batch containing m transition samples. Notice that Algorithm employs the standard practice of using target network
7

---

2.8
Oytamal datinnrry
19:01
aijorttim
10
0
89
10
03
53
-05
23
-10
11
0.
0.6
0.7
0.8
0.9
0.4
0.2
0.4
0.0
1.0
0.4
0.05
0..0
1.0
Figure 2: Left: Comparison with optimal stationary policies. Middle: Ground truth CVaR at 20- Right: Ground truth CVaR at Ø2.
0' which is periodically updated from 0. The key difference from the ordinary quantile-regression distributional Q-learning is our target action selection strategy for choosing aj. (Step I(a)). For other implementation details, we refer the reader (Dabney et 2018b).
Algorithm 2 Quantile Regression Distributional Q-Learning for static CVaR Input: 0, 0', mini-batch (Ik, Sk: Ck Tke xx) for = ==11... ..
1. For each k = 1...m,
(a) dgg - arg maxa W (x/ ExTte aa)))
(b) ( ax) + Tx + 70' (2 a') .j = 1
2. L - 1 Ek=1 No Ens Pr. (T ( ux) - 02 (1(xx,a))
3. Output V..
Empirical Results
Unless otherwise stated, we implement [ and and represent our policies using a neural network with two hidden layers, with ReLU activation. All our experiments use Adam the stochastic gradient optimizer. For each action, the output consists Additional details and results, well the complete code reproduce results be found in the supplementary material.
100 quantile values.
4.1 Synthetic Data
We first evaluate our proposed algorithm in a simple task where we know the optimal stationary policy for any CVaR level. The MDP has 4 states Zo, #1, T1, 23 where state To is the initial state and 2y is a terminal state. Each state has two actions On and 01. Action ag generates an immediate reward following Gaussian (11, 1) and action has immediate reward A (0.8, 0.4?). Clearly, do gives better expected reward but with higher variance. Each action always moves the state from x; 0.63, the optimal stationary policy is to choose action
1- We use 0.9 for this task. For an in all states, while for a < 0.62, the optimal stationary policy is to choose action aj in all states. We compare our proposed algorithm with the optimal stationary policy at various levels of CVaR. Figure 2 (left) shows the results.
The Markov action-selection strategy corresponds to T.D. which is the strategy proposed by Dabney al. (2018)). Our proposed strategy corresponds to Algorithm and 2 We clearly that the proposed strategy outperforms the Markov strategy at all tested CVaR levels. Further insights are revealed in Figure 2 (middle and right). These the ground truth CVaR values for all the stationary policies, where [1 00 means always choosing action aj in and ag in the next two states. Notice the switching point around 0.625 in the middle plot and around a 0.83 in the right plot. The Markov action-selle selictine strategy will choose action aj in T2 for a < 0.83 since this is the better
'W omit any comparison with gradient methods for CVaR, whose gradient estimation involves the use of the Lagrangian. A discussion of the pros and cons of value-based vs policy-gradient methods would distract much from main focus here.

---

action if one ignores the rewards collected since the beginning. However, this results in a rather conservative strategy since the optimal strategy should still favor ag 42 for 0.625.
4.2
Option Trading
0.05
Risk-neutral
Risk-neute
action-selection
Markov action-selection
0.04
Proposed algorithm
0.03
Proposed algorithm
0.03
0.02
0.01
0.01
0.00
0.00
alpha level
level
Figure 3: Test results on synthetic (Left) and real (Right) prices.
We evaluate our proposed algorithm on the non-trivial real-world task of option trading, commonly used
domain for risk-sensitive RL (Li al., 2009 Chow and Ghavamzadchi 2014 Tamar et al. 2017). In particular, we tackle the task of learning an exercise policy for American options. This can be formulated
discounted finite-horizon MDP with continuous states and actions. The state includes the price of stock at time t. well as the number of steps to the maturity date, which we set to 100. The first action, "hold", will always move the state one time step forward with zero reward, while the second action, "execute", will generate an immediate reward max K - xx} and enter terminal state. K is the strike price. In our experiments, we use 1 and always normalize the prices such that "execute". We set y = 0.999, which corresponds to a non-zero daily risk-free interest rate.
1. At / T 1, all actions will be interpreted as
We use actual daily closing prices for the top 10 Dow components from 2005 to 2019. Prices from 2005-2015 are used for training and prices from 2016-2019 for testing. To allow training on unlimited data, we follow (Li et al. 2009) and create stock price simulator using the geometric Brownian motion (GBM) model. The GBM model assumes that the log-ratio of prices follows Gaussian distribution log #+++ ~ ~N(H o² /2, with parameters μ and a, which we estimate from the real training data.
For each algorithm, each stock and each CVaR level, trained 3 policies using different random seeds. The policies are then tested on the synthetic data (generated using the same training model) for 1000 episodes. The policies are further tested on the real data, using 10 episodes, each with 100 consecutive days of closing prices, covering the years of test period. All results are averaged over the 3 policies and over the 10 stocks.
Figure ssows the test results on synthetic (Left) as well as real data (Right). The proposed strategy clearly performs better across various CVaR levels. The gap is significant at lower levels. Also included are the results from the nest-nl strategy, trained using a = 1, and tested on a values. This corresponds to the standard action-sclectio strategy based on the expected return, which performs badly at low a levels.
4.3 Atari Games
Atari games have been a popular domain for benchmarking deep RL algorithms. Dabney et al. (2018a) evaluated risk-sensitive approaches several games but acknowledged that qualitatively the outcome did not always match their expectations. We believe that the difficulty in interpreting the results is due to the many confounding factors play, e.g. total training time, exploration strategies, learning rate etc. Here, we compare the effect of our proposed approach with that of the Markov action-selection strategy in terms of the total discounted return starting from the initial state. This means that for discount factor y < 1, the initial stage of the game plays a much more significant role than later stages.
We use the same implicit quantile networks architecture as in (Dabney al. 2018a). We use the implementation by Fujita al. (2021) and made the slight modifications needed for Algorithms ]

---

Random 30M steps
Random seed 30Mil steps
Random seed 30MI steps
10-
6-
Rest-neutral
Rakneeurra
Counte
4-
2-
2-
200
250
300
350
400
450
100
200
300
400
500
200
250
300
350
400
Markot
Morkov
Morcow
Count
10-
5-
2000
250
300
350
400
450
100
200
300
400
500
700
250
300
350
400
Proposed
Proposed
30
Proposed
Count
20
20-
10-
0-
200
250
300
35.0
400
430
100
200
300
400
500
200
250
300
350
400
discounted
discounted
Total discounted
Figure 4: Results on Atari game Asterix after 30-million training steps. Each plot is histogram of the discounted returns over 100 evaluation episodes.
and We optimize for 0.25-CVaR for both risk-averse strategies, with > 0.99. Figure 4 shows the performance of the learned policies after going through 30 million steps of the game Asterix, across random seeds. Each plot is histogram of the returns over 100 evaluation runs. We can clearly see that the proposed approach achieves more concentrated distribution of returns compared to both the risk-neutral and the Markov strategies, and overall with better 0.25-CVaR. In this particular example, we observe that the Markov strategy actually performs worse than the risk-neutral strategy, both in terms of the mean and the 0.25 2VR,,,,,, although the resulting policies still have more concentrated distribution of returns in the test runs. The outcomes may change if we train for a larger number of steps, but we leave future work for more extensive empirical investigations. Additional results can be found in the supplementary material.
5 Conclusion and Future Work
This work points out a problem that arises existing methods that extend distributional RL the CVaR risk measure, and proposes an approach with better theoretical properties. We have shown that the proposed approach for learning a CVaR-optimized policy works a variety of task domains and can produce better risk-averse policies. Furthermore, the proposed algorithms can be easily incorporated into existing distributional RL frameworks.
On simple tasks for which we know the optimal CVaR policies, we have shown empirically that our approach managed converge these optimal policies. Theoretically, however, the convergence properties of the proposed Tp operator remain unknown the general settings and we believe this is an interesting problem for future works. In terms of the practical use of the proposed algorithms, one open problem is regarding the repeated use of the stored threshold s in the replay buffer after significant change in the return distribution of the policy. We believe that more adaptive way of utilizing s during training can lead to improvement in the learning efficiency in terms speed and stability. In terms of exploration, may also be beneficial to our approach.
optimistic bias such that proposed by Keramati et al. (2020)
References
Artzner, P., Delbaen, F., Eber, J.-M., and Heath, D. (1999). Coherent measures of risk. Mathematical Finance, 93):203-228.
Barth-Maron, G., Hoffman, M., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lillicrap, T. (2018), Distributed distributional deterministic policy gradients.
10

---

Bäuerle, N. and Ott. J. (2011). Markov decision processes with vveraee-value-ar at-risk criteria. Mathematical Methods of Operations Research, 74(3):661-37.
Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 1., pages 449 458.
Bernhard, J., Pollok, S., and Knoll, A. (2019). Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In IEEE Intelligent Vehicles Symposium.
Borkar, V. (2001). A sensitivity formula for risk-sensitive cost and the actor-critic algorithm. Systems & Control Letters, 44(5):339- 346.
Chow, Y. and Ghavamzadeh, M. (2014). Algorithms for evar optimization in mdps. In Advances in Neural Information Processing Systems 27, pages 3509-3517,
Chow, Y. and Pavone, M. (2014). A framework for time-consistent, risk-averse model predictive control: Theory and algorithms. In 2014 American Control Conference, pages 4204-4211.
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. (2018a). Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference Machine Learning.
Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. (2018b). Distributional reinforcement learning with quantile regression. In Proceedings of the Thirly-Second AAAI Conference Artificial Intelligence, (AAAI-18), pages 2892-2901.
Fujita, Y., Nagarajan, P., Kataoka, T., and Ishikawa, (2021). Chainerr:: deep reinforcement learning library. Journal of Machine Learning Research, 22(77):1-14.
Howard, R. A. and Matheson, J. E. (1972). Risk-sensitive markov decision processes. Management Science, 18(7):356-36)
Huang, W. and Haskell, W. B. (2020). Stochastic approximation for risk-aware markov decision processes. IEEE Transactions on Automatic Control, pages 1-1.
Keramati, R., Dann, C., Tamkin, A., and Brunskill, E. (2020). Being optimistic to be conservative: Quickly learning evar policy, Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4436-4443
L.A., P. and Ghavamzadeh, M. (2013). Actor-critic algorithms for risk-sensitive mdps. In Burges, C. J. C., Bottou, I., Welling, M., Ghahramani, 7.,, and Weinherger, K. Q., editors, Advances in Neural Information Processing Systems 26, pages 252-260.
Li, Y., Szepesvari, C., and Schuurmans, D. (2009). Learning exercise policies for american options. In Proceedings of the Twelth International Conference Artificial Intelligence and Statistics, pages 352-359.
Mavrin, B., Yao, H., Kong, L., Wu, K., and Yu, Y. (2019). Distributional reinforcement learning for efficient exploration. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 4424-4433
Mnih, V., Kavukcuoglu, K., Silver, D.,. Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):529 533.
Morimura, T., Sugiyama, M.. Kashima, H., Hachiya, H., and Tanaka, T. (2010). Parametric return density estimation for reinforcement learning. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI' 10, page 368-375.
Petrik, M. and Subramanian, D. (2012). An approximate solution method for large risk-averse markov decision processes. In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, UAI' 12, page 11.
11

---

Rockafellar, R. T. and Uryasev, S. (2000). Optimization of conditional value-at-risk, Journal of Risk, 2:21-41.
Ruszezynski, A. (2010). Risk-averse dynamic programming for markov decision processes. Math. Program., 125(2))235-261.
Stanko, and Macek, K. (2019). Risk-averse distributional reinforcement learning: A evar opti- mization approach. In Proceedings of the 11th International Joint Conference Computational Intelligence (IJCCI), pages 412-423.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction. The MIT Press, second edition.
Tamar, A., Chow, Y., Ghavamzadeh, M., and Mannor, S. (2017). Sequential decision making with coherent risk. IEEE Transactions on Automatic Control, 62(7):3323-3338.
Tamar, A., Di Castro, D., and Mannor, S. (2012). Policy gradients with variance related risk criteria.
In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, page 1651-1658, Madison, WI, USA. Omnipress.
Tamar, A., Glassner, Y., and Mannor, S. (2015). Optimizing the evar via sampling. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI' 15, page 2993-2999, AAAI Press.
Watkins, C.J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning, 8(3):779-292
Yang. D., Zhao, L., Lin, Z.. Qin, T., Bian, J., and Liu, T.-Y. (2019). Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neural Information Processing Systems 32.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes]] Our theoretical contributions are presented as Propositions in Section 2 and our empirical contributions are presented in Section 3.
(b) Did you describe the limitations of your work? [Yes] The problem setup and the assumptions are carefully stated before presenting the theoretical results. In particular, the results in Section 2.3 remain open for settings that violate our assumptions.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our results improves one's 's ability to learn risk-sensitive policies, any direct societal impact would be minimal.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] Supplemental material
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, cloud provider)? [Yes] Supplemental material
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you'n using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "obbB_RLZf6j", "reviewer": "Reviewer_wokp", "summary": "The authors study the problem of learning risk-sensitive policies in the distributional RL setting. They adopt Conditional Value at Risk (CVaR) as the risk measure throughout the paper. They begin with a problem overview and discuss the differences between static and dynamic CVaR problem formulations. After introducing the requisite background on distributional RL, the authors propose a naïve method of learning a risk-sensitive policy, called the Markov action-selection strategy, where the action selection procedure aims to maximize the $\\alpha$-CVaR over return distributions at a given state. It is then shown that a distributional Bellman operator employing the Markov action-selction strategy does not converge to a dynamic or static CVaR optimal policy, even if the optimal CVaR policy is stationary and Markov. \n\nWith this result, the authors’ aim is thus to determine if a static CVaR policy may be properly optimized in the distributional RL framework. They begin by constructing an augmented MDP from the original, with the addition of a thresholding parameter. This parameter is initially set equal to the $\\alpha$-quantile of the initial return distribution and is repeatedly updating by subtracting off observed rewards, thereby effectively keeping a history of the “return-to-go” to reach the initial threshold. The introduction of this new parameter poses a new problem: it is no longer sufficient to retain a single value distribution for the original MDPs state-action pairs, since the value distributions in the augmented MDP depend on the threshold parameter, which may be observed to be many different values for any given state-action pair. To navigate this problem, the authors assume the existence of a CVaR optimal augmented policy that is stationary and Markov in the original MDP, and show that the stationarity and Markov properties allow for one to recover an optimal CVaR policy in the original MDP. They go on to show that given the optimal original policy’s value distribution, Algorithm 1 executes the optimal original policy by keeping track of the thresholding parameter’s history. It is shown thereafter that the action selection procedure in Algorithm 1 may be used in a distributional Bellman operator, and that this new operator has a fixed point that is equal to the return distribution associated with the optimal CVaR policy from the original un-augmented MDP. The authors note here that it is still an open question whether the new Bellman operator is a contractive operator on the entire space of return distributions i.e. that starting from any return distribution, the repeated application of the operator would result in the eventual fixed point that is equal to the return distribution of the CVaR optimal policy. \n\nThe authors then present an algorithm that optimizes a neural network policy using the newly proposed distributional Bellman operator. Experiments are presented showing that the proposed algorithm outperforms the Markov action-selection and risk neutral policies in a synthetic data experiment, an options trading experiment, and in Atari games.\n", "questions": "N/A", "limitations": "-The paper does discuss the theoretical limitations of the work, namely the absence of a proof of contractivity of the proposed Bellman operator.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-The paper introduces novel and interesting theoretical results that advance the theory of CVaR distributional RL\n\n-The research presented in the paper lays groundwork for important future research, e.g. proving the contractive property of the proposed Bellman operator.\n\n-The paper does a very good job of clearly stating the need, contribution, and relevance of the work. This is reinforced by the detailed background provided in Section 2\n\n\nWeaknesses:\n\n-The paper ends abruptly and has no concluding remarks. \n\n-The discussion and analysis of the empirical results is limited. There is very little explanation offered that would allow the reader to construct an intuitive reasoning for the obtained results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g1vmJHExwiz", "reviewer": "Reviewer_XVo8", "summary": "The authors tackle the challenge of learning a risk averse policy (CVaR) in a distributional RL setting.\nWhile prior works have presented a naive method for optimizing the CVaR, this is shown (both theoretically and practically) to not optimize the correct objective.\nIn this work, they present a theoretically sound method for iteratively learning a CVaR maximizing policy and back these claims with experiments.", "questions": "1) The comparison to Dabney et. al. 2018 (a) seems slightly misleading.\nThe goal of a risk-sensitive method is to maximize the lower-tail of the returns. Dabney et. al. 2018 (a) might not optimize the CVaR properly, but their results do show competitive performance in ATARI domains (often outperforming mean optimization).\n\nIn these experiments, we see that Markov selection is under-performing compared to risk-neutral.\n\nWhat happens in simpler domains / when the algorithm is run using the same parameters as Dabney et. al. 2018 (a) and for a longer period of time (commonly 200m steps)?\n\n2) What happens in other domains? does this method still work or is it limited to Asterix?", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Present a theoretically backed 1-step iterative update scheme for learning a CVaR maximizing policy in a distributional RL setting.\n- Present a comparison and analysis of prior methods showing that naive application of the 1-step CVaR does not solve the objective.\n- Experiments on synthetic data + option trading + ATARI.\n\nWeakness:\n\n- While this work performs analysis on atari, this analysis is very limited. While I appreciate the simpler experiments that enable a more theoretical analysis of the method, complex domains enable a more convincing empirical justification.\nI'd like to see more experiments in such domains. If compute is an issue, the authors can always resort to tasks such as continuous control / minatari / cartpole / and other simpler tasks.\n- Paper lacks a conclusion and future work section. While there are no rules that require such structure, I do feel that this is warranted and well accepted in the community. I'd propose to move some proofs to the appendix in order to allocate room for discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JewpRpxfK9U", "reviewer": "Reviewer_GmTT", "summary": "This paper considers learning a risk-sensitive policy based on CVaR via distributional RL. A modified distributional optimal operator is proposed with fixed-point satisfying CVaR optimality and an algorithm is proposed which combines QRDQN and the proposed operator. Experiments are executed on both synthetic and real-world data to present the performance of the proposed algorithm.", "questions": "1. Although the operator $\\tilde{\\mathcal{T}}_{\\psi}$ defined in (10) may not result in convergence of distribution, I was wondering if $C_{\\alpha}(\\tilde{\\mathcal{T}}_{\\psi} U)$ converges to $C_{\\alpha}(Z^{\\pi^*})$ for all $U$. \n2. The authors compare their proposed algorithm, which is based on QRDQN (Dabney et al. 2018b) with IQN(Dabney et al. 2018a). I think it would be more fair to use the same baseline structure to approximate the distribution function, either IQN or QRDQN.\n3. The experiments on Atari games show that the proposed algorithm achieves a more concentrated distribution. Is there any benefit or effect with the concentration?", "limitations": "This paper requires that the rewards are drawn from a countable set and this is not a common assumption. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThis paper is well written and easy to follow. It indicates that the standard distributional Bellman operator does not work for the CVaR optimal policy and makes a proper modification. \n\nWeakness: \nThe theoretical results are not solid enough to guarantee the convergence of the algorithm. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7T2NAUkLiZu", "reviewer": "Reviewer_2T3t", "summary": "The paper presents a new method for robust reinforcement learning that optimizes the dynamic and static CVaR. First the paper proves that the current action selection rule in distributional RL does not converge to the CVaR objective. The paper presents a new method that they show learns to optimiz static and dynamic CVaR. Experiments show that the proposed approach does better at the task of option trading than risk neutral and markov action selection startegies", "questions": "- What is meant by distributional RL does not converge to static or dynamic RL \n- What about other literature on robust policy optimization? The paper can compare their method to these existing approaches \n    -  Risk-constrained reinforcement learning with percentile risk criteria\n    - Risk-sensitive and robust decision-making: a CVaR optimization approach\n    - Tractable objectives for robust policy optimization\n", "limitations": "I feel the paper currently lacks a good contrast to existing methods for robust RL, principally and experimentally. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Interesting and relevant problem\n+ Interesting theoretical and experimental results\n- It is difficult to see how the method compares or positions itself to other papers in the robust RL literature\n- The evaluation seem to be missing some relevant baselines as the paper points out itself that there is related work that can tackle static and dynamci CVaR", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept