Distributional Reinforcement Learning for Risk-Sensitive Policies
Authors: Shiau Hong Lim
Ilyas Malik
Venue:  (2021)

Abstract: We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator with respect to a risk-based action-selection strategy overestimates the dynamic, Markovian CVaR. The resulting policies can however still be overly conservative and one often prefers to learn an optimal policy based on the static, non-Markovian CVaR. To this end, we propose a modification to the existing algorithm and show that it can indeed learn a proper CVaR-optimized policy. Our proposed approach is a simple extension of standard distributional RL algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empirically show that our proposed algorithm is able to produce a family of risk-averse policies that achieves a better tradeoff between risk and the expected return.


Under review as a conference paper at ICLR 2021
DISTRIBUTIONAL REINFORCEMENT LEARNING FOR RISK- POLICIES
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator with respect to risk- based vetiion-eleccion strategy overestimates the dynamic, Markovian CVaR. The resulting policies can however still be overly conservative and one often prefers to learn an optimal policy based on the static, non-Markovian CVaR. To this end, we propose modification to the existing algorithm and show that it can indeed learn a proper CVaR-optimized policy. Our proposed approach a simple extension standard distributional RL algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empiri- cally show that proposed algorithm is able to produce family of risk-averse policies that achieves a better tradeoff between risk and the expected return.
INTRODUCTION
In standard reinforcement learning (RL) (Sutton & Barto 2018). one seeks to learn a policy that maximizes an objective, usually the expected total discounted rewards or the long-term average re- wards. In stochastic domains, especially when the level of uncertainty involved is high, maximizing the expectation may not be the most desirable since the solution may have high variance and occa- sionally performs badly. In such scenarios one may choose to learn a policy that is more risk-averse and avoids bad outcomes, even though the long-term average performance is slightly lower than the optimal.
In this work we consider optimizing the conditional value-at-risk (CVaR) (Rockafellar & Uryasev 2000), a popular risk measure, widely used in financial applications, and is increasingly being used in RL. The CVaR objective focuses the lower tail of the return and is therefore more sensitive to rare but catastrophic outcomes. Various settings and RL approaches have been proposed to solve this
problem (Petrik Subramanian, 2012 Chow Ghavamzadeh| [014 | Chow & Pavone [0014] Tamar
ct al.
| Tamar et al. (2017 77 Huang & Haskell 2020)) ). Most of the proposed approaches, however,
involve more complicated algorithms than standard RL algorithms such as Q-learning (Watkins &
Dayan 1992] and its deep variants, e.g. DQN (Mnih et al. 2015).
Recently, the distributional approach RL (Bellemare et al. 2017 Morimura et al. 2010) has received increased attention due to its ability to learn better policies than the standard approaches in many challenging tasks (Dabney al.] 22088a by Yang et al. 2019). Instead of learning a value function that provides the expected return of each state-action pair, the distributional approach learns the entire return distribution each state-action pair. While this computationally more costly, the approach itself is simple extension to standard RL and is therefore easy to implement and able to leverage many of the advances in deep RL.
Since the entire distribution is available, naturally considers exploiting this information to opti- mize for an objective other than the expectation. Dabney et al. (2018a) presented a simple way to do so for family of risk measures including the CVaR. The theoretical properties of such approach, however, are not clear. In particular, it is not clear whether the algorithm converges to any particular variant of CoaR-optimal policy. We address this issue in this work.
Our main contribution is to first show that the proposed algorithm in (Danney et al. 8a) overes- timates the dynamic, Markovian CVaR but empirically can be as conservative. It has been demon-

---

Under review as a conference paper at ICLR 2021
strated that this variant of CVaR can be overly conservative in many scenarios (Tamar et al. 2017 Yu et al.; 2017), and one may prefer the static CVaR instead as the objective. Our second contri- bution to propose modified algorithm that can help achieve this. Empirically, we show that the proposed approach learns policies that perform better in terms of the overall CVaR objective on both synthetic and real-world problems.
We close the introduction section with some references to related works. We formally present our problem setup as well as our main analytical results in Section 2 algorithm and finally, Section 4 presents our empirical results.
Section 3 describes our proposed
1.1 RELATED WORKS
The literature on distributional RL has been greatly expanded recently (Morimura et al. 2010 Belle-
mare et al. 2017 Barth-Maron et al. 20018) Dabney et al.] 201 2 works focus on the modeling aspects, such as the choice of representations for the value distribu- tions. The approach has been used to enhance exploration in RL (Mavrin et al. [019 Zhang & Yao.
b Yang al. 2019). Most of these
2019) and in risk-sensitive applications [Wang et al. 2019; Bernhard et 2019).
Solving Markov decision processes (MDP) with risk-sensitive objectives have been addressed in many works (Howard & Matheson] 1972. Ruszezynski, 2010 Bäuerle & Ou (01T), including RL approaches (Borkar 2001] Tamar al. 2012 L.A. & Ghavamzadeh 2013). In particular, Chow & GGavamzadeh (2014): Tamar et al. (2015) deal with the static CVaR objectives while Petrik &
Subramanian (2012); Chow & Pavone (2014) deal with the dynamic CVaR objectives. [Tamar et al. (2177) proposed policy-gradient approach that deals with both the static and the dynamic CVaR objectives. Closest ours is the work by Stanko & Macek (2019). Their proposed approach also makes use of distributional RL but it is not clear whether their action selection strategy properly optimizes either the static or the dynamic CVaR.
PROBLEM SETUP AND MAIN RESULTS
We consider discrete-time Markov decision process with state space X and action space A. For simplicity we assume that X and A are finite, although our results and algorithm can be read- ily extended to more general state-action spaces. We assume that the rewards are bounded and drawn from a countable set R C R. Given states It, Ix+1 E X for any t {0.1.... probability of receiving reward rt- E R and transitioning to Ø1+1 after executing a₁ A in zt is given by p(r,,I/+1 [2, az). Without loss of generality we assume fixed initial state mn- un- less stated otherwise. Given a policy π H + P(A), where H is the set of all histories so far hy = (Fo, do, To, 1: (1. 2i.) E H, and P(A) the space of distributions over A, its expected total discounted reward over time is given by
the
V
where E
E (0, 1) is a discount factor. The superscript π in the expectation indicates that the actions a, are drawn from T (he). The subscript p indicates that the rewards and state transitions induced by
In standard RL, we aim to find policy that maximizes V It well-known that there exists a deterministic stationary policy π X + A whose decisions depend only on the current state, that gives optimal V*, and therefore one typically works in the space of stationary deterministic policies. Key to a synnamc-programming solution to the above problem is the use of a value function Q" (xx a) = E Ie r'rel No = I, 00 a), which satisfies the Bellman equation
Vx,a, a, (z) Σ p(r, 2'12, a) 1 7Q (2', ((()))
The optimal value (x)x)) = Q" (x, a) for any optimal policy .r" satisfies the Bellman optimality equation
Vx,a, (²(x,a)= E(rr,'111, a) [r max ('(2', .a)]
(2)
2

---

Under review as a conference paper at ICLR 2021
Furthermore, for any Q-function show that the operator 7 defined by T²) := E p(r, 2' (r, 7Q(=', (x)))] is a 7-contraction in the sup-norm ||Q]| := max a [Q ar) with fixed-point satisfying T One can therefore start with arbitrary Q-function and repeatedly apply T or stochastic approximation, to learn Q". An analogous operator T can also be shown to be a --contraction with fixed-point satisfying
- g(o,)) < <c, Va, a),
2.1 STATIC AND DYNAMIC CVAR
The expected return V is risk-neutral in the sense that it does not take into account the inherent variability of the return. In many application scenarios, one may prefer a policy that is more risk- averse, with better sensitivity risk (CVaR), which is a popular risk measure that satisfies the properties of being coherent (Artzner al. 1999). The a-level CVaR for random real-valued variable Z, for (0, 1]. is given by |Rockafellar & Uryasev 2000
bad outcomes. In this work, we focus the conditional value-at-
1 EI(( Z)+1
Cz(Z)= max SER
where (x)+ = max (ar, 0}. Note that we are concerned with Z that represents returns (the higher, the better), so this particular version of CVaR focuses on the lower tail the distribution. In particular, the function s + _ z)+] is concave in and the maximum is always attained at the a-level quantile, defined as
qa (Z) = inf(s: (s: Pr(Z - s) a).
For 1, C. reduces to the standard expectation. In the case Z is absolutely continuous, have the intuitive Ca(2) = E < qal.
Our target random variable is the total discounted return Z" = Er= r'r, of a policy म, and our objective is to find a policy that maximizes C., (Z*),, where the optimal CVaR is given by
-E p -=-))=')
max max X
In the context where Z is accumulated multiple time steps, the objective 3 corresponds to maximizing the so-called static CVaR. This objective is time-inconsistent in the sense that the op- timal policy may be history-dependent and therefore non-Markov. This is, however, perfectly ex- pected since the optimal behavior in the later time steps may depend on how much rewards have been accumulated thus far more risky actions can be taken if one has already collected sufficiently large total rewards, and vice versa. From the point of view of dynamic programming, an alterna- tive, time-consistent or Markovian version of CVaR may more convenient. A class such risk measures was proposed by |Ruszczynski (211)), and we shall refer to this version of CVaR as the dynamic CVaR, defined recursively as
VT,
Dd (a,o(2,a) = Calrela 2,0; a],
VH, x, a, 0, D Dar a) = Calri DD T ((z++1)) = x, a = a). and
VT, Z.a, (T = lim D (z,
T-300
It can be shown Ruszczynski maximizing D (ar, for all whose dynamic CVaR is given by D = D In particular, the operator TD,a defined by
2010 that there exists stationary deterministic optimal policy n",
TD a) = C. max D( d') x, a, =a]
(4)
for E Q is y in sup-norm with fixed- ekent-pinnt satisfying
VI,a, (², a) Calra + max (7, x, a).
(5)
We use slightly different definition from that Ruuzezyynsii 2010), but conceptually they essentially
the same.

---

Under review as a conference paper at ICLR 2021
The dynamic CVaR, however, be overly conservative in many cases. We illustrate this with some empirical results in Section 4 In such cases may be favorable to use the static CVaR. Bäuerle & Ott (20IT) suggest an iterative process that can be used to solve for the optimal static CVaR policy. The approach is based on 3):
For a fixed s, one can solve for the optimal policy with respect to max, E8-(s -*)*}.
2. For a fixed и, the optimal is given by the a-level quantile of 2"
3. Repeat until convergence.
Step one above can be done by solving an augmented MDP with states ग = (x, E X x R, where is a moving threshold keeping track of the accumulated rewards far. In particular, this MDP has no rewards and state transition is given by p(0, (x', ( (), s),a) == p(r, x x2|2 a). Solving this augmented MDP directly using RI., however, can result in poor sample efficiency since each example (ær a, r, x') may need to be experienced many times under different threshold s. In this work, propose alternative solution using the approach of distributional RL.
2.2 DISTRIBUTIONAL RL
In standard RL, typically learns the (= a) value for each (I, a) through some form of temporal-difference learning (Sutton & Barto 2018). In distributional RL (Bellemare et al. 2017
one instead tries to learn the entire distribution of possible future return 2 2"(x;, a) for each (2r, a).
The Q-value can then be extracted by simply taking the expectation Q" (xx a) E (x, a)).
The objects of learning distribution functions U Z (Z X ((R)) E Z a)]9]
20, Vz, q 2 1}. For any state-action pair (z, a), we use U(x, a) to denote random variable with the respective distribution. Let 7 be the distributional Bellman operator on Z such that
=2 :2 R+ (X'''xXxx'"
where D denotes equality in distribution, generated by the random variables R, X' X' induced by p(r, x' 12|2, a). We use the notation T instead of when referring to a distributional operator, where U=U(x, a) is random variable. (Bellemare et al. 2017) show that 7 is a 7-contraction in Z in the following distance metric
d(U,v) = sup (U,, 2,0
where W is the L-Wasserstein distance between the distributions of a) and V(x, Further- more, the operator T defined by
jU(2,
A'= ara max (X',
(6)
can be shown to be y-contraction in Q in sup-norm under element expectation, i.c.,
|ETu - ET I BIU - 1V
where ETU= e Q such that (TU = (x, a)), and EU, EV. E/V all similarly defined. In general, 7 is not expected to be contraction in the space of distributions for the obvious reason that multiple optimal policies can have very different distributions of the total return even though they all have the same expected total return.
Since one keeps the full distribution instead of just the expectation, natural way to exploit this to extract more than just the expectation from each distribution. In particular, in (6) one can select the action based on measures other than the expectation E (U' a')]. This is done by Dabney al. (2018a) where distortion measure the expectation used to select actions using various risk measures on Uxx,, a'), including the CVaR. If we replace (xU a')] with (Z(U(z', aa)], one may guess that it converges the optimal dynamic CVaR policy satisfying (5). This is however, not true in general. We now show that choosing actions using C, (U a') results in overestimating the dynamic CVaR value D and D
4

---

Under review as a conference paper at ICLR 2021
Proposition 1. Let U Z. Let Ca(U) Q such that Ca (a) == Ca (x) defined in
Let TD,a as
The distributional Bellman operator TDaa given by
TD,aU(z,a) D RR R+ A' = A(X) = arg max Cx(U(X',
d'
satisfies
Vx, a, (Z(,D,aU(R,aa) 2 (T),aC)
Similarly, for a fixed n, we have that
Vx, CalTD,a (x, a)] 2 CoL)(())
Proof. We will use the properties of CVaR as coherent risk measure (Artzner et al., 1999). In particular, C, is concave in Z (recall that we use the lower-tail version) where AA [0, 1]. +
2 AC, + (1 ( and satisfies both translation invariance and positive homogeneity, where Ve, À E R, X 2 0, C. (AZ + + (Z)
Ca Ca.TD,U Ca
pptra I. +
Σ
A(z)))
Σ p(r, 2 1z, prrr,
= E + (X
- Ca RR + CC, A(X)))]
(TD.aC.
where the coherent properties of C. in (a) and (b), and in (c) use the fact that the expectation is Ca for a = and upperbounds all other C.. The proof for the fixed n case is analogous.
casy to construct an example where the inequalitics in Proposition are strict and that TD. converges to policy that is different from the optimal DG policy. Unfortunately, through empirical observations, TD. still results in policies that are closer to those optimizing the dynamic CVaR rather than the static CVaR.
It is now natural to ask whether we can optimize for the static CVaR instead while still staying within the framework of distributional RL. Recall that it is possible to optimize for the static CVaR by solving an augmented MDP as part of an iterative process. Instead of explicitly augmenting the state space, we rely on the distributions U E Z to implicitly "store" the information needed. This approach will make the most of each transition example from (ar, a), since it updates an entire distribution, and indirectly the entire set of states (x, s) for all in the augmented MDP. For this, the action selection strategy in (6) plays critical role.
Given U Z and E. define C(U, s) E Q such that Ç(U, (x, a) Bi-(s (xx +}. We define a distributional Bellman operator for the threshold as follows,
R U, 8 - (X',a).
(z) R R R A' A
arg max
The following result shows that at least for fixed target threshold, improvement is guaranteed after each application of T..
Proposition 2. For any U,V E Z, and any E TR,
||<T).Uss) -((osvs)) S 7suu IsU,,s' - -(VV,
5

---

Under review as a conference paper at ICLR 2021
Proof. For each (r,
-
(f- ( - (1,"))))
Σ Ep(r,z'(z,a)
(-((--- dU(('' A (r,z')) / (5-r-1) aV A/ (r,,"'))
Σ ((r,2'le,,))
=y
ra
Σ raa
p(r, (x, maxx U, 7
(U,-==) (x', d') max & ( v ( (I', (e"d"))
(a)
Eplr, 1z, (xa) max 5 [ (1,==)) (x', - (v v (e',
o'
7
Sy sup a',
KCU (z),, d') <(V, ((',''
where in (a) we use triangle inequality and the fact that max, f(a) -max, g(a')l S max If(a) g(a)].
Since we only keep one distribution for each ar, we can only apply T. for a single chosen s during each update. Applying To can potentially change <(T.U,s')aa a) for any other s' and there no guarantee that similar improvement happens for these s'. Recall that we seek to optimize the long-term CVaR, where the "optimal" is actually the a-quantile of the long-term return. We therefore propose the following operator.
Vx,a, a, T.U(I, a) : 2 TT. (Ua, a).
This can be easily implemented through distributional RL, which we describe in the next section.
3 ALGORITHM
Our proposed algorithm is based on distributional Q-learning using quantile regression Dabney al. 20186). It can be easily adapted to any other variants of distributional RL. Algorithm shows the main algorithm for computing the loss over a mini-batch containing m transition samples. Here, each distribution ((xr,a) a) is represented by N quantiles 0 (01 ... .a....)., ). each corresponds
quantile level f 1-0.5 N The quantile function (f) can therefore be easily extracted from 6. The loss function is based on quantile regression, where pu) = uu( duco) where 6u<00 = 1 if < 0 and otherwise. The key difference from the ordinary quantile-regression distributional Q-learning is our target action selection strategy for choosing a'g (Step ((a) and (b)). For other implementation details, we refer the reader to (Dabney et al., 2018b).
Algorithm 1 Quantile Regression Distributional Q-Learning for static CVaR Input: 7, a, 0, 0', mini-batch (Ik, ak, The 2%) for 1...
1. For each k = 1...m,
(a) 8 + (0( az))
(b) dg arggmax, ((8', SK-Th (ark, (20
(c) [0, ak) t- rh + 7700 (5 (12)
2.
m Em 1 N² Ens Pt, (+,(TTTT (TO, (Ik; ax) 0,( ax))
3. Output VoL.
The execution of policy defined by requires an additional state information s, which summarizes the rewards collected so far. This is not part of the MDP state and can easily be updated after

---

Under review as a conference paper at ICLR 2021
observing each reward. At the start of new episode, is reset. The complete algorithm for executing the policy in one episode is given Algorithm
Algorithm 2 Policy execution for static CVRR for one episode
Input: 7, ox 0
1. x Initial state
2. arg maxe ((0,9. (z,
3. < (0( a))
4. While not terminal state,
(a) Execute in æ, observe reward r and next state
(b) &mr
(c) f
(d) t arg maxa (),,,)
4 EMPIRICAL RESULTS
implement Algorithm D and 2 and represent our policies using a neural network with two hidden layers, with ReLU activation. All our experiments use Adam the stochastic gradient optimizer with learning rate 0.0001. For each action, the output consists of N 100 quantile values. The complete code reproduce results is included in the supplementary material.
4.1 SYNTHETIC DATA
We first evaluate our proposed algorithm in a simple task where we know the optimal stationary policy for any CVaR level. The MDP has states To, 42, 23 where state zy the initial state and is a terminal state. Each state has two actions ag and a1. Action generates an immediate reward following a Gaussian (11, (1, 1) and action an has immediate reward A (0.8, 0.42). Clearly, dg gives better expected reward but with higher variance. Each action always moves the state from to T1-1- J. We use 7 = 0.9 for this task.
For > 0.63, the optimal stationary policy is to choose action dg in all states, while for < 0.62, the optimal stationary policy is to choose action dı in states. We compare our proposed algorithm for static CVaR with the optimal stationary policy at various levels of CVaR. Figure [ (left plot) shows the results.
Stalic
(C01)
Dynumi:
0.5
10 01
0.0
01
2.8
11
2.3
0.2
0.3
0.2
0.5 Krhh level
2.5
0.2
2.2
1877 2001
1177a m00
Figure 1: Left: Comparison with optimal stationary policies. Middle: Ground truth CVaR at Zi-
Right: Ground truth CVaR at Ø22
We label the results for our proposed algorithm "Static" while the action-selection strategy based on (Dabney et al. 201 "Dynamic". We clearly see that the "Static" version outperforms "Dy- namic" at all tested CVaR levels. In fact, our algorithm performs even better than the optimal sta- tionary policy. Recall that the optimal CVaR policy may be non-stationary, where the actions in later states depend the rewards collected far. This example shows that learning using Algorithm and execution using Algorithm 2 can indeed result in non-stationary policies by storing the "extra" information within the value distribution.

---

Under review as a conference paper at ICLR 2021
Further insights are revealed in the middle and right plots of Figure These the ground truth CVaR values for all the stationary policies, where [100] means always choosing action di in 20 and dg in the next two states. Notice the switching point around 0.625 in the middle plot and around a = 0.83 in the right plot. The "Dynamic" CVaR action-selection strategy will choose action an in 22 for a 0.83 since this is the better action if one ignores the rewards collected since the beginning. However, this results in rather conservative strategy since the optimal strategy should still favor ag 22 for 0.625.
4.2 OPTION TRADING
We evaluate our proposed algorithm on the non-trivial real-world task of option trading, commonly used as a test domain for risk-sensitive RI. ([. et al., 2009) Chow Ghavamzzdehi 2014 Tamar et al.) (017). In particular, This can be formulated as discounted finite-horizon MDP with continuous states and two actions. The state 2; includes the price of stock time t, well the number of steps the maturity date, which we set to T = 100. The first action, "hold", will always move the state one time step forward with zero reward, while the second action, "execute". will generate immediate reward max(0 K - J and enter a terminal state. K is the strike price. In our experiments, we use K and always normalize the prices such that xo 1. At l T 1, all actions will be interpreted as "execute"". We set 7 0.999, which corresponds to non-zero daily risk-free interest rate.
tackle the task of learning
exercise policy for American options.
We use actual daily closing prices for the top 10 Dow components from 2005 2019. Prices from 2005-2015 are used for training and prices from 20 2016-2019 for testing. To allow training on unlimited data, we follow (Li et al. (0099 and create stock price simulator using the geometric Brownian motion (GBM) model. The GBM model assumes that the log-ratio of prices follows Gaussian distribution log #++1 N( from the real training data.
with parameters ju and a, which we estimate
For each algorithm, each stock and each CVaR level, we trained 3 policies using different random seeds. The policies are then tested on the synthetic data (generated using the same training model) for 1000 episodes. The policies are further tested on the real data, using 100 episodes, each with 100 consecutive days of closing prices. The episode's start and end dates are evenly spread out over the 4 years of test period. All results are averaged over the 3 policies and over the 10 stocks.
Figures 2 and 3 show the test results on synthetic and real data, respectively. Again, we label the algorithms "Static" and "Dynamic" as in the previous section. Clearly, when tested on both synthetic and test data, the "Static" algorithm performs better across various CVaR level. The gap is especially significant at lower levels. Also included are the results from training using all values. This corresponds to the standard action-selection strategy based on the expected return. The learned strategies perform badly low levels, suggesting that they taking too much risk.
1, and tested
Static Dynamic
9.330
Shilt Dynamic Eesectaton on
3.05
Expectation
0.025
0.020
0.03
0.015
9.010
0.03
0.505
2.5
0.6
0.7
0.5
4bhs Irne
abtl level
Figure 2: Test results on synthetic data
Figure 3: Test results on real data
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Philippe Artzner, Freddy Delbuen, Jean-Mare Eber, and David Heath. Coherent measures risk. Mathematical Finance, 93):203-2 1999.
Gabriel Barth-Maron, Matthew Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. 04 2018.
N. Bäuerle and Ott. Markov decision processes with average-value-at-ris criteria. Math- ematical Methods of Operations Research, 74(3):361-379 2011. ISSN 1432-2994. doi: 00.10077/00118-011-033777
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 449-458. JML...rrg 2017.
J. Bernhard, S. Pollok, and A. Knoll. Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In 2019 IEEE In- telligent Vehicles Symposium (IV), pp. 2148 2155, 2019.
V.S. Borkar. A sensitivity formula for risk-sensitive cost and the actor-critic algorithm. Sys- tems & Control Letters, 44(5):339 346, 2001. ISSN 0167-6911. doi: https://doi.or// URL httt::/t:://ww sci encediciiiie rect. www.coocccc.... htt///ciencc/ee//
articile/p11/5501666111100
Y. Chow and M. Pavone. A framework for time-consistent, risk-averse model predictive control: Theory and algorithms. In 2014 American Control Conference, pp. 4204 4211, 2014.
Yinlam Chow and Mohammad Ghavamzadeh. mdps. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3509- 3517. Curran Associates, Inc., 2014. URL 1196--alooittan fooerrtooooo-o-----oo---o---o--
Algorithms for cvar optimization in
nips. cc/paper/
Will Dabney, Georg Ostrovski. David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096-1103 Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018a. PMLR. BRLphhtt:////ceeeeee
mlr. press / vBC/ 1780/dalee dabneyl Maannyyy88 html
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the Thirty-Secome Second AAAI Confer- ence on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelli- gence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2892-2901, 2018b. URL
TTT.AAAAAAAAAA//PPP. https: www. wwww.aaai.c. org/ ocs index..in.. httpa//neengp///AA//////////////
Ronald A. Howard and James E. Matheson. Risk sessisensitive markov decision processes. Management Science, (87)):356-369 1972.
W. Huang and W. B. Haskell. Stochastic approximation for risk-aware markov decision processes.
IEEE Transactions on Automatic Control, pp. 1-1, 2020.
Prashanth L.A. and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q- Weinberger (eds.), Advances Neural Information Processing Systems 26, pp. 252- 260. Curran Associates, Inc., 2013. URL http: http::::/papprs..ipp.. nips. CC/ ct..pp..ccppprrrrr/pp/ppp paper/ 1997-actoo--oriticaaloooo -ir wdw.. pdd
Yuxi Li, Csaba Szepesvari, and Dale Schuurmans. Learning exercise policies for american options. In David van Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference Artificial Intelligence and Staristics, volume 5 of Proceedings of Machine Learning Research, pp. 352-355 359, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
html

---

Under review as a conference paper at ICLR 2021
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional re- inforcement learning for efficient exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pp. 4424 4434, Long Beach, California, USA, 09-15 ngs ml wwtppp/sww.saaeeeeeeeeeee
Juun 2019. PMLR. URL htt..http///prooce
9a html
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland. Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http:://ax.....orr/111.. http////aauree42222
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. Proceedings of the Twenty- Sixth Conference Uncertainty in Artificial Intelligence, UAI' 10, pp. 368-375, Arlington, Vir- ginia, USA, 2010. AUAI Press. ISBN 9780974903965.
Marek Petrik and Dharmashankar Subramanian. An approximate solution method for large risk- markov decision processes. In Proceedings of the Twenty-Fighth Conference Uncer- tainty in Artificial Intelligence, 12 pp. 805-814, Arlington, Virginia, USA, 2012. AUAI Press. ISBN 9780974903989.
R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization conditional value-at-risk. Journal of Risk, 2:21-41, 2000.
Andrzej Ruszczynski. Risk-averse dynamic programming for markov decision processes. Math. Progrum., 125(2):235-261 October 2010. ISSN 0025-5610.
Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: evar opti-
mization approach. pp. 412 412-423, 01 2019. doi: 10.5200008815564412222
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL htta:://incompleeeeeeee htt.p//aw....tttooo//eeeeeebbeee 2nd.
html
A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Sequential decision making with coherent risk. IEEE Transactions on Automatic Control, 62(7):3323-3333 2017.
Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICMIL'12, pp. 1651-1658, Madison, WI, USA, 2012. Omnipress. ISBN
9781450312851.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling, In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, AAAI 15, pp. 2993-2992999 AAAI Press, 2015. ISBN 0262511299
Wang, Li, S. Shekhar, and W. Northrop. Uncertainty estimation with distributional rein- forcement learning for applications in intelligent transportation systems: A case study. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3822-3827, 2019.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279-292, May 1992. ISSN 1573-0565.
doi: 10. N0.1007/BF00992698, URL hatps:://cc...... /ttp://///11..001100101000)
BF BPC0992698
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie- Yan Liu. Fully pa- rameterized quantile function for distributional reinforcement learning. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gareett (eds.), Advances in Neural Information Processing Systems 32, pp. 6193-6202.
Curran Associates. Inc., 2019. URL http:://apeeee . www.apprr..iss.ccc/opaap 8850- 0850-fully- wwwllylparamtttrieeeeeeeeee seweriizzdquuunileeeee fonctiinnnnnnno forna--oor-ooo htg.::ae-.aaaataeeeeeooooooo----
learni
pdf
10

---

Under review as a conference paper at ICLR 2021
P. Yu, W. B. Haskell, and H. Xu. Dynamic programming for risk- sequential optimization. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pp. 4934 4939,4999900 2017.
Shangtong Zhang and Hengshuai Yao. QUOTA:
the quantile option architecture for reinforcement learning. In The Thirt AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty- First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pp. 5797-5804, 2019. doi: (0.1169//aiiiv333000 2300./33....110...311177..7 URL https: /doi www. /0. 609 aaai v33101 330015997


Reviews:
[{"review_id": "yXbo1LMuSmm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper consider the problem of learning a risk-averse policy base on CVaR measure using distributional reinforcement learning. The main contributions of this paper are twofold. First, they show that the standard distributional RL algorithm overestimate the dynamic, Markovian CVaR, which might be too conservative. Secondly, they propose a modified algorithm that can learn a proper CVaR-optimized policy based on static, non-Markovian CVaR. \n\nOverall this paper is well-written and easy to follow. The problem is well-motivated, and the proofs of the main propositions are clean and easy to check. \n\nHowever, I have two main concerns on this paper. First, the theory part of this paper (proposition 1 & 2) are quite straight forward, and the modifications to the existing algorithm in [1] are mild, thus the novelty of this work is somewhat limited. Second, the option trading experiment train on a mixture of real stock prices and simulated stock prices (the authors use simulated data to allow training on unlimited data), and we don’t know the exact size of training set, this seems a bit wired to me. Is there any sampling complexity guarantee for the proposed algorithm? How does the performance of the algorithm scale with the training sample size？\n\n[1]  Will Dabney, Mark Rowland, Marc G. Bellemare, and R´emi Munos. Distributional reinforcement learning with quantile regression.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "q1IBUUxuS0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First I wanted to thank authors for putting the manuscript together, I enjoyed reading it. \n\nSummary : Authors proposed using DRL to find a good (optimal) CVaR policy, by pointing that the optimal CVaR policy is non-stationary, and DRL can be leveraged to learn and execute this kind of policies. In addition they showed just taking a max over CVaR rather than E will not result in an optimal policy. \n\nStrength : \n1. Paper is well written, it's easy to follow and it provides necessary background for the reader to follow. \n2. Important Issue : I believe finding a scalable way to optimize for CVaR is an important problem to tackle, as most of the previous work are not scalable (e.g. Chow et al)\n\nWeakness/ Concerns:\n1. Convergence: I have a concern about the convergence of the algorithm, by applying the proposed algorithm, is there any guarantee for convergence? Even in the case of policy evaluation and not control (not taking max, picking action on \\pi) is there a convergence guarantee? Or if not can authors provide intuition/ reason why is that the case?\n\n2. Novelty: Reading the paper, I challenge the novelty of current algorithm/ manuscript. Or maybe I had a hard time pinpointing it, to the best of my knowledge most of the claims have been already known, can authors please explain what they think their main contribution is? (I'm happy to change my score given the explanation)\n\n3. Experiments: The premise of using DRL for CVaR is mainly \"scalability\" so that we can solve larger problems (state space mainly). However, this is not reflected in the experiments, and experiments are in small domains. I think the paper can benefit from an experiment in larger state space. \n\nScore: \nAt this point I think the manuscript is not ready for a publication, however, I did enjoy reading it, and I think it's a great work so far with potentials. I am happy to change my score given authors response to my concerns. \n\nThanks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fpJ9pXJFiKQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about risk-sensitive RL based on the CVaR risk measure. This paper is mainly based on the work presented in Dabney et al. in 2018 which is about distributional RL for a family of risk measures which includes CVaR as well. The main motivation for this work was the point that the method presented in Dabney et al. 2018 overestimates the dynamics and could be excessively conservative in certain scenarios. Authors have proposed to use static CVaR instead and have developed algorithms to do that.\n\nThis paper has solid theoretical results (propositions 1 and 2). Authors have identified a problem in Dabney et al. and proposed an algorithm to resolve it.\n\nThe major issue though, is about the evaluation and experimental results. authors have provided results on a synthetic dataset and a real dataset related to options trading. Even on the real data results, for larger values of alpha, Dabney et al. 2018 has outperformed the proposed approach. In order to make this paper ready for a venue such as ICLR, authors should provide a more comprehensive evaluation of their methods. At least, it is expected to show the performance of their approach and its comparison vs Dabney et al. on several Atari games. Otherwise, the contribution of this paper would be limited.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VJ1fsYrqONl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of learning the optimal policy for static CVaR with distributional RL.\nThe authors underline the difference between static CVaR which is the CVaR on the whole cumulative discounted rewards and the dynamic CVaR which is a Markovian alternative proposed by (Ruszczynski 2010).\nAs pointed out by the authors, the optimal policy for static CVaR must take into account the accumulated rewards to take decisions: the optimal policy is hence not stationary.  A nice property of dynamic CVaR is that its optimal policy is stationary.\nA key result of the paper is to show that optimal solution for the dynamic CVaR is suboptimal for static CVaR (Proposition 1).\nLeveraging on this remark, the authors propose an algorithm based on distributional (quantile regression) to solve the static CVaR problem. The obtained policy is stationary on an augmented MDP where the states are decorated with he reward collected so-far.\nThe experiments on synthetic and real data underline the relevance of the proposed approach.\n\nI really enjoyed reading this theoretical paper. It is very well written and easy to follow. The ideas presented are interesting.\nMy only criticisms 'or questions' are about the state augmentation:\n1. As mentioned on page 4 it would be sample inefficient but one could solve the augmented MDP directly. Why not provide this approach in your experiments ?\n2. As mentioned on page 5 you are using the distribution estimates to \"store the information needed\", instead of using state augmentation. But estimating a distribution is a costly task. Do you have any argument to justify that this approach is indeed more \"sample efficient\" than a naive state augmentation version without distributional estimates ? Any convergence bound ?\n\nMinor remark:\nI would be nice to have at least a small conclusion or perspective\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found found the paper well motivated and well written, they found both the theoretical contributions limited in novelty and the experiments too rudimentary to be insightful.
Decision: Reject